Delete the object if there are no more listeners

This will prevent a memory leak in some cases.