Remove entries with an empty registry

For Component-based listeners, check if the registry is empty after
removing a listener. If it is, remove the registry from the map.