Fix EE issue

When listeners are explicitly removed with removeListener, the registration with the source zone isn't properly removed.

@kraman This is a bug fix against the "old" implementation. The fast implementation has the same issue but I couldn't figure out how to port the fix - do you have any suggestions?