Remove all references to a listener (leak fix)

When we remove a listener, we need to remove *all* its references. One
was missing in BaseCastManager.

This had the side-effect to call several times the listener callbacks
when the listener was removed then re-added.