[fix] Remove only our listeners

We should only remove the listeners that we added, or it is possible
that we remove a listener that is used by the underlying real-time
framework to do internal cleanup.