Use a more-correct logic for removing listeners

Avoid leaking listeners.