Avoids silly concurrent modification exception

* Avoid removing the listener whilst being called during an iteration of
  the listener collection.