Don't call EndListening() on already destructed listeners.

Change-Id: I9bda35f2246de9d37077dda33c710b89ee008e5a