Close the underlying Listener when shutting down