Fix interrupt thread when listener is removed