Remove abandoned listeners