Fix bug when removing non existent listeners