Fix cleanup when last listener removed