Clean up listener list on disconnect.