Remove all listeners on a channel departure