Remove irrelevant IRC listeners