Handle user join/part/quits more efficiently