Only do user removal from a channel when they drop their connection if they were in the channel