Remove remote users from channels when they're disconnecting