Actually remove user from channel when he leaves.