Remove users from our cache if they leave all of our channels.