Properly check setting and unsetting of modes if they're channel statuses