Only unsubscribe from channels if subscribed