Fix crash trying to unset the permanent channel mode during channel syncs of empty channels