Fix desync when empty permanent channels are dropped on ircds that no have permanent channel mode