Fix potential out of sync error and destroy the channel if nobody is in it