Remove a channel if it doesn't exist

This speeds up startup a little if there was an invalid channel still in
the list.