Clean the peer dirty flag also when switching to a channel.