Stop checking if the peer is there once we remove it from the network.