Remove PeerInfo from cache on dtor.