Make the PeerInfo cache contain weak pointers.