Make the PeerInfo cache contain weak pointers.
Remove PeerInfo from cache on dtor.