My understanding is this. At one extreme, say we use only one replica, the distribution of the peers may not even at all, which causes skewed hash results, i.e., some peers become hotposts. At the other extreme, say we use a lot of replicas, we may end up with a lot of small buckets, and adding/removing a peer may cause many keys to hash to different values (like you mentioned). So this number needs to be a good balance between these two scenarios.