graphite-project/carbon — pull request #30 by slackhappy (2012-08-21 03:54:17+00:00)
Title: improve CH routing performance when replfactor=1

I noticed when processing metrics at a rate of 300K-400K per minute per relay per minute, using Consistent Hashing with a replication factor of 1, get_nodes kept showing up as a hot spot.

I think this is because it the ch algorithm exhausts the possible slots of a given metric, regardless of how many replicas are needed. If the common case is 1 or 2 replicas, but the total number of shards is higher, this leads to extra cycles.

My test cluster has 6 cache nodes (2 per machine).

If I throw a println "hi" at the top of the while loop, here's what I see:

import carbon.hashing
ch = carbon.hashing.ConsistentHashRing([])
ch.add_node(('graphite-test-1', 'cache0'))
ch.add_node(('graphite-test-1', 'cache1'))
ch.add_node(('graphite-test-2', 'cache0'))
ch.add_node(('graphite-test-2', 'cache1'))
ch.add_node(('graphite-test-3', 'cache0'))
ch.add_node(('graphite-test-3', 'cache1'))

[x for x in ch.get_nodes('api.fleet.local.machines')]
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi

I think that's 13 "hi's".

This change to use a generator causes the loop to only execute once if only one replica is needed (a. While I have additionally split the load out to several processes, I've found that this change has helped improve the throughput quite a bit (sorry - no hard numbers at the moment, but I can get them).