Package: uhashring
uhashringimplementsconsistent hashingin pure Python.Consistent hashing is mostly used on distributed systems/caches/databases as this avoid the total reshuffling of your key-node mappings when adding or removing a node in your ring (called continuum on libketama). More information and details about this can be found in theliteraturesection.This full featured implementation offers:a lot ofconvenient methodsto use your consistent hash ring in real world applications.simpleintegrationwith other libs such as memcache through monkey patching.a fullketamacompatibility if you need to use it (see important mention below).all the missing functions in the libketama C python binding (which is not even available on pypi) for ketama users.possibility touse your own weight and hash functionsif you don’t care about the ketama compatibility.instance-oriented usageso you can use your consistent hash ring object directly in your code (see advanced usage).nativepypy support, since this is a pure python library.tests of implementation, key distribution and ketama compatibility.Per node weight is also supported and will affect the nodes distribution on the ring.Python 2 EOLIf you need Python 2 support, make sure to useuhashring==1.2as v1.2 is the last release that will support it.IMPORTANTSince v1.0uhashringdefault has changed to use a md5 hash function with 160 vnodes (points) per node in the ring.This change was motivated by the fact that the ketama hash function has more chances of collisions and thus requires a complete ring regeneration when the nodes topology change. This could lead to degraded performances on rapidly changing or unstable environments where nodes keep going down and up. The md5 implementation provides a linear performance when adding or removing a node from the ring!Reminder: when usinguhashringwith the ketama implementation and get 40 vnodes and 4 replicas = 160 points per node in the ring.UsageBasic usageuhashringis very simple and efficient to use:fromuhashringimportHashRing# create a consistent hash ring of 3 nodes of weight 1hr=HashRing(nodes=['node1','node2','node3'])# get the node name for the 'coconut' keytarget_node=hr.get_node('coconut')Ketama usageSimply set thehash_fnparameter toketama:fromuhashringimportHashRing# create a consistent hash ring of 3 nodes of weight 1hr=HashRing(nodes=['node1','node2','node3'],hash_fn='ketama')# get the node name for the 'coconut' keytarget_node=hr.get_node('coconut')Advanced usagefromuhashringimportHashRing# Mapping of dict configs# Ommited config keys will get a default value, so# you only need to worry about the ones you neednodes={'node1':{'hostname':'node1.fqdn','instance':redis.StrictRedis(host='node1.fqdn'),'port':6379,'vnodes':40,'weight':1},'node2':{'hostname':'node2.fqdn','instance':redis.StrictRedis(host='node2.fqdn'),'port':6379,'vnodes':40},'node3':{'hostname':'node3.fqdn','instance':redis.StrictRedis(host='node3.fqdn'),'port':6379}}# create a new consistent hash ring with the nodeshr=HashRing(nodes)# set the 'coconut' key/value on the right host's redis instancehr['coconut'].set('coconut','my_value')# get the 'coconut' key from the right host's redis instancehr['coconut'].get('coconut')# delete the 'coconut' key on the right host's redis instancehr['coconut'].delete('coconut')# get the node config for the 'coconut' keyconf=hr.get('coconut')Default node configurationuhashringoffers advanced node configuration for real applications, this is the default you get for every added node:{'hostname':nodename,'instance':None,'port':None,'vnodes':40,'weight':1}Adding / removing nodesYou can add and remove nodes from your consistent hash ring at any time.fromuhashringimportHashRing# this is a 3 nodes consistent hash ringhr=HashRing(nodes=['node1','node2','node3'])# this becomes a 2 nodes consistent hash ringhr.remove_node('node2')# add back node2hr.add_node('node2')# add node4 with a weight of 10hr.add_node('node4',{'weight':10})Customizable node weight calculationfromuhashringimportHashRingdefweight_fn(**conf):"""Returns the last digit of the node name as its weight.

    :param conf: node configuration in the ring, example:
        {
         'hostname': 'node3',
         'instance': None,
         'nodename': 'node3',
         'port': None,
         'vnodes': 40,
         'weight': 1
        }
    """returnint(conf['nodename'][-1])# this is a 3 nodes consistent hash ring with user defined weight functionhr=HashRing(nodes=['node1','node2','node3'],weight_fn=weight_fn)# distribution with custom weight assignmentprint(hr.distribution)# >>> Counter({'node3': 240, 'node2': 160, 'node1': 80})Customizable hash functionfromuhashringimportHashRing# import your own hash function (must be a callable)# in this example, MurmurHash v3frommmh3importhashasm3h# this is a 3 nodes consistent hash ring with user defined hash functionhr=HashRing(nodes=['node1','node2','node3'],hash_fn=m3h)# now all lookup operations will use the m3h hash functionprint(hr.get_node('my key hashed by your function'))HashRing optionsnodes: nodes used to create the continuum (see doc for format).hash_fn: use this callable function to hash keys, can be set to ‘ketama’ to use the ketama compatible implementation.vnodes: default number of vnodes per node.weight_fn: user provided function to calculate the node’s weight, gets the node conf dict as kwargs.replicas: use this to change ketama ring replicas (default: 4)Available methodsadd_node(nodename, conf): add (or overwrite) the node in the ring with the given config.get(key): returns the node object dict matching the hashed key.get_key(key): alias of the current hashi method, returns the hash of the given key.get_instances(): returns a list of the instances of all the configured nodes.get_node(key): returns the node name of the node matching the hashed key.get_node_hostname(key): returns the hostname of the node matching the hashed key.get_node_instance(key): returns the instance of the node matching the hashed key.get_node_port(key): returns the port of the node matching the hashed key.get_node_pos(key): returns the index position of the node matching the hashed key.get_node_weight(key): returns the weight of the node matching the hashed key.get_nodes(): returns a list of the names of all the configured nodes.get_points(): returns a ketama compatible list of (position, nodename) tuples.get_server(key): returns a ketama compatible (position, nodename) tuple.hashi(key): returns the hash of the given key (on ketama mode, this is the same as libketama).iterate_nodes(key, distinct): hash_ring compatibility implementation, same as range but returns tuples as a generator.print_continuum(): prints a ketama compatible continuum report.range(key, size, unique): returns a (unique) list of max (size) nodes’ configuration available in the consistent hash ring.regenerate: regenerate the ring from the current nodes configuration, useful only when usingweight_fn.remove_node(nodename): remove the given node from the ringAvailable propertiesconf: dict of all the nodes and their configuration.continuum: same as ring.distribution: counter of the nodes distribution in the consistent hash ring.nodes: same as conf.ring: hash key/node mapping of the consistent hash ring.size: size of the consistent hash ring.Integration (monkey patching)You can benefit from a consistent hash ring usinguhashringmonkey patching on the following libraries:python-memcachedimportmemcachefromuhashringimportmonkeymonkey.patch_memcache()mc=memcache.Client(['node1:11211','node2:11211'])InstallationPypiUsing pip:$pipinstalluhashringGentoo LinuxUsing emerge:$sudoemerge-auhashringBenchmarkUsage of the ketama compatible hash (default) has some performance impacts.
Contributions are welcome as to ways of improving this !There is a big performance gap in the hash calculation between
the ketama C binding and its pure python counterpart.Python 3 is doing way better than python 2 thanks to its
native bytes/int representation.Quick benchmark, for 1 million generated ketama compatible keys:python_ketama C binding: 0.8427069187164307 spython 2: 5.462762832641602 spython 3: 3.570068597793579 spypy: 1.6146340370178223 sWhen using python 2 and ketama compatibility is not important, you
can get a better hashing speed using the other provided hashing.hr = HashRing(nodes=[], compat=False)Quick benchmark, for 1 million generated hash keys:python 2: 3.7595579624176025 spython 3: 3.268343687057495 spypy: 1.9193649291992188 sLiteratureconsistent hashing:https://en.wikipedia.org/wiki/Consistent_hashingweb caching paper:http://www8.org/w8-papers/2a-webserver/caching/paper2.htmlresearch paper:http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.23.3738distributed hash table:https://en.wikipedia.org/wiki/Distributed_hash_tableLicenseBSD