Package: binary-trie
Binary TriePython binary trie implementation, helping with XOR distances.Author:Guillaume MichelPurposeIPFSandlibp2pare built upon the Kademlia DHT, which uses the XOR distance as a distance metric between keys. As explained in thisblogpost, the XOR distance is tricky to understand and represent. This distance metric is odd for it is non linear.
For instance, $2 \oplus 3 = 1$ but $3 \oplus 4 = 7$. Ordering $N_8$ by XOR distance to 2 gives the following result: $[2,3,0,1,6,7,4,5]$One popular representation for XOR distance is the Binary Trie. A binary brie is a simpleTriewith keyspace $\lbrace0,1\rbrace^n$ with $n$ being the size of the keyspace. The perfect visual representation of the XOR distance would be a N-dimensional binary trie. The example below displays a binary trie containing the keys $[2,3,4,6,7,9,11,13]$.Each node in the Trie can be associated with some metadata for convenience.UsageInstallpipinstallbinary-trieImportfrombinary_trieimportTrie,bytes_to_bitstring,bitstring_to_bytes,int_to_bitstring,bitstring_to_intCreating an empty trietrie=Trie()Adding keysTheadd(key)method takes a bitstring as input. The bitstring can either be provided directly, or be computer from anintorbytesusing theint_to_bitstring()andbytes_to_bitstring()functions. Note that thelparameter represent the bit length of the binary representation. It is important that all keys share the same bit length. The bit length can be omitted inbytes_to_bistring()when working with bit lengths multiple of 8.trie.add("0010")trie.add(4*"0")trie.add(int_to_bitstring(3,l=4))trie.add(bytes_to_bitstring(b'\x0e',l=4))The add function returnsTrueon success, andFalseif the key was already in the trie (not inserted).Finding keysThefind(key)method returnsTrueif the provided key is in the Trie,Falseotherwise.trie.contains("0010")# Truetrie.contains("0100")# FalseKey depthThedepth(key)method returns the depth of the provided key, if it is in the Trie and-1otherwise. The depth of a trie node is defined as the number of its direct ancestors, up to the root of the trie.trie.depth("0")# 1trie.depth("0010")# 3trie.depth("0111")# 4trie.depth("1101")# 2trie.depth("11")# -1Finding the closest keys to a targetThen_closest_keys(key, n)method returns thenclosest keys to the provided key in the trie, according to the XOR distance. The keys are sorted according to the distance to the target key. Note that only leaves of the trie will be returned, not intermediary nodes.trie.n_closest_keys("0001",1)# ["0000"]trie.n_closest_keys("0010",3)# ["0010", "0011", "0000"]Prefix matchingTheprefix_match_keys(prefix)will return the list of keys of all leaves of the Trie matching the providedprefix.trie.prefix_match_keys("00")# ["0000", "0010", "0011"]trie.prefix_match_keys("1111")# []Attaching metadata to Trie nodesIt is possible to attach anobjectas metadata to a trie leaf node.classMyObj(object):def__init__(self,key,name):self.key=keyself.name=nametrie=Trie(MyObj)obj=MyObj(int_to_bitstring(10,4),"Node 10")trie.add(obj.key,obj)trie.find(obj.key).name# "Node 10"trie.n_closest(obj.key,1)# [obj]trie.prefix_match(obj.key[:2])# [obj]Note that thefind(key)method is similar to thecontains(key)method, but returns the associatedmetadataif any, instead of returning abool.
Then_closest(key, n)method is similar to then_closest_keys(key, n)method, but returns the list ofmetadataassociated with the closest keys, instead of the list of keys.PredicatesIt is possible to assign some boolean variables tometadataobjects, and make use of them using predicate inn_closest(),n_closest_keys(),prefix_match()andprefix_match_keys()methods to filter the results.classMyObj(object):def__init__(self,key,name,some_bool):self.key=keyself.name=nameself.some_bool=some_booltrie=Trie(MyObj)nodeIDs=[0,1,2]# ["0000", "0001", "0010"]foriinnodeIDs:obj=MyObj(int_to_bitstring(i,4),"Node "+str(i),i%2==0)trie.add(obj.key,obj)trie.n_closest_keys("0001",2,predicate=lambdan:n.some_bool)# ["0000", "0010"]trie.prefix_match_keys("000",predicate=lambdan:n.some_bool)# ["0000"]# Note that the key "0001" matched both requests, but wasn't taken into# consideration as it doesn't satisfy the predicateHelpersThere are 4 helpers functions to facilitate the use of this implementation with keys being not onlybitstring, but alsobytesorint. These helper functions help translatebytesandinttobitstringand reciprocally.defbytes_to_bitstring(data:bytes,l:int=8*len(data))->bytes:...bytes_to_bitstring(b'\xff\x00')# "1111111100000000"bytes_to_bitstring(b'\xf3',l=4)# "0011"defbitstring_to_bytes(bs:str)->bytes:...bitstring_to_bytes("1111111100000000")# b'\xff\x00'bitstring_to_bytes("0011")# b'\x03'defint_to_bitstring(i,l:int)->bytes:...int_to_bitstring(6,4)# "0110"int_to_bitstring(6,16)# "0000000000000110"defbitstring_to_int(bs:str)->int:...bitstring_to_int("0110")# 6bitstring_to_int("0000000000000110")# 6EncodingThe following functions help uniquely encode a specific binary prefix, that are bistrings of arbitrary size. The bitstrings can be encoded inintor inunsigned varint. The mapping goes as follow:bitstring       code        varint
""          ->  0       ->  00000000 (0x00)
"0"         ->  1       ->  00000001 (0x01)
"1"         ->  2       ->  00000010 (0x02)
"00"        ->  3       ->  00000011 (0x03)
...
"0000000"   ->  127     ->  01111111 (0x7f)
"0000001"   ->  128     ->  10000000 00000001 (0x8001)
...defencode_bitstring(bitstring:str)->int:...encode_bitstring("1")# 2encode_bitstring("010")# 9encode_bitstring("0000001")# 128defdecode_bitstring(code:int)->str:...decode_bitstring(2)# "1"decode_bitstring(9)# "010"decode_bitstring(128)# "0000001"defbitstring_to_varint(bitstring:str)->bytes:...bitstring_to_varint("1")# b'\x02'bitstring_to_varint("010")# b'\x09'bitstring_to_varint("0000001")# b'\x80\x01'defvarint_to_bitstring(varint_bytes:bytes)->str:...varint_to_bitstring(b'\x02')# "1"varint_to_bitstring(b'\x09')# "010"varint_to_bitstring(b'\x80\x01')# "0000001"