Package: hirola
NumPy vectorized hash table written in C for fast (roughly 10x faster)set/dictlike operations.Free software:MIT licenseDocumentation:https://hirola.readthedocs.io/Source code:https://github.com/bwoodsend/hirola/Releases:https://pypi.org/project/hirola/Ahirola.HashTableis todictwhatnumpy.arrayis tolist.
By imposing some constraints, vectorising, and translating into C, the speed
can be improved dramatically.
For hirola, these constraints are:Keys must all be of the same predetermined type and size.The maximum size of a table must be chosen in advance and managed explicitly.To get any performance boost, operations should be done in bulk.Elements can not be removed.If any of the above are not satisfied for your use case then don’t use
hirola.InstallationInstall Hirola with pip:pip install hirolaQuickstartHashTableAt the rawest and dirtiest level lives theHashTableclass.
AHashTablecan be though of as adictbut with only an enumeration for
values.
To construct an empty hash table:importnumpyasnpfromhirolaimportHashTabletable=HashTable(20,# <--- Maximum size for the table - up to 20 keys."U10",# <--- NumPy dtype - strings of up to 10 characters.)Keys may be added individually…>>>table.add("cat")0… But it’s much more efficient to add in bulk.
The return value is an enumeration of when each key was first added.
Duplicate keys are not re-added.>>>table.add(["dog","cat","moose","gruffalo"])array([1,0,2,3])Multidimensional inputs give multidimensional outputs of matching shapes.>>>table.add([["rabbit","cat"],...["gruffalo","moose"],...["werewolf","gremlin"]])array([[4,0],[3,2],[5,6]])Inspect all keys added so far via thekeysattribute.
(Note that, unlikedict.keys(), it’s a property instead of a method.)>>>table.keysarray(['cat','dog','moose','gruffalo','rabbit','werewolf','gremlin'],dtype='<U10')Key indices can be retrieved withtable.get(key)or justtable[key].
Again, retrieval is NumPy vectorised and is much faster if given large arrays of
inputs rather than one at a time.>>>table.get("dog")1>>>table[["moose","gruffalo"]]array([2,3])Like the Python dict,
usingtable[key]raises aKeyErrorif keys are missing
but usingtable.get(key)returns a configurable default.
Unlike Python’s dict, the default is-1.>>>table["tortoise"]KeyError:"key = 'tortoise' is not in this table.">>>table.get("tortoise")-1>>>table.get("tortoise",default=99)99>>>table.get(["cat","bear","tortoise"],default=[100,101,102])array([0,101,102])Choosing amaxsizeUnlike Python’ssetanddict,Hiroladoes not manage its size
automatically by default
(althoughit can be reconfigured to).
To prevent wasted resizing (which is what Python does under the hood),
you have full control of and responsibility for how much space the table uses.
Obviously the table has to be large enough to fit all the keys in it.
Additionally, when a hash table gets to close to full it becomes much slower.
Depending on how much you favour speed over memory you should add 20-50% extra
headroom.
If you intend to a lot of looking up of the same small set of values then it can
continue to run faster if you increasemaxto 2-3x its minimal size.Structured key data typesTo indicate that an array axis should be considered as a single key,
use NumPy’s structured dtypes.
In the following example, the data type(points.dtype, 3)indicates that a 3D point - a triplet of floats -
should be considered as one object.
Seehelp(HashTable.dtype)for more information of specifying dtypes.
Only the last axis or last axes may be thought of as single keys.
For other setups, first convert withnumpy.transpose().importnumpyasnpfromhirolaimportHashTable# Create a cloud of 3D points with duplicates. This is 3000 points in total,# with up to 1000 unique points.points=np.random.uniform(-30,30,(1000,3))[np.random.choice(1000,3000)]# Create an empty hash table.# In practice, you generally don't know how many unique elements there are# so we'll pretend we don't either an assume the worst case of all 3000 are# unique. We'll also give 25% padding for speed.table=HashTable(len(points)*1.25,(points.dtype,3))# Add all points to the table.ids=table.add(points)Duplicate-free contents can be accessed fromtable.keys:>>>table.keys# <--- These are `points` but with no duplicates.array([[3.47736554,-15.17112511,-9.51454466],[-6.46948046,23.64504329,-16.25743105],[-27.02527253,-16.1967225,-10.11544157],...,[3.75972597,1.24130412,-8.14337206],[-13.62256791,11.76551455,-13.31312988],[0.19851678,4.06221179,-22.69006592]])>>>table.keys.shape(954,3)Each point’s location intable.keysis returned bytable.add(),
similarly tonumpy.unique(...,return_args=True).>>>ids# <--- These are the indices in `table.keys` of each point in `points`.array([0,1,2,...,290,242,669])>>>np.array_equal(table.keys[ids],points)TrueLookup the indices of points without adding them usingtable.get().Handling of nearly full hash tablesHashTables become very slow when almost full.
As of v0.3.0, an efficiency warning will notify you if a table exceeds 90% full.
This warning can be reconfigured into an error, silenced or set to resize the
table automatically to make room.
These are demonstrated in the example constructors below:# The default: Issue a warning when the table is 90% full.HashTable(...,almost_full=(0.9,"warn"))# Disable all "almost full" behaviours.HashTable(...,almost_full=None)# To consider a table exceeding 80% full as an error use:HashTable(...,almost_full=(0.8,"raise"))# To automatically triple in size whenever the table exceeds 80% full use:HashTable(...,almost_full=(0.8,3.0))Resizing tables is slow which is why it’s not enabled by default.
It should be avoided unless you really have no idea how big your table will need
to be.RecipesAHashTablecan be used to replicate adict,setor acollections.Counter.
These might turn into their own proper classes in the future or they might not.Using aHashTableas adictAdictrequires a second array for values.
The output ofHashTable.add()andHashTable.get()should be used as
indices ofvalues:importnumpyasnpfromhirolaimportHashTable# The `keys` - will be populated with names of African countries.countries=HashTable(40,(str,20))# The `values` - will be populated with the names of each country's capital city.capitals=np.empty(countries.max,(str,20))Add or set items using the patternvalues[table.add(key)] = value:capitals[countries.add("Algeria")]="Al Jaza'ir"Or in bulk:new_keys=["Angola","Botswana","Burkina Faso"]new_values=["Luanda","Gaborone","Ouagadougou"]capitals[countries.add(new_keys)]=new_valuesLike Python dicts, overwriting values is exactly the same as writing them.Retrieve values withvalues[table[key]]:>>>capitals[countries["Botswana"]]'Gaborone'>>>capitals[countries["Botswana","Algeria"]]array(['Gaborone',"Al Jaza'ir"],dtype='<U20')View all keys and values withtable.keysandvalues[:len(table)].
AHashTableremembers the order keys were first added so this dict is
automatically a sorted dict.# keys>>>countries.keysarray(['Algeria','Angola','Botswana','Burkina Faso'],dtype='<U20')# values>>>capitals[:len(countries)]array(["Al Jaza'ir",'Luanda','Gaborone','Ouagadougou'],dtype='<U20')Depending on the usage scenario,
it may or may not make sense to want an equivalent todict.items().
If you do want an equivalent,
usenumpy.rec.fromarrays([table.keys,values[:len(table)]]),
possibly adding anames=option:>>>np.rec.fromarrays([countries.keys,capitals[:len(countries)]],...names="countries,capitals")rec.array([('Algeria',"Al Jaza'ir"),('Angola','Luanda'),('Botswana','Gaborone'),('Burkina Faso','Ouagadougou')],dtype=[('countries','<U20'),('capitals','<U20')])If the keys and values have the same dtype thennumpy.c_works too.>>>np.c_[countries.keys,capitals[:len(countries)]]array([['Algeria',"Al Jaza'ir"],['Angola','Luanda'],['Botswana','Gaborone'],['Burkina Faso','Ouagadougou']],dtype='<U20')Using aHashTableas asetTo get set-like capabilities from aHashTable,
leverage thecontains()method.
For these examples we will experiment with integer multiples of 3 and 7.importnumpyasnpof_3s=np.arange(0,100,3)of_7s=np.arange(0,100,7)We’ll only require one array to be converted into a hash table.
The other can remain as an array.
If both are hash tables, simply use one table’skeysattribute as the array.fromhirolaimportHashTabletable_of_3s=HashTable(len(of_3s)*1.25,of_3s.dtype)table_of_3s.add(of_3s)Usetable.contains()as a vectorised version ofin.>>>table_of_3s.contains(of_7s)array([True,False,False,True,False,False,True,False,False,True,False,False,True,False,False])From the above, the common set operations can be derived with following:set.intersection()- Values in the array and in the set:>>>of_7s[table_of_3s.contains(of_7s)]array([0,21,42,63,84])Set subtraction - Values in the array which are not in the set:>>>of_7s[~table_of_3s.contains(of_7s)]array([7,14,28,35,49,56,70,77,91,98])set.union()- Values in either the table or in the tested array (with no
duplicates):>>>np.concatenate([table_of_3s.keys,of_7s[~table_of_3s.contains(of_7s)]],axis=0)array([0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75,78,81,84,87,90,93,96,99,7,14,28,35,49,56,70,77,91,98])Using aHashTableas acollections.CounterFor this example,
let’s give ourselves something a bit more substantial to work on.
Counting word frequencies in Shakespeare’s Hamlet play is the
trendy example forcollections.Counterand it’s what we’ll use too.fromurllib.requestimporturlopenimportreimportnumpyasnphamlet=urlopen("https://gist.githubusercontent.com/provpup/2fc41686eab7400b796b/raw/b575bd01a58494dfddc1d6429ef0167e709abf9b/hamlet.txt").read()words=np.array(re.findall(rb"([\w']+)",hamlet))A counter is just adictwith integer values and adictis just a hash
table with a separate array for values.fromhirolaimportHashTableword_table=HashTable(len(words),words.dtype)counts=np.zeros(word_table.max,dtype=int)The only new functionality that is not defined inusing a hash table as a dictis the ability to count keys as they are added.
To count new elements use the rather odd linenp.add(counts, table.add(keys), 1).np.add.at(counts,word_table.add(words),1)This line does what you might expectcounts[word_table.add(words)] += 1to
do but, due to the way NumPy works,
the latter form fails to increment each count more than once ifwordscontains duplicates.Use NumPy’s indirect sorting functions to get most or least common keys.# Get the most common word.>>>word_table.keys[counts[:len(word_table)].argmax()]b'the'# Get the top 10 most common words. Note that these are unsorted.>>>word_table.keys[counts[:len(word_table)].argpartition(-10)[-10:]]array([b'it',b'and',b'my',b'of',b'in',b'a',b'to',b'the',b'I',b'you'],dtype='|S14')# Get all words in ascending order of commonness.>>>word_table.keys[counts[:len(word_table)].argsort()]array([b'END',b'whereat',b"griev'd",...,b'to',b'and',b'the'],dtype='|S14')A Security NoteUnlike the builtinhash()used internally by Python’ssetanddict,hiroladoes not randomise a hash seed on startup
making an online server runninghirolamore vulnerable to denial of service
attacks.
In such an attack, the attacker clogs up your server by sending it requests that
he/she knows will cause hash collisions and therefore slow it down.
Whereas a Python hash table’s size is always predictably the next power of 8
abovelen(table) * 3 / 2, ahirola.HashTable()may be any size meaning
that you can make an attack considerably more difficult by adding a little
randomness to the sizes of your hash tables.
But if your writing an online server
which performs dictionary lookup based on user input
and your user-base doesn’t like you much
or you have some very spiteful below-the-belt competitors
then I recommend that you don’t use this library.CreditsThis package was initially created withCookiecutterand a fork of theaudreyr/cookiecutter-pypackageproject template.