Package: diskhash
Disk-based hashtableA simple disk-based hash table (i.e., persistent hash table).It is a hashtable implemented on memory-mapped disk, so that it can be loaded
with a singlemmap()system call and used in memory directly (being as fast
as an in-memory hashtable once it is loaded from disk).The code is in C, wrappers are provided for Python, Haskell, and C++. The
wrappers follow similar APIs with variations to accommodate the language
specificity. They all use the same underlying code, so you can open a hashtable
created in C from Haskell, modify it within your Haskell code, and later open
the result in Python.Cross-language functionality will only work for simple types where you can
control their binary representation (64-bit integers, for example).Reading does not touch the disk representation at all and, thus, can be done on
top of read-only files or using multiple threads (and different processes will
share the memory: the operating system does that for you). Writing or modifying
values is, however, not thread-safe.ExamplesThe following examples all create a hashtable to store longs (int64_t), then
set the value associated with the key"key"to 9. In the current API, the
maximum size of the keys needs to be pre-specified, which is the value15below.Raw C#include<stdio.h>#include<inttypes.h>#include"diskhash.h"intmain(void){HashTableOptsopts;opts.key_maxlen=15;opts.object_datalen=sizeof(int64_t);char*err=NULL;HashTable*ht=dht_open("testing.dht",opts,O_RDWR|O_CREAT,&err);if(!ht){if(!err)err="Unknown error";fprintf(stderr,"Failed opening hash table: %s.\n",err);return1;}longi=9;dht_insert(ht,"key",&i);long*val=(long*)dht_lookup(ht,"key");printf("Looked up value: %l\n",*val);dht_free(ht);return0;}The C API relies on error codes and error strings (the&errargument above).
The header file hasdecent
documentation.HaskellIn Haskell, you have different types/functions for read-write and read-only
hashtables. Read-write operations areIOoperations, read-only hashtables are
pure.Read write example:importData.DiskHashimportData.Intmain=doht<-htOpenRW"testing.dht"15htInsertRWht"key"(9::Int64)val<-htLookupRW"key"htprintvalRead only example (htLookupROis pure in this case):importData.DiskHashimportData.Intmain=doht<-htOpenRO"testing.dht"15letval::Int64val=htLookupRO"key"htprintvalPythonPython's interface is based on thestruct
module. For example,'ll'refers to a pair of 64-bit ints (longs):importdiskhashtb=diskhash.StructHash("testing.dht",15,'ll','rw')tb.insert("key",1,2)print(tb.lookup("key"))The Python interface is currently Python 3 only. Patches to extend it to 2.7
are welcome, but it's not a priority.C++In C++, a simple wrapper is defined, which provides a modicum of type-safety.
You use theDiskHash<T>template. Additionally, errors are reported through
exceptions (bothstd::bad_allocandstd::runtime_errorcan be thrown) and
not return codes.#include<iostream>#include<string>#include<diskhash.hpp>intmain(){constintkey_maxlen=15;dht::DiskHash<uint64_t>ht("testing.dht",key_maxlen,dht::DHOpenRW);std::stringline;uint64_tix=0;while(std::getline(std::cine,line)){if(line.length()>key_maxlen){std::cerr<<"Key too long: '"<<line<<"'. Aborting.\n";return2;}constboolinserted=ht.insert(line.c_str(),ix);if(!inserted){std::cerr<<"Found repeated key '"<<line<<"' (ignored).\n";}++ix;}return0;}StabilityThis isbetasoftware. It is good enough that I am using it, but the API can
change in the future with little warning. The binary format is versioned (the
magic string encodes its version, so changes can be detected and you will get
an error message in the future rather than some silent misbehaviour.Automated unit testingensures that
basic mistakes will not go uncaught.LimitationsYou must specify the maximum key size. This can be worked around either by
pre-hashing the keys (with a strong hash) or using multiple hash tables for
different key sizes. Neither is currently implemented in diskhash.You cannot delete objects. This was not a necessity for my uses, so it was
not implemented. A simple implementation could be done by marking objects as
"deleted" in place and recompacting when the hash table size changes or with
an explicitdht_gc()call. It may also be important to add functionality to
shrink hashtables so as to not waste disk space.The algorithm is a rather naïve implementation of linear addression. It would
not be hard to switch torobin hood
hashingand this may indeed happen in the near future.License: MIT