Package: pypocketmap
pypocketmapNOTE: this package is in beta. The current repo only contains implementations
for string-keyed maps.A high performance python hash table library that consumes significantly less
memory than Python Dictionaries. It currently supports Python 3.6+. It is forked fromMicrodictto remove the limitation on the
length of string keys, and add utility methods likegetandsetdefault.
It also has a slightly different table design, making use of code ported to
C fromabseil-cpp.BenchmarksThe latest charts are athttps://observablehq.com/@dylan-burati-ws/pypocketmap-benchmarksInstallation and BuildingYou can install using pip:pip install pypocketmap.pypocketmap is tested to work on Linux, Mac OSX, and Windows systems. You will need
GCC 7+ on linux/mac osx systems and Visual C++ 14+ compiler on Windows systems to
build the package. For the best performance use on a 64 bit system.UsageThe following code snippet shows common uses of the library.>>>importpypocketmapaspkm# Generates a dictionary with string keys and signed 64 bit integer values.>>>d=pkm.create(str,int)>>>d=pkm.create(pkm.string,pkm.i64)# or explicitly# Works just like a python dictionary, although insertion order != iteration order>>>d["a"]=2>>>"a"ind,"b"ind(True,False)# `get` default value can be any type>>>d.get("a"),d.get("b"),d.get("b",False)(2,None,False)# `setdefault` default value must be an int>>>d.setdefault("a",-10)2>>>d.setdefault("b",-10)-10>>>d.update({"okc":1997,"yhf":2002})>>>d<pypocketmap[str,int64]:{'b':-10,'okc':1997,'a':2,'yhf':2002}>>>>[kforkind],list(d.keys())(['b','okc','a','yhf'],['b','okc','a','yhf'])>>>list(d.values()),list(d.items())([-10,1997,2,2002],[('b',-10),('okc',1997),('a',2),('yhf',2002)])>>>len(d)4>>>d.clear()>>>len(d)0How it worksThe C partsThe map part of pypocketmap is just a C port of abseil-cpp's SwissTable, which is explained nicely in this blog post series:https://www.kylematsuda.com/blog/writing_a_hashmap_part_2.The memory savings (pocket part) are partially due to an optimization for small strings - when stored in
the key or value array of the map, a string is a 16-bytepacked_str_t. This is a union:contained: the data fits in 15 bytes, and the length and sentinel bit1fit in 1 byte.spilled: the struct holds achar *pointing to the data, and the length and sentinel bit0take up
the remaining 8 bytes.4 bytes are padding on 32-bit platforms; 8 bytes end up unused because the longer lengths would fail
to allocate.The C++ standard library and Rust cratebyteyarndo something similar - I learned about this from
the crate author's blog post:https://mcyoung.xyz/2023/08/09/yarns/.The Python C API partsThe filestr_int64_Py.cis the only Python module definition which should
be edited. That file andabstract.huse C macros andtypedefto fake generics.
Many Python-specific blocks in the former file are also annotated with customtemplate!comments, which
I came up with for this library's Java equivalent,pocketmap.Thetemplate!comments are the reason for the nearly equivalent*_Py.cfiles in the repo. Essentially,
the macros and comments implement the following traits for the key and value types. Since the
"implementations" are just inlined into the files, this is prone to errors. It is probably doable in C++,
but for now I'm choosing to add tests rather than convert the C code.traitValue{typePacked;// the `&` operator indicates something borrowed.// physically, both Self and &Self can be a struct { char* data, uint64_t len }// semantically, &Self means that the data field is readonly, and its not safe// to use the pointer once the source of the borrow is possibly freed/dropped.// ===/// Converts the possibly-borrowed value to an owned value independent of any/// PyObjects and stores it.fnp_set(arr:*mutSelf::Packed,index:usize,elem:&Self);/// Frees anything alloc'd in a previous `p_set(arr, index)`./// No-op for everything except spilled strings.fnp_unset(arr:*mutSelf::Packed,index:usize);/// Borrows the value - the caller is responsible for making sure `arr[index]` is/// set, and stays set while the returned &Self is in use.fnp_get(arr:*Self::Packed,index:usize)->&Self;/// Converts the Python object to a C value.fnfrom_py(obj:&PyObject)->Result<&Self,()>;/// Convert the C value to a new Python object which owns its data. The caller/// owns the returned Python object, and should eventually either decrement its/// reference count, or return it as the result of a pymethodfunc.fnas_py(&self)->PyObject;/// For `repr`.fnfmt(&self,f:&mut_PyUnicodeWriter)->Result<(),()>;/// For `setdefault` when the value arg is left out.fndefault()->&'staticSelf;/// For `__eq__` when acting as the value type. For pretty much all methods/// when acting as the key type.fneq(&self,other:&Self)->bool;}traitKey:Value+Hash{}Not yet supportedAutomatic or manualshrink_to_fitConcurrent modification exceptions if an iterator is used after the table has been rehashed. The
implementation can skip elements or yield them twice if used incorrectly.updateshould work on any arg whenPyDict_Checkreturns trueorPyMapping_Keysreturns non-null__or__and__ior__operators can be implemented with thisTheMETH_FASTCALLconvention is
stable since Python 3.10, and it should be possible to alter the *Py.c files to use it in place of
METH_VARARGS when compiling for 3.10+.Additional overflow checking whenLLONG_MAX != INT64_MAXor (more likely)LONG_MAX != INT32_MAX