Package: bytetrie
BytetrieA fast, dependency-free, self-compressing trie with radix 256 in pure python.Bytetrie allows fast prefix search in a large corpus of keys. Each key can be
associated with arbitrary data. It features fast lookup times at the cost of
expensive insertion. A Bytetrie is best used if it can be pre-filled with data.
However, due to its in-band compression it can be also used for on-the-fly
updates.KeysKeys are byte strings. Therefore, each node in the trie can have up to 256
children (the radix). Keys do work well with utf-8 and other encodings as long
as the encoding is consistent and deterministic. That is, grapheme clusters
are always encoded to the same byte sequence -- even if the standard allows for
ambiguity. Usually that's a non-issue as long as the same encoder is used for
insertion and lookup.Since prefix search in unicode strings is one of the most common use-cases of
bytetrie, a unicode layer on top of bytetrie isplanned.DataBytetrie can associate arbitrary python objects with keys. Data (or rather a
reference thereof) is kept in-tree. No further processing is done.In addition, bytrie allows multi-valued tries. Every key is then associated with
a sequence of arbitrary objects.PerformanceDespite being in pure python bytetrie isfast. Sifting through the fullgeonames"allCountries" dataset for
places starting withViennatakes a mere 512µs. That's not even a
millisecond for searching through 12,041,359 places. For comparison, a warmed-up
ripgrep search through the same dataset takes three orders of magnitude (400ms)
longer on the same machine.On the downside, building the trie takes about 20 minutes and considerable
memory. Also, the performance is mostly trumped by the time it takes to collect
terminal nodes. The higher up the trie the search ends (and hence the more
results the prefix search yields) the longer it takes. There are several
low-hanging fruits left and further performance improvements are in thepipeline.DependenciesNone. That's the point.Getting startedInstall bytetrie viapip.pip install -U bytetrieThe public interface isByteTriewith the two methodsinsertandfind.
Find returns a list ofTerminalsfrom which thekeyand thevalueof the
node can be retrieved.frombytetrieimportByteTriet=ByteTrie(multi_value=True)t.insert(b"Hallo","Dutch")t.insert(b"Hello","English")t.insert(b"Hug","Gaelic")t.insert(b"Hallo","German")t.insert("Hē".encode("utf-8"),"Hindi")t.insert("Halló".encode("utf-8"),"Icelandic")t.insert(b"Hej","Polish")t.insert(b"Hei","Romanian")t.insert(b"Hujambo","Swahili")t.insert(b"Hej","Swedish")t.insert(b"Helo","Welsh")print("Where to say 'Hi' with 'He'?")print(f"{[(n.key(),n.value())fornint.find(b'He')]}")# [(b'Hei', ['Romanian']), (b'Hej', ['Swedish', 'Polish']), (b'Helo', ['Welsh']), (b'Hello', ['English'])]print("Where to say 'Hi' with 'Ha'?")print(f"{[(n.key().decode(),n.value())fornint.find(b'Ha')]}")# [('Halló', ['Icelandic']), ('Hallo', ['German', 'Dutch'])]print("Where to say 'Hi' with 'Hē'?")print(f"Say 'Hi' with utf-8:{[(n.key().decode(),n.value())fornint.find('Hē'.encode())]}")# [('Hē', ['Hindi'])]ContributeIf you want to contribute tobytetriefeel free to send patches to
dev[at]friedl[dot]net. Alternatviely, you can issue a pull request on GitHub
which will be cherry picked into my tree. If you plan significant long-term
contributions drop me a mail for access to the incubator repository.Github UsersIf you are visiting this repository on GitHub, you are on a mirror ofhttps://git.friedl.net/incubator/bytetrie. This mirror is regularily updated
with my other GitHub mirrors.Like with my other incubator projects, once I considerbytetriereasonable
stable the main tree will move to GitHub.