Package: ant-datrie
datrieSuper-fast, efficiently stored Trie for Python (2.x and 3.x).
Useslibdatrie.Installationpip install datrieUsageCreate a new trie capable of storing items with lower-case ascii keys:>>> import string
>>> import datrie
>>> trie = datrie.Trie(string.ascii_lowercase)trievariable is a dict-like object that can have unicode keys of
certain ranges and Python objects as values.In addition to implementing the mapping interface, tries facilitate
finding the items for a given prefix, and vice versa, finding the
items whose keys are prefixes of a given string. As a common special
case, finding the longest-prefix item is also supported.WarningFor efficiency you must define allowed character range(s) while
creating trie.datriedoesn’t check if keys are in allowed
ranges at runtime, so be careful! Invalid keys are OK at lookup time
but values won’t be stored correctly for such keys.Add some values to it (datrie keys must be unicode; the examples
are for Python 2.x):>>> trie[u'foo'] = 5
>>> trie[u'foobar'] = 10
>>> trie[u'bar'] = 'bar value'
>>> trie.setdefault(u'foobar', 15)
10Check if u’foo’ is in trie:>>> u'foo' in trie
TrueGet a value:>>> trie[u'foo']
5Find all prefixes of a word:>>> trie.prefixes(u'foobarbaz')
[u'foo', u'foobar']

>>> trie.prefix_items(u'foobarbaz')
[(u'foo', 5), (u'foobar', 10)]

>>> trie.iter_prefixes(u'foobarbaz')
<generator object ...>

>>> trie.iter_prefix_items(u'foobarbaz')
<generator object ...>Find the longest prefix of a word:>>> trie.longest_prefix(u'foo')
u'foo'

>>> trie.longest_prefix(u'foobarbaz')
u'foobar'

>>> trie.longest_prefix(u'gaz')
KeyError: u'gaz'

>>> trie.longest_prefix(u'gaz', default=u'vasia')
u'vasia'

>>> trie.longest_prefix_item(u'foobarbaz')
(u'foobar', 10)Check if the trie has keys with a given prefix:>>> trie.has_keys_with_prefix(u'fo')
True

>>> trie.has_keys_with_prefix(u'FO')
FalseGet all items with a given prefix from a trie:>>> trie.keys(u'fo')
[u'foo', u'foobar']

>>> trie.items(u'ba')
[(u'bar', 'bar value')]

>>> trie.values(u'foob')
[10]Get all suffixes of certain word starting with a given prefix from a trie:>>> trie.suffixes()
[u'pro', u'producer', u'producers', u'product', u'production', u'productivity', u'prof']
>>> trie.suffixes(u'prod')
[u'ucer', u'ucers', u'uct', u'uction', u'uctivity']Save & load a trie (values must be picklable):>>> trie.save('my.trie')
>>> trie2 = datrie.Trie.load('my.trie')Trie and BaseTrieThere are two Trie classes in datrie package:datrie.Trieanddatrie.BaseTrie.datrie.BaseTrieis slightly faster and uses less
memory but it can store only integer numbers -2147483648 <= x <= 2147483647.datrie.Trieis a bit slower but can store any Python object as a value.If you don’t need values or integer values are OK then usedatrie.BaseTrie:import datrie
import string
trie = datrie.BaseTrie(string.ascii_lowercase)Custom iterationIf the built-in trie methods don’t fit you can usedatrie.Stateanddatrie.Iteratorto implement custom traversal.NoteIf you usedatrie.BaseTrieyou needdatrie.BaseStateanddatrie.BaseIteratorfor custom traversal.For example, let’s find all suffixes of'fo'for our trie and get
the values:>>> state = datrie.State(trie)
>>> state.walk(u'foo')
>>> it = datrie.Iterator(state)
>>> while it.next():
...     print(it.key())
...     print(it.data))
o
5
obar
10PerformancePerformance is measured fordatrie.Trieagainst Python’s dict with
100k unique unicode words (English and Russian) as keys and ‘1’ numbers
as values.datrie.Trieuses about 5M memory for 100k words; Python’s dict
uses about 22M for this according to my unscientific tests.This trie implementation is 2-6 times slower than python’s dict
on __getitem__. Benchmark results (macbook air i5 1.8GHz,
“1.000M ops/sec” == “1 000 000 operations per second”):Python 2.6:
dict __getitem__: 7.107M ops/sec
trie __getitem__: 2.478M ops/sec

Python 2.7:
dict __getitem__: 6.550M ops/sec
trie __getitem__: 2.474M ops/sec

Python 3.2:
dict __getitem__: 8.185M ops/sec
trie __getitem__: 2.684M ops/sec

Python 3.3:
dict __getitem__: 7.050M ops/sec
trie __getitem__: 2.755M ops/secLooking for prefixes of a given word is almost as fast as__getitem__(results are for Python 3.3):trie.iter_prefix_items (hits):      0.461M ops/sec
trie.prefix_items (hits):           0.743M ops/sec
trie.prefix_items loop (hits):      0.629M ops/sec
trie.iter_prefixes (hits):          0.759M ops/sec
trie.iter_prefixes (misses):        1.538M ops/sec
trie.iter_prefixes (mixed):         1.359M ops/sec
trie.has_keys_with_prefix (hits):   1.896M ops/sec
trie.has_keys_with_prefix (misses): 2.590M ops/sec
trie.longest_prefix (hits):         1.710M ops/sec
trie.longest_prefix (misses):       1.506M ops/sec
trie.longest_prefix (mixed):        1.520M ops/sec
trie.longest_prefix_item (hits):    1.276M ops/sec
trie.longest_prefix_item (misses):  1.292M ops/sec
trie.longest_prefix_item (mixed):   1.379M ops/secLooking for all words starting with a given prefix is mostly limited
by overall result count (this can be improved in future because a
lot of time is spent decoding strings from utf_32_le to Python’s
unicode):trie.items(prefix="xxx"), avg_len(res)==415:        0.609K ops/sec
trie.keys(prefix="xxx"), avg_len(res)==415:         0.642K ops/sec
trie.values(prefix="xxx"), avg_len(res)==415:       4.974K ops/sec
trie.items(prefix="xxxxx"), avg_len(res)==17:       14.781K ops/sec
trie.keys(prefix="xxxxx"), avg_len(res)==17:        15.766K ops/sec
trie.values(prefix="xxxxx"), avg_len(res)==17:      96.456K ops/sec
trie.items(prefix="xxxxxxxx"), avg_len(res)==3:     75.165K ops/sec
trie.keys(prefix="xxxxxxxx"), avg_len(res)==3:      77.225K ops/sec
trie.values(prefix="xxxxxxxx"), avg_len(res)==3:    320.755K ops/sec
trie.items(prefix="xxxxx..xx"), avg_len(res)==1.4:  173.591K ops/sec
trie.keys(prefix="xxxxx..xx"), avg_len(res)==1.4:   180.678K ops/sec
trie.values(prefix="xxxxx..xx"), avg_len(res)==1.4: 503.392K ops/sec
trie.items(prefix="xxx"), NON_EXISTING:             2023.647K ops/sec
trie.keys(prefix="xxx"), NON_EXISTING:              1976.928K ops/sec
trie.values(prefix="xxx"), NON_EXISTING:            2060.372K ops/secRandom insert time is very slow compared to dict, this is the limitation
of double-array tries; updates are quite fast. If you want to build a trie,
consider sorting keys before the insertion:dict __setitem__ (updates):            6.497M ops/sec
trie __setitem__ (updates):            2.633M ops/sec
dict __setitem__ (inserts, random):    5.808M ops/sec
trie __setitem__ (inserts, random):    0.053M ops/sec
dict __setitem__ (inserts, sorted):    5.749M ops/sec
trie __setitem__ (inserts, sorted):    0.624M ops/sec
dict setdefault (updates):             3.455M ops/sec
trie setdefault (updates):             1.910M ops/sec
dict setdefault (inserts):             3.466M ops/sec
trie setdefault (inserts):             0.053M ops/secOther results (note thatlen(trie)is currently implemented
using trie traversal):dict __contains__ (hits):    6.801M ops/sec
trie __contains__ (hits):    2.816M ops/sec
dict __contains__ (misses):  5.470M ops/sec
trie __contains__ (misses):  4.224M ops/sec
dict __len__:                334336.269 ops/sec
trie __len__:                22.900 ops/sec
dict values():               406.507 ops/sec
trie values():               20.864 ops/sec
dict keys():                 189.298 ops/sec
trie keys():                 2.773 ops/sec
dict items():                48.734 ops/sec
trie items():                2.611 ops/secPlease take this benchmark results with a grain of salt; this
is a very simple benchmark and may not cover your use case.Current Limitationskeys must be unicode (no implicit conversion for byte strings
under Python 2.x, sorry);there are no iterator versions of keys/values/items (this is not
implemented yet);it is painfully slow and maybe buggy under pypy;library is not tested with narrow Python builds.ContributingDevelopment happens at github:https://github.com/pytries/datrie.Feel free to submit ideas, bugs, pull requests.Running tests and benchmarksMake suretoxis installed and run$ toxfrom the source checkout. Tests should pass under Python 2.7 and 3.4+.$ tox -c tox-bench.iniruns benchmarks.If you’ve changed anything in the source code then
make surecythonis installed and run$ update_c.shbefore eachtoxcommand.Please note that benchmarks are not included in the release
tar.gz’s because benchmark data is large and this
saves a lot of bandwidth; use source checkouts from
github or bitbucket for the benchmarks.Authors & ContributorsSeehttps://github.com/pytries/datrie/graphs/contributors.This module is based onlibdatrieC library by Theppitak Karoonboonyanan
and is inspired byfast_trieRuby bindings,PyTriepure
Python implementation andTree::TriePerl implementation;
some docs and API ideas are borrowed from these projects.LicenseLicensed under LGPL v2.1.
CHANGES
=======0.8.4 (2022-03-30)forked from pytries/datrie .rename ant-datrieFix Decode string based on byteorder of systemAdjust TrieBase load0.8.2 (2020-03-25)Future-proof Python support by making cython a build time dependency and
removing cython generated c files from the repo (and sdist).Fix collections.abc.MutableMapping importCI and test updatesAdjust library name to unbreak some linkers0.8.1 (skipped)This version intentionally skipped0.8 (2019-07-03)Python 3.7 compatibility; extension is rebuilt with Cython 0.29.11.Trie.get function;Python 2.6 and 3.3 support is dropped;removed patch to libdatrie which is no longer required;testing and CI fixes.0.7.1 (2016-03-12)updated the bundled C library to version 0.2.9;implementedTrie.__len__in terms oftrie_enumerate;rebuilt Cython wrapper with Cython 0.23.4;changedTrieto implementcollections.abc.MutableMapping;fixedTriepickling, which segfaulted on Python2.X.0.7 (2014-02-18)bundled libdatrie C library is updated to version 0.2.8;new.suffixes()method (thanks Ahmed T. Youssef);wrapper is rebuilt with Cython 0.20.1.0.6.1 (2013-09-21)fixed build for Visual Studio (thanks Gabi Davar).0.6 (2013-07-09)datrie is rebuilt with Cython 0.19.1;iter_prefix_values,prefix_valuesandlongest_prefix_valuemethods fordatrie.BaseTrieanddatrie.Trie(thanks Jared Suttles).0.5.1 (2013-01-30)Recently introduced memory leak inlongest_prefixandlongest_prefix_itemis fixed.0.5 (2013-01-29)longest_prefixandlongest_prefix_itemmethods are fixed;datrie is rebuilt with Cython 0.18;misleading benchmark results in README are fixed;State._walk is renamed to State.walk_char.0.4.2 (2012-09-02)Update to latest libdatrie; this makes.keys()method a bit slower but
removes a keys length limitation.0.4.1 (2012-07-29)cPickle is used for saving/loadingdatrie.Trieif it is available.0.4 (2012-07-27)libdatrieimprovements and bugfixes, including C iterator API support;custom iteration support usingdatrie.Stateanddatrie.Iterator.speed improvements:__length__,keys,valuesanditemsmethods should be up to 2x faster.keys longer than 32768 are not supported in this release.0.3 (2012-07-21)There are no new features or speed improvements in this release.datrie.newis deprecated; usedatrie.Triewith the same arguments;small test & benchmark improvements.0.2 (2012-07-16)datrie.Trieitems can have any Python object as a value
(Triefrom 0.1.x becomesdatrie.BaseTrie);longest_prefixandlongest_prefix_itemsare fixed;save&loadare rewritten;setdefaultmethod.0.1.1 (2012-07-13)Windows support (upstream libdatrie changes are merged);license is changed from LGPL v3 to LGPL v2.1 to match the libdatrie license.0.1 (2012-07-12)Initial release.