Package: noahong
Python Aho-Corasick implementationnoahongis a Python implementation of theAho-Corasickalgorithm for string matching, based on a fork of theNoAhoC++ implementation.noahongsupports macOS, Linux, Windows on Python 3.6+.APIThe first thing to do is to instantiate aNoAhoobject and add some keys to it (optionally with different payloads for each).fromnoahongimportNoAhotrie=NoAho()# fill with .add()trie.add("foo","id_foo")trie.add("foobar","id_foobar")# or fill with __setitem__trie["bar"]="id_bar"Once you have added the different keys and their payloads, theNoAhoobject needs to be compiled:trie.compile()Once it is compiled, keys can no longer be added to theNoAhoobject.noahongthen exposes four functions to find matching substrings in text:find_shorttrie.find_short(text)finds the first substring oftextthat is matched by a key added to thetrie.It returns a tuple(start, stop, payload)such that:payloadis the object inserted withtrie.add()startandstopare indices of the match in thetext:text[start:stop] == keyFor example, using the abovetrie:trie.find_short("something foo")# returns (10, 13, 'id_foo')# "something foo"[10:13] == "foo"and returns the first match even though a longer match may start at the same position:trie.find_short("something foobar")# returns (10, 13, 'id_foo')find_longtrie.find_long(text)finds the first longest substring oftextthat is matched by a key added to thetrie.For example, using the abovetrie:trie.find_long("something foobar")# returns (10, 16, 'id_foobar')findall_*Bothfind_shortandfind_longhave afindall_shortandfindall_longcounterparts that allow you to iterate on all non-overlapping matches found
in the text:forxintrie.findall_long("something foo bar foobar"):print(x)# prints# (10, 13, 'id_foo')# (14, 17, 'id_bar')# (18, 24, 'id_foobar')Because matches are non-overlapping:list(trie.findall_short("foobar"))==[(0,3,"id_foo"),(3,6,"id_bar")]whereas:list(trie.findall_long("foobar"))==[(0,6,"id_foobar")]PayloadsNoAhotries accept any Python object as a payload:trie=NoAho()trie.add("foo",0)trie.add("bar",CustomClass())trie.add("baz",lambdax:x)The same payload can be associated with different keys.Notice that the non-pickablelambda x: xpayload works because
there is no serialization involved here.Length and inclusionNoAhotrie objects also expose the number of keys withlen:len(trie)And, when they are compiled, they can be used to test for key inclusion:"foo"intrieThe number of nodes in the underlying Trie can be recovered withtrie.nodes_count()MappedNoAhoIn order to save memory,noahongexposes aMappedmatching object which can be written to disk and later loaded directly to memory to perform matches with a smaller memory footprint.TheMappedobject exposes different finding methods and only supports integer payloads.Construct it by adding keys and payloads to aNoAhoobject:fromnoahongimportNoAho,Mappedtrie=NoAho()trie.add("baz","id_baz")trie.compile()trie.write("./test.matcher")mapped_trie=Mapped("./test.matcher")Themapped_trieobject exposes afindall_anchoredfunction that iterates overanchoredmatches, matches that can be found within boundaries set with a special "anchor" character\u001F.This is useful to restrict matches to be found only between, say, spaces:trie=NoAho()trie.add("foo",0)trie.add("bar",1)trie.compile()trie.write("./test.matcher")mapped_trie=Mapped("./test.matcher")mapped_trie.findall_anchored("\u001Fbar\u001F\u001Ffoo\u001F\u001Ffoobar\u001F")# returns [(1, 4, 1), (6, 9, 0), (11, 14, 0)]Notice how"bar"is not found in the final"foobar"because it is not present between "anchor" characters.It is possible to place anchor characters in the keys:trie=NoAho()trie.add("foo\u001F\u001Fbar",0)trie.add("foo",1)trie.add("bar",2)trie.compile()trie.write("./test.matcher")mapped_trie=Mapped("./test.matcher")mapped_trie.findall_anchored("\u001Ffoo\u001F\u001Fbar\u001F")# returns [(1, 9, 0)]In this case, the longest key found between anchors is returned.InstallationDevpinoahongis available on devpi:pip install noahongPython 3noahongcan be installed manually:python3 setup.py installLegacy READMEYou can find more information on the package and C++ implementation by reading the
legacy README foundhere.