Package: pyahocorasick
pyahocorasickis a fast and memory efficient library for exact or approximate
multi-pattern string search meaning that you can find multiple key strings
occurrences at once in some input text.  The strings “index” can be built ahead
of time and saved (as a pickle) to disk to reload and reuse later.  The library
provides anahocorasickPython module that you can use as a plain dict-like
Trie or convert a Trie to an automaton for efficient Aho-Corasick search.pyahocorasickis implemented in C and tested on Python 3.6 and up.
It works on 64 bits Linux, macOS and Windows.Thelicenseis BSD-3-Clause. Some utilities, such as tests and the pure Python
automaton are dedicated to the Public Domain.TestimonialsMany thanks for this package. Wasn’t sure where to leave a thank you note but
this package is absolutely fantastic in our application where we have a library
of 100k+ CRISPR guides that we have to count in a stream of millions of DNA
sequencing reads. This package does it faster than the previous C program we
used for the purpose and helps us stick to just Python code in our pipeline.Miika (AstraZeneca Functional Genomics Centre)https://github.com/WojciechMula/pyahocorasick/issues/145Download and source codeYou can fetchpyahocorasickfrom:GitHubhttps://github.com/WojciechMula/pyahocorasick/Pypihttps://pypi.python.org/pypi/pyahocorasick/Conda-Forgehttps://github.com/conda-forge/pyahocorasick-feedstock/Thedocumentationis published athttps://pyahocorasick.readthedocs.io/Quick startThis module is written in C. You need a C compiler installed to compile native
CPython extensions. To install:pip install pyahocorasickThen create an Automaton:>>> import ahocorasick
>>> automaton = ahocorasick.Automaton()You can use the Automaton class as a trie. Add some string keys and their associated
value to this trie. Here we associate a tuple of (insertion index, original string)
as a value to each key string we add to the trie:>>> for idx, key in enumerate('he her hers she'.split()):
...   automaton.add_word(key, (idx, key))Then check if some string exists in the trie:>>> 'he' in automaton
True
>>> 'HER' in automaton
FalseAnd play with theget()dict-like method:>>> automaton.get('he')
(0, 'he')
>>> automaton.get('she')
(3, 'she')
>>> automaton.get('cat', 'not exists')
'not exists'
>>> automaton.get('dog')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyErrorNow convert the trie to an Aho-Corasick automaton to enable Aho-Corasick search:>>> automaton.make_automaton()Then search all occurrences of the keys (the needles) in an input string (our haystack).Here we print the results and just check that they are correct. TheAutomaton.iter()method return the results as two-tuples of theend indexwhere a
trie key was found in the input string and the associatedvaluefor this key. Here
we had stored as values a tuple with the original string and its trie insertion
order:>>> for end_index, (insert_order, original_value) in automaton.iter(haystack):
...     start_index = end_index - len(original_value) + 1
...     print((start_index, end_index, (insert_order, original_value)))
...     assert haystack[start_index:start_index + len(original_value)] == original_value
...
(1, 2, (0, 'he'))
(1, 3, (1, 'her'))
(1, 4, (2, 'hers'))
(4, 6, (3, 'she'))
(5, 6, (0, 'he'))You can also create an eventually large automaton ahead of time andpickleit to
re-load later. Here we just pickle to a string. You would typically pickle to a
file instead:>>> import pickle
>>> pickled = pickle.dumps(automaton)
>>> B = pickle.loads(pickled)
>>> B.get('he')
(0, 'he')See also:FAQ and Who is using pyahocorasick?https://github.com/WojciechMula/pyahocorasick/wiki/FAQ#who-is-using-pyahocorasickDocumentationThe full documentation including the API overview and reference is published onreadthedocs.OverviewWith anAho-Corasick automatonyou can efficiently search all occurrences of multiple strings (the needles) in an
input string (the haystack) making a single pass over the input string. With
pyahocorasick you can eventually build large automatons and pickle them to reuse
them over and over as an indexed structure for fast multi pattern string matching.One of the advantages of an Aho-Corasick automaton is that the typical worst-case
and best-caseruntimesare about the same and depends primarily on the size
of the input string and secondarily on the number of matches returned.  While
this may not be the fastest string search algorithm in all cases, it can search
for multiple strings at once and its runtime guarantees make it rather unique.
Because pyahocorasick is based on a Trie, it stores redundant keys prefixes only
once using memory efficiently.A drawback is that it needs to be constructed and “finalized” ahead of time
before you can search strings. In several applications where you search for
several pre-defined “needles” in a variable “haystacks” this is actually an
advantage.Aho-Corasick automatonsare commonly used for fast multi-pattern matching
in intrusion detection systems (such as snort), anti-viruses and many other
applications that need fast matching against a pre-defined set of string keys.Internally an Aho-Corasick automaton is typically based on a Trie with extra
data for failure links and an implementation of the Aho-Corasick search
procedure.Behind the scenes thepyahocorasickPython library implements these two data
structures:  aTrieand an Aho-Corasick
string matching automaton. Both are exposed through theAutomatonclass.In addition to Trie-like and Aho-Corasick methods and data structures,pyahocorasickalso implements dict-like methods: The pyahocorasickAutomatonis aTriea dict-like structure indexed by string keys each
associated with a value object. You can use this to retrieve an associated value
in a time proportional to a string key length.pyahocorasick is available in two flavors:a CPythonC-based extension, compatible with Python 3 only. Use older
version 1.4.x for Python 2.7.x and 32 bits support.a simpler pure Python module, compatible with Python 2 and 3. This is only
available in the source repository (not on Pypi) under the etc/py/ directory
and has a slightly different API.Unicode and bytesThe type of strings accepted and returned byAutomatonmethods are eitherunicodeorbytes, depending on a compile time settings (preprocessor
definition ofAHOCORASICK_UNICODEas set insetup.py).TheAutomaton.unicodeattributes can tell you how the library was built.
On Python 3, unicode is the default.WarningWhen the library is built with unicode support, an Automaton will store 2 or
4 bytes per letter, depending on your Python installation. When built
for bytes, only one byte per letter is needed.Build and install from PyPiTo install for common operating systems, use pip. Pre-built wheels should be
available on Pypi at some point in the future:pip install pyahocorasickTo build from sources you need to have a C compiler installed and configured which
should be standard on Linux and easy to get on MacOSX.To build from sources, clone the git repository or download and extract the source
archive.Installpip(and itssetuptoolscompanion) and then run (in avirtualenvof
course!):pip install .If compilation succeeds, the module is ready to use.SupportSupport is available through theGitHub issue trackerto report bugs or ask
questions.ContributingYou can submit contributions throughGitHub pull requests.There is a Makefile with a default target that builds and runs tests.The tests can run with apip installe -e .[testing] && pytest -vvsSee also the .github directory for CI tests and workflowAuthorsThe initial author and maintainer is Wojciech Muła.Philippe Ombredanneis Wojciech’s sidekick and helps maintaining,
and rewrote documentation, setup CI servers and did a some work to make this
module more accessible to end users.Alphabetic list of authors:Andrew GrigorevBogdanDavid WoakesEdward BettsFrankie RobertsonFrederik PetersengladtoseeINADA NaokiJan FanPastafarianistPhilippe OmbredanneRenat NasyrovSylvain ZimmerXiaopeng Xuand many others!This library would not be possible without help of many people, who contributed in
various ways.
They createdpull requests,
reported bugs asGitHub issuesor via direct messages, proposed fixes, or spent their valuable time on testing.Thank you.LicenseThis library is licensed under very liberalBSD-3-Clauselicense. Some
portions of the code are dedicated to the public domain such as the pure Python
automaton and test code.Full text of license is available in LICENSE file.Other Aho-Corasick implementations for Python you can considerWhilepyahocorasicktries to be the finest and fastest Aho Corasick library
for Python you may consider these other libraries:py_aho_corasickby JanWritten in pure Python.Poor performance.ahocorapyby abusixWritten in pure Python.Better performance than py-aho-corasick.Using pypy, ahocorapy’s search performance is only slightly worse than pyahocorasick’s.Performs additional suffix shortcutting (more setup overhead, less search overhead for suffix lookups).Includes visualization tool for resulting automaton (using pygraphviz).MIT-licensed, 100% test coverage, tested on all major python versions (+ pypy)noahoby Jeff DonnerWritten in C. Does not return overlapping matches.Does not compile on Windows (July 2016).No support for the pickle protocol.acoraby Stefan BehnelWritten in Cython.Large automaton may take a long time to build (July 2016)No support for a dict-like protocol to associate a value to a string key.ahocorasickby Danny YooWritten in C.seems unmaintained (last update in 2005).GPL-licensed.