Package: boyermoore
Table Of ContentsBoyer-Moore in pure python: search for unicode strings quickly in large filesInstallingSearching for all occurences of a substring in a fileSearching for the first occurence of a substring in a filePerformance / Speed testTest environmentTest methodologyTest resultsContributionsBoyer-Moore in pure python: search for unicode strings quickly in large filesThis is an implementation of the Boyer-Moore substring search algorithm in pure python.It is a shameless copy-paste of the python reference code providedhere,
with modifications to support the following additional features:Searching in files without reading the whole file into memory, allowing handling of large filesFull unicode supportSee theAPI documentationfor more details.InstallingInstall frompip.pip install boyermooreSearching for all occurences of a substring in a file>>> from boyermoore import search_file
>>>
>>> offsets = search_file("pattern!", "file.txt")                 # Find all occurrences of "pattern!" in file "file.txt"
>>> offsets                                                       # Display found occurrences
[12, 456, 10422]                                                  # Pattern occurs at byte offsets 12, 456, and 104222Searching for the first occurence of a substring in a file>>> from boyermoore import search_file
>>>
>>> offsets = search_file("pattern!", "file.txt", greedy=False)   # Find the first occurrence of "pattern!" in file "file.txt"
>>> offsets                                                       # Display found occurrences
[12]                                                              # First occurrence of pattern is at byte offset 12Performance / Speed testThe following section illustrates the average speed of theboyermoore.search_filefunction when searching for a unicode string in files of sizes ranging from 32MB to 4GB.Test environmentThe test was executed using Python 3.7.6 on a Windows 10 system with an Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
and 32 GB of RAM.Test methodologyThe test searches for all occurrences of a fixed unicode string in a series of test files.
The unicode string is:Hello नमस्ते Привет こんにちは(“Hello” in English, followed by the Hindi translation, followed by the Russian translation,
followed by the Japanese translation)Each test file has 2 occurrences of the unicode string, one at the very beginning (byte offset of 0)
and one at the very end (byte offset of [file_length - pattern_length]).Test resultsThe following table shows the times taken to search for all occurences of the unicode
string “Hello नमस्ते Привет こんにちは” inside test files of various sizes.File sizeTime (seconds)32 MB0.3164 MB0.55128 MB1.07256 MB1.96512 MB3.871 GB7.564 GB31.01ContributionsContributions are welcome, please open a pull request athttps://github.com/eriknyquist/boyermooreand ensure that:All existing unit tests pass (run tests viapython setup.py test)New unit tests are added to cover any modified/new functionality (runpython code_coverage.pyto ensure that coverage is above 98%)You will need to install packages required for development, these are listed indev_requirements.txt:pip install -r dev_requirements.txtIf you have any questions about / need help with contributions or unit tests, please
contact Erik ateknyquist@gmail.com.