Package: pyre2
Table of ContentsSummaryInstallationCompiling from sourcePlatform-agnostic building with condaBackwards CompatibilityDocumentationUnicode SupportPerformanceCurrent StatusTestsCreditsSummarypyre2 is a Python extension that wrapsGoogle’s RE2 regular expression library.
The RE2 engine compiles (strictly) regular expressions to
deterministic finite automata, which guarantees linear-time behavior.Intended as a drop-in replacement forre. Unicode is supported by encoding
to UTF-8, and bytes strings are treated as UTF-8 when the UNICODE flag is given.
For best performance, work with UTF-8 encoded bytes strings.InstallationNormal usage for Linux/Mac/Windows:$ pip install pyre2Compiling from sourceRequirements for building the C++ extension from the repo source:A build environment withgccorclang(e.g.sudoapt-getinstallbuild-essential)Build tools and libraries: RE2, pybind11, and cmake installed in the build
environment.On Ubuntu/Debian:sudoapt-getinstallbuild-essentialcmakeninja-buildpython3-devcython3pybind11-devlibre2-devOn Gentoo, install dev-util/cmake, dev-python/pybind11, and dev-libs/re2For a venv you can install the pybind11, cmake, and cython packages from PyPIOn MacOS, use thebrewpackage manager:$ brew install -s re2 pybind11On Windows use thevcpkgpackage manager:$ vcpkg install re2:x64-windows pybind11:x64-windowsYou can pass some cmake environment variables to alter the build type or
pass a toolchain file (the latter is required on Windows) or specify the
cmake generator.  For example:$ CMAKE_GENERATOR="Unix Makefiles" CMAKE_TOOLCHAIN_FILE=clang_toolchain.cmake tox -e deployFor development, get the source:$ git clone git://github.com/andreasvc/pyre2.git
$ cd pyre2
$ make installPlatform-agnostic building with condaAn alternative to the above is provided via thecondarecipe (use theminiconda installerif you don’t havecondainstalled already).Backwards CompatibilityThe stated goal of this module is to be a drop-in replacement forre, i.e.:try:
    import re2 as re
except ImportError:
    import reThat being said, there are features of theremodule that this module may
never have; these will be handled through fallback to the originalremodule:lookahead assertions(?!...)backreferences (\\nin search pattern)W and S not supported inside character classesOn the other hand, unicode character classes are supported (e.g.,\p{Greek}).
Syntax reference:https://github.com/google/re2/wiki/SyntaxHowever, there are times when you may want to be notified of a failover. The
functionset_fallback_notificationdetermines the behavior in these cases:try:
    import re2 as re
except ImportError:
    import re
else:
    re.set_fallback_notification(re.FALLBACK_WARNING)set_fallback_notificationtakes three values:re.FALLBACK_QUIETLY(default),re.FALLBACK_WARNING(raise a warning),
andre.FALLBACK_EXCEPTION(raise an exception).DocumentationConsult the docstrings in the source code or interactively
through ipython orpydoc re2etc.Unicode SupportPythonbytesandunicodestrings are fully supported, but note thatRE2works with UTF-8 encoded strings under the hood, which means thatunicodestrings need to be encoded and decoded back and forth.
There are two important factors:whether aunicodepattern and search string is used (will be encoded to UTF-8 internally)theUNICODEflag: whether operators such as\wrecognize Unicode characters.To avoid the overhead of encoding and decoding to UTF-8, it is possible to pass
UTF-8 encoded bytes strings directly but still treat them asunicode:In [18]: re2.findall(u'\w'.encode('utf8'), u'Mötley Crüe'.encode('utf8'), flags=re2.UNICODE)
Out[18]: ['M', '\xc3\xb6', 't', 'l', 'e', 'y', 'C', 'r', '\xc3\xbc', 'e']
In [19]: re2.findall(u'\w'.encode('utf8'), u'Mötley Crüe'.encode('utf8'))
Out[19]: ['M', 't', 'l', 'e', 'y', 'C', 'r', 'e']However, note that the indices inMatchobjects will refer to the bytes string.
The indices of the match in theunicodestring could be computed by
decoding/encoding, but this is done automatically and more efficiently if you
pass theunicodestring:>>> re2.search(u'ü'.encode('utf8'), u'Mötley Crüe'.encode('utf8'), flags=re2.UNICODE)
<re2.Match object; span=(10, 12), match='\xc3\xbc'>
>>> re2.search(u'ü', u'Mötley Crüe', flags=re2.UNICODE)
<re2.Match object; span=(9, 10), match=u'\xfc'>Finally, if you want to match bytes without regard for Unicode characters,
pass bytes strings and leave out theUNICODEflag (this will cause Latin 1
encoding to be used withRE2under the hood):>>> re2.findall(br'.', b'\x80\x81\x82')
['\x80', '\x81', '\x82']PerformancePerformance is of course the point of this module, so it better perform well.
Regular expressions vary widely in complexity, and the salient feature ofRE2is
that it behaves well asymptotically. This being said, for very simple substitutions,
I’ve found that occasionally python’s regularremodule is actually slightly faster.
However, when theremodule gets slow, it getsreallyslow, while this module
buzzes along.In the below example, I’m running the data against 8MB of text from the colossal Wikipedia
XML file. I’m running them multiple times, being careful to use thetimeitmodule.
To see more details, please see theperformance script.TestDescription# total runsretime(s)re2time(s)%retimeregextime(s)%regextimeFindall URI|EmailFind list of ‘([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/[^ ]*)?|([^ @]+)@([^ @]+)’26.2620.1312.08%5.1192.55%Replace WikiLinksThis test replaces links of the form [[Obama|Barack_Obama]] to Obama.1004.3740.81518.63%1.17669.33%Remove WikiLinksThis test splits the data by the <page> tag.1004.1530.2255.43%0.53742.01%Feel free to add more speed tests to the bottom of the script and send a pull request my way!Current StatusThe tests show the following differences with Python’sremodule:The$operator in Python’srematches twice if the string ends
with\n. This can be simulated using\n?$, except when doing
substitutions.Thepyre2module and Python’sremay behave differently with nested groups.
Seetests/test_emptygroups.txtfor the examples.Please report any further issues withpyre2.TestsIf you would like to help, one thing that would be very useful
is writing comprehensive tests for this. It’s actually really easy:Come up with regular expression problems using the regular python ‘re’ module.Write a session in python traceback formatExample.Replace yourimport rewithimport re2 as re.Save it with astest_<name>.txtin the tests directory. You can comment on it however you like and indent the code with 4 spaces.CreditsThis code builds on the following projects (in chronological order):Google’s RE2 regular expression library:https://github.com/google/re2Facebook’s pyre2 github repository:http://github.com/facebook/pyre2/Mike Axiak’s Cython version of this:http://github.com/axiak/pyre2/(seems not actively maintained)This fork adds Python 3 support and other improvements.