Package: retrie
retrieretrieoffers fast methods to match and replace (sequences of) strings based on efficient Trie-based regex unions.TrieInstead of matching against a simple regex union, which becomes slow for large sets of words, a more efficient regex pattern can be compiled using aTriestructure:fromretrie.trieimportTrietrie=Trie()trie.add("abc","foo","abs")asserttrie.pattern()=="(?:ab[cs]|foo)"# equivalent to but faster than "(?:abc|abs|foo)"trie.add("absolute")asserttrie.pattern()=="(?:ab(?:c|s(?:olute)?)|foo)"trie.add("abx")asserttrie.pattern()=="(?:ab(?:[cx]|s(?:olute)?)|foo)"trie.add("abxy")asserttrie.pattern()=="(?:ab(?:c|s(?:olute)?|xy?)|foo)"ATriemay be populated with zero or more strings at instantiation or viaTrie.add, from which method chaining is possible. Two instances can be merged with the+(new instance) and+=(in-place update) operators. Instances will compare equal if their data dictionaries are equal.trie=Trie()trie+=Trie("abc")assert(trie+Trie().add("foo")==Trie("abc","foo")==Trie(*["abc","foo"])==Trie().add(*["abc","foo"])==Trie().add("abc","foo")==Trie().add("abc").add("foo"))InstallationThis pure-Python, OS independent package is available onPyPI:$pipinstallretrieUsageFor documentation, seeretrie.readthedocs.io.The following objects are all subclasses ofretrie.retrie.Retrie, which handles filling the Trie and compiling the corresponding regex pattern.BlacklistTheBlacklistobject can be used to filter out bad occurences in a text or a sequence of strings:fromretrie.retrieimportBlacklist# check out docstrings and methodshelp(Blacklist)blacklist=Blacklist(["abc","foo","abs"],match_substrings=False)blacklist.compiled# re.compile(r'(?<=\b)(?:ab[cs]|foo)(?=\b)', re.IGNORECASE|re.UNICODE)assertnotblacklist.is_blacklisted("a foobar")asserttuple(blacklist.filter(("good","abc","foobar")))==("good","foobar")assertblacklist.cleanse_text(("good abc foobar"))=="good  foobar"blacklist=Blacklist(["abc","foo","abs"],match_substrings=True)blacklist.compiled# re.compile(r'(?:ab[cs]|foo)', re.IGNORECASE|re.UNICODE)assertblacklist.is_blacklisted("a foobar")asserttuple(blacklist.filter(("good","abc","foobar")))==("good",)assertblacklist.cleanse_text(("good abc foobar"))=="good  bar"WhitelistSimilar methods are available for theWhitelistobject:fromretrie.retrieimportWhitelist# check out docstrings and methodshelp(Whitelist)whitelist=Whitelist(["abc","foo","abs"],match_substrings=False)whitelist.compiled# re.compile(r'(?<=\b)(?:ab[cs]|foo)(?=\b)', re.IGNORECASE|re.UNICODE)assertnotwhitelist.is_whitelisted("a foobar")asserttuple(whitelist.filter(("bad","abc","foobar")))==("abc",)assertwhitelist.cleanse_text(("bad abc foobar"))=="abc"whitelist=Whitelist(["abc","foo","abs"],match_substrings=True)whitelist.compiled# re.compile(r'(?:ab[cs]|foo)', re.IGNORECASE|re.UNICODE)assertwhitelist.is_whitelisted("a foobar")asserttuple(whitelist.filter(("bad","abc","foobar")))==("abc","foobar")assertwhitelist.cleanse_text(("bad abc foobar"))=="abcfoo"ReplacerTheReplacerobject does a fast single-pass search & replace for occurrences ofreplacement_mapping.keys()with corresponding values.fromretrie.retrieimportReplacer# check out docstrings and methodshelp(Replacer)replacement_mapping=dict(zip(["abc","foo","abs"],["new1","new2","new3"]))replacer=Replacer(replacement_mapping,match_substrings=True)replacer.compiled# re.compile(r'(?:ab[cs]|foo)', re.IGNORECASE|re.UNICODE)assertreplacer.replace("ABS ...foo... foobar")=="new3 ...new2... new2bar"replacer=Replacer(replacement_mapping,match_substrings=False)replacer.compiled# re.compile(r'\b(?:ab[cs]|foo)\b', re.IGNORECASE|re.UNICODE)assertreplacer.replace("ABS ...foo... foobar")=="new3 ...new2... foobar"replacer=Replacer(replacement_mapping,match_substrings=False,re_flags=None)replacer.compiled# on py3, re.UNICODE is always enabled# re.compile(r'\b(?:ab[cs]|foo)\b')assertreplacer.replace("ABS ...foo... foobar")=="ABS ...new2... foobar"replacer=Replacer(replacement_mapping,match_substrings=False,word_boundary=" ")replacer.compiled# re.compile(r'(?<= )(?:ab[cs]|foo)(?= )', re.IGNORECASE|re.UNICODE)assertreplacer.replace(". ABS ...foo... foobar")==". new3 ...foo... foobar"DevelopmentRunmake helpfor options like installing for development, linting and testing.