Package: pytextrust
PyTextRustmain:develop:Library defined to achieve easily high performance on regex and text processing inside Python, being built as a direct Wrapper of Rust regex and text crates.On short text, sparsity of found elements is the common denominator, this library focuses on algorithms that aknowledge this sparsity and efficiently achieves good performance from simple Python API calls to Rust optimized logics.Give some happinessFeaturesSpecial caseThis lib has special treatment for texts that only contain[a-zA-Z0-9ñç ]plus accented vocals, allowing to use non unicode matching over those texts. This is particularly convenient for some Automatic Speech Recognition outputs.In every place that it is possibly to provide it, this:unicode:False-> removes unicode chars from matching, making matching much more efficient (x6 - x12 it is easly achieved).substitute_bound:True-> substitutes in patternsr"\b"forr"(?-u:\b)"as recommendedheresubstitute_latin_char: True -> substitutes in patternspkg::constants::LATIN_CHARS_TO_REPLACEforpkg::constants::LATIN_CHARS_REPLACEMENT, to allow the use of non unicode variant without losing the ability to match texts and patterns that contain those latin chars (careit projects them intopkg::constants::LATIN_CHARS_TO_REPLACEboth in patterns and texts).FindFind patterns in texts, possibly parallelizing by chunks of either patterns or texts.It uses efficientregex::RegexSetthat reduces the cardinality of the patterns in the matching phase.The structure of finding function is:Rust phase:Try to compile inregex::Regexfor the list of patterns. Get the list of valid ones and invalid ones.Compileregex::RegexSetwith valid patterns and apply over the list of texts. This gives which ones have match over the texts.Operate compiledregex::Regex, finding them over all the texts for the subset of pairs that have matched in theregex::RegexSet.Try to compile invalid patterns withfancy_regex::Regexand find matches over the texts. It reduces final invalid
patterns list that is given back to python.Give matches of valid patterns and invalid patterns back to Python.Python phase:Try to apply all failed patterns, finding them over all the texts. It usesregexpackage
that has expanded pattern support overrebuilt-in package.Return the final result.Calling examplesLiteral replacerThis is a very concrete function to perform high performance literal replacement using Rustaho_corasickimplementation. It accepts parallelization by chunks of text.It uses Rustaho_corasickto perform replacements, adding a layer of bounding around literals to replace through theis_boundedparameter.Ifis_boundedisTruethen before replacing the literal found, it is checked that any of[A-Za-z0-9_](expanded with accents and special word chards that can be checked inpkg::unicode::check_if_word_bytes) is around the literal.Matching types can be chosen over available ones inaho_corasick::MatchKind, being the default oneaho_corasick::MatchKind::LeftmostLongest.More atdoc/notebook/doc/literal_replacer.ipynbin the repository.Calling examplesfrompytextrust.replacerimportreplace_literal_patterns,MatchKindreplace_literal_patterns(literal_patterns=["uno","dos"],replacements=["1","2"],text_to_replace=["es el numero uno o el Dos yo soy el veintiuno"],is_bounded=True,case_insensitive=True,match_kind=MatchKind.LeftmostLongest)returns the replaced text and the number of replacements(['es el numero 1 o el 2 yo soy el veintiuno'], 2)EntitiesEntities are found by overlapping and have a hierarchichal folder structure.Literal entities: fast only literal based entities. Those entities are based in literal alternations, and
are built from a list of strings, is like matching (lit_1|...|lit_N)`. Can be:Private: only used by regex entities by composition. The only interest on them is for composition so those are only matched not finded.Public: calculated and reported. Those reports enforce that matched boundaries are\b, just if the literal matching where\b(lit_1|...|lit_N)\b.Tech note: positions reported by aho corasick should be mapped from byte to char position.Regex entities: a list of regex patterns, possibly containing literal entities calls with template language. For example ifmonthis a literal entity,
Then\d+ of \d+ of {{month}}is a possible entity. The regex entities that depend positively (no negative lookback or lookahead), only are searched on the texts where the literal entity has been found, minimizing computational weight.Feeding of entity matches:From python list of objects, where each object is equivalent to the file JSON loaded. Each object contains a fieldkindwith one of two values:reorlit.From local folder with folders:Structured hierarchically.Steps of entity recognition:Load the entity system:Deserialize all defined entities.BuildLiteralEntityPool. There are public and private literal entities:Private literal entitieswill not be reported only used internally by regex entities.Public literal entitieswill be reported as entities.NOTE: the bound of the literal public entities is calculated after all as Aho Corasick has not bound allowed.BuildRegexEntityPoolusing literals fromLiteralEntityPool, then there are two kinds of regex entitiesThe ones that use any literal entity.The ones that do not use any literal entity.Process texts and get entities:Get literal entity raw index matches.Literal-based regex entities perform find if the ordered set of matches of literal entities is satisfied from literal entities results.Non literal-based regex entities find is performed usingregex::RegexSetEnsemble together public literal entities, literal-based regex entities and non literal-based regex entities and give output.A pattern in a regex entity has two type of categorizations:Pattern that can be compiled atregexcrate:Pattern with at least one positive capture group related to a literal entity. Match will be decided by aho corasick and literal entity order. This is a regex wereentities::extract_required_template_structurethrows a non-empty vector.Pattern that does not fit the previous case, this pattern will be matched throughRegexSet. This is a pattern withentities::extract_required_template_structurethrowing an empty vector.Regex that can not be compiled byregexcrate will receive a direct find fromfancy_regexcrate. This pattern
receives an Error fromentities::extract_required_template_structure.Naming convention for entity files is:Calling examplesCICDThis repository pretends to be a perfect CICD example for a Python+Rust lib based onpyo3. Any suggestions (caching, badges, anything, ...) just let me know by issue :)Useful docLearning docThe Rust Programming LanguageRust CookBookRust by examplePyO3MaturinA comparison of regex enginesReference Rust pattern matching packageshttps://docs.rs/fst/latest/fst/, particularlyhttps://docs.rs/fst/latest/fst/#example-searching-multiple-sets-efficientlyfor entitieshttps://docs.rs/regex-automata/latest/regex_automata/https://docs.rs/aho-corasick/0.7.18/aho_corasick/https://docs.rs/regex-syntax/latest/regex_syntax/Performance adviceshttps://github.com/rust-lang/regex/blob/master/PERFORMANCE.md#unicode-word-boundaries-may-prevent-the-dfa-from-being-usedthere is no problem with using non-greedy matching or having lots of alternations in your regexhttps://github.com/rust-lang/regex/blob/master/PERFORMANCE.md#resist-the-temptation-to-optimize-regexesBenchmark by Rust regex author