Package: pyreparse
PyReParsePyReParse is a library the helps to ease the development processes of parsing
complex reports, which have enough structure for effective regular expression
processing.PyReParse is a library that helps one create parsing engines for formed text reports. I had a such a need when I was tasked to parse a Financial Institution's archived transaction reports where the databases that held this data no-longer existed. So the data in the report forms was the only data available to re-create the original database. Thus, regular-expressions were used to find and capture certain field values, and validation calculations were needed to ensure that the data going into the database was complete and accurate.Benefits...The benefits of using PyReParse include...The use of a standard data structure for holding regular expressions.
Associated to the regexp are additional flags and fields that help to reduce the
number of times a given regexp is executed.Regexp processing can be expensive. The goal is to run regexp matches only when they
are needed. So if you know that the pattern for regexp A always occurs before
regexp B, you can use the data structure to specify that regexp B should not
be used until after regexp A triggers.All regular expressions and their associated properties are in one data structure.Additional benefits include the ability to crosscheck a non-matching line which
a simpler regexp that can catch lines that should have matched but did not,
due to a need to tweak the main regexp, or possibly a corrupt input line.Logic for counting report lines and sections within a report.PyReParse uses named-capture-groups and returns captured values in a dictionary. This eases the ability to capture values for transformation and storage.One can associate a RegExp pattern to a callback so that one can perform custom calculations, validations, and transforations to the captured values of interest.Basic Usage PatternSet up the named-regexp-pattern(s) with named-capture-groups data structure, along with associated properties (seeexamplein test code...):Flags:Only onceStart of SectionTrigger ON/OFFtrigger matching on or off based on another named regexpOptional Quick-Check RegExp)If the current named-regexp fails to match the given line. The quick-check regexp is run, and if a match occurs, warns that a regexp may have missed a line. So either the named-regexp is wrong, or the quick-check is produced a false positive.callback(<PyReParse_Instance>, <regexp_pattern_name>)On match, run the stated callback function to perform validations and processing logic. In fact, all processing logic can be implemented within callbacks.The Callback function is called when a match occurs and after fields have been captured.Callbacks can be used for field validation and event correlation, as the PyReParase instance (which contains the states of all regexp/fields), is available to the callback.Write the document processing logic...If all processing logic is implemented as callbacks, the main logic would look like...(TODO: Callbacks implemented soon...)# Import PyRePrasefrompyreparseimportPyReParseasPRP# Define callback functions...defon_pattern001(prp_instance,pat_name):iffld_name!='pattern001':print(f'Got wrong pattern name [{pat_name}].')# Define our Regular Expression Patterns Data Structure...regexp_pats={'pattern_001':{InDEX_RE:'^Test\s+Pattern\s+(?P<pat_val>\d+)'<INDEX_RE...>:'value',<INDEX_RE...>:'value',INDEX_RE_CALLBACK:on_pattern_001...},...}# Create and Instance of PyRePraseprp=PRP(<regexp_pats>)# Open the input file...withopen(file_path,'r')astxt_file:# Process each line of the input file...forlineintxt_file:# This call on prp.match(<input_line>) to process the line# against our data structure of regexp patterns.match_def,matched_fields=prp.match(line)With or without Callback, you can trigger logic when name-regexp fields match using (seetestsas an example)......# Open the input file...withopen(file_path,'r')astxt_file:# Process each line of the input file...forlineintxt_file:# This call on prp.match(<input_line>) to process the line# against our data structure of regexp patterns.pattern_name,matched_fields=prp.match(line)# Then, we have logic based on which pattern matched,# and/or values in captured fields...ifmatch_def[0]='pattern_001':...elifmatch_def[0]='pattern_002':...Please check outpyreparse_example.py, you can used this code as a template to guide you in the creation of your own parsing engine.The PyReParse Data Structure of PatternsFlagsCoding Triggers...A trigger is a line of logic that references counters or pattern-names. Triggers can use the full depth of python expressions, and are compiled to a call back function for efficiency. The purpose of the trigger is to simply return true or false. For thetrigger-on, the expression should return true if the RegExp Pattern is to be evaluated against the current and following lines. Fortrigger-off, it should evaluate to True so that it is not evaluated for the current and subsequent lines.< Counters >Counters are synbolic names that are enclosed in Less-Than and Greater-Than signs.The following is te current list of supported report counters...<REPORT_LINE>The<REPORT_LINE>counter increments by 1 for each line that thematch()method is called on.<SECTION_NUMBER>The<SECTION_NUMBER>counter increments by 1 for each time amatch()occurs on a pattern that has the flagPyReParse.FLAG_NEW_SECTION.<SECTION_LINE>The<SECTION_LINE>counter increments by 1 for each line that thematch()method is called on that is part of a section.All counters start at 0.{Pattern_Names}Pattern names are symbolic references to the RegExp Patterns in the current PyReParse data structure.
Each Pattern may be associated to triggers that tell the matcher when or when not to execute a match on a given pattern. Triggers improve the efficiency of RegExp processing by reducing the number of Regular Expressions that are executed on any given line. This can be very effective when processing a huge number of documents. The pattern name evaluates to True if the pattern has been matched, and False if the pattern as not been matched since the last "NEW_SECTION", A "New Section" occurs when a pattern that has the flagPyReParse.FLAG_NEW_SECTIONmatches the current line, and it triggers the reset of section counters.Coding Callbacks...You may also code callbacks that are executed when a pattern matches. The callback function is called when a pattern matches, and after the fields have been captured. The callback function is passed the PyReParse instance, and the name of the pattern that matched. The callback function can then use the PyReParse instance to access any currently captured fields, and perform any processing logic field value updates.## License...Apache 2.0