Package: pyrsona
pyrsonaText data file validation and structure management using thepydanticandparsePython packages.InstallationInstall usingpip install pyrsona.A Simple ExampleFor the text fileexample.txt:operator name: Jane Smith
country: NZ
year: 2022

ID,Time,Duration (sec),Reading
1,20:04:05,12.2,2098
2,20:05:00,2.35,4328The followingpyrsonafile structure model can be defined:frompyrsonaimportBaseStructurefrompydanticimportBaseModelfromdatetimeimporttimeclassExampleStructure(BaseStructure):structure=("operator name:{operator_name}\n""country:{country}\n""year:{}\n""\n""ID,Time,Duration (sec),Reading\n")classmeta_model(BaseModel):operator_name:strcountry:strclassrow_model(BaseModel):id:inttime:timeduration_sec:floatvalue:floatTheread()method can then be used to read the file, parse its contents and validate the meta data and table rows:meta,table_rows,structure_id=ExampleStructure.read("example.txt")print(meta)#> {'operator_name': 'Jane Smith', 'country': 'NZ'}print(table_rows)#> [{'id': 1, 'time': datetime.time(20, 4, 5), 'value': 2098.0}, {'id': 2,# 'time': datetime.time(20, 5), 'value': 4328.0}]print(structure_id)#> ExampleStructureWhat's going on here:Thestructureclass attribute contains a definition of the basic file structure. This definition includes the meta data lines and table header lines. Any variable text of interest is replaced with curly brackets and a field name, E.g.'{operator_name}', while any variable text that should be ignored is replaced with empty curly brackets, E.g.'{}'. Thestructuredefinition must contain all spaces, tabs and new line characters in order for a file to successfully match it. The named fields in thestructuredefinition will be passed tometa_model.meta_modelis simply apydantic modelwith field names that match the named fields in thestructuredefinition. All values sent tometa_modelwill be strings and these will be converted to the field types defined inmeta_model. Custompydantic validatorscan be included in themeta_modeldefinition as per standard pydantic models.row_modelis also apydantic model. This time the field names do not need to match the header line in thestructuredefinition; however, therow_modelfields do need to be provided in thesame order as the table columns. This allows the table column names to be customised/standardised where the user does not control the file structure itself. Again, custompydantic validatorscan be included in therow_modeldefinition if required.Another ExampleShould the file structure change at some point in the future a new model can be created based on the original model. This is referred to as asub-model, where the original model is theparentmodel.Given the slightly modified file structure ofnew_example.txt:operator name: Jane Smith
country: NZ
city: Auckland
year: 2022

ID,Time,Duration (sec),Reading
1,20:04:05,12.2,2098
2,20:05:00,2.35,4328Attempting to parse this file using the originalExampleStructuremodel will raise aPyrsonaErrordue to the addition of the'city: Auckland'line. In order to successfully parse the file and capture the new'city'field the followingsub-modelshould be defined.frompyrsonaimportBaseStructurefrompydanticimportBaseModelfromdatetimeimporttimeclassNewExampleStructure(ExampleStructure):structure=("operator name:{operator_name}\n""country:{country}\n""city:{city}\n""year:{}\n""\n""ID,Time,Duration (sec),Reading\n")classmeta_model(BaseModel):operator_name:strcountry:strcity:strExampleStructureis still used as the entry point; however,pyrsonawill attempt to parse the file using anysub-modelsthat exist (in this caseNewExampleStructure) before usingExampleStructureitself.meta,table_rows,structure_id=ExampleStructure.read("new_example.txt")print(meta)#> {'operator_name': 'Jane Smith', 'country': 'NZ', 'city': 'Auckland'}print(table_rows)#> [{'id': 1, 'time': datetime.time(20, 4, 5), 'value': 2098.0}, {'id': 2,# 'time': datetime.time(20, 5), 'value': 4328.0}]print(structure_id)#> NewExampleStructureWhat's going on here:A newpyrsonafile structure model is defined based on the originalExampleStructuremodel. This means thatstructure,meta_modelandrow_modelwill be inherited fromExampleStructure. This also provides a single entry point (I.e.ExampleStructure.read()) when attempting to read the different file versions.structureandmeta_modelare redefined to include the new"city: Auckland"meta data line. Alternatively, the originalmeta_modelinExampleStructurecould have been updated to include anoptionalcityfield.Post-processorsIt is sometimes necessary to modify some of the data following parsing by themeta_modelandrow_model. Two post-processing methods are available for this purpose.Using theExampleStructureclass above,meta_postprocessorandtable_postprocessorstatic methods are defined for post-processing the meta data and table_rows, respectively:classExampleStructure(BaseStructure):# Lines omitted for brevity@staticmethoddefmeta_postprocessor(meta):meta["version"]=3returnmeta@staticmethoddeftable_postprocessor(table_rows,meta):# Add a cumulative total and delete the "id" field:total=0forii,rowinenumerate(table_rows):total+=row["value"]row["total"]=totaldel(row["id"])table_rows[ii]=rowreturntable_rowsThe meta data and table_rows are now run through the post-processing stages before being returned, resulting in the following changes:A newversionfield is added to the meta data.Theidfield is deleted from the table_rows and a cumulative total field is added.meta,table_rows,structure_id=ExampleStructure.read("example.txt")print(meta)#> {'operator_name': 'Jane Smith', 'country': 'NZ', 'version': 3}print(table_rows)#> [{'time': datetime.time(20, 4, 5), 'duration_sec': 12.2, 'value': 2098.0,# 'total': 2098.0}, {'time': datetime.time(20, 5), 'duration_sec': 2.35, 'value': 4328.0,# 'total': 6426.0}]print(structure_id)#> NewExampleStructureExtra detailsAll meta lines MUST be includedWhile theparsepackage allows a wildcard'{}'to be used to ignore several lines this can cause a named field to be unexpectedly included in the wildcard section.pyrsonatherefore checks for the presence of a new line character'\n'in the named field values and fails if one is found.Sub-sub-modelsCalling theread()method will first build a list ofpyrsonafile structure models from theparentmodel down.Anysub-modelsof theparentmodel will themselves be checked forsub-models, meaning that every model in the tree below theparentmodel will be used when attempting to parse a file.Each branch of models will be ordered bottom-up so that the deepest nested model in a branch will be used first. Theparentmodel will be the final model used if all others fail.Model namesTheread()method returns astructure_idvariable that matches the model name. Thisstructure_idcan be useful when creating automated tests that sit alongside thepyrsonamodels as it provides a mechanism for confirming that a text file was parsed using the expectedpyrsonamodel where multiplesub-modelsexist.As the number ofsub-modelsgrows a naming convention becomes more important. One option is to set the names of anysub-modelsto a random hexadecimal value prefixed with a single underscore (in case the value begins with a number), E.g.'_a4c15356'. The initial underscore will be removed from model name when returning thestructure_idvalue.parseformatsTheparsepackage allows format specifications to be included alongside the fields, E.g.'{year:d}'. While including these format types in the structure definition is valid, more complex format conversions can be made usingmeta_model. Keeping all format conversions inmeta_modelmeans that all conversions are defined in one place.