Package: sficopaf
A declarative framework to read complex objects made of several files,
using a user-provided library of unitary file parsers.This library provides aframework, not a specific parser ! Although it
comes with a couple unitary file parsers as an example, it is intended
for users that already know how to parse their various files
independently, but who are looking for a higher-level tool to read
complex objects made of several files/folders and potentially requiring
to combine several parsers.Typical use cases of this library :read collections of test cases on the file system - each test case
being composed of several files (for example 2 ‘test inputs’ .csv
files, 1 ‘test configuration’ .cfg file, and one ‘reference test
results’ json file)more generally, read complex objects, for example made of several csv
files (timeseries + descriptive data), combinations of csv and
xml/json files, configuration files, etc.Main featuresDeclarative: youfirstdefine the type of objects to parse - by
creating a class -, then you useparse_collectionorparse_itemon the appropriate folder or file path.Supports several unitary file parsers for the same object type.
Thanks to a combined{Type+Extension}registration, you register
unitary file parsers for a given object typeandfor a given file
extension (for examplestr+.txt). This allows users to
register several parsers for the same object type, supporting various
formats represented by the extensions.Supports complex classes: the main interest of this framework is
its ability to define complex classes that spans across several
files. For example, aMyTestCaseclass that would have two fieldsinput: DataFrameandexpected_output: str. The class
constructor is introspected in order to find therequiredandoptionalfields and their names. Fields may be objects or
collections (that should be declared with thetypingmodule such
asDict[str, Foo]) in order for the framework to keep track of
the underlying collection types)Recursive: fields may themselves be collections or complex types.
In which case they are represented by several files.Supportstwo main file mapping flavours:flat, where all items are represented as files in the same
folder (even fields and collection elements)wrapped, where all items that represent collections or complex
types are represented by folders, and all ready-to-parse items are
represented by files.Safe: files are opened and finally closed by the framework, your
parsing function may exit without closingLazy-parsing: TODO, a later version will allow to only trigger
parsing when objects are read, in the case of collectionsInstallationRecommended : create a clean virtual environmentWe strongly recommend that you use condaenvironmentor pipvirtualenv/venvin order to better manage packages. Once you are in
your virtual environment, open a terminal and check that the python
interpreter is correct:(Windows)>wherepython(Linux)>whichpythonThe first executable that should show up should be the one from the
virtual environment.Installation stepsThis package is available onPyPI. You may therefore usepipto
install from a release>pipinstallsficopafUninstallingAs usual :>pipuninstallsficopafExamplesBasic: theop_functiontest casesa - example overviewIn this very simple example, we will parse ‘test cases’ for an imaginary
function that performs operations :op_function(a:int, b:int, operation:str ='+')->output:int.Each of our ‘test case’ items will be made of several things: *
mandatory input data (here,aandb) * optional configuration
(here,operation) * mandatory expected result (here,output)We would like these things stored in separate files. Typically the
reason is that you will want to separate the various formats that you
wish to use:csv,xml,json…In this first example we decide to store all items in separate files. So
our data folder structure looks like this:test_cases├──case1│├──input_a.txt│├──input_b.txt│└──output.txt├──case2│├──input_a.txt│├──input_b.txt│├──options.txt│└──output.txt└──case3├──input_a.txt├──input_b.txt├──options.cfg└──output.txt(this data is available in the source code of this project, in foldertest_data/demo)Note that the configuration file is optional. Here, onlycase2andcase3will have a non-default configuration.You may also have noticed that the configuration file is present with
two different extensions :.txt(in case2) and.cfg(in case3).
This framework allows to register several file extensions for the same
type of object to parse. Each extension may have its own parser
function.b - base types and parsers registration - simpleFirst import the package and create a root parser.importsficopafassfroot_parser=sf.RootParser()Then register a parser function for all items that will be represented
assinglefiles.In this example, all inputs and outputs areintso we create a first
function to parse anintfrom a text file:fromioimportTextIOBasedefparse_int_file(file_object:TextIOBase)->int:integer_str=file_object.readline()returnint(integer_str)and we register it:root_parser.register_extension_parser(int,'.txt',parse_int_file)Note that the parsing framework automatically opens and closes the file
for you, even in case of exception.c - base types and parsers registration - proxies and item collectionsWe also need to be able to read aconfiguration, that we would like
to be aDict[str, str]in order for it to later contain more
configuration options.Unfortunately this type is an ‘item collection’ type (dict, list, set,
tuple), so we have to create our own custom wrapper class, in order for
the framework not to think that it has to read each<key, value>pair as a separate file. Indeed by default the framework considers that
all ‘item collection’ types are collections of files.classOpConfig(dict):"""
An OpConfig object is a Dict[str, str] object
"""def__init__(self,config:Dict[str,str]):super(OpConfig,self).__init__()self.__wrapped_impl=config# here you may wish to perform additional checks on the wrapped objectunrecognized=set(config.keys())-set('operation')iflen(unrecognized)>0:raiseValueError('Unrecognized options : '+unrecognized)# Delegate all calls to the implementation:def__getattr__(self,name):returngetattr(self.__wrapped_impl,name)This is named a dynamic proxy. TheOpConfigclass extends thedictclass, but delegates everything to the underlyingdictimplementation provided in the constructor.Note: this pattern is very useful to use this library, even if the
underlying class is not an ‘item collection’ type. Indeed, this is a
good way to create specialized versions of generic objects created by
your favourite parsers. For example two ``pandas.DataFrame`` might
represent a training set, and a prediction table. Both objects, although
similar (both are tables with rows and columns), might have very
different contents (column names, column types, number of rows, etc.).
We can make this fundamental difference appear at the parsing level, by
creating two classes.Back to our example, we propose two formats for theOpConfig: * one.txtformat where the first row will directly contain the value for
theoperation* one.cfgformat where the configuration will be
available in aconfigparserformat, and for which we want to reuse
the existing parser.fromtypingimportDictdefparse_configuration_txt_file(file_object:TextIOBase)->OpConfig:return{'operation':file_object.readline()}defparse_configuration_cfg_file(file_object:TextIOBase)->OpConfig:importconfigparserconfig=configparser.ConfigParser()config.read_file(file_object)returndict(config['main'].items())Once again, we finally register the parsers:root_parser.register_extension_parser(OpConfig,'.txt',parse_configuration_txt_file)root_parser.register_extension_parser(OpConfig,'.cfg',parse_configuration_cfg_file)d - main complex type and final parsing executionFinally, we define theOpTestCaseobject. Its constructor should
reflect the way we want to dispatch the various pieces of information in
separate files, as well as indicate the files the are optional:classOpTestCase(object):def__init__(self,input_a:int,input_b:int,output:int,options:OpConfig=None):self.input_a,self.input_b,self.output=input_a,input_b,outputifoptionsisNone:self.op='+'else:self.op=options['operation']def__str__(self):returnself.__repr__()def__repr__(self):returnstr(self.input_a)+' '+self.op+' '+str(self.input_b)+' =? '+str(self.output)Parsing is then straightforward: simply provide the root folder, the
object type, and the file mapping flavour.results=root_parser.parse_collection('./test_data/demo',OpTestCase)The output shows the created test case objects:pprint(results){'case1':1+2=?3,'case2':1+3=?4,'case3':1-2=?-1}Advanced topicsFlat modeIn our example we used folders to encapsulate object fields and item
collections. This isflat_mode=False. Alternatively you may wish to
setflat_mode=True. In this case the folder structure should be
flat, as shown below. Item names and field names are separated by a
configurable character string. For example to parse the following tree
structure:.├──case1--input_a.txt├──case1--input_b.txt├──case1--output.txt├──case2--input_a.txt├──case2--input_b.txt├──case2--options.txt├──case2--output.txt├──case3--input_a.txt├──case3--input_b.txt├──case3--options.cfg└──case3--output.txtyou’ll need to callresults=root_parser.parse_collection('./test_data/demo_flat',OpTestCase,flat_mode=True,sep_for_flat='--')pprint(results)Note that the dot may be safely used as a separator too.Item collectionsThe parsing framework automatically detects any object that is of a
‘item collection’ type (dict,list,set, and currentlytupleis not supported). These types should be well defined
according to thetypingmodule: for example let’s imagine that we
have an additionalinput_cin our example, with typetyping.Dict[str, typing.List[int]].classOpTestCaseColl(object):def__init__(self,input_a:int,input_b:int,output:int,input_c:Dict[str,List[int]]=None,options:OpConfig=None):self.input_a,self.input_b,self.output=input_a,input_b,outputifoptionsisNone:self.op='+'else:self.op=options['operation']self.input_c=input_corNonedef__str__(self):returnself.__repr__()def__repr__(self):returnstr(self.input_a)+' '+self.op+' '+str(self.input_b)+' =? '+str(self.output)+' '+str(self.input_c)Forflat_mode=True: * dictionary keys are read from the text
behind the separator afterinput_c(so below,keyAandkeyBare the key names) * list items are indicated by any character
sequence, but that sequence is not kept when creating the list object
(below,item1anditem2will not be kept in the output list).├──case1--input_a.txt├──case1--input_b.txt├──case1--output.txt├──case2--input_a.txt├──case2--input_b.txt├──case2--options.txt├──case2--output.txt├──case3--input_a.txt├──case3--input_b.txt├──case3--input_c--keyA--item1.txt├──case3--input_c--keyA--item2.txt├──case3--input_c--keyB--item1.txt├──case3--options.cfg└──case3--output.txtresults=root_parser.parse_collection('./test_data/demo_flat_coll',OpTestCaseColl,flat_mode=True,sep_for_flat='--')pprint(results['case3'].input_c)Results:{'keyA':[-1,-1],'keyB':[-1]}Forflat_mode=False: * we already saw that complex objects are
represented by folders (for examplecase1,case2andcase3)
* item collections are, too :input_cis a folder * dictionary
keys are read from the files or folder names (so below,keyAandkeyBare the key names, and since their content is a complex or
collection object they are folders themselves) * list items are
indicated by files or folders with any name, but that name is not kept
when creating the list object (below,item1anditem2are not
kept in the output list, only their contents is).├──case1│├──input_a.txt│├──input_b.txt│└──output.txt├──case2│├──input_a.txt│├──input_b.txt│├──options.txt│└──output.txt└──case3├──input_a.txt├──input_b.txt├──input_c│├──keyA││├──item1.txt││└──item2.txt│└──keyB│└──item1.txt├──options.cfg└──output.txtresults=root_parser.parse_collection('./test_data/demo_coll',OpTestCaseColl,flat_mode=False)pprint(results['case3'].input_c)Results:{'keyA':[-1,-1],'keyB':[-1]}Finally, note that it is not possible to mix collection and
non-collection items together (for example,Union[int, List[int]]is
not supported)See AlsoCheckherefor
other parsers in Python, that you might wish to register as unitary
parsers to perform specific file format parsing (binary, json,
custom…) for some of your objects.DevelopersPackagingThis project usessetuptools_scmto synchronise the version number.
Therefore the following command should be used for development snapshots
as well as official releases:pythonsetup.pyegg_infobdist_wheelrotate-m.whl-k3Releasing memotwineuploaddist/*-rpypitesttwineuploaddist/*