Package: yaprak
Yet Another PRocessing and Analyzing KitThis is a small package for simplifying batch data processing and analysis.This project is licensed under the terms of the MIT license.InstallationpipinstallyaprakDocumentationDocumentationUse CaseYaprak can be used for running the same set of functions on a set of input files. For instance, when processing a dataset with multiple files each with time-series data, typically all the data should be preprocessed with the same set of filters before any other operation can be carried out. Similarly, the filtered data needs to be processed for a given application in a similar manner.Use ExampleTo use yaprak, in addition to the input files, at least two more files are needed:A Python file with a class derived from Yaprak.A JSON file for configuring a run (file paths, methods to be called (processes), and parameters to pass to these methods (referred to asprocess_specin the Python code, and any other user variables relevant to the use case)Class inheriting YaprakFirst, define a class derived from Yaprak (myClass in the example below). 4 methodsmustbe defined in this class:__init__(self, config): The initialization method has one parameterconfig, which is the name of the JSON configuration file. This method should also call the initialization method for Yaprak.load(self, file): This method describes how to load one input file (file). This is specific to the use case. For instance, in the example below, there is only one integer value in each file, so theload()method implements the loading of the file and extracts the integer value from it. In another case, the input files can be.csvfiles or binary files, and the load function should implement the relevant load operation.save(self, file): Similar to load this method describes how to save the processed data to an output file, and it is specific to the application.report(self): If there is any information to be reported pertinent to the current file, thereportfunction can be used for this purpose.Any other method that is needed to process the data, should also be described in this class (or derived from another class, here). These methods should include only one parameter other thanself, which isprocess_spec.For instance, theadd(self, process_spec)method below adds two numbers. One is from the process_spec (which is read from the parameter list for this method from the configuration file (process_spec['a1'])), and the other one is read from the input file (self.file_input).To use the class, only two steps are needed:yap = myClass('myconfig.json'): Define an instance (yap) of the derived class (myClass) passing the configuration file name (myconfig.json) as the parameter.yap.run(): Run the batch, which will sequentially read each file, execute all the processes described in the configuration file on each input file, report any information regarding the currently processed file, and save the output to an output file.fromyaprak.yaprakimportYaprakclassmyClass(Yaprak):def__init__(self,config):Yaprak.__init__(self,config)self.total=0self.product=0self.file_input=Nonedefload(self,file):withopen(file,'r')asf:self.file_input=int(f.readline())print(self.file_input)defsave(self,file):withopen(file,'w')asf:f.write("Sum is "+str(self.total)+"\n")f.write("Product is "+str(self.product))defreport(self):print('Finished processing')defadd(self,process_spec):self.total=process_spec['a1']+int(str(self.file_input))defmul(self,process_spec):self.product=process_spec['m1']*self.config['m_global']yap=myClass('myconfig.json')yap.run()JSON Configuration FileThe configuration file below (myconfig.json) is a standard JSON file and consists of the following sections:processes: This list has one entry for each process (i.e., method in the derived class) to be executed for all inputs. The processes in this list are executed in the given order. For each process, the keyprocessshould have the value of the name of the method (e.g. add, mul). The keyapplyis a Boolean and should be set totruefor processes to be executed andfalsefor processes to be skipped. Any other parameter (currently only numbers, and strings) to the method can also be passed as key-value pairs for each process. For instance, for theaddprocess below, an integer parametera1is given.This approach of configuring processes has two features, which can be achieved without any changes to the Python code:The order of the processes can be changed.The parameter values of the processes can be changed.The processes can be applied selectively.In addition to theprocesseslist, the following itemsmustalso exist in the JSON file:A list ofIDs, representing each file with an ID string (This could be the prefix of the file name for instance).A list of all the input files in the listinFileList, andA list of all the output files in the listoutFileList.TheinFileListandoutFileListmay define absolute paths. Alternatively, paths for input files (inPath) and output files (outPath) can be defined and the file names in the file lists can be relative.Any other parameter useful to the application can be configured in this file as key-value pairs and can be accessed within the derived class from theconfigdictionary.{"processes":[{"process":"add","a1":10,"apply":true},{"process":"mul","m1":20,"apply":true}],"m_global":100,"inPath":"./inputs/","outPath":"./outputs/","IDs":["1","2","3"],"inFileList":["1.dat","2.dat","3.dat"],"outFileList":["1.txt","2.txt","3.txt"]}