Package: xileh
XilehModule for optimizing pipeline with genetic algorithmsGeneral structureThe idea of this is to abstract complex processing pipelines based of thescipy.pipelinesetup,
but with extended data capacities going beyond a feature matrixXand potential labels / regression
targetsy.
For this end, the abstraction is based of two main components, thepipelineand thepipelinedata.
The pipeline consists of functions acting upon the pipeline data. Functions are added to the pipeline
during initialization and the pipeline is later evaluated given apipelinedataentity.SchematicWhy build something new?As with every project, the first question should be, if there is anything already available which serves the purpose...
If looking at libraries in two main dimensions, i.e.builting functionalityandspecificity / restrictions, we would end up with the following graph IMHO.Usually one would not want to end up in the lower left quadrant of such a split, but herexilehaims to be exactly there. I.e. provide only a very limited scope of functionality built-in, but allow for maximal freedom of customization by being as general as possible.Take for example thesklearn.Pipelinewhichxilehis motivated on. It is great to build, store and evalute all kind of machine learning piplines, but it is very specific in which type (shape) of data it can process and how it its doing it (fit/transform). Of coursesklearnas a whole has a huge bunch of functionaly implemented but then would need to have different processing steps glued together by custom code, no longer allowing for this single object describing the full pipeline.Other libraries likehydraare based of (a single) config file(s) which provide a good overview and single point of truth for the processing, but hence this again requires a quite specific format and might thus hinder rapid prototyping.So xileh deliberatly wants to:impose as little restrictions to you workflow as possible (dealing with arbitrary data objects and python functions)integrate easily with a function's based workflow during developmentprovide a single source of truth for the processingenable reuseabilty of whole pipelines by strongly motivating compositionGetting StartedInstallationClone themasterbranch, or for compatebility of storing basicmnedata entities, use themnebranchgitclonegit@github.com:bsdlab/xileh.gitFrom within the folder, install to your pip repo or just link, depeding on your liking.pipinstall.# orpipinstalldevelopPipelinedataThexileh.core.pipelinedata.xPDataimplements the pipelinedata container, which contains of three elements:A data entity, usually an array or array-like. Possible also a list of otherxPDataentities.Header information as a dictionary containing information about the whole data set used to describe the data in 1.Meta information as a dictionary containing meta data about 1., such as configuration info for the processing, aggregated measure, or per record meta data. Note that if a dictionary value has a shape property, it is assumed that it should contain a per record (i.e. 1 meta elements per value) info and is thus checked for its length upon initialization. So to provide an array which does not match the length of 1. you have to pack it into an object without shape property first.All 1.-3. are potentially extended by processing steps within the pipeline. However, processing should in general not change any entities provided during initialization, but create new copies. There might be reasons however, where a change is required (e.g. memory constraint). Overwriting is thus not restricted.Initializing a containerfromxilehimportxPData# single data container -> nothing can be addedpdata=xPData("somedata",name='mycontainer')# a container which can be dynamically extended, if the data arg is of type listpdata=xPData([],name='mycontainer')# with header or meta datapdata=xPData([],name='cont1',header={'description':'some useful container'})# nested initializationpdata=xPData([xPData([1,2,3,4],name='array_1')],name='outer_container_1')#### Get and set`pytho# those getters are epdata.get_by_name('array_1')==pdata['array_1']==pdata.array_1# setting can be done accordinglypdata.get_by_name('array_1').data=np.ones(3)pdata['array_1'].data=np.ones(3)pdata.array_1.data=np.ones(3)# or overwriting whole containerstd=xPData(np.ones(3),name='new_array')pdata.get_by_name('array_1')=tdpdata['array_1']=tdpdata.array_1=tdAdd# basic way of adding a new container via get by name. Without the# create_if_missing=True, a None value would be returnednewc=pdata.get_by_name('newcont',create_if_missing=True)newc.data=np.ones(123)newc.header['description']='A few ones for important reasons'# or more concisepdata.add(np.ones,'newcont',header={'description':'A few ones for important reasons'})Deletepdata.delete_by_name('newcont')Some UtilitySimply calling the container prints some overview info reflecting the hierarchy of stored data and indicating :  if it is aleaf-container.In[10]:pdata=xPData(...:[...:xPData([1,2,3,4],name='array_1'),...:xPData('string',name='string_1'),...:xPData(pd.DataFrame(),name='df1')...:],...:name='outer_container_1'...:)In[11]:pdataOut[11]:xPDataobjectat0x7f8b023b3880-withsize32outer_container_1:|array_1:list|string_1:<class'str'>|df1:<class'pandas.core.frame.DataFrame'>'--------------------Only container names:In[12]:pdata.get_container_names()Out[12]:['outer_container_1','array_1','string_1','df1']Dict of containers and types:In[15]:pdata.get_containers()Out[15]:{'outer_container_1':[{'array_1':[]},{'string_1':str},{'df1':pandas.core.frame.DataFrame}]}In[14]:pdata.gc()Out[14]:{'outer_container_1':[{'array_1':[]},{'string_1':str},{'df1':pandas.core.frame.DataFrame}]}Save and loadIn[16]:pdata.save('./test_container')In[17]:fromxileh.core.pipelinedataimportfrom_containerIn[18]:newc=from_container('./test_container/')In[19]:newcOut[19]:xPDataobjectat0x7f8b024e71c0-withsize32outer_container_1:|array_1:list|string_1:<class'str'>|df1:<class'pandas.core.frame.DataFrame'>'--------------------PipelinesPipelines are concatenations of functions operating onxPDataobjects. Therefore, all pipeline functions shoud follow the pattern outlined belowFunctions signaturedefadd_data_entity(pdata,name='myname',**kwargs):"""Add a simply data entity to the containerNOTE: A valid function for processing within a pipeline will have onlyone arg, which is the xPData objects that is processed, but can haveany number of kwargs, which will also be logged with the pipeline object"""pdata.add([1,2,3]*size,name)# Note, every function used in a pipeline needs to return the pdata object# --> this is to make explicit, that the function modifies the pdata objectreturnpdataInitializationpl=xPipeline('test_pipeline',log_eval=False)pl.add_steps(('add_to_data',add_data_entity),('add_to_data_2',add_data_entity,{'name':'another_test'}),)RepresentationCalling the pipeline object will list on the stepsIn[5]:plOut[5]:<xileh.core.pipeline.xPipelineobjectat0x7f877d135cf0>Pipelinename:test_pipelineSteps:->'add_to_data'->'add_to_data_2'Calling for the.stepsattribute will also show the internal list of tuples (<name>,<function>,<kwargs_dict>)In[12]:pl.stepsOut[12]:[('add_to_data',<function__main__.add_data_entity(pdata,name='myname',**kwargs)>,{}),('add_to_data_2',<function__main__.add_data_entity(pdata,name='myname',**kwargs)>,{'name':'another_test'})]It is also possible to look just at a specific step by looking for its nameIn[14]:pl.get_step('add_to_data_2')Out[14]:(('add_to_data_2',<function__main__.add_data_entity(pdata,name='myname',**kwargs)>,{'name':'another_test'}),1)Evaluating the pipelinePipelines use the.eval()method to process datapdata=xPData([],name='outercontainer')pl.eval(pdata)# will process the steps 'add_to_data' and 'add_to_data_2'Reuse in other modulesIf pipelines are defined on a global scope in a script or module, you can simple import it:fromxilehimportxPData,xPipelinefrommy_module.my_scriptimportmy_plaspre_pl# ================================= Functions =================================defprint_foo(pdata):fornameinpdata.get_container_names():print(f"{name}-{pdata[name].data}")# ================================= Pipeline ==================================pl=xPipeline('test_pipeline',log_eval=False)pl.add_steps(('print_before',print_foo),*pre_pl.steps,('print_after',print_foo))ModificationSteps can either be replaced completelyIn[17]:deffoo(pdata,k='abc'):...:returnpdata...:In[18]:pl.replace_step('add_to_data',('new_step',foo))In[19]:plOut[19]:<xileh.core.pipeline.xPipelineobjectat0x7f877d135cf0>Pipelinename:test_pipelineSteps:->'new_step'->'add_to_data_2'->'test_step'or their kwargs can be modifiedIn[20]:pl.add_steps(('test_step',foo))In[21]:pl.set_step_kwargs('test_step',k='cde')In[22]:pl.stepsOut[22]:[('add_to_data',<function__main__.add_data_entity(pdata,name='myname',**kwargs)>,{}),('add_to_data_2',<function__main__.add_data_entity(pdata,name='myname',**kwargs)>,{'name':'another_test'}),('test_step',<function__main__.foo(pdata,k='abc')>,{'k':'cde'})]and of course they can be deletedIn[23]:pl.remove_step('new_step')In[24]:plOut[24]:<xileh.core.pipeline.xPipelineobjectat0x7f877d135cf0>Pipelinename:test_pipelineSteps:->'add_to_data_2'->'test_step'In[25]:pl.remove_steps(['add_to_data_2','test_step'])In[26]:plOut[26]:<xileh.core.pipeline.xPipelineobjectat0x7f877d135cf0>Pipelinename:test_pipelineSteps:->Early StoppingStopping an evaluation early can be achieved by settingearly_stopparameter
within thepdata.header. This would be done in any pipeline function
which you would want to trigger an early stop (e.g. you load data and nothing
is found -> skip all processing).pdata.header['early_stop']=TrueWorking with xilehA general example of how a script could be structured is provided in this templateTemplate