Package: adcpipeline
ADC PipelineThere are a lot of different steps involved in data science projects. For example: fixing data quality issues, feature engineering, parameter tuning and reporting your results. Although these steps are often approximately the same, the exact approach per step isn't. The data and the goal of the project determine the way you manipulate your data and what model you need. In turn, your model choice determines what kind of parameter tuning you need to do and how you are going to present your results. In other words, there are a lot of data-science-paths to walk. The more you program, the more you might get drowned in an ever increasing amount of if-statements, giving the feeling that you lose grip on the structure in your project. This package aims to solve that problem, by offering a more structured way of working.InstallationYou can install with pip:pip install adcpipelineBasic principlesTo structure your project, you need to follow three steps:Build your ownPipelineclass.Loading your (run) configuration.Running the pipeline.Below, each step will be explained.1. Build your ownPipelineclassFrom theadcpipelinepackage import thePipelineBaseclass:from adcpipeline import PipelineBaseAnd build your ownPipelineclass by inheriting from PipelineBase:class Pipeline(PipelineBase):
    passThis doesn't do anything yet, so let's add a few steps in ourPipeline. We do this by adding methods we want to execute when we run the Pipeline. The example below adds three methods to this specificPipeline:class Pipeline(PipelineBase):
    def print_text_from_argument(self, text='asfd'):
        print(text)

    def print_predefined_text(self):
        print('Predefined text')

    def n_times_squared(self, value: int, n: int):
        result = value
        for i in range(0, n):
            result = result ** 2
        print(f'Squaring the number {value} for {n} times in a row gives = {result}')2. Loading your (run) configuration.When we want to instantiate thePipeline, we need to pass the data as an argument (df) and we need to pass our run configuration as an argument (method_settings):p = Pipeline(df=data, method_settings=method_settings)The variabledatacan be any data, as long as it is a Pandas DataFrame. Themethod_settingsvariable is a list containing dictionaries, which define (in order) how all the methods are going to be executed once ourPipelineruns. Each dictionary contains the method (name) that needs to be called. The values are a dictionary of arguments (names) with their corresponding argument value that is going to be passed to the method. An example will make things clear:method_settings = [
    {'print_text_from_argument': {'text': 'This is the text passed to the method'}},
    {'print_text_from_argument': {'text': 1}},
    {'print_predefined_text': None},
    {'n_times_squared': {'value': 2, 'n': 2}},
    {'print_text_from_argument': {'text': 'Same method is called again, but later in the pipeline'}}
]Here we see that the methodprint_text_from_argumentis called two times with atextargument. Thistextargument is different each time. After that the other two methods are called and lastly,print_text_from_argumentis called one last time.Themethod_settingsas defined in the example above takes up a lot of lines and every time we make an additionalmethod_settings, we get more lines of code. It is therefore recommended to load themethod_settingsfrom a configuration file instead. You can define your pipeline settings in a.yamlfile and let the pipeline class load this file:p = Pipeline.from_yaml_file(df=data, path=f'{root_dir}/configs/<YOUR_METHOD_SETTINGS>.yaml')The.yamlfile would then look like this:pipeline:
  - print_text_from_argument: {text: 'This is the text passed to the method'}
  - print_text_from_argument: {text: 1}
  - print_predefined_text:
  - n_times_squared: {value: 2, n: 2}
  - print_text_from_argument: {text: 'Same method is called again, but later in the pipeline'}3. Running the pipeline.Withmethod_settingsdefined in step 2, we can now run ourPipeline:p.run()And that's it! By making multiplemethod_settingswe can define several ways to run ourPipeline, without altering any of our code or writing any if statement. For example, during exploratory data analysis, it might be nice to try different things without constantly changing our code. We could then do something along the following lines:p1 = Pipeline.from_yaml_file(df=data, path=f'{root_dir}/configs/<YOUR_METHOD_SETTINGS_1>.yaml')
p2 = Pipeline.from_yaml_file(df=data, path=f'{root_dir}/configs/<YOUR_METHOD_SETTINGS_2>.yaml')
p3 = Pipeline.from_yaml_file(df=data, path=f'{root_dir}/configs/<YOUR_METHOD_SETTINGS_3>.yaml')

p1.run()
p2.run()
p3.run()Where eachYOUR_METHOD_SETTINGS_<N>.yamldefines themethod_settingsperPipeline.
Alternatively the pipeline ships with arun_or_load()method, which can save and load the result of a pipeline from
a .pkl file. This can be useful if you did not change the content of the pipeline, but need to rerun your script.method_settings = [
    {'print_text_from_argument': {'text': 'This is the text passed to the method'}},
    {'print_text_from_argument': {'text': 1}},
    {'print_predefined_text': None},
    {'n_times_squared': {'value': 2, 'n': 2}},
    {'print_text_from_argument': {'text': 'Same method is called again, but later in the pipeline'}}
]
p = Pipeline(df=data, method_settings=method_settings, filename='my_pipeline')
p.run()  # Executes the pipeline, saves the results in cache/my_pipeline.pkl
# Some other code
p.run_or_load()  # Does not execute the pipeline but loads the content of cache/my_pipeline.pkl
# Loads the result of the first function from a pkl file and executes the remaining 4 functions
p.run_or_load(load_cache_from_step=1)Advanced usageThemethod_settingsdictionary is converted to actual methods with their corresponding arguments. These are saved as lambda's in the propertymethod_list, which are called in order by therunmethod. You can call the methods from this list directly if you want.ThePipelineBaseclass contains several magic methods, so that it can be used as a list. For instance,p[0]will return the first item in themethod_settingsproperty. For more info, see the magic methods in thePipelineBaseclass.If you have (mostly) the same data manipulations for eachPipeline, you can probably use just a single class as described above. However, if this class becomes extremly large and large portions of the code are evident to be only applicable to certain types of pipelines, you might consider multiple inheritance. For example, you might have completely different methods in yourPipelinefor classification models and regression models. So you might build aPipelineclass as above, but make two extra classes -PipelineClassificationandPipelineRegression- that inherit from yourPipelineclass. Another example is that you maybe have timeseries and non-timeseries data. Here, too, you might consider using multiple inheritance if that seems logical.Other codeThere is some other code in this repository used directly byPipelineBaseor that might be useful to you. To name a few: there is a DatabaseConnection class which is a small wrapper around sqlalchemy and there is a method to load loggers. This is not explicitly explained in the README, but can be used.