Package: datapipeliner
datapipelinerdvc-ready data pipelinesProvides data pipelines that are especially suited to incoporation intoDVC.
Allows model and data processing experiment production support.Data pipelines are described in a YAML configuration file.The pipelines can be parameterized by tags names.
These can be programatically controlled via a commands in a dvc params.yaml file - which allows
for data processing experiments to be integrated into a dvc experiment environment ie the data processing stages
can become features in the experiment.The YAML file can define standard pdpipe stages as well as custom stages defined
in a .py file. Parameters to the stages are defined in the YAML file.Installationpip install datapipelinerRequirementsThis package manages YAML configurations withconfuse, which itself depends onpyYAML. Pipeline stages and pipelines are generated withpdpipe, andengardeis an
optional dependency forverify_all-,verify_any-, andengarde-type stages.DetailsThe pipeline is defined inconfig.yaml. This file contains information
aboutsources, files from which the data is drawn,pipelinesand their stages, and
thesinks, files to which the transformed data is written. Custom-made functions may
be defined in a standard*.pyfile/module, which must take apandas.DataFrameas
input and return apandas.DataFrameas output. Pipeline stages are generated from
these custom functions by specifying them and their keyword arguments inconfig.yaml.The fileconfig.yamlcontrols all aspects of the pipeline, from data discovery, to
pipeline stages, to data output. If the environment variableDATAPIPELINERDIRis not
specified, then then it will be set to the current working directory. The fileconfig.yamlshould be put in theDATAPIEPLINEDIR, and data to be processed should be
in that directory or its subdirectories.ExampleThe directory structure of this example is as follows:example/
    config.yaml
    custom_functions.py
    example.py
    raw
        products_storeA.csv
        products_storeB.csv
    output
        products_storeA_processed.csv
        products_storeB_processed.csvThe contents ofconfig.yamlis as follows (paths are relative to the location ofconfig.yaml, i.e. theDATAPIPELINERDIR):sources:
  example_source:
    file: raw/products*.csv
    kwargs:
      usecols:
        - items
        - prices
        - inventory
    index_col: items

sinks:
  example_sink:
    file: output/*_processed.csv

pipelines:
  example_pipeline:

  - type: transform
      function: add_to_col
      tag: add
      kwargs:
        col_name: prices
        val: 1.5
      staging:
        desc: Adds $1.5 to column 'prices'
        exmsg: Couldn't add to 'prices'.

    - type: pdpipe
      function: ColDrop
      kwargs:
        columns: inventory
      staging:
        exraise: false

    - type: verify_all
      check: high_enough
      tag: verify
      kwargs:
        col_name: prices
        val: 19
      staging:
        desc: Checks whether all 'prices' are over $19.The modulecustom_functions.pycontains:custom_functions.py

    def add_to_col(df, col_name, val):
        df.loc[:, col_name] = df.loc[:, col_name] + val
        return df

    def high_enough(df, col_name, val):
        return df.loc[:, col_name] > valFinally, the contents of the fileexample.py:import custom_functions
import datapipeliner as dpp

src = dpp.Source("example_source")  # generate the source from `config.yaml`
snk = dpp.Sink("example_sink")  # generate the sink from `config.yaml`.

# generate the pipeline from `config.yaml`.
line = dpp.Line("example_pipeline", custom_functions)

# connect the source and sink to the pipeline, print what the pipeline will do, then run
# the pipeline, writing the output to disk. capture the input/output dataframes if desired.
pipeline = line.connect(src, snk)
print(pipeline)
(dfs_in, dfs_out) = line.run()Runningexample.pygeneratessrc,snk, andlineobjects. Then, thesrcandsnkare connected to an internalpipeline, which is apdpipe.PdPipeLineobject.
When this pipeline is printed, the following output is displayed:A pdpipe pipeline:
[ 0]  Adds $1.5 to column 'prices'
[ 1]  Drop columns inventory
[ 2]  Checks whether all 'prices' are over $19.The function of this pipeline is apparent from the descriptions of each stage. Some
stages have custom descriptions specified in thedesckey ofconfig.yaml. Stages
of typepdpipehave their descriptions auto-generated from the keyword arguments.The commandline.run()pulls data fromsrc, passes it throughpipeline, and
drains it tosnk. The returnsdfs_inanddfs_outshow that came in fromsrcand what went tosnk. In addition toline.run(), the firstnstages of the
pipeline can be tested on filemfrom the source withline.test(m,n).Output from ExampleThis is.\raw\products_storeA.csvbefore it is drawn into the source:itemspricesinventorycolorfoo195redbar243greenbaz227blueThis is.\raw\products_storeA.csvafter it is drawn into the source with the argumentusecols = ["items", "prices", "inventory"]specified inconfig.yaml:itemspricesinventoryfoo195bar243baz227The output from the pipeline is sent to.\products_storeA_processed.csv. The arguments
specified byconfig.yamlhave been applied. Namely,priceshave been incremented by1.5, theinventorycolumn has been dropped, and then a check has been made that allpricesare over19.itemspricesfoo20.5bar25.5baz23.5If theverify_allstep had failed, an exception would be raised, and the items that
did not pass the check would be returned in the exception message. Say, for example,
that thevalargument was21instead of19:AssertionError: ('high_enough not true for all',
prices  items        
foo      20.5)Direct Dataframe InjectionAdditionally is is possible to call the pipeline directly with a dataimport custom_functions
import datapipeliner as dpp
import pandas as pd

tags =  'add;verify'

df_in = pd.read_csv("myfile.csv")

# generate the pipeline from `config.yaml`.
line = dpp.Line("example_pipeline", custom_functions, tags)

df_out= line.runDataFrame(df_in)ProvenanceThis project was created as a fork of the excellent pdpipewrench. A big thanks to blakeNaccarato /
pdpipewrench.