Package: simple-dag
🎯 simple_dagWelcome tosimple_dag! Here, we provide the easiest way to create a pipeline in an orchestration-agnostic manner. Just decorate your functions with our@transformdecorator! 🎉Free software: MIT license🚀 Getting Startedgit clone https://github.com/leokster/simple_dag.git
cd simple_dag
python3.10 -m venv venv
source venv/bin/activate
pip install simple_dag
venv/bin/dagit -f examples/dag.py💡 The Main IdeasWhat is a DAG? 🤔:A DAG, or Directed Acyclic Graph, represents a set of functions (the nodes) and their dependencies (the edges). It allows us to execute many functions, which depend on each other, in a specific order.Aren't there already many DAG libraries?:Absolutely, but most of them are tightly coupled to specific orchestration frameworks and require a very specific way to define a DAG. This makes it challenging to switch between frameworks. Our library, however, is different! 🎈What is the goal of this library?:Our library aims to offer a simple and streamlined way to define a DAG in a framework-agnostic manner. This means you can switch between frameworks without having to rewrite your DAG. As of now, we support Dagster and direct execution. 🎯What is a transform?:In the context of a data pipeline, a transform is a function that takes some data as input and produces some new data as output. It's like the magic wand in your data pipeline. 🪄Show me some code! 👩‍💻:Imagine we have a transformation where we read a CSV file, filter the data, and write it to a new CSV file. The@transformdecorator marks a function as a transformation function.PandasDFInputandPandasDFOutputprepare the data for the transformation and write the post-transformation data, respectively.dfis the input data andoutputis the output data.import os
from simple_dag import transform, PandasDFInput, PandasDFOutput

@transform(
        df=PandasDFInput(
                os.path.join("data/curated/ds_salaries_2023.csv"),
        ),
        output=PandasDFOutput(
                os.path.join("data/curated/ds_salaries_2023_ES.csv"),
        ),
)
def create_2023_salaries_ES(df, output: PandasDFOutput):
df = df[df["company_location"] == "ES"]
output.write_data(df, index=False)@transform:This decorator indicates that a function is a transformation. It acceptsInputandOutputarguments. Please note, theOutputarguments are passed directly to your function, while theInputarguments are processed by theInputclass and then the resultant data is passed to your function.Input:Inputs prepare the data for your function. Currently, we support the following inputs:PandasDFInput: Reads a pandas dataframe from a CSV file. The function receives this data as a pandas dataframe.BinaryInput: Reads a binary file. The function receives this data as a bytes object.SparkDFInput: Reads a Spark dataframe from a parquet file (Experimental). The function receives this data as a Spark dataframe.Output:Outputs write the data after your function has processed it. TheOutputobjects have awrite_datamethod, which can be used in your function to write the data. Currently, we support the following outputs:PandasDFOutput: Writes a pandas dataframe to a CSV file.BinaryOutput: Writes a binary file.SparkDFOutput: Writes a Spark dataframe to a parquet file (Experimental).