Package: pemi
Welcome to Pemi’s documentation!Pemi is a framework for building testable ETL processes and workflows.  Users
definepipesthat define how to collect, transform, and deliver data.  Pipes
can be combined with other pipes to build out complex and modular data pipelines.
Testing is a first-class feature of Pemi and comes with a testing API to allow for
describing test coverage in a manner that is natural for data transformations.Full documentation on readthedocsInstall PemiPemi can be installed from pip:pip install pemiConcepts and FeaturesPipesThe principal abstraction in Pemi is thePipe.  A pipe can be composed
ofData Sources,Data Targets, and otherPipes.  When
a pipe is executed, it collects data form the data sources, manipulates that data,
and loads the results into the data targets.  For example, here’s a simple
“Hello World” pipe. It takes a list of names in the form of a Pandas DataFrame
and returns a Pandas DataFrame saying hello to each of them.importpandasaspdimportpemifrompemi.fieldsimport*classHelloNamePipe(pemi.Pipe):# Override the constructor to configure the pipedef__init__(self):# Make sure to call the parent constructorsuper().__init__()# Add a data source to our pipe - a pandas dataframe called 'input'self.source(pemi.PdDataSubject,name='input',schema=pemi.Schema(name=StringField()))# Add a data target to our pipe - a pandas dataframe called 'output'self.target(pemi.PdDataSubject,name='output')# All pipes must define a 'flow' method that is called to execute the pipedefflow(self):self.targets['output'].df=self.sources['input'].df.copy()self.targets['output'].df['salutation']=self.sources['input'].df['name'].apply(lambdav:'Hello '+v)To use the pipe, we have to create an instance of it:pipe = HelloNamePipe()and give some data to the source named “input”:pipe.sources['input'].df = pd.DataFrame({
    'name': ['Buffy', 'Xander', 'Willow', 'Dawn']
})nameBuffyXanderWillowDawnThe pipe performs the data transformation when theflowmethod is called:pipe.flow()The data target named “output” is then populated:pipe.targets['output'].dfnamesalutationBuffyHello BuffyXanderHello XanderWillowHello WillowDawnHello DawnData SubjectsData SourcesandData Targetsare both types ofData
Subjects.  A data subject is mostly just a reference to an object
that can be used to manipulate data.  In the [Pipes](#pipes) example
above, we defined the data source called “input” as using thepemi.PdDataSubjectclass.  This means that this data subject refers
to a Pandas DataFrame object.  Calling thedfmethod on this data subject
simply returns the Pandas DataFrame, which can be manipulated in all the ways
that Pandas DataFrames can be manipulated.Pemi supports 3 data subjects natively, but can easily be extended to support others.  The
3 supported data subjects arepemi.PdDataSubject- Pandas DataFramespemi.SaDataSubject- SQLAlchemy Enginespemi.SparkDataSubject- Apache Spark DataFramesSchemasA data subject can optionally be associated with aSchema.
Schemas can be used to validate that the data object of the data
subject conforms to the schema.  This is useful when data is passed
from the target of one pipe to the source of another because it
ensures that downstream pipes get the data they are expecting.For example, suppose we wanted to ensure that our data had fields calledidandname.
We would define a data subject like:from pemi.fields import *

ds = pemi.PdDataSubject(
    schema=pemi.Schema(
        id=IntegerField(),
        name=StringField()
    )
)If we provide the data subject with a dataframe that does not have a field:df = pd.DataFrame({
    'name': ['Buffy', 'Xander', 'Willow']
})

ds.df = dfThen an error will be raised when the schema is validated (which happens automatically when
data is passed between pipes, as we’ll see below):ds.validate_schema()
#=> MissingFieldsError: DataFrame missing expected fields: {'id'}We’ll also see later that defining a data subject with a schema also
aids with writing tests.  So while optional, defining data subjects
with an associated schema is highly recommended.Referencing data subjects in pipesData subjects are rarely defined outside the scope of a pipe as done
in [Schemas](#schemas).  Instead, they are usually defined in the
constructor of a pipe as in [Pipes](#pipes).  Two methods of thepemi.Pipeclass are used to define data subjects:sourceandtarget.  These methods allow one to specify the data subject class
that the data subject will use, give it a name, assign a schema, and
pass on any other arguments to the specific data subject class.For example, if we were to define a pipe that was meant to use an
Apache Spark dataframe as a source:spark_session = ...
class MyPipe(pemi.Pipe):
    def __init__(self):
        super().__init__()

        self.source(
            pemi.SparkDataSubject,
            name='my_spark_source',
            schema=pemi.Schema(
                id=IntegerField(),
                name=StringField()
            ),
            spark=spark_session
        )Whenself.sourceis called, it builds the data subject from the options provided
and puts it in a dictionary that is associated with the pipe.  The spark data frame
can then be accessed from within the flow method as:def flow(self):
    self.sources['my_spark_source'].dfTypes of PipesMost user pipes will typically inherit from the mainpemi.Pipeclass.  However,
the topology of the pipe can classify it according to how it might be used.  While
the following definitions can be bent in some ways, they are useful for describing
the purpose of a given pipe.ASource Pipeis a pipe that is used to extract data from some
external system and convert it into a Pemi data subject.  This data
subject is thetargetof thesourcepipe.ATarget Pipeis a pipe that is used to take a data subject and
convert it into a form that can be loaded into some external system.
This data subject is thesourceof thetargetpipe.ATransformation Pipeis a pipe that takes one or more data sources,
transforms them, and delivers one more target sources.AJob Pipeis a pipe that is self-contained and does not specify any
source or target data subjects.  Instead, it is usually composed of other
pipes that are connected to each other.Pipe ConnectionsA pipe can be composed of other pipes that are each connected to each
other.  These connections for a directed acyclic graph (DAG).  When
then connections between all pipes are executed, the pipes that form
the nodes of the DAG are executed in the order specified by the DAG
(in parallel, when possible – parallel execution is made possible
under the hood viaDask graphs).  The data
objects referenced by the node pipes’ data subjects are passed between
the pipes according.As a minimal example showing how connections work, let’s define
a dummy source pipe that just generates a Pandas dataframe with
some data in it:class MySourcePipe(pemi.Pipe):
    def __init__(self):
        super().__init__()

        self.target(
            pemi.PdDataSubject,
            name='main'
        )

    def flow(self):
        self.targets['main'].df = pd.DataFrame({
            'id': [1,2,3],
            'name': ['Buffy', 'Xander', 'Willow']
        })And a target pipe that just prints the “salutation” field:class MyTargetPipe(pemi.Pipe):
    def __init__(self):
        super().__init__()

        self.source(
            pemi.PdDataSubject,
            name='main'
        )

    def flow(self):
        for idx, row in self.sources['main'].df.iterrows():
            print(row['salutation'])Now we define a job pipe that will connect the dummy source pipe to
our hello world pipe and connect that to our dummy target pipe:class MyJob(pemi.Pipe):
    def __init__(self):
        super().__init__()

        self.pipe(
            name='my_source_pipe',
            pipe=MySourcePipe()
        )
        self.connect('my_source_pipe', 'main').to('hello_pipe', 'input')

        self.pipe(
            name='hello_pipe',
            pipe=HelloNamePipe()
        )
        self.connect('hello_pipe', 'output').to('my_target_pipe', 'main')

        self.pipe(
            name='my_target_pipe',
            pipe=MyTargetPipe()
        )

    def flow(self):
        self.connections.flow()In the flow method we callself.connections.flow().  This calls theflowmethod of each pipe defined in the connections graph and
transfers data between them, in the order specified by the DAG.The job pipe can be executed by calling itsflowmethod:MyJob().flow()
# => Hello Buffy
# => Hello Xander
# => Hello WillowFurthermore, if you’re running this in a Jupyter notebook, you can see a graph of the
connections by running:import pemi.dot
pemi.dot.graph(MyJob())Referencing pipes in pipesReferencing pipes within pipes works the same way as for data sources and targets.
For example, if we wanted to run theMyJobjob pipe and then look at the
source of the “hello_pipe”:job  = MyJob()
job.flow()
job.pipes['hello_pipe'].sources['input'].dfWhere to go from hereFull documentation on readthedocsContributingIf you want to contribute to the development of Pemi, you’ll need to be able to run the test
suite locally.  To get started, copy the example environment file to a file you can
edit locally if needed:>>> cp example.env .envAll of the tests are run inside of a docker container, which you can build using>>> inv buildOnce the containers are built, spin up the containers to run the tests>>> inv upAnd then run the tests using something like (you may prefer different pytest options):>>> inv test --opts="-s -x -vv --tb=short --color=yes tests"The test container also launches a local Jupyter notebook server.  This can be a convenient tool to
have when developing Pemi.  To access the notebook severs, just visithttp://localhost:8890/labin a web browser (the specific port can be configured in the.envfile).Take down the container using>>> inv down