Package: pipewelder
Pipewelder is a framework that provides a command-line tool and Python
API to manageAWS Data
Pipelinejobs from flat files.
Simple uses it as a cron-like job scheduler.Sourcehttps://github.com/SimpleFinance/pipewelderDocumentationhttp://pipewelder.readthedocs.orgPyPIhttps://pypi.python.org/pypi/pipewelderOverviewPipewelder aims to ease the task of scheduling jobs by defining very
simple pipelines which are little more than an execution schedule,
offloading most of the execution logic to files in S3. Pipewelder uses
Data Pipeline’s concept ofdata
stagingto pull input files from S3 at the beginning of execution and to upload
output files back to S3 at the end of execution.If you follow Pipewelder’s directory structure, all of your pipeline
logic can live in version-controlled flat files. The included
command-line interface gives you simple commands to validate your
pipeline definitions, upload task definitions to S3, and activate your
pipelines.InstallationPipewelder is available fromPyPIviapipand is compatible with Python 2.6, 2.7, 3.3, and 3.4:pip install pipewelderThe easiest way to get started is to clone the project from GitHub, copy
the example project from Pipewelder’s tests, and then modify to suit:gitclonehttps://github.com/SimpleFinance/pipewelder.gitcp-rpipewelder/tests/test_datamy-pipewelder-projectIf you’re setting up Pipewelder and need help, feel free to email the
author.DevelopmentTo do development on Pipewelder, clone the repository and runmaketo install dependencies and run tests.Directory StructureTo use Pipewelder, you provide a template pipeline definition along with
one or more directories that correspond to particular pipeline
instances. The directory structure looks like this (seetest_datafor a working example):pipeline_definition.json
pipewelder.json <- optional configuration file
my_first_pipeline/
    run
    values.json
    tasks/
        task1.sh
        task2.sh
my_second_pipeline/
...Thevalues.jsonfile in each pipeline directory specifies parameter
values that are used modify the template definition including the S3
paths for inputs, outputs, and logs. Some of these values are used
directly by Pipewelder as well.A`ShellCommandActivity<http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html>`__
in the template definition simply looks for an executable file namedrunand executes it.runis the entry point for whatever work
you want your pipeline to do.Often, yourrunexecutable will be a wrapper script to execute a
variety of similar tasks. When that’s the case, use thetaskssubdirectory to hold these definitions. These tasks could be text files,
shell scripts, SQL code, or whatever else yourrunfile expects.
Pipewelder givestasksfolder special treatment in that the CLI will
make sure to remove existing task definitions when uploading files.Using the Command-Line InterfaceThe Pipewelder CLI should always be invoked from the top-level directory
of your definitions (the directory wherepipeline_definition.jsonlives). If your directory structure matches Pipewelder’s expectations,
it should work without further configuration.As you make changes to your template definition orvalues.jsonfiles, it can be useful to check whether AWS considers your definitions
valid:$ pipewelder validateOnce you’ve defined your pipelines, you’ll need to upload the files to
S3:$ pipewelder uploadFinally, activate your pipelines:$ pipewelder activateAny time you change thevalues.jsonorpipeline_definition.json,
you’ll need to run theactivatesubcommand again. Because active
pipelines can’t be modified, theactivatecommand will delete the
existing pipeline and create a new one in its place. The run history for
the previous pipeline will be discarded.AcknowledgmentsPipewelder’s package structure is based onpython-project-template.