Package: outflow
Outflow is a framework that helps you build and run task workflows.The api is as simple as possible while still giving the user full control over the definition and execution of the
workflows.Feature highlight :Simple but powerful APISupport forparallelized and distributed executionCentralizedcommand line interfacefor your pipeline commandsIntegrateddatabaseaccess, sqlalchemy models and alembic migrationsExecutions and exceptions logging fortracabilityStrict type and input/output checking for arobustpipelineCheck out ourdocumentationfor more information.InstallingInstall and update usingpip:pip install -U outflowQuick startOne file starterFirst, create apipeline.pyscript:# -- pipeline.pyfromoutflow.core.commandsimportCommand,RootCommandfromoutflow.core.pipelineimportPipelinefromoutflow.core.tasksimportas_task# with the as_task decorator, the function will be automatically converted into a Task subclass# the signature of the function, including the return type, is used to determine task inputs and outputs@as_taskdefGetValues()->{"word1":str,"word2":str}:return{"word1":"Hello","word2":"world!"}# default values can also be used as inputs@as_taskdefConcatenate(word1:str,word2:str)->{"result":str}:result=f"{word1}{word2}"returnresult# you can return the value directly if your task has only one output# A task can have side-effects and returns nothing@as_taskdefPrintResult(result:str):print(result)@RootCommand.subcommand()classHelloWorld(Command):defsetup_tasks(self):# instantiate the tasksget_values=GetValues()concatenate=Concatenate(word2="outflow!")# you can override task inputs value at instantiationprint_result=PrintResult()# build the workflowget_values>>concatenate>>print_result# instantiate and run the pipelinewithPipeline(root_directory=None,settings_module="outflow.core.pipeline.default_settings",force_dry_run=True,)aspipeline:result=pipeline.run()and run your first Outflow pipeline:$ python pipeline.py hello_worldA robust, configurable and well-organized pipelineYou had a brief overview of Outflow's features and you want to go further. Outflow offers command line tools to help you to start your pipeline project.First, we will need to auto-generate the pipeline structure -- a collection of files including the pipeline settings, the database and the cluster configuration, etc.$ python -m outflow management create pipeline my_pipelineThen, we have to create a plugin -- a dedicated folder regrouping the commands, the tasks as well as the description of the database (the models)$ python -m outflow management create plugin my_namespace.my_plugin --plugin_dir my_pipeline/plugins/my_pluginIn the my_pipeline/settings.py file, add your new plugin to the plugin list:PLUGINS=['outflow.management','my_namespace.my_plugin',]and run the following command:$ python ./my_pipeline/manage.py my_pluginYou'll see the following output on the command line:* outflow.core.pipeline.pipeline - pipeline.py:325 - INFO - No cluster config found in configuration file, running in a local cluster
 * my_namespace.my_plugin.commands - commands.py:49 - INFO - Hello from my_pluginYour pipeline is up and running. You can now start adding new tasks and commands.ContributingFor guidance on setting up a development environment and how to make a contribution to Outflow, see thecontributing guidelines.