Package: jobflow
jobflowDocumentation|PyPI|GitHub|PaperJobflow is a free, open-source library for writing and executing workflows. Complex
workflows can be defined using simple python functions and executed locally or on
arbitrary computing resources using theFireWorksworkflow manager.Some features that distinguish jobflow are dynamic workflows, easy compositing and
connecting of workflows, and the ability to store workflow outputs across multiple
databases.Is jobflow for mejobflow is intended to be a friendly workflow software that is easy to get started with,
but flexible enough to handle complicated use cases.Some of its features include:A clean and flexible Python API.A powerful approach to compositing and connecting workflows — information passing
between jobs is a key goal of jobflow. Workflows can be nested allowing for a natural
way to build complex workflows.Integration with multiple databases (MongoDB, S3, GridFS, and more) through theMaggmapackage.Support for theFireWorksworkflow management system, allowing workflow
execution on multicore machines or through a queue, on a single machine or multiple
machines.Support for dynamic workflows — workflows that modify themselves or create new ones
based on what happens during execution.Workflow modelWorkflows in jobflows are made up of two main components:AJobis an atomic computing job. Essentially any python function can beJob,
provided its input and return values can be serialized to json. Anything returned by the job is
considered an "output" and is stored in the jobflow database.AFlowis a collection ofJobobjects or otherFlowobjects. The connectivity
between jobs is determined automatically from the job inputs. The execution order
of jobs is automatically determined based on their connectivity.Python functions can be easily converted in toJobobjects using the@jobdecorator.
In the example below, we define a job to add two numbers.fromjobflowimportjob,Flow@jobdefadd(a,b):returna+badd_first=add(1,5)add_second=add(add_first.output,5)flow=Flow([add_first,add_second])flow.draw_graph().show()The output of the job is accessed using theoutputattribute. As the job has not
yet been run,outputcontains be a reference to a future output. Outputs can be used
as inputs to other jobs and will be automatically "resolved" before the job is
executed.Finally, we created a flow using the twoJobobjects. The connectivity between
the jobs is determined automatically and can be visualised using the flow graph.Installationjobflowis a Python 3.9+ library and can be installed usingpip.pipinstalljobflowQuickstart and tutorialsTo get a first glimpse of jobflow, we suggest that you follow our quickstart tutorial.
Later tutorials delve into the advanced features of jobflow.Five-minute quickstart tutorialIntroduction to jobflowDefining Jobs using jobflowNeed help?Ask questions about jobflow on thejobflow support forum.
If you've found an issue with jobflow, please submit a bug report onGitHub Issues.What’s new?Track changes to jobflow through thechangelog.ContributingWe greatly appreciate any contributions in the form of a pull request.
Additional information on contributing to jobflow can be foundhere.
We maintain a list of all contributorshere.Licensejobflow is released under a modified BSD license; the full text can be foundhere.CitationIf you use Jobflow in your work, please cite it as follows:"Jobflow: Computational Workflows Made Simple", A.S. Rosen, M. Gallant, J. George, J. Riebesell, H. Sahasrabuddhe, J.X. Shen, M. Wen, M.L. Evans, G. Petretto, D. Waroquiers, G.‑M. Rignanese, K.A. Persson, A. Jain, A.M. Ganose,Journal of Open Source Software, 9(93), 5995 (2024) DOI:10.21105/joss.05995AcknowledgementsJobflow was designed by Alex Ganose, Anubhav Jain, Gian-Marco Rignanese, David Waroquiers, and Guido Petretto. Alex Ganose implemented the first version of the package. Later versions have benefited from the contributions of several research groups. A full list of contributors is availablehere.