Package: dagonstar
DagOnStar (aka DagOn*)DagOnStar (Direct acyclic graph On anything) is a lightweight Python library implementing
a workflow engine able to execute parallel jobs represented by direct acyclic graphs on
any combination of local machines, on-premise high-performance computing clusters,
containers, and cloud-based virtual infrastructures.DagOnStar is used as the primary workflow engine to run real-world production-level
applications.DagOnStar is in production at theCenter for Monitoring and Modeling Marine and Atmosphereapplications hosted at the University of Naples "Parthenope".MotivationThanks to the advent of public, private, and hybrid clouds, the democratization of
Computational resources changed the rules in many science fields.
For decades, one of the efforts of computer scientists and computer engineers was the
development of tools able to simplify access to high-end computational resources by
computational scientists. However, nowadays, any science field can be considered
"computational" if the availability of powerful but easy-to-manage workflow
engines is crucial.AcknowledgmentsThe following initiatives support DAGonStar development:Research contract "Mytilus farming System with High-Performance Computing and Artificial Intelligence"
(MytilEx, CUP I63C23000180002, funded by the Campania Region, Veterinary sector) -
DAGonStar orchestrates the production workflow to deliver daily 168 weather, marine, pollutants,
and farmed mussels contamination forecasts and predictions.PWAEuroHPC H2020 project "Adaptative Multi-tier Intelligent data manager for Exascale"
(ADMIRE, 956748-ADMIRE-H2020-JTI-EuroHPC-2019-1, funded by the European Commission) -
WP7: DAGonStar orchestrates the Environmental Application delivering on-demand weather, marine,
and pollutants simulations and forecasts on the Campania Region (Italy).linkCite DAGonStarWorkflow engineSánchez-Gallegos, Dante Domizzi, Diana Di Luccio, Sokol Kosta, J. L. Gonzalez-Compean, and Raffaele Montella.
"An efficient pattern-based approach for workflow supporting large-scale science: The DagOnStar experience."
Future Generation Computer Systems 122 (2021): 187-203.linkBarron-Lugo, J.A., Gonzalez-Compean, J. L., Carretero, J., Lopez-Arevalo, I., & Montella, R. (2021).
A novel transversal processing model to build environmental big data services in the cloud.
Environmental Modelling & Software, 144, 105173.linkSánchez-Gallegos, Dante D., Diana Di Luccio, José Luis Gonzalez-Compean, and Raffaele Montella.
"Internet of things orchestration using dagon workflow engine."
In 2019 IEEE 5th world forum on internet of things (WF-IoT), pp. 95-100. IEEE, 2019.linkSánchez-Gallegos, Dante D., Diana Di Luccio, J. L. Gonzalez-Compean, and Raffaele Montella.
"A microservice-based building block approach for scientific workflow engines: Processing large data volumes with dagonstar."
In 2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pp. 368-375. IEEE, 2019.linkMontella, Raffaele, Diana Di Luccio, and Sokol Kosta.
"Dagon: Executing direct acyclic graphs as parallel jobs on anything."
In 2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 64-73. IEEE, 2018.linkApplicationsMellone, Gennaro, Ciro Giuseppe De Vita, Enrico Zambianchi, David Expósito Singh,
Diana Di Luccio, and Raffaele Montella.
"Democratizing the computational environmental marine data science: using the high-performance cloud-native computing for inert transport and diffusion lagrangian modelling."
In 2022 IEEE International Workshop on Metrology for the Sea; Learning to Measure Sea Health Parameters (MetroSea), pp. 267-272. IEEE, 2022.linkDe Vita, Ciro Giuseppe, Gennaro Mellone, Aniello Florio,
Catherine Alessandra Torres Charles, Diana Di Luccio,
Marco Lapegna, Guido Benassai, Giorgio Budillon, and Raffaele Montella.
"Parallel and hierarchically-distributed Shoreline Alert Model (SAM)."
In 2023 31st Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 109-113. IEEE, 2023.linkMontella, Raffaele, Diana Di Luccio, Ciro Giuseppe De Vita, Gennaro Mellone,
Marco Lapegna, Gloria Ortega, Livia Marcellino, Enrico Zambianchi, and Giulio Giunta.
"A highly scalable high-performance Lagrangian transport and diffusion model for marine pollutants assessment."
In 2023 31st Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 17-26. IEEE, 2023.linkSurveysAldinucci, Marco, Giovanni Agosta, Antonio Andreini, Claudio A. Ardagna,
Andrea Bartolini, Alessandro Cilardo, Biagio Cosenza et al.
"The Italian research on HPC key technologies across EuroHPC."
In Proceedings of the 18th ACM international conference on computing frontiers, pp. 178-184. 2021.linkInstallationgitclonehttps://github.com/DagOnStar/dagonstar.gitcddagonstarvirtualenvvenv.venv/bin/activatepipinstall-rrequirements.txtexportPYTHONPATH=$PWD:$PYTHONPATHTroubleshootingOn some MacOS installations, pycrypto fails to automatically install.
Usually this is due togmplibrary missing in the default include and library path.
Before launching the requirements install, find the location of the missing library.
Then export the CFLAGS as in the example below (The actual path could be different):export"CFLAGS=-I/usr/local/Cellar/gmp/6.2.1_1/include -L/usr/local/Cellar/gmp/6.2.1_1/lib"DemoCopy the configuration file in the examples directory.cpdagon.ini.sampleexamples/dagon.inicdexamplesEdit the ini file matching your system configuration.Task oriented workflow.The workflow is defined as tasks and their explicit dependencies.pythontaskflow-demo.pyData oriented workflow.The workflow is defined by data dependencies (task dependencies are automatically resolved)pythondataflow-demo.pyBatch Task FlowfromdagonimportWorkflowfromdagon.taskimportTaskType,DagonTask# Create the orchestration workflowworkflow=Workflow("Taskflow-Demo")taskA=DagonTask(TaskType.BATCH,"Tokio","/bin/hostname >tokio.out")taskB=DagonTask(TaskType.BATCH,"Berlin","/bin/date")taskC=DagonTask(TaskType.BATCH,"Nairobi","/usr/bin/uptime")taskD=DagonTask(TaskType.BATCH,"Mosco","cat workflow://Tokio/tokio.out")workflow.add_task(taskA)workflow.add_task(taskB)workflow.add_task(taskC)workflow.add_task(taskD)taskB.add_dependency_to(taskA)taskC.add_dependency_to(taskA)taskD.add_dependency_to(taskB)taskD.add_dependency_to(taskC)workflow.run()Batch Data FlowfromdagonimportWorkflowfromdagon.taskimportDagonTask,TaskType# Check if this is the mainif__name__=='__main__':# Create the orchestration workflowworkflow=Workflow("DataFlow-Demo")# The task ataskA=DagonTask(TaskType.BATCH,"A","mkdir output; hostname > output/f1.txt")# The task btaskB=DagonTask(TaskType.BATCH,"B","echo $RANDOM > f2.txt; cat workflow:///A/output/f1.txt >> f2.txt")# The task ctaskC=DagonTask(TaskType.BATCH,"C","echo $RANDOM > f2.txt; cat workflow:///A/output/f1.txt >> f2.txt")# The task dtaskD=DagonTask(TaskType.BATCH,"D","cat workflow:///B/f2.txt >> f3.txt; cat workflow:///C/f2.txt >> f3.txt")# add tasks to the workflowworkflow.add_task(taskA)workflow.add_task(taskB)workflow.add_task(taskC)workflow.add_task(taskD)workflow.make_dependencies()# run the workflowworkflow.run()Meta workflowfromdagonimportWorkflowfromdagon.dag_tpsimportDAG_TPSfromdagon.taskimportDagonTask,TaskTypeworkflow=Workflow("DataFlow-Demo-Server")workflow.set_dry(False)# Set the dry# The task ataskA=DagonTask(TaskType.BATCH,"A","mkdir output;echo 'A1,A2,A3' > output/f1.csv")# The task btaskB=DagonTask(TaskType.BATCH,"B","echo 'B1,B2,B3' > f2.csv; cat workflow:///A/output/f1.csv >> f2.csv")# The task ctaskC=DagonTask(TaskType.BATCH,"C","echo 'C1,C2,C3' > f2.txt; cat workflow:///A/output/f1.csv >> f2.csv")# The task dtaskD=DagonTask(TaskType.BATCH,"D","cat workflow:///B/f2.csv >> f3.csv; cat workflow:///C/f2.csv >> f3.csv")#second workflowworkflow2=Workflow("DataFlow-transversal")workflow2.set_dry(False)# Set the dry# The task EtaskE=DagonTask(TaskType.BATCH,"E","mkdir output;echo 'E1,E2,E3' > output/f1.csv")# The task ftaskF=DagonTask(TaskType.BATCH,"F","echo 'F1,F2,F3' > f2.csv; cat workflow://DataFlow-Demo-Server/A/output/f1.csv >> f2.csv; cat workflow:///E/output/f1.csv >> f2.csv")# The task gtaskG=DagonTask(TaskType.BATCH,"G","cat workflow:///F/f2.csv >> f3.csv; cat workflow://DataFlow-Demo-Server/C/f2.csv >> f3.csv")# add tasks to the workflow 1workflow.add_task(taskA)workflow.add_task(taskB)workflow.add_task(taskC)workflow.add_task(taskD)# add tasks to the workflow 2workflow2.add_task(taskE)workflow2.add_task(taskF)workflow2.add_task(taskG)#list of the workflowsmetaworkflow=DAG_TPS("NewDAG")metaworkflow.add_workflow(workflow)metaworkflow.add_workflow(workflow2)metaworkflow.make_dependencies()metaworkflow.run()