Package: scriptflow
scriptflowSmall library that allows scheduling scripts asyncrhonously on different platforms. Think of it as a Make when you can write the dependencies as python code, and that can run locally, on an HPC or in the cloud (cloud is not implemented just yet).The status is very experimental. I will likely be changing the interface as I go.Goals:works on windows / osx / linuxdescribe dependencies as python code (using await/async)describe scripts with input/output as codeclean terminal feedback (using rich)task retrycheck that output was generatednotifications (using light webserver atscriptflow.lamadon.com)send status to central web serviceresume flowsclean outputnamed runsstore run informationoutput diagnostic / reporting (tracing how files were created)simpler interface with default head executor and awaitable tasksskip computation based on timestamp of inputs and outpusload and store tasks resultsremove existing output of task if task is started (issue with failing tasks that look like they worked)executors :local excutor using subprocessHPC excutor (monitoring qsub)docker Executoraws executor (probably using Ray)dask executoradd check on qsub return valuesselect flow by name from terminal? scripts can create tasks, not sure how to await them.reporting:input and output hashesstart and end datetimesnotification systemallow to send messagesallow for runsallow to send messages with html content like imageswriting tasks and flowscache flows in addition to caching tasks (avoid same task getting scheduled from 2 places)a functional api for task creation with hooksa functional api for flowscontroller could parse the log file for results (looking for specific triggers)allow for glob output/inputprovide simple toml/json interface for simple tasks and flowsuseshlexto parse command from stringsclipass arguments to flowscreate portable executableSimple flow example:Create a filesflow.pywith:importscriptflowassf# set main optionssf.init({"executors":{"local":{"maxsize":5}},'debug':True})# example of a simple step that combines outcomesdefstep2_combine_file():withopen('test_1.txt')asf:a=int(f.readlines()[0])withopen('test_2.txt')asf:b=int(f.readlines()[0])withopen('final.txt','w')asf:f.write("{}\n".format(a+b))# define a flow called sleepitasyncdefflow_sleepit():i=1task1=sf.Task(cmd=f"""python -c "import time; time.sleep(5); open('test_{i}.txt','w').write('5');" """,outputs=f"test_{i}.txt",name=f"solve-{i}")i=2task2=sf.Task(cmd=f"""python -c "import time; time.sleep(5); open('test_{i}.txt','w').write('5');" """,outputs=f"test_{i}.txt",name=f"solve-{i}")awaitsf.bag(task1,task2)task_final=sf.Task(cmd="python -c 'import sflow; sflow.step2_combine_file()'",outputs=f"final.txt",inputs=[*t1.get_outputs(),*t1.get_outputs()])awaittask_finalthen create a local env, activate, install and run!python3-mvenvenvsourceenv/bin/activate
pipinstallscriptflow
scritpflowrunsleepitLife cycle of a taskthe task object is created. All properties can be edited.the task is sent to an executor. At this point, the properties of the task are frozen. They can be read, copied but not changed. A unique ID id created from the task from its command and its inputs. The task can be sent by using thestart()method, or it will be sent automatically when awaited.the task is awaited, and hence execution is blocked until the task is finished. Nothing can be done at that stage. Again, the task is automatically sent at this stage if it has not be done before. Also note that several tasks can be awaited in parallel by bagging them withsf.bag(...).the task is completed, the await returns. The task has now it's output attached to it, it can be used in the creation of other tasks.Inspiration / AlternativesI have tried to use the following three alternatives which are all truly excelent!pydoitnextflowsnakemakeThere were use cases that I could not implement cleanly in the dataflow model of nextflow. I didn't like that snakemake relied on file names to trigger rules, I was constently juggling complicated file names. Pydoit is really great, but I couldn't find how to extend it to build my own executor, and I always found myself confused writing new tasks and dealing with dependencies.Developingthe package is managed using poetry, install poetry first thenpoetry install

# run example
cd examples/simple-local
poetry run scriptflow run sleepit

# run tests with coverate
poetry run python -m pytest --cov=scriptflow
poetry run coverage xml
poetry run codecov -t <token>Docker images to try the different schedulersPBSslurm=======