Package: job-orchestration
#Job OrchestrationWhat is the point in this package?Whilst running long experiments forhttps://aclanthology.org/2021.eacl-main.219/There was a number of characteristics /
pain points that felt unaddressed by the current open source packages.I want to run large number of experiments that are different in some small number of ways that can be controlled via
config.I want to be able to set of a list of experiments that should be run and not have to wait for experiment X to finish
before I kick off experiment YI want to be able to run these in parallel when appropriate.I don't want the failure of one experiment to effect another experiment.These experiments naturally broke down into a set of atomic tasks and when the 4th Task Failed I wanted to be able
to not have to repeat the first 3 experiments.If I get a future result that was surprising, I want to be able to go back to an earlier experiment andLook at the raw output including logs from that experiment.Rerun in exactly the same setup as before (including being able to revert my local version of the code base to that
point in time).Be able to compare what has changed in the code base between the two runs.If there was a train model as part of this experiment I wanted this to still be present.While the experiments are running I want an easy centralised place that I can see progress some estimation of how long
was left.I want validation to run prior to the experiment running. For example, I didn't want to train a model for X hours only
to find that the result didn't save due to a missing path.This package is an attempt to provide a basic framework to start to address these pain points. It was also important that
the package was Lightweight and didn't increase the run time significantly. Also while it was accepted that this package
would require some level opinionation to achieve these goals there was an aim to under index on this for the main feature
of the package where possible. Finally, we only considered experiments that could be run on a single machine scaling out
to multiple machines can be achieved by just partitioning the config files.Get Started:Download the package by running the following commands.pip install job-orchestrationThe first thing that you will need to do is set theJOB_ORCHESTRATION_WORKSPACEenvironment variable to tell the
module where you will write logs / output / etc too.Interface:To enable this module to run your code we require that you follow the convention that you have a file called Tasks
which contains a class that implements theTaskBaseclass. So the class needs a constructor which takes in the config
dictionary and provides a run method to do the actual work. Optionally, you can also provide a validate method
(Recommended that you do ).Config Files:To control the execution of your program you will need to provide one or more yaml config files. As a minimum we require
that you provide us with the following fields:
githubRepository: Url of the git repo that your project is under.
pathToTasks: This is the parent file of the Tasks.py file.
outputDir: Path to write the output too it - this will be a sub folder of the Output Directory.
tasks: A list of the tasks that need to be performed by the library. These must contain at least:
- id - must be unique in the set of tasks in this config file.
- method - The name of the class in the Tasks file to execute.These fields plus any others that are present will be passed through to as a dictionary object to the task
constructor and so can be used to control execution there.Now the next question is where do I put this config? You must put it in the "ConfigSources" sub-directory under the
path given by theJOB_ORCHESTRATION_WORKSPACEenvironment variable. (Pro-tip once you have set the environment variable
just runpython -m job_orchestrationto create all the subdirectories).Now we need to ready this config file to do this runpython -m job_orchestration -action readyConfigs- this will validate and setup the output location for any configs
in the ConfigSources directory before moving them to the ConfigsToRun directory.Finally, we need to run a worker which will run the tasks one by one in each of the yaml files. The worker will run
and pick up a config file from "ConfigsToRun" directory and attempt to run it and repeat.python -m job_orchestration -action readyConfigsFor more clarity please see the example Tasks.py and config file in the /demo folder: