Package: opensafely-jobrunner
OpenSAFELY job runnerA job runner is a service that encapsulates:the task of checking out an OpenSAFELY study repo;executing actions defined in itsproject.yamlconfiguration file when
requested via a jobs queue; andstoring its results in a particular locations.Quickrefs:PlaybooksEnd users will find more information in theOpenSAFELY documentation.Operating principlesIn production, this software runs as a loop on a secure server within the
infrastructure of the primary data provider.  It polls anOpenSAFELY job
server, looking for requests to run
jobs.Jobs belong to aworkspace. This describes the git repo containing the
OpenSAFELY-compliant project under execution; the git branch, and kind of
database to use. The workspace also acts as a kind of namespace for
partitioning outputs of its jobs.An OpenSAFELY-compliant repo must provide aproject.yamlfile which
describes how a requested job should be converted into a command (& arguments)
that can be run in a subprocess on the secure server.  It incorporates the idea
of dependencies, so an action that generates a chart might depend on an action
that extracts data from the databaseforthat chart.  See theActions referencefor more information.An action can defineoutputs; these are persisted on disk and made available
to subsequent actions in the workspace, and users who have permission to log
into the server and view the raw files.The runner takes care of executing dependencies in order. By default, it skips
re-running a dependency whose previous run produced output that still exists in
the production environment.  The runner also reports status back to the job
server, redacting possibly-sensitive information.The runner is bundled as part of theopensafely-clitool so users
can test their actions locally.Job structureThe job server serves jobs as JSON in the following format. First, a job must
belong to a workspace:{"workspace":{"name":"my workspace","repo":"https://github.com/opensafely/job-integration-tests","branch":"master","db":"full"}}Possible values for"db"are "full", "slice", and "dummy".A workspace is a way of associating jobs related to a given combination of
branch, repository and database. To enqueue a job, a client POSTs JSON like
this:{"backend":"tpp","action_id":"do_thing","workspace_id":1}Consuming jobsA job runner is service installed on a machine that has access to a given
backend. It receives jobs from the server and consumes those whosebackendvalue matches the
value of the currentBACKENDenvironment variable.It must also define three environment variables which are an RFC1838 connection
URL; these correspond to thedbrequested in the job's workspace definition,
and as such are namedFULL_DATABASE_URL,SLICE_DATABASE_URL, andDUMMY_DATABASE_URL.When a job is found, the following happens:The corresponding repo is fetched. Private repos are accessed using
thePRIVATE_REPO_ACCESS_TOKENsupplied in the environment.Itsproject.yamlis parsed:Individualactionsare extracted from this fileA dependency graph is calculated for the requested action; for example, an
action might depend on three previous actions before it can be runEach action in the graph is checked to see if it needs to be runActions that either: (a) already have output generated from a previous
run; (b) are currently running; (c) failed on their last run do not
need to be runIf a dependency has failed, then the requested action failsIf the dependency needs to be run, a new job is pushed to the queue, and the
current job is postponedIf an action has no dependencies needing to be run, then itsdocker runis
executedOn completion, a status code and message are reported back to the job
server. On success, a list of output file locations are also posted. On
failure, the message has any potentially-sensitive information redacted, and is associated with
a unique string so that a user with requisite permissions can log into the
production environment and examine the docker logs for the full error.Output locationsEvery action defines a list ofoutputswhich are persisted to a permanent
storage location.  The project author must categorise these outputs as eitherhighly_sensitiveormoderately_sensitive.  Any pseudonymised data which may
be highly disclosive (e.g. without low number redaction) should be classed ashighly_sensitive; data which the author believes could be released following
review should be classed asmoderately_sensitive. This design allows tiered
levels of permissions for collaborators to review data outputs. For example, the
study author would usually have access tohighly_sensitivematerial for
debugging; but other collaborators could have access tomoderately_sensitivedata to prepare it for release (for which it is planned to add aminimally_sensitivecategory).Outputs are therefore persisted to filesystem paths according to the following
environment variables:# A location where cohort CSVs (one row per patient) should be# stored. This folder must exist.HIGH_PRIVACY_STORAGE_BASE=/home/opensafely/high_security# A location where script outputs (some for publication) should be# storedMEDIUM_PRIVACY_STORAGE_BASE=/tmp/outputs/medium_securityProject.yamlA valid project file looks like this:version:"3.0"expectations:population_size:1000actions:generate_study_population:run:cohortextractor:latest generate_cohort --study-definition study_definitionoutputs:highly_sensitive:cohort:output/input.csvrun_model:run:stata-mp:latest analysis/model.doneeds:[generate_study_population]outputs:moderately_sensitive:model:models/cox-model.txtfigure:figures/survival-plot.pngSee theproject pipeline documentationfor a detailed
description of the project.yaml setup.Local actions developmentThecohortextractorcommand-line toolimports this library, and implements the action-parsing-and-running functionality as a series of
synchronous docker commands, rather than asynchronously via the job queue.For developersPlease seethe additional information.