Package: analysis-runner
Analysis runnerThis tool helps tomake analysis results reproducible,
by automating the following aspects:Allow quick iteration using an environment that resembles production.Only allow access to production datasets through code that has been reviewed.Link the output data with the exact program invocation of how the data has
been generated.One of our main workflow pipeline systems at the CPG isHail Batch. By default, its
pipelines are defined by running a Python programlocally. This tool instead lets you run the "driver" on Hail Batch itself.Furthermore, all invocations are logged together with the output data, as well asAirtableand the sample-metadata server.When using the analysis-runner, the batch jobs are not run under your standard
Hail Batchservice account user(<USERNAME>-trial). Instead, a separate Hail Batch account is
used to run the batch jobs on your behalf. There's a dedicated Batch service
account for each dataset (e.g. "tob-wgs", "fewgenomes") and access level
("test", "standard", or "full", as documented in the team docsstorage policies),
which helps with bucket permission management and billing budgets.Note that you can use the analysis-runner to start arbitrary jobs, e.g. R scripts. They're just launched in the Hail Batch environment, but you can use any Docker image you like.The analysis-runner is also integrated with our Cromwell server to run WDL based workflows.CLIThe analysis-runner CLI can be used to start pipelines based on a GitHub repository,
commit, and command to run.First, make sure that your environment provides Python 3.10 or newer:>python3--version
Python3.10.7If the installed version is too old, on a Mac you can usebrewto update. E.g.:brewinstallpython@3.10Then install theanalysis-runnerPython package usingpip:python3-mpipinstallanalysis-runnerRunanalysis-runner --helpto see usage information.Make sure that you're logged into GCP:gcloudauthapplication-defaultloginIf you're in the directory of the project you want to run, you can omit the--commitand--repositoryparameters, which will use your current git remote and
commit HEAD.For example:analysis-runner\--dataset<dataset>\--description<description>\--access-level<level>\--output-dir<directory-within-bucket>\script_to_run.pywitharguments<level>corresponds to anaccess levelas defined in the storage policies.<directory-within-bucket>doesnotcontain a prefix likegs://cpg-fewgenomes-main/. For example, if you want your results to be stored ings://cpg-fewgenomes-main/1kg_pca/v2, specify--output-dir 1kg_pca/v2.If you provide a--repository, you MUST supply a--commit <SHA>, e.g.:analysis-runner\--repositorymy-approved-repo\--commit<commit-sha>\--dataset<dataset>\--description<description>\--access-level<level>--output-dir<directory-within-bucket>\script_to_run.pywithargumentsFor more examples (including for running an R script and dataproc), see theexamplesdirectory.GitHub AuthenticationIf you are submitting an analysis-runner job that needs to clone a private repository owned by populationgenomics on GitHub (eg submitting a script to analysis-runner from a private repository), please make sure that your configuration file contains the following section:[infrastructure]git_credentials_secret_name='<ask_software_team_for_secret_name>'git_credentials_secret_project='<ask_software_team_for_secret_project>'If you are specifying multiple configuration files, please make sure that this section appears in the final right-most config to avoid these settings being overwritten.Custom Docker imagesThe default driver image that's used to run scripts comes with Hail and some statistics libraries preinstalled (see the correspondingHail Dockerfile). It's possible to use any custom Docker image instead, using the--imageparameter. Note that any such image needs to contain the critical dependencies as specified in thebaseimage.For R scripts, we add the R-tidyverse set of packages to the base image, see the correspondingR Dockerfileand theR examplefor more details.Helper for Hail BatchThe analysis-runner package has a number of functions that make it easier to run reproducible analysis through Hail Batch.This is installed in the analysis runner driver image, ie: you can access the analysis_runner module when running scripts through the analysis-runner.Checking out a git repository at the current commitfromcpg_utils.hail_batchimportget_batchfromanalysis_runner.gitimport(prepare_git_job,get_repo_name_from_current_directory,get_git_commit_ref_of_current_repository,)b=get_batch(name='do-some-analysis')j=b.new_job('checkout_repo')prepare_git_job(job=j,# you could specify a name here, like 'analysis-runner'repo_name=get_repo_name_from_current_directory(),# you could specify the specific commit here, eg: '1be7bb44de6182d834d9bbac6036b841f459a11a'commit=get_git_commit_ref_of_current_repository(),)# Now, the working directory of j is the checkout out repositoryj.command('examples/bash/hello.sh')Running a dataproc scriptfromcpg_utils.hail_batchimportget_batchfromanalysis_runner.dataprocimportsetup_dataprocb=get_batch(name='do-some-analysis')# starts up a cluster, and submits a script to the cluster,# see the definition for more information about how you can configure the cluster# https://github.com/populationgenomics/analysis-runner/blob/main/analysis_runner/dataproc.py#L80cluster=dataproc.setup_dataproc(b,max_age='1h',packages=['click','selenium'],cluster_name='My Cluster with max-age=1h',)cluster.add_job('examples/dataproc/query.py',job_name='example')DevelopmentYou can ignore this section if you just want to run the tool.To set up a development environment for the analysis runner using pip, run
the following:pipinstall-rrequirements-dev.txt

pre-commitinstall--install-hooks

pipinstall--editable.DeploymentAdd a Hail Batch service account for all supported datasets.Copy the Hail tokensto the Secret Manager.Deploy theserverby invoking thedeploy_serverworkflowmanually.Deploy theAirtablepublisher.Publish theCLI tool and libraryto PyPI.The CLI tool is shipped as a pip package. To build a new version,
we usebump2version.
For example, to increment the patch section of the version tag 1.0.0 and make
it 1.0.1, run:gitcheckout-badd-new-version
bump2versionpatch
gitpush--set-upstreamoriginadd-new-version# Open pull requestopen"https://github.com/populationgenomics/analysis-runner/pull/new/add-new-version"It's important the pull request name start with "Bump version:" (which should happen
by default). Once this is merged intomain, a GitHub action workflow will build a
new package that will be uploaded to PyPI, and become available to install withpip install.