Package: dsub
dsub: simple batch jobs with DockerOverviewdsubis a command-line tool that makes it easy to submit and run batch scripts
in the cloud.Thedsubuser experience is modeled after traditional high-performance
computing job schedulers like Grid Engine and Slurm. You write a script and
then submit it to a job scheduler from a shell prompt on your local machine.Todaydsubsupports Google Cloud as the backend batch job runner, along with a
local provider for development and testing. With help from the community, we'd
like to add other backends, such as a Grid Engine, Slurm, Amazon Batch,
and Azure Batch.Getting starteddsubis written in Python and requires Python 3.7 or higher.The last version to support Python 3.6 wasdsub0.4.7.For earlier versions of Python 3, usedsub0.4.1.For Python 2, usedsub0.3.10.Pre-installation stepsCreate a Python virtual environmentThis is optional, but whether installing from PyPI or from github,
you are strongly encouraged to use aPython virtual environment.You can do this in a directory of your choosing.python3 -m venv dsub_libs
    source dsub_libs/bin/activateUsing a Python virtual environment isolatesdsublibrary dependencies from
other Python applications on your system.Activate this virtual environment in any shell session before runningdsub.
To deactivate the virtual environment in your shell, run the command:deactivateAlternatively, a set of convenience scripts are provided that activate the
virutalenv before callingdsub,dstat, andddel. They are in thebindirectory. You can
use these scripts if you don't want to activate the virtualenv explicitly in
your shell.Install the Google Cloud SDKWhile not used directly bydsubfor thegoogle-v2orgoogle-cls-v2providers, you are likely to want to install the command line tools found in theGoogle
Cloud SDK.If you will be using thelocalprovider for faster job development,
youwillneed to install the Google Cloud SDK, which usesgsutilto ensure
file operation semantics consistent with the Googledsubproviders.Install the Google Cloud SDKRungcloud initgcloudwill prompt you to set your default project and to grant
credentials to the Google Cloud SDK.InstalldsubChooseoneof the following:Install from PyPIIf necessary,install pip.Installdsubpip install dsubInstall from githubBe sure you have git installedInstructions for your environment can be found on thegit website.Clone this repository.git clone https://github.com/DataBiosphere/dsub
cd dsubInstall dsub (this will also install the dependencies)python -m pip install .Set up Bash tab completion (optional).source bash_tab_completePost-installation stepsMinimally verify the installation by running:dsub --help(Optional)Install Docker.This is necessary only if you're going to create your own Docker images or
use thelocalprovider.MakefileAfter cloning the dsub repo, you can also use theMakefileby running:makeThis will create a Python virtual environment and installdsubinto a
directory nameddsub_libs.Getting started with the local providerWe think you'll find thelocalprovider to be very helpful when building
yourdsubtasks. Instead of submitting a request to run your command on a
cloud VM, thelocalprovider runs yourdsubtasks on your local machine.Thelocalprovider is not designed for running at scale. It is designed
to emulate running on a cloud VM such that you can rapidly iterate.
You'll get quicker turnaround times and won't incur cloud charges using it.Run adsubjob and wait for completion.Here is a very simple "Hello World" test:dsub \
   --provider local \
   --logging "${TMPDIR:-/tmp}/dsub-test/logging/" \
   --output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \
   --command 'echo "Hello World" > "${OUT}"' \
   --waitNote:TMPDIRis commonly set to/tmpby default on most Unix systems,
although it is also often left unset.
On some versions of MacOS TMPDIR is set to a location under/var/folders.Note: The above syntax${TMPDIR:-/tmp}is known to be supported by Bash, zsh, ksh.
The shell will expandTMPDIR, but if it is unset,/tmpwill be used.View the output file.cat "${TMPDIR:-/tmp}/dsub-test/output/out.txt"Getting started on Google Clouddsubsupports the use of two different APIs from Google Cloud for running
tasks. Google Cloud is transitioning fromGenomics v2alpha1toCloud Life Sciences v2beta.dsubsupports both APIs with the (old)google-v2and (new)google-cls-v2providers respectively.google-v2is the current default provider.dsubwill be transitioning to makegoogle-cls-v2the default in coming releases.The steps for getting started differ slightly as indicated in the steps below:Sign up for a Google account andcreate a project.Enable the APIs:For thev2alpha1API (provider:google-v2):Enable the Genomics, Storage, and Compute APIs.For thev2betaAPI (provider:google-cls-v2):Enable the Cloud Life Sciences, Storage, and Compute APIsProvidecredentialssodsubcan call Google APIs:gcloud auth application-default loginCreate aGoogle Cloud Storagebucket.The dsub logs and output files will be written to a bucket. Create a
bucket using thestorage browseror run the command-line utilitygsutil,
included in the Cloud SDK.gsutil mb gs://my-bucketChangemy-bucketto a unique name that follows thebucket-naming conventions.(By default, the bucket will be in the US, but you can change or
refine thelocationsetting with the-loption.)Run a very simple "Hello World"dsubjob and wait for completion.For thev2alpha1API (provider:google-v2):dsub \
    --provider google-v2 \
    --project my-cloud-project \
    --regions us-central1 \
    --logging gs://my-bucket/logging/ \
    --output OUT=gs://my-bucket/output/out.txt \
    --command 'echo "Hello World" > "${OUT}"' \
    --waitChangemy-cloud-projectto your Google Cloud project, andmy-bucketto
the bucket you created above.For thev2betaAPI (provider:google-cls-v2):dsub \
    --provider google-cls-v2 \
    --project my-cloud-project \
    --regions us-central1 \
    --logging gs://my-bucket/logging/ \
    --output OUT=gs://my-bucket/output/out.txt \
    --command 'echo "Hello World" > "${OUT}"' \
    --waitChangemy-cloud-projectto your Google Cloud project, andmy-bucketto
the bucket you created above.The output of the script command will be written to theOUTfile in Cloud
Storage that you specify.View the output file.gsutil cat gs://my-bucket/output/out.txtBackend providersWhere possible,dsubtries to support users being able to develop and test
locally (for faster iteration) and then progressing to running at scale.To this end,dsubprovides multiple "backend providers", each of which
implements a consistent runtime environment. The current providers are:localgoogle-v2 (the default)google-cls-v2 (new)More details on the runtime environment implemented by the backend providers
can be found indsub backend providers.Differences betweengoogle-v2andgoogle-cls-v2Thegoogle-cls-v2provider is built on the Cloud Life Sciencesv2betaAPI.
This API is very similar to its predecessor, the Genomicsv2alpha1API.
Details of the differences can be found in theMigration Guide.dsublargely hides the differences between the two APIs, but there are a
few difference to note:v2betais a regional service,v2alpha1is a global serviceWhat this means is that withv2alpha1, the metadata about your tasks
(called "operations"), is stored in a global database, while withv2beta, the
metadata about your tasks are stored in a regional database. If your operation
information needs to stay in a particular region, use thev2betaAPI
(thegoogle-cls-v2provider), and specify the--locationwhere your
operation information should be stored.The--regionsand--zonesflags can be omitted when usinggoogle-cls-v2The--regionsand--zonesflags fordsubspecify where the tasks should
run. More specifically, this specifies what Compute Engine Zones to use for
the VMs that run your tasks.With thegoogle-v2provider, there is no default region or zone, and thus
one of the--regionsor--zonesflags is required.Withgoogle-cls-v2, the--locationflag defaults tous-central1, and
if the--regionsand--zonesflags are omitted, thelocationwill be
used as the defaultregionslist.dsubfeaturesThe following sections show how to run more complex jobs.Defining what code to runYou can provide a shell command directly in the dsub command-line, as in the
hello example above.You can also save your script to a file, likehello.sh. Then you can run:dsub \
    ... \
    --script hello.shIf your script has dependencies that are not stored in your Docker image,
you can transfer them to the local disk. See the instructions below for
working with input and output files and folders.Selecting a Docker imageTo get started more easily,dsubuses a stock Ubuntu Docker image.
This default image may change at any time in future releases, so for
reproducible production workflows, you should always specify the image
explicitly.You can change the image by passing the--imageflag.dsub \
    ... \
    --image ubuntu:16.04 \
    --script hello.shNote: your--imagemust include theBashshell interpreter.For more information on using the--imageflag, see theimage section in Scripts, Commands, and DockerPassing parameters to your scriptYou can pass environment variables to your script using the--envflag.dsub \
    ... \
    --env MESSAGE=hello \
    --command 'echo ${MESSAGE}'The environment variableMESSAGEwill be assigned the valuehellowhen
your Docker container runs.Your script or command can reference the variable like any other Linux
environment variable, as${MESSAGE}.Be sure to enclose your command string in single quotes and not double
quotes. If you use double quotes, the command will be expanded in your local
shell before being passed to dsub. For more information on using the--commandflag, seeScripts, Commands, and DockerTo set multiple environment variables, you can repeat the flag:--env VAR1=value1 \
--env VAR2=value2You can also set multiple variables, space-delimited, with a single flag:--env VAR1=value1 VAR2=value2Working with input and output files and foldersdsub mimics the behavior of a shared file system using cloud storage
bucket paths for input and output files and folders. You specify
the cloud storage bucket path. Paths can be:file paths likegs://my-bucket/my-filefolder paths likegs://my-bucket/my-folderwildcard paths likegs://my-bucket/my-folder/*See theinputs and outputsdocumentation for more details.Transferring input files to a Google Cloud Storage bucket.If your script expects to read local input files that are not already
contained within your Docker image, the files must be available in Google
Cloud Storage.If your script has dependent files, you can make them available to your script
by:Building a private Docker image with the dependent files and publishing the
image to a public site, or privately to Google Container Registry or
Artifact RegistryUploading the files to Google Cloud StorageTo upload the files to Google Cloud Storage, you can use thestorage browserorgsutil. You can also run on data
that’s public or shared with your service account, an email address that you
can find in theGoogle Cloud Console.FilesTo specify input and output files, use the--inputand--outputflags:dsub \
    ... \
    --input INPUT_FILE_1=gs://my-bucket/my-input-file-1 \
    --input INPUT_FILE_2=gs://my-bucket/my-input-file-2 \
    --output OUTPUT_FILE=gs://my-bucket/my-output-file \
    --command 'cat "${INPUT_FILE_1}" "${INPUT_FILE_2}" > "${OUTPUT_FILE}"'In this example:a file will be copied fromgs://my-bucket/my-input-file-1to a path on the data diskthe path to the file on the data disk will be set in the environment variable${INPUT_FILE_1}a file will be copied fromgs://my-bucket/my-input-file-2to a path on the data diskthe path to the file on the data disk will be set in the environment variable${INPUT_FILE_2}The--commandcan reference the file paths using the environment variables.Also in this example:a path on the data disk will be set in the environment variable${OUTPUT_FILE}the output file will written to the data disk at the location given by${OUTPUT_FILE}After the--commandcompletes, the output file will be copied to the bucket pathgs://my-bucket/my-output-fileMultiple--input, and--outputparameters can be specified and
they can be specified in any order.FoldersTo copy folders rather than files, use the--input-recursiveandoutput-recursiveflags:dsub \
    ... \
    --input-recursive FOLDER=gs://my-bucket/my-folder \
    --command 'find ${FOLDER} -name "foo*"'Multiple--input-recursive, and--output-recursiveparameters can be
specified and they can be specified in any order.Mounting "resource data"While explicitly specifying inputs improves tracking provenance of your data,
there are cases where you might not want to expliclty localize all inputs
from Cloud Storage to your job VM.For example, if you have:a large set of resource filesyour code only reads a subset of those filesruntime decisions of which files to readORa large input file over which your code makes a single read passORa large input file that your code does not read in its entiretythen you may find it more efficient or convenient to access this data by
mounting read-only:a Google Cloud Storage bucketa persistent disk that you pre-create and populatea persistent disk that gets created from aCompute Engine Imagethat you
pre-create.Thegoogle-v2andgoogle-cls-v2providers support these methods of
providing access to resource data.Thelocalprovider supports mounting a
local directory in a similar fashion to support your local development.Mounting a Google Cloud Storage bucketTo have thegoogle-v2orgoogle-cls-v2provider mount a Cloud Storage bucket
usingCloud Storage FUSE,
use the--mountcommand line flag:--mount RESOURCES=gs://mybucketThe bucket will be mounted into the Docker container running your--scriptor--commandand the location made available via the environment variable${RESOURCES}. Inside your script, you can reference the mounted path using the
environment variable. Please readKey differences from a POSIX file systemandSemanticsbefore using Cloud Storage FUSE.Mounting an existing peristent diskTo have thegoogle-v2orgoogle-cls-v2provider mount a persistent disk that
you have pre-created and populated, use the--mountcommand line flag and the
url of the source disk:--mount RESOURCES="https://www.googleapis.com/compute/v1/projects/your-project/zones/your_disk_zone/disks/your-disk"Mounting a persistent disk, created from an imageTo have thegoogle-v2orgoogle-cls-v2provider mount a persistent disk created from an image,
use the--mountcommand line flag and the url of the source image and the size
(in GB) of the disk:--mount RESOURCES="https://www.googleapis.com/compute/v1/projects/your-project/global/images/your-image 50"The image will be used to create a new persistent disk, which will be attached
to a Compute Engine VM. The disk will mounted into the Docker container running
your--scriptor--commandand the location made available by the
environment variable${RESOURCES}. Inside your script, you can reference the
mounted path using the environment variable.To create an image, seeCreating a custom image.Mounting a local directory (localprovider)To have thelocalprovider mount a directory read-only, use the--mountcommand line flag and afile://prefix:--mount RESOURCES=file://path/to/my/dirThe local directory will be mounted into the Docker container running your--scriptor--commandand the location made available via the environment
variable${RESOURCES}. Inside your script, you can reference the mounted
path using the environment variable.Setting resource requirementsdsubtasks run using thelocalprovider will use the resources available on
your local machine.dsubtasks run using thegoogle,google-v2, orgoogle-cls-v2providers can take advantage
of a wide range of CPU, RAM, disk, and hardware accelerator (eg. GPU) options.See theCompute Resourcesdocumentation for details.Job IdentifiersBy default,dsubgenerates ajob-idwith the formjob-name--userid--timestampwhere thejob-nameis truncated at 10 characters
and thetimestampis of the formYYMMDD-HHMMSS-XX, unique to hundredths of a
second. If you are submitting multiple jobs concurrently, you may still run into
situations where thejob-idis not unique. If you require a uniquejob-idfor this situation, you may use the--unique-job-idparameter.If the--unique-job-idparameter is set,job-idwill instead be a unique 32
character UUID created byhttps://docs.python.org/3/library/uuid.html. Because
some providers require that thejob-idbegin with a letter,dsubwill
replace any starting digit with a letter in a manner that preserves uniqueness.Submitting a batch jobEach of the examples above has demonstrated submitting a single task with
a single set of variables, inputs, and outputs. If you have a batch of inputs
and you want to run the same operation over them,dsuballows you
to create a batch job.Instead of callingdsubrepeatedly, you can create
a tab-separated values (TSV) file containing the variables,
inputs, and outputs for each task, and then calldsubonce.
The result will be a singlejob-idwith multiple tasks. The tasks will
be scheduled and run independently, but can bemonitoredanddeletedas a group.Tasks file formatThe first line of the TSV file specifies the names and types of the
parameters. For example:--env SAMPLE_ID<tab>--input VCF_FILE<tab>--output OUTPUT_PATHEach addition line in the file should provide the variable, input, and output
values for each task. Each line beyond the header represents the values for a
separate task.Multiple--env,--input, and--outputparameters can be specified and
they can be specified in any order. For example:--env SAMPLE<tab>--input A<tab>--input B<tab>--env REFNAME<tab>--output O
S1<tab>gs://path/A1.txt<tab>gs://path/B1.txt<tab>R1<tab>gs://path/O1.txt
S2<tab>gs://path/A2.txt<tab>gs://path/B2.txt<tab>R2<tab>gs://path/O2.txtTasks parameterPass the TSV file to dsub using the--tasksparameter. This parameter
accepts both the file path and optionally a range of tasks to process.
The file may be read from the local filesystem (on the machine you're callingdsubfrom), or from a bucket in Google Cloud Storage (file name starts with
"gs://").For example, supposemy-tasks.tsvcontains 101 lines: a one-line header and
100 lines of parameters for tasks to run. Then:dsub ... --tasks ./my-tasks.tsvwill create a job with 100 tasks, while:dsub ... --tasks ./my-tasks.tsv 1-10will create a job with 10 tasks, one for each of lines 2 through 11.The task range values can take any of the following forms:mindicates to submit taskm(line m+1)m-indicates to submit all tasks starting with taskmm-nindicates to submit all tasks frommton(inclusive).LoggingThe--loggingflag points to a location fordsubtask log files. For details
on how to specify your logging path, seeLogging.Job controlIt's possible to wait for a job to complete before starting another.
For details, seejob control with dsub.RetriesIt is possible fordsubto automatically retry failed tasks.
For details, seeretries with dsub.Labeling jobs and tasksYou can add custom labels to jobs and tasks, which allows you to monitor and
cancel tasks using your own identifiers. In addition, with the Google
providers, labeling a task will label associated compute resources such as
virtual machines and disks.For more details, seeChecking Status and Troubleshooting JobsViewing job statusThedstatcommand displays the status of jobs:dstat --provider google-v2 --project my-cloud-projectWith no additional arguments, dstat will display a list ofrunningjobs for
the currentUSER.To display the status of a specific job, use the--jobsflag:dstat --provider google-v2 --project my-cloud-project --jobs job-idFor a batch job, the output will list allrunningtasks.Each job submitted by dsub is given a set of metadata values that can be
used for job identification and job control. The metadata associated with
each job includes:job-name: defaults to the name of your script file or the first word of
your script command; it can be explicitly set with the--nameparameter.user-id: theUSERenvironment variable value.job-id: identifier of the job, which can be used in calls todstatandddelfor job monitoring and canceling respectively. SeeJob Identifiersfor more
details on thejob-idformat.task-id: if the job is submitted with the--tasksparameter, each task
gets a sequential value of the form "task-n" wherenis 1-based.Note that the job metadata values will be modified to conform with the "Label
Restrictions" listed in theChecking Status and Troubleshooting Jobsguide.Metadata can be used to cancel a job or individual tasks within a batch job.For more details, seeChecking Status and Troubleshooting JobsSummarizing job statusBy default, dstat outputs one line per task. If you're using a batch job with
many tasks then you may benefit from--summary.$ dstat --provider google-v2 --project my-project --status '*' --summary

Job Name        Status         Task Count
-------------   -------------  -------------
my-job-name     RUNNING        2
my-job-name     SUCCESS        1In this mode, dstat prints one line per (job name, task status) pair. You can
see at a glance how many tasks are finished, how many are still running, and
how many are failed/canceled.Deleting a jobTheddelcommand will delete running jobs.By default, only jobs submitted by the current user will be deleted.
Use the--usersflag to specify other users, or'*'for all users.To delete a running job:ddel --provider google-v2 --project my-cloud-project --jobs job-idIf the job is a batch job, all running tasks will be deleted.To delete specific tasks:ddel \
    --provider google-v2 \
    --project my-cloud-project \
    --jobs job-id \
    --tasks task-id1 task-id2To delete all running jobs for the current user:ddel --provider google-v2 --project my-cloud-project --jobs '*'Service Accounts and Scope (Google providers only)When you run thedsubcommand with thegoogle-v2orgoogle-cls-v2provider, there are two different sets of credentials to consider:Account submitting thepipelines.run()request to run your command/script on a VMAccount accessing Cloud resources (such as files in GCS) when executing your command/scriptThe account used to submit thepipelines.run()request is typically your
end user credentials. You would have set this up by running:gcloud auth application-default loginThe account used on the VM is aservice account.
The image below illustrates this:By default,dsubwill use thedefault Compute Engine service accountas the authorized service account on the VM instance. You can choose to specify
the email address of another service acount using--service-account.By default,dsubwill grant the following access scopes to the service account:https://www.googleapis.com/auth/bigqueryhttps://www.googleapis.com/auth/computehttps://www.googleapis.com/auth/devstorage.full_controlhttps://www.googleapis.com/auth/genomicshttps://www.googleapis.com/auth/logging.writehttps://www.googleapis.com/auth/monitoring.writeIn addition,the APIwill always add this scope:https://www.googleapis.com/auth/cloud-platformYou can choose to specify scopes using--scopes.Recommendations for service accountsWhile it is straightforward to use the default service account, this account also
has broad privileges granted to it by default. Following thePrinciple of Least Privilegeyou may want to create and use a service account that has only sufficient privileges
granted in order to run yourdsubcommand/script.To create a new service account, follow the steps below:Execute thegcloud iam service-accounts createcommand. The email address
of the service account will besa-name@project-id.iam.gserviceaccount.com.gcloud iam service-accounts create "sa-name"Grant IAM access on buckets, etc. to the service account.gsutil iam ch serviceAccount:sa-name@project-id.iam.gserviceaccount.com:roles/storage.objectAdmin gs://bucket-nameUpdate yourdsubcommand to include--service-accountdsub \
   --service-account sa-name@project-id.iam.gserviceaccount.com
   ...What next?See the examples:Custom scriptsDecompress filesFastQCSamtools indexSee more documentation for:Scripts, Commands, and DockerInput and Output File HandlingLoggingCompute ResourcesCompute QuotasJob ControlRetriesChecking Status and Troubleshooting JobsBackend providers