Repository: DataBiosphere/toil

Issue:
Kubernetes batch system doesn't provide Docker for CWL workflows that use Docker
On Gitter, it was reported that if you take our Kubernetes batch system and try to use it to run a CWL workflow that uses a DcokerRequirement, it will fail.

The Kubernetes batch system does not privilege and start up Docker inside the pod, because we haven't usually needed it: at UCSC we use user-mode Singularity to run containers when on Kubernetes. But `cwlrunner` insists on using Docker, and the Singularity code isn't part of Toil itself while Docker-running code is.

We should make it possible for the Kubernetes batch system to launch all workers as privileged, if the user asks for it, and to start up a Docker daemon before running jobs. Then, we can make the CWL execution code ask for this, either when it sees that the workflow is going to need Docker, all the time unless instructed not to, or maybe just when instructed to do so, depending on what seems most user-friendly.

The right way to do this would be a needs-docker requirement on the jobs, I think, but we don't really have the infrastructure to support one-off annotation requirements like that yet. So I think it has to be all or nothing.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-662)
┆Issue Number: TOIL-662



Discussion:
We could also change `cwlrunner` to use user-mode Singularity like we do, when it can't use Docker.
There is already support for singularity in `toil-cwl-runner` (and `cwltool`) via the `--singularity` flag.

A PR to cwltool to automatically try singularity if `docker` fails would be very welcome! After meeting that PR then the additional functionality would flow to `toil-cwl-runner` with a one line PR to update the cwltool version pin.

FYI @adamnovak `toil-cwl-runner` uses `cwltool`, not `cwl-runner`.`cwl-runner` is the generic entry point for any CWL aware workflow engine.
Oh OK.

Maybe we can make `toil-cwl-runner` automatically infer `--singularity` when it sees `--batchSystem kubernetes`.

Does `cwltool` have the functionality to make sure that no more than one Singularity command is trying to fetch a container at a time for a given cache directory?

With Singularity on Kubernetes we usually recommend using the feature we have to mount a directory from the host into the pod to use as a Singularity cache, so each job doesn't have to re-fetch the container. But `singularity` itself requires external synchronization on a shared cache, so we had to write that in our workflows.
@adamnovak the code for pulling Singularity images is [here](https://github.com/common-workflow-language/cwltool/blob/main/cwltool/singularity.py#L152), there is no locking. A simple test with this tool:

```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

requirements:
  DockerRequirement:
    dockerPull: busybox

inputs:
  message:
    type: string
    default: "message"
    inputBinding:
      position: 1

outputs:
  output:
    type: stdout

baseCommand: [ echo ]
```
and this script

```
#!/bin/bash

CWL_SINGULARITY_CACHE=/tmp/cwl/cache
export CWL_SINGULARITY_CACHE
if [ -d $CWL_SINGULARITY_CACHE ] ; then
    rm -rf $CWL_SINGULARITY_CACHE
fi
mkdir $CWL_SINGULARITY_CACHE

cwltool --singularity test_pull.cwl & cwltool --singularity test_pull.cwl
```
Shows this result:
```
$ bash test_pull.sh 
INFO /home/pvh/anaconda3/envs/cwltool/bin/cwltool 3.0.20200530110633
INFO Resolved 'test_pull.cwl' to 'file:///tmp/cwl/test_pull.cwl'
INFO /home/pvh/anaconda3/envs/cwltool/bin/cwltool 3.0.20200530110633
INFO Resolved 'test_pull.cwl' to 'file:///tmp/cwl/test_pull.cwl'
INFO ['singularity', 'pull', '--force', '--name', '/tmp/cwl/cache/busybox.sif', 'docker://busybox']
INFO ['singularity', 'pull', '--force', '--name', '/tmp/cwl/cache/busybox.sif', 'docker://busybox']
INFO [job test_pull.cwl] /tmp/ha68cs9l$ singularity \
    --quiet \
    exec \
    --contain \
    --pid \
    --ipc \
    --home \
    /tmp/ha68cs9l:/OvDltV \
    --bind \
    /tmp/mx6a976k:/tmp:rw \
    --pwd \
    /OvDltV \
    /tmp/cwl/cache/busybox.sif \
    echo \
    message > /tmp/ha68cs9l/1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584
INFO [job test_pull.cwl] /tmp/curn1nzn$ singularity \
    --quiet \
    exec \
    --contain \
    --pid \
    --ipc \
    --home \
    /tmp/curn1nzn:/nMGZid \
    --bind \
    /tmp/3rq97h_2:/tmp:rw \
    --pwd \
    /nMGZid \
    /tmp/cwl/cache/busybox.sif \
    echo \
    message > /tmp/curn1nzn/1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584
INFO [job test_pull.cwl] completed success
{
    "output": {
        "location": "file:///tmp/cwl/1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584",
        "basename": "1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584",
        "class": "File",
        "checksum": "sha1$1133e3acf0a4cbb9d8b3bfd3f227731b8cd2650b",
        "size": 8,
        "path": "/tmp/cwl/1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584"
    }
}
INFO Final process status is success
INFO [job test_pull.cwl] completed success
{
    "output": {
        "location": "file:///tmp/cwl/1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584",
        "basename": "1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584",
        "class": "File",
        "checksum": "sha1$1133e3acf0a4cbb9d8b3bfd3f227731b8cd2650b",
        "size": 8,
        "path": "/tmp/cwl/1c5048ae62dd8df5f4f43fbb295aa64d4bc4b584"
    }
}
INFO Final process status is success
```
So no errors. Further inspection suggests that `singularity pull` at least attempts to be atomic, i.e. there is no file in the cache dir between `singularity pull` starting and it completing. 
The equivalent discussion for WDL/Cromwell: https://github.com/broadinstitute/cromwell/issues/5063
BTW. a [comment](https://sylabs.io/guides/3.6/user-guide/build_env.html) in the Singularity docs suggests that `singularity pull` uses atomic rename as I suspected.
With regards to the singularity cache, seems that with singularity version 3.6 there is:

> A new cache structure is used that is concurrency safe on a filesystem that supports atomic rename. 

https://github.com/hpcng/singularity/blob/master/CHANGELOG.md#new-features--functionalities-2

See also https://github.com/biowdl/singularity-permanent-cache
@mr-c That's great! Maybe we can just upgrade the Singularity version that Toil ships to 3.6 and call it done.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -30,7 +30,7 @@ def runSetup():
     gcs = 'google-cloud-storage==1.6.0'
     gcs_oauth2_boto_plugin = 'gcs_oauth2_boto_plugin==1.14'
     apacheLibcloud = 'apache-libcloud==2.2.1'
-    cwltool = 'cwltool==3.0.20200807132242'
+    cwltool = 'cwltool==3.0.20201017180608'
     galaxyToolUtil = 'galaxy-tool-util'
     htcondor = 'htcondor>=8.6.0'
     kubernetes = 'kubernetes>=10, <11'
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1623,6 +1623,9 @@ def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
             'Please specify a jobstore with the --jobStore option when '
             'specifying a provisioner.')
 
+    if options.batchSystem == 'kubernetes':
+        options.singularity = True
+
     use_container = not options.no_container
 
     if options.logLevel:


Test patch: