Repository: DataBiosphere/toil

Issue:
cwltoil does not pass setEnv variables to cwl job
When running cwltoil with the `--no-container` option, jobs fail if the specified CWL `baseCommand` is not found in the runtime `PATH`. This is expected, but it seems there's no way to provide that path to cwltoil. There is a `--setEnv` argument that appears to be for this purpose (setting environment variables on jobs), but cwltoil does not honor it.

Running `cwltool`, the environment (including PATH) is inherited from the parent process/shell, so binaries specified in `baseCommand` are found if they're in the `PATH`.

cwltool command: 

```
PATH=$PATH:/home/dcl9/bin cwltool --no-container --outdir /home/dcl9/ggr-output/ /home/dcl9/GGR-cwl/qc/fastqc.cwl fastqc_job.json
```

results: [cwltool-succeed.txt](https://github.com/BD2KGenomics/toil/files/245337/cwltool-succeed.txt)

cwltoil command:

```
cwltoil --jobStore /data/tmp/toil-jobstore/ --no-container --setEnv PATH=/home/dcl9/bin --outdir /home/dcl9/ggr-output/ /home/dcl9/GGR-cwl/qc/fastqc.cwl fastqc_job.json
```

results: [cwltoil-fail.txt](https://github.com/BD2KGenomics/toil/files/245335/cwltoil-fail.txt)

```
2016-05-02 10:11:58,668 WARNING:toil.leader: P/p/jobGD9RLU: [job 28436624] /tmp/toil-24191727-ee15-4522-93b3-6be6f484b9a6/tmpIsrHjR/localTempDir/tyO2hO6/out$ fastqc -o /var/spool/cwl --noextract --format fastq /var/lib/cwl/job378892700_tmpSLsWRh/SRC1_ab2859_GR220236_6_160309_S18.tX_repX.fastq --threads 1 --dir /tmp
2016-05-02 10:11:58,668 WARNING:toil.leader: P/p/jobGD9RLU: INFO:cwltool:[job 28436624] /tmp/toil-24191727-ee15-4522-93b3-6be6f484b9a6/tmpIsrHjR/localTempDir/tyO2hO6/out$ fastqc -o /var/spool/cwl --noextract --format fastq /var/lib/cwl/job378892700_tmpSLsWRh/SRC1_ab2859_GR220236_6_160309_S18.tX_repX.fastq --threads 1 --dir /tmp
2016-05-02 10:11:58,668 WARNING:toil.leader: P/p/jobGD9RLU: 'fastqc' not found
2016-05-02 10:11:58,669 WARNING:toil.leader: P/p/jobGD9RLU: ERROR:cwltool:'fastqc' not found
2016-05-02 10:11:58,669 WARNING:toil.leader: P/p/jobGD9RLU: [job 28436624] completed permanentFail
```

fastqc_job.json:

```
{"input_fastq_file":{ "class": "File", "path": "/data/projects/GGR/data/chip_seq/processed_raw_reads/iter-1/SRC1_ab2859_GR220236_6_160309_S18.tX_repX.fastq" }}
```

fastqc.cwl: https://github.com/Duke-GCB/GGR-cwl/blob/master/qc/fastqc.cwl



Discussion:
@tetron, there seem to be two issues here. One is that cwltoil does not seem to honor `--setEnv` and the other one is that `PATH` is incorrect, and that's the one @dleehr wants to work around with `--setEnv`. The workaround is likely going to be ugly and brittle, because `--setEnv` isn't additive. If you did `--setEnv PATH=$PATH:/foo` you'd send the PATH from the leader to the workers which may not work if they are on different machines.

Hi, 

I'm experiencing this issue as well, `cwltoil` doesn't locate my `bcbio_nextgen.py` installation:

```
login01 /scratch/gcarrasco/tests/test_bcbio_cwl ~> cwltoil --setEnv PATH=$PATH --setEnv HOME=$HOME --no-container run_info-cwl-workflow/main-run_info-cwl.cwl run_info-cwl-workfl
ow/main-run_info-cwl-samples.json

...
2016-05-12 13:22:20,552 WARNING:toil.leader: g/Q/jobGOhpMH:     'bcbio_nextgen.py' not found
2016-05-12 13:22:20,552 WARNING:toil.leader: g/Q/jobGOhpMH:     ERROR:cwltool:'bcbio_nextgen.py' not found
...
```

`cwltool` _does_ work with the equivalent command `cwltool --verbose --preserve-environment PATH HOME --no-container run_info-cwl-workflow/main-run_info-cwl.cwl run_info-cwl-workflow/main-run_info-cwl-samples.json`

What's bcbio_nextgen.py and who/what is trying to invoke it? How do we know it is actually invoked without a path and not like, say, './bcbio_nextgen.py` which would bypass PATH lookup?

@guillermo-carrasco It would be nice to have the full source, the full output from a run with --logDebug and proof that `bcbio_nextgen.py` is indeed on PATH before you invoke cwltoil.

Using the single machine batch system (the default) `--setEnv PATH=$PATH --setEnv HOME=$HOME` is redundant because all environment variables should be inherited by default. @tetron, is there a way to print environment variables in CWL jobs?

Hi @hannes-ucsc , 

> What's bcbio_nextgen.py and who/what is trying to invoke it? How do we know it is actually invoked without a path and not like, say, './bcbio_nextgen.py` which would bypass PATH lookup?

`bcbio_nextgen.py` is the pipeline that `cwltool` should trigger. Please have a look at [this](http://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html#running) document to see how I'm running `cwltool`.

In the workflow definitions, you can see that the command is not a relative path:

```
baseCommand:
- bcbio_nextgen.py
- runfn
- batch_for_variantcall
- cwl
```

> It would be nice to have the full source, the full output from a run with --logDebug and proof that bcbio_nextgen.py is indeed on PATH before you invoke cwltoil.

```
 login01 /scratch/gcarrasco/tests/test_bcbio_cwl ~> which bcbio_nextgen.py
/cm/shared/apps/bcbio/20150720-devel/tools/bin/bcbio_nextgen.py
```

When running `cwltool` standalone, the example works fine:

```
login01 /scratch/gcarrasco/tests/test_bcbio_cwl ~> cwltool --verbose --preserve-environment PATH HOME --no-container run_info-cwl-workflow/main-run_info-cwl.cwl run_info-cwl-workflow/main-run_info-cwl-samples.json
```

Complete result file: [cwltool.txt](https://github.com/BD2KGenomics/toil/files/263324/cwltool.txt)

Instead, if I run through `cwltoil` with what _I think_ its the equivalent command, I get the error described above:

```
cwltoil --setEnv PATH=$PATH --setEnv HOME=$HOME --no-container run_info-cwl-workflow/main-run_info-cwl.cwl run_info-cwl-workflow/main-run_info-cwl-samples.json
```

Complete result file:  [cwltoil.txt](https://github.com/BD2KGenomics/toil/files/263328/cwltoil.txt)

That's why I think that the problem is with `cwltoil` not setting the environment properly when calling `cwltool`.

Hope that's more useful, 

Thanks!

what do you think about this @tetron ?

Yeah, I'm dumbfounded by this. In single machine mode --setEnv PATH=$PATH isn't needed because the worker process inherits the leader process' environment (cwltoil being the leader here) by default.

Could it be that cwltoil.py needs to programmatically do what --preserve-environment does for cwltoil?

Yes I imagine, and that's what I though `--setEnv` does, but it may be trickier than that :-/

s/does for cwltoil/does for cwltool/ in my earlier comment

I'm tinkering with this in a [branch](https://github.com/Duke-GCB/toil/tree/try-preserve-environment) and made some progress today.

It doesn't seem to be a problem with `--setEnv` actually, we just need some CLI arguments propagated to `CWLJob.run()`.

`CWLJob.run()` calls [cwltool.main.single_job_executor()](https://github.com/BD2KGenomics/toil/blob/e1fcdf6e1eb3877d95513b79cae3c42446eecce3/src/toil/cwl/cwltoil.py#L212), which eventually calls down to [cwltool.job.run()](https://github.com/common-workflow-language/cwltool/blob/d55f6e0b8e4b4467600345e6048561c236684adb/cwltool/job.py#L63) with some kwargs (outdir, tmpdir, use_container).

When I [include preserve-environment and set use_container to False](https://github.com/Duke-GCB/toil/blob/806ae4534a0f6dadb768c42e8b0b70dd881acb49/src/toil/cwl/cwltoil.py#L216), my above fastqc workflow runs.

I'll take a crack at passing preserve-environment along (defaulting to `["PATH"]` as cwltool does), as well as honoring no-container.

@dleehr have you had a look at my current PR #863 ? If he problem is with `--preserve-environment` we could actually add this as an option to `cwltoil` and pass it through `executor_options`. What do you think?

@guillermo-carrasco That looks like exactly what we need, great!


Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -518,6 +518,11 @@ def main(args=None, stdout=sys.stdout):
     parser.add_argument("--basedir", type=str)
     parser.add_argument("--outdir", type=str, default=os.getcwd())
     parser.add_argument("--version", action='version', version=version)
+    parser.add_argument("--preserve-environment", type=str, nargs='+',
+                    help="Preserve specified environment variables when running CommandLineTools",
+                    metavar=("VAR1,VAR2"),
+                    default=("PATH",),
+                    dest="preserve_environment")
 
     # mkdtemp actually creates the directory, but
     # toil requires that the directory not exist,
@@ -565,8 +570,8 @@ def main(args=None, stdout=sys.stdout):
         adjustFiles(job, lambda x: x.replace("file://", ""))
         stdout.write(json.dumps(
             cwltool.main.single_job_executor(t, job, options.basedir, options,
-                                             conformance_test=True, use_container=use_container),
-                                             indent=4))
+                                             conformance_test=True, use_container=use_container,
+                                             preserve_environment=options.preserve_environment), indent=4))
         return 0
 
     if not options.basedir:
@@ -582,7 +587,7 @@ def importDefault(tool):
         t.visit(importDefault)
 
         builder = t._init_job(job, os.path.dirname(os.path.abspath(options.cwljob)))
-        (wf1, wf2) = makeJob(t, {}, use_container=use_container)
+        (wf1, wf2) = makeJob(t, {}, use_container=use_container, preserve_environment=options.preserve_environment)
         adjustFiles(builder.job, lambda x: "file://%s" % x if not urlparse.urlparse(x).scheme else x)
         adjustFiles(builder.job, functools.partial(writeFile, toil.importFile, {}))
         wf1.cwljob = builder.job


Test patch: