Repository: DataBiosphere/toil

Issue:
toil writes to /tmp although --tmpdir-prefix is set
Hi. I'm running toil (3.19.0) like:
```
$ /toil-cwl-runner --cleanWorkDir always \               
        --logFile $(CURDIR)/log --outdir $(CURDIR)/results \                
        --jobStore file:///$(CURDIR)/$(RUN)/job_store \               
        --workDir $(CURDIR)/work --tmpdir-prefix=$(CURDIR)/tmp --clean always                  
```
My pipeline creates quite some huge intermediate files, which toil writes to `/tmp`. I can't have that in `/tmp` since no space, but `--tmpdir-prefix` doesn't seem to have much effect. 



Discussion:
it also happens when I set `--tmp-outdir-prefix`
This is probably related:

https://gist.github.com/gijzelaerr/eccd3cb159b312e2e60810e28fea20cf

I'm running toil on a cluster where `/tmp` is not shared. i've set both `tmpdir` and `tmpout-dir` prefixes, but still toil looks in `/tmp`
setting a temp dir using a `TMPDIR` environmental variable seems to do the trick.
I suspect the offending code is here:

https://github.com/DataBiosphere/toil/blob/6a18b8a6a4ad885729ef9fa53542e0a6fd4b6c0a/src/toil/cwl/cwltoil.py#L585-L588

Your temp files are all under directories with "out_tmpdir" in the name, and this is where that is in the code.

The underlying problem I think is confusion between `workdir` and `options.workDir` here:

https://github.com/DataBiosphere/toil/blob/6a18b8a6a4ad885729ef9fa53542e0a6fd4b6c0a/src/toil/cwl/cwltoil.py#L1119-L1130

We try and override the working directory when a `--tmpdir-prefix` is set, but we only seem to modify it in the options, and the actual local `workdir`, which stays under /tmp, is what is getting passed along to the jobs to work in. Setting `TMPDIR` in the environment is affecting that initial `mkdtemp()` and getting it put in the right place.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1112,22 +1112,40 @@ def main(args=None, stdout=sys.stdout):
         "--no-read-only", action="store_true", default=False,
         help="Do not set root directory in the container as read-only")
 
+    if args is None:
+        args = sys.argv[1:]
+
+    # Problem: we want to keep our job store somewhere auto-generated based on
+    # our options, unless overridden by... an option. So we will need to parse
+    # options twice, because we need to feed the parser the job store.
+    
+    # Propose a local workdir, probably under /tmp.
     # mkdtemp actually creates the directory, but
     # toil requires that the directory not exist,
+    # since it is going to be our jobstore,
     # so make it and delete it and allow
     # toil to create it again (!)
     workdir = tempfile.mkdtemp()
     os.rmdir(workdir)
 
-    if args is None:
-        args = sys.argv[1:]
-
-    # we use workdir as jobStore:
+    # we use workdir as default default jobStore:
     options = parser.parse_args([workdir] + args)
 
-    # if tmpdir_prefix is not the default value, set workDir too
+    # if tmpdir_prefix is not the default value, set workDir if unset, and move
+    # workdir and the job store under it
     if options.tmpdir_prefix != 'tmp':
-        options.workDir = options.tmpdir_prefix
+        workdir = tempfile.mkdtemp(dir=options.tmpdir_prefix)
+        os.rmdir(workdir)
+        # Re-parse arguments with the new default jobstore under the temp dir.
+        # It still might be overridden by a --jobStore option
+        options = parser.parse_args([workdir] + args)
+        if options.workDir is None:
+            # We need to override workDir because by default Toil will pick
+            # somewhere under the system temp directory if unset, ignoring
+            # --tmpdir-prefix.
+            #
+            # If set, workDir needs to exist, so we directly use the prefix
+            options.workDir = options.tmpdir_prefix
 
     if options.provisioner and not options.jobStore:
         raise NoSuchJobStoreException(


Test patch: