Repository: DataBiosphere/toil

Issue:
[CWL] Needs method to avoid copying files when a shared filesystem is available
Paths of `File` and `Directory` objects residing on an identified shared file system should be exempted from copying and just passed directly to the node.

To identify these shared filesystem a `cwltoil` command line option specifying the root path could be added.

Inspiration: 
> hcraT @hcraT 12:59
> Hi I'm writing a pipeline in cwl to run some analyses on biological data. I was able to use toil to run some tests.
> However I realized that toil copies my input files that are quite big. It is possible to avoid this behavior?
> I run my jobs on a batch system.
> I actually fed in as input to the pipeline symbolic links to the actual files.
> Thanks

https://gitter.im/bd2k-genomics-toil/Lobby?at=58f09d818fcce56b20ff77e4

> Pavlo Lutsik @lutsik 14:08
> I echo @hcraT. I probably have a very similar gridEngine-based setup, and copying of large files was a pain. I also implemented a similar workaround: simply eliminated the File type, working with string paths only. I had to adapt all the cwl wrappers and write my own cleanup steps, but the payoff in terms of performance was tremendous.

https://gitter.im/bd2k-genomics-toil/Lobby?at=58f0ada6f22385553d3e440f





Discussion:
@cket and all -- I've been doing some larger scale tests with cwltoil and am also running into what I believe is the same issue. Steps like variantcalling are very slow to spin up because it appears to be copying the input BAM files over to each step before running.

I'm happy to help try to dig into this but am not sure about where to start:

- Is copying everything to the isolated run directory the expected behavior of Toil? Or is this specific to the CWL implementation?
- The later, how does Toil decide when to copy and when to re-use an existing local file?
- Is it possible to also do something similar for S3 filestores on AWS, to avoid downloading and staging multiple times on the same machine?

Thanks for any suggestions and pointers.
Hi @mr-c, @chapmanb, I've been looking into this issue.

> Is copying everything to the isolated run directory the expected behavior of Toil? Or is this specific to the CWL implementation?

No matter what job store is being used (local filesystem, AWS/S3, etc) Toil will, if using caching (the default behavior), check to see if the target of `readGlobalFile` already exists locally. If it does then Toil will create a new hard link to the file in the job's work directory instead of re-downloading or copying the file. 

> The later, how does Toil decide when to copy and when to re-use an existing local file?

With caching Toil will always re-use an existing local file. Obviously this doesn't work if the user isn't using caching (or can't use caching due to using a batch system that doesn't support caching), or if the file has been removed from the cache due to lack of space. 

The final case is when the user specifically requests a mutable copy of the file, but in that case the file still isn't re-downloaded but it would be copied instead of just hard linked. 

>Is it possible to also do something similar for S3 filestores on AWS, to avoid downloading and staging multiple times on the same machine?

This shouldn't be happening since the caching logic is independent of the storage. 

I want to expand on a couple expected Toil behaviors as well. CWLToil does currently copy input files into the job store at the beginning of the workflow. We could change this to do hard links instead, when applicable, and that could save us some space & time. 

Independent of caching, Toil with a file job store will default to hard linking files from the job store rather than copying them. It will only copy files if the job store is on a file system separate from the one the job is executing on. 

Hopefully that provides background on what is and isn't expected in Toil. @chapmanb will also have a conversation later today that will probably involve this issue and how to move forward with addressing it.
CJ;
Thanks so much for the details and expected behavior, and for discussing last week. To follow up I ran the GA4GH bcbio validation workflow (NA12878 germline calling on chromosome 20 -- https://github.com/bcbio/bcbio_validation_workflows) on a single AWS machine with 8 cores. The slow section is variant calling where bcbio splits into 200 jobs to run variant calling with multiple callers on regions of the genome. Each job should use 1 core and 3Gb of memory so we should be able to run 8 in parallel. Here is the debug log for the run with all of the details to see if there's anything useful we can pick from it:

https://gist.github.com/chapmanb/9630a99a12e361932965c1e9ad5163ef

There is not a lot there to go on, mostly a bunch of messages from the waiting jobs. The only thing I noticed was acquiring and releasing locks on the filestore files. If all of these job read from a single BAM file, can they run in parallel and all read it at the same time?

Is there any other debugging output I can get from the jobstore or anywhere else to provide more help debugging why there is so much waiting relative to variant callers running? Thanks much.
Hi @chapmanb, whats standing out to me right now is the message

```DEBUG:toil.worker:No more jobs can run in series by this worker, it's got 199 children```

thats repeated after each job. The first message correctly (?) lists 200 children, but am I correct in assuming the subsequent steps shouldn't have 199 children each?

It kinda seems like the job DAG is wrong - maybe the parent has 200 variant calling children, but those are children of _each other_ in such a way that makes them run serially?

That explains the slowness as well as the log messages. 
CJ;
Thanks much for looking at this. I don't think the 200 children are children of each other, at least that's not the way it's wired up in the CWL. What happens is that the original sample gets split into the 200 parts, those can run in parallel (ideally up to 8 at a time in this setup) and then the 200 samples get merged back together in the next step. Is it possible that's what's meant by the message -- that it can't go forward with any other parts of the workflow until these 200 finish?

I've been watching it during the run and things do run in parallel but just not efficiently. For instance, here's 3 variant calling jobs (1 samtools, 2 freebayes) all going strong at the same time:
```
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
32227 ubuntu    20   0  252952 228028   3892 R 100.0  0.7   0:21.65 samtools
31511 ubuntu    20   0   97864  82404   5260 R  99.7  0.3   3:03.51 freebayes
31059 ubuntu    20   0   92028  76688   5352 R  99.3  0.2   5:09.73 freebayes
```
But that's pretty rare, in general you only see 1 or 2 cores getting much use at a time while there are a lot of _toil_worker processes idling without doing much.

It's been running a lot longer so I have more logging goodness here if anything else useful pops out:

https://s3.amazonaws.com/chapmanb/testcases/toil/NA12878-platinum-chr20-toil.log.gz

From my side, is there any sort of API or approach to dig into the job store? Do finished jobs store metrics about the run? Longer term I'd love to have more control over job stores to be able to do things like expire all variant calling steps to allow re-running from the alignments (for debugging cases like these), but I'm not sure how to get started with the directory full of arbitrarily named pickle files.

Thanks for any thoughts or pointers to dig more.
CJ;
I dug into this a bit more by running the same CWL on the rabix bunny runner to confirm everything is wired together correctly in terms of parallelism and now I understand where the `199 children` messages come from. There are 3 variant callers, each with 199 (well, 200 but whatever) jobs that can start after the initial job splits that variantcaller and sample into genomic regions to process. So I think the 3 messages about that come from the initial split job, then the runner is free to dispatch any of the 600 jobs together. Bunny did allow running all these in parallel (well, it launched 600 jobs in parallel on an 8 core machine which is a separate issue, but there's no inherit dependency between the jobs).

Hope this helps some with explaining what is going on. Thanks again for all the discussion and help.
@chapmanb thanks for following up with the scheduling, that helps explain the behavior. I think it's important that you saw the toil worker processes even if they are idle. That is indicative that toil is scheduling & parallelizing those jobs properly, which I think supports your original diagnosis that the slowness is due to file provisioning.

I also found a traceback in the logs that further support this: 
```
F/D/jobkKebyf    Traceback (most recent call last):
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 340, in main
F/D/jobkKebyf        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1289, in _runner
F/D/jobkKebyf        returnValues = self._run(jobGraph, fileStore)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1234, in _run
F/D/jobkKebyf        return self.run(fileStore)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 279, in run
F/D/jobkKebyf        adjustFilesWithSecondary(cwljob, functools.partial(getFile, fileStore, inpdir, index=index))
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/process.py", line 179, in adjustFilesWithSecondary
F/D/jobkKebyf        adjustFilesWithSecondary(rec[d], op)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/process.py", line 182, in adjustFilesWithSecondary
F/D/jobkKebyf        adjustFilesWithSecondary(d, op, primary)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/process.py", line 179, in adjustFilesWithSecondary
F/D/jobkKebyf        adjustFilesWithSecondary(rec[d], op)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/process.py", line 174, in adjustFilesWithSecondary
F/D/jobkKebyf        rec["path"] = op(rec["path"], primary=primary)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 157, in getFile
F/D/jobkKebyf        shutil.copyfile(srcPath, dstPath)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/shutil.py", line 84, in copyfile
F/D/jobkKebyf        copyfileobj(fsrc, fdst)
F/D/jobkKebyf      File "/mnt/work/bcbio/bcbio-vm/anaconda/lib/python2.7/shutil.py", line 52, in copyfileobj
F/D/jobkKebyf        fdst.write(buf)
F/D/jobkKebyf    IOError: [Errno 28] No space left on device
F/D/jobkKebyf    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-10-0-0-181
```
I think that explains why the behavior we're seeing is different from what I expected. CWLToil is using `getFile`, which can optionally copy the file when provisioning. I'm not sure how easy that will be to work around.

Finally, to answer your questions, there unfortunately isn't a good way to do dig into the job store. I've done it by checking the logging messages that map the job's name to file store IDs in the job store, but that can be pretty labor intensive. 

Hope this makes sense, and let me know if you have any ideas about `getFile`!
CJ;
Great find, thanks for identifying that the copy comes from within cwltool. This looks to be a bug introduced when refactoring this code, the `copy` keyword argument got moved to a different name but never changed here. As a result the `if copy` gets the `copy` module and always returns true, doing a copy by default. I fixed this and swapped over to use symlinks instead but this exposed some problems with the ctwltool interaction. cwltool resolves symlinks by default so we end up with references to the internal Toil store names (`tmp12345.tmp`) instead of the actual file names. I have a couple of PRs in to cwltool to resolve this, then will try to finalize the fix here. I appreciate the help, it's awesome to have a practical path forward on this one.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -40,7 +40,6 @@
 import sys
 import logging
 import copy
-import shutil
 import functools
 
 # Python 3 compatibility imports
@@ -127,7 +126,16 @@ def resolve_indirect(d):
     else:
         return res
 
-def getFile(fileStore, dir, fileTuple, index=None, export=False, primary=None, rename_collision=False):
+def getFile(fileStore, dir, fileTuple, index=None, export=False, primary=None, rename_collision=False,
+            existing={}):
+    """Extract input file from Toil jobstore.
+
+    Uses standard filestore to retrieve file, then provides a symlink to it
+    for running. If export is True (for final outputs), it gets copied to
+    the final location.
+
+    Keeps track of files being used locally with 'existing'
+    """
     # File literal outputs with no path, from writeFile
     if fileTuple is None:
         raise cwltool.process.UnsupportedRequirement("CWL expression file inputs not yet supported in Toil")
@@ -153,18 +161,21 @@ def getFile(fileStore, dir, fileTuple, index=None, export=False, primary=None, r
     else:
         srcPath = fileStore.readGlobalFile(fileStoreID)
         if srcPath != dstPath:
-            if copy:
-                shutil.copyfile(srcPath, dstPath)
+            if os.path.exists(dstPath):
+                if index.get(dstPath, None) != fileStoreID:
+                    raise Exception("Conflicting filesStoreID %s and %s both trying to link to %s" % (index.get(dstPath, None), fileStoreID, dstPath))
             else:
-                if os.path.exists(dstPath):
-                    if index.get(dstPath, None) != fileStoreID:
-                        raise Exception("Conflicting filesStoreID %s and %s both trying to link to %s" % (index.get(dstPath, None), fileStoreID, dstPath))
-                else:
-                    os.symlink(srcPath, dstPath)
+                os.symlink(srcPath, dstPath)
+                existing[srcPath] = dstPath
             index[dstPath] = fileStoreID
     return dstPath
 
-def writeFile(writeFunc, index, x):
+def writeFile(writeFunc, index, existing, x):
+    """Write output files back into Toil jobstore.
+
+    'existing' is a set of files retrieved as inputs from getFile. This ensures
+    they are mapped back as the same name if passed through.
+    """
     # Toil fileStore references are tuples of pickle and internal file
     if isinstance(x, tuple):
         return x
@@ -174,6 +185,7 @@ def writeFile(writeFunc, index, x):
         return None
     else:
         if x not in index:
+            x = existing.get(x, x)
             if not urlparse.urlparse(x).scheme:
                 rp = os.path.realpath(x)
             else:
@@ -200,8 +212,7 @@ def addFilePartRefs(p):
         nameroot, nameext = os.path.splitext(basename)
         for k, v in [("dirname", dirname,), ("basename", basename),
                      ("nameroot", nameroot), ("nameext", nameext)]:
-            if k not in p:
-                p[k] = v
+            p[k] = v
     return p
 
 def locToPath(p):
@@ -276,8 +287,11 @@ def run(self, fileStore):
 
         # Copy input files out of the global file store, ensure path/location synchronized
         index = {}
-        adjustFilesWithSecondary(cwljob, functools.partial(getFile, fileStore, inpdir, index=index))
+        existing = {}
+        adjustFilesWithSecondary(cwljob, functools.partial(getFile, fileStore, inpdir, index=index,
+                                                           existing=existing))
         cwltool.pathmapper.adjustFileObjs(cwljob, pathToLoc)
+        cwltool.pathmapper.adjustFileObjs(cwljob, addFilePartRefs)
 
         # Run the tool
         opts = copy.deepcopy(self.executor_options)
@@ -297,8 +311,7 @@ def run(self, fileStore):
         cwltool.pathmapper.adjustFileObjs(output, functools.partial(computeFileChecksums,
                                                                     cwltool.stdfsaccess.StdFsAccess(outdir)))
         # Copy output files into the global file store.
-        adjustFiles(output, functools.partial(writeFile, fileStore.writeGlobalFile, {}))
-
+        adjustFiles(output, functools.partial(writeFile, fileStore.writeGlobalFile, {}, existing))
         return output
 
 
@@ -606,6 +619,15 @@ def unsupportedInputCheck(p):
     if p.get("contents") and (not p.get("path") and not p.get("location")):
         raise cwltool.process.UnsupportedRequirement("CWL File literals not yet supported in Toil")
 
+def unsupportedRequirementsCheck(requirements):
+    """Check for specific requirement cases we don't support.
+    """
+    for r in requirements:
+        if r["class"] == "InitialWorkDirRequirement":
+            for l in r.get("listing", []):
+                if isinstance(l, dict) and l.get("writable"):
+                    raise cwltool.process.UnsupportedRequirement("CWL writable InitialWorkDirRequirement not yet supported in Toil")
+
 def unsupportedDefaultCheck(tool):
     """Check for file-based defaults, which don't get staged correctly in Toil.
     """
@@ -658,6 +680,7 @@ def main(args=None, stdout=sys.stdout):
     try:
         t = cwltool.load_tool.load_tool(options.cwltool, cwltool.workflow.defaultMakeTool,
                                         resolver=cwltool.resolver.tool_resolver, strict=useStrict)
+        unsupportedRequirementsCheck(t.requirements)
     except cwltool.process.UnsupportedRequirement as e:
         logging.error(e)
         return 33
@@ -710,7 +733,7 @@ def importDefault(tool):
             cwltool.pathmapper.adjustDirObjs(tool, locToPath)
             cwltool.pathmapper.adjustFileObjs(tool, locToPath)
             adjustFiles(tool, lambda x: "file://%s" % x if not urlparse.urlparse(x).scheme else x)
-            adjustFiles(tool, functools.partial(writeFile, toil.importFile, {}))
+            adjustFiles(tool, functools.partial(writeFile, toil.importFile, {}, {}))
         t.visit(importDefault)
 
         if options.restart:
@@ -735,14 +758,14 @@ def importDefault(tool):
             cwltool.pathmapper.adjustDirObjs(builder.job, pathToLoc)
             cwltool.pathmapper.adjustFileObjs(builder.job, pathToLoc)
             cwltool.pathmapper.adjustFileObjs(builder.job, addFilePartRefs)
-            adjustFiles(builder.job, functools.partial(writeFile, toil.importFile, {}))
+            adjustFiles(builder.job, functools.partial(writeFile, toil.importFile, {}, {}))
             wf1.cwljob = builder.job
             outobj = toil.start(wf1)
 
         outobj = resolve_indirect(outobj)
 
         try:
-            adjustFilesWithSecondary(outobj, functools.partial(getFile, toil, outdir, index={},
+            adjustFilesWithSecondary(outobj, functools.partial(getFile, toil, outdir, index={}, existing={},
                                                                export=True, rename_collision=True))
             cwltool.pathmapper.adjustFileObjs(outobj, pathToLoc)
         except cwltool.process.UnsupportedRequirement as e:


Test patch: