Repository: DataBiosphere/toil

Issue:
toil-cwl-runner downloads input files twice when --disableCaching option is present
I'm running a single-machine `toil-cwl-runner` workflow. I am investigating how files are downloaded from jobstore for GSoC project regarding [#3469](https://github.com/DataBiosphere/toil/issues/3469). The CWL workflow does only a `cat` command:
```
#!/usr/bin cwl-runner
#stream.cwl file

cwlVersion: v1.2
class: CommandLineTool
baseCommand: cat

stdout: $(inputs.output_filename)

inputs:
  input_file:
    type: File
    streamable: true
    inputBinding:
      position: 1
  output_filename:
    type: string?
    default: output.txt

outputs:
  out:
    type: stdout
```

I'm running with the following options:
```toil-cwl-runner --disableCaching --debugWorker --logLevel DEBUG --retryCount 1 --workDir=workdir --outdir=outdir --jobStore=aws:eu-west-1:toil-stream  stream.cwl stream-s3.yml```
I also have `TMPDIR` set, but nothing is written there.
I'm running Toil version `5.3.0a1-c4476778aa867caabe7202a84e65697ff7d501eb`

It seems that `toil-cwl-runner` downloads the input file twice in the work directory. First download is in `ToilFsAccess._abs` function: https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/cwl/cwltoil.py#L725 
Second download is in: https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/cwl/cwltoil.py#L750

Both functions call `nonCachingFileStore.readGlobalFile` which downloads the file:
https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/fileStores/nonCachingFileStore.py#L83

Is there a reason why the second call could not use the mapping from the first time?
https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/fileStores/nonCachingFileStore.py#L92

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-934)
┆Issue Number: TOIL-934

toil-cwl-runner downloads input files twice when --disableCaching option is present
I'm running a single-machine `toil-cwl-runner` workflow. I am investigating how files are downloaded from jobstore for GSoC project regarding [#3469](https://github.com/DataBiosphere/toil/issues/3469). The CWL workflow does only a `cat` command:
```
#!/usr/bin cwl-runner
#stream.cwl file

cwlVersion: v1.2
class: CommandLineTool
baseCommand: cat

stdout: $(inputs.output_filename)

inputs:
  input_file:
    type: File
    streamable: true
    inputBinding:
      position: 1
  output_filename:
    type: string?
    default: output.txt

outputs:
  out:
    type: stdout
```

I'm running with the following options:
```toil-cwl-runner --disableCaching --debugWorker --logLevel DEBUG --retryCount 1 --workDir=workdir --outdir=outdir --jobStore=aws:eu-west-1:toil-stream  stream.cwl stream-s3.yml```
I also have `TMPDIR` set, but nothing is written there.
I'm running Toil version `5.3.0a1-c4476778aa867caabe7202a84e65697ff7d501eb`

It seems that `toil-cwl-runner` downloads the input file twice in the work directory. First download is in `ToilFsAccess._abs` function: https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/cwl/cwltoil.py#L725 
Second download is in: https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/cwl/cwltoil.py#L750

Both functions call `nonCachingFileStore.readGlobalFile` which downloads the file:
https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/fileStores/nonCachingFileStore.py#L83

Is there a reason why the second call could not use the mapping from the first time?
https://github.com/DataBiosphere/toil/blob/6b882c6ed44378a67bd3e7722ff7bba6ab9289ea/src/toil/fileStores/nonCachingFileStore.py#L92

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-934)
┆Issue Number: TOIL-934



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -746,6 +746,12 @@ def _abs(self, path: str) -> str:
             destination = super(ToilFsAccess, self)._abs(path)
         return destination
 
+    def size(self, path: str) -> int:
+        if path.startswith("toilfs:"):
+            return self.file_store.getGlobalFileSize(FileID.unpack(path[7:]))
+        else:
+            return super().size(path)
+
 
 def toil_get_file(
     file_store: AbstractFileStore, index: dict, existing: dict, file_store_id: str


Test patch: