Question: Hi,

Have a CWL defined that needs to return a set of files (all filename extensions) so I have defined my outputs as:

outputs:
mapped_out:
type: File
outputBinding:
glob: $(inputs.sample).bam
secondaryFiles:
- .bai
- .bas
- .md5
- .met
- .maptime

I've tried a couple of variations of the json:

{
...
"mapped_out": {
"path": "/tmp/mapped.bam",
"class": "File"
},
...
}

Yeilded one file provisioned to /tmp/mapped.bam

This version (based on [alea-createGenome.cwl][1] & [alea-alignReads-job.json][2]) didn't stage anything:

{
...
"mapped_out": "/tmp/mapped",
...
}

Everything seems to have compelted in the cwltool side:

Final process status is success
{
"mapped_out": {
"checksum": "sha1$53bb0c4abb07013393891cb50a3feec4c6381304",
"basename": "insilico_21.bam",
"location": "file:///home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam",
"path": "/home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam",
"secondaryFiles": [
{
"checksum": "sha1$ef6f2cf70e11d7d0be17b79dfb02eb1277e43b41",
"basename": "insilico_21.bam.bai",
"location": "file:///home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.bai",
"path": "/home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.bai",
"class": "File",
"size": 1370120
},
{
"checksum": "sha1$4bf5068040c0e2a350aa21fa299f6567230bfbeb",
"basename": "insilico_21.bam.bas",
"location": "file:///home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.bas",
"path": "/home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.bas",
"class": "File",
"size": 1973
},
{
"checksum": "sha1$4a60424144f5283c4e9cf74deb214597cac8bae8",
"basename": "insilico_21.bam.md5",
"location": "file:///home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.md5",
"path": "/home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.md5",
"class": "File",
"size": 32
},
{
"checksum": "sha1$63139bed16686c6be0dd5469342af1dac8795260",
"basename": "insilico_21.bam.met",
"location": "file:///home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.met",
"path": "/home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.met",
"class": "File",
"size": 1521
},
{
"checksum": "sha1$39f641f432b510034fb96b3e73569f5fc1824521",
"basename": "insilico_21.bam.maptime",
"location": "file:///home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.maptime",
"path": "/home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.maptime",
"class": "File",
"size": 279
}
],
"class": "File",
"size": 42245405
}
}

Any help gratefully recieved.

Thanks,
Keiran

[1]: https://github.com/common-workflow-language/workflows/blob/master/tools/alea-createGenome.cwl
[2]: https://github.com/common-workflow-language/workflows/blob/master/test/alea-alignReads-job.json

Answer: Ah, I think I understand the confusion here. Apologies since I think we created it.

1) Dockstore input JSON can (optionally) include output parameters in order to provision files to locations like S3, icgc-storage, ftp. This is an artifact of Dockstore's beginnings in the pan-cancer project where we always wrote workflows that look like "download from GNOS/S3 -> do processing -> upload to GNOS/S3"

In other words, you should be able to do this to upload bamstats_report, an output to s3:

$ cat sample_configs.json
{
"bam_input": {
"class": "File",
"path": "https://s3.amazonaws.com/oconnor-test-bucket/sample-data/NA12878.chrom20.ILLUMINA.bwa.CEU.low_coverage.20121211.bam"
},
"bamstats_report": {
"class": "File",
"path": "s3://oicr.temp/bamstats.zip"
}
}
dockstore tool launch --entry quay.io/collaboratory/dockstore-tool-bamstats:1.25-6_1.0 --json sample_configs.json

And you should be able to do this to just leave the results in place on your local host

$ cat sample_configs2.json
{
"bam_input": {
"class": "File",
"path": "https://s3.amazonaws.com/oconnor-test-bucket/sample-data/NA12878.chrom20.ILLUMINA.bwa.CEU.low_coverage.20121211.bam"
}
}
$ dockstore tool launch --entry quay.io/collaboratory/dockstore-tool-bamstats:1.25-6_1.0 --json sample_configs2.json

This is a red herring though.

2) It looks like Dockstore has a bug/missing feature where we probably missed that output parameters (in the CWL) can also specify secondary files. While the secondary files look like they're being generated properly coming out of cwltool (in /home/ubuntu/./datastore/launcher-ccd381b4-c475-4770-b88b-bebd2b06439c/outputs/insilico_21.bam.*) , they aren't being moved further along to /tmp/mapped.* as we would have expected.

We're adding this as an issue https://github.com/ga4gh/dockstore/issues/544