Repository: DataBiosphere/toil

Issue:
Upload final output files to AWS S3/Azure Blob/Google Cloud Storage
Currently (and please correct me if I'm wrong) Toil will spit out its
output files at the head node where you run a workflow. This is fine for
many cases, but sometimes it's more convenient to have all of the output
files in cloud storage.

One way of facilitating this is to add a runtime argument which lets us
specify an output file endpoint.

An implementation of this, written by @Irrationone, which works for
(only) CWL and Azure can be seen here:

https://github.com/BD2KGenomics/toil/compare/master...Irrationone:304682394b5687a88f1f48415e2e462138e83e76

This changeset makes a new object, a "final job store", which is another
job store which essentially represents the final storage destination for
the output files.

The downside of this is that it conflates a final storage destination
with a job store, which may not be desired; and that the final job store
contains much more functionality than is necessary for its purpose.

The upside of making the final storage destination another job store is
that it only requires adding in minimal code.

Anyway, any discussion on how to implement will be much appreciated.

┆Issue is synchronized with this [JIRA Story](https://ucsc-cgl.atlassian.net/browse/TOIL-164)
┆Issue Number: TOIL-164



Discussion:
`fileStore.exportFile(p.resolved[7:], destUrl)` could place the files at any `destUrl`.  I think this could be a useful option for cwl (users can use the `fileStore.exportFile()` in their script directly with native python), though I think the exporting to any generic user-specified bucket rather than a jobstore would be preferable (especially if wanting to export to a bucket that is already being used for other things).

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -357,7 +357,7 @@ def run(self, fileStore):
         return resolve_indirect(self.cwljob)
 
 
-def toilStageFiles(fileStore, cwljob, outdir, index, existing, export):
+def toilStageFiles(fileStore, cwljob, outdir, index, existing, export, destBucket=None):
     """Copy input files out of the global file store and update location and
     path."""
 
@@ -367,6 +367,21 @@ def toilStageFiles(fileStore, cwljob, outdir, index, existing, export):
     for f, p in pm.items():
         if not p.staged:
             continue
+
+        # Deal with bucket exports
+        if destBucket:
+            # Directories don't need to be created if we're exporting to
+            # a bucket
+            if p.type == "File":
+                # Remove the staging directory from the filepath and
+                # form the destination URL
+                unstageTargetPath = p.target[len(outdir):]
+                destUrl = '/'.join(s.strip('/') for s in [destBucket, unstageTargetPath])
+
+                fileStore.exportFile(p.resolved[7:], destUrl)
+
+            continue
+
         if not os.path.exists(os.path.dirname(p.target)):
             os.makedirs(os.path.dirname(p.target), 0o0755)
         if p.type == "File":
@@ -915,6 +930,8 @@ def main(args=None, stdout=sys.stdout):
                     metavar=("VAR1 VAR2"),
                     default=("PATH",),
                     dest="preserve_environment")
+    parser.add_argument("--destBucket", type=str,
+                        help="Specify a cloud bucket endpoint for output files.")
     # help="Dependency resolver configuration file describing how to adapt 'SoftwareRequirement' packages to current system."
     parser.add_argument("--beta-dependency-resolvers-configuration", default=None)
     # help="Defaut root directory used by dependency resolvers configuration."
@@ -1049,9 +1066,20 @@ def setSecondary(fileobj):
 
         outobj = resolve_indirect(outobj)
 
-        toilStageFiles(toil, outobj, outdir, fileindex, existing, True)
-
-        visit_class(outobj, ("File",), functools.partial(compute_checksums, cwltool.stdfsaccess.StdFsAccess("")))
+        # Stage files. Specify destination bucket if specified in CLI
+        # options. If destination bucket not passed in,
+        # options.destBucket's value will be None.
+        toilStageFiles(
+            toil,
+            outobj,
+            outdir,
+            fileindex,
+            existing,
+            export=True,
+            destBucket=options.destBucket)
+
+        if not options.destBucket:
+            visit_class(outobj, ("File",), functools.partial(compute_checksums, cwltool.stdfsaccess.StdFsAccess("")))
 
         stdout.write(json.dumps(outobj, indent=4))
 


Test patch: