Repository: DataBiosphere/toil

Issue:
Running CWL workflows with `--destBucket` produces output with `location`s that don't point there.
I got the Toil WES server to run this command:

```
toil-cwl-runner \
--batchSystem=aws_batch \
--awsBatchQueue=arn:aws:batch:us-west-2:318423852362:job-queue/BatchTaskBatchJobQueue15-rh2BbLlNWpzhCZsn \
--awsBatchRegion=us-west-2 \
--destBucket=s3://agc-318423852362-us-west-2/project/Demo/userid/anovak4rBHiA/context/myContext/toil-execution/38c6ccafd6d942daa42ef5dd2c80b9a9 \
--clean=always \
--jobStore=aws:us-west-2:toil-3d983bb1-1935-4bdf-8c4c-cc96d27d2497 \
/opt/work/workflows/38c6ccafd6d942daa42ef5dd2c80b9a9/execution/nontrivial.cwl \
/opt/work/workflows/38c6ccafd6d942daa42ef5dd2c80b9a9/execution/wes_inputs.json
```

I used AGC and the "nontrivial" CWL example workflow from commit 4033ad7ae96e167857501ff6d1b3d857f002765b.

But I got an output that looks like this:

```
{
    "output": {
        "location": "file:///opt/work/workflows/38c6ccafd6d942daa42ef5dd2c80b9a9/execution/compressed.gz",
        "basename": "compressed.gz",
        "nameroot": "compressed",
        "nameext": ".gz",
        "class": "File",
        "checksum": "sha1$8cc374b6fcd2e737207f35d7b331c00369a0def1",
        "size": 274771
    }
}
```

I think this is because, when we stage the output files into the destination bucket, we set their locations without reference to the `destBucket` parameter:

https://github.com/DataBiosphere/toil/blob/97064368d6049d5e1eb4cacbc7c6dce3487ced2d/src/toil/cwl/cwltoil.py#L1792-L1794

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1153)
┆friendlyId: TOIL-1153

Running CWL workflows with `--destBucket` produces output with `location`s that don't point there.
I got the Toil WES server to run this command:

```
toil-cwl-runner \
--batchSystem=aws_batch \
--awsBatchQueue=arn:aws:batch:us-west-2:318423852362:job-queue/BatchTaskBatchJobQueue15-rh2BbLlNWpzhCZsn \
--awsBatchRegion=us-west-2 \
--destBucket=s3://agc-318423852362-us-west-2/project/Demo/userid/anovak4rBHiA/context/myContext/toil-execution/38c6ccafd6d942daa42ef5dd2c80b9a9 \
--clean=always \
--jobStore=aws:us-west-2:toil-3d983bb1-1935-4bdf-8c4c-cc96d27d2497 \
/opt/work/workflows/38c6ccafd6d942daa42ef5dd2c80b9a9/execution/nontrivial.cwl \
/opt/work/workflows/38c6ccafd6d942daa42ef5dd2c80b9a9/execution/wes_inputs.json
```

I used AGC and the "nontrivial" CWL example workflow from commit 4033ad7ae96e167857501ff6d1b3d857f002765b.

But I got an output that looks like this:

```
{
    "output": {
        "location": "file:///opt/work/workflows/38c6ccafd6d942daa42ef5dd2c80b9a9/execution/compressed.gz",
        "basename": "compressed.gz",
        "nameroot": "compressed",
        "nameext": ".gz",
        "class": "File",
        "checksum": "sha1$8cc374b6fcd2e737207f35d7b331c00369a0def1",
        "size": 274771
    }
}
```

I think this is because, when we stage the output files into the destination bucket, we set their locations without reference to the `destBucket` parameter:

https://github.com/DataBiosphere/toil/blob/97064368d6049d5e1eb4cacbc7c6dce3487ced2d/src/toil/cwl/cwltoil.py#L1792-L1794

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1153)
┆friendlyId: TOIL-1153



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1669,6 +1669,9 @@ def toilStageFiles(
 ) -> None:
     """
     Copy input files out of the global file store and update location and path.
+
+    :param destBucket: If set, export to this base URL instead of to the local
+           filesystem.
     """
 
     def _collectDirEntries(
@@ -1789,9 +1792,21 @@ def _realpath(
                         n.write(p.resolved.encode("utf-8"))
 
     def _check_adjust(f: CWLObjectType) -> CWLObjectType:
-        f["location"] = schema_salad.ref_resolver.file_uri(
-            pm.mapper(cast(str, f["location"]))[1]
-        )
+        # Figure out where the path mapper put this
+        mapped_location: MapperEnt = pm.mapper(cast(str, f["location"]))
+        if destBucket:
+            # Make the location point to the destination bucket.
+            # Reconstruct the path we exported to.
+            base_name = mapped_location.target[len(outdir) :]
+            dest_url = "/".join(s.strip("/") for s in [destBucket, base_name])
+            # And apply it
+            f["location"] = dest_url
+        else:
+            # Make the location point to the place we put this thing on the
+            # local filesystem.
+            f["location"] = schema_salad.ref_resolver.file_uri(
+                mapped_location.target
+            )
 
         if "contents" in f:
             del f["contents"]
diff --git a/src/toil/lib/aws/__init__.py b/src/toil/lib/aws/__init__.py
--- a/src/toil/lib/aws/__init__.py
+++ b/src/toil/lib/aws/__init__.py
@@ -50,25 +50,37 @@ def get_aws_zone_from_metadata() -> Optional[str]:
     Get the AWS zone from instance metadata, if on EC2 and the boto module is
     available. Otherwise, gets the AWS zone from ECS task metadata, if on ECS.
     """
-    if running_on_ec2():
-        try:
-            # Use the EC2 metadata service
-            import boto
-            from boto.utils import get_instance_metadata
-            return get_instance_metadata()['placement']['availability-zone']
-        except (KeyError, ImportError):
-            pass
-    elif running_on_ecs():
+
+    # When running on ECS, we also appear to be running on EC2, but the EC2
+    # metadata service doesn't seem to be contactable. So we check ECS first.
+
+    if running_on_ecs():
         # Use the ECS metadata service
         logger.debug("Fetch AZ from ECS metadata")
         try:
             resp = json.load(urlopen(os.environ['ECS_CONTAINER_METADATA_URI_V4'] + '/task', timeout=1))
             logger.debug("ECS metadata: %s", resp)
             if isinstance(resp, dict):
+                # We found something. Go with that.
                 return resp.get('AvailabilityZone')
         except (json.decoder.JSONDecodeError, KeyError, URLError) as e:
-            logger.debug("Skipping ECS metadata due to error: %s", e)
-            pass
+            # We're on ECS but can't get the metadata. That's odd.
+            logger.warning("Skipping ECS metadata due to error: %s", e)
+    if running_on_ec2():
+        # On EC2 alone, or on ECS but we couldn't get ahold of the ECS
+        # metadata.
+        try:
+            # Use the EC2 metadata service
+            import boto
+            from boto.utils import get_instance_metadata
+            logger.debug("Fetch AZ from EC2 metadata")
+            return get_instance_metadata()['placement']['availability-zone']
+        except ImportError:
+            # This is expected to happen a lot
+            logger.debug("No boto to fetch ECS metadata")
+        except (KeyError, URLError) as e:
+            # We're on EC2 but can't get the metadata. That's odd.
+            logger.warning("Skipping EC2 metadata due to error: %s", e)
     return None
 
 def get_aws_zone_from_boto() -> Optional[str]:


Test patch: