Repository: DataBiosphere/toil

Issue:
Cache temporary IAM credentials obtained from instance metadata
The patch needs to be injected in the leader and the worker. The latter would activate it for user scripts. S3AM should get it, too. So naturally, this should go into bd2k-python-lib.


Discussion:
https://github.com/BD2KGenomics/s3am/issues/23

@cket and @jvivian this is the metadata caching hack I was referring to.

So we are starting to see the issue in the wild now:

```
HTTPError: HTTP Error 429: Too Many Requests
ERROR:boto:Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/boto/utils.py", line 210, in retry_url
    r = opener.open(req, timeout=timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
```

Occurs the Mesos sandbox as the primary exception. There are also secondary backtraces but they are harder to read because multiple processes write to the same log file. A single backtrace must have these lines in it (arbitrary order):

```
      File "/usr/local/bin/_toil_worker", line 11, in <module>
      File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1100, in __init__
      File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 555, in __init__
      File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 569, in __init__
      File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 200, in __init__
      File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 376, in get_credentials
      File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 395, in _populate_keys_from_metadata_server
      File "/usr/local/lib/python2.7/dist-packages/boto/regioninfo.py", line 187, in connect
      File "/usr/local/lib/python2.7/dist-packages/boto/s3/__init__.py", line 43, in connect
      File "/usr/local/lib/python2.7/dist-packages/boto/s3/__init__.py", line 73, in connect_to_region
      File "/usr/local/lib/python2.7/dist-packages/boto/sdb/connection.py", line 122, in __init__
      File "/usr/local/lib/python2.7/dist-packages/boto/sdb/__init__.py", line 54, in connect_to_region
      File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 612, in getJobStore
      File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 628, in resumeJobStore
      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 197, in __init__
      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 620, in _connectSimpleDB
      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 632, in _connectS3
      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 92, in main
```

The current workaround is to bump the `--defaultCores` option to limit the number of worker running concurrently on any given node.
Use https://github.com/BD2KGenomics/bd2k-python-lib/blob/master/src/bd2k/util/ec2/credentials.py#L45

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -674,7 +674,9 @@ def _setProvisioner(self):
             self._provisioner = CGCloudProvisioner(self.config, self._batchSystem)
         elif self.config.provisioner == 'aws':
             logger.info('Using AWS provisioner.')
+            from bd2k.util.ec2.credentials import enable_metadata_credential_caching
             from toil.provisioners.aws.awsProvisioner import AWSProvisioner
+            enable_metadata_credential_caching()
             self._provisioner = AWSProvisioner(self.config, self._batchSystem)
         else:
             # Command line parser shold have checked argument validity already
@@ -695,7 +697,9 @@ def getJobStore(cls, locator):
             from toil.jobStores.fileJobStore import FileJobStore
             return FileJobStore(rest)
         elif name == 'aws':
+            from bd2k.util.ec2.credentials import enable_metadata_credential_caching
             from toil.jobStores.aws.jobStore import AWSJobStore
+            enable_metadata_credential_caching()
             return AWSJobStore(rest)
         elif name == 'azure':
             from toil.jobStores.azureJobStore import AzureJobStore
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -76,6 +76,14 @@ def main():
     from toil.lib.bioio import getTotalCpuTime
     from toil.lib.bioio import getTotalCpuTimeAndMemoryUsage
     from toil.job import Job
+    try:
+        import boto
+    except ImportError:
+        pass
+    else:
+        # boto is installed, monkey patch it now
+        from bd2k.util.ec2.credentials import enable_metadata_credential_caching
+        enable_metadata_credential_caching()
     ##########################################
     #Input args
     ##########################################


Test patch: