Repository: DataBiosphere/toil

Issue:
AWS token expiry due to (presumable) leader thread contention
The temporary IAM credentials used to download the hot-deployed user script can expire if the leader's _refreshUserScript thread doesn't get scheduled within a frame of 2 minutes. This will cause jobs to fail basically as fast as the leader can schedule them:

https://joel.armstro.ng/graftoil/dashboard/snapshot/FZqfN5f18jJPGO4FU1NzuW0LnupObmMe

(the failed jobs are marked as non-preemptable after failure because of a temporary modification I made.)

Thankfully this is pretty much self-correcting once the refresh thread does get scheduled, though every task prior to that needs to run, fail and be rescheduled. But do we really need that refresh thread at all? I'm not too surprised that it missed a 2 minute window, python threads [aren't scheduled in an even remotely sane manner](http://www.dabeaz.com/python/UnderstandingGIL.pdf).

Sorry if this is a stupid question--I don't know much about AWS. But couldn't we use [AWS Signature v4 and presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) to sidestep this whole credential refresh issue? It seems like boto2 has supported this since 2014. I'm eager to get rid of any unneeded threads.
AWS token expiry due to (presumable) leader thread contention
The temporary IAM credentials used to download the hot-deployed user script can expire if the leader's _refreshUserScript thread doesn't get scheduled within a frame of 2 minutes. This will cause jobs to fail basically as fast as the leader can schedule them:

https://joel.armstro.ng/graftoil/dashboard/snapshot/FZqfN5f18jJPGO4FU1NzuW0LnupObmMe

(the failed jobs are marked as non-preemptable after failure because of a temporary modification I made.)

Thankfully this is pretty much self-correcting once the refresh thread does get scheduled, though every task prior to that needs to run, fail and be rescheduled. But do we really need that refresh thread at all? I'm not too surprised that it missed a 2 minute window, python threads [aren't scheduled in an even remotely sane manner](http://www.dabeaz.com/python/UnderstandingGIL.pdf).

Sorry if this is a stupid question--I don't know much about AWS. But couldn't we use [AWS Signature v4 and presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) to sidestep this whole credential refresh issue? It seems like boto2 has supported this since 2014. I'm eager to get rid of any unneeded threads.
Toil will attempt to refresh user script after it's been deleted from the job store
I think this failure is only cosmetic (i.e. it can't possibly affect the actual execution of the workflow), but rarely, the user script will be refreshed *during* deletion of the job store. If the user script file has already been deleted, the thread fails:

```
Attempting to delete the job store
Binding to job store domain 'toil-registry'.
Refreshing user script resource Resource(name='site-packages', pathHash='c3e8ab34e898b8be2df76ef059db6a0a', url=u'https://cactus-jcarmstr-nematodes-preempt--files.s3-us-west-2.amazonaws.com/75a44249-39c1-5221-a08d-382368802ee6?Signature=g9vqYJmMQZOdi%2B43TQWACUussqY%3D&Expires=1519242938&AWSAccessKeyId=ASIAJKAZFMBJB27B77RA&versionId=4S8J.1ZRklPag0DA2p1IdambkZWjEO0t&x-amz-security-token=FQoDYXdzEKz//////////wEaDBy0AKmvzG7xxZHj/CK3AyKtTLc5hMGbYFr/ULsYXQokxooL4xhPMu7EdYUM7kYv81Fwjjpm1wuVI8H8AFYnODCzN/iUwK/Lpm06l1/LsJO5L0G8y8ckEPpkTpgJzWqeMhEoQbcUyz0KbjYjkzyepqkdJEU9O7L6L8tS0UIiaHB2mjQSHUmY5CeRdXumANqYH95cyrlFFqSKleUzo2ihXGDhmwzDVE2VdF66BVCn19l5pwO7DHbCEZ5oukYzBT6R/N6WmV2Z/TFAhqkLRvfol/jRL6uMtYMGMXUykgJ3gOok7MejHs3fENbldXBskfYq3U9YcTPhUun3ptQyM22RfYRl9NYnTSECLTM2/qY5b%2BLoauudZHk78RuhwfTerMUZMaFk9q9BitRTdJ91h%2B5q6RqabMT/O4Mkexl%2B9UHgFOEhFsYdMlDegRCe5sRBbJOf2APVmsFrxMh5qEl%2BMtO13AZc0hynLsQNsm//smbS8AAh8EUtSKUs2nADLRUtF1LD%2Bg2fGoksI5w9eNJgHtR/lb3WwoyTrYTbs3EUOC5UASLdDAQZy8k9/tVoi16m74oJJ2Dc7Z99joRTGMWLC/pY9IfPeh0ydAUoo6OyxQU%3D', contentHash='57d8fd7557107e69bfc8f5e7b2e05774').
Exception in thread refreshUserScript:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 894, in _refreshUserScript
    userScriptResource = userScriptResource.refresh(self._jobStore)
  File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 96, in refresh
    url=jobStore.getSharedPublicUrl(sharedFileName=self.pathHash),
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 620, in getSharedPublicUrl
    return self.getPublicUrl(self._sharedFileID(sharedFileName))
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 616, in getPublicUrl
    return key.generate_url(expires_in=self.publicUrlExpiration.total_seconds())
AttributeError: 'NoneType' object has no attribute 'generate_url'

Binding to job store domain 'toil-registry'.
Successfully deleted the job store
```

Not sure if this is worth fixing, because it doesn't seem to actually cause a problem, but I think the solution would be to hang onto the thread, spawned here:

https://github.com/BD2KGenomics/toil/blob/1c74a23a5e8237f763f70e8c675e957080622501/src/toil/common.py#L870

and shut it down before job store deletion.


Discussion:
Just realized the credential refresh is because the *instance* has temporary credentials. That sucks. Still, maybe we could set the ACL to public on just the user script, or at least use a separate process rather than thread to refresh the credentials.
Just realized the credential refresh is because the *instance* has temporary credentials. That sucks. Still, maybe we could set the ACL to public on just the user script, or at least use a separate process rather than thread to refresh the credentials.


Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -874,33 +874,6 @@ def _setupHotDeployment(self, userScript=None):
             userScriptResource = userScript.saveAsResourceTo(self._jobStore)
             logger.debug('Injecting user script %s into batch system.', userScriptResource)
             self._batchSystem.setUserScript(userScriptResource)
-            thread = Thread(target=self._refreshUserScript,
-                            name='refreshUserScript',
-                            kwargs=dict(userScriptResource=userScriptResource))
-            thread.daemon = True
-            thread.start()
-
-    def _refreshUserScript(self, userScriptResource):
-        """
-        Periodically refresh the user script in the job store to prevent credential
-        expiration from causing the public URL to the user script to expire.
-        """
-        while True:
-            # Boto refreshes IAM credentials if they will be expiring within the next five
-            # minutes, but it will only check the expiry if and when credentials are needed to
-            # sign an actual AWS request. This means that we should be refreshing the user script
-            # at least every 5 minutes. Note that refreshing the user script in the job store
-            # involves an S3 request requiring credentials and therefore also triggers refreshing
-            # the IAM role credentials. In the worst case, refresh() is called 5 minutes plus
-            # epsilon before IAM credential expiration. The resource is refreshed three minutes
-            # after that, leaving two minutes plus epsilon generating a new signed URL, this time
-            # with refreshed IAM role credentials. This consideration only applies to AWS and
-            # Boto2, of course. See https://github.com/BD2KGenomics/toil/issues/1372.
-            time.sleep(3 * 60)
-            logger.debug('Refreshing user script resource %s.', userScriptResource)
-            userScriptResource = userScriptResource.refresh(self._jobStore)
-            logger.debug('Injecting refreshed user script %s into batch system.', userScriptResource)
-            self._batchSystem.setUserScript(userScriptResource)
 
     def importFile(self, srcUrl, sharedFileName=None):
         self._assertContextManagerUsed()
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -24,6 +24,8 @@
 import base64
 import hashlib
 import itertools
+import urlparse
+import urllib
 
 # Python 3 compatibility imports
 from six.moves import xrange, cPickle, StringIO, reprlib
@@ -32,15 +34,9 @@
 from bd2k.util import strict_bool
 from bd2k.util.exceptions import panic
 from bd2k.util.objects import InnerClass
-from boto.sdb.domain import Domain
-from boto.s3.bucket import Bucket
-from boto.s3.connection import S3Connection
-from boto.sdb.connection import SDBConnection
-from boto.sdb.item import Item
 import boto.s3
 import boto.sdb
 from boto.exception import S3CreateError
-from boto.s3.key import Key
 from boto.exception import SDBResponseError, S3ResponseError
 from concurrent.futures import ThreadPoolExecutor
 
@@ -621,7 +617,20 @@ def getPublicUrl(self, jobStoreFileID):
         for attempt in retry_s3():
             with attempt:
                 key = self.filesBucket.get_key(key_name=jobStoreFileID, version_id=info.version)
-                return key.generate_url(expires_in=self.publicUrlExpiration.total_seconds())
+                key.set_canned_acl('public-read')
+                url = key.generate_url(query_auth=False,
+                                       expires_in=self.publicUrlExpiration.total_seconds())
+                # boto doesn't properly remove the x-amz-security-token parameter when
+                # query_auth is False when using an IAM role (see issue #2043). Including the
+                # x-amz-security-token parameter without the access key results in a 403,
+                # even if the resource is public, so we need to remove it.
+                scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
+                params = urlparse.parse_qs(query)
+                if 'x-amz-security-token' in params:
+                    del params['x-amz-security-token']
+                query = urllib.urlencode(params, doseq=True)
+                url = urlparse.urlunsplit((scheme, netloc, path, query, fragment))
+                return url
 
     def getSharedPublicUrl(self, sharedFileName):
         assert self._validateSharedFileName(sharedFileName)


Test patch: