Repository: DataBiosphere/toil

Issue:
Resource Download throw: HTTPError: HTTP Error 403: Forbidden



Discussion:
During a run I got 2 failures from toil. 

```
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:       Traceback (most recent call last):
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 237, in main
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           jobStore=jobStore)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 851, in _execute
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           returnValues = self.run(fileStore)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 976, in run
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           userFunction = self._getUserFunction()
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 943, in _getUserFunction
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           userFunctionModule = userFunctionModule.localize()
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 358, in localize
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           resource.download( callback=stash )
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 138, in download
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           self._save( tempDirPath )
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 227, in _save
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           self._download( localFile )
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 209, in _download
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           with closing( urlopen( self.url ) ) as content:
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           return _opener.open(url, data, timeout)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/lib/python2.7/urllib2.py", line 410, in open
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           response = meth(req, response)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           'http', request, response, code, msg, hdrs)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/lib/python2.7/urllib2.py", line 448, in error
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           return self._call_chain(*args)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           result = func(*args)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:         File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:           raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
WARNING:toil.leader:bbdd4a75-77b3-4d89-b55f-61a535e9a314:       HTTPError: HTTP Error 403: Forbidden
```

I'd bet money (but not a lot) that this is due to clock skew relative to AWS.

edit: or an issue where the module url is saved for longer than 60m. Was the batch running for more than an hour?

I think you were right, this is time related but I don't think necessarily due to clock skew. Currently urls generated during the run expire in 1 hour, which is probably too low. I'll up that expiration limit

Maybe they shouldn't expire at all, unless there is careful renewal of the expiry times. It would suck to have a long-running batch break because the urls for some resources expired. I'll be running batches that take weeks.

Unfortunately AWS requires that they expire. I think the next best thing is setting the limit to something like 20 years, which is what I am planning on


Resolving patch:
diff --git a/src/toil/jobStores/awsJobStore.py b/src/toil/jobStores/awsJobStore.py
--- a/src/toil/jobStores/awsJobStore.py
+++ b/src/toil/jobStores/awsJobStore.py
@@ -161,7 +161,9 @@ def getPublicUrl(self, jobStoreFileID):
         Encrypted file urls are currently not supported
         """
         key = self.files.get_key(key_name=jobStoreFileID)
-        return key.generate_url(expires_in=3600)  # one hour
+        # There should be no practical upper limit on when a job is allowed to access a public
+        # URL so we set the expiration to 20 years.
+        return key.generate_url(expires_in=60 * 60 * 24 * 365 * 20)
 
     def getSharedPublicUrl(self, FileName):
         jobStoreFileID = self._newFileID(FileName)


Test patch: