Repository: DataBiosphere/toil

Issue:
testAutoScaledCluster sometimes is throttled to death
Example of the error trace taken from http://jenkins.cgcloud.info/job/toil-pull-request-integration-tests/1360/testReport/junit/src.toil.test.provisioners.aws.awsProvisionerTest/AWSRestartTest/testAutoScaledCluster/
```
INFO:toil.test:Setting up toil.test.provisioners.aws.awsProvisionerTest.AWSRestartTest.testAutoScaledCluster ...
INFO:toil.lib.bioio:Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
INFO:toil.utils.toilLaunchCluster:Using aws provisioner.
Traceback (most recent call last):
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/venv/bin/toil", line 9, in <module>
    load_entry_point('toil', 'console_scripts', 'toil')()
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/src/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/src/toil/utils/toilLaunchCluster.py", line 93, in main
    vpcSubnet=config.vpcSubnet)
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/src/toil/provisioners/aws/awsProvisioner.py", line 114, in launchCluster
    profileARN = self._getProfileARN(ctx)
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/src/toil/provisioners/aws/awsProvisioner.py", line 792, in _getProfileARN
    profile = ctx.iam.create_instance_profile(iamRoleName)
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/venv/local/lib/python2.7/site-packages/boto/iam/connection.py", line 1103, in create_instance_profile
    return self.get_response('CreateInstanceProfile', params)
  File "/mnt/ephemeral/workspace/toil-pull-request-integration-tests/venv/local/lib/python2.7/site-packages/boto/iam/connection.py", line 102, in get_response
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>6e497298-7924-11e7-86b9-d7d1cdd81d37</RequestId>
</ErrorResponse>
```

https://forums.aws.amazon.com/thread.jspa?messageID=406788&#406788 says that the recommended response is a truncated exponential back off.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-829)
┆Issue Number: TOIL-829



Discussion:
Someone with access to the Jenkins account should contact AWS support (select the phone option), and ask if we can get our IAM rate limit raised, or even what exactly our limit is.

Slightly longer-term, we should add a retry to every call to `setup_iam_ec2_role`. And possibly de-parallelize the jenkins-it tests a bit, or just stagger their start by a minute or so. Unfortunately that won't help all the PRs currently open, unless they merge in that fix.
@joelarmstrong I was planning on just doing a retry. I can have that out pretty soon. @ejacox asked me to contact AWS support; what do you mean exactly by "access to the Jenkins account"?
I mean, anyone who can actually log in to the jenkins aws account (don't know the account name). The easiest way to contact support is to log in to the console and contact support from there.
Also it looks like the throttling may happen on any of the boto calls in the step (in #1769 it was on `setup_iam_ec2_role`, in #1805 it was `create_instance_profile`) so my plan was to just put a retry inside the whole function.
ah, didn't see that. yeah, that is a much better idea.
It looks like we have been accumulating IAM roles as well...
> Increased IAM Error Rates
> We have identified the root cause and taken steps to resolve the issue. Error rates and latencies have improved and we continue to work towards full recovery.
>  
> Please see our Service Health Dashboard for more information.

(copied and pasted from the support page)

Looks like this is an Amazon issue. Still, the retry wouldn't hurt

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -781,34 +781,50 @@ def groupNotFound(e):
     def _getProfileARN(cls, ctx):
         def addRoleErrors(e):
             return e.status == 404
-        roleName = 'toil'
-        policy = dict(iam_full=iamFullPolicy, ec2_full=ec2FullPolicy,
-                      s3_full=s3FullPolicy, sbd_full=sdbFullPolicy)
-        iamRoleName = ctx.setup_iam_ec2_role(role_name=roleName, policies=policy)
 
-        try:
-            profile = ctx.iam.get_instance_profile(iamRoleName)
-        except BotoServerError as e:
-            if e.status == 404:
-                profile = ctx.iam.create_instance_profile(iamRoleName)
-                profile = profile.create_instance_profile_response.create_instance_profile_result
-            else:
-                raise
-        else:
-            profile = profile.get_instance_profile_response.get_instance_profile_result
-        profile = profile.instance_profile
-        profile_arn = profile.arn
-
-        if len(profile.roles) > 1:
-                raise RuntimeError('Did not expect profile to contain more than one role')
-        elif len(profile.roles) == 1:
-            # this should be profile.roles[0].role_name
-            if profile.roles.member.role_name == iamRoleName:
-                return profile_arn
-            else:
-                ctx.iam.remove_role_from_instance_profile(iamRoleName,
-                                                          profile.roles.member.role_name)
-        for attempt in retry(predicate=addRoleErrors):
+        def throttleError(e):
+            return isinstance(e, BotoServerError) and e.status == 400 and e.error_code == 'Throttling'
+
+        def truncExpBackoff():
+            # as recommended here https://forums.aws.amazon.com/thread.jspa?messageID=406788#406788
+            yield 0
+            t = 1
+            while t < 1024:
+                yield t
+                t *= 2
+            while True:
+                yield t
+
+        for attempt in retry(delays=truncExpBackoff(), predicate=throttleError):
             with attempt:
-                ctx.iam.add_role_to_instance_profile(iamRoleName, iamRoleName)
+                roleName = 'toil'
+                policy = dict(iam_full=iamFullPolicy, ec2_full=ec2FullPolicy,
+                              s3_full=s3FullPolicy, sbd_full=sdbFullPolicy)
+                iamRoleName = ctx.setup_iam_ec2_role(role_name=roleName, policies=policy)
+
+                try:
+                    profile = ctx.iam.get_instance_profile(iamRoleName)
+                except BotoServerError as e:
+                    if e.status == 404:
+                        profile = ctx.iam.create_instance_profile(iamRoleName)
+                        profile = profile.create_instance_profile_response.create_instance_profile_result
+                    else:
+                        raise
+                else:
+                    profile = profile.get_instance_profile_response.get_instance_profile_result
+                profile = profile.instance_profile
+                profile_arn = profile.arn
+
+                if len(profile.roles) > 1:
+                        raise RuntimeError('Did not expect profile to contain more than one role')
+                elif len(profile.roles) == 1:
+                    # this should be profile.roles[0].role_name
+                    if profile.roles.member.role_name == iamRoleName:
+                        return profile_arn
+                    else:
+                        ctx.iam.remove_role_from_instance_profile(iamRoleName,
+                                                                  profile.roles.member.role_name)
+                for attempt in retry(predicate=addRoleErrors):
+                    with attempt:
+                        ctx.iam.add_role_to_instance_profile(iamRoleName, iamRoleName)
         return profile_arn


Test patch: