Repository: DataBiosphere/toil

Issue:
aggressive AWS autoscaler polling causes 503 response and pipeline failure
The autoscaler seems to be polling EC2 too often and causing API limits to be exceeded:

```
BotoServerError: 503 Service Unavailable
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>RequestLimitExceeded</Code><Message>Request limit exceeded.</Message></Error></Errors><RequestID>8648be89-3344-4b3c-af07-74c1cd014a81</RequestID></Response>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 248, in check
    scalerThread.join(timeout=0)
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 406, in tryRun
    preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/abstractProvisioner.py", line 156, in setNodeCount
    workerInstances = self._getWorkersInCluster(preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 555, in _getWorkersInCluster
    entireCluster = self._getNodesInCluster(both=True)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 552, in _getNodesInCluster
    return self.__getNodesInCluster(self.ctx, self.clusterName, both=both)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 537, in __getNodesInCluster
    'instance-state-name': 'pending'})
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 622, in get_only_instances
    next_token=next_token)
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 681, in get_all_reservations
    [('item', Reservation)], verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1170, in get_list
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1116, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1028, in _mexe
    raise BotoServerError(response.status, response.reason, body)
BotoServerError: BotoServerError: 503 Service Unavailable
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>RequestLimitExceeded</Code><Message>Request limit exceeded.</Message></Error></Errors><RequestID>8648be89-3344-4b3c-af07-74c1cd014a81</RequestID></Response>
```

Retrying this request might be a bit redundant since boto is probably retrying automatically. The autoscaler thread dies on this error, causing total failure of the pipeline--instead I think it should always catch the exception, back off for a minute or two, then retry the current autoscaler loop. (OK, this behaves almost identically to a retry, but the pipeline should never fail just because the autoscaler is getting throttled.)

Frustratingly, AWS documentation doesn't say *what* exactly the limits are. But maybe increasing the polling interval a bit would help.

----
edit: oops, I see now that the scaler interval is configurable with a commandline option. Still, I think the default should be increased, and the exception should be caught, but it might be tricky to figure out how to catch it (without doing a simple retry) without putting AWS-specific code in the generic clusterScaler.py.
aggressive AWS autoscaler polling causes 503 response and pipeline failure
The autoscaler seems to be polling EC2 too often and causing API limits to be exceeded:

```
BotoServerError: 503 Service Unavailable
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>RequestLimitExceeded</Code><Message>Request limit exceeded.</Message></Error></Errors><RequestID>8648be89-3344-4b3c-af07-74c1cd014a81</RequestID></Response>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 248, in check
    scalerThread.join(timeout=0)
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 406, in tryRun
    preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/abstractProvisioner.py", line 156, in setNodeCount
    workerInstances = self._getWorkersInCluster(preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 555, in _getWorkersInCluster
    entireCluster = self._getNodesInCluster(both=True)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 552, in _getNodesInCluster
    return self.__getNodesInCluster(self.ctx, self.clusterName, both=both)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 537, in __getNodesInCluster
    'instance-state-name': 'pending'})
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 622, in get_only_instances
    next_token=next_token)
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 681, in get_all_reservations
    [('item', Reservation)], verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1170, in get_list
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1116, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1028, in _mexe
    raise BotoServerError(response.status, response.reason, body)
BotoServerError: BotoServerError: 503 Service Unavailable
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>RequestLimitExceeded</Code><Message>Request limit exceeded.</Message></Error></Errors><RequestID>8648be89-3344-4b3c-af07-74c1cd014a81</RequestID></Response>
```

Retrying this request might be a bit redundant since boto is probably retrying automatically. The autoscaler thread dies on this error, causing total failure of the pipeline--instead I think it should always catch the exception, back off for a minute or two, then retry the current autoscaler loop. (OK, this behaves almost identically to a retry, but the pipeline should never fail just because the autoscaler is getting throttled.)

Frustratingly, AWS documentation doesn't say *what* exactly the limits are. But maybe increasing the polling interval a bit would help.

----
edit: oops, I see now that the scaler interval is configurable with a commandline option. Still, I think the default should be increased, and the exception should be caught, but it might be tricky to figure out how to catch it (without doing a simple retry) without putting AWS-specific code in the generic clusterScaler.py.


Discussion:
I suspect that the limits vary depending on the actual EC2 instance that is doing the requests. Do you know that information, or the size of the cluster when this error occurred?
The [AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) claim the limits are account-wide, unfortunately. So I guess it isn't necessarily that the autoscaler triggers these limits all by itself, but that getting throttled causes the thread to die.

The cluster was only 4 (m3.xlarge) worker nodes plus 1 (t2.medium) leader node at that point. The pipeline had been running for about 15 minutes.
I suspect that the limits vary depending on the actual EC2 instance that is doing the requests. Do you know that information, or the size of the cluster when this error occurred?
The [AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) claim the limits are account-wide, unfortunately. So I guess it isn't necessarily that the autoscaler triggers these limits all by itself, but that getting throttled causes the thread to die.

The cluster was only 4 (m3.xlarge) worker nodes plus 1 (t2.medium) leader node at that point. The pipeline had been running for about 15 minutes.

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -87,7 +87,7 @@ def __init__(self):
         self.maxPreemptableNodes = 0
         self.alphaPacking = 0.8
         self.betaInertia = 1.2
-        self.scaleInterval = 10
+        self.scaleInterval = 30
         self.preemptableCompensation = 0.0
         
         # Parameters to limit service jobs, so preventing deadlock scheduling scenarios
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -22,6 +22,7 @@
 
 import time
 
+from bd2k.util.retry import retry, never
 from bd2k.util.threading import ExceptionalThread
 
 from toil.batchSystems.abstractBatchSystem import AbstractScalableBatchSystem
@@ -59,6 +60,16 @@ def __init__(self, config, batchSystem):
         self.statsPath = config.clusterStats
         self.scaleable = isinstance(self.batchSystem, AbstractScalableBatchSystem)
 
+    @staticmethod
+    def retryPredicate(e):
+        """
+        Return true if the exception e should be retried by the cluster scaler
+
+        :param e: exception raised during execution of setNodeCount
+        :return: boolean indicating whether the exception e should be retried
+        """
+        return never(e)
+
     def shutDown(self, preemptable):
         if not self.stop:
             # only shutdown the stats threads once
@@ -154,22 +165,24 @@ def setNodeCount(self, numNodes, preemptable=False, force=False):
                 the `numNodes` argument. It represents the closest possible approximation of the
                 actual cluster size at the time this method returns.
         """
-        workerInstances = self._getWorkersInCluster(preemptable)
-        numCurrentNodes = len(workerInstances)
-        delta = numNodes - numCurrentNodes
-        if delta > 0:
-            log.info('Adding %i %s nodes to get to desired cluster size of %i.', delta, 'preemptable' if preemptable else 'non-preemptable', numNodes)
-            numNodes = numCurrentNodes + self._addNodes(workerInstances,
-                                                        numNodes=delta,
-                                                        preemptable=preemptable)
-        elif delta < 0:
-            log.info('Removing %i %s nodes to get to desired cluster size of %i.', -delta, 'preemptable' if preemptable else 'non-preemptable', numNodes)
-            numNodes = numCurrentNodes - self._removeNodes(workerInstances,
-                                                           numNodes=-delta,
-                                                           preemptable=preemptable,
-                                                           force=force)
-        else:
-            log.info('Cluster already at desired size of %i. Nothing to do.', numNodes)
+        for attempt in retry(predicate=self.retryPredicate):
+            with attempt:
+                workerInstances = self._getWorkersInCluster(preemptable)
+                numCurrentNodes = len(workerInstances)
+                delta = numNodes - numCurrentNodes
+                if delta > 0:
+                    log.info('Adding %i %s nodes to get to desired cluster size of %i.', delta, 'preemptable' if preemptable else 'non-preemptable', numNodes)
+                    numNodes = numCurrentNodes + self._addNodes(workerInstances,
+                                                                numNodes=delta,
+                                                                preemptable=preemptable)
+                elif delta < 0:
+                    log.info('Removing %i %s nodes to get to desired cluster size of %i.', -delta, 'preemptable' if preemptable else 'non-preemptable', numNodes)
+                    numNodes = numCurrentNodes - self._removeNodes(workerInstances,
+                                                                   numNodes=-delta,
+                                                                   preemptable=preemptable,
+                                                                   force=force)
+                else:
+                    log.info('Cluster already at desired size of %i. Nothing to do.', numNodes)
         return numNodes
 
     def _removeNodes(self, instances, numNodes, preemptable=False, force=False):
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -63,6 +63,19 @@ def __init__(self, config, batchSystem):
         self.masterPublicKey = self.setSSH()
         self.tags = self._getLeader(self.clusterName).tags
 
+    @staticmethod
+    def retryPredicate(e):
+        return AWSProvisioner.throttlePredicate(e)
+
+    @staticmethod
+    def throttlePredicate(e):
+        # boto/AWS gives multiple messages for the same error...
+        if e.status == 503 and 'Request limit exceeded' in e.body:
+            return True
+        elif e.status == 400 and 'Rate exceeded' in e.body:
+            return True
+        return False
+
     def setSSH(self):
         if not os.path.exists('/root/.sshSuccess'):
             subprocess.check_call(['ssh-keygen', '-f', '/root/.ssh/id_rsa', '-t', 'rsa', '-N', ''])
@@ -244,7 +257,9 @@ def _getLeader(cls, clusterName, wait=False, zone=None):
     def _addTags(cls, instances, tags):
         for instance in instances:
             for key, value in iteritems(tags):
-                instance.add_tag(key, value)
+                for attempt in retry(predicate=AWSProvisioner.throttlePredicate):
+                    with attempt:
+                        instance.add_tag(key, value)
 
     @classmethod
     def _waitForNode(cls, instance, role):
@@ -420,7 +435,11 @@ def _terminateInstances(cls, instances, ctx):
 
     @classmethod
     def _terminateIDs(cls, instanceIDs, ctx):
-        ctx.ec2.terminate_instances(instance_ids=instanceIDs)
+        logger.info('Terminating instance(s): %s', instanceIDs)
+        for attempt in retry(predicate=AWSProvisioner.throttlePredicate):
+            with attempt:
+                ctx.ec2.terminate_instances(instance_ids=instanceIDs)
+        logger.info('Instance(s) terminated.')
 
     def _logAndTerminate(self, instances):
         self._terminateInstances(instances, self.ctx)
@@ -499,27 +518,38 @@ def _addNodes(self, instances, numNodes, preemptable=False):
 
         instancesLaunched = []
 
-        if not preemptable:
-            logger.info('Launching %s non-preemptable nodes', numNodes)
-            instancesLaunched = create_ondemand_instances(self.ctx.ec2, image_id=self._discoverAMI(self.ctx),
-                                      spec=kwargs, num_instances=1)
-        else:
-            logger.info('Launching %s preemptable nodes', numNodes)
-            kwargs['placement'] = getSpotZone(self.spotBid, self.instanceType.name, self.ctx)
-            # force generator to evaluate
-            instancesLaunched = list(create_spot_instances(ec2=self.ctx.ec2,
-                                                           price=self.spotBid,
-                                                           image_id=self._discoverAMI(self.ctx),
-                                                           tags={'clusterName': self.clusterName},
-                                                           spec=kwargs,
-                                                           num_instances=numNodes,
-                                                           tentative=True)
-                                     )
-            # flatten the list 
-            instancesLaunched = [item for sublist in instancesLaunched for item in sublist]
-        wait_instances_running(self.ctx.ec2, instancesLaunched)
+        for attempt in retry(predicate=AWSProvisioner.throttlePredicate):
+            with attempt:
+                # after we start launching instances we want to insure the full setup is done
+                # the biggest obstacle is AWS request throttling, so we retry on these errors at
+                # every request in this method
+                if not preemptable:
+                    logger.info('Launching %s non-preemptable nodes', numNodes)
+                    instancesLaunched = create_ondemand_instances(self.ctx.ec2, image_id=self._discoverAMI(self.ctx),
+                                                                  spec=kwargs, num_instances=numNodes)
+                else:
+                    logger.info('Launching %s preemptable nodes', numNodes)
+                    kwargs['placement'] = getSpotZone(self.spotBid, self.instanceType.name, self.ctx)
+                    # force generator to evaluate
+                    instancesLaunched = list(create_spot_instances(ec2=self.ctx.ec2,
+                                                                   price=self.spotBid,
+                                                                   image_id=self._discoverAMI(self.ctx),
+                                                                   tags={'clusterName': self.clusterName},
+                                                                   spec=kwargs,
+                                                                   num_instances=numNodes,
+                                                                   tentative=True)
+                                             )
+                    # flatten the list
+                    instancesLaunched = [item for sublist in instancesLaunched for item in sublist]
+
+        for attempt in retry(predicate=AWSProvisioner.throttlePredicate):
+            with attempt:
+                wait_instances_running(self.ctx.ec2, instancesLaunched)
+
+        # request throttling retry happens internally to these two methods to insure proper granularity
         AWSProvisioner._addTags(instancesLaunched, self.tags)
         self._propagateKey(instancesLaunched)
+
         logger.info('Launched %s new instance(s)', numNodes)
         return len(instancesLaunched)
 
@@ -527,9 +557,11 @@ def _propagateKey(self, instances):
         if not self.config.sseKey:
             return
         for node in instances:
-            # since we're going to be rsyncing into the appliance we need the appliance to be running first
-            ipAddress = self._waitForNode(node, 'toil_worker')
-            self._rsyncNode(ipAddress, [self.config.sseKey, ':' + self.config.sseKey], applianceName='toil_worker')
+            for attempt in retry(predicate=AWSProvisioner.throttlePredicate):
+                with attempt:
+                    # since we're going to be rsyncing into the appliance we need the appliance to be running first
+                    ipAddress = self._waitForNode(node, 'toil_worker')
+                    self._rsyncNode(ipAddress, [self.config.sseKey, ':' + self.config.sseKey], applianceName='toil_worker')
 
     @classmethod
     def _getBlockDeviceMapping(cls, instanceType):
@@ -547,9 +579,10 @@ def _getBlockDeviceMapping(cls, instanceType):
     @classmethod
     def __getNodesInCluster(cls, ctx, clusterName, preemptable=False, both=False):
         pendingInstances = ctx.ec2.get_only_instances(filters={'instance.group-name': clusterName,
-                                                               'instance-state-name': 'pending'})
+                                                       'instance-state-name': 'pending'})
+
         runningInstances = ctx.ec2.get_only_instances(filters={'instance.group-name': clusterName,
-                                                               'instance-state-name': 'running'})
+                                                       'instance-state-name': 'running'})
         instances = set(pendingInstances)
         if not preemptable and not both:
             return [x for x in instances.union(set(runningInstances)) if x.spot_instance_request_id is None]


Test patch: