Repository: DataBiosphere/toil

Issue:
AWSProvisioner/preemptable-scaler can be killed by 400'd spot request
```
ip-172-31-3-112.us-west-2.compute.internal 2016-10-21 04:20:06,404 preemptable-scaler INFO toil.provisioners.aws.awsProvisioner: Adjusting cluster size by 1
ip-172-31-3-112.us-west-2.compute.internal 2016-10-21 04:20:07,617 preemptable-scaler INFO toil.provisioners.aws.awsProvisioner: Launching 1 preemptable nodes
Exception in thread preemptable-scaler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 361, in tryRun
    preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 66, in setNodeCount
    self._addNodes(instancesToLaunch, preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 283, in _addNodes
    clusterName=self.clusterName, spec=kwargs, num_instances=instancesToLaunch))
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 446, in create_spot_instances
    ec2.create_tags([x.id for x in requests], {'clusterName': clusterName})
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 4203, in create_tags
    return self.get_status('CreateTags', params, verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSpotInstanceRequestID.NotFound</Code><Message>The spot instance request ID 'sir-12345678' does not exist</Message></Error></Errors><RequestID>aaaaaaaa-bbbb-cccc-dddddddddddd</RequestID></Response>
```

AWSProvisioner/preemptable-scaler can be killed by 400'd spot request
```
ip-172-31-3-112.us-west-2.compute.internal 2016-10-21 04:20:06,404 preemptable-scaler INFO toil.provisioners.aws.awsProvisioner: Adjusting cluster size by 1
ip-172-31-3-112.us-west-2.compute.internal 2016-10-21 04:20:07,617 preemptable-scaler INFO toil.provisioners.aws.awsProvisioner: Launching 1 preemptable nodes
Exception in thread preemptable-scaler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 361, in tryRun
    preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 66, in setNodeCount
    self._addNodes(instancesToLaunch, preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 283, in _addNodes
    clusterName=self.clusterName, spec=kwargs, num_instances=instancesToLaunch))
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 446, in create_spot_instances
    ec2.create_tags([x.id for x in requests], {'clusterName': clusterName})
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 4203, in create_tags
    return self.get_status('CreateTags', params, verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSpotInstanceRequestID.NotFound</Code><Message>The spot instance request ID 'sir-12345678' does not exist</Message></Error></Errors><RequestID>aaaaaaaa-bbbb-cccc-dddddddddddd</RequestID></Response>
```



Discussion:
One more eventual consistency problem to deal with. Adding a retry loop should fix this.

One more eventual consistency problem to deal with. Adding a retry loop should fix this.


Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -438,12 +438,19 @@ def create_spot_instances(ec2, price, image_id, spec, clusterName,
 
     :rtype: Iterator[list[Instance]]
     """
+    def spotRequestNotFound(e):
+        return e.error_code == "InvalidSpotInstanceRequestID.NotFound"
+
     for attempt in retry_ec2(retry_for=a_long_time,
                              retry_while=inconsistencies_detected):
         with attempt:
             requests = ec2.request_spot_instances(price, image_id, count=num_instances, **spec)
 
-    ec2.create_tags([x.id for x in requests], {'clusterName': clusterName})
+    for requestID in (request.id for request in requests):
+        for attempt in retry_ec2(retry_while=spotRequestNotFound):
+            with attempt:
+                ec2.create_tags([requestID], {'clusterName': clusterName})
+
     num_active, num_other = 0, 0
     # noinspection PyUnboundLocalVariable,PyTypeChecker
     # request_spot_instances's type annotation is wrong


Test patch: