Repository: buildbot/buildbot

Issue:
Buildbot gives up on EC2 spot instance requests before EC2 does
This ticket is a migrated Trac ticket [2935](http://trac.buildbot.net/ticket/2935)

People contributed to the original ticket: @unknown_contributor, @djmitche
Ticket created on: `Oct 09 2014`
Ticket last modified on: `Dec 19 2015`

---

When Eight receives a spot request status code other than `pending-evaluation`, `pending-fulfillment`, or `fulfilled`, it concludes that the spot request has failed and gives up on it.  However, [[several status codes|http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances-bid-status.html#spot-instances-bid-status-lifecycle]] are non-terminal, and EC2 may still fulfill the request at a later time.  Nine knows that `price-too-low` is non-terminal, and so cancels the request when giving up on it, but does not do this for other non-terminal status codes.

As a result, EC2 may launch instances that are not tracked by Buildbot.  These will remain running and costing money until the spot price exceeds the bid price, at which point EC2 will automatically terminate the instance.  To avoid this, Buildbot needs to cancel spot requests when it gives up on them.


    2014-10-09 01:17:00-0400 [-] E[[C2LatentBuildSlave]] el6-amd64 requesting spot instance
    2014-10-09 01:18:06-0400 [-] E[[C2LatentBuildSlave]] el6-amd64 has waited 1 minutes for spot request sir-022rlcrg
    2014-10-09 01:18:37-0400 [-] E[[C2LatentBuildSlave]] el6-amd64 failed to fulfill spot request sir-022rlcrg with status capacity-oversubscribed
    2014-10-09 01:18:37-0400 [-] Buildslave el6-amd64 detached from testsuite-el6-amd64
    2014-10-09 01:18:37-0400 [-] while preparing slavebuilder:
            Traceback (most recent call last):
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
                current.result = callback(current.result, *args, **kw)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
                _inlineCallbacks(r, g, deferred)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
                result = result.throw[[ExceptionIntoGenerator]](g)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throw[[ExceptionIntoGenerator]]
                return g.throw(self.type, self.value, self.tb)
            --- <exception caught here> ---
              File "/home/buildbot/env/local/lib/python2.7/site-packages/buildbot/process/builder.py", line 335, in _start[[BuildFor]]
                ready = yield slavebuilder.prepare(self.builder_status, build)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
                result = context.call(ctx, function, *args, **kwargs)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in call[[WithContext]]
                return self.currentContext().call[[WithContext]](ctx, func, *args, **kw)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in call[[WithContext]]
                return func(*args,**kw)
              File "/home/buildbot/env/local/lib/python2.7/site-packages/buildbot/buildslave/ec2.py", line 364, in _request_spot_instance
                request = self._wait_for_request(reservations[0])
              File "/home/buildbot/env/local/lib/python2.7/site-packages/buildbot/buildslave/ec2.py", line 443, in _wait_for_request
                request.id, request.status)
            buildbot.interfaces.[[LatentBuildSlaveFailedToSubstantiate]]: (u'sir-022rlcrg', <Status: capacity-oversubscribed>)
            
    2014-10-09 01:18:37-0400 [-] slave <Build testsuite-el6-amd64> can't build <[[LatentSlaveBuilder]] builder='testsuite-el6-amd64'> after all; re-queueing the request
    
    [...]
    
    2014-10-09 01:32:37-0400 [Broker,32,127.0.0.1] slave 'el6-amd64' attaching from I[[Pv4Address]](TCP, '127.0.0.1', 59401)
    2014-10-09 01:32:37-0400 [Broker,32,127.0.0.1] Slave el6-amd64 received connection while not trying to substantiate.  Disconnecting.




---





Discussion:
https://github.com/buildbot/buildbot/commit/58de958616898818f1a14a1736e6ca7bfe550114#diff-5f8fc956a57a61b9d3a894e07da94455R454 added support for this for price-too-low.  It looks like it would be easy enough to just cancel any request with an unrecognized status code, too.  I can add that, but can't really test it.

Resolving patch:
diff --git a/master/buildbot/worker/ec2.py b/master/buildbot/worker/ec2.py
--- a/master/buildbot/worker/ec2.py
+++ b/master/buildbot/worker/ec2.py
@@ -599,5 +599,8 @@ def _wait_for_request(self, reservation):
             log.msg('%s %s failed to fulfill spot request %s with status %s' %
                     (self.__class__.__name__, self.workername,
                      request['SpotInstanceRequestId'], request_status))
+            # try to cancel, just for good measure
+            self.ec2.meta.client.cancel_spot_instance_requests(
+                SpotInstanceRequestIds=[request['SpotInstanceRequestId']])
             raise LatentWorkerFailedToSubstantiate(
                 request['SpotInstanceRequestId'], request_status)


Test patch: