buildbot/buildbot — pull request #1019 by bhearsum (2013-12-20 19:19:32+00:00)
Title: Check slave before trying to access it - fixes race condition in slave shutdown

From https://bugzilla.mozilla.org/show_bug.cgi?id=923155#c11 and #c12:
some more digging into this today and we _think_ there's a race condition in normal (that is, non-graceful) shutdown. We correlated two instances of this happening with slaves in shutdown. Specifically:

From buildbot-master51:
71952 2013-10-18 07:42:08-0700 [Broker,38873,10.134.57.229] BuildSlave.detached(tst-linux64-ec2-016)
71953 2013-10-18 07:42:09-0700 [-] starting build using slave
71954 2013-10-18 07:42:09-0700 [-] Unhandled Error

And from tst-linux64-ec2-016:
5926 2013-10-18 07:42:08-0700 [Broker,client] lost remote
5927 2013-10-18 07:42:08-0700 [Broker,client] Lost connection to buildbot-master51.srv.releng.use1.mozilla.com:9201
5928 2013-10-18 07:42:08-0700 [Broker,client] Stopping factory
5929 2013-10-18 07:42:08-0700 [-] Main loop terminated.
5930 2013-10-18 07:42:08-0700 [-] Server Shut Down.

From buildbot-master52:
18063 2013-10-18 09:01:41-0700 [Broker,38810,10.134.56.83] BuildSlave.detached(tst-linux32-ec2-096)
18064 2013-10-18 09:01:43-0700 [Broker,38812,10.134.57.241] BuildSlave.detached(tst-linux32-ec2-029)
18065 2013-10-18 09:01:44-0700 [-] starting build using slave
18066 2013-10-18 09:01:44-0700 [-] Unhandled Error

And from tst-linux32-ec2-029:
435 2013-10-18 09:01:41-0700 [Broker,client] lost remote
436 2013-10-18 09:01:41-0700 [Broker,client] Lost connection to buildbot-master52.srv.releng.use1.mozilla.com:9201
437 2013-10-18 09:01:41-0700 [Broker,client] Stopping factory
438 2013-10-18 09:01:41-0700 [-] Main loop terminated.
439 2013-10-18 09:01:41-0700 [-] Server Shut Down.

The times on the second instance don't correlate _quite_ as well as the first, but that could be due to clock differences between the master and the slave. It's certainly clear that in both cases there are slaves shutting down directly before the master tries to start the job. So the master may be trying to start the job with that slave - which looks okay when it calls startBuild() - but then the slave shuts down before startBuild() can finish initiating the job.

Assuming the assessment in comment #11 is correct, I think this patch should fix us up...by making sure we don't throw in prepare() and return False instead, the "if not ready" block from _prepared should run, and reset the claim: https://hg.mozilla.org/build/buildbot/file/b22c537e9f18/master/buildbot/process/builder.py#l837.