Repository: buildbot/buildbot

Issue:
Worker stuck in building state after worker disconnection
The connection got lost but the build went on, or rather kept waiting for worker feedback.
I tried to stop the build by clicking the stop button but it does not work and stays in yellow/building.
It only adds new lines with:
`no reason`
or
`[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ]`

I'm working on giving a fuller log and precise repro instructions.




Discussion:
I tried the following which did not reproduce the error:
* start a build honored by a libvirt worker,
* virsh destroy worker
* the build was correctly cancelled
When I stopped the master, I got the following logs:
2021-01-08 12:38:08+0100 [-] Initiating clean shutdown
2021-01-08 12:38:08+0100 [-]  <Build my_build number:666 results:cancelled>: stopping build: Master Shutdown 5
2021-01-08 12:38:08+0100 [-] RemoteCommand.interrupt <RemoteShellCommand '[......]'> Master Shutdown
2021-01-08 12:38:08+0100 [-]  but this RemoteCommand is already inactive
Unfortunately I'm missing the logs when the build started... there is a jump in logged data between two twistd.log files
Which version of Buildbot have you seen this bug in? There have been a fix to a similar issue in the latent worker implementation recently, so it may be this is already fixed.
This was with 2.10.0 which included #5599 if that is the fix you thought about.
Unfortunately, I miss the log at the time of the issue and it did not reproduce since then, mostly because I diminished the pressure on the latent worker.
I'll make sure to report if the same issue happen again and the logs are available.
Thanks, having a confirmation would be great.
We've been seeing this lately as well, pretty much exclusively on our Windows workers -- apparently some of their network connections are flaky and they end up cancelling themselves, but the master thinks they are still running. (Clicking "Stop" in the web ui just provides an endless "Stopping..." that never actually stops.) Stopping and restarting the master fixes these, but obviously that's not an acceptable fix.

I tried experimenting with shortening keepalive and keepalive_interval but that had no effect. Would love to find a way to fix this as the 'zombie' jobs are a big nuisance. Suggestions welcome; also, if providing logs would help diagnose, please let me know.

As an example, here's the tail of twisted.log from a windows worker that just had this happen (i.e.: the worker's network flaked for a minute; the web UI for the worker shows the status as "Canceled" but the Workers status page still shows the job as throbbing-yellow-capsule, and the "Stop" button doesn't do anything):

```
2021-04-01 15:20:23-0700 [Broker,client]   using PTY: False
2021-04-01 15:21:03-0700 [-] sending app-level keepalive
2021-04-01 15:21:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:22:03-0700 [-] sending app-level keepalive
2021-04-01 15:22:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:23:03-0700 [-] sending app-level keepalive
2021-04-01 15:23:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:24:03-0700 [-] sending app-level keepalive
2021-04-01 15:24:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:25:03-0700 [-] sending app-level keepalive
2021-04-01 15:25:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:26:03-0700 [-] sending app-level keepalive
2021-04-01 15:26:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:27:03-0700 [-] sending app-level keepalive
2021-04-01 15:27:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:28:03-0700 [-] sending app-level keepalive
2021-04-01 15:28:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:29:03-0700 [-] sending app-level keepalive
2021-04-01 15:29:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:30:03-0700 [-] sending app-level keepalive
2021-04-01 15:30:03-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:30:40-0700 [Broker,client] WorkerForBuilder._ackFailed: WorkerForBuilder.sendUpdate
2021-04-01 15:30:40-0700 [Broker,client] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]

2021-04-01 15:30:40-0700 [Broker,client] WorkerForBuilder._ackFailed: WorkerForBuilder.sendUpdate
2021-04-01 15:30:40-0700 [Broker,client] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]

2021-04-01 15:30:40-0700 [Broker,client] WorkerForBuilder._ackFailed: WorkerForBuilder.sendUpdate
2021-04-01 15:30:40-0700 [Broker,client] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]

2021-04-01 15:30:40-0700 [Broker,client] lost remote step
2021-04-01 15:30:40-0700 [Broker,client] stopCommand: halting current command <buildbot_worker.commands.shell.WorkerShellCommand object at 0x000001EB9F326A60>
2021-04-01 15:30:40-0700 [Broker,client] command interrupted, attempting to kill
2021-04-01 15:30:40-0700 [Broker,client] using TASKKILL PID /F /T to kill pid 6860
2021-04-01 15:30:40-0700 [Broker,client] taskkill'd pid 6860
2021-04-01 15:30:40-0700 [-] Scheduling retry 1 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 2.4276185084175483 seconds.
2021-04-01 15:30:40-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:30:40-0700 [-] command finished with signal None, exit code 1, elapsedTime: 616.420314
2021-04-01 15:30:40-0700 [-] would sendStatus but not .running
2021-04-01 15:30:40-0700 [-] WorkerForBuilder.commandComplete None
2021-04-01 15:30:42-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:03-0700 [-] Scheduling retry 2 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 2.615052142782245 seconds.
2021-04-01 15:31:03-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:06-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:06-0700 [-] Scheduling retry 3 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 3.750537772489395 seconds.
2021-04-01 15:31:06-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:09-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:31-0700 [-] Scheduling retry 4 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 5.930810313276111 seconds.
2021-04-01 15:31:31-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:36-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:36-0700 [-] Scheduling retry 5 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 8.589732518396882 seconds.
2021-04-01 15:31:36-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:45-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:45-0700 [-] Scheduling retry 6 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 11.451941565413545 seconds.
2021-04-01 15:31:45-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:57-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:31:57-0700 [-] Scheduling retry 7 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 17.743804267365913 seconds.
2021-04-01 15:31:57-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:32:14-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:32:35-0700 [-] Scheduling retry 8 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 26.604889648326324 seconds.
2021-04-01 15:32:35-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:33:02-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:33:02-0700 [-] Scheduling retry 9 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 38.491694040154265 seconds.
2021-04-01 15:33:02-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:33:40-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:34:02-0700 [-] Scheduling retry 10 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 58.387752671230466 seconds.
2021-04-01 15:34:02-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:35:00-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:35:00-0700 [-] Scheduling retry 11 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 60.03861585096624 seconds.
2021-04-01 15:35:00-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:36:00-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:36:21-0700 [-] Scheduling retry 12 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 60.74238562665125 seconds.
2021-04-01 15:36:21-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:37:22-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:37:22-0700 [-] Scheduling retry 13 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x000001EB9F082CA0> in 60.9445337721182 seconds.
2021-04-01 15:37:22-0700 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:38:23-0700 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x000001EB9F082820>
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] message from master: attached
2021-04-01 15:38:33-0700 [Broker,client] Connected to buildmaster; worker is ready
2021-04-01 15:38:33-0700 [Broker,client] sending application-level keepalives every 60 seconds
2021-04-01 15:39:33-0700 [-] sending app-level keepalive
2021-04-01 15:39:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:40:33-0700 [-] sending app-level keepalive
2021-04-01 15:40:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:41:33-0700 [-] sending app-level keepalive
2021-04-01 15:41:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:42:33-0700 [-] sending app-level keepalive
2021-04-01 15:42:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:43:33-0700 [-] sending app-level keepalive
2021-04-01 15:43:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:44:33-0700 [-] sending app-level keepalive
2021-04-01 15:44:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:45:33-0700 [-] sending app-level keepalive
2021-04-01 15:45:33-0700 [Broker,client] Master replied to keepalive, everything's fine
2021-04-01 15:46:33-0700 [-] sending app-level keepalive
2021-04-01 15:46:33-0700 [Broker,client] Master replied to keepalive, everything's fine
```
This is still happening very regularly for us, and it's becoming a big issue -- the stuck workers are 'zombies' that can't get new jobs assigned to them (since the master thinks they are still working). Restarting the master clears this up, but that means interrupting all the other (non-zombie) workers. 

As a stopgap, is there any way to tell the master to flush the work queue of a specific worker without having to stop and restart? That would at least give us a less-bad workaround.


@steven-johnson Could you compress the master log and send it to povilas@radix.lt? There may be some clues of what's happening.

By the way, are the affected workers running on libvirt or on bare hardware?
> Could you compress the master log and send it to [povilas@radix.lt](mailto:povilas@radix.lt)? There may be some clues of what's happening.

Yes, but I'll probably have to do some twiddling with our setup (in off hours) to get something that isn't polluted with lots of irrelevant stuff, so it may take a day or two. (Do you also want the worker logs?)
 
> By the way, are the affected workers running on libvirt or on bare hardware?

Bare hardware. (We do occasionally see this behavior on other buildbots, eg linux on bare hardware, but 95+% of the occurrences are on Windows.)
> (Do you also want the worker logs?)

Actually yes, I didn't consider that I might not be able to easily tell when the worker disconnected just from master side.
[FAILURES.zip](https://github.com/buildbot/buildbot/files/6461324/FAILURES.zip)

OK, the enclosed zip as the twistd.log files from the master and two workers.
The master is a mac latop; both workers are running Win10.

Both workers are tasked with running a heavy load (rebuilding LLVM from source).

The master also has a other tasks running that consume meaningful CPU.

Let me know if this helps.


The master log shows a lot of those exceptions:

```
'NoneType' object has no attribute 'info'
	/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/twisted/internet/defer.py:1445:_inlineCallbacks
	/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/twisted/internet/defer.py:47:run
	/Users/srj/Library/Python/3.6/lib/python/site-packages/buildbot/process/properties.py:491:getRenderingFor
	/Users/srj/Library/Python/3.6/lib/python/site-packages/buildbot/process/properties.py:209:render
	--- <exception caught here> ---
	/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/twisted/internet/defer.py:167:maybeDeferred
	/Users/srj/Library/Python/3.6/lib/python/site-packages/buildbot/process/properties.py:533:getRenderingFor
	/Users/srj/Library/Python/3.6/lib/python/site-packages/buildbot/process/build.py:778:getWorkerInfo
	]]
	--- <exception caught here> ---
	/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/twisted/internet/defer.py:47:run
	/Users/srj/Library/Python/3.6/lib/python/site-packages/buildbot/process/properties.py:493:getRenderingFor
	]]
	]]
	]]
```
it looks like there is some property rendering that is triggered that requires some worker info. This actuall raises an exception because at that point, the worker is disconnected and the build's worker is set to None.
There is no info in the log on what is the actual origin of this property rendering :-/


maybe a simple fix to this is to decide getWorkerInfo should rather return {} if worker is disconnected:

```patch
diff --git a/master/buildbot/process/build.py b/master/buildbot/process/build.py
index af80404f5..5ca51a2e3 100644
--- a/master/buildbot/process/build.py
+++ b/master/buildbot/process/build.py
@@ -775,4 +775,6 @@ class Build(properties.PropertiesMixin):
                                             lambda: self.finished)

     def getWorkerInfo(self):
-        return self.workerforbuilder.worker.info
+        if self.workerforbuilder.worker is not None:
+            return self.workerforbuilder.worker.info
+        return {}
```


> There is no info in the log on what is the actual origin of this property rendering :-/

Where would add logging on that?

> maybe a simple fix to this is to decide getWorkerInfo should rather return {} if worker is disconnected:

Are you suggesting I try that out locally? (Happy to do so, just not clear)
> > There is no info in the log on what is the actual origin of this property rendering :-/
> 
> Where would add logging on that?

I think adding some log here might give some more clue of what we are trying to render:
https://github.com/buildbot/buildbot/blob/master/master/buildbot/process/properties.py#L682

Then, there is this old PR on twisted that is supposed to fix the traceback generation, that was never merged
https://github.com/twisted/twisted/pull/988

Maybe you can try repro with that version of twisted.

> 
> > maybe a simple fix to this is to decide getWorkerInfo should rather return {} if worker is disconnected:
> 
> Are you suggesting I try that out locally? (Happy to do so, just not clear)

If you have a simple repro, I guess this would be worth to see of your build correctly manage to finish.

We can see a bit earlier in the log the connection break:
```log
2021-05-11 11:34:23-0700 [-] RemoteCommand.interrupt <RemoteShellCommand '['ninja', '-C', 'D:/build_bot/worker/llvm-13-x86-64-windows/llvm-build', '-j', '14', 'install']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
	]
2021-05-11 11:34:23-0700 [-] RemoteCommand.disconnect: lost worker
2021-05-11 11:34:23-0700 [Broker,2,104.132.0.72] Got workerinfo from 'win-worker-1'
2021-05-11 11:34:23-0700 [-] bot attached
2021-05-11 11:34:23-0700 [Broker,2,104.132.0.72] Worker win-worker-1 attached to llvm-12-x86-64-windows/win-worker-1
```

We can see here that the master detects connection lost and the worker reconnects in the same second.

Franckly I am not sure so much time shall be spent in making buildbot fail properly for such network flakyness. That time would be better spent in fixing the flackyness IMHO.

The pb protocol is not resistant to network flakyness, this is a shame, but too hard to fix at that point.
I hope we can fix this issue with the next protocol @mokibit  is working on.

> Franckly I am not sure so much time shall be spent in making buildbot fail properly for such network flakyness

I think it's worth it. There will always be weird environments with plenty of things being unreliable. Buildbot should handle them properly.

Regarding the failure, I suspect that the cause may be rendering of the build properties that is done before each step in `setBuildProperties()`.

@steven-johnson I think you can apply the patch suggested by Pierre as an interim solution. It may not help though. In the mean time I will look for a proper solution to shut down the master-side build completely as soon as worker disconnects.
> @steven-johnson I think you can apply the patch suggested by Pierre as an interim solution. It may not help though. In the mean time I will look for a proper solution to shut down the master-side build completely as soon as worker disconnects.

Unfortunately, no, it doesn't seem to:
- when flakiness kills a task, the "Stop" button is still inoperative (just does "Stopping..." forever)
- The dead zombie task still hangs around on master forever
> I think it's worth it. There will always be weird environments with plenty of things being unreliable. Buildbot should handle them properly.

+1

Resolving patch:
diff --git a/master/buildbot/process/build.py b/master/buildbot/process/build.py
--- a/master/buildbot/process/build.py
+++ b/master/buildbot/process/build.py
@@ -197,7 +197,7 @@ def getWorkerCommandVersion(self, command, oldversion=None):
         return self.workerforbuilder.getWorkerCommandVersion(command, oldversion)
 
     def getWorkerName(self):
-        return self.workerforbuilder.worker.workername
+        return self.workername
 
     @staticmethod
     def setupPropertiesKnownBeforeBuildStarts(props, requests, builder,
@@ -260,6 +260,7 @@ def setupWorkerBuildirProperty(self, workerforbuilder):
     def setupWorkerForBuilder(self, workerforbuilder):
         self.path_module = workerforbuilder.worker.path_module
         self.workername = workerforbuilder.worker.workername
+        self.worker_info = workerforbuilder.worker.info
 
     @defer.inlineCallbacks
     def getBuilderId(self):
@@ -294,6 +295,11 @@ def startBuild(self, workerforbuilder):
 
         worker = workerforbuilder.worker
 
+        # Cache the worker information as variables instead of accessing via worker, as the worker
+        # will disappear during disconnection and some of these properties may still be needed.
+        self.workername = worker.workername
+        self.worker_info = worker.info
+
         log.msg("{}.startBuild".format(self))
 
         # TODO: this will go away when build collapsing is implemented; until
@@ -322,7 +328,7 @@ def startBuild(self, workerforbuilder):
                                    self.sources, self.number)
 
         # then narrow WorkerLocks down to the right worker
-        self.locks = [(l.getLockForWorker(workerforbuilder.worker.workername),
+        self.locks = [(l.getLockForWorker(self.workername),
                        a)
                       for l, a in self.locks]
         metrics.MetricCountEvent.log('active_builds', 1)
@@ -775,4 +781,4 @@ def waitUntilFinished(self):
                                             lambda: self.finished)
 
     def getWorkerInfo(self):
-        return self.workerforbuilder.worker.info
+        return self.worker_info


Test patch:
diff --git a/master/buildbot/test/fake/worker.py b/master/buildbot/test/fake/worker.py
--- a/master/buildbot/test/fake/worker.py
+++ b/master/buildbot/test/fake/worker.py
@@ -37,6 +37,7 @@ class FakeWorker:
     def __init__(self, master):
         self.master = master
         self.conn = fakeprotocol.FakeConnection(self)
+        self.info = properties.Properties()
         self.properties = properties.Properties()
         self.defaultProperties = properties.Properties()
         self.workerid = 383
diff --git a/master/buildbot/test/integration/test_worker.py b/master/buildbot/test/integration/test_worker.py
--- a/master/buildbot/test/integration/test_worker.py
+++ b/master/buildbot/test/integration/test_worker.py
@@ -22,8 +22,12 @@
 from buildbot.machine.base import Machine
 from buildbot.process.buildstep import BuildStep
 from buildbot.process.factory import BuildFactory
+from buildbot.process.properties import Interpolate
 from buildbot.process.results import CANCELLED
+from buildbot.process.results import RETRY
 from buildbot.test.fake.latent import LatentController
+from buildbot.test.fake.step import BuildStepController
+from buildbot.test.fake.worker import WorkerController
 from buildbot.test.util.integration import RunFakeMasterTestCase
 
 try:
@@ -214,6 +218,37 @@ def test_worker_reconfigure_with_new_builder(self):
         # reconfig should succeed
         yield self.reconfig_master(config_dict)
 
+    @defer.inlineCallbacks
+    def test_step_with_worker_build_props_during_worker_disconnect(self):
+        """
+        We need to handle worker disconnection and steps with worker build properties gracefully
+        """
+        controller = WorkerController(self, 'local')
+
+        stepcontroller = BuildStepController()
+
+        config_dict = {
+            'builders': [
+                BuilderConfig(name="builder", workernames=['local'],
+                              properties={'worker': Interpolate("%(worker:numcpus)s")},
+                              factory=BuildFactory([stepcontroller.step])),
+            ],
+            'workers': [controller.worker],
+            'protocols': {'null': {}},
+            'multiMaster': True,
+        }
+
+        yield self.setup_master(config_dict)
+
+        builder_id = yield self.master.data.updates.findBuilderId('builder')
+        yield self.create_build_request([builder_id])
+
+        yield controller.connect_worker()
+        self.reactor.advance(1)
+        yield controller.disconnect_worker()
+        self.reactor.advance(1)
+        yield self.assertBuildResults(1, RETRY)
+
     @defer.inlineCallbacks
     def test_worker_os_release_info_roundtrip(self):
         """