juju/juju — pull request #646 by wallyworld (2014-09-01 05:36:04+00:00)
Title: Make provisioner task more robust

Maybe fixes: https://bugs.launchpad.net/juju-core/+bug/1354027

With debug turned on, it could be seen that the provisioner task was being notified of containers added to a host, which the provisioner then started to process. However, at this time, the machine records were not necessarily all written to the database yet. It turned out that for one container, its status document was not yet written, so the provisioner saw an error and ignored the container forever. A couple of seconds after this happened, the status record was written.

So unfortunately, we have no transactions so there's no guarantee of a consistent view of the state model. This branch accounts for the possibility that status may be missing by recording a list of churning machines. Such machines are retried after a delay (5 seconds), which should be enough to ensure a consistent view the next time. If not, the machines are added back to the churning list.

I think this is occurring for the Landscape guys because they use the deployer to smash a whole bunch of changes together into state.

As a drive by, if a dying machine gets a NotFound when it is marked as Dead, then this will just continue on as if the machine is dead.