coreos/fleet — pull request #928 by bcwaldon (2014-09-29 23:19:36+00:00)
Title: Engine Versioning

As detailed in #924, there exists a breaking change that causes upgrades from v0.6.2 to v0.7.0+ to fail. This is due to the fact that the engine's approach to scheduling actually changed. The goal of this PR is to be able to handle this sort of incompatible change in the engine gracefully moving forward.

The most sane way to do this is to associate a version number with the engine, and to assert that only the highest-versioned engines in a cluster can assume the lead role (i.e. make scheduling decisions). The cluster agrees on what the highest version is using a key in etcd. The engine's primary job is to ensure that the value stored in this key is greater than or equal to its own version. If the value is ever less than a given engine's version, that engine must attempt to upgrade the value before continuing operation. If the value is greater than the engine's version, the engine must not attempt to acquire the lead role.

Now we do have one nasty detail to deal with, and that is how an engine that has already acquired the lead role is notified that the agreed-upon highest-version has increased. Since this upgrade mechanism is not going to be used often, the best solution is to allow the higher-version engines to forcefully acquire the lock held by the lower-version engine. Since the lower-version engine already uses a leasing mechanism to maintain his ownership of the lead role, the subsequent renew operation will fail and the engine will stop scheduling work.

The last piece to address is the fact that current engines out in the wild know nothing about versioning, and aren't going to gate their startup on checking the highest version in the cluster. We can actually assign a virtual version number of `0` to them, and use `1` to represent the new engine behavior. This new engine will understand that it should be the lead after updating the value stored in etcd to `1`, and will attempt to break any locks held by the old code. Now, if the new nodes die for some reason, yes, an old version can pick the lock back up and continue. This is not the end of the world, however, as the old code is actually forwards-compatible with the new code.