ceph/ceph — pull request #1148 by gregsfortytwo (2014-01-27 18:14:10+00:00)
Title: Wip 7215 quorum features

This patchset switches the monitor to always use the agreed-upon quorum features when it encodes anything to disk — necessary so that if it syncs to an older daemon, the older daemon can still read the data.
In order to support this, we: persist the quorum features we care about (and have used) to disk, to prevent a too-old daemon from starting up on our store; ignore election messages from daemons which do not support these features; and set up some machinery to make this all behave nicely, and to politely tell the new-but-not-new-enough daemons that they do not support some necessary features.

I tested this manually. The story is a little messier than I'm presenting here because I found some bugs, but basically:
1) Build an old version of Ceph and start a vstart cluster.
2) Build this branch and restart two nodes and make sure that they did not update the quorum features.
3) Build old code again and restart one of the restarted daemons, and make sure it could still read its store.
4) Build this branch and get two daemons running on it, then shut down the old daemon. Make sure the quorum upgraded its features.
5) Turn on an old daemon and make sure that it did not trigger elections on the new mons, and did not crash.
6) Add a "test feature" to the set, build, restart, make sure it behaved appropriately until the upgrade happened.
7) Build this branch vanilla, turn on last daemon, and make sure it shut down cleanly and output the missing feature.

Hurray!

I'm working on a teuthology test that covers this stuff as well, but unfortunately I've run into unrelated bugs so it's not ready yet. :(