ceph/ceph — pull request #925 by gregsfortytwo (2013-12-09 19:25:23+00:00)
Title: Wip mon api

This pull request establishes the basis of a monitor versioning API. When a leader is elected, it sends its command set to the peons and they advertise that to clients (and enforce it as the allowed set). If the peon gets a command it doesn't understand, or whose local description differs from the leader's, it forwards the request. If the peons don't get a command set, they assume it's a "classic" (ie, Dumpling or older) monitor, and advertise the classic command set. This lets any cluster which includes this code advertise a unified set of commands, even if the monitors differ.

The last set extends this unification to handle classic monitors as peons (rather than leaders). Each peon shares its local command set when deferring, and if the leader doesn't get a command set from one or more monitors, it falls back to sharing the "classic" command set rather than what it actually supports. This command set sharing also allows more sophisticated techniques in the future without requiring all monitors to be updated: the leader could take an intersection of each command set and advertise those, or advertise the one it recognizes as oldest, or a command set which excludes deprecated versions of commands (even if the cluster can still handle them).

This has been tested manually on my dev box: I turned on a cluster with the code, then compiled dumpling and restarted a monitor. The first time through this exposed some bugs, but after fixing those and restarting the new monitors, the cluster advertised the dumpling command set, and the cluster responded to pg split commands correctly (I manually verified that the monitors were preprocessing messages and then forwarding them, rather than forwarding them when they weren't recognized). When I turned off the dumpling monitor, it advertised the new command set.
Then I recreated the cluster and verified that read-only commands were not forwarded to the leader (so the nodes were recognizing them as a match). I changed one of the commands, restarted the leader, and verified that that command was forwarded to it despite being a read-only command.