mon/MonitorStore: store global version (gv) alongside paxos values

Keep a parallel set of files mapping paxos states (NNN of foo/NNN) to a
global version number, stored at foo_gv/NNN.  Provide per-state accessors
and wire into put_bl_sn_map() and trim().

Signed-off-by: Sage Weil <sage@inktank.com>
mon: allow all paxos instances to recover before going writeable

Wait for all paxos instances to settle (recover, and commit any proposed
but uncommitted value) before any of them go writeable.  This will allow
the leader to choose global versions responsibly.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: provide global paxos version helpers

For each election epoch, we wait for paxos machines to settle, and then
start allocating global paxos version numbers.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: include global versions in paxos, slurp messages

Signed-off-by: Sage Weil <sage@inktank.com>
mon: specify/allocate a global version at begin time

Each time the leader proposes a new state, it allocates a new global
version number to go with it.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: share global version with proposal, collect, commits, slurp, etc.

Share the global version through all of the paxos messages that share
data, and when slurping.  This ensures that committed values get the gv's
associated with them, and that proposed values, if they are later committed
by a new leader, will retain their original gv, and that new gv's will only
be allocated after that -- once paxos settles and things go writeable.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: fix paxos completions

If we are already recovered, trigger completions immediately.  Otherwise,
wait, and when we do recover trigger them all.  This fixes a number of
things, most notably making active events only trigger when everyone is
recovered.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: track when peon paxos machines activate

Make note of when peons get leases, indicating that all paxos machines have
recovered.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: move start-up feature check into static helper

Signed-off-by: Sage Weil <sage@inktank.com>
mon: move feature read/write into helpers

Signed-off-by: Sage Weil <sage@inktank.com>
mon: refactor monitor feature checks

Three helpers:
 - legacy features (if file isn't present)
 - required features
 - supported features

Write out the feature file on startup with legacy values if it isn't
present, so that everything else can assume it is there.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: share quorum features on election victory

Inform peons what the quorum features are.  Previously only the leader
knew, which was of limited utility.

This is one piece of #2690.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: set new incompat GV feature when paxos stabilizes for the first time

This is a marker that future versions will use to know whether they can
safely convert the monitor data to the new format.  If the GV feature is
not present, they will refuse to convert.

Also set the ondisk GV feature at the same time.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: do not issue global versions if quorum does not support the feature

Signed-off-by: Sage Weil <sage@inktank.com>
mon: move setting of ondisk GV feature into helper

Signed-off-by: Sage Weil <sage@inktank.com>
mon: require MON_GV protocol feature

Require the MON_GV feature when

 - we see the ondisk feature is set on bootup
 - we enable the ondisk feature

This means that once we form a quorum with the feature and enable it on
disk, there is no going back; we won't be able to talk to old monitors
without the feature, and a downgrade won't be possible.

Hopefully, in practice, any monitors with old code will be up at the time
we are upgrading, such that the quorum will not include the feature and we
won't make the transition.  Otherwise, if they are down, and the remaining
nodes have the feature and enable it, and the old code starts up, it won't
be able ot join until it is upgraded to the new code as well.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: clean up recovered_leader() checks

Assert they are called only once per machine per election epoch.  Fix the
recovered_peon() caller to do that.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: include auth mon's paxos version on latest gv calculation

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Merge remote-tracking branch 'gh/wip-mon-gv'

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Conflicts:
	src/include/ceph_features.h