mon: Paxos: s/get_version()/get_last_committed()/

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: sync_start_chunks() to use sync_provider instead of arg

The argument is redundant as it's always 'sync_provider'.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: rework sync_store_init()

Instead of just initiating a store for a sync, also check the current
store for any on-going sync state and make sure we backup the most
up-to-date monmap we can find, along with the paxos versions from the last
clean state we saw (either from the current store if no sync is on-going,
or from our previous store state if a sync is on-going).

Also make sure we clear out the store before we start the sync; we've also
added a function to make this process the same from other places that
might need to do it, and make sure we compact the store if 'mon compact
on sync' is set to true.

Fixes: #5256 (partially)
Backport: cuttlefish

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: rework preinit's checks for an on-going sync.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: use appropriate func to clear store on sync_requester_abort()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: if sync on-going, use backup paxos vers to match probe replies

Considering that after an aborted sync we will end up with a cleared store,
we know that our paxos state has been zeroed out.  This means that our
first and last committed versions are 0.  This also means that we would
be willing to sync from *any* monitor with a version > 0.

However!  The fact is that this monitor may have seen some previous paxos
state.  If so, we must guarantee that we only sync from a monitor with a
higher paxos version than the last consistent state we at some point had.

What is this trying to avoid, you ask?  Well, consider we have 3 monitors
(A, B and C) and we are converting them from bobtail to cuttlefish.

Now imagine that at the time of the upgrade we would have
  A    -> version 10
  B, C -> version 20

And consider the following chain of events:

  - A converts from bobtail and starts
  - B converts from bobtail; post-conversion, crashes, restarts and needs to sync
B syncs from A (only mon available) <--- this is where the last_committed bit is important!!! ****
A, B form quorum (version 10)
C converts (version 20) and starts
hilarity ensues

---

Now imagine that we add a new monitor D to an already existing quorum
composed of A and B.  In-between adding D to the monmap and bringing it
up, C came back to life and started synchronizing from B.  However, all of
a sudden, the DC's network got fubared, and B is now unreachable!  C aborts
its sync due to it timing out, and to avoid an inconsistent state, its
store is cleared.

So, to gather our thoughts, we now have an up-to-date monitor A available,
a

-- Every time I'm closing in on a decent argument, things just blow up on
-- my face.  The sync appears to be solid as hell, but we've seen this
-- happening!  A has version 10, B has version 11, C has version 1; C
-- syncs from A, form a quorum with v 10; B comes up with v 11 and
-- map weirdness happens!
-- Just can't find a justification for that! Gah

Now imagine that from the remaining quorum, a given monitor C's disk fails
and has to be re-mkfs'd.  Now, to top it off, those two monitors that had
been left running, D and E, and with which C would form a quorum, become
unresponsive as the DC's network just got fubared.  But wait, you have been
able to bring A and B up!

So here's the thing: if we allow C to synchronize from A and B, they make
just the right number to form a quorum.  But as A and B are out-of-date,
the resulting quorum will be starting off from a past state, and that's a
really bad to happen to the cluster.  So we want to limit

This avoids that we end up synchronizing from a monitor that only saw a

Conflicts:
	src/mon/Monitor.cc
mon: OSDMonitor: don't ignore apply_incremental()'s return on UfP [1]

apply_incremental() may return -EINVAL.  Don't ignore it.

[1] UfP = Update from Paxos

Fixes: #5343

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
test: test_store_tool: global init before using LevelDBStore

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: MonitorDBStore: allow transactions to erase whole prefixes.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: use tx erase_prefix() instead during sync_clear_store()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>