mon/Paxos: put source mon id in a temp variable

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bb046ed01ecf58b8c87eeeee2e00a476e6fba467)
mon/Paxos: verify all new peons are still contiguous at end of round

During the collect phase we verify that each peon has overlapping or
contiguous versions as us (and can therefore be caught up with some
series of transactions).  However, we *also* assimilate any new states we
get from those peers, and that may move our own first_committed forward
in time.  This means that an early responder might have originally been
contiguous, but a later one moved us forward, and when the round finished
they were not contiguous any more.  This leads to a crash on the peon
when they get our first begin message.

For example:

 - we have 10..20
 - first peon has 5..15
   - ok!
 - second peon has 18..30
   - we apply this state
 - we are now 18..30
 - we finish the round
   - send commit to first peon (empty.. we aren't contiguous)
   - send no commit to second peon (we match)
 - we send a begin for state 31
   - first peon crashes (it's lc is still 15)

Prevent this by checking at the end of the round if we are still
contiguous.  If not, bootstrap.  This is similar to the check we do above,
but reverse to make sure *we* aren't too far ahead of *them*.

Fixes: #9053
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e5ce5f0dcec9bbe9ed4a6b41758ab7802614810)
mon/Paxos: share state and verify contiguity early in collect phase

We verify peons are contiguous and share new paxos states to catch peons
up at the end of the round.  Do this each time we (potentially) get new
states via a collect message.  This will allow peons to be pulled forward
and remain contiguous when they otherwise would not have been able to.
For example, if

  mon.0 (leader)  20..30
  mon.1 (peon)    15..25
  mon.2 (peon)    28..40

If we got mon.1 first and then mon.2 second, we would store the new txns
and then boot mon.1 out at the end because 15..25 is not contiguous with
28..40.  However, with this change, we share 26..30 to mon.1 when we get
the collect, and then 31..40 when we get mon.2's collect, pulling them
both into the final quorum.

It also breaks the 'catch-up' work into smaller pieces, which ought to
smooth out latency a bit.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c54f1e4d66b22bad715ac17e9baa72ab93e48c46)
mon/Paxos: fix off-by-one in last_ vs first_committed check

peon last_committed + 1 == leader first_committed is okay.  Note that the
other check (where I clean up whitespace) gets this correct.

Fixes: #9301 (partly)
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d81cd7f86695185dce31df76c33c9a02123f0e4a)
mon: re-bootstrap if we get probed by a mon that is way ahead

During bootstrap we verify that our paxos commits overlap with the other
mons we will form a quorum with.  If they do not, we do a sync.

However, it is possible we pass those checks, then fail to join a quorum
before the quorum moves ahead in time such that we no longer overlap.
Currently nothing kicks up back into a probing state to discover we need
to sync... we will just keep trying to call or join an election instead.

Fix this by jumping back to bootstrap if we get a probe that is ahead of
us.  Only do this from non probe or sync states as these will be common;
it is only the active and electing states that matter (and probably just
electing!).

Fixes: #9301
Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c421b55e8e15ef04ca8aeb47f7d090375eaa8573)
Merge pull request #2657 from ceph/wip-9053-9301-firefly

mon: backport two paxos fixes to firefly

Reviewed-by: Joao Luis <joao@redhat.com>