mon: fix scrub vs paxos race: refresh on commit, not round completion

Consider:

 - paxos starts a commit N+1
 - a majority of the peers ack it
  - paxos::commit() writes N+1 it to disk
  - tells peers to commit
 - peers commit N+1, *and* refresh_from_paxos(), and generate N+1 full map
 - leader does _scrub on N+1, without latest full osdmap
 - peers do _scrub on N+1, with latest full osdmap
 - leader finishes paxos gather, does refresh_from_paxos()
 -> scrub fails.

Fix this by doing the refresh_from_paxos() at commit time and not when
the paxos round finishes.  We move the refresh out of finish_proposal
and into its own helper, and update all callers accordingly.  This
keeps on-disk state more tightly in sync with in-memory state and
avoids the need for a e.g., kludgey workaround in the scrub code.

We also simplify the bootstrap checks a bit by doing so immediately
and relying on the normal bootstrap paxos reset paths to clean up
any waiters.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: separate proposal commit from the end of the round

Each commit should match with exactly one proposal; finish it when we
actually commit it and make sensible asserts.

The old finish_proposal() turns into finish_round(), and performs
generic checks and cleanup associated with the transition from
updating -> active.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/PaxosService: assert not proposing in propose_pending

Signed-off-by: Sage Weil <sage@inktank.com>
mon/PaxosService: consolidate resetting in restart()

We had duplicated code in election_finished() and restart(), and it was
incomplete.  Put it all in reset() only (the mon should have called
restart() long before the election finishes).  Note that we cannot
assert as much in election_finished() because another service may have
just cross-proposed.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: once sync full is chosen, make sure we don't change our mind

It is possible for a sequence like:

 - probe
 - first probe reply has paxos trim that indicates a full sync is
   needed
 - start sync
 - clear store
 - something happens that makes us abort and bootstrap (e.g., the
   provider mon restarts
 - probe
 - first probe reply has older paxos trim bound and we call an election
 - on election completion, we crash because we have no data.

Non-determinism of the probe decision aside, we need to ensure that
the info we share during probe (fc, lc) is accurate, and that once we
clear the store we know we *must* do a full sync.

Fixes: #5621
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
mon: set peon state to electing if other mons call an election

Previously we would call mon->reset() and set various flags (like
exited_quorum timestamp), but the state would remain PEON.  Make an
explicit join_election() callback and set the state there, and add
asserts in reset() (renamed to be private) so that we ensure all
callers are well-behaved.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: drop win_election() _reset() kludge and strengthen assertions

This is only there for the benefit of win_standalone_election(), but it
doesn't need it, it clutters the code, and weakens our assertions.

Now the only win_election() callers are win_standalone_election() (which
is a single path that just did _reset()) and from the elector.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: set forwarded message recv stamp

Set it to the stamp of the MForward that carried us.  One could argue
we really want the original receive stamp on the origin, but that is
not available to us, and this is better than nothing.

In particular, this gives 'ceph log ...' commands a timestamp when they
are forwarded via a peon.  The stamp is still between when the request
is sent and when it is committed/acked, so all is well from the
client's perspective.

Signed-off-by: Sage Weil <sage@inktank.com>