mon/PaxosService: do not prepare new pending if still proposing

The _active callback can get called while are already proposing.  If
that happens, we should not prepare a fresh new pending but should
wait for the previous proposal to finish.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: bootstrap peon too if monmap updates

If we get a monmap update, the leader bootstraps.  Peons should do the
same.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
mon: fix scrub vs paxos race: refresh on commit, not round completion

Consider:

 - paxos starts a commit N+1
 - a majority of the peers ack it
  - paxos::commit() writes N+1 it to disk
  - tells peers to commit
 - peers commit N+1, *and* refresh_from_paxos(), and generate N+1 full map
 - leader does _scrub on N+1, without latest full osdmap
 - peers do _scrub on N+1, with latest full osdmap
 - leader finishes paxos gather, does refresh_from_paxos()
 -> scrub fails.

Fix this by doing the refresh_from_paxos() at commit time and not when
the paxos round finishes.  We move the refresh out of finish_proposal
and into its own helper, and update all callers accordingly.  This
keeps on-disk state more tightly in sync with in-memory state and
avoids the need for a e.g., kludgey workaround in the scrub code.

We also simplify the bootstrap checks a bit by doing so immediately
and relying on the normal bootstrap paxos reset paths to clean up
any waiters.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/PaxosService: remove redundant check

_active checks is_active() and rescheduled itself already.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/PaxosService: assert not proposing in propose_pending

Signed-off-by: Sage Weil <sage@inktank.com>
mon/PaxosService: consolidate resetting in restart()

We had duplicated code in election_finished() and restart(), and it was
incomplete.  Put it all in reset() only (the mon should have called
restart() long before the election finishes).  Note that we cannot
assert as much in election_finished() because another service may have
just cross-proposed.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: fix RECONNECT_SEQ behavior

Calling handle_ack() here has no effect because we have already
spliced sent messages back into our out queue.  Instead, pull them out
of there and discard.  Add a few assertions along the way.

Signed-off-by: Sage Weil <sage@inktank.com>