mon: reimplement a simpler sync

The previous sync implementation was highly stateful and very complex.
This made it very hard to understand and to debug, and there were bugs
still lurking in the timeout code (at least).

Replace it with something much simpler:

 - sync providers are almost stateless.  they keep an iterator, identified
   by a unique cookie, that times out in a simple way.
 - sync requesters sync from whomever they fancy.  namely anyone with newer
   committed paxos state.

There are a few extra fields that might allow sync continuation later, but
this is complex and not necessary at this point.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: config min paxos txns to keep separately

We were using paxos_max_join_drift to control the minimum number of
paxos transactions to keep around.  Instead, make this explicit, and
separate from the join drift.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: remove unnecessary trim enable/disable

The sync no longer cares if we trim Paxos versions as we go, as long as we
don't trim so fast that we fall behind between GET_CHUNK messages, which
we can consider a tuning problem.

Remove this extra complexity!

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: move consistent check into Paxos::init()

Signed-off-by: Sage Weil <sage@inktank.com>
mon: add --force-sync startup option

Signed-off-by: Sage Weil <sage@inktank.com>
mon/MonitorDBStore: drop unused single prefix synchronizer

Signed-off-by: Sage Weil <sage@inktank.com>
mon: apply payload size to incremental paxos states, not just the key scan

This applies a uniform message size (soft limit) during sync.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: implement simple 'scrub' command

Compare all keys within the sync'ed prefixes across members of the quorum
and compare the key counts and CRC for inconsistencies.

Currently this is a one-shot inefficient hammer.  We'll want to make this
work in chunks before it is usable in production environments.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: rev the internal mon protocol

This captures the new sync and scrub.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: factor out sync_clear_store_state() helper

Signed-off-by: Sage Weil <sage@inktank.com>
mon: only clear in_sync flag on mon start; do not discard other state

There is other important stuff here, like the monmap.  Don't throw it
away!

Signed-off-by: Sage Weil <sage@inktank.com>
mon: drop single-use is_sync_on_going() check

Signed-off-by: Sage Weil <sage@inktank.com>
mon: preserve last_committed_floor across sync

Add a paranoid check to prevent us from forgetting how far ahead our
last_committed was when we sync.  This prevents an i'll-timed sync
from allowing paxos to warp back in time.

See: #5256
Signed-off-by: Sage Weil <sage@inktank.com>