mon: Paxos: create function to queue a proposal for later proposal

Instead of directly appending a proposal onto the proposal queue, we
should instead use 'queue_proposal()'. This allows us to not only avoid
duplicating the queuing code, but most importantly it provides a single
place where we can make any future changes we may deem necessary.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: assert on finish_proposal() if mon is not the leader

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: trim through Paxos

Instead of directly modifying the store whenever we want to trim our Paxos
state, we should do it through the Paxos itself, proposing the trim to the
quorum and commit it once accepted.

This enforces three major invariants that we will be able to leverage later
on during the store synchronization:

 1) The Leader will set the pace for trimming across the system. No one
    will trim their state unless they are committing the value proposed by
    the Leader;

 2) Following (1), the whole system (at least all the mons that are part of
    the quorum) will trim at the same time. There will be no divergent
    states due to trimming on different monitors.

 3) Each trim will be kept as a transaction in the Paxos' store, allowing
    us to obtain a consistent state during synchronization, by shipping the
    Paxos versions to the other monitor and applying them. We could have
    incurred in an inconsistent state if the trimming was happening at any
    time and without being logged; but by going through Paxos this concern
    no longer applies.

The trimming itself may be triggered each time a proposal finishes, which
is the time at which we know we have committed a new version on the store.

It shall be triggered iff we are sure we have enough versions on the store
to fill the gap of any monitor that might become alive and still hasn't
drifted enough to require synchronization. Roughly speaking, we will check
if the number of available versions is higher than 'paxos_max_join_drift'.

Furthermore, we added a new option, 'paxos_trim_tolerance', so we are able
to avoid trimming every single time the above condition is met -- which
would happen every time we trimmed a version, and then proposed a new one,
and then we would trim it again, etc. So, just tolerate a couple of commits
before trimming again.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: MonitorDBStore: add clear() function to remove all keys from the store

We also provide a finer-grained function that should receive a set of
strings containing exceptions. Basically, any key in the store with the
same prefix as any value in this exceptions set will not be removed.

This is useful to clear the Paxos' and the services' state from the store
without messing the Monitor-specific keys, which if removed would make the
monitor unusable.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: PaxosService: cleanup some old commented out lines

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: PaxosService: add trim through Paxos support on services

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
test: cli: mon-store-tool: test trims

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: remove 'stashed version' references/usage

We haven't been using stashed versions since we unified the paxos among
all services, but we left these references lingering. We are now removing
them because they are nothing more than kludge in the source, and being
unused can lead to confusion.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: remove the 'force' argument on trim_to() function

This was used by the old slurp mechanism. We are getting rid of slurp
and we won't be needing to force trims.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: update Peon's first/last committed version after state change

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>