mon: Paxos: create function to queue a proposal and use it

Instead of creating a proposal and add it to the queue directly on
propose_new_value(), we now use a function that does just that. This will
become useful when we use it on trim_to() to queue a proposal to trim the
Paxos state.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: do not allow a peon to trim the Paxos versions

We are going ahead and make the trimming a Paxos proposal. We cannot allow
a Peon to be responsible for his own trimming, as it should come from the
Leader in proposal form.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: assert on finish_proposal() if mon is not the leader

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: trim through Paxos

Instead of directly modifying the store whenever we want to trim our Paxos
state, we should do it through the Paxos itself, proposing the trim to the
quorum and commit it once accepted.

This enforces three major invariants that we will be able to leverage later
on during the store synchronization:

 1) The Leader will set the pace for trimming across the system. No one
    will trim their state unless they are committing the value proposed by
    the Leader;

 2) Following (1), the whole system (at least all the mons that are part of
    the quorum) will trim at the same time. There will be no divergent
    states due to trimming on different monitors.

 3) Each trim will be kept as a transaction in the Paxos' store, allowing
    us to obtain a consistent state during synchronization, by shipping the
    Paxos versions to the other monitor and applying them. We could have
    incurred in an inconsistent state if the trimming was happening at any
    time and without being logged; but by going through Paxos this concern
    no longer applies.

The trimming itself may be triggered each time a proposal finishes, which
is the time at which we know we have committed a new version on the store.

It shall be triggered iff we are sure we have enough versions on the store
to fill the gap of any monitor that might become alive and still hasn't
drifted enough to require synchronization. Roughly speaking, we will check
if the number of available versions is higher than 'paxos_max_join_drift'.

Furthermore, we added a new option, 'paxos_trim_tolerance', so we are able
to avoid trimming every single time the above condition is met -- which
would happen every time we trimmed a version, and then proposed a new one,
and then we would trim it again, etc. So, just tolerate a couple of commits
before trimming again.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: MonitorDBStore: add clear() function to remove all keys from the store

We also provide a finer-grained function that should receive a set of
strings containing exceptions. Basically, any key in the store with the
same prefix as any value in this exceptions set will not be removed.

This is useful to clear the Paxos' and the services' state from the store
without messing the Monitor-specific keys, which if removed would make the
monitor unusable.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: PaxosService: cleanup some old commented out lines

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: PaxosService: add trim through Paxos support on services

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
test: cli: mon-store-tool: test trims

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: remove 'stashed version' references/usage

We haven't been using stashed versions since we unified the paxos among
all services, but we left these references lingering. We are now removing
them because they are nothing more than kludge in the source, and being
unused can lead to confusion.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: remove the 'force' argument on trim_to() function

This was used by the old scrub mechanism. We are getting rid of the scrub
and we won't be needing to force trims.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: update Peon's first/last committed version after state change

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
messages: MMonSync: Monitor Synchronization message (WIP)

This commit message will be updated with the full description once this is
no longer WIP. It is mostly done, but there are certainly things that will
end up going away and other things that will be introduced.

(update as of 08/10/12: This design is mostly complete; will change the
commit message as soon as we are sure nothing is missing)

msg: Message: Add msg type id for MMonSync

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Paxos: add support to enable/disable trimming

This support is essential for the Monitor's store synchronization
mechanisms.

We must revisit this patch, since we have yet to make 'trim_enable()'
trim the Paxos state if it is needed.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: Add initial support for synchronization (WARNING: WIP)

This is very on-going work, and I've actually stopped working on it for the
time being while addressing other pieces of this puzzle. And I feel I can't
emphasize this enough: this is on-going work.

There is a lot of stuff missing, some other stuff was completely left in
a limbo of half-done/half-undone, and it will certainly NOT WORK.

Also, there are a couple of things that are to be replaced, some other
stuff will be stripped away.

(update as of 08/10/12: the Monitor class compiles; will not work as the
'handle_sync()' function is commented out)

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: messaging for synchronizing two monitor stores

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>