leveldb: add compact() method

This will compact the entire store; it will be slow!

Signed-off-by: Sage Weil <sage@inktank.com>
mon: --compact argument, config option to compact the store on start

Signed-off-by: Sage Weil <sage@inktank.com>
mon: compact leveldb on bootstrap

This is an opportunistic time to optimize our local data since we are
out of quorum.  It serves as a safety net for cases where leveldb's
automatic compaction doesn't work quite right and lets things get out
of hand.

Anecdotally we have seen stores in excess of 30GB compact down to a few
hundred KB.  And a 9GB store compact down to 900MB in only 1 minute.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: compact leveldb on bootstrap

This is an opportunistic time to optimize our local data since we are
out of quorum.  It serves as a safety net for cases where leveldb's
automatic compaction doesn't work quite right and lets things get out
of hand.

Anecdotally we have seen stores in excess of 30GB compact down to a few
hundred KB.  And a 9GB store compact down to 900MB in only 1 minute.

Signed-off-by: Sage Weil <sage@inktank.com>
leveldb: add compact_prefix method

Signed-off-by: Sage Weil <sage@inktank.com>
mon: add compact_prefix transaction operation

Add a prefix compaction opteration to the transaction that will be
performed after the transaction applies.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: compact PaxosService prefix on trim

Each time we trim a PaxosService, have leveldb compact so that the
space from removed states is reclaimed.

This is probably not optimal if leveldb's heuristics are doing the right
thing, but it currently appears as if they are not.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: compact on trim

Compact the paxos keys when we trim old paxos states.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: enable 'mon compact on trim' by default; trim in larger increments

This resolves the leveldb growth-without-bound problem observed by
mikedawson, and all the badness that stems from it.  Enable this by
default until we figure out why leveldb is not behaving better.

While we are at it, trim more states at a time.  This will make
compaction less frequent, which should help given that there is some
overhead unrelated to the amount of deleted data.

Fixes: #4815
Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'wip-mon-compact' into next

Reviewed-by: Samuel Just <sam.just@inktank.com>