leveldb: add compact() method

This will compact the entire store; it will be slow!

Signed-off-by: Sage Weil <sage@inktank.com>
mon: --compact argument, config option to compact the store on start

Signed-off-by: Sage Weil <sage@inktank.com>
mon: compact leveldb on bootstrap

This is an opportunistic time to optimize our local data since we are
out of quorum.  It serves as a safety net for cases where leveldb's
automatic compaction doesn't work quite right and lets things get out
of hand.

Anecdotally we have seen stores in excess of 30GB compact down to a few
hundred KB.  And a 9GB store compact down to 900MB in only 1 minute.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: compact leveldb on bootstrap

This is an opportunistic time to optimize our local data since we are
out of quorum.  It serves as a safety net for cases where leveldb's
automatic compaction doesn't work quite right and lets things get out
of hand.

Anecdotally we have seen stores in excess of 30GB compact down to a few
hundred KB.  And a 9GB store compact down to 900MB in only 1 minute.

Signed-off-by: Sage Weil <sage@inktank.com>
leveldb: add compact_prefix method

Signed-off-by: Sage Weil <sage@inktank.com>
mon: add compact_prefix transaction operation

Add a prefix compaction opteration to the transaction that will be
performed after the transaction applies.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: compact PaxosService prefix on trim

Each time we trim a PaxosService, have leveldb compact so that the
space from removed states is reclaimed.

This is probably not optimal if leveldb's heuristics are doing the right
thing, but it currently appears as if they are not.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: compact on trim

Compact the paxos keys when we trim old paxos states.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: enable 'mon compact on trim' by default; trim in larger increments

This resolves the leveldb growth-without-bound problem observed by
mikedawson, and all the badness that stems from it.  Enable this by
default until we figure out why leveldb is not behaving better.

While we are at it, trim more states at a time.  This will make
compaction less frequent, which should help given that there is some
overhead unrelated to the amount of deleted data.

Fixes: #4815
Signed-off-by: Sage Weil <sage@inktank.com>
osd: Rename members and methods related to stat publish

pg_stats_lock to pg_stats_publish_lock
pg_stats_valid to pg_stats_publish_valid
pg_stats_stable to pg_stats_publish
update_stats() to publish_stats_to_osd()
clear_stats() to clear_publish_stats()

Signed-off-by: David Zafman <david.zafman@inktank.com>
osd: read kb stats not tracked?

In read cases track stats in PG::unstable_stats
Include unstable_stats in write_info() and publish_stats_to_osd()
For now this information may not get persisted

fixes: #2209

Signed-off-by: David Zafman <david.zafman@inktank.com>
Set num_rd, num_wr_kb and num_wr in various places that needed it

Signed-off-by: David Zafman <david.zafman@inktank.com>
mon: change leveldb block size to 64K

#leveldb on freenode says > 2MB is nonsense (it might explain the weird
behavior we saw).  Riak tuning guide suggests 256KB for large data block
environments.  Default is 8KB.  64KB seems sane for us.

Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'wip-2209' into next

Reviewed-by: Samuel Just <sam.just@inktank.com>
elector: trigger a mon reset whenever we bump the epoch

We need to call reset during every election cycle; luckily we
can call it more than once. bump_epoch is (by definition!) only called
once per cycle, and it's called at the beginning, so we put it there.

Fixes #4858.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Merge branch 'wip-mon-compact' into next

Reviewed-by: Samuel Just <sam.just@inktank.com>
ObjectCacher: wait for all reads when stopping flusher

Stopping the flusher is essentially the shutdown step for the
ObjectCacher - the next thing is actually destroying it.

If we leave any reads outstanding, when they complete they will
attempt to use the now-destroyed ObjectCacher. This is particularly a
problem with rbd images, since an -ENOENT can instantly complete many
readers, so the upper layers don't wait for the other rados-level
reads of that object to finish before trying to shutdown the cache.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Revert "mon: when electing, be sure acked leaders have new enough stores to lead"

This was somehow broken -- out-of-date leaders were being elected -- and
we've decided smaller band-aids are more appropriate. We don't completely
revert the MMonElection changes, though -- there have been user clusters
running the code which includes these messages so we can't pretend it
never happened. We can make them clearly unused in the code, though.

This reverts commit fcaabf1a22723c571c10d402464071c6405607c0.

Signed-off-by: Greg Farnum <greg@inktank.com>
Revert "mon: update assert for looser requirements"

We reverted the gating by paxos sequences, so now we don't
need to look at them at all.

This reverts commit 1e6f02b337767012aeb387da9582cd7ad5a03084.
Signed-off-by: Greg Farnum <greg@inktank.com>
mon: if we get our own sync_start back, drop it on the floor.

We have timeouts that will clean everything up, and this can happen
in some cases that we've decided are legitimate. Hopefully we'll
be able to do something else later.

Signed-off-by: Greg Farnum <greg@inktank.com>