Reduce ObjectCacher flush overhead

Flush op in ObjectCacher will iterate the whole active object set, each
dirty object also may own several BufferHead. If the object set is large,
it will consume too much time.

Use dirty_bh instead to reduce overhead. Now only dirty BufferHead will
be checked.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Add rbdcache max dirty object option

Librbd will calculate max dirty object according to rbd_cache_max_size, it
doesn't suitable for every case. If user set image order 24, the calculating
result is too small for reality. It will increase the overhead of trim call
which is called each read/write op.

Now we make it as option for tunning, by default this value is calculated.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Wait tx state buffer in flush_set

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
ObjectCacher: fix dirty_or_tx_bh logic in bh_set_state()

The else-if chain here was wrong. Handling dirty or tx buffers and
errors should be in independent conditions.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
ObjectCacher: fix bh_{add,remove} dirty_or_tx_bh accounting

tx buffers need to go on the bh_lru_rest as well, and removing erases
(not inserts) them into dirty_or_tx_bh.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Only write bufferhead when it's dirty

The TX state bh should be skipped because the bh should be inflight. We only
need to write dirty bh. And TX and dirty state bh both should be waited until
flushed.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Merge pull request #2149 from yuyuyu101/wip-flush-set

Fix dup bh_write for TX state bh

Tested-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Original changeset 

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
mds: separate inode recovery queue from MDCache

Refactor to:
* have somewhere to put some logic for doing
  background recovery in future.
* trim a few lines from the oversized MDCache.cc
  whereever we can.

Signed-off-by: John Spray <john.spray@redhat.com>
mds: remove unused purge_prealloc_ino

Signed-off-by: John Spray <john.spray@redhat.com>
mds: make MDS::replay_done clearer

... and add some assertions.

Signed-off-by: John Spray <john.spray@redhat.com>
mds: refactor MDS boot

* Make boot_start private.
* Define boot stages in enum, replace int with type.
* Merge steps 0 and 1, 0 always fell through to 1.
* starting_done was only ever reached by a fall through
  from the previous step, so call it directly from there.
* since fall through is no longer needed, replace switch with
  if/else

Signed-off-by: John Spray <john.spray@redhat.com>
mds: introduce explicit DaemonState instead of int

Signed-off-by: John Spray <john.spray@redhat.com>
mds: handle replaying old format journals

To get back to the reformatting procedure that otherwise
occurs during MDLog::open, introduce an MDLog::reopen call
that MDS can use in the standbyreplay->standby transition
for the special case where the journal is old.

Fixes: #8869

Signed-off-by: John Spray <john.spray@redhat.com>
mds: remove some rogue "using namespace std;"

Signed-off-by: John Spray <john.spray@redhat.com>