Reduce ObjectCacher flush overhead

Flush op in ObjectCacher will iterate the whole active object set, each
dirty object also may own several BufferHead. If the object set is large,
it will consume too much time.

Use dirty_bh instead to reduce overhead. Now only dirty BufferHead will
be checked.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Add rbdcache max dirty object option

Librbd will calculate max dirty object according to rbd_cache_max_size, it
doesn't suitable for every case. If user set image order 24, the calculating
result is too small for reality. It will increase the overhead of trim call
which is called each read/write op.

Now we make it as option for tunning, by default this value is calculated.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Wait tx state buffer in flush_set

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
ObjectCacher: fix dirty_or_tx_bh logic in bh_set_state()

The else-if chain here was wrong. Handling dirty or tx buffers and
errors should be in independent conditions.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
ObjectCacher: fix bh_{add,remove} dirty_or_tx_bh accounting

tx buffers need to go on the bh_lru_rest as well, and removing erases
(not inserts) them into dirty_or_tx_bh.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Only write bufferhead when it's dirty

The TX state bh should be skipped because the bh should be inflight. We only
need to write dirty bh. And TX and dirty state bh both should be waited until
flushed.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Merge pull request #2149 from yuyuyu101/wip-flush-set

Fix dup bh_write for TX state bh

Tested-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Original changeset 

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>