osd: delay push ack until commit

Delay sending the push reply until the write commits to disk.  This helps
throttle recovery traffic by ensuring we aren't buffering too many pushes
and starving other IO.  This is important because the op queue and io
queues are not priorities like the msgr queue is), and the network may be
much faster than the disk.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: finish pull op only after commit

Finish the recovery op only after it commits to disk.  This helps throttle
the pace of pulls so that the primary does not get ahead of its own local
storage.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: make OSD a config observer

Signed-off-by: Sage Weil <sage@inktank.com>
osd: make osd_max_backfills dynamically adjustable

Signed-off-by: Sage Weil <sage@inktank.com>
filestore: disable extra committing queue allowance

The motivation here is if there is a problem draining the op queue
during a sync.  For XFS and ext4, this isn't generally a problem: you
can continue to make writes while a syncfs(2) is in progress.  There
are currently some possible implementation issues with btrfs, but we
have not demonstrated them recently.

Meanwhile, this can cause queue length spikes that screw up latency.
During a commit, we allow too much into the queue (say, recovery
operations).  After the sync finishes, we have to drain it out before
we can queue new work (say, a higher priority client request).  Having
a deep queue below the point where priorities order work limits the
value of the priority queue.

Signed-off-by: Sage Weil <sage@inktank.com>
debug recovery reservations
osd: debug msg prio, cost
debug latency too
osd: use Message::get_cost() function for queueing

The data payload is a decent proxy for cost in most cases, but not all.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: set PULL subop cost to size of requested data

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add OpRequest flag point when commit is sent

With writeahead journaling in particular, we can get requests that
stay in the queue for a long time even after the commit is sent to the
client while we are waiting for the transaction to apply to the fs.
Instead of showing up as 'waiting for subops', make it clear that the
client has gotten its reply and it is local state that is slow.

Signed-off-by: Sage Weil <sage@inktank.com>
note puller's max chunk in pull requests

this lets us calculate a cost value
common/PrioritizedQueue: check queues in highest-to-lowest order

If there are multiple queues with sufficient tokens, we currently
always choose the lowest-priority queue.  This seems like the opposite
of what we want, particularly given that dequeue is stateless and will
allow a low-priority queue with surplus tokens to 'starve' a higher-
priority queue.

Signed-off-by: Sage Weil <sage@inktank.com>
common/PrioritizedQueue: buckets -> tokens

Signed-off-by: Sage Weil <sage@inktank.com>
common/PrioritizedQueue: add min cost, max tokens per bucket

Two problems.

First, we need to cap the tokens per bucket.  Otherwise, a stream of
items at one priority over time will indefinitely inflate the tokens
available at another priority.  The cap should represent how "bursty"
we allow a given bucket to be.  Start with 4MB for now.

Second, set a floor on the item cost.  Otherwise, we can have an
infinite queue of 0 cost items that start over queues.  More
realistically, we need to balance the overhead of processing small items
with the cost of large items.  I.e., a 4 KB item is not 1/1000th as
expensive as a 4MB item.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: simplify asok to single callback

Signed-off-by: Sage Weil <sage@inktank.com>
common/PrioritizedQueue: dump state to Formatter

Signed-off-by: Sage Weil <sage@inktank.com>
osd: dump op priority queue state via admin socket

Signed-off-by: Sage Weil <sage@inktank.com>
osd: make last state for slow requests more informative

Report on the last event string, and pass in important context for the
op event list, including:

 - which peers were sent sub ops and we are waiting for
 - which pg queue we are delayed by

Signed-off-by: Sage Weil <sage@inktank.com>
osd: show last few events for slow requests

Signed-off-by: Sage Weil <sage@inktank.com>