osd: make OSD a config observer

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 9230c863b3dc2bdda12c23202682a84c48f070a1)

Conflicts:

	src/osd/OSD.cc
osd: make osd_max_backfills dynamically adjustable

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 101955a6b8bfdf91f4229f4ecb5d5b3da096e160)
os/FileStore: allow filestore_queue_max_{ops,bytes} to be adjusted at runtime

The 'committing' ones too.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit cfe4b8519363f92f84f724a812aa41257402865f)
osd: target transaction size 300 -> 30

Small transactions make pg removal nicer to the op queue.  It also slows
down PG deletion a bit, which may exacerbate the PG resurrection case
until #3884 is addressed.

At least on user reported this fixed an osd that kept failing due to
an internal heartbeat failure.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 1233e8617098766c95100aa9a6a07db1a688e290)
filestore: filestore_queue_max_ops 500 -> 50

Having a deep queue limits the effectiveness of the priority queues
above by adding additional latency.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 40654d6d53436c210b2f80911217b044f4d7643a)
osd: debug msg prio, cost, latency

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit bec96a234c160bebd9fd295df5b431dc70a2cfb3)
osd: use Message::get_cost() function for queueing

The data payload is a decent proxy for cost in most cases, but not all.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e8e0da1a577e24cd4aad71fb94d8b244e2ac7300)
osd: set PULL subop cost to size of requested data

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a1bf8220e545f29b83d965f07b1abfbea06238b3)
osd: add OpRequest flag point when commit is sent

With writeahead journaling in particular, we can get requests that
stay in the queue for a long time even after the commit is sent to the
client while we are waiting for the transaction to apply to the fs.
Instead of showing up as 'waiting for subops', make it clear that the
client has gotten its reply and it is local state that is slow.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit b685f727d4c37a26cb78bd4a04cce041428ceb52)
note puller's max chunk in pull requests

this lets us calculate a cost value
(cherry picked from commit 128fcfcac7d3fb66ca2c799df521591a98b82e05)
common/PrioritizedQueue: buckets -> tokens

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit c549a0cf6fae78c8418a3b4b0702fd8a1e4ce482)
common/PrioritizedQueue: add min cost, max tokens per bucket

Two problems.

First, we need to cap the tokens per bucket.  Otherwise, a stream of
items at one priority over time will indefinitely inflate the tokens
available at another priority.  The cap should represent how "bursty"
we allow a given bucket to be.  Start with 4MB for now.

Second, set a floor on the item cost.  Otherwise, we can have an
infinite queue of 0 cost items that start over queues.  More
realistically, we need to balance the overhead of processing small items
with the cost of large items.  I.e., a 4 KB item is not 1/1000th as
expensive as a 4MB item.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 6e3363b20e590cd9df89f2caebe71867b94cc291)
common/PrioritizedQueue: dump state to Formatter

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 514af15e95604bd241d2a98a97b938889c6876db)
osd: simplify asok to single callback

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 33efe32151e04beaafd9435d7f86dc2eb046214d)
osd: dump op priority queue state via admin socket

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 24d0d7eb0165c8b8f923f2d8896b156bfb5e0e60)
osd: make last state for slow requests more informative

Report on the last event string, and pass in important context for the
op event list, including:

 - which peers were sent sub ops and we are waiting for
 - which pg queue we are delayed by

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a1137eb3e168c2d00f93789e4d565c1584790df0)
osd: refactor ReplicatedPG::do_sub_op

PULL is the only case where we don't wait for active.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 23c02bce90c9725ccaf4295de3177e8146157723)
osd: improve sub_op flag points

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 73a969366c8bbd105579611320c43e2334907fef)
osd: reduce op thread heartbeat default 30 -> 15 seconds

If the thread stalls for 15 seconds, let our internal heartbeat fail.
This will let us internally respond more quickly to a stalled or failing
disk.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 61eafffc3242357d9add48be9308222085536898)
osd: do not reply to ping if internal heartbeat is not healthy

If we find that our internal threads are stalled, do not reply to ping
requests.  If we do this long enough, peers will mark us down.  If we are
only transiently unhealthy, we will reply to the next ping and they will
be satisfied.  If we are unhealthy and marked down, and eventually recover,
we will mark ourselves back up.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ad6b231127a6bfcbed600a7493ca3b66c68484d2)