osd: track per-pg epochs, min

Add some simple tracking so that we can quickly determine what the min
pg osdmap epoch is.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 81e4c47722255ac3d46f701a80e104cc390e766c)
osd: fix pg epoch floor tracking

If you call erase() on a multiset it will delete all instances of a value;
we only want to delete one of them.  Fix this by passing an iterator.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a52a855f6c92b03dd84cd0cc1759084f070a98c2)
osd: prevent pgs from getting too far ahead of the min pg epoch

Bound the range of PG epochs between the slowest and fastest pg
(epoch-wise) with 'osd map max advance'.  This value should be set to
something less than 'osd map cache size' so that the maps we are
processing will be in memory as many PGs advance forward in time in
loose synchrony.

This is part of the solution to #7576.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit cf25bdf6b0090379903981fe8cee5ea75efd7ba0)
osd: add sanity check/warning on a few key configs

Warn when certain config values are set to bad values.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f3ec7d0b23fdee39a34bda7595cd2a79c08daf8a)
osd: introduce simple sleep during scrub

This option is similar to osd_snap_trim_sleep: simply inject an optional
sleep in the thread that is doing scrub work.  This is a very kludgey and
coarse knob for limiting the impact of scrub on the cluster, but can help
until we have a more robust and elegant solution.

Only sleep if we are in the NEW_CHUNK state to avoid delaying processing of
an in-progress chunk.  In this state nothing is blocked on anything.
Conveniently, chunky_scrub() requeues itself for each new chunk.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit c4e8451cc5b4ec5ed07e09c08fb13221e31a7ac6)
common/io_priority: wrap ioprio_set() and gettid()

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a2b49110ef65efd526c3430ad03c988ca9dde768)
common/Thread: allow io priority to be set for a Thread

Ideally, set this before starting the thread.  If you set it after, we
could potentially race with create() itself.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 1b8741022c5a2ebae38905215dadee696433e931)
common/WorkQueue: allow io priority to be set for wq

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dd6badcb5eedfec6748b3e6ca4d46e3b266038f6)

Conflicts:

	src/common/WorkQueue.cc
osd: allow io priority to be set for the disk_tp

The disk_tp covers scrubbing, pg deletion, and snap trimming

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit d9073f486527ca13cdb2774745c4c63c218333ad)
Merge pull request #2203 from ceph/wip-scrub-firefly

backport scrub throttling to firefly

Reviewed-by: Samuel Just <sam.just@inktank.com>