osd: introduce simple sleep during scrub

This option is similar to osd_snap_trim_sleep: simply inject an optional
sleep in the thread that is doing scrub work.  This is a very kludgey and
coarse knob for limiting the impact of scrub on the cluster, but can help
until we have a more robust and elegant solution.

Only sleep if we are in the NEW_CHUNK state to avoid delaying processing of
an in-progress chunk.  In this state nothing is blocked on anything.
Conveniently, chunky_scrub() requeues itself for each new chunk.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit c4e8451cc5b4ec5ed07e09c08fb13221e31a7ac6)
common/io_priority: wrap ioprio_set() and gettid()

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 705713564bebd84ad31cc91698311cf2fbd51a48)

Conflicts:
	src/common/Makefile.am
common/Thread: allow io priority to be set for a Thread

Ideally, set this before starting the thread.  If you set it after, we
could potentially race with create() itself.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 01533183e7455b713640e001962339907fb6f980)
common/WorkQueue: allow io priority to be set for wq

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 5e4b3b1f1cb870f39fc7cfb3adeae93e078d9057)

Conflicts:
	src/common/WorkQueue.cc
osd: fix filestore perf stats update

Update the struct we are about to send, not the (unlocked!) one we will
send the next time around.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 4afffb4a10a0bbf7f2018ef3ed6b167c7921e46b)
ceph-disk: partprobe before settle when preparing dev

Two users have reported this fixes a problem with using --dmcrypt.

Fixes: #6966
Tested-by: Eric Eastman <eric0e@aol.com>
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0f196265f049d432e399197a3af3f90d2e916275)
rgw: allocate enough space for bucket instance id

Fixes: #8608
Backport: dumpling, firefly
Bucket instance id is a concatenation of zone name, rados instance id,
and a running counter. We need to allocate enough space to account zone
name length.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit d2e86a66ca55685e04ffbfaa58452af59f381277)
Merge pull request #2014 from ceph/wip-scrub-dumpling

osd: scrub priority updates for dumpling

Reviewed-by: Loic Dachary <loic@dachary.org>
qa/workunits/suites/fsx.sh: don't use zero range

Zero range is not supported by cephfs.

Fixes: #8542
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 2dec8a810060f65d022c06e82090b4aa5ccec0cb)
Revert "qa/workunits/suites/fsx.sh: don't use zero range"

This reverts commit 583e6e3ef7f28bf34fe038e8a2391f9325a69adf.

We're using a different fsx source, which doesn't support the
same options as our git-based one does.

Signed-off-by: Greg Farnum <greg@inktank.com>
librbd: skip zeroes when copying an image

This is the simple coarse-grained solution, but it works well in
common cases like a small base image resized with a bunch of empty
space at the end. Finer-grained sparseness can be copied by using rbd
{export,import}-diff.

Fixes: #6257
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 824da2029613a6f4b380b6b2f16a0bd0903f7e3c)
Support latest qemu iotest code

Modified qemu-iotests workunit script to check for versions
that use the latest qemu (currently only Trusty).  Limit the
tests to those that are applicable to rbd.

Fixes: 7882
Signed-off-by: Warren Usui <warren.usui@inktank.com>
(cherry picked from commit 606e725eb5204e76e602d26ffd113e40af2ee812)
Use new git mirror for qemu-iotests

Fixes: 8191
Signed-off-by: Warren Usui <warren.usui@inktank.com>
(cherry picked from commit ddf37d903f826f3e153d8009c716780453b68b05)
rgw: don't try to wait for pending if list is empty

Fixes: #8846
Backport: firefly, dumpling

This was broken at ea68b9372319fd0bab40856db26528d36359102e. We ended
up calling wait_pending_front() when pending list was empty.
This commit also moves the need_to_wait check to a different place,
where we actually throttle (and not just drain completed IOs).

Reported-by: Sylvain Munaut <s.munaut@whatever-company.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit f9f2417d7db01ecf2425039539997901615816a9)
rgw: improve delmited listing of bucket

If found a prefix, calculate a string greater than that so that next
request we can skip to that. This is still not the most efficient way to
do it. It'll be better to push it down to the objclass, but that'll
require a much bigger change.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit e6cf618c257f26f97f60a4c1df1d23a14496cab0)
rgw: add NextMarker param for bucket listing

Partially fixes #8858.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 924686f0b6593deffcd1d4e80ab06b1e7af00dcb)
rgw: account common prefixes for MaxKeys in bucket listing

To be more in line with the S3 api. Beforehand we didn't account the
common prefixes towards the MaxKeys (a single common prefix counts as a
single key). Also need to adjust the marker now if it is pointing at a
common prefix.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 82d2d612e700f94a0bb2d9fb7555abf327be379b)
rgw: list extra objects to set truncation flag correctly

Otherwise we end up returning wrong truncated value, and no data on the
next iteration.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit dc417e477d4ad262885c6b5f5987cf06d63b159d)
rgw: dump prefix unconditionally

As part of issue #8858, and to be more in line with S3, dump the Prefix
field when listing bucket even if bucket is empty.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d7209c11251d42227608bc54cc69232ef62ffe80)

Conflicts:
	src/rgw/rgw_rest_s3.cc
utf8: export encode_utf8() and decode_utf8()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 49fc68cf8c3122c878ea9503c9c74d7046bc9c6f)