qa: rbd map-snapshot-io: udevadm settle

Udev runs blkid on device close, thwarting any rbd unmap that
immediately follows use of the device.  Explicitly settle for now.

See #4183.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: restrict pool size to 1..10

See: #4159
Signed-off-by: Sage Weil <sage@inktank.com>
v0.57
osd: pull requeued requests off one at a time

Pull items off the finished queue on at a time.  In certain cases, an
event may result in new items betting added to the finished queue that
will be put at the *front* instead of the back.  See latest incarnation
of #2947.

Note that this is a significant changed in behavior in that we can
theoretically starve if an event keeps resulting in new events getting
generated.  Beware!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
osd: requeue pg waiters at the front of the finished queue

We could have a sequence like:

- op1
- notify
- op2

in the finished queue.  Op1 gets put on waiting_for_pg, the notify
creates the pg and requeues op1 (and the end), op2 is handled, and
finally op1 is handled.  That breaks ordering; see #2947.

Instead, when we wake up a pg, queue the waiting messages at the front
of the dispatch queue.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
os/FileStore: check replay guard on src for collection rename

This avoids a problematic sequence like:

     - rename A/ -> B/
     - remove B/1...100
     - destroy B/
     - create A/
     - write A/101...
     <crash>
     - replay A/ -> B/
     - remove B/1...100  (fails but tolerated)
     - destroy B/        (fails with ENOTEMPTY)

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
PG: remove weirdness log for last_complete < log.tail

In the case of a divergent object prior to log.tail,
last_complete may end up before log.tail.

Backport: bobtail
Fixes #4174
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
doc/release-notes: v0.57

Signed-off-by: Sage Weil <sage@inktank.com>
qa: sample test for new replication tests

Signed-off-by: Joe Buck <jbbuck@gmail.com>
testing: updating hadoop-internal test

Small tweaks to the hadoop-internal test
to better use existing environment varaibles.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Noah Watkins <noahwatkins@gmail.com>
Merge branch 'next'
Merge branch 'master' of https://github.com/ceph/ceph
test/bufferlist: fix warning

In file included from test/bufferlist.cc:31:0:
../src/gtest/include/gtest/gtest.h: In function ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = unsigned int, T2 = int]’:
../src/gtest/include/gtest/gtest.h:1300:30: instantiated from ‘static testing::AssertionResult testing::internal::EqHelper::Compare(const char*, const char*, const T1&, const T2&) [with T1 = unsigned int, T2 = int, bool lhs_is_null_literal = false]’
test/bufferlist.cc:1604:227: instantiated from here
warning: ../src/gtest/include/gtest/gtest.h:1263:3: comparison between signed and unsigned integer expressions [-Wsign-compare]

Signed-off-by: Sage Weil <sage@inktank.com>
Merge remote-tracking branch 'gh/wip-4159'
qa: mon/pool_ops.sh: fix pool tests

The '! command' doesn't fail properly, even with -e, in bash (wtf!).

Also, the last pool deletion command succeeds because the pool
'--yes-i-really-really-mean-it' doesn't exist.  So drop that test.

Signed-off-by: Sage Weil <sage@inktank.com>
qa: mon/pool_ops.sh: test pool set size

Signed-off-by: Sage Weil <sage@inktank.com>
osd: introduce HASHPSPOOL pool flag, feature to avoid overlapping pg placements

The existing code will overlay the placement of PGs from pools because
it simply adds the ps to the pool as the CRUSH input.  That means that
the layout/placement for pg 0.10 == 1.9 == 2.8 == 3.7 == 4.6 == ...,
which is not optimal.

Instead, use hash(ps, poolid).  The avoids the initial problem of
the sequence being adjacent to other pools.  It also avoids the (small)
possibility that hash(poolid) will drop us somewhere in the output
number space where our sequence of outputs overlaps with some other
pool; instead, out output sequence will be a fully random (for a well-
behaved hash).

Use the multi-input hash functions used by CRUSH for this.

Default to the legacy behavior for now.  We won't enable this until
deployed systems and kernel code catch up.

Fixes: #4128
Signed-off-by: Sage Weil <sage@inktank.com>
mon: move OSDMap feature bit calculation into an OSDMap method

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: note OSDHASHPSPOOL feature when pool FLAG_HASHPSPOOL is set

This allows the osd and mon to enforce feature bits on their connections.

Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'wip-pool'

Reviewed-by: Samuel Just <sam.just@inktank.com>