osd: do not make pg_pool_t incompat when hit_sets are enabled

If we enable HitSet tracking, the OSD needs to know this, but clients do
not care.  Setting the compat version is too heavyweight as it locks out
older kernels (*any* currents, currently) who are unaffected by the new
fields.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph_test_rados_api_tier: scrub while cache tier is missing clones

Trigger a scrub to verify that we can handle a cache tier that is missing
some clones.  We rely on the test harness to notice the error, and we do
not confirm that the scrub happened.  In practice this is plenty of time,
however.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not assert on clone_size mismatch

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: improve clone vs head checking

- notice when we are missing a clone (that isn't at the end of the list)
- notice when we are missing a clone on the last object in the scrub map
- do not assert when we are missing a clone

There is still more we could do to improve this (like noticing one missing
clone but still checking the others), but we'll leave that aside for just
a moment...

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: tolerate missing clones in cache pools

A few cases:

- As we are working through the list, if we see a clone that is lower than
  the next one we were expecting, we should be able to skip them.
- If we see a head, we can skip all of the rest of the clones.
- If we get to the end and next_clone was set, we can ignore it.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/OSDMonitor: prevent setting hit_set unless all OSDs support it

We are using OSD_CACHEPOOL as a proxy for the support for the tiering
OSDMap infrastructure.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/OSDMonitor: require OSD_CACHEPOOL feature before using tiering features

The OSDs need to support this feature before we allow users to turn it
on.  This is similar to what the erasure pool support does.

Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'wip-7875'

Reviewed-by: Samuel Just <sam.just@inktank.com>
Merge pull request #1547 from ceph/wip-cache-scrub

osd: improve scrub checks on clones; tolerate missing clones on cache pools

Fixes: #7885
Reviewed-by: Samuel Just <sam.just@inktank.com>
doc: fix typos in tiering dev doc

Signed-off-by: Loic Dachary <loic@dachary.org>
Merge pull request #1549 from dachary/wip-doc

doc: fix typos in tiering dev doc
Merge pull request #1555 from ceph/wip-7835

ReplicatedPG::make_writeable: fill in ssc on clone

Reviewed-by: Sage Weil <sage@inktank.com>
Merge pull request #1554 from ceph/wip-7828

ReplicatedPG:: s/_delete_head/_delete_oid, adjust head_exists iff is_hea...

Reviewed-by: Sage Weil <sage@inktank.com>
Merge pull request #1553 from ceph/wip-7874

ReplicatedPG: disable clone subsets for cache pools

Reviewed-by: Sage Weil <sage@inktank.com>
Merge remote-tracking branches 'origin/wip-sam-testing2', 'upstream/wip-7659', 'origin/wip-7904', 'origin/wip-7858', 'origin/wip-7837' and 'origin/wip-7831' into wip-sam-testing