mon/OSDMonitor: improve no-op cache_mode set check

If we have a pending pool value but the cache_mode hasn't changed, this is
still a no-op (and we don't need to block).

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
osd/osd_types: add pg_pool_t FLAG_COMPLETE_CLONES

Set a flag on the pg_pool_t when we change cache_mode NONE.  This
is because object promotion may promote heads without all of the clones,
and when we switch the cache_mode back those objects may remain.  Do
this on any cache_mode change (to or from NONE) to capture legacy
pools that were set up before this flag existed.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/ReplicatedPG: do not complain about missing clones when INCOMPLETE_CLONES is set

When scrubbing, do not complain about missing cloens when we are in a
caching mode *or* when the INCOMPLETE_CLONES flag is set.  Both are
indicators that we may be missing clones and that that is okay.

Fixes: #8882
Signed-off-by: Sage Weil <sage@redhat.com>
osd/ReplicatedPG: observed INCOMPLETE_CLONES when doing clone subsets

During recovery, we can clone subsets if we know that all clones will be
present.  We skip this on caching pools because they may not be; do the
same when INCOMPLETE_CLONES is set.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/ReplicatedPG: observe INCOMPLETE_CLONES in is_present_clone()

We cannot assume that just because cache_mode is NONE that we will have
all clones present; check for the absense of the INCOMPLETE_CLONES flag
here too.

Signed-off-by: Sage Weil <sage@redhat.com>