osd/PG: trigger scrub via scrub schedule, must_ flags

When a scrub is requested, flag it and move it to the front of the
scrub schedule instead of immediately queuing it.  This avoids
bypassing the scrub reservation framework, which can lead to a heavier
impact on performance.

Signed-off-by: Sage Weil <sage@inktank.com>
osdmap: make replica separate in default crush map configurable

Add 'osd crush chooseleaf type' option to control what the default
CRUSH rule separates replicas across.  Default to 1 (host), and set it
to 0 in vstart.sh.  This captures the desired effect of 7ea5d84f but
avoids breaking vstart, and gives users of mkcephfs (or any other
cluster creation tool) flexibility.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: move scrub schedule random backoff to seperate helper

Separate this from the load check, which will soon vary dependon on the
PG.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: remove useless osd_scrub_min_interval check

This was already a no-op: we don't call PG::scrub_sched() unless it has
been osd_scrub_max_interval seconds since we last scrubbed.  Unless we
explicitly requested in, in which case we don't want this check anyway.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: change scrub min/max thresholds

The previous 'osd scrub min interval' was mostly meaningless and useless.
Meanwhile, the 'osd scrub max interval' would only trigger a scrub if the
load was sufficiently low; if it was high, the PG might *never* scrub.

Instead, make the 'min' what the max used to be.  If it has been more than
this many seconds, and the load is low, scrub.  And add an additional
condition that if it has been more than the max threshold, scrub the PG
no matter what--regardless of the load.

Note that this does not change the default scrub interval for less-loaded
clusters, but it *does* change the meaning of existing config options.

Fixes: #3786
Signed-off-by: Sage Weil <sage@inktank.com>