crush: add chooseleaf_vary_r tunable

The current crush_choose_firstn code will re-use the same 'r' value for
the recursive call.  That means that if we are hitting a collision or
rejection for some reason (say, an OSD that is marked out) and need to
retry, we will keep making the same (bad) choice in that recursive
selection.

Introduce a tunable that fixes that behavior by incorporating the parent
'r' value into the recursive starting point, so that a different path
will be taken in subsequent placement attempts.

Note that this was done from the get-go for the new crush_choose_indep
algorithm.

This was exposed by a user who was seeing PGs stuck in active+remapped
after reweight-by-utilization because the up set mapped to a single OSD.

Signed-off-by: Sage Weil <sage@inktank.com>
crush: add infrastructure around new chooseleaf_vary_r tunable

- encoding
- feature bit
- decompile/compile

Signed-off-by: Sage Weil <sage@inktank.com>
crush: add SET_CHOOSELEAF_VARY_R step

This lets you adjust the vary_r tunable on a per-rule basis.

Signed-off-by: Sage Weil <sage@inktank.com>
crush: add infrastructure around SET_CHOOSELEAF_VARY_R rule step/command

This will let you vary the vary_r tunable on a per-rule basis.

Signed-off-by: Sage Weil <sage@inktank.com>
crushtool: new cli tests for the vary-r tunable

These illustrate the variation in mapping results as the vary_r tunable
is adjusted.  In particular, for a legacy cluster, we can use a value like
4 to minimize the number of PGs that move while now getting the valid
mappings that we need.

The crushmap here comes from a customer who was seeing a bad mapping for
certain pgs after some OSDs were reweighted by utilization.

Signed-off-by: Sage Weil <sage@inktank.com>
crush: add firefly tunables baseline test

This is a user's map that gives different results when the vary_r tunable
is adjusted.

Signed-off-by: Sage Weil <sage@inktank.com>