osdmaptool: --test-map-pgs-dump

Signed-off-by: Sage Weil <sage@redhat.com>
crush: recalculate straw scalers during a reweight

The crushtool --reweight function triggers a fresh calculation of bucket
weights so that they are always the sum of the item weights.  In the
straw bucket case, the weights were updated but the corresponding straw
scalers were not being recalculated.  The result is that there was not
effect on placement in adjusted buckets until the next time a bucket item's
weight was adjusted.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
crushtool/CrushTester: output utilization even with 1 batch

Signed-off-by: Sage Weil <sage@redhat.com>
crush/CrushTester: add new --show-mappings option

This makes --show-utilization and --show-statistics usable.

Signed-off-by: Sage Weil <sage@redhat.com>
crush/CrushWrapper: dump chooseleaf_vary_r with other tunables

Signed-off-by: Sage Weil <sage@redhat.com>
crush: add straw_calc_version tunable

It doesn't do anything, yet.

Signed-off-by: Sage Weil <sage@redhat.com>
crush: default to straw_calc_version 1

Stick with bobtail tunables where it affects compatibility.  Use v1 of
straw_calc, though, since that does not, and we want the best for new
clusters.

Signed-off-by: Sage Weil <sage@redhat.com>
crush: pass crush_map * to various builder methods

In particular, we will need it for crush_calc_straw().

Signed-off-by: Sage Weil <sage@redhat.com>
crush/builder: break out new version 1 of crush_calc_straw

No change, yet.

Signed-off-by: Sage Weil <sage@redhat.com>
crush: fix distortion of straw scalers by 0-weight items

The presence of a 0-weight item in a straw bucket should have no effect
on the placement of other items.  Add a test validating that and fix
crush_calc_straw() to fix the distortion.

Note that this effects the *precalculation* of the straw bucket inputs and
does not effect the actually mapping process given a compiled or encoded
CRUSH map, and only when straw_calc_version == 1 (i.e., the admin opted in
to the new behavior).

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
crush: fix crush_calc_straw() scalers when there are duplicate weights

The straw bucket was originally tested with uniform weights and with a
few more complicated patterns, like a stair step (1,2,3,4,5,6,7,8,9).  And
it worked!

However, it does not behave with a pattern like
 1, 2, 2, 3, 3, 4, 4

Strangely, it does behave with
 1, 1, 2, 2, 3, 3, 4, 4

and more usefully it does behave with
 1, 2, 2.001, 3, 3.001, 4, 4.001

That is, the logic that explicitly copes with weights that are duplicates
is broken.

The fix is to simply remove the special handling for duplicate weights --
it isn't necessary and doesn't work correctly anyway.

Add a test that compares the mapping result of  [1, 2, 2, 3, 3, ...] with
[1, 2, 2.001, 3, 3.001, ...] and verifies that the difference is small.
With the fix, we get .00012, whereas the original implementation gets
.015.

Note that this changes the straw bucket scalar *precalculated* values that
are encoded with the map, and only when the admin opts into the new behavior.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
mon: add 'osd crush {get,set}-tunable <name> [value]' commands

For now, just add the straw_calc_version tunable.

Signed-off-by: Sage Weil <sage@redhat.com>
crush/builder: a note about the original crush_calc_straw()

Signed-off-by: Sage Weil <sage@redhat.com>
crush: move default bucket choice into CrushWrapper helper

Signed-off-by: Sage Weil <sage@redhat.com>
crush: add a straw2 bucket type

This is an improved straw bucket that correctly avoids any data movement
between items A and B when neither A nor B's weights are changed.  Said
differently, if we adjust the weight of item C (including adding it anew
or removing it completely), we will only see inputs move to or from C,
never between other items in the bucket.

Notably, there is not intermediate scaling factor that needs to be
calculated.  The mapping function is a simple function of the item weights.

Unfortunately, we need a natural log, which is expensive.  We use a lookup
table here.

Signed-off-by: Sage Weil <sage@redhat.com>
crush: introduce CRUSH_V4 feature bit

Introduce a feature bit that reflects whether straw2 buckets are in use.
Clients need to support them before they can be used!

Signed-off-by: Sage Weil <sage@redhat.com>
crush: add allowed_bucket_types tunable

This tunable is a bitmask indicating which bucket types are allowed.
For now, the only purpose is to affect get_default_bucket_type(), which
will try to pick a type that is supported, with a preference for straw2 or
straw buckets.  In the future, we likely want something a bit more
sophisticated that reflects whether the bucket is expected to be fixed-size
or not.

We could have added a default_bucket_type field to accomplish this, but I
think that would prove to be too limiting in the future, and this
accomplishes the same thing.

Note that if the admin selects the 'hammer' tunables, that means clients
and servers will support straw2 and that will be the preferred choice. The
default is still bobtail tunables, which means new clusters created on
hammer will not use straw2 just yet.

Signed-off-by: Sage Weil <sage@redhat.com>