crush: eliminate CRUSH_MAX_SET result size limitation

This is only present to size the temporary scratch arrays that we put on
the stack.  Let the caller allocate them as they wish and remove the
limitation.

Signed-off-by: Sage Weil <sage@inktank.com>
crush: return CRUSH_ITEM_UNDEF for failed placements with indep

For firstn mode, if we fail to make a valid placement choice, we just
continue and return a short result to the caller.  For indep mode, however,
we need to make the position stable, and return an undefined value on
failed placements to avoid shifting later results to the left.

Signed-off-by: Sage Weil <sage@inktank.com>
crush: use breadth-first search for indep mode

Signed-off-by: Sage Weil <sage@inktank.com>
crush/CrushWrapper: add dump_tree()

This is based on the OSDMap::print_tree(), but is a bit simpler because
we do not have up/down information at this level.

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: add note about r in recursive choose

Signed-off-by: Sage Weil <sage@inktank.com>
crush: add a few unit tests for INDEP mode

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: strip firstn conditionals out of crush_choose, rename

Now that indep is handled by crush_choose_indep, rename crush_choose to
crush_choose_firstn and remove all the conditionals.  This ends up
stripping out *lots* of code.

Note that it *also* makes it obvious that the shenanigans we were playing
with r' for uniform buckets were broken for firstn mode.  This appears to
have happened waaaay back in commit dae8bec9 (or earlier)... 2007.

Signed-off-by: Sage Weil <sage@inktank.com>
osd, crush: add 'erasure' pool/pg type

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: do not shift result when removing nonexistent osds

If it is a replicated pool, remove and shift to the left.  For erasure
pools, replace nonexistent items with CRUSH_ITEM_NONE.

Signed-off-by: Sage Weil <sage@inktank.com>
mon, crush: add mode to "osd crush rule create-simple ..."

Add a mode (firstn or indep) to the create-simple command.  Make it
optional and default to firstn (for compatiblity and simplicity).

Note: a "default=..." option for mon commands would be easier.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/osd_types: pg_pool_t: fix /// -> ///< comments

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: clarify numrep vs endpos

Pass numrep (the width of the result) separately from the number of results
we want *this* iteration.  This makes things less awkward when we do a
recursive call (for chooseleaf) and want only one item.

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: pass parent r value for indep call

Pass down the parent's 'r' value so that we will sample different values in
the recursive call when the parent tries multiple times.  This avoids doing
useless work (calling multiple times and trying the same values).

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: new SET_CHOOSE_LEAF_TRIES command

Explicitly control the number of sample attempts, and allow the number of
tries in the recursive call to be explicitly controlled via the rule. This
is important because the amount of time we want to spend looking for a
solution may be rule dependent (e.g., higher for the wide indep pool than
the rep pools).

(We should do the same for the other tunables, by the way!)

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: fix up the indep tests

Fix indentation.
Simplify+fix the changed vs moved calculation.
Use the new SET_CHOOSE_LEAF_TRIES command.

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: apply chooseleaf_tries to firstn mode too

Parameterize the attempts for the _firstn choose method, and apply the
rule-specified tries count to firstn mode as well.  Note that we have
slightly different behavior here than with indep:

 If the firstn value is not specified for firstn, we pass through the
 normal attempt count.  This maintains compatibility with legacy behavior.
 Note that this is usually *not* actually N^2 work, though, because of the
 descend_once tunable.  However, descend_once is unfortunately *not* the
 same thing as 1 chooseleaf try because it is only checked on a reject but
 not on a collision.  Sigh.

 In contrast, for indep, if tries is not specified we default to 1
 recursive attempt, because that is simply more sane, and we have the
 option to do so.  The descend_once tunable has no effect for indep.

Signed-off-by: Sage Weil <sage@inktank.com>
crush/mapper: add SET_CHOOSE_TRIES rule step

Since we can specify the recursive retries in a rule, we may as well also
specify the non-recursive tries too for completeness.

Signed-off-by: Sage Weil <sage@inktank.com>
crush/CrushWrapper: set chooseleaf_tries to 5 for 'simple' indep rules

When making a generic indep rule, set the recursive retry to 5.  This gives
better overall results.

Signed-off-by: Sage Weil <sage@inktank.com>
crush/CrushCompiler: [de]compile set_choose[leaf]_tries rule step

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: fix feature calculation for CACHEPOOL

We need to include the faeture in the mask.

Signed-off-by: Sage Weil <sage@inktank.com>