osd/OSDMap: apply primary_affinity to mapping

The behavior is a bit different for replicated and indep/erasure mode.
In the first case, we are rearranging the result.  In the second case,
we can just set the primary argument to the right value.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: add primary_affinity feature bit

Indicate that we support it.  Indicate when an OSDMap requires it.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph_psim: some futzing to test primary_affinity

- map to acting
- count first position, primary

Signed-off-by: Sage Weil <sage@inktank.com>
mon/OSDMonitor: add 'mon osd allow primary affinity' bool option

By default, disallow adjustment of primary affinity unless the user has
opted in by adjusting their monitor config.  This will avoid some user
pain because inadvertantly setting the affinity will prevent older clients
from connecting to and using the cluster.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: remove bad assert

You can have an erasure poool with all CRUSH_ITEM_NONE and primary == -1.
acting is not empty.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: include primary affinity in OSDMap::print

Signed-off-by: Sage Weil <sage@inktank.com>