osd/OSDMap: add osd_primary_affinity fields, accessors, encoding

Signed-off-by: Sage Weil <sage@inktank.com>
mon/OSDMonitor: add 'osd primary-affinity ...' command

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: apply primary_affinity to mapping

The behavior is a bit different for replicated and indep/erasure mode.

Pass the pps back to the _pg_to_osds() caller so that we can use it as the
seed for the primary affinity.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/OSDMap: add primary_affinity feature bit

Indicate that we support it.  Indicate when an OSDMap requires it.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph_psim: some futzing to test primary_affinity

- map to acting
- count first position, primary

Signed-off-by: Sage Weil <sage@inktank.com>
mon/OSDMonitor: add 'mon osd allow primary affinity' bool option

By default, disallow adjustment of primary affinity unless the user has
opted in by adjusting their monitor config.  This will avoid some user
pain because inadvertantly setting the affinity will prevent older clients
from connecting to and using the cluster.

Signed-off-by: Sage Weil <sage@inktank.com>