Monitor: use a single static accessor for getting CompatSet features off disk

This way we don't need to maintain two working code paths.

Signed-off-by: Greg Farnum <greg@inktank.com>
Monitor: introduce a function that translates quorum features into disk features

After an election, we call apply_quorum_to_compatset_features(). It translates
from the quorum features into monitor disk state features we care about and
adds them to the disk store. This prevents an older daemon from starting up
on a store it doesn't understand.

While strictly speaking we don't need to add the EC feature until we create
an EC pool, all monitors which speak the new OSDMap encoding also support EC
pools, and having more than one feature makes the pattern clearer.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSDMonitor: do not allow creation of EC pools if the quorum can't handle it

Signed-off-by: Greg Farnum <greg@inktank.com>
Monitor: add a function to get the required quorum features from the local compatset

We will use this to avoid deferring to people who can't read our store.

Signed-off-by: Greg Farnum <greg@inktank.com>
Elector: ignore messages from mons without required feature capabilities

We maintain a list of required_features which the other monitor's features
must supply. This starts out at 0 and is initialized from the monitor's
list of features whenever we start electing.
Despite the scary sound of "just ignore it", this is safe: the monitor
will only record features as required once a quorum has formed in which
every monitor supports them. After that happens, monitors which do not
support those features will be unable to read the whole mon store/understand
the pg reports/whatever else, so letting them into the quorum would be buggy
behavior.
So if we ignore a monitor, it will not be able to start nor join
an election round with anybody who was in our quorum -- that is, the
ignored monitor cannot form a separate quorum. By ignoring it here, we
also prevent it from endlessly calling elections against the real
quorum.

Unfortunately there is no way to communicate to old monitors that they
cannot join the quorum -- there are no existing messages for that purpose,
and eg adding a new op to the MMonElection message will just cause it
to crash, which we don't want to do either.

Fixes: #7215

Signed-off-by: Greg Farnum <greg@inktank.com>
Elector: send an OP_NAK MMonElection to old peers who support it

Only new monitors support receiving OP_NAK from a peer without crashing, but
when we add new required features in the future, our monitors can accept
an OP_NAK message which tells them what features they're missing. Then they
will print out an error message and shut down.
(Unfortunately, doing a clean shutdown from here would require a lot of
infrastructure, so we just call exit(0).)

Signed-off-by: Greg Farnum <greg@inktank.com>
mon: do not use CEPH_FEATURES_ALL for things that touch the disk

We want to encode with our quorum_features instead. Remaining uses of
CEPH_FEATURES_ALL are:
1) when the Elector is sharing its supported features
2) in a MonMap function which is used by monmaptool
3) In the Monitor for winning a standalone election, for ephemeral data,
  and for doing mkfs (when we necessarily don't have quorum_features).
4) When doing ceph-mon --inject-monmap (we don't persist the quorum_features
  to disk, so we can't use them here).
5) in MMonElection, for doing the default monmap encoding (which is
  re-encoded later if the final features don't match CEPH_FEATURES_ALL).
6) As the default encode features for the OSDMap (the monitor always
  supplies quorum_features instead).

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge pull request #1148 from ceph/wip-7215-quorum-features

Add monitor quorum features

Reviewed-by: Sage Weil <sage@inktank.com>