ceph/ceph — pull request #1236 by jecluis (2014-02-13 17:58:06+00:00)
Title: mon: Monitor: initial quorum must contain all mons on mon_initial_members

If the monitor has never joined a quorum, and it has a populated
mon_initial_members list in the conf, then it should expect to only be
able to form quorum with all the monitors from that list.

The only exception to this would be the case where a monitor is freshly
added to the cluster through 'ceph mon add' but its conf file contains
the mon_initial_members list anyway. In this case, the monitor will be
able to join the cluster even if a portion of those monitors are offline,
as long as there's an already formed quorum or another monitor shares an
updated version of the monmap.

This patch is meant to fix a scenario as seen in #7212.

For an initial deployment of three monitors (a, b and c):

- mon.a and mon.b start
- mon.a probes mon.b and mon.c
- mon.c starts
- mon.b probes mon.a and mon.c
- mon.c probes mon.a and mon.b
- mon.b replies to mon.a and mon.c
- mon.a replies to mon.b
- mon.c starts election with mon.b
- mon.b and mon.a start an election
- mon.b defers to mon.a
- mon.a and mon.b form a quorum
- mon.c bootstraps, probes, gets updated monmap with mon.a
- mon.c starts an election with mon.a and mon.b
- mon.a ignores mon.c's election: not in the monmap because mon.a never
actually probed mon.c, thus didn't get mon.c into the monmap.
- mon.b notices an old election epoch from mon.c, assumes mon.c just
started, starts new election with mon.c and mon.a; defers to mon.a
- mon.c never gets into the quorum, so it just keeps triggering elections
and causing havoc in the cluster.

By limiting the initial quorum formation, we guarantee that all monitors
will eventually probe and get replies from all other monitors, and will
end up with the same monmap (unless due to configuration error), thus
guaranteeing that this and similar scenarios are not triggered.

This behavior seems to have been present since the introduction of the
mon_initial_members list and initial mommap generation mechanism. However,
only now did we start noticing this being triggered by ceph-deploy suites
in QA. It is difficult to trigger it manually with ceph-deploy, and even
more so to trigger with a manual deployment of monitors -- so that may be
the reason why it hasn't been noticeable in the past.

Fixes: 7212

Signed-off-by: Joao Eduardo Luis