opened
fiibbb
username_0
I'm new to etcd. I've been looking into this [example](https://github.com/etcd-io/etcd/blob/master/clientv3/concurrency/example_election_test.go) of using etcd for leader election. I'm a bit confused with what this example illustrates. It seems like a leader has to call `Resign` to concede the leadership to another agent participating the campaign. In the case of leader loosing network connection to etcd cluster, wouldn't this render the system as hanging because the old leader can not "give up" leadership, thus no new leader can win the campaign?
What kind of safety guarantee does the leader election primitives provide?
issue
created
hexfusion
username_1
@username_0 leader election is started when a randomized election timer expires, no leader interaction is required to give up leadership. Actually for an election timer to timeout the node would not receive an empty ApendEntries RPC which serves as a heartbeat from the leader. So if the leader goes down a new election would happen automatically once the first nodes election timer times out. At that point, that node will become candidate at the next term votes for itself and then sends out RequestVote RPC's to the other members. If it recieves votes from the majority of the cluster it will then become leader for that term.

Here is a nice visual of the process: http://thesecretlivesofdata.com/raft/ and more info on raft including the paper. https://raft.github.io/
comment
created
fiibbb
username_0
@username_1 Thanks a lot for the explanation. I believe we are talking about two subtly different things though.

In my original question, I'm confused with how clients of etcd may use etcd for leader election among client agents, as opposed to leader election among etcd nodes. I'm familiar with how leader election within Raft works, but I'm less familiar with using consensus systems for leader election among client agents.

In the [example](https://github.com/etcd-io/etcd/blob/master/clientv3/concurrency/example_election_test.go) I linked, this is done via the concurrency API that comes with etcd client. In this example, leader is calling `Resign` to proactively relinquish leadership ([here)(https://github.com/etcd-io/etcd/blob/master/clientv3/concurrency/example_election_test.go#L78)].

What this tells me is that either 1), the leader would not be able to relinquish leadership in the case of network partition (the `Resign` call never makes to etcd), thus other nodes can not "win" their campaign, in which case the system stops making progress as leader is now offline. Or 2), if the etcd cluster decides that the leader is not longer online (eg via noticing the lease owned by the leader expires), and further allows a different node participating in the campaign becomes new leader, then there's almost guaranteed to be a split-brain since the original leader may not know that a second leader has been elected.

Thinking about this again, it seems like leader election primitives provided by consensus systems such as etcd or zookeeper can never prevent split-brain. Raft internally doesn't prevent split-brain either -- more than one Raft node can "think" itself as the leader at any given moment, Raft simply guarantees that when a split-brain happen, the leaders not in the quorum can not commit their writes.

I may be missing something important here. It seems counter-intuitive that leader election primitives are either prone to stop making progress or prone to split-brain in the case of network partition. If someone could either confirm that this is the case, or that I'm missing something, that would be very helpful.
comment
created
thrawn01
username_2
Hi @username_1 I've recently been digging into the `concurrency.Election` library myself.

Leader elections use `concurrency.Session` to create and hold a lease with the etcd server. The `Session` runs a goroutine that keeps the lease alive as long as connectivity is not interrupted longer than the lease TTL. (Which is configurable).  When the call to `election.Campaign()` succeeds in wining the election it uses the lease id to write the entry, such that if a connection interruption occurs and the client is unable to keep the lease alive for longer than the configured TTL a new client will be elected leader.

Unfortunately the election library does not monitor the session liveliness and it's up to the client implementor to watch for a `session.Done()` signal which indicates a new session should be recreated. 

Hope this helps!
comment
created
pjhdld
username_3
In the case of a network failure between the leader and other nodes, the leader will also enter the election timeout situation, and then become candidate election, the other two nodes enter the election timeout and elect the leader
comment
created
dhanaraj0
username_4
Did you get an answer to original question? If yes can you post that here, it might be helpful to others as well.
comment
created
fiibbb
username_0
I have since worked on quite a few other leader election systems, and I think I have an answer to my original question.

Typically leader election systems DO NOT prevent split brain from happening. In fact split-brain is common when leader election systems switch between leaders. This is by design.

For systems that have to guard against split-brain, the workload depending on leader (agents that receive instructions from leader) must also use the fencing-token technique to ensure that the instructions are coming from a valid leader, instead of a stale one. This is explained in more detail in Martin Kleppmann's popular article: [How to do distributed locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html)
comment