Fig. 5. Zab protocol summary

the initial history and complete Phase 2 once they receive a commit message for the new leader proposal.

One interesting optimization is to use a leader election primitive that selects a leader that has the latest epoch and has accepted the transaction with highest zxid among a quorum of processes. Such a leader provides directly the initial history of the new epoch.

# B. Leading

A leader proposes operations by queuing them to all con-nected followers. To achieve high throughput and low latency, the leader has a steady stream of proposals to the followers. By the channel properties, we guarantee that followers receive proposals in order. In our implementation, we use TCP connec-tions to exchange messages between processes. If a connection to a given follower closes, then the proposals queued to the connection are discarded and the leader considers the corresponding follower down.

Detecting crashes through connections closing was not a suitable choice for us. Timeout values for a connection can be of minutes or even hours, depending on the operating system configuration and the state of the connection. To mutually detect crashes in a fine-grained and convenient manner, avoiding operating system reconfiguration, leader and followers exchange periodic heartbeats. If the leader does not receive heartbeats from a quorum of followers within a timeout interval, the leader renounces leadership of the epoch, and transitions to the ELECTION state. Once it elects a leader, it starts a new iteration of the algorithm, and starts a new iteration of the protocol proceeding to Phase 1.

# C. Following

When a follower emerges from leader election, it connects to the leader. To support a leader, a follower f acknowledges its new epoch proposal, and it only does so if the new epoch proposed is later than f.p. A follower only follows one leader at a time and stays connected to a leader as long as it receives heartbeats within a timeout interval. If there is an interval with no heartbeat or the TCP connection closes, the follower abandons the leader, transitions to ELECTION and proceeds to Phase 1 of the algorithm.

Figure 5 shows the protocol a follower executes to support a leader. The follower sends its current epoch f.a in a current epoch message (CEPOCH(f.a)) to the leader. The leader sends a new epoch proposal (NEWEPOCH(e)) once it receives a current epoch message from a quorum Q of followers. The new proposed epoch e must be greater than the current epoch of any follower in Q. A follower acknowledges the new epoch proposal with its latest epoch and highest zxid, which the leader uses to select the initial history for the new epoch.

In Phase 2, a follower acknowledges the new leader pro-posal \( ( N E W L E A D E R ( e , I _ { e } ) \)) by setting its \( f , G \) value to e and accepting the transactions in the initial history. Note that once a follower accepts a new epoch proposal for an epoch e, it does not send an acknowledgement for any other new epoch proposal for the same epoch e. This property guarantees that no two processes can become established leaders for the same epoch e. Once it receives a commit message from the leader for the new leader proposal, the follower completes Phase 2 and proceeds to Phase 3. In Phase 3, the follower receives new proposals from the leader. A follower adds new proposals to its history and acknowledges them. It delivers these proposals when it receives commit messages from the leader.

Note that a follower and a leader follow the recovery protocol both when a new leader is emerging and when a follower connects to an established leader. If the leader is already established, the \( N E W L E A D E R ( e , I _ { e } ) \) proposal has already been committed so any acknowledgements for the \( N E W L E A D E R ( e , I _ { e } ) \) proposal are ignored.

# D. Liveness

Zab requires the presence of a leader to propose and commit operations. To sustain leadership, a leader process \( \ell \) needs to be able to send messages to and receive messages from followers. In fact, process \( \ell \) requires that a quorum of followers are up and select \( \ell \) as their leader to maintain its leadership. This requirement follows closely the properties