\( \langle { e ^ { \prime } , \langle { v , z } \rangle } \rangle \), \( e p o c h ( z ) = e ^ { \prime } \), and z succeeds all zxid values previously broadcast in \( e ^ { \prime } \).

Step \( \ell . \).3.2: Upon receiving acknowledgments from a quo-rum of followers to a given proposal \( \langle { e ^ { \prime } , \langle { v , z } \rangle } \rangle \), the leader sends a commit \( C O M M T ( e ^ { \prime } , \langle v , z \rangle ) \) to all follow-ers.

Step \( f \).3.1: Follower f initially invokes \( r e a d y ( e ^ { \prime } ) \) if it is leading.

Step \( f \).3.2: Follower f accepts proposals from \( \ell \) following reception order and appends them to \( h _ { f } \) .

Step \( f \).3.3: Follower f commits a transaction \( \langle v , z \rangle \) by invoking \( a b d e l i \nu e r ( \langle v , z \rangle ) \) once it receives \( C O M M T ( e ^ { \prime } , \langle v , z \rangle ) \) and it has committed all transactions \( \langle v ^ { \prime } , z ^ { \prime } \rangle \) such that \( \langle v ^ { \prime } , z ^ { \prime } \rangle \in h _ { f } \) , \( z ^ { \prime } \prec _ { z } z \).

Step \( \ell . \).3.3: Upon receiving a CEPOCH(e) message from follower f while in Phase 3, leader \( \ell \) proposes back \( N E W E P O C H ( e ^ { \prime } ) \) and \( N E W L E A D E R ( e ^ { \prime } , I _ { e ^ { \prime } } \circ \beta _ { e ^ { \prime } } ) \).

Step \( \ell \).3.4: Upon receiving an acknowledgement from f of the \( N E W L E A D E R ( e ^ { \prime } , I _ { e ^ { \prime } } \circ \beta _ { e ^ { \prime } } ) \) proposal, it sends a commit message to f. Leader \( \ell \) also makes \( Q \leftarrow Q \cup \{ f \} \).

✷

Note that a realization of this protocol does not re-quire sending complete histories with \( A C K – E ( f . a , h _ { f } ) \) and \( N E W L E A D E R ( e ^ { \prime } , I _ { e ^ { \prime } } ) \), only the last transaction identifier in the history followed by missing transactions. It is also possible to omit values in acknowledgements and commit messages in Phase 3 to reduce the size of messages.

The following section discusses the Zab protocol in more detail along with some implementation aspects.

# V. ZAB IN DETAIL

In our implementation of Zab, a Zab process can be looking for a leader (ELECTION state), following (FOLLOWING state), or leading (LEADING state). When a process starts, it enters the ELECTION state. While in this state the process tries to elect a new leader or become a leader. If the process finds an elected leader, it moves to the FOLLOWING state and begins to follow the leader. Processes in the FOLLOWING state are followers. If the process is elected leader, it moves to the LEADING state and becomes the leader. Given that a pro-cess that leads also follows, states LEADING and FOLLOW-ING are not exclusive. A follower transitions to ELECTION if it detects that the leader has failed or relinquished leadership, while a leader transitions to ELECTION once it observes that it no longer has a quorum of followers supporting its leadership.

The basic delivery protocol is similar in spirit to two phase commit [16] without aborts. The primary picks values to broadcast in FIFO order and creates a transaction \( \langle v , z \rangle \). Upon receiving a request to broadcast a transaction, a leader pro-poses \( \langle e , \langle v , z \rangle \rangle \) following the order of zxid of the transactions. The followers accept the proposal and acknowledge by sending an \( A C K ( e , \langle v , z \rangle ) \) back to the leader. Note that a follower does not send the acknowledgment back until it writes the proposal to local stable storage. When a quorum of processes have accepted the proposal, the leader issues a \( C O M M I T ( e , \langle v , z \rangle ) \). When a process receives a commit message for a proposal \( \langle e , \langle v , z \rangle \rangle \), the process delivers all undelivered proposals with zxid \( z ^ { \prime } \), \( z ^ { \prime } \prec _ { z } z \).

Co-locating the leader and the primary on the same process has practical advantages. The primary-backup scheme we use requires that at most one process at a time is able to generate updates that can be incorporated into the service state. A primary propagates state updates using Zab, which in turn requires a leader to initiate proposals. Leader and primary correspond to different functionality, but they share a common requirement: election. By co-locating them, we do not need separate elections for primary and leader. Also important is the fact that calls to broadcast transactions are local when they are co-located. We consequently co-locate leader and primary.

# A. Establishing a new leader

Leader election occurs in two stages. First, we run a leader election algorithm that outputs a new process as the leader. We can use any protocol that, with high probability, chooses a process that is up and that a quorum of processes selects. This property can be fulfilled by an \( \Omega \) failure detector [17].

Figure 5 shows the events for both the leader and followers when establishing a new leader. An elected leader does not become established for a given epoch e until it completes Phase 2, in which it successfully achieves consensus on the proposal history and on itself as the leader of e. We define a established leader and a established epoch as follows:

Definition V.1. (Established leader) A leader \( \ell _ { e } \) is estab-lished for epoch e if the \( N E W L E A D E R ( e , I _ { e } ) \) proposal of \( \ell _ { e } \) is accepted by a quorum Q of followers.

Definition V.2. (Established epoch) An epoch e is established if there is an established leader for e.

Once a process determines that it is a prospective leader by inspecting the output of the leader election algorithm, it starts a new iteration in Phase 1. It initially collects the latest epoch of a quorum of followers Q, proposes a later epoch, and collects the latest epoch and highest zxid of each of the followers in Q. The leader completes Phase 1 once it selects the history from a follower f with latest epoch and highest zxid in a \( A C K { - } E ( f . a , h _ { f } ) \). These steps are necessary to guarantee that once the prospective leader completes Phase 1, none of the followers in Q accept proposals from earlier epochs. Given that the history of a follower can be arbitrarily long, it is not efficient to send the entire history in a \( A C K { - } E ( f . a , h _ { f } ) \). The last zxid of a follower is sufficient for the prospective leader to determine if it needs to copy transactions from any given follower, and only copies missing transactions.

In Phase 2, the leader proposes itself as the leader of the new epoch and sends a \( N E W L E A D E R ( e , I _ { e } ) \) proposal, which contains the initial history of the new epoch. As with \( A C K – E ( f . a , h _ { f } ) \), it is not necessary to send the complete initial history, but instead only the transactions missing. A leader becomes established once it receives the acknowledgments to the new leader proposal from a quorum of followers, at which point it commits the new proposal. Followers deliver

Fig. 5. Zab protocol summary

the initial history and complete Phase 2 once they receive a commit message for the new leader proposal.

One interesting optimization is to use a leader election primitive that selects a leader that has the latest epoch and has accepted the transaction with highest zxid among a quorum of processes. Such a leader provides directly the initial history of the new epoch.

# B. Leading

A leader proposes operations by queuing them to all con-nected followers. To achieve high throughput and low latency, the leader has a steady stream of proposals to the followers. By the channel properties, we guarantee that followers receive proposals in order. In our implementation, we use TCP connec-tions to exchange messages between processes. If a connection to a given follower closes, then the proposals queued to the connection are discarded and the leader considers the corresponding follower down.

Detecting crashes through connections closing was not a suitable choice for us. Timeout values for a connection can be of minutes or even hours, depending on the operating system configuration and the state of the connection. To mutually detect crashes in a fine-grained and convenient manner, avoiding operating system reconfiguration, leader and followers exchange periodic heartbeats. If the leader does not receive heartbeats from a quorum of followers within a timeout interval, the leader renounces leadership of the epoch, and transitions to the ELECTION state. Once it elects a leader, it starts a new iteration of the algorithm, and starts a new iteration of the protocol proceeding to Phase 1.

# C. Following

When a follower emerges from leader election, it connects to the leader. To support a leader, a follower f acknowledges its new epoch proposal, and it only does so if the new epoch proposed is later than f.p. A follower only follows one leader at a time and stays connected to a leader as long as it receives heartbeats within a timeout interval. If there is an interval with no heartbeat or the TCP connection closes, the follower abandons the leader, transitions to ELECTION and proceeds to Phase 1 of the algorithm.

Figure 5 shows the protocol a follower executes to support a leader. The follower sends its current epoch f.a in a current epoch message (CEPOCH(f.a)) to the leader. The leader sends a new epoch proposal (NEWEPOCH(e)) once it receives a current epoch message from a quorum Q of followers. The new proposed epoch e must be greater than the current epoch of any follower in Q. A follower acknowledges the new epoch proposal with its latest epoch and highest zxid, which the leader uses to select the initial history for the new epoch.

In Phase 2, a follower acknowledges the new leader pro-posal \( ( N E W L E A D E R ( e , I _ { e } ) \)) by setting its \( f , G \) value to e and accepting the transactions in the initial history. Note that once a follower accepts a new epoch proposal for an epoch e, it does not send an acknowledgement for any other new epoch proposal for the same epoch e. This property guarantees that no two processes can become established leaders for the same epoch e. Once it receives a commit message from the leader for the new leader proposal, the follower completes Phase 2 and proceeds to Phase 3. In Phase 3, the follower receives new proposals from the leader. A follower adds new proposals to its history and acknowledges them. It delivers these proposals when it receives commit messages from the leader.

Note that a follower and a leader follow the recovery protocol both when a new leader is emerging and when a follower connects to an established leader. If the leader is already established, the \( N E W L E A D E R ( e , I _ { e } ) \) proposal has already been committed so any acknowledgements for the \( N E W L E A D E R ( e , I _ { e } ) \) proposal are ignored.

# D. Liveness

Zab requires the presence of a leader to propose and commit operations. To sustain leadership, a leader process \( \ell \) needs to be able to send messages to and receive messages from followers. In fact, process \( \ell \) requires that a quorum of followers are up and select \( \ell \) as their leader to maintain its leadership. This requirement follows closely the properties