\( \langle { e ^ { \prime } , \langle { v , z } \rangle } \rangle \), \( e p o c h ( z ) = e ^ { \prime } \), and z succeeds all zxid values previously broadcast in \( e ^ { \prime } \).

Step \( \ell . \).3.2: Upon receiving acknowledgments from a quo-rum of followers to a given proposal \( \langle { e ^ { \prime } , \langle { v , z } \rangle } \rangle \), the leader sends a commit \( C O M M T ( e ^ { \prime } , \langle v , z \rangle ) \) to all follow-ers.

Step \( f \).3.1: Follower f initially invokes \( r e a d y ( e ^ { \prime } ) \) if it is leading.

Step \( f \).3.2: Follower f accepts proposals from \( \ell \) following reception order and appends them to \( h _ { f } \) .

Step \( f \).3.3: Follower f commits a transaction \( \langle v , z \rangle \) by invoking \( a b d e l i \nu e r ( \langle v , z \rangle ) \) once it receives \( C O M M T ( e ^ { \prime } , \langle v , z \rangle ) \) and it has committed all transactions \( \langle v ^ { \prime } , z ^ { \prime } \rangle \) such that \( \langle v ^ { \prime } , z ^ { \prime } \rangle \in h _ { f } \) , \( z ^ { \prime } \prec _ { z } z \).

Step \( \ell . \).3.3: Upon receiving a CEPOCH(e) message from follower f while in Phase 3, leader \( \ell \) proposes back \( N E W E P O C H ( e ^ { \prime } ) \) and \( N E W L E A D E R ( e ^ { \prime } , I _ { e ^ { \prime } } \circ \beta _ { e ^ { \prime } } ) \).

Step \( \ell \).3.4: Upon receiving an acknowledgement from f of the \( N E W L E A D E R ( e ^ { \prime } , I _ { e ^ { \prime } } \circ \beta _ { e ^ { \prime } } ) \) proposal, it sends a commit message to f. Leader \( \ell \) also makes \( Q \leftarrow Q \cup \{ f \} \).

✷

Note that a realization of this protocol does not re-quire sending complete histories with \( A C K – E ( f . a , h _ { f } ) \) and \( N E W L E A D E R ( e ^ { \prime } , I _ { e ^ { \prime } } ) \), only the last transaction identifier in the history followed by missing transactions. It is also possible to omit values in acknowledgements and commit messages in Phase 3 to reduce the size of messages.

The following section discusses the Zab protocol in more detail along with some implementation aspects.

# V. ZAB IN DETAIL

In our implementation of Zab, a Zab process can be looking for a leader (ELECTION state), following (FOLLOWING state), or leading (LEADING state). When a process starts, it enters the ELECTION state. While in this state the process tries to elect a new leader or become a leader. If the process finds an elected leader, it moves to the FOLLOWING state and begins to follow the leader. Processes in the FOLLOWING state are followers. If the process is elected leader, it moves to the LEADING state and becomes the leader. Given that a pro-cess that leads also follows, states LEADING and FOLLOW-ING are not exclusive. A follower transitions to ELECTION if it detects that the leader has failed or relinquished leadership, while a leader transitions to ELECTION once it observes that it no longer has a quorum of followers supporting its leadership.

The basic delivery protocol is similar in spirit to two phase commit [16] without aborts. The primary picks values to broadcast in FIFO order and creates a transaction \( \langle v , z \rangle \). Upon receiving a request to broadcast a transaction, a leader pro-poses \( \langle e , \langle v , z \rangle \rangle \) following the order of zxid of the transactions. The followers accept the proposal and acknowledge by sending an \( A C K ( e , \langle v , z \rangle ) \) back to the leader. Note that a follower does not send the acknowledgment back until it writes the proposal to local stable storage. When a quorum of processes have accepted the proposal, the leader issues a \( C O M M I T ( e , \langle v , z \rangle ) \). When a process receives a commit message for a proposal \( \langle e , \langle v , z \rangle \rangle \), the process delivers all undelivered proposals with zxid \( z ^ { \prime } \), \( z ^ { \prime } \prec _ { z } z \).

Co-locating the leader and the primary on the same process has practical advantages. The primary-backup scheme we use requires that at most one process at a time is able to generate updates that can be incorporated into the service state. A primary propagates state updates using Zab, which in turn requires a leader to initiate proposals. Leader and primary correspond to different functionality, but they share a common requirement: election. By co-locating them, we do not need separate elections for primary and leader. Also important is the fact that calls to broadcast transactions are local when they are co-located. We consequently co-locate leader and primary.

# A. Establishing a new leader

Leader election occurs in two stages. First, we run a leader election algorithm that outputs a new process as the leader. We can use any protocol that, with high probability, chooses a process that is up and that a quorum of processes selects. This property can be fulfilled by an \( \Omega \) failure detector [17].

Figure 5 shows the events for both the leader and followers when establishing a new leader. An elected leader does not become established for a given epoch e until it completes Phase 2, in which it successfully achieves consensus on the proposal history and on itself as the leader of e. We define a established leader and a established epoch as follows:

Definition V.1. (Established leader) A leader \( \ell _ { e } \) is estab-lished for epoch e if the \( N E W L E A D E R ( e , I _ { e } ) \) proposal of \( \ell _ { e } \) is accepted by a quorum Q of followers.

Definition V.2. (Established epoch) An epoch e is established if there is an established leader for e.

Once a process determines that it is a prospective leader by inspecting the output of the leader election algorithm, it starts a new iteration in Phase 1. It initially collects the latest epoch of a quorum of followers Q, proposes a later epoch, and collects the latest epoch and highest zxid of each of the followers in Q. The leader completes Phase 1 once it selects the history from a follower f with latest epoch and highest zxid in a \( A C K { - } E ( f . a , h _ { f } ) \). These steps are necessary to guarantee that once the prospective leader completes Phase 1, none of the followers in Q accept proposals from earlier epochs. Given that the history of a follower can be arbitrarily long, it is not efficient to send the entire history in a \( A C K { - } E ( f . a , h _ { f } ) \). The last zxid of a follower is sufficient for the prospective leader to determine if it needs to copy transactions from any given follower, and only copies missing transactions.

In Phase 2, the leader proposes itself as the leader of the new epoch and sends a \( N E W L E A D E R ( e , I _ { e } ) \) proposal, which contains the initial history of the new epoch. As with \( A C K – E ( f . a , h _ { f } ) \), it is not necessary to send the complete initial history, but instead only the transactions missing. A leader becomes established once it receives the acknowledgments to the new leader proposal from a quorum of followers, at which point it commits the new proposal. Followers deliver