# P4CEMaker: automated hardware acceleration of consensus protocols

Paul Breuil, Baptiste Lepers

► To cite this version:

Paul Breuil, Baptiste Lepers. P4CEMaker: automated hardware acceleration of consensus protocols. ICDCS 2025 - IEEE 45th International Conference on Distributed Computing Systems, Jul 2025, Glasgow, United Kingdom. pp.681-691, 10.1109/ICDCS63083.2025.00072. hal-05317064

HAL Id: hal-05317064

https://inria.hal.science/hal-05317064v1

Submitted on 16 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# P4CEMaker: automated hardware acceleration of consensus protocols

Paul Breuil*†, Baptiste Lepers†

*École polytechnique, France. †Inria, France.

**Abstract**—P4CEMaker is a novel system designed to semi-automatically accelerate existing RDMA-based consensus protocols through the use of a programmable switch.

Central to the design of P4CEMaker is the insight that, despite the diverse algorithmic approaches employed by consensus protocols (fault detection, leader election, ...), the protocols fundamentally rely on a shared set of networking operations, such as the scattering and the gathering of values, which can be accelerated by programmable switches.

P4CEMaker is implemented in two components. The first is a dynamic analysis tool that automatically detects the aforementioned network operations and gives the developer precise information, in the form of a call graph, to understand where and how they are executed in a consensus protocol's code. The second is a versatile hardware acceleration library, enabling the execution of these operations in hardware with minimal code modifications.

We used P4CEMaker to accelerate four different consensus protocols, achieving up to 2 times performance improvement in around a day of work per protocol.

***Index Terms***—rdma, consensus, programmable switch, smr, tofino

## I. INTRODUCTION

Consensus is a cornerstone of distributed systems, and minimizing its overhead is important for performance and efficiency. Most low-overhead consensus protocols leverage the remote direct memory access (RDMA) protocol [1], [38], [14], [23], and recent work further increase the efficiency through the use of programmable switches [12].

Using programmable switches to accelerate protocols allows reaching remarkably low latency and high throughput but requires non-trivial development efforts. For instance, P4CE [12], which took the endeavour to accelerate the Mu protocol [1] using programmable switches, necessitated a one-year development period. The effort was divided between understanding the logic of the existing software implementation of Mu and writing libraries to accelerate the exchange of RDMA messages at the programmable switch level.

This paper explores a semi-automated approach to accelerate RDMA-based consensus protocols using programmable switches. Key to our approach is the realization that most consensus protocols share logical network operations, such as value multicasting or response counting, which can be automatically identified and accelerated at the switch level.

Our approach consists of two key components. The first component uses dynamic analysis. It finds the code sections of existing consensus protocols that are responsible for the multicast and counting of consensus messages. The second

component is a configurable library that allows a developer to leverage a programmable switch to multicast RDMA packets to a set of replicas, and to count their answers. Using these two components, a developer can analyze the code base of an existing consensus protocol and patch it to be accelerated by a programmable switch.

We implement these two components in a tool that we call P4CEMaker. We demonstrate the efficiency of P4CEMaker by analysing and patching the code base of four existing consensus protocols. We reproduce the results manually achieved by P4CE [12], using P4CEMaker to analyze and patch Mu [1]. We also accelerate the Sift [23], Hermes [22] and Velos [15] protocols. The four protocols implement widely different consensus logic (e.g., leader-based vs. leaderless, identical nodes vs. dissaggregated architecture) and various types of network requests, demonstrating the generality of our approach. Each protocol required around a day to analyze and patch, with an average of 60 lines of code changed per protocol. The accelerated protocols offer up to 2x higher throughput than their un-accelerated counterparts.

The main contributions of this work are:

* A methodology to find bricks of common network logic between consensus protocols.

* The automated acceleration of these bricks through the use of a programmable switch.

* The acceleration of four existing network protocols: Mu [1], Sift [23], Velos [15] and Hermes [22].

The rest of the paper is organized as follows. We provide some background on RDMA, consensus protocols and programmable switches in Section II. We present the design of P4CEMaker in Section III and its implementation in Section IV. We then discuss related works in Section V. Finally, evaluation results are discussed in Section VI before concluding in Section VII.

## II. BACKGROUND

We provide background on the underlying concepts and technologies that P4CEMaker builds upon, such as RDMA, consensus protocols, and programmable switches.

### A. Remote Direct Memory Access

Below, we introduce RDMA concepts useful to understand the studied protocols, their differences, and the mechanisms used for hardware acceleration. InfiniBand [16] provides a complete RDMA specification and implementation, supported
by numerous vendors [5], [17], [33]. We refer to the InfiniBand
specification for the rest of this paper.

**Transport Mode.** The protocols we study use various transport modes. Mu, Velos and Sift use RDMA Reliable connection mode. Using reliable connections, a source server can directly read and write from the memory of a destination server, without involving the CPU of the destination server. Reliable connections do not natively support multicasting data. Hermes uses RDMA Unreliable Datagrams. Unreliable datagram supports multicasting, although direct memory access is not possible anymore.

**Different requests.** RDMA requests are transmitted through a bidirectional communication channel known as a Queue Pair (QP). In Reliable Connection mode, servers associate virtual address ranges with a QP, enabling these ranges to be directly accessed by remote servers. Access to these registered memory regions is controlled through a remote key (R_key), which serves as a capability token granting permission to perform RDMA operations on the specified memory range.

Mu and Sift use RDMA READ and WRITE commands to directly read and write data from remote servers. Velos uses Compare and Swap (CAS) operations to atomically modify the memory of the remote server.

Unreliable Datagram mode is more akin to standard UDP connections. Hermes uses RDMA SEND commands to send data to a destination server. The data is appended to a pre-registered memory region and the destination server is woken up to process the packet.

## B. Consensus protocols

Consensus protocols aim to achieve agreement on a data value. We present the four network consensus protocols accelerated in this paper, Mu, Velos, Sift and Hermes.

1) **Mu:** Mu is a leader-based consensus protocol [1]. Each replica owns a log, and the leader appends data to the logs. In the fast path, the leader sends a single RDMA WRITE to each replica. Once the RDMA WRITEs have been ACKed by a majority of the replicas, the leader knows that the data has been appended to a majority of the logs and consensus is reached.

The correctness of the approach is ensured by carefully setting the RDMA read/write permissions to the logs, to ensure that a single leader server has access to the replicas' logs at a given time.

**How to accelerate Mu:** Mu offers optimal latency (single round trip per consensus) but offers suboptimal bandwidth. Indeed, the leader sends the data to every replica, and so it shares its available bandwidth between the replicas. The protocol can be accelerated by having the leader send a single RDMA WRITE request to the programmable switch. The switch then transparently multicasts the request to all the replicas and counts their ACKs to only transmit one. Figure 1 illustrates the messages exchanged in the un-accelerated version of Mu vs. accelerated version.

2) **Velos:** Velos is a leader-based consensus protocol [15]. Velos uses the CAS RDMA primitive to achieve faster failover

<figure><img src="image_2.png" /><figcaption>Fig. 1: Message sequence diagram of the fast path in the Mu protocol, without acceleration (left) and with acceleration (right).</figcaption></figure>

compared to Mu; however, this approach incurs a performance overhead on the normal execution path. In the normal path, the leader first prepares a slot for a given value by sending a CAS request to each replica and, once it has received a quorum of acknowledgements, accepts the value with another CAS request. An additional WRITE request can be added to the accept phase to transmit a value bigger than the 64 bits limit of the RDMA CAS.

**How to accelerate Velos:** Like Mu, Velos does not offer optimal bandwidth as the leader shares its bandwidth between all the replicas. The protocol can be accelerated by having the leader send, for each request, a single RDMA CAS request to the programmable switch. The switch then transparently multicasts the request to all the replicas and counts their ACKs to only transmit one. Figure 2 illustrates the messages exchanged in the un-accelerated version of Velos vs. accelerated version.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Message sequence diagram of the fast path in the Velos protocol, without acceleration (top) and with acceleration (bottom).</figcaption></figure>

3) **Sift:** Sift is a leader-based consensus protocol and key-value store optimised for reducing cost and complexity rather than performance [23]. Sift proposes using high-performance coordinator nodes, that supports the high CPU usage of
the consensus protocol, and low-performance cheap memory
nodes for data storage.

In the fast path, when the coordinator receives a PUT
request to insert a (key, value) pair in the key-value store,
the coordinator sends an RDMA WRITE to every memory
node participating in the consensus. Once a quorum of RDMA
acknowledgments has been received, the leader acknowledges
the client's request (making client-side latency only a single
network round trip). Then a further RDMA WRITE is sent to
the memory nodes to apply the request in the background.

**How to accelerate Sift:** Similarly to Mu, Sift does not offer optimal bandwidth as every write request is sent multiple times. For both network round trips, the protocol can be accelerated by having the leader send a single WRITE request to the programmable switch for each broadcast round. The switch then transparently multicasts the request and counts their ACKs (see Figure 3).

<figure><img src="image_4.png" /><figcaption>Fig. 3: Message sequence diagram of the fast path in the Sift protocol, without acceleration (top) and with acceleration (bottom).</figcaption></figure>

4) *Hermes*: Hermes is a leaderless consensus protocol and key-value store that uses RDMA Unreliable Datagram SENDS [22]. In Hermes, all nodes play a symmetrical role. When a node receives a write request, an invalidation message is sent to all other nodes in the system. Once every node has acknowledged the invalidation message, the write is completed by a validation message. Fault tolerance is guaranteed through the use of a logical timestamp in invalidation messages.

**How to accelerate Hermes:** The protocol can be accelerated so that the nodes send a single validation and invalidation message to the switch. Then, the switch transparently multicasts the messages (see Figure 4). Unlike `READ`, `WRITE` and `CAS` commands, `SEND` commands are not acknowledged at the RDMA protocol level. Instead, each replica manually crafts an answer packet that is sent back as an acknowledgment using

the RDMA `SEND` command. These packets contain per replica information and cannot be aggregated in hardware.

<figure><img src="image_5.png" /><figcaption>Fig. 4: Message sequence diagram of the fast path in the Hermes protocol, without acceleration (top) and with acceleration (bottom).</figcaption></figure>

### C. Programmable switches

Programmable switches allow arbitrary packet switching and modification at network speed. These switches are divided into two layers: the *control plane* and *data plane*. The data plane is a programmable application-specific integrated circuit (ASIC) which enables the in-network computation capabilities of the switch. The control plane is a regular processor running a linux operating system used to control how the data plane processes packets.

The control plane is much more versatile than the data plane, allowing for the use of any programming language, but is significantly slower.

The data plane of our switch is programmed using the P4₁₆ programming language [4]. P4₁₆ defines a *pipeline* to process incoming packets. This pipeline is divided in two parts: the ingress and the egress (see Figure 5). Each part contains a parser, a series of stages used to operate on the network packets and a deparser. Between the ingress and the egress lies a buffer and the replication engine. The replication engine duplicates packets across the output ports of the switch. This means that packet routing and filtering should be done in the ingress, while machine specific packet modification belongs in the egress.

Intel Tofino [18] switches use the P4₁₆ programming language and enable stateful computation capabilities via the use of *registers*. Registers allow storing a value at a specific index and reading back this value at a later time. This allows the switch to adapt its behaviour using information received in packets without expensive control plane calls.
<figure><img src="image_6.png" /><figcaption>Fig. 5: Portable switch architecture pipeline.</figcaption></figure>

### III. DESIGN

The goal of P4CEMaker is to automatically detect if a consensus protocol can be accelerated and to assist developers in patching their code. To that end, P4CEMaker first uses dynamic analysis to detect network operations that can be accelerated at the switch level. When such operations are found, the code locations that trigger the operations are reported to the developer. P4CEMaker then provides a library that offers a convenient interface to accelerate the most common network operations and to patch the code with minimal effort.

#### A. Detecting common networking logic and finding code locations

P4CEMaker uses dynamic analysis to detect two core operations used by consensus protocols:

* The multicast of a value to all nodes participating in the consensus.

* The counting of acknowledgments, in response to a multicast, used to check if a majority of the participants have received a request.

Both operations can be accelerated by programmable switches [27], [8], [38], [14], [23], [22], [21]. A single run of the consensus algorithm is necessary to find the code locations responsible for these operations.

**a) Multicasts:** To detect multicasts, P4CEMaker captures every RDMA request sent to the replicas and analyses their content. When multiple requests containing the same data are sent in succession, a multicast is detected.

To help developers accelerate multicast operations, P4CEMaker provides the code locations that send multicasted requests, and the call chains that lead to these locations. Each call chain comes with a frequency indicator, allowing a developer to find the most common code paths that lead to a multicast. The information is useful because many protocols multicast diverse messages (e.g., in Mu both the fast path, the slow path and the "keep alive" requests are multicasted).

For some protocols, a simple call chain is not sufficient to understand the logic of the code, because RDMA commands are built through a chain of callbacks. In that case, P4CEMaker provides a full call graph that contains both the callers and callees of each function of the call chain.

**b) Counting acknowledgements:** The counting of acknowledgements happens *after* receiving the acknowledgment. P4CEMaker reports a call graph of the functions called after receiving an ACK. The graph is weighed according to the frequency of the chosen paths, allowing the easy distinction between the normal counting path, and error paths.

#### B. P4 library

The P4 library of P4CEMaker is its most complex component, as designing programmable switch code that is versatile and portable across different protocol paradigms and transport modes is challenging. The goal of P4CEMaker is to be as transparent as possible on the protocol side to minimize the porting effort.

The P4 library offers an API to build multicast groups. A multicast group is a group of machines that contain a leader and a certain number of replicas. One of these replicas is the *source* replica, while the others are *mirror* replicas. When the leader sends a certain packet to the source replica, the P4 code *multicasts* it to all mirror replicas, then *counts* all received acknowledgments of these packets, so only one is sent back to the leader.

Previous work on accelerating RDMA multicasts in hardware required establishing new connections between the switch and the replicas [12]. Naively, one might think this is a better option as the multicasted and non multicasted connections are independent and thus easier to manage. However, this approach has two main issues. The first is that using two separate connections loses the sequential guarantee on a single RDMA reliable connection, thus necessitating complex concurrent programming to avoid race conditions. The second issue is that building an RDMA connection is complicated and requires both extensive modifications of the protocol code and complex code in the programmable switch. Our library uses a much simpler approach. The leader establishes connections with the replicas, as in the normal, non-accelerated case, and then sends sufficient information to the switch so that the switch can use the connections pre-established by the leader. The leader sends a single packet to the switch, which replicates the packet so that all replicas receive it as if it were sent directly by the leader. This choice minimizes changes at the protocol level.

**a) Creating a multicast group:** To establish a multicast group, the leader provides the switch with the metadata of the connection to the source replica, along with the metadata of the connections to all mirror replicas.

**b) Multicasts:** When the switch receives a request from the leader, it multicasts it to the replicas. For that, the switch first duplicates the request, and then tailors each copy to each replica. At a high level, it rewrites the transport layer headers and the RDMA metadata of the packet to that of the destination. Our design also allows the leader to send non-replicated packets – in that case the switch simply transmits the packet to a single destination.
c) *Counting acknowledgements:* Some multicasted packets are acknowledged. For these requests, all replicas reply to the switch with an acknowledgment message, which can be positive in case of success, or negative in case of a failure. The switch waits for a quorum of positive acknowledgements before transmitting a single acknowledgment to the leader. If one of the servers sends a negative acknowledgment (NAK), the switch forwards it immediately to the leader. The leader can then deactivate the multicasting and revert to its regular fault recovery mechanisms. Multicasting can then be reestablished once all failures have been managed.

## IV. IMPLEMENTATION

We describe the implementations of the two components of P4CEMaker: the protocol analysis tool and the switch multicast library.

### A. Protocol analysis

The protocol analysis tool is implemented using 750 lines of C++ and uses the Intel Pin API [19] to capture function calls. During the execution of a consensus protocol, P4CEMaker maintains a buffer that stores the most recent 20 function calls, including their names and caller/callee relationships.

P4CEMaker utilizes the contents of this buffer to construct execution paths both leading to the `ibv_post_send` function (responsible for sending RDMA requests) and following the `ibv_poll_cq` function (responsible for receiving RDMA acknowledgments). The paths are not strict call chains: they also capture "side functions". For instance, if the function `a()` calls a function `b()` before calling `ibv_post_send`, the path contains both the call to `b()` and to `ibv_post_send`.

To detect multicast messages, P4CEMaker computes the hash of the data sent via `ibv_post_send`. If two subsequent calls to `ibv_post_send` have the same hash, then P4CEMaker infers that the application is performing a multicast operation and records the path leading to the `ibv_post_send` call.

In the RDMA protocol, all messages sent over the network have a `wr_id` (Work Request ID). Acknowledgments contain the `wr_id` of the message they acknowledge. When a multicast is detected, P4CEMaker also remembers the `wr_id` of the multicasted data. At every call of `ibv_poll_cq` P4CEMaker checks if the received acknowledgement corresponds to the `wr_id` of a previously identified multicasted request. If a match is found, P4CEMaker records the execution path following the `ibv_poll_cq` call (i.e., the 20 functions calls after `ibv_poll_cq`).

Offline, P4CEMaker then computes the frequency of paths leading to a multicast and following acknowledgments. This analysis enables the identification of frequently used "fast paths" and less common "slow or error paths." Finally, P4CEMaker outputs two weighted graphs that highlight the most frequent execution paths.

### B. P4 library

Previous efforts exist at multicasting RDMA packets using programmable switches [12]. Our P4 library reuses some of

the building blocks of these efforts, but significantly revisits the API used to create multicast groups to minimize the overhead of patching applications. We also add support for RDMA over Converged Ethernet (RoCE) v1, the unreliable datagram transport mode and allow for some non-multicasted packets to coexist with multicasted packets.

1) *Usage:* To enable multicasting for a specific process, the relevant connection metadata must be transmitted to the switch control plane. Our library requires only a minimal amount of information for a given configuration. Notably, this information constitutes a subset of the data already required for sending a packet, making it readily available within the code of a consensus protocol. When a multicast group is first enabled, a request is sent to the control plane of the programmable switch (via TCP). The control plane then generates and initializes the necessary data structures in the data plane to facilitate multicasting.

Patching a protocol requires only few lines of code, exemplified in Listing 1 for Reliable Connections. After creating connections to a set of replicas, a server registers these connections with the switch (lines 3-14 of Listing 1). Registering connections consists in creating a new broadcast group (line 3), and, for each replica, in collecting the relevant connection metadata (line 6, the IP, the Queue Pair, the `R_key` and the address of the readable/writable buffer) and registering the replica either as the source replica (line 9) or as a mirror replica (line 11). Once all replicas have been registered, the switch is configured (line 14) and the multicast is activated (line 15).

Once the multicast is activated, packets sent to the source replica are multicasted to all replicas. Patching the sending side is done by sending packets only to the source replica (line 24).

2) *Features:* The library supports multicasting across both reliable connections and unreliable datagram transport modes. Both RoCE v1 and RoCE v2 are supported.

a) *Supporting the coexistence of multicasted and unicasted messages:* Our library allows a machine to send both multicasted packets and unicasted packets on the same queue pair. In the first case, the switch will transparently duplicate the packets then aggregate the acknowledgements. In the second case, the packet is left untouched and sent to its destination.

Special care must be taken when multicasting RDMA packets in reliable connections, as the connection is *stateful*. In particular, reliability in RDMA is enforced through the use of a packet sequence number (PSN). When a queue pair is created, it is given a random number to use as the PSN for the first packet (as a consequence, the connections to the different replicas have a different starting PSN). The PSN is then incremented every time a packet is sent on a given queue pair. Any missing or out-of-order PSN will cause an error and a retransmission. The switch must thus keep track of packets sent on each queue pair and update the PSNs accordingly.

When a protocol only sends multicasted messages, keeping track of the PSNs is easy: there is a fixed offset between the PSN of the source replica, and the PSN of a given mirror
**Listing 1** Consensus protocol code patched to use the bypass library.

```c
1 void initConnections() {
2 ...
3 + auto group = new MulticastGroup();
4 + boot firstConnection = true;
5 + for (auto &c: connections) {
6 +   auto channel = new Channel(cremoteIp, c.qp,
                                  cremoteKey, cremoteBuffer);
7 +   if (firstConnection) {
8 +     firstConnection = false;
9 +     group.setSourceChannel(channel);
10 +   } else {
11 +     group.addMirrorChannel(channel);
12 +   }
13 + }
14 + group.configureSwitch();
15 + group.enableMulticast();
16 ...
17 }
18
19 void sendOperation() {
20 ...
21 - for (auto &connection : connections) {
22 -   sendPacket(connection, packet);
23 - }
24 + sendPacket(connection[0], packet);
25 ...
26 }
```

replica. The offset can be computed when creating a multicast group and communicated to the switch. However, when both multicast messages and unicast messages coexist on the same queue pair, the offsets between the various replicas can drift.

Keeping track of this drift, while simple in theory, is made actually complex because of the limitations inherent to hardware programming. For example, a naive approach would be to store, for each mirror replica, a drift with the source replica. Upon receiving a unicast message, the source replica would increase the drift of all mirror replicas. This simple approach, however, is not possible to implement on our switch, as the switch does not allow incrementing multiple elements of an array.

Instead, this calculation requires introducing a *multicast sequence number* as an intermediary value. The full algorithm to translate the PSN of an ACK received from a mirror replica to the PSN of the ACK sent to the leader is presented in Listing 2. We first initialize the necessary data structures (lines 1-6), in particular we set `multicast_sequence_number` to 0 and store for each mirror its current PSN. Upon receiving a multicast packet, we increment `multicast_sequence_number` and store the PSN of this packet into the map `ack_psn`. Upon receiving an acknowledgement from a mirror replica, we subtract the offset to get its multicast sequence number (line 13), then retrieve and set the translated PSN (line 14).

To distinguish packets that are meant to be multicast and packets that are meant to be unicast, we use the `R_key` field of the RDMA `WRITE` packets. Multicast packets use a custom `R_key` (one could think of that `R_key` as the `R_key`

**Listing 2** Algorithm to translate acknowledgement PSN.

```c
1 Initialisation:
2   offsets = []
3   ack_psn = []
4   multicast_sequence_number = 0
5   for each mirror replica:
6     offsets[mirror] = current_psn_mirror
7   On receiving multicast packet:
8     multicast_sequence_number++
9     ack_psn[multicast_sequence_number] = headers.psn
10    On receiving acknowledgement from mirror replica:
11     packet_multicast_sequence_number = headers.psn
12     - offset[mirror]
13     headers.psn = ack_psn[packet_multicast_sequence_number]
14   On receivingacknowledgement from mirror replica:
15   packet_multicast_sequence_number = headers.psn
16   ...
17 }
18
19 void sendOperation() {
20 ...
21 - for (auto &connection : connections) {
22 -   sendPacket(connection, packet);
23 - }
24 + sendPacket(connection[0], packet);
25 ...
26 }
```

of the switch). When the switch receives a packet with that `R_key`, it duplicates the packet, and modifies the `R_key` of the replicated packets to match that of the replicas. When the switch receives a packet with another `R_key`, it simply forwards it to its destination.

This functionality adds another layer of complexity in the implementation, since not all packets have an available `R_key` field. In particular, when sending a request bigger than the maximum transmission unit (MTU), the RDMA protocol splits the request into several small packets, and only the first contains an `R_key`. We maintain an additional table in the data plane to store, for each queue pair, whether a packet belonging to a large `WRITE` is meant to be multicast.

The library currently only supports mixing unicast and multicast packets on the queue pair of the source replica, and not on the queue pair of the mirror replicas. In practice, we found this condition to not be restrictive.

## V. RELATED WORK

a) *Network acceleration:* Programmable switches and SmartNICs have been used to accelerate multiple consensus protocols [36], [20], [10], [8], [9], [30], [27], [3], [25], [26] and other workloads [13], [37], [31], [32], [6], [40], [11], [7], [39], [24].

NetPaxos [10] and P4xos [8] accelerate the original Paxos protocol using P4 programmable Tofino switches. Each switch plays a specific Paxos role (proposer, acceptor, ...) and protocol decision is implemented inside the switches. P4xos offers a protocol specific hardware acceleration requiring numerous switches and a specialised network topology. NOPaxos proposes using an ordered unreliable multicast to replace the necessity of an expensive consensus and implements this idea using programmable switches. HoverRaft [27] introduces a Raft implementation where, similarly to P4CEMaker, network bottlenecks are eliminated through programmable switch multicast and aggregations. P4CEMaker proposes automating such approaches for RDMA-based protocols. NetLR [26] accelerates Hermes by using a programmable switch as a
coordinator for the various replicas and replacing all replica-to-replica traffic, thus removing the network bottleneck.

Other endeavours have proposed consensus hardware acceleration using SmartNICs [3], [25], reducing leader CPU overhead but not addressing the leader network link bottleneck. István et al. propose accelerating a consensus protocol using FPGAs by running all of the protocol logic in hardware and allowing for much greater performance [20]. However, their approach doesn't reduce network traffic and is complementary to the one proposed by P4CEMaker.

Programmable switches have been used to accelerate numerous other workloads. Sapio et al. [37] present a hardware packet aggregation for scaling distributed machine learning leveraging the power of programmable switches. NetReduce [31], [32] showcases using FPGAs to accelerate deep neural network training over RDMA. P4COM [6] proposes using programmable switches for in-network computation and accelerates a MapReduce workload using a Tofino switch. Gallium [40] is a compiler allowing automatic acceleration of middlebox programs using a P4 programmable switch. P4SC [7] presents a framework to implement service function chains (i.e., load balancing, firewalling, ...) in programmable switches. Wang et al. [39] use P4 to aggregate and disaggregate small IOT packets in a programmable switch data plane.

**b) RDMA:** Using RDMA is key to the low latency and high throughput of message exchanges and is now commonly used to achieve consensus. In this paper we focused on accelerating Mu [1], Sift [23] and Hermes [22] as representative of protocols with different replication strategies. Our approach could be applied to accelerate other RDMA-based consensus protocols [38], [35]. APUS [38] is another leader-based consensus protocols that uses RDMA *WRITE* to replicate a value across a set of nodes. P4CEMaker would be capable of accelerating these broadcasted writes but wouldn't be able to accelerate the custom RDMA *WRITE* based acknowledgements used by APUS. RotorcRaft [35] is an RDMA-based implementation of the Raft protocol [34] optimized to reduce the bottleneck on the leader. P4CEMaker could be used to make these optimizations more performant.

**c) Accelerating RDMA workloads:** RDMA multicast has been implemented in software [2], [28], and the ability to multicast at the protocol level has been discussed [29]. RDMC [2] multicasts a packet using a succession of point-to-point communication. A node, upon receiving a packet, will duplicate it across a predetermined set of nodes, spreading the information across the network. Such replica coordination is also used in RamCast [28] to provide atomic multicasts. However, these two methods incur a CPU overhead cost in packet replication and a latency cost due to their pipelined design. Extensions to the RDMA protocol allowing each replica to accept identical RDMA packets [29] would bring optimal throughput using a passive optical cross-connect fabric to packet duplication. This technique would however not be able to perform acknowledgement counting.

# VI. EVALUATION

In this section, we evaluate the developer effort necessary to accelerate a protocol using P4CEMaker and present performance results. The evaluation aims at answering the following questions:

* Is P4CEMaker able to correctly detect the network requests that can be accelerated? Is P4CEMaker able to significantly reduce the development time of an accelerated protocol?

* Does the hardware acceleration provided by P4CEMaker increase performance?

## A. Experimental setup and results

We tested the consensus protocols on 3 machines. The first two are Dell PowerEdge R7515 machines, each with an AMD EPYC 7302P CPU (32 cores) and 32 GiB of DDR4 random access memory (RAM). The third machine is a Supermicro SYS-520P-WTR with an Intel Xeon Gold 6326 CPU (16 cores) and 64 GiB of DDR4 RAM. These are all connected through an Edgecore Wedge 100BF-32X programmable switch that contains a 2-pipe first generation Intel Tofino ASIC. All machines are equipped with NVIDIA ConnectX-5 network cards interfaced through PCI Express 3.0 x16 links. Each card is directly connected to the programmable switch using 100 Gbit/s Ethernet.

P4CEMaker was evaluated using the available open source implementations of Hermes, Sift and Mu. Because the source of Velos is not publicly available, we built a custom implementation, following the algorithmic description of the original article.

We executed P4CEMaker on all four protocols. The maximum observed overhead, compared to a non-instrumented run, was 30%, which is acceptable since P4CEMaker only needs to be executed once. P4CEMaker was able to automatically detect requests that could be accelerated in all four protocols. For Mu, Velos and Sift, both multicasted requests and received acknowledgements were detected. In Hermes, since there is no protocol acknowledgement in unreliable datagram, only multicasted requests were detected. In all cases, the functions needed to be patched were included in the output of the tool.

Using P4CEMaker we were able to accelerate Mu, Velos, Sift and Hermes in only a day of work. We are thus able to achieve much higher development velocity than similar approaches using a bespoke hardware acceleration (e.g., a similar work to accelerate Mu required a full year of work [12]). Moreover, this development velocity was achieved without prior knowledge of the consensus protocol code base. Table I presents the number of lines of code necessary to patch each protocol and the maximum performance improvement brought by the patch. Patching a protocol required a maximum of 73 lines of code change. The patches improve performance by up to 2×. In the remainder of this section, we detail the patching effort and the performance gains for each protocol.
<table><caption>TABLE I: Patching effort and performance improvement brought by hardware acceleration.</caption><thead><tr><td>Protocol</td><td>Patch size (lines of code)</td><td>Max. performance improvement (with 2 replicas)</td></tr></thead><tbody><tr><td>Mu</td><td>52</td><td>2x</td></tr><tr><td>Velos</td><td>40</td><td>2x</td></tr><tr><td>Hermes</td><td>73</td><td>45%</td></tr><tr><td>Sift</td><td>55</td><td>30%</td></tr></tbody></table>

## B. Mu

1) Protocol patching: Mu broadcasts multiple messages: consensus queries in the slow and fast path, and heartbeats. While the code of Mu is well-written and functions have explicit names, it is a large code base (over 12000 lines of code). P4CEMaker simplifies the analysis by producing the call graph presented in Figure 6, which shows the various calls leading to the `ibv_post_send` function. We clearly see on the graph the 4 functions involved in the critical path, highlighted in red. The `propose` and `fastWrite` functions are called 5 times more often than the greyed out functions, and the `postSendSingle` is called 10 times more often. Given that we executed Mu with 2 replicas, this hints that the for loop that iterates over the various replicas is in the `fastWrite` function. The figure illustrates how P4CEMaker simplifies the work of the developer, that can directly ignore greyed paths, whereas a more traditional study would have required a more thorough and time-consuming analysis. The necessary change to the code so only one request is sent is presented in listing 3, showcasing how we only need to replace a for loop with its first execution. Configuring the multicast prior to sending queries also caused no significant challenge, as the necessary data is readily available in the code.

<figure><img src="image_7.png" /><figcaption>Fig. 6: Mu multicasted request call graph produced by P4CEMaker. Edge width symbolizes the frequency of the path in the call graph.</figcaption></figure>

P4CEMaker also helps in modifying the protocol to wait for a single acknowledgment instead of a quorum of acknowledgments. This step generally involves finding an integer being incremented for every received ACK and checking it against an expected number of ACKs. P4CEMaker's instrumentation is useful in this step, as it allows tracking the code being executed after receiving an acknowledgement. For Mu, we get the graph presented in Figure 7, which illustrates the most common code

<caption>**Listing 3** Patch of Mu to send requests only to the source replica.</caption>

```
1 void fastWrite(...) {
2   ...
3   - for (auto &c : connections) {
4     auto &c = connections[0];
5     auto ok = c.rc->postSendSingle(...);
6   }
7   ...
8 }
```

pathes executed after receiving an acknowledgement. Note that there are 5 instances of the function call `ibv_poll_cq` in the Mu codebase and numerous other paths taken by the code which are not shown here as they were filtered out by P4CEMaker, greatly simplifying the analysis. In this case, the function to patch is `fastConsume` and the necessary patch presented in listing 4.

<figure><img src="image_8.png" /><figcaption>Fig. 7: Mu acknowledgement call graph produced by P4CEMaker. Edge width symbolizes the frequency of the path in the call graph.</figcaption></figure>

<caption>**Listing 4** Patch of Mu to await only a single acknowledgment.</caption>

```
1 void fastConsume(...) {
2   ...
3   - int expected_nr = outstanding_req *
               replicas_size + quorum_size;
4     int expected_nr = outstanding_req + 1;
5   ...
6 }
```

2) Performance: Using P4CEMaker, we were able to reproduce performance results of [12] on Mu achieving 2 times the goodput of the non-accelerated version (see Figure 8). The goodput is the number of useful bytes sent per second, which excludes protocol overhead bits. Using P4CEMaker,
we can achieve consensus at link speed (11.5GB/s) for values above 500B. The performance gain is expected: in the non-accelerated Mu, the link from the leader to the switch is a bottleneck. Since the experimental setup uses one leader and two replicas, P4CEMaker halves the necessary bandwidth required to achieve consensus, which results in doubling the goodput. Using a setup with more replicas could allow for further performance improvements, as showcased in [12], where a 4 times performance improvement is obtained using four replicas.

<figure><img src="image_9.png" /><figcaption>Fig. 8: Mu goodput depending on item size. The version accelerated by P4CEMaker maximizes the available network bandwidth, while the non-accelerated version is limited by the link from the leader to the switch.</figcaption></figure>

### C. Hermes

1) **Protocol patching:** Hermes is the most complex protocol to patch (73 lines of code changed). Figure 9 illustrates the call graph used by Hermes to gather the metadata of the multicasted messages. Hermes relies on a generic library to send its messages. To create a message, the library calls both its own functions and callbacks set by the user of the library. P4CEMaker simplifies the analysis. Looking at Figure 9, we notice that the *wings* library calls the user's callbacks in the *wings issue pkts* function. By reading the source code of the callbacks, it is easy to see that the `inv_skip_or_get_sender_id` and `val_skip_or_get_sender_id` functions actually return the id of the destination node of the request. Thus, we modified the first function to return the ID of the source replica, and to skip the other replicas.

Since Hermes uses custom requests for acknowledgements, they can't be hardware accelerated using a generic library and no changes for counting acknowledgements are necessary.

2) **Performance:** When the load is balanced between all nodes, the bottleneck is the network down-link (i.e., the link from the switch to a node), on which hardware acceleration has no impact. Nevertheless, minor, but repeatable, performance improvements can be observed (see Figure 10a). However,

when the load is imbalanced between nodes, hardware acceleration improves performance by up to 45% (see Figure 10b). With lower write ratios, since reads are local, Hermes is CPU-bound and only minor improvements can be observed. With higher ratios however, the un-accelerated protocol is network-bound and benefits from the acceleration. The theoretical 2x maximum improvement is not reached because Hermes still becomes CPU-bound before being able to fully saturate the network bandwidth.

### D. Velos

1) **Protocol Patching:** The patching effort of Velos is similar to that of Mu. The patch required only 40 lines of code and half a day of work. (Note that, even though we use a custom implementation of Velos, the person that patched the protocol is not the person that implemented the protocol.)

2) **Performance:** Similarly to Mu, we were able to double the performance of Velos using P4CEMaker (see Figure 11 that presents the goodput of Velos). The reason is the same: the bottle-neck was the bandwidth of the link between the leader and the switch, and P4CEMaker allows the leader to halve the number of messages sent on that link. The accelerated version of Velos almost achieves consensus at link speed. The 1GB/s difference is explained by the overhead of the compare-and-swap operations performed by Velos.

### E. Sift

1) **Protocol Patching:** Due to Sift separating coordinator nodes (which manage consensus), and memory nodes (which store data), the protocol involves more complex network exchanges than the previously presented protocols. Again, P4CEMaker helps in narrowing down the search of multicast and acknowledgments. Patching Sift required 55 lines of code and a day of work.

Sift is a key value store managing both GET and PUT requests. GET requests involve sending a single `READ` request to one of the memory nodes, while PUT requests are multicasted

<figure><img src="image_10.png" /><figcaption>Fig. 9: Hermes call graph leading to multicasts.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>(a) Relative speedup of accelerated execution when using a balanced load.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Absolute throughput when using an imbalanced load.</figcaption></figure>

<caption>Fig. 10: Hermes throughput depending on the write ratio.</caption>

across all memory nodes. Thus, only the latter need to be hardware accelerated. Sift uses the same queue pair both GET and PUT queries, meaning that non multicasted messages (reads) coexist with multicasted messages. P4CEMaker identified a few functions:

* `submitWriteRequest` and `submitUnloggedWriteRequest` which can be multicasted.

* `processWriteCompletion` and `processLogWriteCompletion` which need to be modified to wait for a single acknowledgement.

Patching these functions required 55 lines of code. The main difference with the previous patches is that the `submitWriteRequest` function enables multicasting on the source replica upon entry, and disables multicasting upon exit so that the messages sent by other functions on the same queue pair are not multicasted.

<figure><img src="image_13.png" /><figcaption>Fig. 11: Velos goodput depending on item size.</figcaption></figure>

2) Performance: In Sift, hardware acceleration improves performance by up to 30% (see Figure 12). Enabling and disabling multicasting on a queue pair simply consists in setting or clearing a flag on a data structure and has no observable overhead. Using hardware acceleration actually lowers the CPU and memory load per operation because fewer packets need to be generated by the protocol, and fewer acknowledge need to be counted. Sift uses a pipeline design, which is bottlenecked by the network functions responsible for creating and counting packets. Reducing the number of such operations explains the performance improvements.

<figure><img src="image_14.png" /><figcaption>Fig. 12: Sift write throughput depending on number of client threads.</figcaption></figure>

## VII. CONCLUSION

We have shown how multiple consensus protocols share common network operations that can be accelerated in hard-
ware. We have presented P4CEMaker. P4CEMaker uses dy-
namic analysis to find these network operations and offers
a P4 library with an easy-to-use API to accelerate these
operations. Using this method, we have patched four pre-
existing consensus protocols and have achieved up to 2×
performance gains.

REFERENCES

[1] Marcos K. Aguilera, Naama Ben-David, Rachid Guerraoui, Virendra J. Marathe, Athanasios Xygkis, and Igor Zablotchi. Microsecond consensus for microsecond applications. In *OSDI*, pages 599–616. USENIX Association, 2020.

[2] Jonathan Behrens, Sagar Jha, Ken Birman, and Edward Tremel. RDMC: A reliable RDMA multicast for large objects. In *DSN*, pages 71–82. IEEE Computer Society, 2018.

[3] Giacomo Belocchi, Valeria Cardellini, Aniello Cammarano, and Giuseppe Bianchi. Paxos in the NIC: hardware acceleration of distributed consensus protocols. In *DRCN*, pages 1–6. IEEE, 2020.

[4] Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, and David Walker. P4: programming protocol-independent packet processors. *Comput. Commun. Rev.*, 44(3):87–95, 2014.

[5] Broadcom Inc. RDMA over Converged Ethernet feature in Ethernet NIC controllers, 2022.

[6] Ge Chen, Gaoxiong Zeng, and Li Chen. P4COM: in-network computation with programmable switches. CoRR, abs/2107.13694, 2021.

[7] Xiang Chen, Dong Zhang, Xiaojun Wang, Kai Zhu, and Haifeng Zhou. P4SC: towards high-performance service function chain implementation on the p4-capable device. In *IM*, pages 1–9. IFIP, 2019.

[8] Huynh Tu Dang, Pietro Bressana, Han Wang, Ki Suh Lee, Noa Zilberman, Hakim Weatherspoon, Marco Canini, Fernando Pedone, and Robert Soulé. P4xos: Consensus as a network service. *IEEE/ACM Trans. Netw.*, 28(4):1726–1738, 2020.

[9] Huynh Tu Dang, Marco Canini, Fernando Pedone, and Robert Soulé. Paxos made switch-y. *Comput. Commun. Rev.*, 46(2):18–24, 2016.

[10] Huynh Tu Dang, Daniele Sciascia, Marco Canini, Fernando Pedone, and Robert Soulé. NetPaxos: consensus at network speed. In *SOSR*, pages 5:1–5:7. ACM, 2015.

[11] Joeri de Ruiter and Caspar Schutijser. Next-generation internet at terabit speed: SCION in P4. In *CoNEXT*, pages 119–125. ACM, 2021.

[12] Rémi Dulong, Nathan Felber, Pascal Felber, Gilles Felber, Baptiste Lepers, Valerio Schiavoni, Gaël Thomas, and Sébastien Vaucher. P4CE: Consensus over RDMA at Line Speed. In *2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)*, 2024.

[13] Aoxiang Feng, Dezun Dong, Fei Lei, Junchao Ma, Enda Yu, and Ruiqi Wang. In-network aggregation for data center networks: A survey. *Comput. Commun.*, 198:63–76, 2023.

[14] Rachid Guerraoui, Antoine Murat, and Athanasios Xygkis. Velos: One-sided paxos for RDMA applications. CoRR, abs/2106.08676, 2021.

[15] Rachid Guerraoui, Antoine Murat, and Athanasios Xygkis. Velos: One-sided paxos for rdma applications, 2021.

[16] InfiniBand Trade Association. *InfiniBand Architecture Specification Volume 1*, 2020.

[17] Intel Corporation. Which Intel Ethernet network adapters support iWARP and RoCE v2?, 2021.

[18] Intel Corporation. Intel Tofino intelligent fabric processors, 2022.

[19] Intel Corporation. Pin - a dynamic binary instrumentation tool, 2024.

[20] Zsolt István, David Sidler, Gustavo Alonso, and Marko Vukolic. Consensus in a box: Inexpensive coordination in hardware. In *NSDI*, pages 425–438. USENIX Association, 2016.

[21] Joseph Izraelevitz, Gaukas Wang, Rhett Hanscom, Kayli Silvers, Tamara Silbergleit Lehman, Gregory Chockler, and Alexey Gotsman. Acuerdo: Fast atomic broadcast over rdma. In *Proceedings of the 51st International Conference on Parallel Processing*, ICPP '22, New York, NY, USA, 2023. Association for Computing Machinery.

[22] Antonios Katsarakis, Vasilis Gavrielatos, M.R. Siavash Katebzadeh, Arpit Joshi, Aleksandar Dragojevic, Boris Grot, and Vijay Nagarajan. Hermes: A fast, fault-tolerant and linearizable replication protocol. In

Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '20, page 201–217, New York, NY, USA, 2020. Association for Computing Machinery.

[23] Mikhail Kazhamiaka, Babar Naveed Memon, Chathura Kankanamge, Siddhartha Sahu, Sajjad Rizvi, Bernard Wong, and Khuzaima Daudjee. Sift: resource-efficient consensus with RDMA. In *CoNEXT*, pages 260–271. ACM, 2019.

[24] Daehyeok Kim, Zaoxing Liu, Yibo Zhu, Changhoon Kim, Jeongkeun Lee, Vyas Sekar, and Srinivasan Seshan. TEA: enabling state-intensive network functions on programmable switches. In *SIGCOMM*, pages 90–106. ACM, 2020.

[25] Daehyeok Kim, Amir Saman Memaripour, Anirudh Badam, Yibo Zhu, Hongqiang Harry Liu, Jitu Padhye, Shachar Raindel, Steven Swanson, Vyas Sekar, and Srinivasan Seshan. Hyperloop: group-based nic-offloading to accelerate replicated transactions in multi-tenant storage systems. In *SIGCOMM*, pages 297–312. ACM, 2018.

[26] Gyuyeong Kim and Wonjun Lee. In-network leaderless replication for distributed data stores. *Proc. VLDB Endow.*, 15(7):1337–1349, 2022.

[27] Marios Kogias and Edouard Bugnion. HoverRaft: achieving scalability and fault-tolerance for microsecond-scale datacenter services. In *EuroSys*, pages 25:1–25:17. ACM, 2020.

[28] Long Hoang Le, Mojtaba Eslahi-Kelorazi, Paulo R. Coelho, and Fernando Pedone. RamCast: RDMA-based atomic multicast. In *Middleware*, pages 172–184. ACM, 2021.

[29] Kin-Wai Leong, Zhilong Li, and Yunqu Leon Liu. Reliable multicast using remote direct memory access (RDMA) over a passive optical cross-connect fabric enhanced with wavelength division multiplexing (WDM). *APISPA Transactions on Signal and Information Processing*, 8:e25, 2019.

[30] Jialin Li, Ellis Michael, Naveen Kr. Sharma, Adriana Szekeres, and Dan R. K. Ports. Just say NO to Paxos overhead: Replacing consensus with network ordering. In *OSDI*, pages 467–483. USENIX Association, 2016.

[31] Shuo Liu, Qiaoling Wang, Junyi Zhang, Qinliang Lin, Yao Liu, Meng Xu, Ray C. C. Cheung, and Jianfei He. Netreduce: Rdma-compatible in-network reduction for distributed DNN training acceleration. CoRR, abs/2009.09736, 2020.

[32] Shuo Liu, Qiaoling Wang, Junyi Zhang, Wenfei Wu, Qinliang Lin, Yao Liu, Meng Xu, Marco Canini, Ray C. C. Cheung, and Jianfei He. In-network aggregation with transport transparency for distributed training. In *ASPLOS (3)*, pages 376–391. ACM, 2023.

[33] NVIDIA Corporation. ConnectX SmartNICs, 2022.

[34] Diego Ongaro and John Ousterhout. In search of an understandable consensus algorithm. In *Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference*, USENIX ATC'14, page 305–320, USA, 2014. USENIX Association.

[35] Xuecheng Qi, Huiqi Hu, Xing Wei, and Aoying Zhou. Rotorcraft: Scalable follower-driven raft on rdma. In *Arnab Bhattacharya, Janice Lee Mong Li, Divyakant Agrawal, P. Krishna Reddy, Mukesh Mohania, Anirban Mondal, Vikram Goyal, and Rage Uday Kiran, editors, Database Systems for Advanced Applications*, pages 293–308, Cham, 2022. Springer International Publishing.

[36] Amedeo Sapio, Ibrahim Abdelaziz, Abdulla Aldilaijan, Marco Canini, and Panos Kalnis. In-network computation is a dumb idea whose time has come. In *HotNets*, pages 150–156. ACM, 2017.

[37] Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan R. K. Ports, and Peter Richtárik. Scaling distributed machine learning with in-network aggregation. In *NSDI*, pages 785–808. USENIX Association, 2021.

[38] Cheng Wang, Jianyu Jiang, Xusheng Chen, Ning Yi, and Heming Cui. APUS: fast and scalable paxos on RDMA. In *SoCC*, pages 94–107. ACM, 2017.

[39] Shie-Yuan Wang, Chia-Ming Wu, Yi-Bing Lin, and Ching-Chun Huang. High-speed data-plane packet aggregation and disaggregation by P4 switches. *J. Netw. Comput. Appl.*, 142:98–110, 2019.

[40] Kaiyuan Zhang, Danyang Zhuo, and Arvind Krishnamurthy. Gallium: Automated software middlebox offloading to programmable switches. In *SIGCOMM*, pages 283–295. ACM, 2020.