# Energy-Efficient Right-Sizing of Kafka-like Message Brokers for IoT Workloads

Govind KP, Guillaume Pierre, Romain Rouvoy

<figure><img src="image_1.png" /></figure>

## To cite this version:

Govind KP, Guillaume Pierre, Romain Rouvoy. Energy-Efficient Right-Sizing of Kafka-like Message Brokers for IoT Workloads. ICPE 2026 - 17th ACM/SPEC International Conference on Performance Engineering, ACM; SPEC, May 2026, Florence, Italy. 10.1145/3777884.3797015. hal-05512966

HAL Id: hal-05512966

https://inria.hal.science/hal-05512966v1

Submitted on 16 Feb 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Energy-Efficient Right-Sizing of Kafka-like
Message Brokers for IoT Workloads

Govind KP

Guillaume Pierre

Romain Rouvoy

Inria, Univ. Rennes, CNRS, IRISA

Univ. Rennes, Inria, CNRS, IRISA

Univ. Lille, Inria, CNRS, CRISTAL

Rennes, France

Rennes, France

Lille, France

research@govind.contact

guillaume.pierre@irisa.fr

romain.rouvoy@inria.fr

Abstract

IoT data pipelines rely on message brokers, such as Apache Kafka and Redpanda for continuous telemetry ingestion. When it comes to capacity planning of these systems, the absence of clear sizing guidance often leads to conservative over-provisioning and unnecessary energy use. We present a calibration-based methodology for energy-efficient right-sizing of Kafka-compatible clusters for IoT ingest. Using a small set of initial experiments on 3–4 nodes, we fit a performance model that predicts maximum sustainable throughput and per-node power, enabling operators to choose the smallest cluster that satisfies a target ingest rate with headroom while minimizing energy consumption.

We substantiate the approach with an experimental study of Kafka and Redpanda across three hardware generations (HDD, SATA SSD, NVMe), varying partition counts, node counts, and resource limits. We find that storage technology is the primary determinant of throughput, horizontal scaling is near-linear, and vertical CPU scaling yields diminishing returns; the two brokers exhibit distinct energy proportionality properties. On previously unseen hardware, the model predicts throughput and power with median errors under 10% and 7%, respectively.

Our results provide a practical, reproducible capacity-planning
workflow that maps IoT workload requirements (message size and
rate) to concrete, energy-aware deployment decisions.

CCS Concepts

• Information systems → Message queues.

Keywords

Message brokers, capacity planning, right-sizing, energy-efficiency, IoT.

1 Introduction

Every second, millions of IoT sensor readings, financial transactions, and user interactions flow through distributed messaging systems, which have become foundational infrastructure for managing data at scale. Apache Kafka, introduced in 2011, innovated in this space by treating messages as an append-only log, combining the durability of a database with the throughput of a messaging system [18]. With features such as replay, customizable retention periods, and exactly-once semantics, Kafka has been widely adopted across the industry, and over 80% of Fortune 100 companies rely on it for countless use cases [3].

Despite Kafka’s widespread adoption, deploying and managing message broker systems is operationally complex. Their mission-critical nature means any operational changes must be zero-downtime and cannot cause performance degradation.

This is compounded by two further challenges: First, while message brokers scale horizontally with relative ease, scaling down proves difficult due to potentially very large broker state that requires rebalancing across nodes, a process typically necessitating careful manual intervention. This inflexibility makes the initial deployment decisions very hard to reverse.

Second, administrators face a sheer number of configuration choices that must be made upfront. They must select a messaging framework<fnref n="1" />, a hardware configuration, the number of servers, retention policies, and numerous other configuration parameters that all interact in complex ways.

This deployment challenge has consequences that extend far beyond a single cluster. Data centers already consume a significant percentage of our global electricity, yet servers often operate at just 20-40% utilization—an inefficiency driven largely by our collective inability to accurately provision critical infrastructure [31]. Message brokers exemplify this dilemma perfectly. Lacking clear guidance for upfront capacity planning, organizations often fall back to a “better safe than sorry” approach, tending to over-provision their systems to guarantee performance. While some turn to cloud-hosted Queuing-as-a-Service (QaaS) offerings [1, 7, 22], this merely shifts the configuration burden to QaaS providers who may employ similar over-provisioning strategies.

This paper addresses the following questions: (1) given the ex-
pected characteristics of an IoT messaging workload, which hard-
ware and software configurations will deliver sufficient perfor-
mance while avoiding over-provisioning? And (2), if multiple such
"right-sized" configurations exist, how should administrators choose
between them based on factors like cost and energy efficiency?

The state of the art offers surprisingly little guidance for answer-
ing these questions. Much of this knowledge exists as hard-won,
tacit expertise within organizations, but a systematic, accessible
exploration of the problem has been lacking.

To fill this gap, in this paper we conduct a comprehensive study to characterize the impact of key configuration choices. We empirically measure the performance and energy usage of Apache Kafka and Redpanda<fnref n="2" /> on three distinct hardware setups with different CPUs, disks, memory, and network configurations. We also vary the IoT message workloads and the number of servers to characterize

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.

<footnote n="1">¹Alternatives to Kafka include several API-compatible frameworks such as Redpanda [27], WarpStream [32] and Pravega [26].</footnote>

<footnote n="2">²Redpanda is arguably the most popular alternative to Kafka [5], and it claims massive performance improvements. We leave a study of other similar frameworks for future work.</footnote>
performance bottlenecks and the conditions under which they arise. Our study follows a principled, minimal experimental design that isolates main effects while keeping the study tractable.

*Scope of this study:* To bound the design space, we focus this study on large-scale IoT-style streaming pipelines, where producers continuously emit small- to medium-sized payloads at high rates, and durability is ensured via replication. This class of workloads is representative of telemetry ingest backbones and data pipelines that feed stream processors. While the methodology transfers to other domains, evaluating those specificities (e.g., transactional eventing with strict latency SLOs) is out of scope for this paper.

Our experimental results show that storage technology is the most critical factor determining broker throughput; in particular, NVMe-based deployments achieve up to 5x greater throughput than those using legacy HDDs. We also find that both Kafka and Redpanda scale nearly linearly as brokers are added, offering predictability. However, their energy consumption patterns differ markedly: Kafka's power usage plateaus at moderate loads, whereas Redpanda's consumption scales more proportionally with load.

Based on these findings, we derive a performance model that enables operators to select “right-sized” configurations that precisely match expected performance requirements. Finally, we demonstrate the practical application of this model by predicting the performance and energy usage of a previously unseen hardware configuration with a minimal set of initial experiments, achieving a prediction error lower than 10% for throughput and 7% for energy consumption.

The paper is organized as follows: Section 2 provides background on Apache Kafka and Redpanda. Section 3 presents our experimental evaluations, Section 4 proposes our actionable model, and Section 5 validates it. Finally, Section 6 discusses related work, and Section 7 concludes.

# 2 Background

## 2.1 Log-Based Message Brokers

Message brokers enable asynchronous communication between message producers which generate messages tagged with a topic attribute, and message consumers which subscribe to messages tagged with relevant topics. Traditional message brokers, such as RabbitMQ, implement the AMQP or MQTT protocols and focus primarily on message routing and delivery guarantees. These brokers, however, were not designed for the high-throughput, data-intensive workloads that are common today.

To meet these demands, a new architectural pattern emerged: the log-based message broker. Systems such as Apache Kafka [2] and Redpanda [27] treat data streams as an append-only, immutable log. This design provides high performance, data durability, and the ability for consumers to “replay” messages from any point in history. In production, these brokers operate as a distributed cluster to provide fault tolerance and scalability.

As illustrated in Figure 1, log-based message brokers are organized along the following concepts:

* **Brokers:** These are server processes in charge of storing and managing the logs, handling producer and consumer requests, and ensuring fault tolerance through replication.

<figure><img src="image_4.png" /><figcaption>Figure 1: Kafka's architecture (adapted from [19]).</figcaption></figure>

* **Producers and Consumers:** Applications that publish and subscribe to messages, often organized into consumer groups for workload distribution.

* **Topics:** Each message is associated with a topic, which is a logical channel for data distribution and partitioning.

* **Partitions:** Each topic is divided into partitions, the core unit of parallelism and placement. Producers append to a partition in FIFO order; consumers read from a partition sequentially. No guarantee is provided regarding the delivery order of messages processed in different partitions, even if they are tagged to the same topic. Brokers assign partition leaders to threads/cores (e.g., via I/O and network thread pools in Kafka or thread-per-core in Redpanda), enabling parallel disk and network pipelines. Higher partition counts increase parallelism (more leaders, more outstanding I/O) and can reduce head-of-line blocking, but also add overheads (file handles, metadata, open segments, background compaction) which eventually dominate. In practice, production deployments span from a handful of partitions (ordering-critical topics) up to thousands per broker (high fan-out and throughput), trading throughput gains against management overhead and memory/CPU pressure.

* **Replication:** Each <topic; partition> pair is replicated for fault tolerance, according to the topic's replication factor. For each partition, one broker acts as the leader handling read and write operations, whereas the others act as followers, replicating the data for redundancy.

## 2.2 A Comparison of Kafka and Redpanda

We study Apache Kafka and Redpanda as they represent the most prominent implementations of the log-based model. Because Redpanda implements the Kafka API, it can be used as a drop-in replacement, allowing for a direct comparison of how their underlying architectural differences affect performance and resource utilization. Notably, these systems have converged considerably in recent years, with both now offering similar features through different implementation paths.

*Implementation and Concurrency Model.* Apache Kafka is implemented in Scala and Java, running on the Java Virtual Machine (JVM). This ensures portability but introduces performance overheads from the JVM runtime, such as garbage collection. In contrast,
Redpanda is a C++ application designed for modern multi-core hardware [33]. It employs a thread-per-core model, which dedicates CPU cores and memory resources to specific tasks, aiming to minimize OS-level context switching and reduce latency [10].

*Metadata Management.* Historically, Kafka relied on an external Apache ZooKeeper cluster for metadata management and leader election, a dependency known to add operational complexity [23]. Redpanda, by design, integrates consensus management directly into the broker using a built-in Raft implementation. However, with the introduction of KRaft in recent Kafka versions, both systems now offer self-contained, Raft-based metadata management [20].

*I/O and Durability Model.* The systems' default approaches to disk I/O also differ. Kafka optimizes for high throughput by writing to the filesystem page cache, relying on the OS to flush data to disk asynchronously. This is fast but makes durability dependent on replication, as data in the cache can be lost on power failure. Conversely, Redpanda prioritizes stronger durability guarantees by default. It uses direct I/O to bypass the page cache and performs an 'fsync' on each record batch, ensuring data is persisted to disk immediately. This choice, too, is no longer a fixed distinction; recent Redpanda versions have introduced a configurable write-caching mode that mimics Kafka's behavior, allowing users to explicitly choose their desired trade-off between throughput and durability [29].

# 3 Experimental Observations

Message broker users typically express their requirements in business-oriented terms, such as the message rates and sizes, number of users, and desired parallelism and fault tolerance.

To determine which hardware and software configurations may or may not be able to fulfill these requirements, we begin by systematically exploring the performance characteristics and bottlenecks of Kafka and Redpanda in a variety of configurations. This task is complex due to a vast parameter space spanning software, hardware, cluster topology, and workload.

## 3.1 Experimental Methodology

To help ensure our experimental findings are reproducible and representative of real-world conditions, we follow a methodology grounded in bare-metal testing. Each experimental run requires provisioning bare-metal nodes, deploying a reproducible software stack, provisioning the brokers, warming up the system, and conducting multi-minute throughput probes—a process that takes hours per configuration. Moreover, before each run, we randomly assign roles (e.g., broker or workload generator) to the available physical nodes. These steps help ensure that we are able to generalize our findings and that our results are not biased by the unique characteristics of any single machine.

Our experiments rely on large, server-grade machines that are expensive and difficult to access, making exhaustive factorial designs prohibitively costly. We therefore adopt a one-factor-at-a-time (OFAT) approach [9], varying a single parameter while holding others constant. Initial tests revealed that due to the highly controlled nature of our setup the results were quite stable between runs. Consequently, we determined that five repetitions per configuration

were sufficient to ensure reproducible findings. This strategy allows us to identify the main performance effects with a manageable number of experiments, which is critical given the operational and resource constraints of our setup.

*Hardware Platforms.* As detailed in Table 1, we use server machines from three different generations: (1) Taurus servers represent legacy infrastructure based on spinning magnetic hard disk drives; (2) Grappe servers represent modern infrastructure with SSD storage; and (3) Advance servers represent premium NVMe-based infrastructure. We sourced these machines from the Grid’5000 testbed [4] and OVHcloud. All experiments exercise one broker per physical server, which is a common practice for I/O-bound workloads.

*Software Frameworks.* We conduct our experiments using Apache Kafka v3.8.0 (via Strimzi v0.44.0 configured with ZooKeeper<fnref n="3" />) and compare it against Redpanda v24.1.8. For Redpanda we enable write caching to ensure an apples-to-apples comparison with the same durability guarantees that Kafka provides.

*Deployment and Configuration.* We leverage established mechanisms for provisioning cluster resources [17], deploying Kubernetes clusters, and configuring brokers. To ensure reproducibility across different hardware, we employ NixOS v24.11 [8] running the Linux kernel v6.6.80 as the uniform foundational software stack on all machines.

To bound the experimental scope, we fix the replication factor (RF=3) and the number of topics (n=1) to isolate hardware and framework effects. Notably, we do not evaluate failure scenarios (e.g., performance during node failure or rebalancing); our focus is on steady-state capacity.

Table 2 summarizes the key experimental parameters used across our study, with variations noted where applicable.

*Performance Metrics and Measurement.* We evaluate the systems based on throughput, latency, and power consumption using the following methods:

* **Workload Generation:** We exercise the systems using synthetic workloads based on modified versions of Kafka's benchmark scripts. These are deployed as parallel Kubernetes jobs that produce messages with random byte string payloads at specified rates and sizes. The producer can be horizontally scaled to ensure sufficient load capacity. These synthetic workloads are parameterized to emulate large-scale IoT-style telemetry streams (small to medium payloads at sustained rates) without the confounding factors of trace-specific burstiness. Our goal is capacity characterization; not reproducing a specific production trace.

* **Maximum Sustainable Throughput (MST):** Let $\lambda$ be the producer send rate (messages/s). We define MST as the largest $\lambda^*$ such that (i) no messages are dropped; and (ii) observable queues/lag remain bounded over the measurement window. We identify $\lambda^*$ via a binary search, increasing $\lambda$

<footnote n="3">³We use Kafka in ZooKeeper mode as it is the default in the Strimzi operator release tested. A preliminary comparison with KRaft revealed negligible differences in maximum sustainable throughput for steady-state ingest. We hypothesize that after the initialization of resources, the "hot path" of the data for a producer-heavy workload does not involve metadata operations.</footnote>
<table><caption>Table 1: Cluster Specifications.</caption><thead><tr><th></th><th>Taurus (HDD cluster)</th><th>Grappe (SSD cluster)</th><th>Advance (NVMe cluster)</th></tr></thead><tbody><tr><th>CPU</th><td>2x6-core Intel Xeon E5-2630</td><td>2x20-core Intel Xeon Gold 5218R</td><td>1x16-core AMD EPYC 4584PX</td></tr><tr><th>Memory</th><td>32 GiB</td><td>96 GiB</td><td>128 GiB</td></tr><tr><th>Storage</th><td>299 GB HDD (Dell PERC H710)</td><td>480 GB SSD (Intel SSDSC2KG480G8R)</td><td>1 TB NVMe (Micron 7450 MTFDKCC960TFR)</td></tr><tr><th>Network</th><td>10 Gbps</td><td>25 Gbps</td><td>25 Gbps</td></tr><tr><th>Release Year</th><td>2012</td><td>2020</td><td>2021</td></tr></tbody></table>

<table><caption>Table 2: Key Experimental Parameters.</caption><thead><tr><th>Parameter</th><th>Value / Range</th></tr></thead><tbody><tr><td><strong>Software</strong></td><td></td></tr><tr><td>Kafka version</td><td>v3.8.0 (Strimzi v0.44.0)</td></tr><tr><td>Kafka metadata mode</td><td>ZooKeeper</td></tr><tr><td>Redpanda version</td><td>v24.1.8</td></tr><tr><td>Redpanda cache mode</td><td>Enabled</td></tr><tr><td>Platform</td><td>NixOS v24.11, Linux v6.6.80</td></tr><tr><td><strong>Workload</strong></td><td></td></tr><tr><td>Message size</td><td>4 KiB (32 B to 512 MB explored)</td></tr><tr><td>Replication factor</td><td>3 (fixed)</td></tr><tr><td>Number of topics</td><td>1 (fixed)</td></tr><tr><td>Partitions per broker</td><td>60 (18–3000 explored)</td></tr><tr><td>Workload type</td><td>Synthetic IoT-style telemetry</td></tr><tr><td><strong>Resource Allocation</strong></td><td></td></tr><tr><td>CPU cores per broker</td><td>8 (1–10 explored)</td></tr><tr><td>Memory per broker</td><td>16 GiB (1–25 GiB explored)</td></tr><tr><td>Number of brokers</td><td>3 (1–10 explored)</td></tr><tr><td>Deployment model</td><td>Kubernetes, 1 broker per node</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 2: Example of maximum sustainable message rate for different message sizes. The tested configuration can safely sustain any workload within the green area.</figcaption></figure>

until message drops or unbounded lags are detected, then backing off to the highest stable setting. Figure 2 illustrates the delineation between stable (sustainable) and unstable (message-dropping) operating regions for various message sizes.

* **End-to-End Latency:** We measure client-experienced latency metrics, including acknowledgment latency and round-trip time, by continuously sending and receiving messages on a dedicated topic using KMinion [28].

* **Power Consumption:** The gold standard for measuring power consumption is hardware wattmeters, which were not available across all of the hardware configurations tested. Since previous research has confirmed the high accuracy of RAPL-based software power meters [15], we instead use Scaphandre [14] to measure the total host power consumption. The available RAPL domains cover CPU package and DRAM but not full platform power. We therefore treat RAPL-derived power as a proxy for node power and focus on comparative trends across configurations, not absolute data center power.

## 3.2 Impact of Workload Characteristics

### 3.2.1 Impact of Message Size.
Applications of message brokers span a wide range of use cases, from small control messages to large data payloads. We now investigate how message size affects message rate (in messages/sec) and data throughput (in bytes/sec)

<figure><img src="image_6.png" /><figcaption>Figure 3: Data throughput vs. message size (Kafka, Advance Cluster, 3 brokers).</figcaption></figure>

by varying the message size from 32 B (the smallest useful message size) to 512 MB (the default maximum message size is 1 MB).

Figure 2 shows a near-inverse relationship between message size and message rate. However, Figure 3 reveals a more complex relationship where larger message sizes lead to greater data throughput. This can be explained by more efficient network packet utilization and improved buffering mechanics for larger message sizes.

Based on these findings and practical constraints in our experimental setup, we standardize on a 4096 B (4 KiB) message size in
<figure><img src="image_7.png" /><figcaption>**Figure 4: Impact of workload partitioning with 3 brokers (Advance cluster).**</figcaption></figure>

subsequent experiments. This size represents a pragmatic com-
promise that allows us to meaningfully saturate all hardware con-
figurations while still representing a realistic IoT workload. With
very small messages (e.g., 32 B), we found that the producer clients
themselves became the bottleneck before the broker infrastructure,
preventing us from accurately measuring the true capacity limits
of the systems under test.

3.3 Impact of Partition Count

Partitions are a way for message brokers to balance their load among multiple brokers. However, too large a number of partitions may also reveal system inefficiencies. Figure 4 shows the impact of the number of partitions on system throughput for Kafka and Redpanda on the Advance cluster.

We see that increasing partition count initially improves through-
put, but with diminishing returns. Since disk writes are batched per
producer-partition pair, excessive partitioning can increase resource
contention despite being technically supported. The single-partition
case represents minimal parallelism, though some workloads re-
quire it for ordering guarantees.

Both Kafka and Redpanda see stable throughput across 18–300
partitions per broker, a typical production range. Subsequent ex-
periments standardize on 60 partitions per broker.

3.4 Choosing the Right Framework – Kafka vs. Redpanda

Kafka and Redpanda exhibit distinct performance characteristics
across different hardware configurations. Figure 5 summarizes
Kafka's and Redpanda's throughput on all three hardware con-
figurations. Redpanda seems to be particularly designed to exploit
the properties of NVMe storage, as it outperforms Kafka on the
Advance cluster. However, on the HDD-based Taurus cluster, Red-
panda performs significantly worse than Kafka. On Grappe, the two
frameworks have similar performance, with a small advantage for
Kafka. This suggests that Redpanda's access patterns are designed

<figure><img src="image_8.png" /><figcaption>Figure 5: Kafka's and Redpanda's throughput on the 3 clusters.</figcaption></figure>

for high-speed disk technology, such as NVMe, regardless of CPU
or memory allocation. Kafka is less sensitive to storage technology.

3.5 Impact of Vertical Scaling – CPU and Memory

We now evaluate the impact of quantity of available CPU and mem-
ory resources. By limiting the number of CPU cores and allocated
memory using *cgroups*, we can investigate the impact on message
broker performance. For Redpanda, we investigate only configura-
tions with at least 1 GiB of memory per CPU core, as this is enforced
by the framework during its initialization.

Figure 6 shows that adding CPU cores in the Grappe cluster
brings diminishing returns beyond two cores. This is also true in
the other clusters, suggesting that message brokers are primarily
I/O-bound rather than CPU-bound.

In real-world deployments with multiple concurrent consumers
that potentially lag behind producers, memory requirements in-
crease as messages are buffered longer, while maintaining many
client connections demands more CPU resources for connection
handling. These factors could alter the scaling characteristics we
observed.

Based on these findings, we standardize on 8 CPU cores and
16 GiB of memory for all subsequent experiments, providing suf-
ficient resources while also ensuring consistency across different
hardware configurations.

3.6 Impact of Horizontal Scaling – Number of Brokers

A classical technique to increase message broker throughput is to
add more brokers.

Figure 7 displays throughput as a function of the number of bro-
ker servers in the Grappe cluster. Both Kafka and Redpanda demon-
strate near-linear scaling behavior, which aligns with theoretical
expectations. As long as partitions and topics are evenly distributed
among brokers, message broker systems represent highly scalable
mechanisms due to the parallelizable nature of their workloads
and how they can make near-optimal use of available disk and
network throughput.
<figure><img src="image_9.png" /><figcaption>Figure 6: Impact of vertical scaling (Grape cluster, 3 brokers).</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 7: Impact of horizontal scaling (Grape Cluster).</figcaption></figure>

Both frameworks show similar scaling patterns on the SSD-based
Grape cluster, where Kafka maintains a consistent throughput
advantage over Redpanda. This performance gap becomes more
pronounced as the broker count increases, with Kafka reaching ap-
proximately 1600 MB/s and Redpanda 1400 MB/s at the ten-broker
configuration. The parallel slopes in the graph demonstrate that
both systems can effectively distribute workloads across additional
nodes without introducing significant coordination overhead.

Note that we could not evaluate horizontal scaling on the Ad-
vance cluster due to limited access to expensive servers. However,
based on our observations on other clusters, we expect similar
linear scaling behavior on NVMe storage.

These findings have strong implications for capacity planning:
organizations can confidently address throughput requirements
by adding broker nodes, with each additional node contributing a
predictable increase to the overall throughput.

3.7 End-to-end Message Latency

While a system’s maximum sustainable throughput is a crucial
indicator of its capacity, end-to-end message latency is equally
important for real-time applications. We decided not to evaluate
the end-to-end latency at maximum system throughput as internal
queues begin to fill, causing latency to increase unboundedly—a
phenomenon often referred to as *bufferbloat*.

To provide a fair comparison between Kafka and Redpanda, we
measure the end-to-end message latency under a controlled, identi-
cal load for both systems. We set this load to approximately 30% of
the maximum sustainable throughput of the *slower* of the two sys-
tems on each hardware configuration. This approach ensures that
both brokers operate comfortably within their stable performance
regions, enabling a direct comparison of their inherent latency char-
acteristics under an equivalent, moderate load. We report the 99th
percentile (p99) end-to-end latency, a worst-case metric critical for
applications with strict service-level objectives (SLOs).

Figure 8 shows the p99 latency for Kafka and Redpanda across
our three hardware configurations. On SSD (Grape) and NVMe
(Advance) clusters, Redpanda maintains lower latencies. This ad-
vantage disappears on HDD-based systems (Taurus cluster) where
mechanical disk limitations dominate.

For latency-sensitive applications requiring consistent perfor-
mance under high load, Redpanda deployed on SSD or NVMe stor-
age clearly offers superior characteristics.

3.8 Power Consumption

Energy efficiency is now a critical factor in data center operations,
both for operational cost control and environmental sustainability
goals.

Figure 9 depicts the aggregate power consumption of three bro-
ker servers in the Advance cluster as a function of system load,
by varying the load in 1% increments from 0% to 10% and then
<figure><img src="image_11.png" /><figcaption>Figure 8: End-to-end message latency.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 9: Power consumption as a function of throughput (Advance cluster, 3 brokers).</figcaption></figure>

in 10% increments from 10% to 100% of the maximum sustainable throughput.

Interestingly, the two message broker frameworks behave very differently. Kafka exhibits a linear increase in power consumption up to about 20% of maximum throughput, after which it remains relatively stable regardless of additional load. We believe that this can be attributed to Kafka's architecture using fixed thread pools, which consume resources at a consistent level once they are fully engaged. In contrast, Redpanda shows linear power usage growth up to roughly 50% of maximum capacity, after which it stabilizes.

We observed similar difference in power proportionality in the other clusters, even though the maximum sustainable throughput varies significantly as we have seen before.

# 4 Right-Sizing Message Broker Clusters

Our experimental analysis in Section 3 highlights a number of key findings: (1) Storage technology significantly impacts broker performance, with different systems showing varying efficiency on NVMe vs. HDD storage; (2) Both Kafka and Redpanda demonstrate near-linear throughput scaling with respect to node count; (3) Message brokers are primarily I/O-bound, with limited benefits from vertical CPU scaling; and (4) Energy consumption patterns differ

between brokers, with Kafka showing constant non-idle power draw while Redpanda exhibits near-proportional scaling up to 50% capacity.

These findings allow us to model the performance and energy usage of message broker clusters and predict it for unknown types of hardware. Our methodology takes as input the expected workload characteristics (throughput and message size) and the available hardware configurations (node types, storage options). It then requires one to issue a small number of experimental measurements to assess the processing capacity of the provided machines, their scalability characteristics and their energy profile. Finally, it delivers a small set of optimal configurations that safely handle the expected workload with adequate headroom, and that minimizes energy consumption. A final choice between them would probably also involve other considerations such as financial cost.

## 4.1 Assumptions

Our linear throughput model holds under the following assumptions:

(1) **Homogeneous Clusters:** Broker nodes within a cluster are homogeneous (same CPU, storage, and network).

(2) **Fixed Replication Factor:** The replication factor is fixed at 3 unless stated otherwise.

(3) **Stable Partition Range:** The number of partitions per broker is within the typical range where throughput is stable (as identified in Section 3).

(4) **Consistent Message Size:** The model assumes a given message size or a size-adjusted calibration (per Section 4.2).

When any of these parameters change materially (e.g., a different replication factor or significantly different workload patterns), we recommend re-running the minimal calibration.

## 4.2 Calibration Experiments

The first step requires administrators to assess the capabilities of the provided hardware. This involves conducting a minimal set of experiments to establish performance scaling characteristics and energy consumption patterns:

(1) **Measuring Idle Power:** Measure power consumption ($P_{idle}$) for a set of nodes with the brokers installed but no workload.

(2) **Measuring Scaling:** Measure maximum sustainable throughput on clusters of 3 and 4 nodes<fnref n="4" />. This choice represents a pragmatic trade-off. A minimum of 3 brokers is required for a fault-tolerant production setup, making it a realistic baseline. Using just two data points (3 and 4 nodes) is the minimum required to derive a linear scaling model, thus minimizing the time and cost of the calibration process for practitioners. While more data points would yield a more robust model, our validation in Section 5 shows that this simplified approach is sufficient for reasonably accurate predictions.

* The message size can be set to 4 kB. If the expected workload message size is known in advance, we can

<footnote n="4">⁴If a choice between Kafka and Redpanda has not already been made, a separate measurement of each framework under these conditions is necessary.</footnote>
either make use of results from Section 3.2.1 to adjust for the expected throughput, or run the experiments with the expected message size (keeping in mind that small message sizes can lead to producer saturation rather than broker saturation).

* While measuring the system at full load, we simultaneously record energy consumption metrics during these runs to obtain $P_{max}$ (the per-server power consumption at full load).

## 4.3 Linear Performance Modeling

Based on our findings we can model the expected throughput $T(n)$ for an n-node cluster using a linear model:

$$T(n) = a \cdot n + b \quad (1)$$

Using our earlier calibration points, we can then fit the parameters $a$ and $b$ for this given hardware configuration.

It is worth noting that when using only two measurement points to derive a linear model, any noise or anomalies in those initial measurements will directly affect the model's accuracy. If possible, multiple iterations of the calibration experiments should be run to ensure that the results are consistent and the deviations remain within acceptable limits.

## 4.4 Calculating Optimal Cluster Size

To determine the minimum number of servers required for a specific target throughput, we can simply solve Equation 1 and round the result up to the nearest integer, considering that we can only deploy on entire nodes:

$$n_{min} = \left\lceil \frac{T_{target} - b}{a} \right\rceil \quad (2)$$

## 4.5 Energy Consumption Modeling

We model the power requirements as a linear function of the number of nodes:

$$E(n) = n \cdot P_{max} \quad (3)$$

where $n$ is the number of nodes and $P_{max}$ is the power consumption at maximum load for each node. We assume homogeneous nodes so that $P_{max}$ is identical across brokers.

This linear energy model reflects our findings: at high utilization, both Kafka and Redpanda exhibit power consumption proportional to the number of nodes. Although energy scaling differs at low utilization, right-sized clusters typically operate at medium to high loads, making these differences negligible.

For workloads with significant idle periods, energy use can be estimated by interpolating between $P_{idle}$ and $P_{max}$ from calibration. Alternatively, splitting workloads across multiple smaller clusters can help maintain higher average utilization and improve energy efficiency.

# 5 Validation

To validate our right-sizing model, we apply our methodology to a new cluster from the Grid'5000 testbed called Ecotype. This cluster differs from the previous ones by the fact that its storage relies on an SAS interface for SSD storage. For reference, we also apply the same methodology to the Taurus and Grappe clusters.

<table><caption>Table 3: Derived Linear Model Parameters.</caption><thead><tr><th>System</th><th>Line Equation</th><th>R<sup>2</sup></th></tr></thead><tbody><tr><td>Ecotype-Redpanda</td><td>T(n) = 210.07n - 110.65</td><td>0.678</td></tr><tr><td>Grappe-Redpanda</td><td>T(n) = 132.60n + 51.35</td><td>0.992</td></tr><tr><td>Taurus-Kafka</td><td>T(n) = 44.44n + 30.36</td><td>0.955</td></tr><tr><td>Ecotype-Kafka</td><td>T(n) = 279.51n - 485.94</td><td>0.945</td></tr><tr><td>Grappe-Kafka</td><td>T(n) = 188.54n - 89.74</td><td>0.672</td></tr></tbody></table>

<figure><img src="image_13.png" /><figcaption>Figure 10: Measured vs. predicted throughput scaling for Kafka and Redpanda on the Ecotype cluster.</figcaption></figure>

## 5.1 Throughput Model Validation

We initially benchmarked the Ecotype servers under two configurations with 3 and 4 brokers, and derived linear model parameters from these measurements. We then compare the measured system throughput for other configurations to the model's prediction. Table 3 shows the model parameters, as well as the $R^2$ values. These values show a reasonably good fit between measurements and the models. The model achieves approximately 10% error in predicting throughput across different cluster configurations.

Figure 10 illustrates the measured throughput versus our model's predictions for the Ecotype cluster. It reveals that our model tends to slightly overestimate scaling capacity for this cluster, particularly for Redpanda, while maintaining good prediction accuracy. Interestingly, the actual scaling behavior appears more non-linear than observed in other clusters measurements. Unlike other clusters which maintain the dominance of Kafka or Redpanda across all scales, we see a performance crossover between the two systems in the case of Ecotype. Redpanda demonstrates stronger performance at lower node counts, but Kafka outperforms at greater throughput. Our model successfully captures this intersection point, demonstrating its utility in comparing different message broker systems across various cluster sizes.

## 5.2 Energy Consumption Validation

For a 1,000 MB/s workload with 4 KB messages on the Ecotype-Kafka configuration, our model predicts a power draw of 369.6 Watts for a 6-node configuration. The measured value is 399.4 Watts, representing a prediction error of 7%.
This accuracy validates our energy model for capacity planning. The step-function pattern of energy consumption at higher utilization levels reinforces that right-sizing is critical for energy efficiency, as each new node significantly increases power consumption without proportional throughput benefits.

## 6 Related work

A vast literature explores the performance and runtime adaptation of Data Stream Processing (DSP) systems [6] including Apache Storm, Apache Flink and Kafka Streams, the DSP engine integrated within Kafka [25]. However, DSP systems focus on efficiently analyzing the payload of data stream messages. They usually rely on Kafka for ingesting data from the external world, feeding them to the DSP engine, and ensuring message durability in case of DSP server failures [21].

In contrast, we focus on the Kafka-like brokers themselves. Their purpose is to guarantee the durability of massive volumes of data, therefore any reconfiguration in the set of message broker machines requires rebalancing the partitions' data across the new set of brokers. Doing this without interrupting message ingestion and delivery for the upstream and downstream applications is a very challenging task [30]. For this reason it is important to configure message brokers correctly from the start, and to minimize the number of necessary reconfigurations over the entire system lifecycle.

Some message brokers such as Apache Pulsar aim to improve Kafka's elasticity by removing the need for data rebalancing [11]. However, Pulsar reportedly still falls short of Kafka regarding other important metrics such as system throughput and latency [24].

Some prior work has explored the threshold performance of Kafka [13]. The authors demonstrate that Kafka can support up to 420,000 messages/second on three virtual machines running on commodity hardware. They also highlight non-linear relationships between consumer groups and throughput degradation. However, the paper does not explore the impact of different frameworks such as Redpanda, hardware configurations, message sizes, and scalability beyond three brokers.

A performance model of Kafka was proposed in [34]. This model has inspired ours, but the methodologies differ: the authors first propose a theoretical model based on an understanding of the internal organization of Kafka, then they experimentally evaluate the model using three brokers running on commodity machines. While this provides valuable insights into packet size and metadata overhead in Kafka communications, the empirical validation is limited to a single hardware configuration. In contrast, we preferred first observing the performance characteristics of Kafka and Redpanda through a wide variety of configurations, and then deriving a model from the observations. This allowed us to propose a much simpler model that enables comparisons not only between various message throughput values but also message size, system scale, hardware base as well as energy usage.

Industry often needs to right-size message broker clusters that belong to critical infrastructure which operates 24/7 and where downtime can have significant business impacts. Organizations with sufficient size often develop internal expertise and tooling

in this domain, yet they rarely share their insights publicly. A notable exception is Dropbox, which shared their investigation of Kafka's throughput limits in order to make proper provisioning decisions [16]. Their experimental methodology inspired ours in particular for developing systematic ways to identify system limits and visualizing the safety curve. However, this study remains specific to Dropbox's infrastructure and workload. It also considers only three deployment options to choose from: small (5 brokers), medium (10 brokers) or large (15 brokers).

Finally, some cloud providers develop coarse-grained sizing guidelines. For instance, Amazon Web Services emphasizes network throughput, storage I/O, and inter-broker bandwidth as critical constraints in Kafka clusters [12]. However, these guidelines ignore message broker frameworks, such as Redpanda, and remain specific to deployments using Amazon's *Managed Streaming for Kafka* (MSK) service.

## 7 Conclusion

We have explored the throughput and energy efficiency of Kafka and Redpanda message brokers across various hardware configurations and workload scenarios. Our findings underscore the critical role that hardware selection plays in optimizing both performance and energy consumption. We demonstrated that storage type largely determines system throughput, with NVMe storage outperforming traditional HDDs and SSDs. Redpanda is highly sensitive to storage performance, making it unsuitable for deployment on HDDs. Message size and partitioning strategies also significantly influence throughput: larger messages achieve greater absolute throughput, while partition count has a limited effect on Kafka's performance, highlighting the need for careful workload characterization in deployment planning. The power proportionality analysis indicates that Kafka in particular exhibits non-linear power consumption patterns, with potential for optimization in real-world deployments where systems rarely operate at peak capacity.

These findings led us to a simple throughput model of Kafka and Redpanda for previously unknown hardware configurations. This model can be used to right-size message broker clusters such that they deliver the expected throughput. Our evaluations show predictions with about 10% error on system throughput and 7% on energy usage. These results are sufficiently precise to allow practitioners to right-size their clusters and avoid unnecessary overcapacity or energy consumption.

## Acknowledgments

This work is supported by the "FrugalCloud" Inria and OVHcloud partnership. Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities, as well as other organizations (see https://www.grid5000.fr).

## References

[1] Amazon Web Services, Inc. [n. d.]. Amazon Managed Streaming for Apache Kafka. https://aws.amazon.com/msk/.

[2] Apache Software foundation. [n. d.]. Apache Kafka. https://kafka.apache.org/.

[3] Apache Software Foundation. [n. d.]. Powered By. https://kafka.apache.org/powered-by.

[4] Daniel Balouek, Alexandra Carpen Amarie, Ghislain Charrier, Frédéric Desprez, Emmanuel Jeannot, Emmanuel Jeanvoine, Adrien Lèbre, David Margery, Nicolas
Niclausse, Lucas Nussbaum, Olivier Richard, Christian Pérez, Flavien Quesnel, Cyril Rohr, and Luc Sarzyniec. 2013. Adding Virtualization Capabilities to the Grid'5000 Testbed. In *Cloud Computing and Services Science*, Ivan I. Ivanov, Marten van Sinderen, Frank Leymann, and Tony Shan (Eds.). Communications in Computer and Information Science, Vol. 367. Springer International Publishing, doi:10.1007/978-3-319-04519-1_1

[5] Business Wire. 2024. Redpanda skyrockets in FY24: 300% revenue growth, 179% customer growth and industry-first streaming data advancements. https://www.businesswire.com/news/home/20240206758655/en/Redpanda-Skyrockets-in-FY24-300-Revenue-Growth-179-Customer-Growth-and-Industry-first-Streaming-Data-Advancements.

[6] Valeria Cardellini, Francesco Lo Presti, Matteo Nardelli, and Gabriele Russo Russo. 2022. Runtime Adaptation of Data Stream Processing Systems: The State of the Art. *Comput. Surveys* 54, 115 (Sept. 2022). doi:10.1145/3514496

[7] Digital Ocean, LLC. [n. d.]. Managed Apache Kafka as a Service. https://www.digitalocean.com/products/managed-databases-kafka.

[8] Elco Dolstra and Andres Löh. 2008. NixOS: A purely functional Linux distribution. In *Proc. ACM SIGPLAN*.

[9] Daniel D. Frey, Fredrik Engelhardt, and E.M. Greitzer. 2003. A role for "one-factor-at-a-time" experimentation in parameter design. *Research in Engineering Design* 14, 2 (2003).

[10] Alexander Gallego. 2020. Thread-per-core buffer management for a modern Kafka-API storage system. https://www.redpanda.com/blog/tpc-buffers.

[11] Sijie Guo. 2024. No data rebalance needed! That's why we reimagined Kafka with Apache Pulsar to make it 1000x more elastic. https://streamnative.io/blog/no-data-rebalance-needed-kafka-and-pulsar.

[12] Steffen Hausmann. 2022. Best practices for right-sizing your Apache Kafka clusters to optimize performance and cost. https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-your-apache-kafka-clusters-to-optimize-performance-and-cost/.

[13] Guenter Hesse, Christoph Matthies, and Matthias Uflacker. 2020. How fast can we insert? An empirical performance evaluation of Apache Kafka. In *Proc. ICPADS*. doi:10.1109/ICPADS51040.2020.00089

[14] Hubblo-org. [n. d.]. Scaphandre: Energy consumption metrology agent. https://github.com/hubblo-org/scaphandre.

[15] Mathilde Jay, Vladimir Ostapenco, Laurent Lefevre, Denis Trystram, Anne-Cécile Orgerie, and Benjamin Fichel. 2023. An experimental comparison of software-based power meters: focus on CPU and GPU. In *Proc. IEEE/ACM CCGrid*. doi:10.1109/CCGrid57682.2023.00020

[16] Peng Kang. 2019. Finding Kafka's throughput limit in Dropbox infrastructure. https://dropbox.tech/infrastructure/finding-kafkas-throughput-limit-in-dropbox-infrastructure.

[17] Govind KP, Guillaume Pierre, and Romain Rouvoy. 2023. Studying the Energy Consumption of Stream Processing Engines in the Cloud. In *Proc. IC2E*. doi:10.1109/IC2E59103.2023.00019

[18] Jay Kreps, Neha Narkhede, and Jun Rao. 2011. Kafka: A Distributed Messaging System for Log Processing. In *Proc. NetDB*.

[19] Narayan Kumar. 2020. Kafka Architecture & Internal. https://mail-narayank medium.com/kafka-architecture-internal-d0b3334d1df.

[20] Colin McCabe. 2025. [KAFKA-9119] KIP-500: Replace ZooKeeper with a Metadata Quorum - ASF JIRA. https://issues.apache.org/jira/browse/KAFKA-9119.

[21] Robert Metzger. 2015. Kafka + Flink: A practical, how-to guide. https://wwwveronica.com/blog/kafka-flink-a-practical-how-to.https://www.

[22] Microsoft. [n. d.]. Events hubs. https://azure.microsoft.com/en-us/products/event-hubs.

[23] Neha Narkhede and Flavio Junqueira. 2015. Distributed consensus reloaded: Apache ZooKeeper and replication in Apache Kafka. https://www.confluent.io/blog/distributed-consensus-reloaded-apache-zookeeper-and-replication-in-kafka/.

[24] Alok Nikhil and Vinoth Chandar. 2020. Benchmarking Apache Pulsar, Kafka, and RabbitMQ. https://www.confluent.io/blog/kafka-fastest-messaging-system/.

[25] István Pelle, Bence Szőke, Abdulhalim Fayad, Tibor Cinkler, and László Toka. 2023. A comprehensive performance analysis of stream processing with Kafka in cloud native deployments for IoT use-cases. In *Proc. NOMS*. doi:10.1109/NOMS56928.2023.10154377

[26] Pravega authors. [n. d.]. Pravega – A Reliable Stream Storage System. https://cncf.pravega.io/.

[27] Redpanda. [n. d.]. Redpanda – the streaming data platform for developers. https://www.redpanda.com/.

[28] Redpanda Data Inc. [n. d.]. KMinion: A feature-rich Prometheus exporter for Apache Kafka. https://github.com/redpanda-data/kminion.

[29] Redpanda Data Inc. 2024. Write caching. https://www.redpanda.com/blog/write-caching-performance-benchmark.

[30] Parth Saxena. 2024. Kafka Explained: High Availability and Elastic Clusters (Part 1). https://medium.com/@parthsaxena04/kafka-explained-high-availability-and-elastic-clusters-part-1-e513be2e9dbc

[31] Arman Shehabi, Sarah J. Smith, Alex Hubbard, Alex Newkirk, Nuoa Lei, Md Abu Bakar Siddik, Billie Holecek, Jonathan Koomey, Eric Masanet,

and Dale Sartor. 2024. 2024 United States Data Center Energy Usage Report. Technical Report LBNL-2001637. Lawrence Berkeley National Laboratory. https://eta-publications.lbl.gov/sites/default/files/2024-12/lbnl-2024-united-states-data-center-energy-usage-report.pdf.

[32] WarpStream. [n. d.]. WarpStream – An Apache Kafka compatible data streaming platform. https://www.warpstream.com/.

[33] Noah Watkins. 2022. Engineering Redpanda for multi-core hardware. https://www.redpanda.com/blog/engineering-redpanda-multi-core-hardware.

[34] Han Wu, Zhihao Shang, and Katinka Wolter. 2019. Performance Prediction for the Apache Kafka Messaging System. In *Proc. HPCC*. doi:10.1109/HPCC/SmartCity/DSS.2019.00036