Benchmark Your
Dedicated ApacheKafka® Cluster on
Confluent Cloud
Anna Povzner, Scott Hendricks © 2020 Confluent, Inc.

Table of Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Setting Up Benchmark Clients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Download Your Java Config for the Cluster from the Confluent Cloud UI . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Container/VM Creation for Running Performance Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Benchmark Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Running Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Create Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Execute Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Example Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Topic Cleanup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Performance Troubleshooting Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Benchmarking Your Dedicated Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
What Performance Should I Expect from My Own Workload? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Abstract
Confluent Cloud Dedicated cluster capacity is based on a pricing unit called a CKU(Confluent Unit for Apache Kafka®), where each unit is capable of sustaining 50 MB/s
producer bandwidth and 150 MB/s consumer bandwidth. This white paper reports theresults of the benchmarks we ran on a 2-CKU multi-zone Dedicated cluster, and shows
the ability of a CKU to deliver the stated client bandwidth on AWS, GCP, and Azure
clouds. The paper describes the setup up and execution of our benchmarks in sufficientdetail to make them reproducible.

Introduction
Confluent Cloud Dedicated cluster capacity is based on a pricing unit called a CKU(Confluent Unit for Apache Kafka®). As of April 2020, one CKU is capable of sustaining
50 MB/s producer bandwidth and 150 MB/s consumer bandwidth, where a higher
consumer bandwidth capacity is useful for a publish/subscribe model that lets multiple
consumer groups retrieve their own copy of a topic. One CKU also comes with a set ofupper limits on your workload behavior. Notable upper limits include the maximum
number of simultaneously connected clients (1,000 per CKU) and the maximum
number of topic partitions (3,000 per CKU). Increasing your allocated CKUs will
linearly increase your upper limits. Independent of the number of CKUs, there is also an
upper limit on per-partition producer bandwidth (5 MB/s) and per-partition consumer
bandwidth (15 MB/s).
The specific performance of your clients with a CKU will depend on the behavior ofyour workload. In general, you will not be able to max out all the dimensions of yourworkload behavior and achieve the maximum CKU bandwidth at the same time. For
example, if you reach the partition limit, you will not be able to reach the maximum
CKU bandwidth. The benchmark described in the paper provides an example workloadthat is capable of reaching maximum per-CKU bandwidth. You’ll note that our
example workload uses 108 partitions, fewer than 20 connections, and specific client
configuration. Your own workload is probably different and as a result, your application© 2014-2020 Confluent, Inc.

1

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

bandwidth may be different.
Since the range of possible workload behaviors is large, you may need to benchmark
your own Dedicated cluster with the configurations more relevant to your specificworkload. This paper describes the benchmark that achieves the maximum CKU
bandwidth for all clouds where Confluent Cloud is available: Amazon Web Services(AWS), Microsoft Azure, and Google Cloud Platform (GCP). For benchmarks, we used
off-the-shelf Kafka producer and consumer performance tools, kafka-producer-perftest and kafka-consumer-perf-test. You can use our benchmarks as a baseline toverify the capabilities of your cluster and also as a starting point for benchmarking awider range of workload parameters, such as a different number of partitions andclients, different message sizes, and different client configurations.
To make our benchmarks fully reproducible, this paper first provides a step-by-stepguide for setting up and running the benchmarks that test the capabilities of a 2-CKU
Dedicated cluster, a minimum size for the multi-zone clusters. We present and explain
the results of running these benchmarks on three multi-zone clusters, one per cloud(AWS, GCP, and Azure). Our results show that each setup was able to achieve themaximum 2-CKU bandwidth: 100 MB/s producer and 300 MB/s consumer bandwidth.

Setting Up Benchmark Clients
This section describes the steps for setting up VMs/containers that would run Kafka
performance tools, kafka-producer-perf-test and kafka-consumer-perf-test,
configuring performance tools to be able to access your Dedicated cluster, and theexact commands for running the performance tools. All the steps are relevant for
testing any CKU Dedicated cluster, while our specific examples assume testing a 2CKU cluster.

© 2014-2020 Confluent, Inc.

2

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Download Your Java Config for the Clusterfrom the Confluent Cloud UI
You need to configure performance benchmarks to access your Dedicated cluster.
Since kafka-producer-perf-test and kafka-consumer-perf-test are Java clients,
download your Java config for the cluster from the Confluent Cloud UI as follows.
Select your cluster from the navigation bar:

Click on Tools & client configuration menu:© 2014-2020 Confluent, Inc.

3

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Select Clients at the top:

© 2014-2020 Confluent, Inc.

4

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Generate a New Kafka Cluster API Key & Secret:Expand Java in the list:© 2014-2020 Confluent, Inc.

5

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Check box to show API Keys to replace {{ CLUSTER_API_KEY }} and {{CLUSTER_API_SECRET }} values with the real API Key and Secret you just created:© 2014-2020 Confluent, Inc.

6

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Click Copy to clipboard:© 2014-2020 Confluent, Inc.

7

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Save the contents of the #Kafka config section for later, ignoring every other section.

Container/VM Creation for Running
Performance Benchmarks
The number of containers to run the performance benchmark was chosen as theminimum number of containers that ensures that clients are not a bottleneck, i.e.,there is enough performance capacity for the benchmark producers and consumers topush enough bandwidth to the cluster.
Create two clean confluentinc/cp-kafka containers or VMs for each CKU in AWS or
Azure, or four for each CKU in GCP. These should be running with a minimum of 4cores and isolated from each other and any other heavy workloads. The test is very
CPU intensive, so it is required to run on two separate systems for each CKU on AWS
and Azure, and four separate systems for each CKU on GCP.

© 2014-2020 Confluent, Inc.

8

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent CloudYou should also aim to create these tests in the same cloud region as your Kafka
cluster. This will ensure the numbers match our reported results.

DockerOn Docker, you can run the following command to create a new container runningconfluentinc/cp-kafka. You must note the Container IDs output from eachcommand.

docker run -d confluentinc/cp-kafka tail -f /dev/null

To reproduce the results reported in this paper for a 2-CKU cluster, create fourcontainers on AWS or Azure, and eight containers on GCP.
To get a prompt to run a performance benchmark command in each container, use thefollowing command:

docker exec -it <CONTAINER 1 ID> bash
docker exec -it <CONTAINER 2 ID> bash
docker exec -it <CONTAINER 3 ID> bash
docker exec -it <CONTAINER 4 ID> bash...

Kubernetes
On Kubernetes, you can create a pod by running the kubectl run command. Make sure
you choose a different name for each pod you run.
You may need to tweak the commands below to get them to run on different nodes.
For example, on AWS r5.xlarge (or any other 32GB node type) you can use the--requests 'memory=20Gi' flag to make sure they are on individual nodes. You can© 2014-2020 Confluent, Inc.

9

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloudalso use the --port/--hostport flag combo to schedule these alone, as we have done
below.

kubectl run kafka-benchmark1 \--image confluentinc/cp-kafka \--restart Never \--port 28130 \--hostport 28130 \[ --namespace optional-namespace ] \-- \tail -f /dev/null
kubectl run kafka-benchmark2 \--image confluentinc/cp-kafka \--restart Never \--port 28130 \--hostport 28130 \[ --namespace optional-namespace ] \-- \tail -f /dev/null...

To reproduce the results reported in this paper for a 2-CKU cluster, run the above
commands to create four pods in AWS/Azure or eight pods in GCP:

kafka-benchmark1, …, kafka-benchmark4 [, …, kafka-benchmark8]

To get a prompt to run a performance benchmark command in each container, use thefollowing command:© 2014-2020 Confluent, Inc.

10

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloudkubectl exec -it [ -n optional-namespace ] kafka-benchmark1 -- bash
kubectl exec -it [ -n optional-namespace ] kafka-benchmark2 -- bash
kubectl exec -it [ -n optional-namespace ] kafka-benchmark3 -- bash
kubectl exec -it [ -n optional-namespace ] kafka-benchmark4 -- bash...

Configuration
Important: Copy the configuration you saved from the Confluent Cloud UI to eachsystem in a file called /config, making sure you have replaced the parameterized
sections.
The file should look something like this, but with your actual values.

Benchmark Overview
As our benchmark, we used pre-packaged performance testing tools that ship with
Kafka: kafka-producer-perf-test and kafka-consumer-perf-test. We used thefollowing non-default client configurations and workload specifications:
Configuration / Workload Specification

Setting

Client version

Apache Kafka 2.4 / Confluent Platform
5.4

acks

© 2014-2020 Confluent, Inc.

all

11

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent CloudConfiguration / Workload Specification

Setting

linger.ms

10compression.type

none (default)

Producer record sizevalue = 512 bytes

Number of topic partitions

108

Producer to consumer ratio

1:3

Our benchmarks do not use compression, which is the default client configuration. Thisshows the ability of a CKU to achieve the stated bandwidth over the wire, independentof compression and its resulting compression ratio. We recommend enabling lz4 or
Snappy compression on the producer via compression.type producer configuration,
which would both let you achieve higher client (pre-compression) bandwidth and saveon ingress, egress, and storage costs. The trade-off of using lz4 or Snappy compression
is a slightly higher CPU usage on both clients and brokers, while gzip could result in amore significant CPU overhead.
Our benchmark was designed to test a 2-CKU Dedicated cluster. The number of
producers and consumers was the minimum number of clients that were able to pushenough load given the CPU capabilities of the cloud provider’s instance types runningthe benchmark clients. Our benchmark had four producers (eight on GCP) publishingmessages to one topic with 108 partitions, and three consumer groups, each with four
consumers (eight on GCP), consuming from the same topic. We chose a producer toconsumer ratio of 1:3 in order to get the maximum CKU bandwidth (50:150 MB/s per
CKU). You can use other ratios, but you may not get the maximum CKU bandwidth.
The next section will describe how to execute the benchmarks. Notice that consumers© 2014-2020 Confluent, Inc.

12

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

start at the same time as producers, meaning that all consumers will be reading fromthe tail of the log. If your application has non-real-time consumers, which are reading
further back from the log, you may not achieve the full CKU bandwidth because theworkload will become more disk bound.
We selected 108 partitions as an example of not too small, not too large. You will needat least 10x the number of CKUs partitions to achieve the maximum CKU bandwidth,to account for per-partition bandwidth limits.
We used producer configuration linger.ms=10 (default is 0) to ensure good batchingof records, which optimizes throughput. We usually recommend linger.ms between
five and 10 milliseconds if you care more about latencies, and up to 100 milliseconds if
you care about throughput but have a lot of clients and partitions which may reduce
the ability of each client to batch records together. The record size of 512 bytes was
also chosen as large enough to optimize for throughput.
We chose to benchmark a highly fault-tolerant client configuration, because we expect
it to be a more common scenario for production workloads: producer configurationacks=all (default is 1). Compared to the default, less fault-tolerant configuration, youwill see higher producer latencies, because the producer has to wait for the message tobe replicated to all the partition’s replicas before receiving the acknowledgment.
We provide more suggestions on how to validate performance and optimize yourworkload at the end of this paper.

Running Benchmarks
This section describes the exact set of steps and commands that we used forbenchmarking a 2-CKU cluster.

Create Topic
On only one of the Docker containers or Kubernetes Pods, run the following command© 2014-2020 Confluent, Inc.

13

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloudto create the topic. Make sure to replace <BOOTSTRAP SERVER>:<PORT> with theactual bootstrap server from your configs.

kafka-topics \--bootstrap-server <BOOTSTRAP SERVER>:<PORT> \--command-config /config \--create \--topic perf-test \--partitions 108 \--replication-factor 3

Execute Benchmark
To execute the tests, on each system add the following section to a script and run
them all simultaneously, staggered by a couple seconds. Make sure to replace<BOOTSTRAP SERVER>:<PORT> with the actual bootstrap server from your configs.
Because the final output from the consumers is not a summary of the test, we mustuse awk to calculate the running average of the entire test. The awk command sums theincremental fetch.time.ms (column $8), and takes the final value for the running sumof data.consumed.in.MB (column $3). It then calculates MB/s based on those twovalues.
You must delete and re-create the topic for every rerun due to the way

the consumers are configured here. Also, be sure the tests are notstarted at identical times on different servers to avoid consumer grouprace conditions.

AWS and Azure
These tests are designed to run up to 30 MB/s ingress, 90 MB/s egress per test
container/VM to the Kafka cluster. Please note, traffic will likely not be equal among all© 2014-2020 Confluent, Inc.

14

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloudcontainers/VMs. Some may run at a higher rate on one and a lower rate on another.

kafka-producer-perf-test \--topic perf-test \--record-size 512 \--producer.config config \--throughput 60000 \--num-records 54000000 \--producer-props acks=all linger.ms=10 \| tee /tmp/producer &kafka-consumer-perf-test \--broker-list <BOOTSTRAP SERVER>:<PORT> \--consumer.config /config \--topic perf-test \--messages 53950000 \--group=perf-test-1 \--show-detailed-stats \--hide-header \--timeout 60000 \| tee /tmp/consumer1 &kafka-consumer-perf-test \--broker-list <BOOTSTRAP SERVER>:<PORT> \--consumer.config /config \--topic perf-test \--messages 53950000 \--group=perf-test-2 \--show-detailed-stats \--hide-header \--timeout 60000 \| tee /tmp/consumer2 &kafka-consumer-perf-test \--broker-list <BOOTSTRAP SERVER>:<PORT> \© 2014-2020 Confluent, Inc.

15

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud--consumer.config /config \--topic perf-test \--messages 53950000 \--group=perf-test-3 \--show-detailed-stats \--hide-header \--timeout 60000 \| tee /tmp/consumer3 &wait
echo "Test Results:"
tail -n 1 /tmp/producer | sed 's|.(\(.\))|Producer: \1|g'echo "Consumer 1:" \`cat /tmp/consumer1 \| awk -F"," '{if($8>0){msec+=$8};mb=$3}END{print mb*1000/msec}'`\"MB/sec"
echo "Consumer 2:" \`cat /tmp/consumer2 \| awk -F"," '{if($8>0){msec+=$8};mb=$3}END{print mb*1000/msec}'`\"MB/sec"
echo "Consumer 3:" \`cat /tmp/consumer3 \| awk -F"," '{if($8>0){msec+=$8};mb=$3}END{print mb*1000/msec}'`\"MB/sec"

GCP
Since we run GCP with more client nodes, we need to decrease the amount of traffic
sent per node to keep the test at 15 minutes. These tests are designed to run up to 15
MB/s ingress, 45 MB/s egress per test container/VM to the Kafka cluster. Please note,some containers or VMs may run at a higher consumer rate on one and a lower
consumer rate on another.

© 2014-2020 Confluent, Inc.

16

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloudkafka-producer-perf-test \--topic perf-test \--record-size 512 \--producer.config config \--throughput 30000 \--num-records 27000000 \--producer-props acks=all linger.ms=10 \| tee /tmp/producer &kafka-consumer-perf-test \--broker-list <BOOTSTRAP SERVER>:<PORT> \--consumer.config /config \--topic perf-test \--messages 26950000 \--group=perf-test-1 \--show-detailed-stats \--hide-header \--timeout 60000 \| tee /tmp/consumer1 &kafka-consumer-perf-test \--broker-list <BOOTSTRAP SERVER>:<PORT> \--consumer.config /config \--topic perf-test \--messages 26950000 \--group=perf-test-2 \--show-detailed-stats \--hide-header \--timeout 60000 \| tee /tmp/consumer2 &kafka-consumer-perf-test \--broker-list <BOOTSTRAP SERVER>:<PORT> \--consumer.config /config \--topic perf-test \--messages 26950000 \© 2014-2020 Confluent, Inc.

17

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud--group=perf-test-3 \--show-detailed-stats \--hide-header \--timeout 60000 \| tee /tmp/consumer3 &wait
echo "Test Results:"
tail -n 1 /tmp/producer | sed 's|.(\(.\))|Producer: \1|g'echo "Consumer 1:" \`cat /tmp/consumer1 \| awk -F"," '{if($8>0){msec+=$8};mb=$3}END{print mb*1000/msec}'`\"MB/sec"
echo "Consumer 2:" \`cat /tmp/consumer2 \| awk -F"," '{if($8>0){msec+=$8};mb=$3}END{print mb*1000/msec}'`\"MB/sec"
echo "Consumer 3:" \`cat /tmp/consumer3 \| awk -F"," '{if($8>0){msec+=$8};mb=$3}END{print mb*1000/msec}'`\"MB/sec"

Example Output
For each test, you should see streaming output while the test is running. This can belargely ignored, but designed to show progress is being made:© 2014-2020 Confluent, Inc.

18

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

249231 records sent, 49746.7 records/sec (24.29 MB/sec), 14.1 ms avg latency, 212.0
ms max latency.
2020-03-26 17:47:20:530, 0, 493.1592, 24.1315, 1009990, 49421.2315, 0, 5002, 24.1315,
49421.2315
2020-03-26 17:47:20:630, 0, 496.1494, 24.3395, 1016114, 49847.2306, 0, 5001, 24.3395,
49847.2306
2020-03-26 17:47:25:157, 0, 605.2256, 24.0767, 1239502, 49309.0000, 0, 5000, 24.0767,
49309.0000
251874 records sent, 50364.7 records/sec (24.59 MB/sec), 18.3 ms avg latency, 149.0
ms max latency.
2020-03-26 17:47:25:550, 0, 616.1587, 24.5019, 1261893, 50179.8805, 0, 5020, 24.5019,
50179.8805
2020-03-26 17:47:25:630, 0, 619.0649, 24.5831, 1267845, 50346.2000, 0, 5000, 24.5831,
50346.2000
2020-03-26 17:47:30:157, 0, 729.5664, 24.8682, 1494152, 50930.0000, 0, 5000, 24.8682,
50930.0000
250346 records sent, 50059.2 records/sec (24.44 MB/sec), 11.6 ms avg latency, 130.0
ms max latency.
2020-03-26 17:47:30:550, 0, 739.1626, 24.6008, 1513805, 50382.4000, 0, 5000, 24.6008,
50382.4000
2020-03-26 17:47:30:630, 0, 739.8574, 24.1585, 1515228, 49476.6000, 0, 5000, 24.1585,
49476.6000
2020-03-26 17:47:35:160, 0, 851.5815, 24.3884, 1744039, 49947.4315, 0, 5003, 24.3884,
49947.4315

Once the test finishes, the output should resemble this:

Producer: 24.79 MB/sec, 16.88 ms avg latency, 1851.00 ms max latency, 7 ms 50th, 57
ms 95th, 211 ms 99th, 566 ms 99.9th.
Consumer 1: 24.7757 MB/sec
Consumer 2: 24.7930 MB/sec
Consumer 3: 24.8054 MB/sec

This shows 25 MB/s ingress to the cluster and 75 MB/s egress for this script. Add up
the summary output from all the scripts to get the final total.

© 2014-2020 Confluent, Inc.

19

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Topic Cleanup
To clean up and restart, run this command to delete the topic and start over at the topof this section. Make sure to replace <BOOTSTRAP SERVER>:<PORT> with the actual
bootstrap server from your configs.

kafka-topics \--bootstrap-server <BOOTSTRAP SERVER>:<PORT> \--command-config /config \--topic perf-test \--delete

Results
We ran the benchmarks described in the previous section on 2-CKU multi-zone clusters
created in AWS, GCP, and Azure clouds. 2 CKUs are capable of sustaining 100 MB/s
producer bandwidth and 300 MB/s consumer bandwidth, but may support larger
bandwidth bursts. The bursts may vary over time, and while our results show slightly
higher than sustained CKU bandwidth, such bursts are not guaranteed.
In order to demonstrate the maximum throughput we could achieve with ourbenchmark from a 2-CKU cluster, we configured producers in the benchmarks to
produce with the rate slightly higher than sustained 2-CKU rate: 120 MB/s. Keep this inmind when looking at resulting producer latencies, because clients producing with therate higher than the system’s sustained rate causes longer delays on clients. So, if you
care about short latencies, make sure that your clients do not try to send more
bandwidth than your maximum sustained CKU bandwidth.
The table below reports total producer and consumer bandwidth achieved by runningthe benchmark on each of the cloud types: AWS, GCP, and Azure.

© 2014-2020 Confluent, Inc.

20

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud

Cloud

Producer bandwidth(MB/s)

Consumer bandwidth(MB/s)

AWS

117

351

GCP

106

318

Azure

106

318

We were able to achieve higher bandwidth on AWS because periodic log flushes to disk
were absorbed by EBS burst balance. On other clouds, bandwidth fluctuated between
about 100 MB/s and 150 MB/s, where log flushes to disk caused dips in bandwidthfollowed by bandwidth spikes due to clients catching up to their throughput rate. In allenvironments, consumers were able to keep up with producers, resulting in 3x the
producer bandwidth (3 consumer groups).

Performance Troubleshooting Tips
Benchmarking Your Dedicated Cluster
If you try our benchmark on your own 2-CKU cluster and get somewhat worse
bandwidth than 100 MB/s producer and 300 MB/s consumer bandwidth, make surethat:
• The hypervisors, Docker hosts, or Kubernetes nodes are not overloaded. For
example, multiples of your benchmark containers are not placed on the samenode.
• Your client benchmarks are not set up in the remote region or not going throughan overloaded network or proxy.

© 2014-2020 Confluent, Inc.

21

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloud• On AWS, it is not caused by a warm-up time for Elastic Load Balancer (ELB), in
which case, try re-running the benchmark.
If your cluster is a different size than 2-CKU, you should also be able to get a CKU
maximum bandwidth by creating the appropriate number of benchmark
containers/VMs as described in the "Setting Up Benchmark Clients" section: twice thenumber of CKUs on AWS and Azure, and four times the number of CKUs on GCP. On
each container, execute performance testing tools exactly as described in this paper
for a 2-CKU cluster.
If you checked the above and you are still not seeing 50 MB/s ingress and 150 MB/s
egress per CKU, please contact Confluent Support for assistance.

What Performance Should I Expect fromMy Own Workload?
Your specific performance will depend on many factors including number of clients,
partitions, producer partitioning strategy, rate of the connection attempts from yourapplication layer, and so on. If you want to maximize the bandwidth you get from a
CKU:• Make sure that your clients are not a bottleneck, i.e., they are capable of pushingthe desired load.
• Keep in mind there is a per-partition bandwidth limit. The limit is soft in a sensethat it may be possible to get more than the limit, but achieving a CKU
bandwidth may require a minimum number of partitions based on the CKU
partition limits.
• You should be able to achieve the maximum CKU bandwidth if your workload isreasonably efficient in using bandwidth, without requiring too many requests orconnection attempts. In other words, if your workload behavior starts
approaching one or more upper limits defined by the number of your CKUs, your© 2014-2020 Confluent, Inc.

22

Benchmark Your Dedicated Apache Kafka® Cluster on Confluent Cloudclient bandwidth may decrease as the workload starts saturating CKU processing
capacity. In this scenario, you may either improve the behavior of your clients orpurchase more CKUs. In general, two CKUs will deliver twice the bandwidth of
one CKU.
We recommend the following reading material for further information on designing
your client applications for better throughput and/or latency:• 99th Percentile Latency at Scale with Apache Kafka: while this paper focuses onlatency, architecting for low latency also makes your clients more efficient withusing bandwidth, which is also discussed in the paper.
• Optimizing Your Apache Kafka Deployment provides a comprehensive list ofguidelines for configuring your Kafka deployment to optimize for various goals:
throughput, latency, durability, and availability.

© 2014-2020 Confluent, Inc.

23