# Benchmarking Message Queues
## 4 Methodology
\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{**References**} & \multicolumn{6}{c|}{**Message Queues**} & \multicolumn{6}{c|}{**Metrics**} \\ \cline{2-10}  & **Redis** & **Kafka** & **Active MQ** & **RabbitMQ** & **Pulsar** & **Latency** & **Throughput** & **Persistence** & **Other Metric** \\ \hline [PERSON] et al. [11] & & & & & & & & & ✓ \\ \hline [PERSON] et al. [15] & & & ✓ & & & & & ✓ & \\ \hline [PERSON] et al. [18] & & ✓ & & ✓ & & ✓ & ✓ & & ✓ \\ \hline [PERSON] et al. [19] & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & & \\ \hline [PERSON] et al. [21] & & ✓ & ✓ & & & ✓ & ✓ & & \\ \hline [PERSON] et al. [23] & & & ✓ & ✓ & & & & ✓ \\ \hline [PERSON] et al. [24] & & ✓ & & ✓ & & & & ✓ \\ \hline Our study & ✓ & ✓ & ✓ & ✓ & & ✓ & ✓ & & \\ \hline \end{tabular}
\end{table} 
Table 1: Comparison: our study vs. existing studies. determining the system's overall performance. Therefore, having throughput data in two different forms provides a more comprehensive view of the message queue's performance, taking into account both network bandwidth and message processing efficiency. This can help developers, system architects, and decision-makers to make more informed decisions when selecting or optimizing message queues for specific use cases. For the purpose of running the experiments, all four message queues were installed locally following the official documentation. We used Redis server 7.0.9 [25], RabbitMQ 3.11.10 [26], ActiveMQ Artemis 2.28.0 [27], and Apache Kafka 3.4.0 [28]. We forked [29] the OpenMessaging Benchmark to customize the workloads (data size and the number of messages per second) according to the needs of our experiments. In order to benchmark the four message queues, we used the OpenMessaging Benchmark tool created by The Linux Foundation. It is a highly versatile benchmarking tool written in Java that supports multiple popular message queues and data streaming technologies such as RabbitMQ, ActiveMQ Artemis, Redis, Apache Kafka, Pravega, Pulsar, and many more. It provides a range of pre-built payloads of sizes 100 B, 200 B, 400 B, 1 KB, 2 KB, and 4 KB. Besides these, it also provides the ability to create payloads of other sizes programmatically. Using this tool, we generated detailed metrics for latency and throughput, which are the metrics that we used to compare the four message queues. We used this tool because we wanted a common tool to benchmark the four technologies so as to maintain consistency and eliminate any biases that may arise if we had used benchmarking tools specific to each technology. Using the feature provided by OpenMessaging, we generated data of various sizes. The specifics of how these data sizes were used and how the experiments were conducted are explained in detail in Section 5 (Experiments). After conducting multiple sets of experiments for both throughput and latency, we generated JSON [30] documents containing the results of the experiments. We built Python [31] scripts using the Matplotlib [32] library in order to represent the JSON documents in charts. The scripts can be found under _/bin/charts_ in our forked version [29] of OpenMessaging. We adopted this benchmarking approach for the message queues in our study due to several compelling reasons. Firstly, latency and throughput are crucial metrics that determine the performance of a message queue, making them essential aspects to evaluate. Secondly, the OpenMessaging Benchmark tool, developed by The Linux Foundation, was selected for its credibility in the field of computer science. By utilizing the same tool, we ensure consistent and reliable benchmarking across the four message queues under examination. This approach is vital as using different tools may yield inconsistent results, and some benchmarking tools may exhibit biases when developed by the message queue developers themselves. Furthermore, our benchmarking process generated comprehensive results encompassing various metrics. These metrics include throughput, measured in both the number of events per second and megabytes per second.