## 5 Performance Evaluation
### Experimental Results
#### 5.2.2 Kafka Broker Virtual Network Function Experiment  
The second experiment makes use of both servers. One of them is used to generate traffic by means of the Sangrenel tool [46], which is a well-known Kafka client. The other one is used to receive and process it using a Kafka broker [45], implemented in the VNF under test. The workload produced by Sangrenel consists of a maximum of 20 topics, with a resulting traffic rate approximately equal to 102.4 Mbps. On average, the traffic rate associated with each topic is 5.12 Mbps. The Kafka message size is 800 B. We used constant rate traffic because our goal was to evaluate the peak rate that is sustainable for a long time using our solution with the test application (worst case). This allows for the provision of additional insights on the service capacity of the implemented VNF. In our experiments, the duration of each test was 5 min. For this application, we considered the following data plane KPIs:  
* _I/O message rate_: The handled throughput normalizes to the publication rate. Specifically, it represents the fraction of traffic that is correctly handled by a VNF. A Kafka  
Figure 8: RAM usage and boot time during NetBricks Kata container deployment using QEMU and Firecracker hypervisors. Red lines refer to Firecracker RAM usage, and blue lines refer to QEMU RAM usage. Dotted lines indicate memory consumption for simple container deployment, whereas continuous lines represent RAM usage overhead required by container deployment followed by a VNF call. The two dash–dotted vertical lines show the average container boot time for Firecracker (red) and QEMU (blue). message is considered undelivered if the broker does not return the corresponding ACK message back to the Sangrenel client by the timeout. * _CPU consumption_: the percentage of a CPU core consumed by the VNF and measured on Server #1 using a CPU collector based on the _mpsat_ command. * _Batch write latency_ or simply _latency_: The time spent by the client while waiting for an ACK message from the Kafka broker. Since the Kafka client and broker run in two servers, located in the same rack and belonging to the same 10G LAN, the message propagation time is negligible. It is thus possible to estimate the service response time. In order to minimize latency, in our experiments, the batch size was set as equal to 1 Kafka message. The first performance we analyzed was the I/O message rate, since it allowed us to evaluate the maximum number of topics that could be handled by our VNF running in a Kata container over a QEMU/Firecracker microVM. Figure 9 shows the value of the I/O message rate as a function of the number of Kafka topics used, along with the relevant 95% confidence intervals, which appeared to be almost negligible. The first comment is that the two solutions--that is, Kata + QEMU and Kata + Firecracker--performed in a similar way. In fact, although the blue line, representing Kata + QEMU, was steadily higher than the Kata + Firecracker curve, this happened for values that were not acceptable; that is, for more than five Kafka topics. For instance, for 10 topics, corresponding to a traffic rate of about 51.2 Mbps, the message loss for Kata + QEMU was equal to 3.5%, which was definitely excessive. Instead, up to five topics, corresponding to a rate equal to 25.6 Mbps, which is in line with most of the applications considered in previous sections, the message loss for Kata + QEMU was 0.17%. This was very close to that of Kata + Firecracker, which was equal to 0.41%. Both results were acceptable. It is worth noting that, for one topic, corresponding to 5.12 Mbps, the I/O message rate was equal to 100% of delivered messages. Finally, it is important to point out that this application needs to be written on a disk, which is critical for this virtualization environment. In fact, the storage was not implemented with SSD devices, which is an intrinsic limitation of Firecracker regarding access to storage. It would clearly perform better for other applications.