## 4 Challenges in Using Non-Automated Methodologies for Performance Evaluation  
From a scientific perspective, the most significant issues when comparing automated and orchestrated performance measurement of Kubernetes CNIs to manual evaluations revolve around several key factors:* Consistency: Manual evaluation is highly susceptible to variability due to human factors. Evaluators may follow slightly different procedures or make unintentional errors, leading to inconsistent results. Automation ensures that procedures are followed continuously, providing consistent and reliable data.
* Scalability: Manual evaluation becomes increasingly impractical as the network environment grows in scale. Large-scale Kubernetes deployments require extensive evaluation across numerous nodes and configurations, which is time-consuming and labor-intensive when done manually. On the other hand, automated evaluation can handle large volumes of tests quickly and efficiently without significant human intervention.
* Reproducibility: Reproducibility is another critical factor; automated evaluation can be easily repeated to verify results or evaluations under different conditions, whereas manual assessment is challenging to replicate with the same precision.
* Human Error: Human error is a persistent issue in manual evaluation, as even the most experienced evaluators can make mistakes that skew results. Automation minimizes the risk of such errors, leading to more accurate and trustworthy outcomes.
* Time Efficiency: Automated evaluation has a significant advantage in terms of time efficiency. Manual evaluation can take days or weeks and much manual input. In contrast, orchestrated systems can perform the same tasks quickly and be much more modular, freeing up valuable human resources for more strategic activities.  
We need to expand on time efficiency a bit more, as it becomes more and more critical and the environment becomes more significant. First and foremost, the deployment process for these plugins is very complex. We are not talking about a couple of apt/yum commands; restart a service, and everything is ready. Especially when talking at scale, it is an absolute must to create a deployment strategy that is automated but also robust so that it can handle any issues in a non-destructive way. Suppose you must deploy a Kubernetes CNI plugin to 1000 nodes in an HPC cluster and verify if the CNI is working correctly across all those nodes--how long would that take?  
The second part of the time efficiency story is the performance evaluation part. Our methodology offers a modular approach to evaluating performance--we can easily configure the number of runs. The default settings are five runs, which take roughly 40 h for a set of two servers to evaluate a CNI as the evaluation procedure goes through all tuned performance profiles, package size settings, MTU settings, and TCP/UDP evaluations for both bandwidth and latency. To further elaborate on how important the automation part is, we also made a manual run of all these tests, and it took one person 60 h to finish the same number of tests and an additional 10 h to gather all the data and do a manual analysis for a single CNI.  
While manual evaluation of Kubernetes CNIs can provide valuable insights, the drawbacks related to consistency, scalability, reproducibility, human error, and time efficiency make it less desirable compared to automated and orchestrated approaches. Automation ensures more reliable and consistent results and significantly enhances the ability to handle large-scale environments and complex evaluation scenarios. By minimizing human error and reducing the time required for evaluation, automated methodologies empower organizations to optimize their deployments and allocate their human resources more effectively. Consequently, adopting an automated and orchestrated evaluation approach is crucial for achieving robust and efficient network performance in Kubernetes environments.