# Performance Comparison of H.264 and H.265 Encoders in a 4K FPV Drone Piloting System
## 3 Evaluation Methodology  
This section describes the evaluation process. The main idea is to automate the process for determining the best set of encoding parameters (e.g., codec, GOP size, QP) for a specific use. In other words, if a user has limited bitrate throughput, we wish to provide a set of parameters that provides the best quality at the specified bitrate. Similarly, if a user defines a minimal required quality, we aim to provide a set of parameters that generates a video that matches the quality requirement but with the lowest possible bitrate. To automate the process, we designed a test environment consisting of six Python scripts. * encode.py: This script encodes the input video through all combinations of defined codec, GOP size, QP, and target bitrate parameters. The four parameters are user-defined in separate lists. This script uses FFMPEG [16] to encode videos. * quality_assessment.py: After generating a set of test videos, this script calculates and saves PSNR, SSIM, and BRISQUE scores into a newly created data.csv file. * generate_vmaf.py: As we use FFMPEG to calculate VMAF, for each video, we need to save the console output into a separate text file, for further analysis. * avg_vmaf.py: Using this script, we extract average VMAF scores from previously created files using a regex. We save the scores into a newly created file, data_vmaf.csv. * append_vmaf.py: This script creates another file, data_final.csv. This file is created by appending all the VMAF scores onto data.csv, meaning that data_final.csv contains all the data we need for the analysis, including codec type, target bitrate, actual bitrate, average QP (\(\frac{\text{QP\_max}+\text{QP\_min}}{2}\)), GOP size, and the scores of all four quality metrics. * draw.py: The final step involves visualizing all of the results, which can be challenging when dealing with multi-dimensional data. For a clear interpretation, we draw interactive parallel coordinates and bubble charts using plotly [17]. The scripts need to be executed in the order described in Figure 1. The whole process is computationally very complex, which is why we used a supercomputer, BURA [18], located in Rijeka, Croatia. BURA includes multi-computer and multi-processor systems based on a hybrid computing architecture. The multi-computer cluster has 288 compute nodes, each having two Xeon E5 processors (in total, 24 physical cores). The size of the node memory is 64 GB, and the storage space is 64 GB per node. We deployed all scripts on BURA and performed the simulations remotely. Python scripts were run through SLURM (Simple Linux Utility for Resource Management) [19] scripts, which offer the possibility of specifying the number of threads, nodes, and so on for execution of the linked python script. For each video, encoding was carried out using one node and one thread. The encoding process was fairly fast (around 30 min in total for each raw video) and, so, there was no need for parallelization. Moreover, the onboard drone computer can hardly replicate the computational power of a supercomputer, so only one thread was considered sufficient for encoding. However, quality assessment is computationally very complex; so, for each raw video, 24 scripts were executed on the 24 nodes simultaneously. Each script used one thread for assessing the quality of selected compressed sequence samples, and increasing the number of threads/nodes had no impact on the metric score. Even on BURA, it took approximately 3 min to process a single frame. After obtaining all results and drawing the final graph, we could determine an optimal set of parameters that represents the best match for a specific need. In terms of the availability of reference images, there are three types of image quality metrics: Full-reference (FR), No-reference (NR), and Reduced-reference (RR) metrics. FR metrics require a reference image and calculate scores by comparing the compressed image pixels with the reference image pixels. NR metrics are typically used when the reference image is not available.