## 5 Evaluation
### Results
#### 5.4.1 Results for RQ1  
In prediction tasks across various domains, many learning methods have been proposed to predict outcomes, including the CNN, DeepFM, DeepPerf, and HINNPerf. In addition, traditional machine learning methods such as FM can also be used for prediction tasks. DeepFM [23] is a representative model in the field of click-through rate prediction, while DeepPerf [20] demonstrates superior prediction accuracy in configurable systems. HINNPerf [31], proposed in 2023 as an improved version of DeepPerf, has similarly achieved high prediction precision. The CNN leverages parameter sharing and localized connectivity. These properties enable efficient feature extraction from limited datasets while improving the predictive accuracy. As a well-established deep learning architecture, the CNN currently represents one of the most widely adopted models in the field. In this experiment, we verify the effectiveness of our proposed method by comparing our MovePerf model with the CNN, FM, DeepFM, DeepPerf, and HINNPerf. We evaluate the six models across the four projects shown in Table 4. Note that the datasets for HINNPerf are not considered for this comparison, as they primarily focus on configurable systems, where the features are strongly related to the specific configuration options. For instance, in the case of \(\times 264\), the configuration options include parameters such as the number of reference frames, enabling or disabling the default entropy encoder, and the number of frames for ratecontrol and lookahead. The hyperparameter settings in DeepFM follow [23], those for DeepPerf follow [20], and those for HINNPerf follow [31]. The hyperparameter settings in FM follow the FM component in DeepFM. The number of epochs for the CNN is 100, the number of epochs for MovePerf is 100, and the batch size is 64. Table 4 presents the performance prediction results for the six methods across various datasets. Overall, MovePerf demonstrates a more balanced and effective approach across the datasets. For JUnit, the CNN exhibits a relatively high mean of 15.83% and a small margin of 4.94, indicating that the model's predictions are somewhat erratic and unstable. In contrast, DeepFM significantly reduces the mean to 13.53%, although its margin increases to 10.58. This suggests that, while DeepFM provides more accurate predictions in most cases, it has larger prediction MREs in some instances. HINNPerf shows a mean of 5.96% and a smaller margin of 2.00, indicating improved prediction stability, although a notable prediction MRE remains. Finally, MovePerf performs exceptionally well in this project, achieving a mean of 5.41% and a margin of 0.03, indicating low prediction MREs and high stability. Therefore, for the dataset with a small number of samples and large label variation, MovePerf performs better, reducing the mean by 0.55% versus HINNPerf. For LC-problems, the CNN shows a mean of 6.44% and a margin of 5.64, reflecting relatively low prediction accuracy and some instability. DeepFM shows a mean of 1.77% and achieves a margin of 3.56, suggesting that the prediction MRE of the model is reduced and it can provide a high-accuracy prediction, although its stability is moderate. In comparison, the performance of HINNPerf is superior, with a mean of 0.35% and a margin of 0.10, demonstrating smaller prediction MREs and higher stability, thus showing stronger predictive capabilities. MovePerf gives slightly better results than HINNPerf, with a mean of 0.28% and a margin of only 0.01. Although its predictive accuracy is slightly lower than that of HINNPerf, the prediction MRE remains small, and it excels in terms of stability. Therefore, when dealing with datasets with a very small number of samples and minimal label variation, MovePerf performs even better, reducing the mean by 0.07% compared to HINNPerf. For Concurrency, the CNN has a mean of 12.47% and a margin of 4.39, suggesting moderate prediction MREs with reasonable stability.