## 5 Evaluation
### Results
#### 5.4.5 Results for RQ5  
To evaluate the practicality and feasibility of our approach, it is essential to measure the time consumed by the training and testing processes of MovePerf. All experiments were conducted on a Dell workstation with a 1.19 GHz Intel Core i5 CPU and 8 GB RAM, running 64-bit Windows 10. Due to the differences in the TensorFlow versions used in the models, GPU acceleration may not have functioned properly or been fully utilized. This could have rendered the experiments unstable, potentially resulting in inconsistent or non-reproducible results. Therefore, to ensure the reliability and consistency of the experimental outcomes, we opted to implement the methods under a CPU environment.  
We compared DeepFM, DeepPerf, HINNPerf, and our approach, because CNN and FM models have simpler architectures and much shorter computation times. However, as the experimental results indicate, their predictive accuracy is lower; thus, these methods are excluded from the comparison.  
From the perspective of the model architecture, DeepPerf, DeepFM, and our approach share certain similarities. Since the deep learning neural network (DNN) component used in DeepFM has a relatively simple architecture, it requires less time to train compared to  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Project** & **True Value** & **Predicted Value** & **MRE** \\ \hline \multirow{3}{*}{JUnit} & 477.36 & 501.80 & 5.12 \\  & 469.76 & 442.19 & 5.87 \\  & 472.92 & 496.38 & 4.96 \\ \hline \multirow{3}{*}{LC-problems} & 6117 & 6136.57 & 0.32 \\  & 6105 & 6090.68 & 0.23 \\  & 6100 & 6122.55 & 0.37 \\ \hline \multirow{3}{*}{Concurrency} & 4026.15 & 4155.39 & 3.21 \\  & 3700.09 & 3909.89 & 5.67 \\  & 3155.02 & 2997.69 & 4.99 \\ \hline \multirow{3}{*}{Kevin} & 51,621.00 & 60,824.84 & 17.83 \\  & 102,812.96 & 81,872.22 & 20.34 \\ \cline{1-1}  & 51,697.67 & 62,088.90 & 20.10 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Generalization of the experimental results.  
DeepPerf and MovePerf. When used individually, the FM component used in MovePerf has a smaller time cost, and its contribution to the overall time is negligible. Therefore, the time cost for MovePerf should be roughly equivalent to that of DeepPerf. HINNPerf, which consists of multiple FNN blocks and requires the training of many parameters, requires a longer period of time.  
Furthermore, based on the experimental results, the models' training times are as shown in Figure 9. For all project datasets with MMR, the time cost of DeepFM in searching for optimal hyperparameters and training a model is the shortest, ranging from 5 to 80 min. Then, it takes DeepPerf and our MovePerf 3-126 min to conduct model training and hyperparameter searching, while HINNPerf spends 27-297 min.  
The time costs are slightly larger since we used a CPU for the experiments. When compared to the GPU time used in [31], despite the differences in the number of features (ranging from 9 to 60 features) and the sample sizes (ranging from 192 to 10\({}^{31}\)), the time cost of model training and hyperparameter searching for DeepPerf is from 5 to 13 min, while HINNPerf takes 14 to 16 min. Thus, under the same GPU conditions, the maximum time cost for MovePerf is 13 min, which makes its time cost acceptable.  
In summary, MovePerf demonstrates higher predictive accuracy than other methods, and its time cost in searching for optimal hyperparameters and training a model is reasonable.