## 4 Experiments
### Ablation Experiments
#### 4.3.4 Performance Comparison with Different Surrogate Models  
To verify the effectiveness of RF in our approach, we replaced RF in RFEPPO with other common ML methods and compared them. Since the surrogate model in our method can be replaced with other regression models arbitrarily, we compared our method with LREPPO, KNNEPPO, SVRPPO, REPPO, BAGEPPO, XGBEPPO, ABEPPO, and GBEPPO, which replace RF in RFEPPO with linear regression, KNN, SVR, Ridge, Bagging, XGBoost, AdaBoost, and gradient boost, respectively. For KNN, we set the number of neighbors and the power parameter to a range of \(\{2,3,4,5,6\}\) and \(\{1,2\}\) respectively. For SVR, we used the radial basis function (RBF) kernel. For ridge, we set regularization strength among \(\{0,0.01,0.1,1\}\). For bagging, XGBoost, AdaBoost, gradient boost and RF, we set the number of decision trees among \(\{100,110,\ldots,200\}\). The hyperparameters of these methods were determined by the grid search within their respective ranges, ensuring the final model performance. In this experiment, we chose the Turkiye Student Evaluation dataset for  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Datasets** & **A2C** & **DDPG** & **SAC** & **TD3** & **PPO** \\ \hline  & **Acc (\%)** & **Time (s)** & **Acc (\%)** & **Time (s)** & **Acc (\%)** & **Time (s)** & **Acc (\%)** & **Time (s)** \\ \hline winequality\_white & 67.03 \(\pm\) 1.60 & 479 \(\pm\) 29 & 66.73 \(\pm\) 1.62 & **420**\(\pm\) 192 & 67.38 \(\pm\) 1.40 & 530 \(\pm\) 4 & 67.34 \(\pm\) 2.15 & 422 \(\pm\) 52 & **0.683**\(\pm\) 0.89 & 548 \(\pm\) 34 \\ optdigits & 98.45 \(\pm\) 0.40 & 508 \(\pm\) 21 & 98.40 \(\pm\) 0.24 & 459 \(\pm\) 177 & 98.29 \(\pm\) 0.26 & 576 \(\pm\) 16 & 98.47 \(\pm\) 0.75 & **403**\(\pm\) 42 & **0.985**\(\pm\) 0.29 & 580 \(\pm\) 11 \\ Turkey\_Student\_Evaluation & 87.67 \(\pm\) 0.47 & 334 \(\pm\) 14 & 87.65 \(\pm\) 0.19 & **154**\(\pm\) 37 & 87.60 \(\pm\) 0.74 & 378 \(\pm\) 6 & 87.52 \(\pm\) 0.47 & 292 \(\pm\) 56 & **0.876**\(\pm\) 13 & 363 \(\pm\) 16 \\ balance\_scale & 92.54 \(\pm\) 0.96 & 67 \(\pm\) 2 & 92.58 \(\pm\) 10.9 & 89 \(\pm\) 22 & 92.80 \(\pm\) 2.24 & 112 \(\pm\) 2 & 93.91 \(\pm\) 0.60 & 71.79 \(\pm\) 0.9367 & 12.8 \(\pm\) 103 \(\pm\) 1 \\ Cardiotopography & **89.47**\(\pm\) 1.33 & 262 \(\pm\) 6 & 89.00 \(\pm\) 1.26 & 299 \(\pm\) 57 & 89.36 \(\pm\) 1.91 & 316 \(\pm\) 18 & 88.92 \(\pm\) 0.58 & **245**\(\pm\) 22 & 0.8904 \(\pm\) 0.82 & 296 \(\pm\) 2 \\ monks\_1 & 99.39 \(\pm\) 1.48 & **39**\(\pm\) 1 & 98.86 \(\pm\) 2.44 & 63 \(\pm\) 0.9 & 99.18 \(\pm\) 0.87 & 83 \(\pm\) 2 & 99.64 \(\pm\) 0.87 & 58.2 \(\pm\) 0.997 & 1.21 \(\pm\) 77 \(\pm\) 1 \\ DR\_Debrecen & 73.06 \(\pm\) 1.88 & **59**\(\pm\) 2 & 72.73 \(\pm\) 20.0 & 27 \(\pm\) 73.88 \(\pm\) 1.87 & 105 \(\pm\) 3 & **7.36 validation. As in previous experiments, we ran each method five times and trained 200 episodes each time. The accuracy and time for each method are shown in Figure 9. From this figure, it can be seen that RF has better final accuracy than other models. Although LEPPO, KNNEPPO, and SVRPPO take less time than RFEPPO, the cost of using them is reduced accuracy. Other methods are worse than RF in terms of both time and accuracy. Therefore, we believe that RF is the most suitable surrogate model in our method. In order to further analyze the RF surrogate model, we need to know its predictive performance, thus we collected 100 samples from the first 100 episodes. Each sample was a combination of hyperparameters and the corresponding accuracy of validation set. To compare the performance of these surrogate models, the 100 samples were divided into training and testing samples in the ratio of 8:2. The RMSE results of the test samples are tabulated in Table 8, and the ground truth of the test samples and the predicted results of each model are presented in the Figure 10. It is obvious from Table 8 that RF performs the best. In Figure 10, the blue points represent the true values and the red points represent the predicted results.