## 4 Experiments
### Ablation Experiments
#### 4.3.1 Performance Comparison with and without Curiosity-Driven Method  
To demonstrate that the addition of intrinsic reward has a better effect and can improve accuracy in the test set, we removed the curiosity mechanism and only used the validation set accuracy as reward. We conducted comparison experiments on the balance_scale data set.  
Figure 7: Average validation accuracy across 5 trials on MNIST dataset.  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Datasets** & \multicolumn{1}{c}{**RS**} & \multicolumn{2}{c}{**TPE**} & \multicolumn{2}{c}{**Hyperband**} \\ \hline  & **Acc (\%)** & **Time (h)** & **Acc (\%)** & **Time (h)** & **Acc (\%)** & **Time (h)** \\ \hline MNIST & 99.17 \(\pm\) 0.14 & 1.48 \(\pm\) 0.013 & 99.19 \(\pm\) 0.06 & 1.46 \(\pm\) 0.051 & 98.82 \(\pm\) 0.07 & 1.04 \(\pm\) 0.050 \\ Fashion\_MNIST & 90.56 \(\pm\) 0.59 & 1.48 \(\pm\) 0.016 & 91.70 \(\pm\) 0.24 & 1.53 \(\pm\) 0.040 & 86.85 \(\pm\) 1.29 & 0.98 \(\pm\) 0.004 \\ \hline
**Datasets** & \multicolumn{1}{c}{**BOHB**} & \multicolumn{2}{c}{**MBRL-SDP**} & \multicolumn{2}{c}{**RFEPPO (Our Method)**} \\ \hline  & **Acc (\%)** & **Time (h)** & **Acc (\%)** & **Time (h)** & **Acc (\%)** & **Time (h)** \\ \hline MNIST & 99.10 \(\pm\) 0.02 & 0.94 \(\pm\) 0.046 & 99.24 \(\pm\) 0.04 & 0.76 \(\pm\) 0.010 & **99.33**\(\pm\) 0.05 & **0.75**\(\pm\) 0.002 \\ Fashion\_MNIST & 90.78 \(\pm\) 0.40 & 0.92 \(\pm\) 0.013 & 89.91 \(\pm\) 0.04 & **0.76**\(\pm\) 0.003 & **92.16**\(\pm\) 0.27 & 0.78 \(\pm\) 0.014 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The performance (mean \(\pm\) std) of our method and other conventional algorithms on 2 image datasets.  
Figure 8 shows the effect of intrinsic reward on the overall search process. In Figure 8a, the green line refers to the reward return for each episode using only the accuracy of the validation set as the reward. We can find that the agent's reward has no obvious upward trend with the growth of episodes. We believe that the reason for this phenomenon is that the agent is updated slowly due to too many sparse rewards and little sample utilization. Moreover, better validation set accuracy is obtained at the initial stage of the search process, which we believe is caused by the suitability of the initial state of this dataset. The coral line refers to the reward return for each episode when using the intrinsic reward. We can find an overall upward trend in the reward return for the whole search process. This is because the curiosity-driven method encourages the agent to explore more unfamiliar states, and the more novel the state, the higher the intrinsic reward. The use of intrinsic reward improves the utilization of the sample, which allows the agent to learn more about the environment. Figure 8b shows the accuracy of the validation set corresponding to the best hyperparameter configuration searched in the current episode. We can see that compared to the non-intrinsic reward approach, the use of the intrinsic reward approach can find high-performance hyperparameter configuration quickly and with higher accuracy. Therefore, we believe that intrinsic reward can solve the problem of sparse reward in HPO. The curiosity-driven method encourages the agent to keep exploring unfamiliar states, and finds better hyperparameter configurations faster.