## 1 Introduction
GS is very simple and can be executed in parallel; however, the number of trials increases exponentially with the number of hyperparameters. Compared to GS, RS is more efficient, but it does not guarantee finding the global optimum in the limit, whereas GS never will. A typical BO method is to adopt a Gaussian process (GP) as the surrogate model; however, GP is cubically related to the number of data sizes. Hyperband dynamically allocates resources to each set of configurations and uses SH to stop underperforming configurations. However, it randomly samples new configurations at each step. In order to learn the previously sampled configurations, BOHB combines BO with hyperband, and DEHB combines the evolutionary optimization method of differential evolution (DE ([PERSON] and others, 2018)) with hyperband. However, these hyperband-based methods tend to choose the hyperparameter settings that enable the model to converge quickly in the early stages of  
Figure 1: HPO methods. training. [PERSON] et al. [15] used the reinforcement learning (RL) method and treated HPO as a sequential decision process. In addition, to speed up training, they used MLP as a surrogate model to predict accuracy and dynamically train the agent with true accuracy and predicted accuracy. This approach reduces the number of trials and does not tend to choose a hyperparameter setting that converges quickly in the early stages of training. However, due to the sequential selection of hyperparameters, the reward is obtained only when the last hyperparameter is selected, and the rewards of previous steps are all 0, which causes the problem of sparse reward. Moreover, the MLP model needs to be carefully designed and requires a large sample size to prevent overfitting. Inspired by [15], we adopt random forest (RF) as a surrogate model and use part of the real rewards and part of the predicted rewards from the surrogate model when updating the agent. Moreover, in order to solve the problem of sparse reward, we add the curiosity to provide intrinsic reward. The basic process of RL-based HPO is shown in Figure 2. The agent observes state \(s_{t}\) in the environment at time step \(t\) and then executes action \(a_{t}\) according to the policy. Consequently, the environment returns a reward \(r_{t}\) to the agent, and the state is transferred from \(s_{t}\) to \(s_{t+1}\). For the environment, state, action and reward of this paper, we introduce them in detail in Section 3. In this work, we use the proposed method to optimize the hyperparameters of extreme gradient boosting (XGBoost) [16] on nine tabular datasets and convolutional neural network (CNN) on two image datasets. The main contributions of this paper are as follows:  
* We use the RF algorithm as a surrogate model which can predict rewards and then reduce the consumption of resources. * For the sparse reward problem of HPO, we use curiosity to provide an intrinsic reward that speeds up the convergence and finds better hyperparameter configuration faster. * We use the proposed method to tune the hyperparameters of the XGBoost on nine tabular datasets and CNN models on two image datasets. The experimental results show that our method has higher accuracy and faster convergence compared to other optimization algorithms. The remainder of this paper is organized into four sections. Section 2 introduces the hyperparameter optimization methods and the optimization objective of RL. In Section 3, we describe our proposed method. Section 4 displays and discusses the experiment results. Finally, we draw conclusions in Section 5.