## 2 Related Work  
### Hyperparameter Optimization  
HPO is a task to find the optimal/suboptimal hyperparameters in ML/DL models by using certain algorithms. HPO generally consists of three components: the first is the objective function, i.e., the goal that the algorithm needs to minimize or maximize; the second is the search range, which needs to determine candidates for discrete spaces and  
Figure 2: The process of HPO using reinforcement learning methods. upper and lower bounds for continuous spaces; and the third is the parameters of the algorithm itself, such as the selection of acquisition functions. The definition of HPO is as follows: Let \(\mathcal{A}\) denote a ML or DL algorithm with \(n\) hyperparameters; \(\mathcal{A}_{\lambda}\) is the algorithm \(\mathcal{A}\) with hyperparameter \(\lambda\); \(\Lambda_{n}\) is the search space of the \(n\)-th hyperparameter; and \(\Lambda\) denotes all hyperparameter search spaces. Therefore, given a data set \(\mathcal{D}\), the purpose of HPO is to find  
\[\lambda^{*}=\underset{\lambda\in\Lambda}{argmin}\,\mathbb{E}_{(D_{train},D_{ valid})\sim\mathcal{D}}\mathcal{L}(\mathcal{A}_{\lambda},D_{train},D_{valid}) \tag{1}\]  
where \(\mathcal{L}(\mathcal{A}_{\lambda},D_{train},D_{valid})\) denotes the loss value of model \(\mathcal{A}\) with hyperparameter setting \(\lambda\) on validation data set \(D_{valid}\) that is trained on training set \(D_{train}\). Some popular hyperparameter tuning algorithms have been proposed in recent years and widely used in different fields. **Grid search and random search:** GS defines the search space as a regular grid and evaluates every position in the grid. With a large search range and a small step size, GS has a high probability of discovering the global optimum. However, this method is very computationally intensive and time consuming, especially when the number of hyperparameters to be tuned is relatively large. The idea of RS is similar to GS, but it is generally faster than GS. Instead of testing all values in the grid, RS randomly selects sample points within the search space. RS assumes that if the set of sample points is sufficiently large, then the global optimum or its approximation can also be found with a high probability through random sampling. **Bayesian optimization:** BO is an effective optimization method and generally used for optimizing expensive black-box functions [17]. BO finds the value that minimizes the objective function based on the historical evaluation results by building a surrogate function (probabilistic model). Several surrogate models commonly used in BO are Gaussian process (GP), tree-structured Parzen estimator (TPE) [5], and RF. The Bayesian based approach differs from grid or random search in that it saves a lot of useless work by referring to previous evaluations when trying the next set of hyperparameters. **Evolutionary algorithm-based optimization:** The idea of the EA method comes from the concept of biology. Since natural evolution is a dynamic process occurring in a changing environment, it is applicable to the HPO task, which is also a dynamic process. The covariance matrix adaptation evolution strategy (CMA-ES) [18] is one of the most well-known EA methods. EA can also be used to optimize the neural structure and hyperparameters of deep learning models [19]. **Bandit-based optimization:** It is very expensive to verify the performance of different hyperparameters when the data set is very large or the model is very complex. Common techniques to address this problem include using subsets of the data set, feature dimensionality reduction, reducing training steps, etc. However, the multi-fidelity approach converts this manual heuristic into a formal algorithm that uses an approximation of the real loss function for optimization [20]. Typical bandit-based methods are hyperband and successive halving. The BOHB proposed in [12] utilizes hyperband to decide how many sets of hyperparameters to use at a time and how much budget to allocate to each set of hyperparameters.