## 3 Methodology
### Overall Framework  
Algorithm 1 describes the entire process. For a given task, the agent selects hyperparameters sequentially in each episode in order, then we use the selected hyperparameters to obtain the evaluation results of the \(D_{valid}\) as a reward, and finally use the PPO algorithm to update the parameters \(\theta\) of the agent.  
```
0:\(n\): Number of hyperparameters; \(s_{1}\): Randomly generated distributions; \(\mathcal{D}\): Data set \(\mathcal{D}=\varnothing\);
0: top-1 hyperparameter configuration;
1:while not done do
2:for\(i=1\) to \(\frac{episodes}{2}\)do
3:\(\tau\) = []
4: agent selects \(\lambda\) and obtains \(r_{T}\) on \(D_{valid}\)
5:\(\tau\) = [\((s_{t},a_{t},r_{t}),...,(s_{T},a_{T},r_{T})\)]
6: use \(\tau\) to update agent according to Equation (8)
7: add \((\lambda,r_{T})\) to \(\mathcal{D}\)
8:endfor
9: Use \(\mathcal{D}\) to train random forest
10: Save the current policy \(\pi\)
11:for\(\frac{episodes}{2}+1\) to episodesdo
12:\(\tau\) = []
13: agent selects \(\lambda\)
14:if\(d(\pi,\pi_{\theta})\leqslant 0.1\)then
15: use random forest to predict \(r_{T}^{\prime}\)
16:\(\tau\) = [\((s_{t},a_{t},r_{t}),...,(s_{T},a_{T},r_{T}^{\prime})\)]
17: use \(\tau\) to update agent according to Equation (8)
18: add \((\lambda,r_{T}^{\prime})\) to \(\mathcal{D}\)
19:else
20: obtain real \(r_{T}\) on \(D_{valid}\)
21:\(\tau\) = [\((s_{t},a_{t},r_{t}),...,(s_{T},a_{T},r_{T})\)]
22: use \(\tau\) to update agent according to Equation (8)
23: add \((\lambda,r)\) to \(\mathcal{D}\)
24:endif
25:endfor
26:endwhile
```  
**Algorithm 1** RFEPPO  
At the beginning of the search process, we first collect some samples of the real hyperparameter combinations \(\lambda=[\lambda_{1},\lambda_{2},\ldots,\lambda_{T}]\) and their corresponding validation accuracy \(r_{T}\). After each episode, the combination of hyperparameters and corresponding accuracy \((\lambda,r_{T})\) are collected for training the surrogate model. The agent is updated at the end of each episode using the trajectory \(\tau\), where \(r_{t}\) in the trajectory is \(r_{i}+r_{e}\). After the initial phase, the surrogate model is trained with the collected samples, and the policy \(\pi\) is recorded at this time. In the subsequent search process, the policy \(\pi_{\theta}\) of each episode is compared with the \(\pi\), and the RL agent is updated by the intrinsic reward and real accuracy if it is greater than \(0.1\); otherwise, the agent is learned by the intrinsic reward and the predicted accuracy \(r_{T}^{\prime}\) is made by the surrogate model. _Episodes_ is the total number of times to search for hyperparameters.  
The time complexity of searching for hyperparameters is \(O(E\times t)\), where \(E\) is the number of episodes in the whole process, and \(t\) is the time required for each episode. It is worth noting that \(E=E_{real}+E_{pre}\), \(t\) is divided into \(t_{real}\) and \(t_{pre}\), where \(E_{real}\) is the number of episodes with real rewards, and \(t_{real}\) is the time required for various operations in each episode at this time; \(E_{pre}\) is the number of episodes with rewards predicted by the surrogate model; and \(t_{pre}\) is the time required for various operations in each episode at this time. Thetime required to train the RF is \(O(m\times log(m)\times n\times k)\), where \(m\) is the number of samples, \(n\) is the number of hyperparameters, and \(k\) is the number of decision trees. Therefore, the time complexity of the whole process is \(O(E\times t+m\times log(m)\times n\times k)\).