## 3 Methodology
### Proximal Policy Optimization with Curiosity  
Let \(\theta\) denote the parameters of the agent, and we use the PPO-clip algorithm to update the \(\theta\) in this work. As mentioned in Section 2.2, the policy gradient algorithm is difficult to select an appropriate step size. Too much difference between older and newer policies during the training process is not helpful for learning. PPO is a novel policy gradient algorithm, which can solve the issue of difficulty in determining the step length in the policy gradient algorithm. PPO algorithm introduces a new objective function that can achieve small batch update in multiple training steps. PPO-clip updates the policy with the following objective function:  
\[\theta_{k+1}=argmax_{\theta}\mathop{\mathbb{E}}_{s,\mu\sim n_{b}}[L(s,a, \theta_{k},\theta)] \tag{8}\]  
where \(L\) is given by  
\[L(s,a,\theta_{k},\theta)= min(\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)}A^{\pi_{ \theta_{k}}}(s,a), \tag{9}\] \[clip(\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)},1-\varepsilon,1+\varepsilon)A^{\pi_{\theta_{k}}}(s,a))\]  
where \(\varepsilon\) is a hyperparameter of PPO and we set \(\varepsilon=0.2\), according to the authors' recommendations. Let \(r_{t}(\theta)=\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)}\); the PPO approach is summarized by controlling the ratio \(r_{t}(\theta)\) of the old and new policies to prevent the change of update magnitude from affecting the agent's learning. Clip operation will limit the value of \(r_{t}(\theta)\) to \([1-\varepsilon,1+\varepsilon]\). Furthermore, \(A\) is the advantage function.  
To solve the sparse reward problem, we use the curiosity-based prediction error method to obtain an intrinsic reward. We construct a model that receives the state \(s_{t}\) and action \(a_{t}\) of the current time step \(t\) and predicts the state \(s^{{}^{\prime}}_{t+1}\) of the next time step \(t+1\). Then, the difference between the predicted state and the true next state is used as the intrinsic reward. The agent obtains a reward \(r_{t}\) after executing action \(a_{t}\) at each time step. The reward \(r_{t}\) is composed of \(r_{i}\) and \(r_{e}\), where \(r_{i}\) is the intrinsic reward and \(r_{e}\) is the environment reward. Assuming that each episode has \(T\) time steps, the \(r_{e}\) of time steps 1 to \(T-1\) is 0, while the \(r_{e}\) of time step \(T\) is the validation set accuracy. The \(r_{i}\) is related to the state \(s_{t}\), the action \(a_{t}\) of the current time step \(t\), and the state \(s_{t+1}\) of the next time step \(t+1\). Figure 5 shows how the intrinsic reward is calculated. Since each state is composed of \(\mu\) and \(\sigma\), we need to extend the state to a certain dimension. First, we convert \(s_{t}\) and \(s_{t+1}\) to \(\Phi(s_{t})\) and \(\Phi(s_{t+1})\) with an embedding layer. Then we concatenate the \(\Phi(s_{t})\) and \(a_{t}\), and then feed them to the forward network to predict the \(\Phi^{{}^{\prime}}(s_{t+1})\). The mean squared error (MSE) between the predicted state \(\Phi^{{}^{\prime}}(s_{t+1})\) and the true next state \(\Phi(s_{t+1})\) is used as the intrinsic reward \(r_{i}\) for the current time step \(t\).