## 2 Methodology
### Mirror Environment
Whereas the reward function only gives the immediate reward at a state, the value function \(\mathcal{V}(\mathbf{s};\pi)\) for a state \(\mathbf{s}\) roughly approximates the total amount of reward that can be accumulated over the future episodic-steps starting from state \(\mathbf{s}\) applying policy \(\pi\). Thus, the value function actually encodes the idea of long-term return in the sequential decision process instead of only focusing on the immediate reward. With the value function, the cumulative reward can now be computed for a transition \((\mathbf{s},a,\mathbf{\tilde{s}},r)\) as: \(r+\mathcal{V}(\mathbf{\tilde{s}};\pi)\), i.e., the current reward added to the value of the next state. However, a discount factor \(\gamma\) is used to gradually reduce the values of the next states and to avoid an infinite sum. This discounted cumulative reward is expressed as follows:  
\[R_{\gamma}(\mathbf{s},a,\mathbf{\tilde{s}},r)=r+\gamma\mathcal{V}(\mathbf{\tilde{s}};\pi) \tag{2}\]  
where \(0<\gamma<1\) is used. Note that the value network is also built on top of the final convolutional layer of the policy (see Figure 3). This value network is also updated at each epoch to minimize the difference between the predicted value \(\mathcal{V}(\mathbf{s};\pi)\) and the observed discounted reward \(R_{\gamma}\) in the gathered trajectories applying the policy in the previous epoch. Here, the value network updated in the last epoch is used to compute the discounted rewards for the observed transitions. In PPO, the improvement (i.e., _advantage_) of the current policy \(\pi\) over the previous policy \(\pi_{\text{prev}}\) (i.e., the policy from the previous epoch) is computed by comparing their corresponding cumulative rewards. For a transition \((\mathbf{s},a,\mathbf{\tilde{s}},r)\), the cumulative reward of the current policy \(\pi\) refers to the observed discounted reward \(R_{\gamma}(\mathbf{s},a,\mathbf{\tilde{s}},r)\). On the other hand, the cumulative reward of the previous policy \(\pi_{\text{prev}}\) refers to the value of the current state, i.e., \(\mathcal{V}(\mathbf{s};\pi_{\text{prev}})\) since the value was optimized to approximate the discounted rewards in the previous epoch. Now the advantage can be simply computed as follows:  
\[A(\mathbf{s},a,\mathbf{\tilde{s}},r)=R_{\gamma}(\mathbf{s},a,\mathbf{\tilde{s}},r)-\mathcal{V }(\mathbf{s};\pi_{\text{prev}}). \tag{3}\]  
The training objective of PPO is to maximize this advantage, thereby updating the policy towards improvement in the expected cumulative reward. For more details on the advantage and objective function, please refer to [12]. In our mirror environment, the PPO training objective for maximizing the advantage can be expressed as follows:  
\[ \begin{split}\underset{\pi}{\text{max}}\,\mathbb{E}_{(\mathbf{s},a,\mathbf{\tilde{s}},r)\in\mathbb{T}}[\frac{\pi(\mathbf{s},a)}{\pi_{\text{prev}}(\mathbf{s },a)}A(\mathbf{s},\mathbf{\tilde{s}},r)]\\ +\mathbb{E}_{(\mathbf{s}^{\prime},a^{\prime},\mathbf{\tilde{s}}^{\prime} )\in\mathbb{T}^{\prime}}[\frac{\pi(\mathbf{s}^{\prime},a^{\prime})}{\pi_{\text{ prev}}(\mathbf{s}^{\prime},a^{\prime})}A(\mathbf{s}^{\prime},\mathbf{\tilde{s}}^{\prime},r^{ \prime})]\end{split}  \tag{4}\]  
Usually, the policy ratio \(\pi(\mathbf{s},a)/\pi_{\text{prev}}(\mathbf{s},a)\) in (4) is clipped between \([0.8,1.2]\) to avoid large policy updates [12]. Note that the \(\pi(\mathbf{s},a)\) is the target policy to be optimized, whereas the previous policy \(\pi_{\text{prev}}(\mathbf{s},a)\) is a scalar constant holding the optimal probability of action \(a\) at state \(\mathbf{s}\) as observed while gathering experiences. Thus, this previous policy is simply the policy updated in the last epoch and used for gathering experiences in the current epoch. Therefore, the above objective suggests that the policy for an action is increased compared to the previous policy if the advantage is positive, and decreased otherwise.