Beyond the Boundaries of Proximal Policy Optimization

## Abstract

Abstract Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget.

## 1 Introduction

Proximal policy optimization (PPO) (Schulman et al., ) is ubiquitous within modern reinforcement learning (RL), having found success in domains such as robotics (Andrychowicz et al., ), gameplay (Berner et al., ), and research applications (Mirhoseini et al., ). Given it’s ubiquity, significant research effort has explored the theoretical (Hsu et al., ; Kuba et al., ) and empirical (Engstrom et al., ; Andrychowicz et al., ) properties of PPO.

PPO is an on-policy algorithm; at each iteration it collects a dataset using the current (behavior) policy. This dataset is used to construct a surrogate to the true objective, enabling gradient-based optimization while seeking to prevent large changes in policy between iterations, similar to trust region policy optimization (Schulman et al., ). The solution to the surrogate objective is then taken as the behavior parameters for the following iteration, defining the behavior policy with which to collect the following dataset. The behavior policies are therefore *exactly coupled* with the preceding surrogate objective solution.

In this work we instead consider the inner-loop optimization of each surrogate objective to estimate an update vector, which we name the outer gradient. A trivial result follows that the outer loop of PPO can be viewed to update the behavior parameters using unity learning rate $\sigma=1$ gradient ascent on the outer gradients. Using this insight we propose outer-PPO, a novel variation of PPO that employs an arbitrary gradient-based optimizer in the outer loop of PPO. Outer-PPO *decouples* the estimation and application of updates in way not possible in standard PPO. An illustration of outer-PPO applying a learning rate greater than unity is provided in figure 1. The new behaviors enabled by outer-PPO raise several questions related to implicit design choices of PPO:

1

###### Question 1 .

Is the unity learning rate always optimal?

2

###### Question 2 .

Is the independence (lack of prior trajectory information e.g momentum) of each outer update step always optimal?

3

###### Question 3 .

Is initializing the inner loop surrogate objective optimization at the behavior parameters (without exploiting prior trajectory / momentum) always optimal?

Figure: Figure 1: Diagram of outer-PPO estimating and applying the outer gradient as an update. (i) Transitions are collected with policy $\pi(\bm{\theta}_{k})$ defining a surrogate objective and corresponding ‘trust-region’ (shaded) surrounding $\bm{\theta}_{k}$; inner-loop optimization of the surrogate objective (blue dashed) yields $\bm{\theta}_{k}^{*}$. (ii) Outer-PPO computes outer gradient as $\bm{g}^{O}_{k}\leftarrow\bm{\theta}_{k}^{*}-\bm{\theta}_{k}$. (iii) Outer-PPO updates behavior parameters using an arbitrary gradient based optimizer applied to the outer gradient to give $\bm{\theta}_{k+1}$, in this case gradient ascent with a learning rate $\sigma>1$. Standard PPO can be understood as directly taking $\bm{\theta}_{k+1}\leftarrow\bm{\theta}_{k}^{*}$, or as a special case of outer-PPO corresponding to gradient ascent with learning rate $\sigma=1$.

This work forms an empirical investigation of the aforementioned questions. To motivate this investigation, consider the clipping parameter $\epsilon$ of PPO, controlling the size of the ‘trust region’ within which we seek to restrict our update. If $\epsilon$ is set too low, we restrict ourselves to small policy updates. Conversely, if we set $\epsilon$ too high we decrease the reliability of our update direction. In outer-PPO, introducing an outer learning rate decouples these two effects; we are able to reliably estimate an update vector using a moderate $\epsilon$, but then take step of large magnitude in this direction.

We emphasize that we do not seek to identify the most performant configuration possible but to understand the performance of outer-PPO relative to a well-tuned PPO baseline. To this end we restrict the tuning of outer-PPO to simple grid searches applied to fixed base PPO hyperparameters.

Our contributions are as follows:

- •
We propose *outer proximal policy optimization* (outer-PPO), in which an arbitrary gradient-based optimizer is applied to the ‘outer gradients’ of PPO. By tracking the outer trajectory, outer-PPO further permits a momentum bias to be applied to the inner-loop initialization.
- •
We optimize a PPO baseline through extensive hyperparameter sweeps (total of 38,400 agent trained) on subsets of Brax (6 tasks) (Freeman et al., ), Jumanji (4 tasks) (Bonnet et al., ), and MinAtar (4 tasks) (Young & Tian, ). We open-source the sweep database files to facilitate future research against strongly tuned baselines.
- •
We perform three lightweight outer-PPO grid searches on non-unity outer learning rates, outer Nesterov momentum and biased-initialization, each addressing questions 1, 2 and 3 respectively.
- •
We evaluate the outer-PPO methods against the baseline, using 64 seeds per task over the 14 different tasks. We find non-unity outer learning rates to yield the greatest improvement (5-10%) on both Brax and Jumanji. Outer Nesterov also improves performance on Brax and Jumanji. Biased initialization achieves a moderate improvement on Jumanji alone. No method improves over the baseline on MinAtar.
- •
Given the stated empirical results we conclude the *negative* for questions 1, 2 and 3. Relaxing each of these PPO design choices can lead to consistent, statistically significant improvement of performance over at least one of the evaluated environment suites.
- •
We propose that practitioners able to experiment may explore non-unity outer learning rates given the simplicity (single hyperparameter) and consistent improvement achieved on Brax and Jumanjji.

###### Question 1 .

###### Question 2 .

###### Question 3 .

## 2 Background

### 2.1 Reinforcement Learning

We consider the standard reinforcement learning formulation of a Markov decision process $\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma\rangle$, where $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is the state transition probability function, $r:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathbb{R})$ is the reward function, and $\gamma\in[0,1]$ is the discount factor. We use the notation $\Delta(\bm{X})$ to denote the probability distribution over a set $\bm{X}$. The reinforcement learning objective is to maximize the expected return $\mathbb{E}_{\pi}[G_{t}]=\mathbb{E}_{\pi}[\sum_{t}\gamma^{t}r_{t}]$ given a *policy* $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ defining the agent behavior. In actor-critic policy optimization the policy is explicitly represented as a parametric function $\pi:\mathcal{S}\times\bm{\theta}^{\pi}\rightarrow\Delta(\mathcal{A})$, and a *value function* $V:\mathcal{S}\times\bm{\theta}^{V}\rightarrow\mathbb{R}$ is employed to guide optimization. In deep RL  (Mnih et al., ; Silver et al., ) neural networks are used for the policy and value functions, for ease of notation we consider $\bm{\theta}\in\mathbb{R}^{(d_{\pi}+d_{V})}$ as the concatenation of the respective weight vectors.

### 2.2 Proximal Policy Optimization

Proximal policy optimization was proposed by Schulman et al. (), and has since become one of the most popular algorithms for on-policy reinforcement learning. At each iteration $k$ a dataset of transitions $\mathcal{D}_{k}$ is collected using policy $\pi(\bm{\theta}_{k})$, and advantages $\hat{A}_{k}$ are estimated using generalized advantage estimation (GAE) (Schulman et al., ).
The transition dataset and advantages are then used within an inner optimization loop, in which the policy parameters $\bm{\theta}^{\pi}$ are optimized with respect to a given surrogate objective along with the value parameters $\bm{\theta}^{V}$.
Psuedocode for a single iteration of PPO is provided in algorithm 1, where InnerOptimizationLoop is defined in appendix A. The full algorithm updates parameters iteratively by $\bm{\theta}_{k+1}\leftarrow\textsc{PPOIteration}(\bm{\theta}_{k})$.

Figure: Algorithm 1 Proximal policy optimization iteration

PPO permits the use of any arbitrary surrogate objective, though it is most commonly associated with the *clipped objective* Schulman et al. () stated in equation 1.

$$ $L^{\pi}(\bm{\theta}^{\pi})=\mathbb{E}_{s,a\sim\mathcal{D}_{k}}\left[\min\left( \rho(\bm{\theta}^{\pi})\hat{A},\ \text{clip}(\rho(\bm{\theta}^{\pi}),1- \epsilon,1+\epsilon)\hat{A}\right)\right]$ (1) $$

Here $\rho(\bm{\theta}^{\pi})=\frac{\pi(a|s)}{\pi_{k}(a|s)}$ is the ratio between our current policy $\pi$ and the behavior policy $\pi_{k}$, and $\epsilon$ is the clipping threshold. The value function is similarly optimized using either simple regression. $L^{V}(\bm{\theta}^{V})=(V_{\bm{\theta}_{k}}-V_{\text{targ}})^{2}$ or the clipped objective defined in appendix A.

### 2.3 Trust regions

A *trust region* is a region surrounding an optimization iterate $\bm{\theta}_{k}$ within which we permit our algorithm to update the parameters to $\bm{\theta}_{k+1}$. In TRPO, a trust region surrounding the behavior parameters is explicitly defined as the region in parameter space $\bm{\theta}\in\Theta$ satisfying
$\mathbb{E}_{s\sim\mathcal{D}_{k}}\left[D_{\text{KL}}\left(\pi(\bm{\theta}_{k}|
s)\parallel\pi(\bm{\theta}|s)\right)\right]\leq\delta$. Optimizing subject to this constraint prevents large changes in the policy between successive iterations, and gives rise to a guarantee of *monotonic improvement*. Similarly, if the clipped surrogate objective of PPO is replaced with a KL penalty $L^{\pi}(\bm{\theta})=\mathbb{E}_{s,a\sim\mathcal{D}_{k}}[\rho(\bm{\theta})\hat
{A}-\beta D_{\text{KL}}\left(\pi(\bm{\theta}_{k}|s)\parallel\pi(\bm{\theta}|s)
\right)]$, a trust-region is implicitly defined for some $\delta$. Both TRPO and PPO-KL approximate the natural policy gradient (Kakade, ), (Hsu et al., ); the steepest direction in the non-Euclidean geometry of policy space induced by the Fisher information metric.

Unlike the KL penalized surrogate, the clipped surrogate objective of equation 1 *does not* define a formal trust region. We can however define the region of non-zero gradients, with gradient defined as in equation 2.

$$ $\nabla_{\theta^{\pi}}L^{\pi}(\bm{\theta}^{\pi})=\mathbb{E}_{s,a\sim\mathcal{D} _{k}}\left[\hat{A}\nabla_{\bm{\theta}^{\pi}}\rho(\bm{\theta}^{\pi})\cdot \mathbb{I}\left(\left|\rho(\bm{\theta}^{\pi})-1\right|\leq\epsilon\ \text{or} \ \left(\rho(\bm{\theta}^{\pi})-1\right)\hat{A}\leq 0\right)\right]$ (2) $$

Here $\mathbb{I}(\cdot)$ is an indicator function that equals 1 if and only if the argument is true, and 0 otherwise. Whilst the subspace $\nabla_{\bm{\theta}}L^{\pi}>0$ can be considered analogous to a trust region, it is possible to irreversibly step arbitrarily far beyond this region (Hsu et al., ). Nonetheless, where not ambiguous we shall abuse notation and refer to $\nabla_{\bm{\theta}}L^{\pi}>0$ as the trust region of the clipped surrogate. Whilst not defining a formal trust region, the clipped objective enjoys theoretical motivation as a valid drift function in the mirror learning framework (Kuba et al., ), hence also benefits from monotonic improvement and convergence guarantees.

## 3 Outer-PPO

In equation 3 we define the outer gradient of PPO.

$$ $\bm{g}^{O}(\bm{\theta})=\textsc{PPOIteration}(\bm{\theta})-\bm{\theta}$ (3) $$

The behavior parameter update of PPO $\bm{\theta}_{k+1}\leftarrow\textsc{PPOIteration}(\bm{\theta}_{k})$ can now be equivalently expressed as $\bm{\theta}_{k+1}\leftarrow\bm{\theta}_{k}+\bm{g}^{O}(\bm{\theta}_{k})$. Evidently, PPO is exactly gradient ascent, with a constant learning rate $\sigma=1$, on its outer gradients. With this simple result established, we propose a family of methods employing arbitrary optimizers on the PPO outer loop, denoted as *outer-PPO*. As an illustrating example, a comparison of standard PPO and outer-PPO with non-unity learning rates is provided in algorithms 2 and 3. We additionally propose a closely-related method for biasing the inner estimation loop using the prior (outer) trajectory, denoted as *biased initialization*.

### 3.1 Outer Learning Rates

Varying the outer learning rate scales the update applied to the behavior parameters, as defined in algorithm 3 and illustrated in figure 1. The behavior of scaling the outer gradient can not be directly recovered by varying the PPO hyperparameters.

Figure: Algorithm 2 Standard PPO

Figure: Algorithm 3 Outer-LR PPO

An outer learning rate $\sigma<1$ interpolates between the behavior parameters $\bm{\theta}_{k}$ and inner-loop solution $\bm{\theta}_{k}^{*}$, encoding a lack of trust in the outer gradient estimation. Whilst the magnitude of the outer gradient can be reduced by varying hyperparameters, such as the clipping $\epsilon$ or number of inner loop iterations, the outer gradients are inherently noisy due to stochastic data collection and inner-loop optimization. PPO is additionally able to irreversibly escape its clipping boundary (Engstrom et al., ), and can drift far from the behavior policy given sub-optimal surrogate objective parameters. These effects motivate the exploration of methods that attenuate the outer updates, irrespective of the outer gradient magnitude. In contrast, a learning rate $\sigma>1$ amplifies the update vector, encoding confidence in its direction. Whilst the outer gradient magnitude could be increased by varying the PPO hyperparameters, in particular $\epsilon$, increasing the size of the trust region may lead the policy to drift to beyond the region of policy space where the dataset $\mathcal{D}_{k}$ collected with policy $\pi_{k}$ can be considered representative of the environment dynamics, motivating the amplification of well-estimated outer gradients over increases to trust region size.

### 3.2 Momentum

Whilst permitting novel behavior, outer-LR PPO still only exploits information from a single PPO iteration when updating the parameters. Applying momentum breaks this design choice; instead of directly updating the parameters with the scaled outer gradient $\sigma\bm{g}^{O}_{k}$, we update using the Nesterov momentum rule as in algorithm 4 and illustrated in figure 2(a).

Figure: Algorithm 4 Outer-Nesterov PPO

Figure: (a) Outer-Nesterov PPO. At each iteration Outer-Nesterov PPO estimates an outer gradient $\bm{g}^{O}_{k}$, updates the momentum $\bm{m}_{k}$, and steps the parameters using the Nesterov momentum update. The momentum step therefore *precedes* the construction of the following trust region, since it defines the following behavior policy $\pi(\bm{\theta}_{k+1})$.

In supervised learning momentum is motivated using pathological curvature, and the ability to ‘build up speed’ (Sutskever et al., ). Given that the outer gradient is the solution to a surrogate objective, we do not anticipate pathological curvature presenting to the outer optimizer. However, similar to learning rates $\sigma>1$ the increase in effective learning rate of momentum may assist in learning. Momentum can also be motivated here using resilience to noise; since any given collected dataset will be noisy, the outer gradient is also noisy. As using a learning rate $\sigma<1$ corresponded to a lack of trust in any given outer gradient, using momentum corresponds to a smoothing process, where we at no point solely trust a single outer gradient to be accurate.

### 3.3 Biased Initialization

Outer-PPO Nesterov applies a momentum-based update to the outer loop of PPO. This update occurs *before* the successive iteration’s dataset $\mathcal{D}_{k+1}$ is collected, hence the momentum directly determines the behavior parameters $\pi_{k+1}$ for the following surrogate objective. Beyond the effects of stateful inner-loop optimizers such as Adam (Kingma & Ba, ), each outer gradient estimation is independent of the prior trajectory. In contrast we propose biased initialization to apply an outer momentum-based update *after* data is collected, hence *inside* the following trust region problem as in algorithm 5,where $\bm{m}_{k}=\mu\bm{m}_{k-1}+(1-\mu)\bm{g}^{O}_{k}$ is the momentum vector, and in figure 2(b).

Figure: Algorithm 5 PPO iteration with biased initialization

Biased initialization bears a strong similarity to the conjugate gradient initialization employed in Hessian-free optimization (Martens, ). The primary motivation for such techniques would be to better estimate the update vector in a given budget of inner-loop iterations.

## 4 Experiments

### 4.1 Evaluation Procedure

We experiment on subsets of the Brax (Freeman et al., ), Jumanji (Bonnet et al., ), and MinAtar (Young & Tian, ) environment suites, selected as diverse examples of continuous and discrete control problems. We employ the absolute evaluation procedure recommended by Colas et al. () and Gorsane et al. (). Absolute evaluation entails intermediate evaluations during training and a final, large-scale evaluation using the best policy identified to give the ‘absolute’ performance. We train with a budget of $1\times 10^{7}$ transitions, perform 20 intermediate evaluations, and conduct final evaluation using 1280 episodes.

Recognizing the hyperparameter sensitivity of deep reinforcement learning (Hsu et al., ; Engstrom et al., ; Andrychowicz et al., ), we commit significant resources to establishing a strong PPO baseline and fair evaluation. We sweep for a budget of 600 trials per task using the tree-structured Parzen estimator (Bergstra et al., ; Watanabe, ). Each trial is the mean of 4 agents, trained using seeds randomly sampled from $[0,10000]$, for a total of 2400 agents trained per task during baseline tuning. A total of 11 hyperparameters are tuned, each with extensive ranges considered. Full descriptions of the hyperparameter sweep ranges, and the optimal values identified are provided in appendix C.

After hyperparameter tuning a final 64 agents are trained per environment task, where the set of evaluation seeds is non-overlapping with seeds used for hyperparameter tuning. To compare methods we aggregate performance over the tasks of an environment suite following the procedure recommended by Agarwal et al. (), normalizing with the min/max return found for each task across all trained agents (including sweep agents), a table of which is presented in appendix D.

### 4.2 Defined Experiments

We consider the three outer-PPO methods defined in section 3; outer-LR, outer-Nesterov and biased initialization, addressing questions 1, 2, and 3 respectively. The outer-PPO methods are grid searched using increments of 0.1 for all hyperparameters. Outer-LR has a single hyperparameter; outer learning rate $\sigma$, which is swept over the range [0.1, 4.0] (40 trials). Nesterov-PPO two hyperparameters; $\sigma$ [0.1, 1.0] and momentum factor $\mu$ [0.1, 0.9] (90 trials). Biased initialization also has two hyperparameters; bias learning rate $\alpha$ [0.1, 1.0], bias momentum $\mu$ [0.0, 0.9] (100 trials). The base PPO hyperparameters are frozen from the baseline sweep up to the 500th trial, such that no method is tuned using a budget greater than the 600 trials used by the baseline. The optimal hyperparameters identified for each sweep are provided in the figures of appendix E.

Figure: Figure 3: Aggregate point estimates for Brax (upper), Jumanji (center), and MinAtar (lower). Optimal hyperparameters *per-environment* are used. Normalized to task min/max across all experiments.
Refer to caption: x1.png

Figure: Figure 4: Probability of improvement for Brax (left), Jumanji (center), and MinAtar (right). Optimal hyperparameters *per-environment* are used. Normalized to task min/max across all experiments.
Refer to caption: x4.png

## 5 Results

### 5.1 Empirical performance

We first consider the performance of the three outer-PPO methods, where the optimal hyperparameters identified from the grid sweeps *per-environment* are employed.
In figures 3 and 4 we present the aggregate point estimates and probability of improvement.
Further results including sample efficiency curves are provided in appendix D.

Aggregate point estimates. Outer-LR demonstrates a statistically significant improvement over the PPO baseline on Brax and Jumanji for all point estimates considered (median, IQM, mean, optimality gap). Outer-Nesterov also demonstrates enhanced performance on Brax and Jumanji; this improvement is less substantial than that of outer-LR but remains statistically significant on all point estimates aside from the Brax median. Biased initialization is the weakest of the outer-PPO instantiations, with minor improvements lacking statistical significance on Brax and moderate but significant improvements on Jumanji. No method improves over baseline on MinAtar.

Probability of improvement. All methods have a probability of improvement (over baseline) greater than 0.5. In most cases this improvement is statistically significant, aside from biased initialization on Brax and outer-LR on MinAtar. Notably, outer-LR has a probability of improvement greater than 0.6 on Brax and greater than 0.7 on Jumanji.

Figure: Figure 5: Outer-LR hyperparameter sensivity. Mean normalized return across the Brax (left), Jumanji (center), MinAtar (right) tasks as a function of outer learning rate $\sigma$. Mean of 4 seeds plotted with standard error shaded. Normalized to task min/max across all experiments.
Refer to caption: x7.png

Figure: Figure 6: Outer-Nesterov hyperparameter sensitivity. Mean normalized return across the Brax (left), Jumanji (center), MinAtar (right) tasks as a function of outer learning rate $\sigma$ and outer momentum $\mu$. Mean of 4 seeds plotted. Normalized to task min/max across all experiments.
Refer to caption: x10.png

Figure: Figure 7: Biased initialization hyperparameter sensitivity. Mean normalized return across the Brax (left), Jumanji (center), MinAtar (right) tasks as a function of bias init learning rate $\alpha$ and bias momentum $\mu$. Mean of 4 seeds plotted. Normalized to task min/max across all experiments.
Refer to caption: x13.png

### 5.2 Hyperparameter Sensitivity

In the results of figures 3 and 4, the optimal hyperparameters from each *per-environment* outer-PPO grid search are used. We now consider the sensitivity of outer-PPO to these hyperparameters. In figures 5, 6, and 7 we present the return, *normalized across each environment suite*, as a function of the sweep hyperparameters for outer-LR, outer-Nesterov and biased initialization. Normalization is again performed using the extreme values presented in appendix D. Analogous plots for the individual tasks are provided in appendix E.

#### Outer learning rate.

When normalized across all tasks, Brax has low sensitivity to outer learning rate. The range of values $\sigma\in[0.8,2.0]$ has comparable performance to the peak located at $\sigma=1.6$. Notably, performance is not greatly reduced when using values up to $\sigma=3.0$. Jumanji again exhibits near optimal-performance over a broad range of values $\sigma\in[0.5,2.2]$, with the peak again located at $\sigma=1.6$. Unlike in Brax, performance on Jumanji is greatly diminished for values $\sigma>2.5$. MinAtar has a sharp peak in performance around standard PPO ($\sigma=1.0$), with a rapid decrease in performance for values greater than this.

#### Nesterov.

All three suites have a ridge-like trend in normalized performance, with poor performance where $\sigma$ and $\mu$ are both small or both large. Both Brax and Jumanji have their peak at $(\sigma,\mu)=(0.7,0.5)$, with a relatively broad plateau of near-optimal performance. The peak of MinAtar is at $(\sigma,\mu)=(0.9,0.1)$, with a narrow ridge of near-optimal performance.

#### Biased initialization.

The dominant trend on all three suites is decreasing normalized performance for large bias learning rate $\alpha$. The optima for all suites at either $\alpha=0.1$ (Brax, MinAtar) or $\alpha=0.2$ (Jumajji). There is comparably little variation with respect to bias momentum $\mu$, with the suite optima dispersed through the available range. Jumanji has a broader region of near optimal performance than Brax or MinAtar, covering $\alpha<0.4$.

## 6 Discussion

We now reflect on the questions posed in section 1. The PPO baselines in this work were tuned aggressively for each task, greatly increasing the confidence in the experimental findings. Given the baseline strength, and performance demonstrated in figures 3 and 4, we conclude in the *negative* for all three questions as evidenced by:

- Q1.
Varying the outer learning rate leads to an statistically significant increase on all point estimates on Brax and Jumanji, with corresponding increases to probability of improvement.
- Q2.
Employing Nesterov momentum on the outer loop, with outer learning rate attenuation, achieves statistically significant increases to all point estimates on Brax and Jumanji. We also observe a statistically significant probability of improvement on all three suites.
- Q3.
Momentum-biased initialization achieves statistically significant increase on all point estimates on Jumanji, with a probability of improvement of 0.6 on this suite.

#### Common hyperparameters.

The sensitivity plots in figures 5, 6 demonstrate robust normalized performance across the Brax and Jumanji suites for outer-LR and outer-Nesterov. However, they do not indicate any significant increase in normalized return could be achieved over standard PPO for a set of common hyperparameters shared across a suite. To achieve the improved aggregate metrics in figure 3 it was necessary to use task-specific hyperparameters. We do however emphasize the aggressive, task-specific, tuning of the baseline, and view the robustness of normalized return across a range of hyperparameters as a strength of the methods.

#### Task-specific hyperparameters.

Task-specific hyperparameter sensitivity plots are provided in appendix E. For outer-LR the optimal per-task values for $\alpha$ range between 0.5 (corresponding to cautious updates) and 2.3 (corresponding to confident updates). That values of $\alpha$ up to 2.3 can be optimal is surprising, as an $\alpha$ greater than unity directly violates the trust region established by our previous behavior policy. This precludes the provable monotonic improvement of PPO (Kuba et al., ); by stepping beyond the trust region we may in principle select a policy that is worse than the previous. For outer-Nesterov co-varying $\sigma$ with $\mu$ can be understood through the effective learning rate $\sigma/(1-\mu)$. The task-specific effective learning rate varies from 0.7 to 2.3. Lastly, for biased initialization the sharp peaks in performance on Brax tasks suggest the method suffers from high variance on this suite, hence the hyperparameters selected may not be optimal in expectation. On Jumanji the method is significantly less hyperparameter sensitive as evidenced by the smooth contours, providing an explanation for the performance gap observed between these suites.

#### MinAtar results.

No outer-PPO method improved over baseline on MinAtar. We comment that other works committing substantial resources to baseline tuning on MinAtar have struggled to achieve improvements on the suite Jesson et al. (). Furthermore, the hyperparameter sensitivity plots in figures 5, 6 and 7 demonstrate all methods achieve peak normalized return greater than 0.9 on MinAtar. Since here we are normalizing to the maximum performing agents across all sweeps, this indicates there is less variance in the optimal performance of MinAtar compared to Brax and Jumanji with peak normalized returns around 0.7 and 0.8 respectively. A final explanation for the failure to surpass baseline on MinAtar could be ‘brittle’ base hyperparameters, not suited to the modified dynamics introduced by outer-PPO, supported by the sharp peak observed in outer-LR and concentration of performance in outer-Nesterov about standard PPO in figures 5 and 6.

#### Limitations.

We identify two core limitations to this work; the *fixed transition budget* and the *lack of co-optimization* of base and outer-PPO hyperparameters. We only consider a timestep budget of $1\times 10^{7}$ transitions. Whilst sample efficiency plots are provided in appendix D the hyperparameters have not been tuned to maximize performance in the data-limited regime. Furthermore, we do not consider the asymptotic performance for larger transition budgets, where it is possible the improvement achieved by outer-PPO methods may be diminished. With respect to co-optimization, given the dependence of the outer gradients on the base hyperparameters there is undoubtedly significant interaction between these and the outer-PPO hyperparameters. Exploring these interactions would yield better understanding and potentially improved performance. We additionally highlight the presence of learning rate annealing on the inner Adam instances in all experiments. This implies the outer gradients tend to zero, the implications of which we do not explore in this work.

## 7 Related Work

The usage of the difference between initial parameters and those after gradient-based optimization as a ‘gradient’ has been explored for meta-learning in the Reptile algorithm (Nichol et al., ). Reptile aims to find an initialization that can be quickly fine-tuned across a distribution of tasks. Unlike outer-PPO, which applies this idea within a single RL task, Reptile performs gradient steps on different supervised learning tasks to determine the ‘Reptile gradient’. One could interpret outer-PPO as performing serial Reptile whereby each sampled task is the next PPO iteration alongside the collected dataset.

Whilst to the best of our knowledge we are the first to apply momentum to the outer loop of PPO, momentum-based optimizers such as RMSProp Tieleman & Hinton () and Adam Kingma & Ba () are commonly applied in other areas of RL. Recent work has examined the interaction of momentum based optimizers and RL objectives. Bengio et al. () identify that a change in objective (such as by updating a target network or dataset), may lead to momentum estimates anti-parallel to the current gradient thereby hindering progress, and propose a correction term to mitigate this effect. Asadi et al. () propose to reset the momentum estimates periodically throughout training and demonstrate improved performance on the Atari Learning Environment Bellemare et al. () with Rainbow Hessel et al. () doing so. However, none of these approaches focuses on PPO specifically, and instead address temporal difference learning or value based-methods.

Lastly, the biased initialization explored in this work is similar to the conjugate gradient initialization technique employed in hessian-free optimization Martens (), although this used only the prior iterate and not a momentum vector. Hessian-free optimization can be considered a supervised learning version of TRPO (Schulman et al., ).

## 8 Conclusion

In this work, we introduced outer-PPO, a novel perspective of proximal policy optimization that applies arbitrary gradient-based optimizers to the outer loop of PPO. We posed three key research questions regarding the optimization process in PPO and conducted an empirical investigation across 14 tasks from three environments suites. Our experiments revealed that non-unity learning rates and momentum in the outer loop both yielded statistically significant performance improvements across a variety of evaluation metrics in the Brax and Jumanji environments, with gains ranging from 5-10% over a heavily tuned PPO baseline. Biased initialization provided improvements upon the baseline on Jumanji tasks but not Brax.

The most immediate direction for future research would be the exploration of interactions between base hyperparameters and outer-PPO hyperparameters. Since the optimal base hyperparameters may be unsuited to the modified dynamics of outer-PPO, the co-optimization of hyperparameters may yield performance improvements and deeper understanding of the method. Other possible future directions include the use of outer-PPO with alternatives to the clipped surrogate loss function, such as KL-penalized PPO Hsu et al. () or discovered policy optimization Lu et al. (), and the use of adaptive optimizers on the outer loop such as RMSProp or Adam. Indeed, an ‘outer’ variant of many dual-loop RL algorithms can be defined, and we hope that this work will stimulate further research into optimizing RL algorithms through more sophisticated outer-loop strategies.

## Acknowledgments

This work was supported by a Turing AI World-Leading Researcher Fellowship G111021. Research supported with Cloud TPUs from Google’s TPU Research Cloud (TRC)

## Appendix A Further Details on PPO

### A.1 Inner Optimization Loop

Figure: Algorithm 6 PPO Inner Optimization Loop

Algorithm 6 describes the inner optimization loop of proximal policy optimization, where $L^{\pi}$ and $L^{V}$ are defined in equations 1 and 4 respectively. For notational ease this presentation is slightly simplified. Typically, instead of the gradient ascent steps taken in lines 5 and 6 typically each of $\bm{\theta}^{\pi}$ and $\bm{\theta}^{V}$ are optimized using independent instances of Adam (Kingma & Ba, ), with potentially distinct learning rates $\eta^{\pi}\neq\eta^{V}$.

### A.2 Clipped Value Objective

$$ $L^{V}(\bm{\theta}^{V})=\max\left[\left(V_{\bm{\theta}_{k}}-V_{\text{targ}} \right)^{2},\left(\text{clip}\left(V_{\bm{\theta}_{k}},V_{\bm{\theta}_{k-1}}- \varepsilon,V_{\bm{\theta}_{k-1}}+\varepsilon\right)-V_{\text{targ}}\right)^{2 }\right]$ (4) $$

## Appendix B Implementation details

We implement our experiments using the JAX-based Stoix library (Toledo, ). Our implementation is such that several seeds can be trialed / evaluated simultaneously for the same hyperparameters using a single device. We used Google Cloud TPU (v4-8) for these experiments. The runtime varied between environments, and given different hyperparameters (e.g parallel environments, rollout length, batch size, number of inner epochs) with an average of approximately 10 minutes per 4-seed trial.

## Appendix C Hyperparameters

### C.1 Sweep Ranges

The sweep ranges for baseline hyperparameter sweeps are presented in table 1.

**Table 1: Sweep ranges for baseline hyperparameters.**
| Parameter | Sweep Range |
| --- | --- |
| Parallel environments | $2^{6}$ to $2^{10}$ |
| Rollout | $2^{2}$ to $2^{8}$ |
| Num. epoch | 1 to 16 |
| Num. minibatch | $2^{0}$ to $2^{6}$ |
| Actor learning rate | $1\times 10^{-5}$ to $1\times 10^{-3}$ (log scale) |
| Critic learning rate | $1\times 10^{-5}$ to $1\times 10^{-3}$ (log scale) |
| Discount factor ($\gamma$) | 0.9 to 1.0 |
| GAE $\lambda$ | 0.0 to 1.0 |
| Clip $\epsilon$ | 0.1 to 0.5 |
| Max gradient norm | 0.1 to 5.0 |
| Reward scaling | $0.1$ to $100$ (log scale) |

### C.2 Optimal Values

The optimal values identified by the baseline sweep, up to trial 500, are included in table 2. These values are the ‘base’ hyperparameters used for outer-PPO methods.

**Table 2: Optimal values from baseline sweep up to trial 500**
| Task | Parallel env. | Rollout | Num. epoch | Num. m-batch | Actor lr | Critic lr | Discount $\gamma$ | GAE $\lambda$ | Clip $\epsilon$ | Max g. norm | Reward scale |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ant | 128 | 8 | 2 | 32 | 3.0e-04 | 1.4e-04 | 0.98 | 0.70 | 0.21 | 4.85 | 0.14 |
| halfcheetah | 64 | 64 | 3 | 16 | 3.9e-04 | 4.4e-04 | 0.99 | 0.94 | 0.13 | 2.40 | 0.46 |
| hopper | 64 | 64 | 2 | 64 | 6.3e-04 | 3.6e-04 | 1.00 | 0.96 | 0.17 | 3.54 | 3.95 |
| humanoid | 256 | 64 | 4 | 64 | 1.0e-04 | 1.0e-04 | 0.98 | 0.89 | 0.34 | 3.30 | 0.14 |
| humanoidstandup | 64 | 64 | 3 | 32 | 3.0e-04 | 8.2e-04 | 0.99 | 0.98 | 0.10 | 4.65 | 0.35 |
| walker2d | 256 | 32 | 4 | 64 | 5.4e-04 | 8.2e-04 | 1.00 | 0.92 | 0.12 | 3.74 | 22.54 |
| asterix | 128 | 128 | 3 | 64 | 8.3e-04 | 2.1e-05 | 1.00 | 0.20 | 0.30 | 2.28 | 6.62 |
| breakout | 64 | 16 | 14 | 16 | 1.8e-04 | 1.2e-04 | 0.90 | 0.53 | 0.16 | 0.25 | 5.19 |
| freeway | 64 | 128 | 10 | 2 | 6.9e-04 | 1.3e-04 | 0.98 | 0.70 | 0.15 | 4.71 | 6.64 |
| space_invaders | 128 | 32 | 16 | 2 | 3.0e-05 | 1.1e-04 | 0.98 | 1.00 | 0.25 | 0.35 | 0.61 |
| game_2048 | 1024 | 8 | 9 | 32 | 4.9e-04 | 3.8e-04 | 0.99 | 0.04 | 0.28 | 2.56 | 0.13 |
| maze | 256 | 32 | 7 | 64 | 6.5e-04 | 4.3e-04 | 0.98 | 0.66 | 0.14 | 2.46 | 1.97 |
| rubiks_cube | 64 | 256 | 13 | 4 | 9.0e-04 | 2.2e-04 | 0.99 | 0.55 | 0.14 | 3.45 | 11.03 |
| snake | 1024 | 8 | 11 | 4 | 6.0e-04 | 6.0e-04 | 1.00 | 0.46 | 0.12 | 2.52 | 20.48 |

**Table 3: Optimal hyperparameters per task for each outer-PPO method**
| Task | Outer-LR | Outer-Nesterov | Biased Initialization |  |  |
| --- | --- | --- | --- | --- | --- |
|  | $\sigma$ | $\sigma$ | $\mu$ | $\alpha$ | $\mu$ |
| Ant | 0.5 | 0.7 | 0.2 | 0.1 | 0.8 |
| HalfCheetah | 0.5 | 0.4 | 0.5 | 0.2 | 0.8 |
| Hopper | 1.5 | 0.9 | 0.4 | 0.5 | 0.8 |
| Humanoid | 1.9 | 0.5 | 0.7 | 0.1 | 0.4 |
| HumanoidStandup | 2.1 | 0.5 | 0.3 | 0.5 | 0.8 |
| Walker2d | 2.0 | 0.9 | 0.6 | 0.4 | 0.0 |
| 2048 | 1.3 | 0.8 | 0.4 | 0.3 | 0.9 |
| Snake | 2.3 | 1.0 | 0.4 | 0.7 | 0.5 |
| Rubik’s Cube | 1.7 | 0.5 | 0.7 | 0.4 | 0.3 |
| Maze | 0.9 | 0.9 | 0.0 | 0.1 | 0.5 |
| Asterix | 1.1 | 0.6 | 0.5 | 0.1 | 0.4 |
| Breakout | 1.1 | 0.9 | 0.1 | 0.0 | 0.5 |
| Freeway | 1.6 | 0.9 | 0.3 | 0.2 | 0.5 |
| Space Invaders | 1.3 | 0.8 | 0.2 | 0.1 | 0.9 |

## Appendix D Additional Results

Figure: Figure 8: Performance profiles for Brax (left), Jumanji (center), and MinAtar (right). 6 / 4 / 4 tasks used from Brax / Jumanji / MinAtar respectively. For each task, agents are trained and evaluated using 64 seeds.
Refer to caption: x16.png

Figure: Figure 9: Sample efficiency curves for Brax (left), Jumanji (center), and MinAtar (right).
Refer to caption: x19.png

**Table 4: Minimum and maximum returns used for normalization.**
| Task | Min | Max |
| --- | --- | --- |
| Ant | -2958.14 | 13466.48 |
| Halfcheetah | -587.37 | 7859.28 |
| Hopper | 21.03 | 3697.39 |
| Humanoid | 207.63 | 11851.71 |
| Humanoidstandup | 6686.00 | 71897.67 |
| Walker2d | -32.44 | 2558.61 |
| 2048 | 989.50 | 29084.63 |
| Snake | 0.00 | 92.55 |
| Rubiks Cube | 0.00 | 0.66 |
| Maze | 0.03 | 0.84 |
| Asterix | 0.30 | 64.46 |
| Breakout | 0.00 | 92.86 |
| Freeway | 0.00 | 66.13 |
| Space Invaders | 0.00 | 191.80 |

## Appendix E Sweep Performances

### E.1 Baseline

Figure: (a) ant
Refer to caption: x22.png

Figure: (a) asterix
Refer to caption: x28.png

Figure: (a) game_2048
Refer to caption: x32.png

### E.2 Outer learning rates

Figure: (a) ant
Refer to caption: x36.png

Figure: (a) asterix
Refer to caption: x42.png

Figure: (a) game_2048
Refer to caption: x46.png

### E.3 Nesterov Momentum

Figure: (a) ant
Refer to caption: x50.png

Figure: (a) asterix
Refer to caption: x56.png

Figure: (a) game_2048
Refer to caption: x60.png

### E.4 Biased Initialization

Figure: (a) ant
Refer to caption: x64.png

Figure: (a) asterix
Refer to caption: x70.png

Figure: (a) game_2048
Refer to caption: x74.png

## Appendix F Individual Task Performances

Figure: (a) ant
Refer to caption: x78.png

Figure: (a) asterix
Refer to caption: x84.png

Figure: (a) game_2048
Refer to caption: x88.png

## Peer Reviews

confidence: 2
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. Line 37: why is “exactly coupled” emphasized? Some newly defined terms, like the behaviour parameters, can be emphasized instead.

2. What is the intuition behind the biased initialization? Trust region is used to reduce the off-policy distribution shift, but biased initialization worsens the situation.

3. Why were Brax and Jumanji chosen instead of MuJoCo or the DeepMind suite?

4. Line 303, during the hyperparameter tuning, does each trial represent a choice of the hyperparameters, and does each agent represent a random seed? Can learning curves for the baseline be provided? How is the final performance of the baseline? Could you help me read Figure 10? What are the points and meanings of the x-axis and y-axis?

5. In Figure 3, how is the metric, optimality gap, defined?

6. The result in Figure 4 is very straightforward. How is this probability metric computed? From Figure 3, there is no significant difference between algorithms. However, the probability of the proposed algorithms being better than the baseline is larger than 0.5. What causes the probability measure to lean towards proposed algorithms?

7. Line 313, “normalizing with the min/max return found for each task across all trained agents (including sweep agents).” What are sweep agents? How is the normalization conducted with the min/max return? In line 470, why does a 0.9 peak normalized return imply less variance compared to 0.7 or 0.8?

8. For the outer loop, is the gradient update applied without using any optimizers?

9. Are these outer loop hyperparameters task-dependent? Could they differ a lot for each task?

10. What does a smaller stepsize on a policy improvement direction suggest? One hypothesis can be the PPO’s gradient direction is not toward policy improvement. However, the algorithm won’t learn in this case, and this hypothesis does not seem to hold.
soundness: 3
strengths: The algorithms are clearly introduced with the help of figure representations. The paper makes claims on its empirical performances, which are well supported by the empirical results.
summary: The paper designs a novel framework, called outer-PPO, to further modify PPO’s trust region gradients through an outer loop. PPO conducts several gradient updates using each set of collected data. The proposed framework computes these gradients in the inner loop without updating and combines these gradients into one update with extra stepsize and momentum designs in the outer loop. The proposed algorithms are evaluated on Brax, Jumaji, and MinAtar environments, showing statistically significant improvement on Brax and Jumaji and comparable performance on MinAtar.
weaknesses: The novelty of the algorithm can be explained more. Empirically speaking, outer-PPO with a non-unity learning rate performs best on Brax and Jumaji and functions as the main contribution. However, what is the difference between this proposed algorithm and rescaling the learning rate of the original PPO? Is the empirical result suggesting that stochastic gradient descent can be a better optimizer than the commonly used Adam?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: I ask the authors to clarify what is the advantage of introducing the learning rate instead of modifying $\epsilon$.
soundness: 3
strengths: Originality
--------------

The idea presented in the paper is, as far as I know, novel.

Quality and Clarity
-------------------------

The paper presents the idea in a clear way, highlighting the main research questions well and providing a robust empirical analysis of the proposed algorithm (with very detailed ablation studies). The algorithm is sound and coherent with the investigation objective.

Significance
----------------

I am unsure about the significance of the proposed idea. The paper introduces new hyperparameters that make PPO more complicated rather than more simple. I am unsure whether the complexity introduced is a payoff of the little gains in terms of performance. However, the takeaway message of seeing the outer loop of PPO as a gradient ascent procedure is interesting.
summary: This paper proposes an improvement of Proximal Policy Optimization (PPO), one of the most well-known online policy gradient algorithms.

PPO works by following two nested optimization stages: the inner optimization optimizes a "clipped surrogate objective" that can be seen as a constrained optimization problem, where the optimal solution optimizes the expected advantage while keeping the target policy close to the data; the outer optimization stage simply loops simply updates the target policy and collects new data. 

In PPO, the inner objective is optimized by gradient ascent, while the outer objective is seen simply as a loop.

The authors' idea is to see the inner optimization as a gradient estimation procedure and the outer loop as a gradient ascent procedure. By leveraging this idea, the authors explore different learning rates (which in PPO is inherently set to 1), the application of momentum, and the possibility of "biasing" the initialization of the inner loop by leveraging on the outer gradient ascent.
weaknesses: As I have mentioned above, I think a weakness of the proposed method is the introduction of new hyperparameters (i.e., outer learning rate and momentum) - with what seems to be little payoff.

Furthermore, I am unsure whether the learning rate is necessary: the hyperparameter $\epsilon$ of PPO already provides a mechanism to control "how aggressive" the policy updates are. While I can acknowledge that the learning rate and the $\epsilon$ are two different terms (i.e., the learning rage $\sigma$ acts in the parameter space, while the hyperparameter $\epsilon$ acts in the "policy" space), I can't see what is the advantage of using $\sigma$ in place of modifying $\epsilon$.

Perhaps, as highlighted in the "Strength" section, the main weak point of the paper is the significance.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: For the outer-PPO hyperparameter tuning, did you also first tune using 4 seeds, and then run a final evaluation using 64 new seeds, just like for the standard PPO experiments?

In general, I did not find the paper interesting, and I would expect much larger improvements in performance (2x improvement or something around that) for me to recommend the work for publication. There is no strong justification for why the proposed method solves a fundamental limitation in PPO or why it would lead to a fundamental improvement in performance. I do not expect any practitioner to adopt the proposed algorithm. For this kind of work, I would expect large empirical performance gains for me to recommend it for publication. Therefore, I do not foresee myself changing my score unless such results are provided in the rebuttal. I would recommend aiming to submit the work to a venue that de-emphasizes significance, and merely looks at correctness of the claims.

Typos:
“Given it’s” → “Given its”
Line 195: “appendix ??” the reference link is missing.
soundness: 2
strengths: - The paper is clear.
- There do a lot of experiments, and the results are reported faithfully.
summary: The typical PPO algorithm performs an inner loop optimization at each policy update step. With a fixed trajectory data and current policy parameter $\theta$, it optimizes an update to the policy parameters $\theta’$, then applies the new parameters to get more trajectory data. The current paper proposes to modify this update rule by instead considering $\Delta = \theta’ - \theta$ as a replacement to the gradient in any typical gradient-based optimizer. For example, one can consider the update $\sigma \Delta$. In this case $\sigma=1$ will correspond to the standard PPO, but by changing $\sigma$ it generalizes the algorithm. As the original PPO is included in this class of updates, tuning the parameter is guaranteed to at least not decrease the performance compared to PPO. They compare performance on Brax continuous control tasks, MinAtar and Jumanji, and through an extensive hyperparameter tuning procedure claim 5-10% improved performance for the same number of hyperaprameter optimization trials (there were some differences in the hyperparameter optimization procedure between outer-PPO and regular PPO).
weaknesses: - The performance improvements are at best small (5-10%)
- The performance change could also be due to changes in hyperparameter tuning procedures.
Currently, the authors tune PPO by optimizing 11 hyperparameters for 600 trials (each trial averaged across 4 seeds) using a Tree Structured Parzen method, and then doing a final evaluation at the best parameters for 64 seeds (different from the initial 4 seeds). And the outer-PPO methods are hyperparameter tuned by first doing 500 trials of PPO hyperparameter optimization, then doing an additional 100 trials by a hyperparameter sweep over the outer-PPO parameters. These two optimization methods are inconsistent, as outer-PPO includes a direct sweep over the hyperparameters. Such an inconsistency in tuning may lead to an improved performance for outer-PPO. There is no guarantee that, for example, the standard PPO may also not perform better if it first does 500 trials of optimization with the Tree Structured Parzen, and then does a 100 trial sweep over some chosen important hyperparameters. In general, while I appreciate that the authors tried to tune the parameters exhaustively, 11 hyperparameters are a lot to tune, and with the evaluation noise due to using only 4 seeds it is difficult to create a fully convincing evaluation method where it would be clear that small improvements like 5-10% are due to an improvement in the algorithm, and not some small details in the tuning procedure. Perhaps a simpler experimental protocol would be more convincing. For example, tune PPO  on all tasks, then employ a single fixed outer learning rate (the same one across all tasks). If such a simple procedure lead to an improved performance, it would be more convincing.

- The optimal outer-PPO hyperparameters are different per task.
- No strong justification is given why we should expect the new method to lead to large improvements in performance (whilst it is clear that the performance should not decrease under proper tuning, there is no indication that large improvements are expected.)
- The outer learning rate sensitivity plots show that the peak performance is not much different compared to $\sigma=1$, which corresponds to standard PPO

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: Check the weaknesses section above.
soundness: 2
strengths: 1. The idea of the proposed method (outer-PPO) is interesting.
2. This paper provides a comprehensive evaluation of the proposed method.
summary: This paper introduces outer-PPO, a novel perspective of proximal policy optimization that applies arbitrary gradient-based optimizers to the outer loop of PPO.
weaknesses: 1. In section 5.1, it would make this paper stronger if the authors could explain metrics in detail. 
2. The analysis in section 5.1 could dive deeper into why, in some environments, the methods proposed in this paper work well, while in other environments, these methods are not better than the baseline PPO. For example, the authors could provide some hypotheses and use some experiments to support these hypotheses. 
3. In section 5.2, hyperparameter sensitivity, it would make this paper stronger if the authors could compare the hyperparameter sensitivity of their methods with baseline PPO.
4. Given Figure 8 and Figure 9 in the appendix, it is unclear if the final converged performance of the proposed methods would be better than baseline PPO since it seems all algorithms still haven’t converged.

Suggestions:
1. In the experiments that the authors conducted, it seems the proposed methods do not show better performance than baseline PPO, especially since it is unclear which implementation version of baseline PPO the authors used. Currently, there are many different implementations of PPO, and different implementations of PPO can significantly affect the final performance [1]. For research focus, it is acceptable that the proposed method does not perform better than PPO in general environments. However, it would make the paper stronger if the authors could provide evidence in which situations (such as environments with high dimensional input or output), the proposed method is better than PPO.
2. It would make this paper stronger if the authors could provide the details of the training time of each algorithm with the same number of timesteps.
3. Page 4, Line 195, appendix ??.

[1] The 37 Implementation Details of Proximal Policy Optimization

Meta-review: The paper proposes a new algorithm that creates an inner and outer loop PPO method where the inner loop estimates an update direction and the outer loop actually changes the policy. The paper performs a non-insignificant amount of hyperparameter tuning to make a performance comparison and perform some hyperparameter sensitivity analysis of the new algorithm. The primary concerns of the reviewers are around the relevance of the method and the empirical results being meaningful. To these complaints, I will add that the hyperparameter tuning while more than done in most papers, is not sufficient for accurate statistical accounting of the performance differences. Because the chosen hyperparameters are the result of a stochastic process, this uncertainty needs to be accounted for in the final results. Consequently, the statistical comparisons of the algorithms do not account for this randomness and thus, conclusions cannot be accurately drawn. 

I also want to add that it is not clear what problem this new method is actually solving. It may be novel and it could work better, but it is unclear what problem with policy optimization algorithms is actually being addressed. It is also unclear how this new method actually impacts the optimization process differently than PPO, this is a common complaint made by reviewers and there should be evidence to show this effect.

Decision: Reject