Simple Policy Optimization

## Abstract

Abstract As one of the most important and influential algorithms in reinforcement learning, the Proximal Policy Optimization (PPO) algorithm has demonstrated outstanding performance across various domains. It simplifies the optimization-based importance sampling process of the Trust Region Policy Optimization (TRPO) algorithm through ratio clipping. However, this simplification with ratio clipping does not always effectively enforce trust region constraints. In this paper, we introduce an algorithm named Simple Policy Optimization (SPO), which incorporates a novel clipping method for the KL divergence between the old and new policies. Extensive experimental results in both Atari 2600 and MuJoCo environments show that, compared to PPO, SPO achieves better sample efficiency, extremely low KL divergence, and higher policy entropy, while also being robust to increases in network depth or complexity. More importantly, SPO maintains the simplicity of an unconstrained first-order algorithm. Our code is available at https://github.com/MyRepositories-hub/Simple-Policy-Optimization .

## 1 Introduction

Reinforcement learning has made rapid progress in recent years, becoming a cornerstone tool across a wide range of disciplines.
There are two primary paradigms in deep reinforcement learning, value-based and policy-based. Policy-based model-free reinforcement learning algorithms have been extensively applied across various domains, achieving performances that surpass human capabilities in areas including board games [, , ], video games [, , ], autonomous driving [, ], and intelligent control systems [, , ]. Furthermore, notable algorithms like PPO [] and A3C [] utilize the Actor-Critic (AC) framework [], which integrates these paradigms and serves as a foundational component in numerous fields, including games, robotics, large language models (LLMs), and more. In this paper, we focus on policy-based reinforcement learning algorithms and introduce the Simple Policy Optimization (SPO), which features a novel clipping method for importance sampling.

The Proximal Policy Optimization (PPO) algorithm [] limits the difference between the old and new policies through a ratio clipping operation, which in the Trust Region Policy Optimization (TRPO) algorithm [] is manifested as a hard constraint on expected KL divergence. Ratio clipping circumvents the need to solve the computationally expensive KL divergence constraint problem, thus allowing the PPO algorithm to be effectively extended to large-scale reinforcement learning environments [, ]. Previous studies have explored the relationship between ratio clipping and trust region constraints. For instance, even if the probability ratio is bounded, the corresponding KL divergence does not necessarily be bounded []. The performance improvements in the PPO algorithm are likely primarily dependent on “code-level optimizations” []. Additionally, some works have designed novel probability ratio clipping methods or objective functions [, , , ], or alternated between a standard RL phase and a distillation phase during the training stage to enhance the sample efficiency of PPO [, ].

There are numerous strategies to mitigate excessive KL divergence, leading to various adaptations of the PPO algorithm. For instance, KL divergence can be incorporated as a penalty term within the surrogate objective, complemented by an adaptive penalty coefficient []. During PPO’s training phase, a predefined KL threshold is established. An early stopping mechanism is triggered once this KL divergence threshold is reached []. Alternatively, the early stopping strategy can be based on the deviation in the probability ratio rather than on KL divergence itself [].

However, the aforementioned approaches have not fundamentally altered the characteristics of the PPO algorithm (we acknowledge that the surrogate objective with ratio clipping remains the preferred choice in most cases), nor have they effectively constrained the KL divergence between the old and new policies. We propose the Simple Policy Optimization (SPO) algorithm to address these shortcomings. Notably, SPO does not require the clipping of the probability ratio; instead, it introduces a novel clipping method for the KL divergence between the old and new policies. We highlight three important contributions in SPO:

- •
SPO can effectively limit the average KL divergence between the old and new policies in almost all environments, while the PPO algorithm inevitably results in a high value of KL divergence between the successive policies.
- •
SPO requires the calculation of the KL divergence between successive policies, which does not increase much computational overhead, and still, it maintains the simplicity of an unconstrained first-order algorithm.
- •
The aforementioned characteristics of SPO are robust to increases in network depth or complexity. In contrast, the increases in KL divergence between successive policies of PPO becomes more evident as the network deepens.

Based on the above three points, SPO can achieve better monotonic improvement than PPO, and its first-order property makes it easily scalable to distributed training with many workers, delivering strong performance as well.

## 2 Background

In this section, we introduce the background knowledge of policy gradients in reinforcement learning, as well as the TRPO and PPO algorithms.

### 2.1 Policy Gradient

Reinforcement learning is generally defined by a tuple $(\mathcal{S},\mathcal{A},r,\mathcal{P},\rho_{0},\gamma)$, where $\mathcal{S}$ and $\mathcal{A}$ represent the state space and action space, $r:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward function, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1]$ is the probability distribution of the state transition function, $\rho_{0}:\mathcal{S}\mapsto[0,1]$ is the initial state distribution, and $\gamma\in(0,1]$ is the discount factor.

For each trajectory

$$ $\tau=(s_{0},a_{0},r_{0},\dots,s_{T-1},a_{T-1},r_{T-1},s_{T}),$ (1) $$

the objective function can be written as

$$ $J(\theta)=\mathbb{E}_{\tau\sim p_{\theta}(\cdot)}\left[R(\tau)\right]=\sum_{ \tau}R(\tau)\cdot p_{\theta}(\tau),$ (2) $$

where $\theta$ is the parameters of the policy network, $R(\tau)=\sum_{t=0}^{T-1}\gamma^{t}r_{t}$. Then the policy gradient can be estimated as

$$ $\begin{split}\nabla J(\theta)&=\mathbb{E}_{\tau\sim p_{\theta}(\cdot)}\left[R( \tau)\cdot\nabla\log p_{\theta}(\tau)\right]\\ &=\mathbb{E}_{\tau\sim p_{\theta}(\cdot)}\left\{R(\tau)\cdot\nabla\log\left[ \rho_{0}(s_{0})\cdot\prod_{t=0}^{T-1}\pi_{\theta}(a_{t}|s_{t})\cdot\mathcal{P} (s_{t+1}|s_{t},a_{t})\right]\right\}\\ &=\mathbb{E}_{\tau\sim p_{\theta}(\cdot)}\left\{R(\tau)\cdot\nabla\left[\log \rho_{0}(s_{0})+\sum_{t=0}^{T-1}\log\pi_{\theta}(a_{t}|s_{t})+\log\mathcal{P}( s_{t+1}|s_{t},a_{t})\right]\right\}\\ &=\mathbb{E}_{\tau\sim p_{\theta}(\cdot)}\left[\sum_{t=0}^{T-1}R(\tau)\cdot \nabla\log\pi_{\theta}(a_{t}|s_{t})\right]\approx\frac{1}{N}\sum_{n=1}^{N}\sum _{t=0}^{T_{n}-1}\left[R(\tau^{n})\cdot\nabla\log\pi_{\theta}(a_{t}^{n}|s_{t}^{ n})\right],\\ \end{split}$ (3) $$

and now we can estimate $\nabla J(\theta)$ through the Monte Carlo (MC) method [].

### 2.2 Trust Region Policy Optimization

In TRPO [], the following monotonic improvement theorem is given:

###### Theorem 1 .

Let $\alpha=D_{\rm TV}^{\max}(\pi,\tilde{\pi})$ and $\epsilon=\max_{s,a}\left|A_{\pi}(s,a)\right|$, the following bound holds:

$$ $\eta(\tilde{\pi})\geq L_{\pi}(\tilde{\pi})-\frac{4\epsilon\gamma}{(1-\gamma)^{ 2}}\cdot\alpha^{2},$ (4) $$

where $L_{\pi}$ matches the original objective $\eta$ to first order if the policy $\pi$ is a differentiable function of a parameter vector [].

Noting the relationship between total variation divergence and the KL divergence $D_{\rm TV}(\pi,\tilde{\pi})^{2}\leq D_{\rm KL}(\pi,\tilde{\pi})$, we have the following corollary:

###### Corollary 2 .

Let $D_{\rm KL}^{\max}(\pi,\tilde{\pi})=\max_{s}D_{\rm KL}(\pi,\tilde{\pi})$, the following bound holds:

$$ $\eta(\tilde{\pi})\geq L_{\pi}(\tilde{\pi})-\frac{4\epsilon\gamma}{(1-\gamma)^{ 2}}\cdot D_{\rm KL}^{\max}(\pi,\tilde{\pi}).$ (5) $$

Theoretically, this lower bound is really powerful, this is because if we denote it as

$$ $M_{\pi}(\tilde{\pi})=L_{\pi}(\tilde{\pi})-\frac{4\epsilon\gamma}{(1-\gamma)^{2 }}\cdot D_{\rm KL}^{\max}(\pi,\tilde{\pi}),$ (6) $$

we have $\eta(\tilde{\pi})\geq M_{\pi}(\tilde{\pi})$ and $\eta(\pi)=M_{\pi}(\pi)$, and then we obtain

$$ $\eta(\tilde{\pi})-\eta(\pi)\geq M_{\pi}(\tilde{\pi})-M_{\pi}(\pi).$ (7) $$

This means that we can indirectly optimize the original objective $\eta$ by optimizing the lower bound $M_{\pi}$. Through a heuristic approximation, the maximum KL divergence $D_{\rm KL}^{\max}$ in the Corollary 2 is approximated as the average KL divergence and used as a constraint to obtain the formalization of TRPO algorithm, that is,

$$ $\begin{split}\max_{\theta}\enspace&\mathbb{E}_{(s_{t},a_{t})\sim\pi_{\theta_{ \rm old}}}\left[\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\rm old}}(a_{t}| s_{t})}\cdot\hat{A}(s_{t},a_{t})\right],\\ \text{s.t.}\hskip 2.5pt\enspace&\mathbb{E}\left[D_{\text{KL}}(\pi_{\theta_{\rm old }},\pi_{\theta})\right]\leq\delta.\\ \end{split}$ (8) $$

This constrained optimization problem can be solved using the conjugate gradient algorithm followed by a line search.

###### Theorem 1 .

###### Corollary 2 .

### 2.3 Proximal Policy Optimization

PPO [] is an algorithm that attains the data efficiency and reliable performance of TRPO, while using only first-order optimization. PPO algorithm has two main variants, one of them is called PPO-Penalty, that is,

$$ $\max_{\theta}\enspace\mathop{\mathbb{E}}_{(s_{t},a_{t})\sim\pi_{\theta_{\rm old }}}\left\{\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_{ t})}\cdot\hat{A}(s_{t},a_{t})-\beta\left[D_{\text{KL}}(\pi_{\theta_{\text{old} }},\pi_{\theta})\right]\right\},$ (9) $$

where $\beta$ is adaptive KL penalty coefficient. Another one is called PPO-Clip, which can be formalized as

$$ $\max_{\theta}\enspace\mathop{\mathbb{E}}_{(s_{t},a_{t})\sim\pi_{\theta_{\rm old }}}\Bigg{\{}\min\Bigg{[}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\rm old} }(a_{t}|s_{t})}\cdot\hat{A}(s_{t},a_{t}),\text{clip}\left(\frac{\pi_{\theta}(a _{t}|s_{t})}{\pi_{\theta_{\rm old}}(a_{t}|s_{t})},1\pm\epsilon\right)\cdot\hat {A}(s_{t},a_{t})\Bigg{]}\Bigg{\}}.$ (10) $$

PPO-Clip has been widely adopted by the academic community due to its simple form and reliable performance.

## 3 Method

In TRPO, trust region are imposed by limiting the expectation of the KL divergence between the old and new policies. Here, we denote the KL divergence under state $s_{t}$ as

$$ $d=D_{\text{KL}}\left[\pi_{\theta_{\rm old}}(\cdot|s_{t}),\pi_{\theta}(\cdot|s_ {t})\right]=\sum_{a\in\mathcal{A}}\pi_{\theta_{\rm old}}(a|s_{t})\cdot\log \frac{\pi_{\theta_{\rm old}}(a|s_{t})}{\pi_{\theta}(a|s_{t})},$ (11) $$

and the surrogate objective

$$ $\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})=\frac{\pi_{\theta}(a_{t} |s_{t})}{\pi_{\theta_{\rm old}}(a_{t}|s_{t})}\cdot\hat{A}(s_{t},a_{t}).$ (12) $$

According to the Corollary 2, we hope to optimize the surrogate objective $\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})$ with the KL divergence $d$ as small as possible, as it optimizes the monotonic improvement lower bound in policy iterations. This minimax problem seems to be equivalent to the following form:

$$ $\max_{\theta}\enspace\mathop{\mathbb{E}}_{(s_{t},a_{t})\sim\pi_{\theta_{\rm old }}}\left[\frac{\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})}{d}\right].$ (13) $$

However, note that if we simply using the above form as the objective for the policy network, this can lead to significant problems. Now Assume that the estimated value of the advantage $\hat{A}(s_{t},a_{t})$ is positive, the policy network would strive to minimize the KL divergence in order to maximize the objective. This tendency would result in the network preferring to make only minor adjustments to the parameters, thereby hindering its ability to learn effectively from historical data.

To address this issue, we aim to guide the policy network to avoid situations where the KL divergence $d$ becomes excessively small. For example, we artificially set a threshold $d_{\max}$. When $d\leq d_{\max}$, the objective is the surrogate objective $\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})$. Otherwise, if $d>d_{\max}$, then the objective becomes the ratio form of $(\ref{ratio})$. The formalization will be

$$ $\mathop{\mathbb{E}}_{(s_{t},a_{t})\sim\pi_{\theta_{\rm old}}}\left[\mathfrak{O }_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})\cdot\frac{d_{\rm clip}}{d}\right],$ (14) $$

where $d_{\text{clip}}=\text{clip}(d,0,d_{\max})$, $d_{\max}$ is a hyperparameter that constrains the upper bound of KL divergence. As we expected:

$$ $\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})\cdot\frac{d_{\rm clip}}{ d}=\left\{\begin{aligned} &\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t }),&d\leq d_{\max};\\ &\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})\cdot\frac{d_{\max}}{d}, &d>d_{\max}.\\ \end{aligned}\right.$ (15) $$

Objective $(\ref{positive})$ only considers the case where the estimated value of the advantage $\hat{A}(s_{t},a_{t})$ is positive. However, when $\hat{A}(s_{t},a_{t})$ is negative and $d$ exceeds the threshold, we need to increase the absolute value of objective as a penalty instead of decreasing it. Therefore, by combining both cases, we formalize the objective:

$$ $\mathop{\mathbb{E}}_{(s_{t},a_{t})\sim\pi_{\theta_{\rm old}}}\left[\mathfrak{O }_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})\cdot\left(\frac{d_{\rm clip}}{d}+ \sigma-1\right)\cdot\sigma\right],$ (16) $$

where $\sigma=\text{sign}\left[\hat{A}(s_{t},a_{t})\right]$. We now refer to objective $(\ref{d_clip/d})$ as $J(\theta)$. Obviously, we have

$$ $J(\theta)=\left\{\begin{aligned} &\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{ t},a_{t})\cdot\frac{d_{\text{clip}}}{d},&\enspace\hat{A}(s_{t},a_{t})>0;\\ &\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})\cdot\left(-\frac{d_{ \text{clip}}}{d}+2\right),&\enspace\hat{A}(s_{t},a_{t})<0.\\ \end{aligned}\right.$ (17) $$

Furthermore, if $d\leq d_{\max}$, we have $\left(d_{\text{clip}}/d+\sigma-1\right)\cdot\sigma\equiv 1$, which means that the KL divergence satisfies the trust region constraints, and the goal $J(\theta)$ will be the surrogate objective $\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})$. Otherwise,

$$ $J(\theta)=\left\{\begin{aligned} &\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{ t},a_{t})\cdot\frac{d_{\max}}{d},&\enspace\hat{A}(s_{t},a_{t})>0;\\ &\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})\cdot\left(-\frac{d_{ \max}}{d}+2\right),&\enspace\hat{A}(s_{t},a_{t})<0.\\ \end{aligned}\right.$ (18) $$

Figure: Figure 1: Function curves of $\left(d_{\text{clip}}/d+\sigma-1\right)\cdot\sigma$ when $d_{\max}$ is set to 0.01.
Refer to caption: x1.png

We will provide intuition about the objective (16), suppose that $d_{\max}=0.01$, and the function curves are shown in Figure 1. If $\hat{A}(s_{t},a_{t})>0$ and $d>d_{\max}$, the objective $J(\theta)$ will decrease rapidly (see curve $d_{\text{clip}}/d$), this can force the KL divergence to satisfy the trust region constraints. Otherwise, when $\hat{A}(s_{t},a_{t})<0$ and $d>d_{\max}$, the function curve $-d_{\text{clip}}/d+2$ and $d_{\text{clip}}/d$ are symmetric about the line $y=1$, and $J(\theta)$ will also decrease. Therefore, to maximize the objective $J(\theta)$, SPO will secretly optimize the surrogate objective $\mathfrak{O}_{\theta_{\rm old}}^{\theta}(s_{t},a_{t})$ within the trust region $d\leq d_{\max}$ as much as possible, but it is still an unconstrained first-order algorithm.

The pseudo-code of SPO is shown in Algorithm 1. When $\theta=\theta_{\rm old}$ (this means that the policy network has not been updated yet), we use the original surrogate objective to update the policy network parameters $\theta$. Once the update is performed, the parameters of the new policy network change, then we optimize the objective $(\ref{d_clip/d})$.

Considering that the hyperparameter $d_{\max}$ can be set manually and even dynamically changed at any moment during the training process, SPO can benefit significantly from the Corollary 2. In summary, the characteristics mentioned above enable SPO to potentially achieve better monotonic policy improvement than PPO, and its performance depends on the value of the hyperparameter $d_{\max}$, which corresponds to the degree of conservativeness in policy iterations.

Figure: Algorithm 1 Simple Policy Optimization (SPO)

## 4 Experiments

We report results on the Atari 2600 environment [, ]. The Atari 2600 environment is a popular benchmark in the field of reinforcement learning. It consists of a set of video game environments designed for the Atari 2600 gaming console. These environments serve as testing grounds for developing and evaluating reinforcement learning algorithms and have been widely used because of its diverse set of games, ranging from simple classics like Pong to more complex titles like SpaceInvaders and Breakout. This diversity allows researchers to assess the generalization capabilities of reinforcement learning algorithms across different tasks and complexities.

We preprocessed the Atari 2600 environment, for example, the input to the network is the stacking of the last four frames resized to $84\times 84\times 4$, reward clipping and so on [, ]. In all experiments, we use the hyperparameters provided in the Appendix A unless otherwise specified. For each environment in Atari 2600, we compute and visualize the standard deviation of the algorithm’s performance across three separate runs with different random seeds. Our experiments are divided into five parts:

- 1.
Comparison to PPOs. The first part compares the performance of several PPO variants and the SPO algorithm in the Atari 2600 environment, as well as the changes in KL divergence.
- 2.
Over optimization. In the second part, we increase the number of update epochs from 8 to 1000 in a certain training stage to over-optimize the respective objectives of PPO and SPO, and observe their optimization of surrogate objectives and the growth trend of KL divergence.
- 3.
Increasing network complexity. In the third part, we try to increase the depth of the neural network and observe how the complexity of the network structure affects the performance of PPO and SPO in the training process.
- 4.
The scalability of SPO for continuous action space. In the fourth part, we further tested the scalability of SPO in a continuous action space environment, e.g., MuJoCo [].
- 5.
Sensitivity analysis. Finally, we conducted a sensitivity analysis on the hyperparameter $d_{\max}$ of the SPO algorithm. Due to the limitation of space, please refer to the Appendix B.6 for the detailed content of this section.

Figure: Figure 2: Training curves of different PPO variants and SPO algorithm in Atari 2600 environments. Among them, PPO-C represents PPO-Clip, PPO-P represents PPO-Penalty, and PPO-E represents PPO-Clip with an early stopping strategy.
Refer to caption: x2.png

### 4.1 Comparison to PPOs

For the sake of experimental fairness, the policy network and value network structures of both the SPO and PPO algorithms are entirely identical, see shallow network structure in Figure 4. They share a common convolutional layer designed to compress high-dimensional image inputs into a low-dimensional representation of 512 dimensions. Subsequently, a fully connected layer is connected separately to generate outputs for the policy network (with the output dimension corresponding to the action space) and the value network (with a 1-dimensional output).

Partial experimental results are shown in Figure 2, with more detailed results provided in the Appendix B.4. Here, PPO-Penalty (PPO-P) and PPO-Clip with an early stopping strategy (PPO-E) are two main variants introduced to limit the excessive increase of KL divergence of PPO during the training process. We maintain consistency in the corresponding KL divergence thresholds for these two variants and the threshold $d_{\max}$ for SPO, all set to 0.02. It can be seen that SPO nearly achieves significantly better sample efficiency than other PPO variants in most environments, ranking second only to SPO-E in a few cases, while maintaining the lowest KL divergence throughout the entire training process (see Appendix B.4). In contrast, both PPO-C and PPO-P consistently exhibit significantly high KL divergence. However, PPO-E is also able to effectively limit KL divergence, trailing only slightly behind SPO.

We also notice that SPO exhibits the highest policy entropy during training (see Appendix B.4), which indicates that SPO has a better exploration of the environment. This reveals the fundamental difference between SPO and PPO-E, and we speculate that this may be a by-product of spontaneously limiting the KL divergence (not forcibly). In fact, we conducted a sensitivity analysis on the hyperparameter $d_{\max}$ in Appendix B.6, and we found that a lower value of $d_{\max}$ leads to a higher policy entropy.

Figure: Figure 3: We over-optimize the objectives of PPO and SPO for 1000 epochs in the early stages of training, where $d_{\max}$ is set to 0.01, 0.015, 0.02, 0.025 and 0.03, respectively.
Refer to caption: x3.png

### 4.2 Over Optimization

To further evaluate whether SPO can satisfy trust region constraints under extreme conditions, we select three environments and over-optimize the objectives of PPO and SPO with different $d_{\max}$ for 1000 epochs in the early stages of training, results are shown in Figure 3. Where SPO-$x$ represents the SPO algorithm with the maximum KL divergence $d_{\max}$ set to $x$, PPO-$x$ represents the PPO algorithm with the ratio clipping parameter $\epsilon$ set to $x$. We can see that even in the case of over-optimization, SPO still has a very stable KL divergence, which remains below the threshold, and the surrogate objective gradually increases as the constraints become weaker ($d_{\max}$ increases). In contrast, PPO always leads to excessively high average KL divergence and has relatively poor optimization performance for surrogate objective. Therefore, to a certain extent, this indicates that SPO optimizes the surrogate objective within the trust region as much as possible, and performs better than PPO.

Figure: Figure 4: Shallow network and deeper network structures in Atari 2600 environments.
Refer to caption: x4.png

### 4.3 Increasing Network Complexity

We are interested in understanding how the depth or complexity of the neural networks affects the performance of both PPO and SPO algorithms in the training process, as well as their KL divergence. This subsection aims to answer the following questions: When the network structure becomes more complex, will the ability of the SPO algorithm to limit KL divergence disappear? Will the phenomenon of PPO’s inability to limit KL divergence become more evident? The Two structures are shown in Figure 4.

We did not make any changes to the hyperparameters or other settings, except for increasing the number of network layers. PPO-1 and SPO-1 represent the use of shallow networks, while PPO-2 and SPO-2 represent the use of deeper networks. The results are shown in Figure 5. As the network becomes deeper, we can see that the KL divergence of PPO during the training process becomes larger than that in the shallow networks, sometimes leading to a sharp decline in its performance. In some environments, PPO almost loses its ability to learn effectively (e.g., Breakout and DemonAttack). In contrast, SPO demonstrates strong robustness to changes in network structure. Furthermore, we observed that the average KL divergence curves of SPO-1 and SPO-2 nearly overlap throughout the entire training process, which once again validates the effectiveness of SPO in limiting KL divergence. Additionally, this suggests that SPO is more likely to play a pivotal role in complex, large-scale neural network models, such as the Transformer [] or DMs (Diffusion Models) [].

Figure: Figure 5: Training curves of PPO and SPO algorithms using networks with different depths in Atari 2600 environments, where PPO-1 and SPO-1 represent the use of shallow networks, while PPO-2 and SPO-2 represent the use of deeper networks.
Refer to caption: x5.png

Figure: Figure 6: Training curves in MuJoCo with deeper networks.
Refer to caption: x6.png

### 4.4 The Scalability of SPO for Continuous Action Space

We further tested the scalability of the SPO algorithm in a continuous action space environment, e.g., MuJoCo [], and surprisingly, SPO still performed exceptionally well.

Typically, in the continuous action space, we assume that the output of the policy network follows a multivariate Gaussian distribution, and we further assume that the components are mutually independent. Therefore, the KL divergence between them is the sum of the KL divergences of the individual components.

Now we assume that two random variables follow Gaussian distributions: $p(x)=\mathcal{N}(\mu_{1},\sigma_{1}^{2})$ and $q(x)=\mathcal{N}(\mu_{2},\sigma_{2}^{2})$, then their KL divergence is

$$ $D_{\text{KL}}(p,q)=\log\frac{\sigma_{2}}{\sigma_{1}}+\frac{(\mu_{1}-\mu_{2})^{ 2}+\sigma_{1}^{2}-\sigma_{2}^{2}}{2\sigma_{2}^{2}}.$ (19) $$

The details see Appendix B.7. For each environment in MuJoCo, we compute and visualize the standard deviation of the algorithm’s performance across five separate runs with different random seeds, and we also employ two different network architectures with varying depths (see Figure 16). The results are shown in Figure 6, Table 1, Table 2 and Figure 7.

**Table 1: Average return of the last 10% training steps with shallow networks in MuJoCo.**
| Algorithm | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Humanoid-v4 | HumanoidStandup-v4 | Walker2d-v4 |
| --- | --- | --- | --- | --- | --- | --- |
| PPO | 5216.4 | 1607.5 | 1471.8 | 822.6 | 150445.3 | 3217.5 |
| SPO | 4875.4 | 2459.9 | 1402.6 | 3228.6 | 163008.1 | 3520.9 |

**Table 2: Average return of the last 10% training steps with deeper networks in MuJoCo.**
| Algorithm | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Humanoid-v4 | HumanoidStandup-v4 | Walker2d-v4 |
| --- | --- | --- | --- | --- | --- | --- |
| PPO | 527.4 | 2781.0 | 1182.6 | 576.7 | 136717.4 | 2512.4 |
| SPO | 2679.8 | 3276.0 | 1390.8 | 3231.8 | 152980.9 | 3800.0 |

Figure: Figure 7: Error bar chart (from left to right): PPO, SPO-0.01, SPO-0.02, and SPO-0.03 with deeper networks.
Refer to caption: x7.png

Figure 6 and Table 2 clearly show that SPO outperforms PPO in sample efficiency across all environments when training with deeper neural networks. Furthermore, as can be seen from Figure 7, the KL divergence of SPO remains controlled during the training process, whereas the KL divergence of PPO becomes significantly large, particularly in Ant-v4 and Humanoid-v4 (the gray line in Figure 7 corresponds to a value of 0.02). This divergence led to the collapse of PPO’s performance in these environments.

Due to the limitation of space, please refer to Appendix B.7 for more detailed experimental results. In summary, we speculate that the performance of the PPO algorithm mainly depends on two factors: “code-level optimizations” and a relatively simple network structure. This is why, in commonly used benchmarks, the policy and value networks of PPO are always designed to be relatively simple. While this avoids excessive KL divergence, it also limits the algorithm’s capacity to encode complex tasks.

## 5 Discussion

We propose Simple Policy Optimization (SPO), an unconstrained first-order algorithm for policy optimization. Experiments on the both Atari 2600 and MuJoCo suite demonstrate the utility of the approach, as SPO is better at constraining the KL divergence between successive policies, while also being robust to increases in network depth or complexity. SPO strikes a favorable balance between sample complexity, simplicity and wall-time.

In this work, we do not advocate the use of the surrogate objective with ratio clipping. We show that the ratio clipping method is not effective in limiting the KL divergence between the successive policies, this phenomenon becomes more pronounced as the network deepens. In contrast, SPO achieves higher sample efficiency with lower KL divergence and exhibits robustness to the depth or complexity of the network.

Recently, some techniques employ reinforcement learning to allow LLMs (Large Language Models) to learn human preferences [, , , ], called RLHF (Reinforcement Learning from Human Feedback). A crucial step in this process is to fine-tune the LLMs using policy optimization algorithms such as PPO. To prevent the “catastrophic forgetting” of LLMs, efforts have been focused on developing new algorithms that minimize the KL divergence with the base reference model during the fine-tuning process [, , ]. Given the high parameter complexity of LLMs and the shortcomings of PPO in constraining KL divergence, it will be interesting to explore the application of SPO in RLHF in the future.

## Appendix A Hyperparameters Setting

### A.1 Hyperparameters Setting in Atari 2600

**Table 3: Detailed hyperparameters in Atari 2600 environments.**
| Algorithm | PPO-Clip | PPO-Penalty | PPO-EarlyStop | SPO |
| --- | --- | --- | --- | --- |
| Number of actors | 8 | 8 | 8 | 8 |
| Horizon | 128 | 128 | 128 | 128 |
| Learning rate | $2.5\times 10^{-4}$ | $2.5\times 10^{-4}$ | $2.5\times 10^{-4}$ | $2.5\times 10^{-4}$ |
| Learning rate decay | Linear | Linear | Linear | Linear |
| Optimizer | Adam | Adam | Adam | Adam |
| Total steps | $1\times 10^{7}$ | $1\times 10^{7}$ | $1\times 10^{7}$ | $1\times 10^{7}$ |
| Batch size | 1024 | 1024 | 1024 | 1024 |
| Update epochs | 8 | 8 | 8 | 8 |
| Mini-batch size | 256 | 256 | 256 | 256 |
| Mini-batches | 4 | 4 | 4 | 4 |
| GAE parameter $\lambda$ | 0.95 | 0.95 | 0.95 | 0.95 |
| Discount factor $\gamma$ | 0.99 | 0.99 | 0.99 | 0.99 |
| Clipping parameter $\epsilon$ | 0.2 | $\backslash$ | 0.2 | $\backslash$ |
| Value loss coeff. $c_{1}$ | 1 | 1 | 1 | 1 |
| Entropy loss coeff. $c_{2}$ | 0.01 | 0.01 | 0.01 | 0.01 |
| Threshold of KL divergence | $\backslash$ | 0.02 | 0.02 | $\backslash$ |
| Maximum KL divergence $d_{\max}$ | $\backslash$ | $\backslash$ | $\backslash$ | 0.02 |

### A.2 Hyperparameters Setting in MuJoCo

**Table 4: Detailed hyperparameters in MuJoCo environments.**
| Algorithm | PPO | SPO |
| --- | --- | --- |
| Number of actors | 8 | 8 |
| Horizon | 256 | 256 |
| Learning rate | $3\times 10^{-4}$ | $3\times 10^{-4}$ |
| Learning rate decay | Linear | Linear |
| Optimizer | Adam | Adam |
| Total steps | $3\times 10^{6}$, $1\times 10^{7}$ | $3\times 10^{6}$, $1\times 10^{7}$ |
| Batch size | 2048 | 2048 |
| Update epochs | 10 | 10 |
| Mini-batch size | 512 | 512 |
| Mini-batches | 4 | 4 |
| GAE parameter $\lambda$ | 0.95 | 0.95 |
| Discount factor $\gamma$ | 0.99 | 0.99 |
| Clipping parameter $\epsilon$ | 0.2 | $\backslash$ |
| Value loss coeff. $c_{1}$ | 0.5 | 0.5 |
| Entropy loss coeff. $c_{2}$ | 0.0 | 0.0 |
| Maximum KL divergence $d_{\max}$ | $\backslash$ | 0.02 |

## Appendix B More Experiment Results

### B.1 Training Curves of Return

Figure: Figure 8: Training curves of return in Atari 2600 environments.
Refer to caption: x8.png

### B.2 Training Curves of Average KL Divergence

Figure: Figure 9: Training curves of average KL divergence in Atari 2600 environments.
Refer to caption: x9.png

### B.3 Training Curves of Policy Entropy

Figure: Figure 10: Training curves of policy entropy in Atari 2600 environments.
Refer to caption: x10.png

### B.4 Comparison to PPOs

Figure: (a) Training curves of average KL divergence.
Refer to caption: x11.png

### B.5 Increasing Network Complexity

Figure: (a) Training curves of return.
Refer to caption: x13.png

Figure: Figure 13: Training curves of policy entropy in Atari 2600 environments.
Refer to caption: x15.png

### B.6 Sensitivity Analysis

We conducted a sensitivity analysis on the hyperparameter $d_{\max}$ of SPO, setting it to 0.01, 0.02, and 0.03, respectively. The experimental results are shown in Figure 14 and Figure 15.

We can see that the performance of the SPO algorithm is not highly sensitive to the selection of the hyperparameter $d_{\max}$, and the average KL divergence during the training process gradually increases as $d_{\max}$ increases. Notably, as $d_{\max}$ gradually decreases, the policy entropy of the SPO algorithm gradually increases. This result seems to be expected because the KL divergence $d$ satisfies

$$ $d=\sum_{a\in\mathcal{A}}\pi_{\theta_{\rm old}}(a|s_{t})\cdot\log\frac{\pi_{ \theta_{\rm old}}(a|s_{t})}{\pi_{\theta}(a|s_{t})}=\underbrace{H\left(\pi_{ \theta_{\text{old}}}(\cdot|s_{t}),\pi_{\theta}(\cdot|s_{t})\right)}_{\text{ Cross Entropy}}-\underbrace{H\left(\pi_{\theta_{\text{old}}}(\cdot|s_{t}) \right)}_{\text{Policy Entropy}}.$ (20) $$

The cross entropy is non-negative, and it will be a good choice for the agent to reduce the KL divergence by maintaining a higher policy entropy in the training process.

We believe that there is a fundamental difference between SPO and PPO-E, the latter of which introduces an early stopping strategy to forcibly limit the KL divergence, as SPO always exhibits a higher policy entropy than PPO-E when they have a similar level of KL divergence.

Figure: (a) Error bar charts of return of SPO in Atari 2600 environments during the last 10% training steps, with their $d_{\max}$ set to 0.01, 0.02, and 0.03, respectively.
Refer to caption: x16.png

From Figure 14 we can see that the performance of the SPO algorithm is not highly sensitive to the selection of the hyperparameter $d_{\max}$.

Figure: (a) Training curves of average KL divergence.
Refer to caption: x18.png

From Figure 15 we can see that the average KL divergence during the training process gradually increases as $d_{\max}$ increases. Notably, as $d_{\max}$ gradually decreases, the policy entropy of the SPO algorithm gradually increases. However, we surprisingly found that the change trend of policy entropy in MuJoCo (see Appendix B.7) was opposite to that in Atari 2600 with the change of $d_{\max}$.

### B.7 The Scalability of SPO for Continuous Action Space

We first give the proof of $(\ref{kld in mujoco})$:

###### Proof.

The KL divergence between two given Gaussian distributions $p(x)$ and $q(x)$ is defined as

$$ $\begin{split}&D_{\text{KL}}(p,q)\\ =&\int p(x)\cdot\log\frac{p(x)}{q(x)}\text{d}x\\ =&\int p(x)\cdot\log\left[\frac{\sigma_{2}}{\sigma_{1}}\cdot\exp\left(-\frac{( x-\mu_{1})^{2}}{2\sigma_{1}^{2}}+\frac{(x-\mu_{2})^{2}}{2\sigma_{2}^{2}}\right )\right]\text{d}x\\ =&\int p(x)\cdot\log\frac{\sigma_{2}}{\sigma_{1}}\text{d}x+\int p(x)\cdot\left (-\frac{(x-\mu_{1})^{2}}{2\sigma_{1}^{2}}+\frac{(x-\mu_{2})^{2}}{2\sigma_{2}^{ 2}}\right)\text{d}x\\ =&\log\frac{\sigma_{2}}{\sigma_{1}}+\frac{\sigma_{1}^{2}-\sigma_{2}^{2}}{2 \sigma_{1}^{2}\sigma_{2}^{2}}\cdot\underbrace{\int x^{2}p(x)\text{d}x}_{M}+ \frac{\mu_{1}\sigma_{2}^{2}-\mu_{2}\sigma_{1}^{2}}{\sigma_{1}^{2}\sigma_{2}^{2 }}\cdot\underbrace{\int xp(x)\text{d}x}_{N}+\frac{\mu_{2}^{2}\sigma_{1}^{2}- \mu_{1}^{2}\sigma_{2}^{2}}{2\sigma_{1}^{2}\sigma_{2}^{2}}\cdot\underbrace{\int p (x)\text{d}x}_{1},\\ \end{split}$ (21) $$

where

$$ $M=\mathbb{E}_{x\sim p(x)}(x^{2})=\mathbb{D}_{x\sim p(x)}(x)+\mathbb{E}_{x\sim p (x)}(x)^{2}=\sigma_{1}^{2}+\mu_{1}^{2},\enspace N=\mathbb{E}_{x\sim p(x)}(x)= \mu_{1},$ (22) $$

so we have

$$ $\begin{split}D_{\text{KL}}(p,q)&=\log\frac{\sigma_{2}}{\sigma_{1}}+\frac{ \sigma_{1}^{2}-\sigma_{2}^{2}}{2\sigma_{1}^{2}\sigma_{2}^{2}}\cdot(\sigma_{1}^ {2}+\mu_{1}^{2})+\frac{\mu_{1}\sigma_{2}^{2}-\mu_{2}\sigma_{1}^{2}}{\sigma_{1} ^{2}\sigma_{2}^{2}}\cdot\mu_{1}+\frac{\mu_{2}^{2}\sigma_{1}^{2}-\mu_{1}^{2} \sigma_{2}^{2}}{2\sigma_{1}^{2}\sigma_{2}^{2}}\\ &=\log\frac{\sigma_{2}}{\sigma_{1}}+\frac{(\mu_{1}-\mu_{2})^{2}+\sigma_{1}^{2} -\sigma_{2}^{2}}{2\sigma_{2}^{2}}.\end{split}$ (23) $$

∎

The shallow network and deeper network structures used in MuJoCo are shown in Figure 16.

Figure: Figure 16: Shallow network (left) and deeper network (right) structures in MuJoCo environments.
Refer to caption: x20.png

**Table 5: Average return of the last 10% training steps with shallow networks in MuJoCo.**
| Algorithm | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Humanoid-v4 | HumanoidStandup-v4 | Walker2d-v4 |
| --- | --- | --- | --- | --- | --- | --- |
| SPO-0.01 | 5066.1 | 2328.6 | 1377.1 | 3772.8 | 158407.4 | 3508.6 |
| SPO-0.02 | 4875.4 | 2459.9 | 1402.6 | 3228.6 | 163008.1 | 3520.9 |
| SPO-0.03 | 4628.0 | 2619.4 | 1632.6 | 3168.3 | 159841.8 | 3286.5 |
| PPO | 5216.4 | 1607.5 | 1471.8 | 822.6 | 150445.3 | 3217.5 |

**Table 6: Average return of the last 10% training steps with deeper networks in MuJoCo.**
| Algorithm | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Humanoid-v4 | HumanoidStandup-v4 | Walker2d-v4 |
| --- | --- | --- | --- | --- | --- | --- |
| SPO-0.01 | 3376.3 | 2336.2 | 1532.3 | 3499.8 | 151760.9 | 3016.9 |
| SPO-0.02 | 2679.8 | 3276.0 | 1390.8 | 3231.8 | 152980.9 | 3800.0 |
| SPO-0.03 | 1504.2 | 2855.7 | 1379.8 | 3285.8 | 147222.7 | 3260.6 |
| PPO | 527.4 | 2781.0 | 1182.6 | 576.7 | 136717.4 | 2512.4 |

Figure: (a) Error bar chart of return with shallow networks.
Refer to caption: x21.png

Figure: (a) Ant-v4.
Refer to caption: x23.png

We recorded and plotted the mean and standard deviation of the returns in the last 10% steps of the training process when SPO and PPO algorithms were respectively applied to shallow networks and deeper networks. The results are shown in Table 5, Table 6, Figure 17 and Figure 18.

Figure: (a) Training curves of return.
Refer to caption: x29.png

The training curves for PPO and SPO algorithms in MuJoCo environments with shallow networks are shown in Figure 19.

Figure: (a) Training curves of return.
Refer to caption: x32.png

The training curves for PPO and SPO algorithms in MuJoCo environments with deeper networks are shown in Figure 20.

###### Proof.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: --What is the significance of proposition 3.2 and theorem 3.3?
Could not we use the theorem (3) to conclude that Total Variation divergence is the better choice compare to KL divergence? 


--As the authors rightly mentioned, larger ResNets or deeper MLPs could help better representation learning and, therefore, better performance, as shown in Figures 5 and 6. I have personally observed this to be true in my own experiments with PPO. Could the authors provide a more compelling reason for why PPO in their experiments does not benefit from better representation learning, beyond the claim that PPO's performance deteriorates with an increasing number of parameters? Could tuning hyperparameters help PPO in this case?

--In figure 7, is it possible to match the curve for SPO(epsilon=0.2) if we choose different epsilon for PPO?
soundness: 3
strengths: Paper is well written, problem statement is clear and is easy to follow. Motivations are supported with references and theoretical justifications as well as toy examples.
summary: --Why: 
It has been shown that consistent policy improvement can be achieved through constrained policy optimization or trust region policy optimization. 

--What: 
However, in practice, current methods use approximations to enforce the trust region. In particular, PPO uses clipping of the probability ratio to enforce the trust region constraint. It's simple and works well in practice. However, the author argues and shows that theoretically, this cannot always satisfy the bound. When the probability ratio is out of bounds, this method zeros the gradient, and therefore there isn't any corrective gradient to push the policy back towards the trust region. Thus, this paper investigates better trust region enforcement for PPO.

--How:
Authors propose an objective (eq 15), which directly constrains the total variation divergence instead of KL divergence. They show theoretically that this is a better divergence (see Proposition 3.2 and 3proposition 3.2 .3). They also argue that, in practice, their objective can apply a corrective gradient when the probability ratio goes over the bound (see Figure 2 and Figure 4).
weaknesses: --Originality and Significance
 
the idea of using  total variation divergence is not new and it's been proposed in original TRPO paper already as authors has also mentioned. Also some follow up work on constraint optimization and safe RL has explored similar ideas that worth being mentioned, for example see:

https://arxiv.org/pdf/1705.10528
,https://arxiv.org/pdf/1805.07708

Given that the idea of using total variation divergence is not entirely new, this paper requires strong experimental evidence to support its claims. The authors' decision to use hyperparameters from the original PPO paper seems unfair, considering that changes in the setup necessitate retuning. However, even with this caveat, Figures 9 and the continuous control results show no significant improvement over the original PPO. I believe tuning PPO could achieve better results. Furthermore, the results in Figure 6 are puzzling. A larger ResNet benefits SPO but not PPO. Observing the constant bias in Figure 7, it appears that choosing a different epsilon for PPO might lead to better results. This suggests that the performance difference could be attributed to suboptimal hyperparameter selection for PPO.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: **[Q1] Can you please explain why each assumption in Definition 4.1 is important?** When comparing the objective in (14) to PPO, it seems like the existence of a unique optimal probability ratio in SPO is actually the important feature compared to multiple optimal solutions in PPO.

**[Q2] Could you please explain the purpose of Section 3 and the optimization problem in (15)?** It does not look like (15) is actually being used by SPO. Instead, the main algorithmic contribution is the objective in (14), which is motivated by considering the optimization problem in (17) at every state-action pair. Applying constraints at every state-action pair is more restrictive than the constraint in (15), and also does not account for the dependence across state-action pairs that occurs because $\pi$ must be a probability distribution.

**[Q3] Experimental suggestions:** I understand that these suggestions may not be possible in a limited rebuttal window.
- How does SPO compare to well-tuned versions of PPO across each setting?
- How does SPO compare to PPO using the adaptive learning rate in [3], which is a simple implementation detail used in the literature to prevent probability ratios from becoming very large?
- How does SPO compare to non-parametric policy update methods such as [6, 7]? Similar to SPO, these methods apply updates based on target probability ratios.
soundness: 2
strengths: - **[S1] Simple implementation:** The implementation of SPO requires minimal changes to PPO and considers a simple unconstrained optimization problem.
- **[S2] Robust performance across architectures:** Experiments show that SPO generalizes well across different implementation choices such as network size, compared to PPO which is much more sensitive to these choices. Given the widespread use of PPO, improvements that provide robustness to hyperparameters / architectures are helpful for applying RL training across different applications.
summary: The paper introduces an on-policy RL algorithm called Simple Policy Optimization (SPO). SPO is proposed as an improvement to PPO that can more effectively constrain the probability ratio between the next and current policy. The objective of SPO is designed to update the probability ratio to the boundary of $[1-\epsilon, 1+\epsilon]$ in the direction that improves the surrogate objective. Experiments on Atari and MuJoCo benchmarks in Gymnasium demonstrate that SPO controls the probability ratio and achieves performance that generalizes across different network architectures, compared to PPO which is more sensitive to implementation choices.
weaknesses: **[W1] Limited novelty compared to prior works, with several missing citations to relevant prior works**
- The results presented in the paper have limited theoretical novelty. Section 3 uses results that have already been shown in [1] to rewrite the TV distance, and applies Pinsker’s inequality in ways that have been used previously in [2]. The optimization problem in (15) is also not novel, and can be motivated from the policy improvement bounds proposed in [2] that include an expected TV distance penalty term (instead of a max over states). Section 4 formulates an objective that satisfies the set of assumptions put forth by the authors in Definition 4.1.
- Several methods have been proposed to better constrain the probability ratio in PPO, and SPO represents another implementation of this same idea. [1, 3] apply adaptive learning rates based on TV or KL divergence, [4, 5] propose modifications to the clipping function / rollback mechanisms, and [6, 7] consider non-parametric updates that determine target probability ratios. The SPO objective in (14) is designed to move each data point towards the target probability ratio in Definition 4.1, which is similar to what is accomplished by many of these methods.
- Most of the relevant works mentioned in the 2 previous bullet points are never referenced in the paper ([4] is the only work that is cited in the paper / compared against in the experiments). In general, the paper does not adequately frame its contribution with respect to prior work (there is no Related Work section in the paper).

**[W2] Experiments: contributions are overstated and suboptimal PPO implementation is used for comparison**
- At several points throughout the paper, the authors claim that SPO achieves state-of-the-art performance. However, the results shown in the paper are not state-of-the-art. Several off-policy algorithms have been shown to perform much better than SPO (and on-policy algorithms in general) on these benchmarks, and in some cases it is possible to find other implementations of PPO that achieve better performance than what has been reported in this paper (see the results contained in the references below).
- Experiments compare against PPO for a single set of hyperparameters / implementation choices that may be suboptimal based on other results reported in the literature. While I understand that this shows PPO can be more sensitive to implementation details than SPO, it would be much more convincing to show that SPO can perform similar to a well-tuned version of PPO across different tasks / architectures.

**Minor technical issues:**
- I believe that (8) is not correct due to the min that appears in the PPO objective in (6). The cases must also consider the sign of the advantage function (e.g., see (7) in [4]).
- The connection between TV distance and the probability ratio in (9) / Appendix B requires the assumption that the support of $\tilde{\pi}$ is contained in the support of $\pi$. This technical assumption is not mentioned anywhere in the paper.

---

**References:**

[1] Queeney et al. Generalized Proximal Policy Optimization with Sample Reuse. In NeurIPS 2021.

[2] Achiam et al. Constrained Policy Optimization. In ICML 2017.

[3] Rudin et al. Learning to Walk in Minutes Using Massively Parallel Deep RL. In CoRL 2022.

[4] Wang et al. Truly Proximal Policy Optimization. In UAI 2020.

[5] Cheng et al. Authentic Boundary Proximal Policy Optimization. IEEE Transactions on Cybernetics, 2021.

[6] Vuong et al. Supervised Policy Update for Deep RL. In ICLR 2019.

[7] Song et al. V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control. In ICLR 2020.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: **[Q1]**
In Table 2, SPO does not outperform PPO with a 3-layer network. Could you clarify this? In tasks where SPO underperforms (e.g., Ant, HalfCheetah, Walker2d), the ratio deviation appears to be constrained overly tightly.

**[Q2]**
To me, the transition from Equation (10) to Equation (14) feels somewhat abrupt. Could you elaborate further on this derivation? I understand the motivation behind Equation (14), presumably to demonstrate the bounded nature of the probability ratio as indicated by Theorem 4.3. However, if you could clarify how the $C$ and the max operator in Equation (10) relate to Equation (14), it would provide stronger theoretical support.

**[Q3]**
Is SPO also more applicable to large-scale scenarios and does it still show superiority in terms of ratio deviation? I’m curious about its performance difference from PPO in cases with extensive parallelism, such as environments with many actors like Isaac Gym.

**[Q4]**
When using a 3-layer network, what is the performance difference between SPO-0.3 and SPO-0.2? If SPO-0.2 constrains the ratio deviation overly tightly, causing it to underperform compared to PPO, SPO-0.3 might perform better due to the slightly looser upper bound on ratio deviation. If this is the case, it would provide stronger theoretical support for the approach.
soundness: 3
strengths: * This paper is well-written and clearly structured.
* The theoretical foundation is straightforward (in a positive way), and the implementation is simple, which could enhance reproducibility.
* SPO demonstrates superior performance over PPO, especially when using deeper networks (e.g., 7-layer MLP and ResNet).
summary: This paper introduces an actor-critic algorithm, Simple Policy Optimization (SPO), which achieves a more tight bound on the policy _probability ratio_ by modifying the loss function of Proximal Policy Optimization (PPO) algorithm. The authors demonstrate both theoretically and empirically that SPO imposes a stricter upper bound on the _ratio deviation._ Notably, SPO outperforms PPO across various tasks, especially when applied to policies with deeper network architectures. Although several papers have already pointed out issues with the heuristic clipping technique in PPO and proposed alternatives, I find the authors' approach offers a novel perspective, leading me to lean towards acceptance.
weaknesses: * SPO tends to make the _ratio deviation_ overly small in cases with fewer layers (see Table 2). Could the authors address whether the tighter ratio deviation in SPO may be limiting performance in some simpler tasks, and if so, how this trade-off might be managed?

* More importantly, without a comparison to existing methods like TR-PPO (as mentioned by the authors / Line 90), it is hard to assess how effective SPO is relative to other approaches that aim to address the same clipping issue in PPO. Could the authors include a comparison of SPO with TR-PPO and other recent methods addressing PPO's clipping issues, particularly focusing on performance and ratio deviation metrics across different network depths?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: **Questions for the Authors**  
1. Could you provide confidence intervals and other statistical measures for the experimental results to help clarify performance variance and statistical significance?
2. Were any efforts made to tune the baseline hyperparameters specifically? If so, could you detail the tuning process? Additionally, how much effort was made tuning SPO? 

Ultimately, I liked the paper, think it is a simple yet efficient algorithm and I believe it would be useful and warrants acceptance **if just slightly more validated results are presented.** This should be as simple as presenting the mean, median and interquartile mean of the results as well as using stratified bootstrapping to produce confidence intervals i.e. no further training runs should be done. Due to this, I am giving it a 5 as I do not believe the results can be presented as it currently stands. I look forward to raising the score.
soundness: 4
strengths: 1. **Clarity and Theory**: The paper is clearly written and provides strong theoretical grounding. New theorems and proofs substantiate the methodology, ensuring that the approach is technically sound.
2. **Experimental Rigor**: Extensive experiments on various benchmarks, including a range of baselines, demonstrate SPO's performance relative to other algorithms. The inclusion of pseudo-code and naming the used open-source repositories contributes to the reproducibility of the work. Additionally, hyperparameters are provided.
3. **Algorithm Simplicity and Impact**: SPO is a straightforward modification of PPO by changing the surrogate objective slightly, achieving meaningful performance improvements, particularly in terms of ratio constraint handling. The simplicity of SPO, combined with its impact, makes it valuable for the community.
4. **Comparative Baselines**: The paper employs a wide array of baselines (for the Mujoco tasks at least), giving context to the results and situating SPO’s benefits accurately. Understandably, Atari is very computationally expensive so it is hard to baseline everything as done in Mujoco so I am not too fazed although I would have liked 1 additional algorithm.
summary: The paper introduces a new RL algorithm, Simple Policy Optimisation (SPO), designed to address the challenges present in existing policy gradient methods, particularly PPO and TRPO. The primary goal is to develop a surrogate objective that constrains the policy ratio more effectively than PPO without the computational burden of TRPO's second-order optimisations. SPO achieves this by using TV divergence, combining TRPO's theoretical robustness with PPO's simplicity, and is demonstrated to improve performance across Atari and MuJoCo benchmarks.
weaknesses: 1. **Lack of Confidence Intervals**: The experimental results lack confidence intervals, which limits the ability to assess the performance variance across runs.
2. **Absence of Statistical Tests**: No statistical testing (such as that from the [rliable](https://github.com/google-research/rliable) package) was performed, making it difficult to confirm the statistical significance of the improvements over PPO and other methods. This draws concern as to whether the improvements are statistically significant.
3. **Potential Baseline Tuning Issues**: PPO and similar RL algorithms are known to be highly sensitive to hyperparameters like batch size, mini-batch size, and the number of rollout steps. The paper does not discuss any dedicated hyperparameter tuning for the baselines, raising concerns about potential underperformance. It was noted that hyperparameters were used from the original papers but this is not always a fair approach especially considering certain types of parameters are not always listed such as number of rollout workers.
4. **Code-level Optimisations**: As noted in the paper, code level optimisations can significantly impact the performance of certain algorithms, but the paper does not specify what implementation tricks were used in the baselines and SPO itself.