Simple Policy Optimization
Authors: Zhengpeng Xie
Qiang Zhang
Fan Yang
Marco Hutter
Renjing Xu
Venue: ICML 2025 poster (2025)
Keywords: Model-Free Reinforcement Learning
Policy Optimization
Primary area: reinforcement_learning->online
TL;DR: We introduce a novel model-free reinforcement learning algorithm.

Abstract: Model-free reinforcement learning algorithms have seen remarkable progress, but key challenges remain. Trust Region Policy Optimization (TRPO) is known for ensuring monotonic policy improvement through conservative updates within a trust region, backed by strong theoretical guarantees. However, its reliance on complex second-order optimization limits its practical efficiency. Proximal Policy Optimization (PPO) addresses this by simplifying TRPO's approach using ratio clipping, improving efficiency but sacrificing some theoretical robustness. This raises a natural question: Can we combine the strengths of both methods? In this paper, we introduce Simple Policy Optimization (SPO), a novel unconstrained first-order algorithm. By slightly modifying the policy loss used in PPO, SPO can achieve the best of both worlds. Our new objective improves upon ratio clipping, offering stronger theoretical properties and better constraining the probability ratio within the trust region. Empirical results demonstrate that SPO outperforms PPO with a simple implementation, particularly for training large, complex network architectures end-to-end.

Simple Policy Optimization
Zhengpeng Xie
Qiang Zhang
Fan Yang" "3 Marco Hutter
Renjing Xu'
Abstract
PPO [ResNe: 50)
Model-free reinforcement learning algorithms have seen remarkable progress, but key chal- lenges remain. Trust Region Policy Optimization (TRPO) is known for ensuring monotonic policy improvement through conservative updates within a trust region, backed by strong theoretical guar- antees. However, its reliance on complex second- order optimization limits its practical efficiency. Proximal Policy Optimization (PPO) addresses this by simplifying TRPO's approach using ra- tio clipping, improving efficiency but sacrificing some theoretical robustness. This raises a natu- ral question: Can we combine the strengths of both methods? In this paper, we introduce Simple Policy Optimization (SPO), a novel unconstrained first-order algorithm. By slightly modifying the policy loss used in PPO, SPO can achieve the best of both worlds. Our new objective improves upon ratio clipping, offering stronger theoretical properties and better constraining the probability ratio within the trust region. Empirical results demonstrate that SPO outperforms PPO with a simple implementation, particularly for training large, complex network architectures end-to-end.
500
PPO (Reshius
SPG (Reehwe-50) 50]
SPO (Reshet
Average retum
400
300
200
100
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
1e7
Figure I. Training performance in the Breakout environment. SPO is novel model-free algorithm capable of end-to and-to-end training for extremely deep neural network architectures, positioning itself as promising alternative the well-known PPO algorithm.
been widely adopted by the academic community. One main practical challenge of PG methods is to reduce the variance of the gradients while keeping the bias low (Sut- ton et al., 2000; Schulman et al., 2015b). In this context, a widely used technique is to add bascline when sampling an estimate of the action-valus function (Greensmith et al., 2004). Another challenge of PG methods is to estimate the proper step size for the policy update (Kakade & Langford, 2002; Schulman et al., 2015a). Given that the training data strongly depends on the current policy, a large step size may result in a collapse of policy performance, whereas a small one may impair the sample efficiency of the algorithm.
Code is available at Simple-Policy--ppiittaiiio
1. Introduction
To address these challenges, Schulman al. (2015a) proved that optimizing certain surrogate objective guarantees pol- icy improvement with non-trivial step sizes. Subsequently, the TRPO algorithm was derived through series of approx- imations, which impose a trust region constraint during the policy iterations, leading to monotonic policy improvement in theory. However, given the complexity of second-order optimization, TRPO is highly inefficient and can be hard to extend to large-scale RL environments. Proximal Policy Optimization (PPO) (Schulman al., 2017) is designed to enforce comparable constraints on the difference between successive policies during the training process, while only using first-order optimization. By clipping the current data that exceeds the probability ratio limit to a constant, PPO attempts to remove the high incentive for pushing the cur-
Deep Reinforcement Learning (DRL) has achieved great success in recent years, notably in games (Mnih et al., 2015; Silver et al., 2016; 2017; 2018; Vinyals al., 2019), foun- dation model fine-tuning (Ouyang al., 2022; Black al., 2023), and robotic control (Makoviychuk et al., 2021; Rudin et al., 2022). Policy gradient (PG) methods (Sutton & Barto, 2018; Lehmann, 2024), as a major paradigm in RI, have
'Equal contribution 'The Hong Kong University of Science and Technology (Guangzhou) BBeijing Innovation Center of Hu- manoid Robotics 'ETH Zurich. Correspondence to: Zhengpeng Xie <zhengpengxie @ wakkus--zz.dd....
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Simple Policy Optimization
Data points of MPD
Data points of SPO
Eppo
E (ee.tr)rtt
min (n:(0) A(se. ai), clip (,,6), Ate, as)
Probosoiity relo
Probabilly nti
Advantage
Advanbba
1
[A(s,a))] 1n:(0) 11a
CaPO
E (oe
r((0) ((()))(s))))))))))
23
06
Epoch
Epoch
Figure 2. (Left) The only difference between SPO and PPO is the policy loss, where (0) no (atlss))// /ea (se) and is the probability ratio hyperparameter, making it simple and straightforward to implement SPO based on high-quality PPO implementations. (Right) The optimization behavior of PPO and SPO is visualized, where each scatter point represents the probability ratio of a single data point for a specific training epoch, with its color corresponding to its advantage, and the red line representing the probability ratio bound.
rent policy away from the old one. It has been demonstrated that PPO can be effectively extended to large-scale complex control tasks (Ye et al., 2020; Makoviychuk et al., 2021).
have explored how to enforce trust region constraints ef- ficiently, which are essential for ensuring robust policy improvement. For instance, the widely-used PPO algo- rithm (Schulman et al., 2017) was the first introduce the heuristic clipping technique, effectively avoiding the computationally expensive second-order optimization. This heuristic clipping technique has been widely used in various reinforcement learning algorithms (Queeney et al., 2021; Zhuang et al., 2023; Gan et al., 2024).
Despite its success, the optimization behavior of PPO re- mains insufficiently understood. Although PPO aims to con- strain the probability ratio deviations between successive policies, it often fails to keep these ratios within bounds (Ilyas et al., 2018; Engstrom et al., 2020; Wang et al., 2020). In some tasks, the ratios can even escalate to values as high as 40 (Wang al., 2020). Furthermore, studies have revealed that PPO's performance is highly dependent on "code-level optimizations" (Andrychowicz et al., 2021; Huang et al., 2022a). The implementation of PPO includes numerous code-level details that critically influence its ef- fectiveness (Engstrom et al., 2020; Huang et al., 2022b).
However, empirical evidence from a wide range of studies demonstrates that ratio clipping fails enforce trust region constraints effectively (Wang et al., 2020). To prevent ag- gressive policy updates, previous works have focused on designing adaptive learning rates based on TV divergence or KI. divergence (Heess et al., 2017: Queeney et al., 2021; Rudin et al., 2022), which have been shown to effectively enhance the stability of PPO. On the other hand, code-level optimizations are crucial for the robust performance of PPO (Engstrom et al., 2020). High-quality implementations of PPO involve numerous code details (Huang et al., 2022a;b), making it challenging to accurately assess the core factors that truly affect the algorithm's 's performance.
In this paper, we propose a new model-free RL algorithm named Simple Policy Optimization (SPO) designed more effectively bound probability ratios through a novel objec- tive function. The key differences in optimization behavior between PPO and SPO are illustrated in Figure 2. Our main contributions are summarized as follows:
We theoretically prove that optimizing a tighter per- formance lower bound using Total Variation (TV) di- vergence constrained space results in more consistent policy improvement.
In this work, we argue that heuristic clipping technique can- not enforce trust region constraints (see Figure 2). During PPO's iterations, ratio clipping zeros the gradients of certain data points, which can lead lack of corrective gradients to prevent the policy from escaping the trust region, thus undermining the monotonic improvement guarantee. As a result, PPO requires additional code-level tuning, such as adaptive learning rates or early stopping strategies, to artifi- cially prevent performance collapse. We reveal this inherent flaw of ratio clipping and propose promising alternatives.
To overcome PPO's limitation in constraining probabil- ity ratios, propose new objective function, leading to the development of the proposed SPO algorithm.
Experiments benchmark various policy gradient al- gorithms across different environments, showing that SPO can achieve competitive performance with sim- ple implementation, improved sample efficiency, and easier training of deeper policy networks.
3. Background
3.1. Reinforcement Learning
2. Related Work
Online reinforcement learning is a mathematical frame- work for sequential decision-making which is generally defined by the Markov Decision Process (MDP) M (S, A, r, P, Po: 7), where S and A represent the state space
Since TRPO (Schulman et al., 2015a) theoretically demon- strated monotonic policy improvement, numerous studies
2

---

Simple Policy Optimization
and action space, r S x A + R is the reward function, P S x A x S - [0: 1] is the probability distribution of the state transition function, Po S + [0, 1] is the initial state distribution, while E (0, 1) is the discount factor.
where & = max, (Ewaa(l (s, a)]}, Dry is the Total Variation (TV) divergence.
Using importance sampling on action ~ (+1ss) and ac- cording to the Pinsker's s inequality, we have
Suppose that an agent interacts with the environment fol- lowing policy n, i.e., Of- ~ T-16s) and obtains a trajectory
(alss) x(ass)
== (Su, ag. To. ............ St, at, .t....), where rt- r(s;,a)) The goal of RL
nT* (TT)= 2 1 - E Banpe (1100((11)))
As
learn a policy that maximizes the objec- tive TT( = E 1=== r're,, where the notation E represents the expected return of the trajectory T generated by the agent following policy IT, i.e., so ~ po(-), a ~ (-1s,), = r(s,, Qz), $1+1 ~ P(-|88, ax). The action- value function and value function are defined as
-
18 Bampe())
2 DKL ( (T]())))))
(1 - 7))
At this point, the subscripts of the expectation in (2) are replaced from 8 ~ p=(+)) and a ~ #(-[s) to 8 ~ px(() and a ~ I (-Is), which means that we can now reuse the current data. In TRPO, the lower bound in (4) is indirectly
(5r az) En, Es
*r(s acts) (1)
(i(( = Fa (111)))) (x+(
optimized by solving the following optimization problem:
Given 2, and V=, the advantage function can be expressed as Ar (8,, ax)) = Qz (8%, ax)) - (*t).
(T(a)(()) A(ssn (5)
max Essr 0
TToaa (ae|sr)
3.2. Trust Region Policy Optimization
s.t. E [DKL [[T]] S d.
Classic policy gradient methods cannot reuse data and are highly sensitive to the hyperparameters. To address these issues, in Trust Region Policy Optimization (TRPO), Schul- man et al. (2015a) derived lower bound for policy improve- ment. Before that, Kakade & Langford (2002) first proved the following policy performance difference theorem.
This problem includes constraint where ō is a hyperpa- rameter that limits the KL divergence between successive policies, with Á(s ax) being the estimate of the advantage function, and the objective is called "surrogate objective".
Theorem 3.1. (Kakade & Langford, 2002) Let P(s: = s|mt represents the probability of the t-th state equals to s in trajectories generated by the agent following policy n, and p (s) (1-7) Ei= "PP(s = s|x) represents the normalized discounted visitation distribution. Given any two policies, and ã. their performance difference can be measured by
3.3. Proximal Policy Optimization
Due to the necessity of solving constrained optimization problem (5) in each update, TRPO is highly inefficient and can be challenging to apply to large-scale reinforcement learning tasks
Schulman et al. (2017) proposed a new objective called "clipped surrogate objective", in which the algorithm is named Proximal Policy Optimization (PPO). PPO retains similar constraints of TRPO but is much easier to implement and involves only first-order optimization.
n(+)) - .7())
E Bampe ()aaa(()))) (s,
1 1-7
Σ (=(8) Σ A(dls)- as,
The "'lipped surrogate objective",, also called PPO-Clip, adopts a ratio clipping function. Denote A. = Á(s,, at), the objective of PPO-Clip can be expressed as
where nTT) Er D E rree.
The key insight is that the new policy म will improve (or at least remain constant) as long as it has a nonnegative expected advantage every state s. Then, the following performance improvement lower bound is given: Theorem 3.2. (Achiam et al., 2017) Given any two policies, प and म, the following bound holds:
Jelip = ) {min [ra(0)- (,) A. }.]]. (6)
where
To(ae]se)
(0)
(0( clip (r.(0), 1 c, + c), To T (ae(se))
1
((*) 7(())= 2 - - F EEppp (-11)) (s,
(7)
(3) with Taasa and Ty being the old policy and the current policy. The gradient of PPO-Clip. given the training data (st, a.),
2
E (a [[]v(]||)|s]],
- 71=
3

---

Simple Policy Optimization
can be expressed
follows:
Vori(0)- As, Vor -A, A < (,)) 2 1
A, > 0, 1
ΩT = { Drv (TT ( S δTV. Vs E S), OKL = { DKL ( [+]]{{{]]] S ŌKL, Vs E ES}, where ÓKL. > a predefined threshold. Let drv Z V Vrk, we establish that Skg C Crrv.
(1I)
(eJelip(0)
0,
otherwise.
(8)
In other words, PPO-Clip aims to remove the high incentive for pushing the current policy away from the old one. PPO-
Clip has gained wide adoption in the academic community due to its simplicity and performance.
Proof. For any given ôKL and āt E OKL using Pinsker's inequality, we have DrV(T v DT [T]|s]]ss <
V DOL S STv, therefore स 9KL = Gry, which
4. Methodology
means QKt. C Grr, concluding the proof.
PPO attempts to limit the differences between successive policies through ratio clipping. However, Wang et al. (2020) proved the following theorem:
Additionally, the optimal solution to the lower bound in the TV divergence solution space, Ωrv. is expected to be superior. We now present the following theorem:
Theorem 4.1, (Wang et al., 2020) For discrete action space tasks where [A] 2 3 or continuous action spuce lasks where the output of the policy ITe follows multivariate Gaussian distribution. Let θ {0|1 S ro(0) < 1 + e), we have supage DKL (Ta.aa [T) = +oo for both discrete and continuous action space tasks.
Theorem 4.3. Given the old policy n, and Ωry, QkL pre- sented in Proposition 4.2, let
1
A(als) ([aas)
E Exup ((a0((())))
A (s, a)
7
2
Theorem 4.1 demonstrates that DKL. (To.s [s,] is not necessarily bounded even if the probability ratio ro( is bounded. However, this theorem considers only extreme case involving single data point, which is less typical than the batch processing used in training data. On a broader scale, the heuristic clipping technique employed by PPO aims to bound the TV divergence for sufficient batch sizes (Queeney et al., 2021). This relationship formalized as
(1
7)²
E Esmpe()
[DVV [|]]]||||]
(aass)
(KT((T)
E E (oNpe())anT((ll
A (3.a)
7
(aass)
257
(1 7)2 E Eempe()
DKI. (T]
(12)
be the lower bounds of performance improvement with TV divergence and KL divergence. Let drv 2 v OOL denote
1
A(als)
Ex-p, () (TV
2
T(ass)
(9)
कTV = arg max LTV (+), #LL = arg max KKL (#), #ElK.
Then, the performance improvement lower bound (3) can be rewritten as
RElTr
(13)
then CTV (#TT)) 2 CKL (TKL.)
((T) ()x)== 2 1
i(a s) (T(as)
Exnp, pa ())))
Ag (s)
7
Proof. Since QKI. C ΩTV, we have CTV (#iv)
EY
B samp ((ww)())))))
i(a 8)
2 2 TV (#kL)=
(1 - 7)2
T(a
(10)
1
7 E
Ea-pan (1awT(1s)))
(als)
(s.
T(als)
This explains why PPO attempts to limit the probability ratio | ((])))/<(a{s) - 1/ S 6, as this enforces a TV divergence trust region in expectation.
2 E Esmpi O [rrv (
(1 - 7)2
a (als) (als)
(14)
Finally, we also found that PPO, which aims to bound the TV divergence, can offer larger solution space compared to methods that incorporate a looser KL divergence as constraint (e.g., in TRPO). To illustrate this, we present the following proposition:
Exwpe (.)pp())aNT((11))
Am a)
2 E Benpe()
DKL (
(1 )-
CKI (f
Proposition 4.2. Given the old policy n, define the solu- tion spaces under the TV and KL. divergence constraints as
thus CTV (H 2
CKL ( concluding the proof.

---

Simple Policy Optimization
Algorithm Simple Policy Optimization (SPO)
1: Initialize: Policy and value networks TT, Ve, hyperparameter e, value loss and policy entropy coefficients c1 C2
2: Output: Optimal policy network n
3: while not converged do
4: # Data collection
5: Collect data D {( au, (t, J= using the current policy network
6: # The networks before updating
7: Toaa *T0, Vasia <- V
8: # Estimate the advantage A(st, 0e) based on Voas
9: Use GAE (Schulman al., 2015b) technique estimate the advantage A(s ac)
10: # Estimate the return R²
11:
Re < Valsa (st) A(st, at)
12:
for each training epoch do
13:
# Compute policy loss Cp (This is the only difference between SPO and PPO)
Lp + + Feaaa (oclss) A(sE, 0z) [A(s..a))]
mo(arles)
Reata (0:(8.) - 2
ma(aelse)
14:
26
15:
# Compute policy entropy Le and value loss Lu
16:
Le E H((o(-1ss)) Lu + 2N Ei_I(V(ss) -R,2
17:
# Compute total loss L
18:
L Cp + cifo - c2c
19:
# Update parameters and & through backpropagation, Ao and d6 is the step sizes 0 Ao VeL, o + Φ do VoL
20:
end for end while
22:
Based on the Proposition 4.2 and Theorem 4.3, we have the following conclusion:
with ratio clipping, we propose the following objective:
[Ag
((0) = ) rz(0) A,
[rr 11²
2c
Conclusion
(16)
The details of the objective will be discussed in the following section, and the pseudo-code is shown in Algorithm 1.
Optimizing the lower bound with TV divergence constrains offers more effective solution space than using KI. divergence constrains, leading to better policy improvement.
5. Theoretical Results
In this section, we provide some theoretical insights of the differences between PPO and SPO, demonstrating that SPO can be more effective in constraining probability ratios.
As a result, to optimize the lower bound (10), we aim to solve the following constrained optimization problem:
5.1. Objective Class
max E ((s,,a))~TT sta (0( Ag 0
Simplify the notation by using r and A to represent the probability ratio and the advantage value. Based the previous analysis, our goal is to find objective function f(r, A, é) such that while optimizing the surrogate objective r A, the probability ratio is constrained by Ir 1| S c.
(15)
E
(1) 1 6,
where ri (0) = To(ar (//////////.a (ar())) and Å, A(se, ar).
PPO attempts to satisfy the constraints (15) through ratio clipping, but this does not prevent excessive ratio deviations (demonstrated in Figure 2). The underlying reason is that ratio clipping causes certain data points to stop contributing to the gradients. Over multiple iterations, this can lead to un- controllable updates, as the absence of corrective gradients prevents the policy from recovering. To overcome this issue
According to (15), for any given A * and e > 0, we can write down the following desired optimization problem:
max A,, s.t. | 1 $
(17)
The objective linear, so the optimal solution is r* + sign(A) -c, where (ign()) is the sign function. Motivated by this, we present the following definition:
5

---

Simple Policy Optimization
Definition 5.1 (e-aligned). For any given A + 0 and O, we say that the function [(r, c) is e-aligned, if it is differentiable and convex with respect to r, and attains its maximum value at r = 1 + sign(A)
During the training process of PPO, certain data points that exceed the probability ratio bound cease to provide gradients. In contrast, all data points in SPO contribute gradients that guide the optimization towards the constraint boundary. As training progresses, PPO will accumulate more gray circles that no longer provide gradients and may influenced by the harmful gradients from green circles. This phenomenon could potentially push the gray circles further away from the constraint boundary. In contrast, the gradient directions of all data points in SPO point towards the constraint boundary. This indicates that SPO imposes stronger constraints on the probability ratio.
The objective of PPO in (6) and SPO in (16) can be ex- pressed as
fppo = min [r,, clip(r, -,, + +)
| 2E
(18)
(r 11)
Sapo = r.
It can be obtained that fppo is not e-aligned, as fppo zeros the gradients under some special cases according to (8). For fapo. we have the following theorem:
6. Experiments
Theorem 5.2. Sapo is e-aligned.
We report results on the Atari 2600 (Bellemare et al., 2013; Machado et al., 2018) and MuJoCo (Todorov et al., 2012) benchmarks. In all our experiments, we utilize the RI. li- brary Gymnasium (Towers et al., 2024), which serves as a central abstraction to ensure broad interoperability between benchmark environments and training algorithms.
Proof: Obviously, fapo is differentiable and convex with respect to since fapo is a quadratic polynomial of r. For any given A # 0 and e > 0, let Ssso (r, A, c)/Or = 0, then
Sspo (r, A, Ör
= A - |A| (r ( = 0,
(19)
6.1. Comparing Algorithms
thus r = signg ()) is the optimal solution for Sspa-
Our implementation of SPO is compared against PPO-Clip (Schulman et al., 2017). PPO-Penall (Schulman et al., 2017), SPU (Vuong et al., 2018), PPO-RB (Wang et al., 2020), TR-PPO (Wang al., 2020), TR-PPO-RB (Wang et al., 2020), and RPO (Gan al.. 2024) in MuJoCo bench- mark. We compute the algorithm's 's performance across ten separate runs with different random seeds. In addition, we emphasize that in all comparative experiments involving the same settings for SPO and PPO, the only modification in SPO is replacing the PPO's objective with (16), no fur- ther code-level tuning is applied to SPO, highlighting its simplicity and efficiency.
Note that fapo is not the only objective function that satis- fies the definition. It can be proved that there is simple objective function fsimple = (r- 11- - sign(A) -e)², which is also e-aligned. We will discuss the differences between these two in Section 6.3.
5.2. Analysis of New Objective
We show that the optimization process of SPO can more effectively bound the probability ratio, as can be seen from Figure 3. The largest circular area in the figure represents the boundary on the probability ratio. The green circles represent data points with non-zero gradients during the training process, while the gray circles represent data points with zero gradients.
Due to the absence of human score baselines in MuJoCo (Todorov et al., 2012), we normalize the algorithms' per- formance across all environments using the training data of PPO-Clip, specifically,
normalizze(score) = score - min
PPO
SPO
(20)
max- min
where max and min represent the maximum and minimum validation returns of PPO-Clip during training, respectively.
As suggested in Agarwal et al. (2021), we employ stratified bootstrap confidence intervals assess the confidence inter- vals of the algorithm and evaluate the composite metrics of SPO against other baselines, illustrated in Figure 4. It can be observed that SPO achieved the best performance across nearly all statistical metrics, which fully demonstrates the strong potential of SPO. For the Atari 2600 benchmark (Bellemare et al., 2013), the main results are presented in Appendix and C.
with gracient
(.19) 1) [..]
witoou gradent
dieecin
Figure 3. In PPO, certain data points exhibit zero gradients, while in SPO, all data points generate non-zero gradients that guide
towards the constraint boundary.
6

---

Simple Policy Optimization
Median
IQM
Mean
Optimallly ity Gap
RPO TR-PPO-RB
PPO-RB
TR-PPO SPU
I
SPO
Penaltnally
PPO-Cip
1,0
1.5
2.0
2.5
0.5
1.0
1.5
2.0
2.5
0.5
1.0
1.5
2.0
2.5
0.00
0.15
0.30
0.45
Figure 4, Aggregate metrics on MuJoCo-v4 with 95% CIs based on 6 environments. We collected the returns of cach algorithm over the last 1% training steps across ten random sceds. this context, higher median, IQM and mcan scores and lower optimality gap are better.
Antov4
HaltCheetah--e
Happor-v6
8000
1000
4000
Avengg
1000
2000
2000
500
04
1.00
Timestopo
147
Tet
Humanold--44
Walker2s-v4
200000
6000
190000
Bunaye 1e0m
0000
2000
2000
52000
0.15
1.00
Ttraslegs
147
Timeetaps
let
2PO (7 leyand)
2P0 Leyans)
SPO Lepans)
SPO (3 (3yyaes)
Figure 5.
Training performance of PPO and SPO with different policy network layers in MuJoCo benehmark. The mean and standard deviation are shown across 5 random seeds.
Table 1. Average return of PPO and SPO the last 10% training steps across separate runs with different random seeds, with their maximum ratio deviation during the entire training process.
Environment
Index
PPO
3 layers
SPO
PPO
layors
SPO
Amt-v4
5323.2
4911.3
10028
4672.5
Raboo deviation (L)
0.229
0.101
548.060
HallCheetab
Average Ratio deviation (1)
4550L2
3002.4
2242.3
5307.3
0.225
0.086
1675.340
0,188
Hopper-v4
Ratoo deviation (1)
Average resumn
1119,4
0,161
0.067
113.178
0.194
Humanoid-vd
Average return (t)
795.1
2870.0
614.1
4769.9
Ratio deviation (1)
3689.957
0.179
2411.445 14455
0.191
HumannidSSunnlp--44
Average retum (t)
143908.8
152378.7
92849.7
176928.9
Rato deviation (1)
2547.499
0.182
4018.718
0.187
Walker2d.v4
Ratio deviation (1)
Average ratumn (f)
3362.3
0,170
2870.2
0.970
998.101
1110.9
3008.1
0.157
6.2. Scaling Policy Network
ratio deviation in a batch during the entire training process, i.e., |2] Σ (s.,,.)~> Ira 1/.
To investigate how scaling policy network size impacts the sample efficiency of both PPO and SPO in MuloCo, the number of policy network layers was increased without altering the hyperparameters or other settings. The standard deviation of the algorithm's 's performance was computed and visualized across five separate runs with different random seeds. The results, shown in Figure 5, 9 and Table I, where the ratio deviation indicates the largest value of average
It can be observed that as the network deepens, the per- formance of PPO collapses in most environments, with uncontrollable probability ratio deviations. contrast, the performance SPO outperforms that of shallow networks in almost all environments and constrains the probability ratio deviation effectively. Furthermore, the statistical met- rics of SPO generally outperform PPO's and demonstrate

---

Simple Policy Optimization
Asssuit
Asterix
BeannRider
Spacel nvaders
30000
Ansiiggh rabur
10000
Asemag ream
Ananegi rebure
4000
Avemge netur
20000
10000
0.00
0.50
0.75
1.00
0.00
0.75
1.00
0.00
0.50
0.75
1.00
0.50
0.50
0.75
1.00
mimestppe
Timestaps
Timesteps
Asssult
Asterix
BeanFider
Spacel nvaders
Ressage devettio
Ausoge devaston
10"
devizton
10
10"
10"
10
10
10"
0.00
0.75
1.00
0.00
0.75
1.00
0.00
0.50
0.75
1.00
0.00
0.50
0.75
1.00
Timestops
Timesteps
PROO2 RRsNNos-88
PPO-0.1 (ResNet-18)
SPO-0.2
PPO-02
SPG-02/CNN)
Figure Training performance of SPO using ResNet-18 as the encoder compared the PPO and SPO using default CNN (shown with the reference line). The mean and standard deviation are shown across random seeds, and the red dashed line represents 0.2.
mance in optimizing the surrogate objective, it also leads uncontrollable ratio deviations. In contrast, the two objec- tives that satisfy the e-aligned definition effectively constrain the ratio deviations during the optimization process.
0:10
0.08
2.1
0.0
20
100
200
0.
100
150
200
Furthermore, we also observe that fapo achieves better op- timization of the surrogate objective compared to ssimples while fsimple converges more quickly to the probability ratio boundary. This aligns with expectations, the opti- mization objective of Ssimple only depends on the sign of the advantage values. As result, fsimple pushes each data point equally toward the constraint boundary, which results in the magnitude of the advantage values being less effec- tively utilized compared to Sspo- which makes difficult to efficiently optimize the surrogate objective.
Opoc1
Epoch
FP0
$P0
Strgle oijectees
Figure 7. The optimization behavior of Sppo- Sapos and seimple-
relative robustness variations in network depth and mini- batch size.
We also trained the ResNet-18' 18' (He et al., 2016) as the en- coder on the Atari 2600 benchmark, the results are shown in Figure 6. As the network's 's capacity increases, the perfor- mance of SPO significantly improved. Moreover, SPO can still maintain good probability ratio constraint, thereby benefiting from the theoretical lower bound (10). In contrast, it is challenging to train large neural networks with PPO because the probability ratio cannot be controlled during training, even employing a smaller 0.1.
7. Conclusion
In this paper, we introduce Simple Policy Optimization (SPO), a novel unconstrained first-order algorithm that effec- tively combines the strengths of Trust Region Policy Opti- mization (TRPO) and Proximal Policy Optimization (PPO). SPO maintains optimization within the trust region, bene- fiting from TRPO's theoretical guarantees while preserving the efficiency of PPO. Our experimental results demonstrate that SPO achieves competitive performance across various henchmarks with a simple implementation. Moreover, SPO simplifies the training of deep policy networks, addressing a key challenge faced by existing algorithms. These findings indicate that SPO is a promising approach for advancing model-free reinforcement learning. In future work, SPO holds potential for impactful applications in areas such as language models, robotic control, and financial modeling. With further research and refinement, we believe SPO will drive innovation and breakthroughs across these fields.
6.3. Constraining Ratio Deviation
To further investigate the optimization behavior of different objective functions that satisfy the e-aligned definition, we visualize the optimization process of Sppo- Sspa. and fsimple presented in Section 5.1, on the same batch advantage values initialized from a standard Gaussian distribution, as shown in Figure 7.
We can observe that while PPO achieves the best perfor-
Since Bhatt et al. (2019) demonstrated that batch normaliza- tion harmful RL. training, we removed batch normalization.
8

---

Simple Policy Optimization
Acknowledgements
Heess, N., Tb, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Frez, T., Wang, Z., Eslami, S., et al. Emer- gence of locomotion behaviours in rich environments. arXiv preprint 1707 02286, 2017.
We would like to thank the anonymous ICLR and ICML reviewers for their insightful and constructive comments.
Impact Statement
Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022a.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Huang, S., Dossa, R. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraAšjo, J. G. Cleanrl: High-quuling single-file implementations of deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022b.
References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on ma- chine learning, pp. 22-31. PMLR, 2017.
Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F. Rudolph, L., and Madry, A. Are deep policy gradi- ent algorithms truly policy gradient algorithms. arXiv preprint 1r1iv:181..0233. 2018.
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C.,, and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural in- formation processing systems, 34:29304-29320 2021.
Kakade, S. and Langford, Approximately optimal ap- proximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learn- ing, pp. 267-274, 2002.
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations, 2021.
Lehmann, M. The definitive guide to policy gradients in deep reinforcement learning: Theory, algorithms and im- plementations. arXiv preprint arXiv:2401. /3662, 2024.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M.
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M. and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562 5625,226-221 2018.
The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-2799 2799,22--22220 2013.
Bhatt, A., Argus, M., Amiranashvili, A., and Brox, T. Cross- norm: Normalization for off-policy td reinforcement learning. arXiv preprint arXiv:
Makoviychuk, V., Wawrzyniak, L.., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A.. Handa, A., al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108 10470, 2021.
10, 2019.
Black, K., Janner, M., Du Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
Engstrom, L.., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. Implementation matters in deep policy gradients: case study on ppo and trpo. arXiv preprint arXiv:2005,12722 2020.
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A.. Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Gan, Y., Yan, R., Wu, Z., and Xing, J. Reflective policy optimization. arXiv preprint 14X.::2400003678.. 2024.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 11-27730-27744 2022.
Greensmith, E.,, Bartlett, P. L., and Baxter, J. Variance reduc- tion techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Queeney, J., Paschalidis, Y., and Cassandras, C. G. Gener- alized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34: 11909-11919, 2021.
9

---

Simple Policy Optimization
Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learning to walk in minutes using massively parallel deep rein- forcement learning, In Conference Robot Learning, pp. 91-100. PMLR, 2022.
Vuong. Q., Zhang, Y., and Ross, K. W. Supervised policy update for deep reinforcement learning. arXiv preprint arXiv: 10005117000 1176,, 2018.
Wang. Y., He, H., and Tan, X. Truly proximal policy op- timization. In Uncertainty in artificial intelligence, pp. 113-122. PMLR, 2020.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
Ye, D., Liu, Z., Sun, M., Shi, B., Zhao, P., Wu, H., Yu, H., Yang, S., Wu, X., Guo, Q., et al. Mastering complex control in moba games with deep reinforcement learn ing. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672-6679, 2020.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abheel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02443. 2015b.
Zhuang, Z., Lei, K., Liu, J., Wang, D., and Guo, Y. Be- havior proximal policy optimization. arXiv preprint arXiv:2302.11312 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I,, Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 52947587)4444-88,, 2016.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354-55,, 2017.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel, T., al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Sci- ence, 362(6419)) 140- 144, 2018.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Sutton, R. S., Singh, S., and McAllester, D. Comparing policy-gradient algorithms. IEEE Transactions on Sys- tems, Man, and Cybernetics, 2000.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp- 5026-5033, IEEE, 2012.
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
Vinyals, O., Babuschkin, I., Czamecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575
(7782):350-354, 2019.
10

---

Simple Policy Optimization
A. Atari 2600
300
250
Perforannce inprovement
200
150
50
0
-50
Oper Heto BeetHiie
Upittamn Porttoon
Fitting Chopret
Spec Boning
Narall hectaree Lontin Kingf de
Witsen
Allen
Tearli Staritur Surnet
Bhany Gravior
Dempa Zancon
Figure 8. Final performance of SPO compared to PPO across 35 games in Atari 2600 environment, using default CNN as the encoder.
B. Hyperparameters
Table 2. Detailed hyperparameters used in SPO.
Hyperparameters
Atari 2600 (Bollemare et al., 2013)
MuloCo (Todorov et al., 2012)
Number of workers
8
Horizon
256
Learning
0.00025
0.0003
Learning rate decay
Linear Adam
Linear
Optimizer
Adam
Total steps
10M
10M
2048
Update opochs
10
Mini-butches
4
Mini batch
256
512
GAF parameter X Discount factor
0.95
0.95
0.99
Value loss cnefficient es
0.5
0.5
Entropy loss coefficient 0g
Probability ratio hyperparamcter
0.2
0.2
C. More Results
Median
IQM
Mean
Optimality
SPO 512)
SPO (7 layers, 512)
SPO layers, 64)
SPO (7 layers, 64)
PPO (3 layers, 512)
layers, 512)
PPO (3 layers, 64)-
layers,
2.4
0.00
0.15
0.30
0.45
Figure 9. Aggregate metrics MuJoCo-v4 with 95% CIs based on 6 environments, comparing PPO and SPO with different policy network layers mini-batch sizes using PPO-normalized score.

---

Simple Policy Optimization
Alien
Amidar
Assault
Asterix
Asteroide
1e6
Allan
5000
500
10000
20000
2000
2000
250
5000
10000
1000
1000
0.-
BankHeist
Battle Zone
Rean
Berzerk
Bowling
Boxing
1000
20000
750
40
20
500
10000
2000
250
0
20
Breakout
Carnival
Centipede
ChopperCommend
CrazyCllbber
Defonder
4000
4000
4000
100000
40000
2000
50000
20000
2000
DemonAltack
ElevatorAction
Fish gerbby
Frenway
Frosibite
Gapher
4000
40000
20000
2000
100
-100
Gravilar
Horo
look Hockey
Jamesbond
JourneEsccape
Kangaroo
-5
500
4000
20000
10000
200
-10-
0
-20000
KungFuMaster
Reveezu
Ms Pacman
Game
Phoenix
Pitfall
2000
7600
20
5000
10000
250
1000
167
Pong
Pooyan
PrivateEye
Qbert
Riverraid
RoadRunner
4000
2000
20000
40000
2000
5000
20000
-20
16r
Robolank
Seaquest
Spacelnvaders
StarGunner
Tennis
TimePijot
1900
2000
50000
7500
255
1000
25000
0.0
500
2500
1e7
ter
Tutankham
UpNDown
Venture
VideoPinball
WizardOWWor
Zaxcon
100000
2000
500
100
50000
1000
0
1x7
147
tw7
tw7
1a7
PPO Restiai-18)
SPO (ReaNal-18)
Figure 10. Training performance on Atari 2600 using ResNet- 18 as the encoder, with a fixed learning rate 0.0001. The mean and standard deviation are shown across 3 random seeds.
12


Reviews:
[{"review_id": "3PcARdTiRU", "reviewer": "Reviewer_SqU8", "summary": "This paper studies an alternative of PPO, named Simple Policy Optimization (SPO), by optimizing a tighter performance lower bound using Total Variation (TV) divergence. The authors are concerned with PPO’s limitation in constraining probability ratios, which is an important problem to study.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SqU8,\n\nThank you for your constructive feedback. Below, we will address your concerns.\n\n>PPO is widely used in large-scale RL problems, and KL regularization will be incorporated when the policy NN is complex, such as LLM...\n\nWe appreciate your suggestion. However, the purpose of KL regularization in PPO for LLMs is to prevent reward over-optimization and catastrophic forgetting caused by policy collapse. This does not prevent PPO's ratio deviation issue, as ratio deviation is defined with respect to the initial policy $\\pi_ {\\theta_ {\\mathrm{old}}}$ at each update. Additionally, due to the limited rebuttal period, we may not be able to provide comparative results of SPO-fine-tuned LLMs. We sincerely apologize for this limitation.\n\n>Another interesting ablation will be to decrease PPO's $\\epsilon$ or increase SPO's $\\epsilon$ so that their average ratio deviation is the same and see if the ratio is the main reason for SPO's better performance.\n\n>An ablation on SPO's $\\epsilon$ will help the readers better understand its robustness.\n\nThank you for your comment. We have supplemented the following experiments:\n\n| Environment   | Ant-v4 | Humanoid-v4  | HumanoidStandup-v4 |Walker2d-v4 |\n|--------|-----|------|------|------|\n| SPO ($\\epsilon=0.1$)   | $5045.05\\pm1005.02$| $4866.81\\pm1230.83$| $160103.83\\pm11509.79$ |$3737.86\\pm903.16$ |\n| SPO ($\\epsilon=0.2$)   | $4760.69\\pm849.75$| $4852.76\\pm1414.11$| $178853.55\\pm43151.81$ |$2944.91\\pm1257.06$ |\n| SPO ($\\epsilon=0.3$)   | $4096.41\\pm863.53$| $4532.02\\pm1319.04$ |$156627.08\\pm8519.33$ |$3193.54\\pm1148.36$ |\n| SPO ($\\epsilon=0.4$)   | $3594.85\\pm857.46$| $4555.34\\pm1497.48$ |$165393.59\\pm29599.54$ |$2423.38\\pm1112.68$ |\n| SPO ($\\epsilon=0.5$)   | $3258.7\\pm854.74$| $2516.6\\pm1050.51$ |$149689.79\\pm16572.68$ |$2187.09\\pm989.88$ |\n\nThe results demonstrate that over-large $\\epsilon$ (0.5) consistently leads to performance degradation, which aligns with the theoretical lower bound.\n\n>The paper [1] also studies replacing the divergence term to TV divergence but is not discussed.\n\nThank you for pointing out. We will include this in the related work section in our subsequent revisions.\n\n>The related works that use TV divergence or more general divergence are not thoroughly discussed.\n\nWe appreciate the suggestion. However, to our knowledge, [2] is the only existing work that discusses the relationship between total variation (TV) divergence and the policy ratio, we have discussed this paper in our related work section. KL divergence is first proposed in original TRPO paper [3]. To the best of our knowledge, beyond TV divergence and KL divergence, there appear to be no published works employing more general divergence measures in this context.\n\n>I didn't observe obvious typos. Authors may consider replace \"can not\" to \"cannot\".\n\nThank you for your thorough review. We will carefully incorporate your suggestions.\n\n>Can the authors comment on the potential drawbacks of the proposed method or, more generally, using TV divergence, compared to PPO and TRPO, such as instabilities? And will add KL regularization to PPO make it better?\n\nIn fact, we compared a PPO variant with KL regularization (PPO-Penalty), as shown in Figure 4 of our paper. The results demonstrate that SPO achieves superior performance.\n\nBest,\n\nAuthors\n\n---\n*Reference:*\n\n[1] X Chu et al. A strong on-policy competitor to PPO.\n\n[2] J Queeney et al. Generalized proximal policy optimization with sample reuse.\n\n[3] J Schulman et al. Trust region policy optimization.", "claims_and_evidence": "Most claims are well supported.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I checked the theories that show the properties of the proposed method but did not check the derivation in previous sections.", "experimental_designs_or_analyses": "Yes, the experimental designs are valid. However, it would make the results more convincing and interesting if the following problems were also studied: \n1. PPO is widely used in large-scale RL problems, and KL regularization will be incorporated when the policy NN is complex, such as LLM. By doing so, the concerned limitation of PPO in constraining probability ratios would not be a problem and hardly to be observed in practice. Can the authors compare the above PPO variant and SPO? It would be great if the experiment could be completed for more complex NNs, such as LLM alignment or reasoning. The studied 7-layer MLP is still not deep enough to claim SPO's robustness in larger-scale settings.\n\n2. Another interesting ablation will be to decrease PPO's $\\epsilon$ or increase SPO's $\\epsilon$ so that their average ratio deviation is the same and see if the ratio is the main reason for SPO's better performance.\n\n3. An ablation on SPO's $\\epsilon$ will help the readers better understand its robustness.", "relation_to_broader_scientific_literature": "This paper is related to PPO and other zeroth-order policy gradient methods.", "essential_references_not_discussed": "The paper [1] also studies replacing the divergence term to TV divergence but is not discussed.\n\n[1] Chu et al. \"A Strong On-Policy Competitor To PPO.\"", "other_strengths_and_weaknesses": "Strengths: 1. The paper is clearly written and easy to follow.\\\n2. The paper has a good motivation.\n\nWeaknesses: 1. The related works that use TV divergence or more general divergence are not thoroughly discussed.\\\n2. The experimental results and ablations can be further enhanced. Please see the Experimental Designs Or Analyses part.", "other_comments_or_suggestions": "I didn't observe obvious typos. Authors may consider replace \"can not\" to \"cannot\".", "questions_for_authors": "Can the authors comment on the potential drawbacks of the proposed method or, more generally, using TV divergence, compared to PPO and TRPO, such as instabilities? And will add KL regularization to PPO make it better?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rCdZE6XoMa", "reviewer": "Reviewer_8JyT", "summary": "The paper introduces Simple Policy Optimization (SPO), a new unconstrained first-order reinforcement learning algorithm designed to effectively combine strengths from Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). SPO modifies PPO's objective by proposing a novel surrogate loss that constrains the probability ratio using Total Variation (TV) divergence, theoretically enhancing policy improvement. Empirical evaluations on Atari and MuJoCo benchmarks suggest SPO can achieve better or comparable performance than PPO, particularly when training deeper neural networks.\n\n\n### Update After Rebuttal\n\nI appreciate the authors’ response. However, none of my concerns have been adequately addressed.\n\n**Regarding Theorem 4.3 and the associated conclusion:**\nIt appears that the authors misunderstood the core of my concern. My point is that Theorem 4.3 does *not* support the claim that \"TV can lead to better policy improvement.\" What is actually shown is that the surrogate objective with TV is greater than that with KL, i.e., $L(\\pi_{\\text{TV}}) \\ge L(\\pi_{\\text{KL}})$. This does not imply that the *true* performance of the final policy is better, i.e., $\\eta(\\pi_{\\text{TV}}) \\ge \\eta(\\pi_{\\text{KL}})$. Additionally, Equation (3) should not include a constraint, as the surrogate objective itself does not include one.\n\n**Regarding the rigor of Equations 17–18:**\nWhat I am asking for here is a formal and rigorous proof. I am concerned that the method may be biased due to the added term, and this needs to be clarified with a precise derivation.\n\n**On experimental sufficiency:**\nAs I previously noted, even adding a random component can result in performance improvements on half of the 57 Atari games. The current experimental evidence is therefore not sufficient to substantiate the paper’s claims.\n\nGiven the above concerns, I will maintain my original score. The paper itself may introduce some potential alternative solution to PPO. I hope the author polish their theories and enhance their experiments.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8JyT,\n\nThank you for your comment. Below, we will address your concerns.\n\n>The authors claim that \"TV divergence offers a larger solution space compared to methods incorporating a looser KL divergence constraint.\" However, the fundamental issue is that when optimizing the surrogate objective loss, the paper does not actually enforce a KL divergence constraint. Specifically, in equation (13), restricting the solution space to $\\tilde{\\pi}\\in\\Omega_ {\\mathrm{TV}}$ is incorrect. The authors seem to have confused constraint-based optimization objectives with surrogate loss-based objectives.\n\nThank you for your comment. However, we respectfully disagree with this perspective. In our approach, we introduce the SPO objective in Equation (16) to constrain the probability ratio, thereby ensuring the $\\epsilon$-aligned property defined in Definition 5.1. Furthermore, as established in Equation (9), constraining the probability ratio across batch data inherently bounds the total variation (TV) divergence, which naturally leads to a TV divergence-based trust region. Importantly, Theorem 4.3 demonstrates that the performance improvement lower bound under TV divergence constraints is indeed superior to that under KL divergence constraints.\n\n>The derivation from Eq. (17) to Eq. (18) does not appear rigorous. Specifically, replacing the absolute value $|r-1|$ with the squared term $(r-1)^2$ significantly changes the nature of the function. The paper lacks an analysis of the divergence introduced by this transition from Eq. (17) to Eq. (18). Additionally, it is unclear whether this modification will lead to actual policy performance improvements.\n\n>Definition 5.1 does not appear clearly motivated or sufficiently rigorous. The notion of an \"$\\epsilon$-aligned\" surrogate is not strong enough to guarantee policy improvement. In fact, many surrogate functions can satisfy the condition of being \"$\\epsilon$-aligned\" yet fail to yield any meaningful performance improvement. Therefore, the practical usefulness and theoretical significance of this definition remain unclear.\n\n>Can you rigorously justify why the absolute deviation (eq. 18) is preferred over squared deviation, given significant conceptual differences?\n\nThis might be a misunderstanding of our method. We employ the squared penalty because it is convex with respect to the probability ratio $r$, and its optimal solution naturally lies at the probability ratio boundary $r^*=1+\\mathrm{sign}(A)\\cdot\\epsilon$. This demonstrates our method's effectiveness in constraining the probability ratio, which consequently leads to indirect optimization of the policy's performance improvement lower bound (10).\n\n>How do you explain the contradictory performances of SPO in Figures 1 and 8?\n\nPlease note that Figure 1 presents results using 50 to 101-layer ResNet architectures, while Figure 8 employs the default shallow CNN structure for comparison.\n\n>What are the detailed reasons for selecting only 35 out of the 57 available Atari environments, and how were these environments chosen?\n\nWe deliberately selected a subset of Atari environments where PPO can operate successfully. Among the 57 Atari games, some involve sparse reward settings where neither PPO nor SPO can effectively train reinforcement learning policies. Furthermore, given the substantial computational overhead of Atari experiments, running all environments would be prohibitively time-consuming. We maintain that the environments included in our experiments provide sufficient basis for meaningful algorithmic comparison.\n\n>Could you explicitly compare the performance of SPO with PPO-penalty, clearly identifying whether TV divergence or other modifications provide the primary benefit?\n\nThank you for your suggestion. Figure 4 in the paper includes comparative experiments between PPO-Penalty and other extensive baselines, with the results clearly demonstrating SPO's performance advantages over both PPO-Clip and PPO-Penalty in MuJoCo environments (10 seeds).\n\nBest,\n\nAuthors", "claims_and_evidence": "The main claims—that SPO achieves superior policy improvement by effectively bounding probability ratios using TV divergence—are inadequately supported by the presented evidence. Specifically, the theoretical claim that \"TV divergence offers a larger solution space than KL divergence\" is problematic. The provided proofs confuse constraint-based optimization (eq. 13) and surrogate objective-based optimization (eq. 16). Consequently, the statement that TV constraints yield a \"more effective solution space\" than KL constraints lacks convincing support and clarity. Additionally, the step from eq. (17) to (18) and the treatment of the absolute advantage versus squared difference are neither adequately justified nor analyzed.\n\nPlease also see Theoretical Claims for more detail.", "methods_and_evaluation_criteria": "In my understanding, the proposed approach differs from PPO with  KL Penalty primarily in two aspects:\n\n- It replaces the KL divergence with alternative divergence terms.\n- It introduces an absolute value of the advantage term into the adaptive coefficient.\n\nTherefore, a comparison with this PPO variant would be necessary to clearly demonstrate whether using TV divergence provides any advantage over KL divergence.", "theoretical_claims": "Q 1:\n\nThe paper contains several clear theoretical and logical errors.\n\nThe authors claim that \"TV divergence offers a larger solution space compared to methods incorporating a looser KL divergence constraint.\" However, the fundamental issue is that when optimizing the surrogate objective loss, the paper does not actually enforce a KL divergence constraint. Specifically, in equation (13), restricting the solution space to $\\tilde{\\pi} \\in \\Omega_{TV}$ is incorrect. The authors seem to have confused constraint-based optimization objectives with surrogate loss-based objectives.\n\nConsequently, their conclusion that \"Optimizing the lower bound with TV divergence constraints offers a more effective solution space than using KL divergence constraints, leading to better policy improvement\" is incorrect.\n\nTo clarify this rigorously, if the authors wish to demonstrate that surrogate objective $A$ is superior to surrogate objective $B$, they must:\n\n- Remove artificial restrictions on the solution space, such as $\\tilde{\\pi} \\in \\Omega_{TV}$.\n- Provide a formal proof showing $\\eta(\\pi^*_A) \\geq \\eta(\\pi^*_B)$ rather than relying solely on comparisons using the surrogate loss $\\mathcal{L}$.\n\n\n\n\n\nQ 2:\n\n\n\nThe derivation from Eq. (17) to Eq. (18) does not appear rigorous. Specifically, replacing the absolute value $|r - 1|$ with the squared term $(r - 1)^2$ significantly changes the nature of the function. The paper lacks an analysis of the divergence introduced by this transition from Eq. (17) to Eq. (18). Additionally, it is unclear whether this modification will lead to actual policy performance improvements.\n\n\n\nQ 3:\n\nDefinition 5.1 does not appear clearly motivated or sufficiently rigorous. The notion of an \"$\\epsilon$-aligned\" surrogate is not strong enough to guarantee policy improvement. In fact, many surrogate functions can satisfy the condition of being \"$\\epsilon$-aligned\" yet fail to yield any meaningful performance improvement. Therefore, the practical usefulness and theoretical significance of this definition remain unclear.", "experimental_designs_or_analyses": "The experimental analyses have limitations. The selection of only 35 out of 57 Atari environments raises concerns about potential cherry-picking or hyperparameter sensitivity. The inconsistent results between Figure 1 and Figure 8 suggest sensitivity to hyperparameters, not robustness. \n\n\nAdditionally, the inconsistent performance of PPO-Penalty regarding median metrics versus optimality gaps requires further clarification. Moreover, the method \"PPO-Penalty\" itself is not clearly defined in the paper; I could not find any explicit definition or description provided.\n\nFinally, deeper network experiments compared only to PPO are insufficient to demonstrate the generality of the proposed method; comparisons to additional baselines are required.", "relation_to_broader_scientific_literature": "The paper appropriately situates itself within the literature of TRPO and PPO improvements. However, th theoretical contributions claimed, especially regarding TV versus KL divergence, must be more clearly contextualized with respect to existing results on divergence constraints in reinforcement learning literature.", "essential_references_not_discussed": "Further discussion and comparison with PPO using a KL penalty would be beneficial.", "other_strengths_and_weaknesses": "Strengths include the simplicity of the SPO objective and promising empirical results for deeper neural networks. Weaknesses center primarily around theoretical ambiguities, insufficient comparative analysis, and questionable hyperparameter sensitivity.", "other_comments_or_suggestions": "- Formulations can be streamlined for clarity; currently, some equations are unnecessarily repeated or redundant.\n- Theorem 3.1 is not actively used and does not aid in understanding the proposed SPO method; it could be omitted or properly integrated.", "questions_for_authors": "1. Can you rigorously justify why the absolute deviation (eq. 18) is preferred over squared deviation, given significant conceptual differences?\n2. How do you explain the contradictory performances of SPO in Figures 1 and 8?\n3. What are the detailed reasons for selecting only 35 out of the 57 available Atari environments, and how were these environments chosen?\n4. Could you explicitly compare the performance of SPO with PPO-penalty, clearly identifying whether TV divergence or other modifications provide the primary benefit?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ISQyd5RjCU", "reviewer": "Reviewer_eeAe", "summary": "This paper introduces Simple Policy Optimization (SPO), a first-order algorithm that modifies PPO's policy loss to achieve stronger theoretical properties, particularly in bounding the probability ratios between successive policies. The authors argue that by optimizing a lower bound under TV divergence constraints, SPO provides a more effective solution space than approaches using KL divergence (e.g., TRPO). Empirical results show that SPO performs comparably to PPO across some benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eeAe,\n\nThank you for your constructive feedback. Below, we will address your concerns.\n\n>Partially. The claim regarding improved theoretical properties is plausible, but empirical support for SPO’s superiority over PPO and TRPO is limited.\n\n>The paper lacks a clear demonstration of significant performance improvements over PPO, with most experimental results showing only marginal gains.\n\n>Include a comparison with TRPO, as the paper suggests that SPO combines the advantages of both TRPO and PPO. A head-to-head comparison would strengthen this claim.\n\nThank you for your suggestion. However, as demonstrated in Figures 1, 4, and 5 of our paper, our experiments show that SPO indeed exhibits significant advantages over PPO. Notably, Figure 4 includes comparisons with **a wide range of baselines**. Unfortunately, due to TRPO's computationally expensive second-order optimization, it is typically not used as a baseline in existing works [1, 2, 3].\n\n>Claims regarding the theoretical advantages of SPO over PPO and TRPO are not fully substantiated with rigorous empirical validation.\n\nThank you for your feedback. As shown in Table 1 of our paper, when PPO trains deeper networks, the probability ratio becomes uncontrolled and leads to performance collapse in PPO. Please note that in this experiment, the only variable modified was the network depth - no additional code-level tuning was performed for SPO. We believe this strongly demonstrates SPO's theoretical advantage, as its probability ratio remains effectively constrained without performance degradation even with increasing network depth.\n\n>Key implementation details and hyperparameters (e.g., the reasoning behind comparing 3-layer vs. 7-layer architectures) are insufficiently explained...\n\n>In the experiments, provide more details on architectural choices (e.g., why 3-layer and 7-layer networks were compared)...\n\nThank you for your suggestion. The reviewer's concern primarily relates to the fairness of the experimental setup and the potential possibility that SPO might involve additional hyperparameters or code-level tuning. Regarding this point, please refer to our already uploaded SPO and PPO implementations, where you will find that the only difference between SPO and PPO lies in the policy loss computation (the sole modification in the trainer.py file, with all other settings remaining identical).\n\nAs for the network depth selection, we employed both the default three-layer mlp [$\\mathrm{dim}(\\mathcal{S})$, 64, 64, $\\mathrm{dim}(\\mathcal{A})$] and a randomly selected seven-layer mlp [$\\mathrm{dim}(\\mathcal{S})$, 256, 256, 128, 128, 64, 64, $\\mathrm{dim}(\\mathcal{A})$] to ensure comprehensive evaluation.\n\n>Several experimental setups (e.g., Figure 6) lack clarity in terms of legend explanations and parameter settings. Additionally, the small number of random seeds (only three) undermines the statistical robustness of the results.\n\nThank you for your careful review. In Figure 6, the red dashed line represents the value of 0.2, while all other elements are clearly indicated in the legend.\n\nRegarding the random seeds, due to the computational costs in Atari environments, using 3 seeds is a common choice [4]. Furthermore, for the MuJoCo environments, the results presented in Figure 4 were indeed obtained using 10 random seeds to ensure statistical reliability.\n\n>Q2. Is there a tunable parameter in SPO that allows interpolation between PPO-like and TRPO-like behavior?...\n\nPlease note that TRPO and PPO exhibit fundamentally different optimization behaviors. SPO combines TRPO's theoretical guarantee (monotonic improvement) with PPO's computational efficiency (eliminating the need for second-order optimization).\n\n>Q3. What is the meaning of the values shown in the legends of Figures 5 and 6 (e.g., 0.1, 0.2)?...\n\nThe red dashed lines in these figures represent the hyperparameter epsilon (typically set to 0.2), and we will add the corresponding legend in subsequent revisions. The remaining horizontal lines, which are already labeled in the legend, indicate the performance of the original CNN at convergence (serving as baseline references).\n\n>Q4. Can you clarify the meaning of Theorem 5.1 (that SPO is $\\epsilon$-aligned)? How does this translate into practical advantages in policy optimization?\n\nPlease refer to the performance improvement lower bound (10). Theorem 5.1 demonstrates that SPO can indirectly optimize this lower bound (10) by constraining the probability ratio deviation $\\|\\frac{\\tilde{\\pi}(a|s)}{\\pi(a|s)}-1\\|\\leq\\epsilon$.\n\n\n\n\nBest,\n\nAuthors\n\n---\n*Reference:*\n\n[1] K Cobbe et al. Leveraging procedural generation to benchmark reinforcement learning.\n\n[2] Y Wang et al. Truly proximal policy optimization.\n\n[3] Y Wang et al. Trust region-guided proximal policy optimization.\n\n[4] Y Gan et al. Reflective policy optimization.", "claims_and_evidence": "Partially. The claim regarding improved theoretical properties is plausible, but empirical support for SPO’s superiority over PPO and TRPO is limited.", "methods_and_evaluation_criteria": "Yes, the methods are appropriate for policy optimization tasks, though additional experiments would strengthen the evaluation.", "theoretical_claims": "While the high-level ideas of the theorem seem sound, the integration into the overall argument could be clearer.", "experimental_designs_or_analyses": "Experimental designs are reasonable but lack robustness due to a small number of seeds and limited hyperparameter tuning discussions.", "relation_to_broader_scientific_literature": "Builds upon PPO and TRPO, addressing known issues with PPO's trust region enforcement.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n\n- Proposes a modification to the PPO objective that aims to provide stronger theoretical guarantees.\n- Provides theoretical analysis comparing different divergence constraints and their impact on policy optimization.\n- Experimental validation includes several benchmark environments, which help illustrate the method’s potential in diverse tasks.\n\n**Weaknesses**\n\n- The paper lacks a clear demonstration of significant performance improvements over PPO, with most experimental results showing only marginal gains.\n- Claims regarding the theoretical advantages of SPO over PPO and TRPO are not fully substantiated with rigorous empirical validation.\n- Key implementation details and hyperparameters (e.g., the reasoning behind comparing 3-layer vs. 7-layer architectures) are insufficiently explained. This makes it difficult to assess whether the results stem from algorithmic improvements or implementation choices.\n- Several experimental setups (e.g., Figure 6) lack clarity in terms of legend explanations and parameter settings. Additionally, the small number of random seeds (only three) undermines the statistical robustness of the results.", "other_comments_or_suggestions": "- In the experiments, provide more details on architectural choices (e.g., why 3-layer and 7-layer networks were compared). Discuss whether the variations in performance are due to SPO or other factors.\n- Include a comparison with TRPO, as the paper suggests that SPO combines the advantages of both TRPO and PPO. A head-to-head comparison would strengthen this claim.\n- Provide clearer explanations in the captions of Figures 5, 6, and 7. For example, explain what the numeric values in the legends (e.g., 0.1, 0.2) represent and clarify the meaning of horizontal lines.\n- Expand the experimental evaluation to include more random seeds (at least 10) to enhance the reliability of the results.", "questions_for_authors": "Q1.  How do the proposed SPO results compare directly with TRPO across the same benchmarks? Can you provide experimental comparisons to clarify this?\n\nQ2.  Is there a tunable parameter in SPO that allows interpolation between PPO-like and TRPO-like behavior? If so, how does varying this parameter affect performance?\n\nQ3.  What is the meaning of the values shown in the legends of Figures 5 and 6 (e.g., 0.1, 0.2)? If these values represent $\\epsilon$, how is  $\\epsilon$ specifically related to the ResNet architecture and not to other settings? What do the horizontal lines in these figures represent?\n\nQ4.  Can you clarify the meaning of Theorem 5.1 (that SPO is $\\epsilon$-aligned)? How does this translate into practical advantages in policy optimization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zPdbkxmB4m", "reviewer": "Reviewer_uY8N", "summary": "This paper theoretically identifies a flaw of the clipping technique in PPO's objective and proposes a solution to it. Empirical results show that the proposed solution achieves comparable or better performance on MuJoCo and Atari, and the performance improves as the policy network scales.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer uY8N,\n\nThank you for your positive feedback. Below, we will address your concerns.\n\n>Figure 10: The results on Atari are averaged over only 3 random seeds, which I think is not sufficient.\n\nThank you for your suggestion. Given the high computational costs associated with Atari environments, the use of 3 random seeds represents a well-established practice in the field [2]. Furthermore, due to time constraints during the rebuttal period, we regret that we are unable to conduct additional experiments with more random seeds.\n\n>The results in Figure 8 look stronger than the MuJoCo results and would be better if put in the main paper instead of the appendix.\n\nThank you for your suggestion! Due to space constraints, we have included the key content in the main text (Figure 4 compares a broader range of baselines). If the paper is accepted, we will also incorporate Figure 8 into the main body.\n\n>Could you specify which implementation of PPO you are using (which implementation SPO is based on)?...\n\nOur implementation of PPO is based on the standard CleanRL library: https://github.com/vwxyzjn/cleanrl, with the only modification being the computation of the policy loss in SPO. The complete code for this study has been submitted as part of the **supplementary materials**.\n\n>Have you tried scaling the width of the network besides the depth? I'm curious because a recent paper has shown that scaling the width can yield significant performance improvements on its own [1].\n\nThank you for the additional paper. Our three-layer network follows the default settings: [$\\mathrm{dim}(\\mathcal{S})$, 64, 64, $\\mathrm{dim}(\\mathcal{A})$], while the seven-layer network is both deeper and wider in architecture: [$\\mathrm{dim}(\\mathcal{S})$, 256, 256, 128, 128, 64, 64, $\\mathrm{dim}(\\mathcal{A})$].\n\n>Do you have any idea why PPO tends to perform worse and lose control of the probability ratio as the network scales?\n\nCertainly! We'd be happy to provide a simplified explanation. Generally speaking, deeper networks tend to have larger Lipschitz constants, meaning that even minor changes in the network's input can lead to significant variations in its output. Consequently, when using deeper architectures, PPO's limitations become more pronounced, as slight parameter adjustments may cause the probability ratio to exceed its boundary.\n\n>Could you show the scaling performance in Atari? It would be more convincing if there are multiple benchmarks where the results support the claim.\n\nThank you for your suggestion. We primarily demonstrate the scaling performance in Atari environments through Figures 1 and 6. Specifically, Figure 1 shows that SPO can effectively train ResNets with over 100 layers, while Figure 6 highlights SPO's significant performance improvements using ResNet-18 across four Atari environments. \n\nAdditionally, due to the limited rebuttal period, we may require additional time to obtain the Atari results, potentially during the discussion phase.\n\n>Figure 2 (right): Could you specify the task on which the results are produced?\n\nSure! We generated Figure 2 using the Hopper-v4 environment. In fact, similar results can be reproduced in any environment (e.g., Humanoid-v4).\n\nBest,\n\nAuthors\n\n---\n*Reference:*\n\n[1] J Obando-Ceron et al. In value-based deep reinforcement learning, a pruned network is a good network.\n\n[2] Y Gan et al. Reflective policy optimization.", "claims_and_evidence": "The abstract claims that TRPO has a strong theoretical foundation while PPO has better practical efficiency, and that the proposed method achieves the best of both worlds. This sounds a bit odd because from the empirical results the proposed method actually performs better than PPO (this is even stated at the end of the abstract).", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, I did not.", "experimental_designs_or_analyses": "Figure 10: The results on Atari are averaged over only 3 random seeds, which I think is not sufficient.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The theoretical explanation of why PPO's clipping technique may not be optimal is convincing, and the corresponding illustration in Figure 2 is intuitive.\n- The implementation of the proposed method is simple.\n- The empirical performance of the proposed method is strong, especially on Atari tasks. The fact that the performance improves as the network scales is particularly promising.", "other_comments_or_suggestions": "The results in Figure 8 look stronger than the MuJoCo results and would be better if put in the main paper instead of the appendix.", "questions_for_authors": "- Could you specify which implementation of PPO you are using (which implementation SPO is based on)? I am a little concerned since as also pointed out by this paper, existing work has shown that the performance of PPO is highly dependent on code-level optimization. It would be better if the url of the implementation is included in the paper.\n- Have you tried scaling the width of the network besides the depth? I'm curious because a recent paper has shown that scaling the width can yield significant performance improvements on its own \\[1\\].\n- Do you have any idea why PPO tends to perform worse and lose control of the probability ratio as the network scales?\n- Could you show the scaling performance in Atari? It would be more convincing if there are multiple benchmarks where the results support the claim.\n- Figure 2 (right): Could you specify the task on which the results are produced?\n\n\n\n\\[1\\] Obando-Ceron et al., \"In value-based deep reinforcement learning, a pruned network is a good network\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper propose a new policy gradient algorithm SPO.

The strength includes

(1) simplicity: SPO only requires single-line modification to the code of PPO algorithm.
(2) well-justified theoretical motivation: the paper points out a limitation of PPO in that it can fail to bound the probability ratios between successive policies despite trying to do so. And the proposed SPO update rule solves this issue.
(3) the claim is also backed up by extensive experimental evaluation, including the Atari and Mujuco benchmarks with various neural network architecture.

There is concern raised among the reviewers regarding the theory-practice mismatch. The analysis provided in the paper is on the TV constrained version of the optimization objective while the actual algorithm takes the Lagrangian form. While this is indeed not ideal, it's arguably quite common in the RL literature, such as in the original TRPO and PPO paper.

Overall, despite that, I believe this is a solid contribution to the suite of scalable RL algorithms.
Decision: Accept (poster)