On Stationary Point Convergence of PPO-Clip
Authors: Ruinan Jin
Shuai Li
Baoxiang Wang
Venue: ICLR 2024 poster (2024)
Keywords: PPO
PPO-Clip
stochastic optimization
Primary area: reinforcement learning
TL;DR: We provide a comprehensive analysis that shows the stationary point convergence of PPO-Clip and the convergence rate thereof.

Abstract: Proximal policy optimization (PPO) has gained popularity in reinforcement learning (RL). Its PPO-Clip variant is one the most frequently implemented algorithms and is one of the first-to-try algorithms in RL tasks. This variant uses a clipped surrogate objective function not typically found in other algorithms. Many works have demonstrated the practical performance of PPO-Clip, but the theoretical understanding of it is limited to specific settings. In this work, we provide a comprehensive analysis that shows the stationary point convergence of PPO-Clip and the convergence rate thereof. Our analysis is new and overcomes many challenges, including the non-smooth nature of the clip operator, the potentially unbounded score function, and the involvement of the ratio of two stochastic policies. Our results and techniques might share new insights into PPO-Clip.

Published as a conference paper at ICLR 2024
ON STATIONARY POINT CONVERGENCE OF PPO-CLIP
Ruinan Jin
Shuai Li
The Chinese University of Hong Kong, Shenzhen Einrvinanccuhi edu. en
Shanghai Jiao Tong University shuaili Bouaili8@s jtu. edu. cn
Baoxiang Wang"
The Chinese University of Hong Kong, Shenzhen sriangwang@euhle edu. en
ABSTRACT
Proximal policy optimization (PPO) has gained popularity in reinforcement learn- ing (RL). Its PPO-Clip variant one the most frequently implemented algorithms and is one of the first-to-try algorithms in RL tasks. This variant uses clipped surrogate objective function not typically found in other algorithms. Many works have demonstrated the practical performance of PPO-Clip, but the theoretical un- derstanding of it is limited to specific settings. In this work, we provide compre- hensive analysis that shows the stationary point convergence of PPO-Clip and the convergence rate thereof. Our analysis is new and overcomes many challenges, including the non-smooth nature of the clip operator, the potentially unbounded score function, and the involvement of the ratio of two stochastic policies. Our results and techniques might share new insights into PPO-Clip.
INTRODUCTION
Proximal policy gradient (PPO) one of the most popular algorithms in reinforcement learning (RL) (Schulman et al. (017). For many RI. tasks, one would try PPO as their first attempt, for its generality and stability among different environments as well as the easiness of implementation. As of September 2023, it returns over 5,500 results on GitHub.com if one searches "PPO", This demon- strates enthusiasm for reproducing the algorithm, applying it to different problems, and extending to many variants.
PPO stems from policy gradient, which is proposed by Williams 1992) as one of the seminal works in RL. The idea is to utilize the policy in previous step the new policy to be within a small discrepancy of the old policy. In this way, the policy does not suddenly deviate from its previous iterations by too much. The idea well-compatible with modern deep learning models, such as advantage function estimation and policy parametrization.
off-policy reference, and restrain
There are two primary variants of PPO introduced by Schulman al. (017). For the sake of convenience, we call them PPO-Penalty and PPO-Clip. PPO--enalty approximately solves a KL- constrained update, but penalizes the KL-divergence in the objective function instead of making hard constraint. The penalty coefficient is then automatically adapted over the course of the training so that it scales appropriately. PPO-Clip does not have a Kl.-divergence term in the objective and does not have constraint either. Instead, it relies on clipping operator in the objective function to remove incentives for the new policy to get too far away from the old policy. This use clipped surrogate objective function is not typical in optimization algorithms. In this work, we focus on PPO-Clip.
In contrast the empirical success of PPO-Clip, theoretical analysis of it is limited. Huang et al. (202I) and Yao et al. (2022) have made substantial attempts to discuss the convergence of PPO-Clip under the tabular and the over wverpaarameterized neural network settings. The basic idea is to categorize PPO-Clip as hinge loss problem classification. This analysis relies on the local convexity of the optimization objective, with which typical techniques for hinge loss apply. Liu et al. (2019)
"Corresponding to: Baoxiang Wang

---

Published as a conference paper at ICLR 2024
employed the classic proof based on the neural tangent kernel (NTK) condition to establish the global convergence of over-parann pa-aaattteeized neural networks. Their analysis is for a variant of PPO which no longer involves the ratio of the current policy and the old policy.
In fact, analyzing the original PPO-Clip involves many challenges. First, the algorithm uses the ratio of two stochastic policies, which is unique structure that has off-the-shed recursive inequality available to characterize it. Second, the clip operator is non-smooth, which necessitates the use of divide-and-conquer of probabilistic events. In comparison, treating the clip operation as a hinge loss does not provide more help when the problem is general and not locally convex. Third, the score function of the policy unbounded. typical example the commonly used neural network policy. Without bounded score function, strong conditions such as the gradient dominance lemma are no longer available, and existing techniques analysis of policy gradient will not apply. In addition, the unbounded score function could make the ratio of two policies arbitrarily large, even in the late stages of the optimization process.
In this work, we investigate the vanilla version of PPO-Clip with minimum set of conditions. We formulate PPO-Clip into a general two-scale iterative process, where the old policy Told is synchro- nized every T steps (T could be I, which makes it single-scale). We show that this process will converge to a stationary point of the learning objective, up to a bias term that depends only on the biases incurred by the Markovian sampling and the advantage estimation. When the biases zero, such as when we use Monte-Carle methods, PPO-Clip converges almost surely. We further provide the convergence rate of the gradient norm in terms of average- convergence and subsequence convergence. Technically, our analyses are based on key lemmas that characterize the probabili- ties of the clip events and thereof the recursion of the learning process. This new tool might share insights into analyzing PPO and might be of independent interest.
2
PRELIMINARIES
2.1 MARKOV DECISION PROCESS
We consider a canonical discounted Markov decision process (MDP) setting with finite state space and action space. In this work, an MDP is denoted as M =
where S and A are finite state and action spaces, P: : S x A A(S) denotes the transition kernel, r : S x A R the reward function, and 7 E (0, 1) is the discount factor. Let so be the initial state.
The goal of reinforcement learning is to learn a policy ı that takes s E and outputs dis- tribution <--(s) over the action space, to maximize the expected cumulative discounted reward. For every policy x, define the action value function Q" S A R as (s,
ET,P Ee 7 az)
s,00 = a Here, E.,,(-)) denotes the expectation under subse-
quent actions and states that follow π and P. When the context is clear we write this into Ex (-) instead. Define the state-value function V S - R as (=(s) == Eaar((1s) 12 (s, a)] and the ad- vantage function A : S x A + as (s, == QT (s, V*(s), respectively. this paper, we consider the case where the policy is parametrized by parameter 0 IRd. Define, as the objective function of our optimization,
V(0)= T (8o) ETT
Σ 2 Ar(sh, h==0
Denote V" == maXgeRd ((0) as global maximum of V(0).
2.2 POLICY GRADIENT
When both π and V are parametrized by 0, it is natural to consider gradient ascent methods to optimize the objective function, i.e., 0n+1.1 = 0n,1 ten VV 10,,,), where en > 0 is the learning rate
(step size). The gradient term (V) can be estimated via VV En V In Te, {als) 0 (8, a)
where Te is an estimation of the advantage function TH Williams 1992 Sutton & Barto 2018
The policy gradient (PG) estimation is unbiased whenever is an unbiased estimation of A, for example, the Monte-Carle estimation.
2

---

Published as a conference paper at ICLR 2024
Policy gradient have received a significant line of research and improvement, and together with the emergence of deep neural networks, receives empirical success in many areas (Konda & Tsitsiklis,
2000 Kakade 2001] Schulman et al. 2017 2015 Lilherap al. 2015 |Schulman al. (2177). In
theory, the global convergence of PG is investigated under the gradient dominance condition (Zhang
et al. 2020 2021 Yuan al. 2020: Xu et al. 2020 Huang al. 2020 Fazel et al. 2018 Fatkhullin
ct al. 2023) and soft-max policy condition (Agarwal et al. 2022 Mei et al. (020)). Stationary point
convergence of PG under general policies are proved by Zhang et al. (2020); Fazel al.] (2018).
Wang et al. (2019) study the convergence of neural policy gradient methods. While our setting is quite different PG, we obtain some insights from these lines of theoretical works.
2.3 PROXIMAL POLICY OPTIMIZATION WITH CLIPPED SURROGATE OBJECTIVE
The proximal policy optimization algorithm has different variants (Schulman et al. 2017 Huang al.] 2022, Hsu et al. 2020; Zhang et al. 2019 2022). In this work, we are particularly interested in investigating its variant with clipped surrogate objective which is introduced in the original PPO work. We denote this variant as PPO-Clip in our manuscript. The surrogate objective function
defined (0) Ereas
salals) A Amona (t.a)) where Tea denotes the policy previous [a]s]
iteration, A is an estimation of the advantage function A (als) Taa [a]s)
The surrogate objective function with
the clip operator is defined as LLLP Erre.
min
clip
Selall)
Aree (s,a)) The
T (a|s)
first term in the minimum operator is T.C.J, The second term, clip
matale)
A A (s, a), modifies
Te (a(s) ())
the surrogate objective by clipping the probability ratio the interval [1
50, 1 + 60]. for some
absolute constant dg. PPO-Clip takes the minimum of the clipped and unclipped objective and the final objective a lower bound on the unclipped objective.
In the original PPO work, PPO-Clip considers Tew to be the policy used in the immediate previous iteration. In the current iteration, the algorithm aims find a parameter that maximizes the clipped surrogate objective, using some estimation Exe.. of the expectation Exe. over the Markovian sam- pling. Namely,
Te (a(as)
To(als) Te.m (als)
Areey (e.0))
maximizepER
E ma
min
clip
(1)
Team
(als)
This maximization alone could involve multiple steps updates on 0, forming a double-loop struc- ture of the PPO algorithm. The off-policy nature of the surrogate objective specifically fits into this type of double-loop structure. In practice, such a structure implemented simply by running the optimizer multiple times (e.g., optimizer.step() in PyTorch) for each policy gradient update.
SETTING AND RESULTS
In this section, we aim to prove the global convergence of PPO-Clip to a stationary point. Namely, the gradient of the value function ((0) should converge to zero whenever the estimation bias is zero (Theorem 3.1 and 3.2). We further extend this result to the convergence rate of average-iteration convergence and subsequence convergence (Theorem 3.3
3.1 FORMULATION
Now we are ready to formulate the PPO-Clip variant that we investigate this paper. In this work, we specifically implement the optimization process within one step of policy gradient, namely Equa- tion (I). as T steps of gradient ascent. The algorithm therefore runs a two-scale manner, where the index of policy gradient iteration will be considered asymptotically large, and the index k of the maximization in (I) is considered to be within I...., T. for some small constant T., Notice that all results hold for T 1 as well. We formulate PPO-Clip below.
3

---

Published as a conference paper at ICLR 2024
PPO-Clip:
Brona min {
Mon (a]s) ( a (a/s) To. (a) 8) TB.. (a|s)
A (s.01)
kn, Brck + Cn,K Vonn
clip
(I: 1,2, ...,T
0n+1,1 = 0n,T',
where ETT,,, is estimate of Ero.., and An, (s,,a) a) is the estimated value of the advantage function.
As we have mentioned in the preliminary section PPO, the value of A and the expectation un- der Markovian sampling Exs,.. need to be estimated. Both the estimations can be obtained in an unbiased way using Monte-Carlo sampling, provided that the sampling goes through until the termi- nation of the episode. Practically, PPO rarely implemented such the actor-critic algorithm use biased estimations to trade for other properties such as sample complexity and generalization. Nevertheless, this biasedness can be formulated by the sampling bias term.
pure Monte-Carlo way. Methods
Define the functionals
Exp
Exprrp (neoro(-1sa) (xxAxo(s,a)); E (.) E Beo (ooo-- xxrr (1120)(-1o).aa))
86+1 Pa-leP(-
If Ang is bounded. for which will assume in Assumption 3.2 both of the functionals bounded linear functionals in x functional. The norm of this functional then exists and we denote the norm as |[ )
Therefore, (E,, E (+) also a bounded linear
max [IE2, , [f) where || := (a (f(ssa)), Subsequently, denote the up- per bound of the norm of this functional as
[ E.,, S on.
(3)
When Monte-Carlo sampling is used estimate the expectation É of the Markovian sampling and the advantage function A, the term Φn, = In this work, we will provide general results that depend on the value of én-
To facilitate the analysis, define the filtraation Fi (01.1), F, (8 En-1) (n 2), where the random variable {5n) represents the stochasticity in the sampling of E and the estimation in step n of the outer iterate.
3.2 ASSUMPTIONS
We first present the assumption regarding the policy parameterization of тhe (s(a). This assumption can be realized by setting the policy to be within certain function classes such neural networks. Assumption 3.1. There constant L, such that for any 8, a, the policy no (als) L--mooth regarding 0, i.e., [[Ve,, (als) VTe, (als)] < L||0 02ll A 01, O2.
In comparison, a stronger assumption was used in some previous works on policy gradient (Zhang ct al. 2020 1021 Yuan et al. 2020 Xu et al. 2020; Huang et al. 2020). In the assumption, the score function V In ( a)) has to be bounded and Lipschitz continuous, i.e., |/V In(TT, (sla))
Mj 00 0zll. for all s, a, for some Mj. This assumption does not hold for neural networks in general. As counterexample, let the action space be {0, }} and let 0 = (z, y).
V Ii( (s(a))
x / + --==" v*), Na (1[s) 1/(1+ -e-= Us), (V s). When ||Ø]] + +oo, is evident that V In (al (a|s)) +o in this case. Nevertheless, our conditions in Assumption
Let N2.y (0
3.1 hold under this example.
Our analyses also need a uniform upper bound for the reward function for each state and action. This condition very commonly used in the field reinforcement learning (Zhang et al.. 2021 2020
Yuan et al. 2020 Xu et al. 2020 Agarwal al. (021).
Assumption 3.2. There exists an upper bound f > O, such that for any s, a, the reward bounded by |r(8, a)l <

---

Published as a conference paper at ICLR 2024
It follows immediately this assumption that (s, a)l 2 Rmax 2f/(1 - -)) In this manuscript, whenever A is estimated (denoted as A), truncate the value of that estimate such that [s, a) S Rmux = 2F/(1 7)
Now describe the conditions of the learning rate of PPO-Clip. In this work, require the learning rate to satisfy the Robbins-Monro condition, in order to prove the global convergence of PPO-Clip stationary point. The condition agrees with many commonly used learning rates of PPO in practice.
Assumption 3.3. The learning rate Σ
satisfies
ExS
Ck=l En,k
+00,
T-
k== n.k < +00.
In the case we only need to prove subsequence convergence or average-iterate mean-square Conver- gence, Assumption can be relaxed into Assumption 3.4
Assumption 3.4. T-1 Gn,k 0.
The learning rale {co,k -oo. satisfies
Σte Ek= En.k +00,
Under Assumption 3.3 we can use the classical martingale method to prove convergence, which yield strong last-iterate convergence. However, this approach is not applicable to the under Assumption 3.4 Under Assumption [.4] we are unable to construct sup-martingale by the value of (O,)). Instead, we will need to decompose { (VV(),) into multiple sub coob-prcesses using first entrance times. Then a respective sup-martingale was constructed on each sub-process, and the compounding effect of these sub- cro--prcesses analyzed.
Assumptions on the learning rate can of course be satisfied by our choice of the learning rate. We remark that both the assumptions host wide range of learning rate choices used in practice in reinforcement learning.
3.3 RESULTS
In investigating the convergence of stochastic optimization, it is common construct recursive inequality for the optimization objective. Taking policy gradient as an example, one may construct recursive inequality using Taylor's expansion as
Le
E (V"V-VVV - ((On+t)(F.)) & V -V(0)) - - VVVO. + 2 (VV().)
where en the learning rate, VV estimation of VV. V* the maximum value of the value function (0), and C represents the Lipschitz coefficient of (V()).
Due to the difference between the policy gradient and PPO-Clip. the update rule is by direct estimate of the policy gradient VV(#). In this case, to obtain a recursive inequality similar the above one, we estimate the error between the PPO-Clip update target and VV The aim is to bound this еттог term and mitigate its effect in the following analyses. However, it is not straightforward to estimate this error as one needs to estimate (o(als)/ Ta (als), which was not available in the literature to the best of our knowledge. From this perspective, the analysis of PPO-Clip might be more challenging compared to the analysis of policy gradient.
distribution of the random variable
To be specific, our idea is to prove that the event B = (Ne (al s) me (als) E 10,1 1811-60 66) (1 5n, +20) is a subset of the event C == (o (a|s) S ko), for some scale kn- Then we relax the set of (s, a) pairs that satisfy the clip condition to those that satisfy the event C. Subsequently,
conditionally under the event C. holds, we can estimate the emro terms (denoted as X and Y in the analysis) accurately. This idea generates the following lemma.
5

---

Published as a conference paper at ICLR 2024
Lemma 3.1. If Assumption 3.1 and 3.2 hold, then for the sequence ( n formed by PPO- Clip 2 there is
V - -V(0++1,)) -V-- (V--V(0,,))
Σ Co.k
VV(0m,1)11
1-1
+ 4v 1 + d0 R?
LAAC
Σ
kn
Σ
En+1,8
[VV.)11)
mux
T-1
+ 2/ 22/I
on +
Σ
En,k
Σ
6n,k
+ Sa.
k=1
k=1
where
C AA L 71+ 12L
(1 7)-
(1-7)3
and Fn+1) is martingale-diffreete sequence.
This lemma is our key lemma. It derives recursive inequity that is similar to that of policy gradient. We now provide the insights into finding this lemma, while deferring the full proof to the appendix.
Step 1: We first aim simplify the gradient in PPO using divide-and approach. Let
Bo,k == {1-
1- < T0 (als) No... (a|s)
1 + 60
be the set of (s, a) pairs that do not satisfy the clip operation. We obtain the following equations.
Ton. i (a|s) nyo, (als)
To.. x (als)
V Bn.k
Emenn.
min
clip
A (8,
Ta.. (a|s)
-Eren..
Amon, (0,a)
Blaia) nip
To.. (a|s)
VTO., (als)
Amon, (0,10)
A (8,
Ton.. (als)
Ton. (a|s)
VTO.. (a|s) ViOo., (a|s)
Emo..
-Axon, (8.a0
TB..
Vioo. (a|s)
Ano,, (s.1a) + Xn, Yn.k.
: EExe,,
T&.... (als)
Here, Xn, and Yn,k are defined as the last two terms, i.e. the error terms, on the right-hand side of the second equation.
Note that the first term of the last equation is the policy gradient of the value function at 6,,1, which can be handled effectively. Therefore, we aim bound X,, and Y,, to ensure that the difference between the gradient of PPO and the policy gradient of the value function at 0,,1 is not significant.
Step 2:
For Xn,k, we observe Q./ Br.k. C Cl/Cn, (see appendix for its proof), where
Cn, = { TTe.. (a|s) (V2L Ao)
22L(1 6o)
| in (6.00))
Σ En- Bre,
no 2.1 (a|s)
and Ag = min
22 As such, the indicator function 1 B,,, can be bounded by Ic,, By further noticing the T-smooth condition, have / (a|s)]l S 2 (a|s) (see
/2L -Im(1-pp) b
[m[1+dp]

---

Published as a conference paper at ICLR 2024
Lemma A.1 for proof), which leads to E (IIxm & 2L 22 1+ d0 R max LA|
(als) (v=))))(a5))
Σ End
Bron-t
1
VIA Rmax
where Ryax :- f-(1111)) is defined as an upper bound on the value function.
For Yn, our main objective is to bound ViT... (als) Vire... (als). According to the L-smooth condition, we have To,, (als) - VTo., (a|s)]l & L| On,1 It therefore suffices to control [|0,, 1n In fact, we could obtain
22 + 06)
| (6)10)
|0 On,kll S en
VT... (a|s)
and the proof of this inequality is presented in Lemma
A.2
The bound of Y, is then
k
E (I (I Fi-1) I 2V/2111+ + Rmax
LAl
Σ
Cn
Exe,
(VTn._.((as)) (als)
+ vIA Rmax
Step In this step, we construct a recursive equation for the value function V(O,, Using Taylor's expansion, we obtain
V v(00+1.1) (V* (V+-V(0n..)
(V (=x4t11)((@01111111111 - 0n.1)
+ (VV(On.)) -0 -110 1n.1l1
VV(On-1)) ((0n=))))((0411111)) (0.+1,1- On.1)
(I-?)² (1-7)3 represents the Lipschitz coefficient of (0). By substi- tuting the expressions for X,,k and n, from the previous step into the equation, there
where L := AlRank (1+7) Allann VLL
V V(6m+1,1) (V* V((Bn.))
Σ En,k
[V
T-1
+ 4/ + 60.2 max
VAIC
Σ En,k
En
(0.111
2/ On +
Σ
Cn,k
Armed with the lemma, we are now ready to discuss the convergence of the PPO-Clip algorithm. Our first theorem asserts its global convergence to stationary point, up to the estimation bias on involved the Markovian sampling and the advantage estimation.
The idea of the proof is to discuss the magnitude of VV Let d' == on +0 kn.))
A/AA,,, the fixed bias term 2 A//AA (Et-) (n,)) can be immediately offset by (ΣI T-1 k= cn,k) (0n-1)11 When (Om....) I CA the desired result is
When || (VV(O...)))
straightforwardly satisfied.
Theorem 3.1. If Assumption
3.1
3.2
3.4 hold, then for the sequence {0n,k) tt formed by
+oc.T
PPO-Clip (2) there is
lim VV 8L simsup On a.s.. +00
n-0000

---

Published as a conference paper at ICLR 2024
Proof. Let 0/ On 0 OEEkl Cn,k).
We construct the following events.
A == ((. max A sup 03), [| 8VIA
sup o1 12+1
... (..(IIVV(O..))) max sup
0}},
121
A., (IIVV max(8
sup o 5}}
12:
A, = ( max Al
sup 8}} (0 8V A
sup %1)
2211
(I.V( max
m (8v A
sup o J....
12n
where := and ā is positive constant. Asymptotically, we can obtain the existence of No such that for v > No. there is
C En+1,
E
> VIAlc
Σ En,
Σ Entl.k
VV
Note that here No only depends the parameters 8, Rmax, |A), L and the learning rate 6n itself. Then through Equation (14) the proof Lemma 3.1 when > To, n 2 i, X E Fi, there is
IxnAi. ((n+111)) 1xrA.. ((n,, 1
Σ
6n.)) IxnA:., (0 Lx Sn-
Notice LxnAi, ((0+11,1) Ax ((nn+11). Then,
IxnAiit V(0n+1.1) - IxnA,.. ((Ona)) S
1 4
Σ G.,k) IxnA,.. ( 11
1xnA, Su-
By summing up the above expression from i to L, we obtain
Gn,k E(IxnAA,)) S t00.
Combining the condition Dim. 1 kn.k) = +00, and the fact the sequence [E(1xnA...)) t has limit, have
lim E(IxnA,.) 0,
n ++00
which implies E(1xnA
= 0. According to the arbitrariness of 6, and X, we get lim in (V SLV/Al lim On a.s..
This theorem essentially provides us the guarantee that PPO-Clip makes the gradient norm of the value function subsequence converge almost surely to zero within an Ollim dn) neigh- borhood. In practice, the size of this neighborhood is determined by the advantage network. The more accurate is the advantage function estimation, the smaller is this neighborhood. When the advantage function is estimated unbiasedly, like through Monte-Carlo sampling, the convergence is exact.
In fact, when én, 0, we further improve this result by replacing Assumption 3.4 with Assumption 3.3 which is strictly weaker.
Theorem 3.2. If Assumption 3.7
3.2 3.3 hold and there exists at > O, such that on = 0 (V t),
then for the sequence (pn.k formed by PPO-Clip 2 there
lim
[V,(l a.s., and lim E (O)VV +0000
n +00
8

---

Published as a conference paper at ICLR 2024
This theorem indicates that PPO-Clip has the same convergence property as the best current results available for policy gradient (Zhang et al. 2020). The proof of this theorem is by Lemma [31] and classic martingale methods in stochastic approximation (Robbins & Monro I95I). We defer the proof to Appendix A.3
Now we proceed to present the results regarding the convergence rate. Unlike existing research on the convergence of the policy gradient algorithm which acquired the last-iterate convergence rate [Zhang et al.) 22000] 2021; Yuan et al.) 2020; Xu al., 2020; [uuan al.] (020), i.e., (O,)
O(-,, we provide the convergence rate in terms of the average iterate, i.e.,
1/(2m=1 T k= ER) E E
We remark that this difference is due to the conditions of the analysis. In general, for stochastic gradient descent algorithms, in order provide convergence rate in terms of the last-iterate sense, it is necessary to establish a local or global inequality that connects VV and V. In the study of SGD- type algorithms, examples of such inequalities are P-L conditions and K-L conditions. In the study of the PG algorithm, assumptions like bounded score functions, Fisher non-degenerate assumption, and transfer error bounded conditions are often used, which then imply the gradient dominance condition, i.e., (0)) 2 V (0) + Ebias This condition is similar to the K-L condition. As we have not imposed any such additional conditions, the convergence rate in terms of the average-iterate convergence is the best we could have obtained.
Theorem 3.3. If Asssupption 3./
3.2 3.4
+ hold, then for the sequence {fn, } 1.km formed by PPO
2 there is
n
min m= 2
[BIIV((0.)))))=
0
+0 lim sup sup dn) n
=1=1 Ex
and
Em E Σ & VV
-1=1 &
+ 0 lim supep) 2
t=
Ez
where E, = Ek=l C1,
Proof. By Lemma 3.1 we have
T Σ En,k
R(V* V(On+1.)) B(V - V(On,))) -
E (0
k=1
T-1
2
4. 1 1 do Ro
VIALC
Σ
E V
Em,k
T-1
En (
On + (
T-1
+ 2 A Σ En (
Σ
Grikk
km
where E, = -ET=1 tr, Then, obtain
Σi= Σ 2&EDV = of
E +0
min m= 1.2......
E [EIVV
+
lim lim
E &
The convergence rate under the notion of best-iterate convergence and average- itteate convergence follow, respectively.
Our analyses might provide some insights that PPO could be more efficient than PG, in some
scenarios. To compare them, we set the learning rate in PPO-Clip as 6n. En (V 1 < k
S
T), and the learning rate in PG as en- The convergence rate of PPO-Clip is O
TE 1
+
Oliimsup &n). while the convergence rate of PG is O شي + O(lim sup Φn).
One could observe that both the convergence rates are in the same order, up a factor of T, which the number of steps of off policy updates using Vold- In this way, while PG requires continuous sampling to obtain policy data, PPO-Clip use sample for T steps. Improving the sample efficiency is critical in many reinforcement learning tasks, and when sampling is the bottleneck, PPO-Clip could have an edge over PG.

---

Published as a conference paper at ICLR 2024
4 CONCLUSION AND FUTURE WORKS
We investigate PPO-Clip, one of the most popular RL. algorithms yet to be fully understood in theory. We show that PPO-Clip converges stationary points up a bias term, and when the biases in Markovian sampling and in advantage estimation are zero this bias term is also zero. We further proved the convergence rate of average-iterate convergence and subsequence convergence.
We used minimum set of conditions in our analysis. The conditions on step sizes and policy parametrization could be satisfied by the choices of step sizes and policy class. The condition on bounded reward function is rather common and mild in RI. analysis. The relaxation of conditions incurs several technical challenges to overcome, which motivates us to introduce our key lemma, Lemma 3.1 Our lemma derives the recursive property of the optimization process under the pres- ence the clipped ratio two stochastic policies, and could be of independent interest for analysis on other optimization algorithms.
remains unknown to us the optimality of PPO-Clip could deduced under the current con- ditions, but it would be important for future works to further characterize the optimality of the stationary points that PPO-Clip converges to, Another direction is to investigate other conditions on the policy, such that it is general enough include neural networks but provides improved results on the convergence and the optimality.
ACKNOWLEDGEMMEN
Ruinan Jin and Baoxiang Wang are partially supported by National Natural Science Founda- tion of China (62106213, 72150002, 72394361) and Shenzhen Science and Technology Pro- gram (RCBS2021066) 003556603 (CYJ202103241200 20011032). The work is partially supported by Guangdong Provincial Key Laboratory of Big Data Computing of The Chinese University of Hong Kong, Shenzhen.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):431-4506 2021.
Ilyas Fatkhullin, Anas Barakat, Anastasia Kireeva, and Niao He. Stochastic policy gradient
methods: Improved sample complexity for fisher-non-deennerate policies. arXiv preprint arXiv:2302.01734, 2023.
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradi- ent methods for the linear quadratie regulator. International Conference on Machine Learning. pp. 1467-1476. PMLR, 2018.
Chloe Ching Yun Hsu, Celestine Gendler-Düinnee and Moritz Hardt. Revisiting design choices in proximal policy optimization. arXiv preprint urXiv22099.08997 2020.
Feihu Huang, Shangqian Gao, Jian Pei, and Heng Huang. Momentum- policy gradient meth- ods. In International Conference Machine Learning, pp. 4422-4433, PMLR, 2020.
Nai-Chieh Huang, Ping-Chus Hsich, Kuo-Hao Ho, Yu Yao, Kai-Chun Hu, Liang-Chun Ouyang, and I Wu. Neural ppo-clip attains global optimality: A hinge loss perspective. arXiv preprint Xiv: 13.19799 2021.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022. URL https:///ccrr-/ boolgrr-llo--- track. wwt.gggihhhbhi..oooaoo ppo imp ieteenttationndeetile
Ruinan Jin, Yu Xing, and Xingkang He. On the convergence of mSGD and AdaGrad for stochastic optimization. In International Conference on Learning Representations, 2022.
Sham M Kakade. A natural policy gradient. Advances in Neural Information Processing Systems, 14, 2001.
Vijay Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, pp. 1008-10144 2000.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: arXiv:1509.00 ptt.:tt099..997 2015.
Boyi Liu, Qi Cai. Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimiza- tion attains globally optimal policy. Advances in Neural Information Processing Systems, 32: 10565-10576. 2019.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. International Conference on Machine Learning, pp- 6820-6829. PMLR, 2020.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemal- ical Statistics, pp. 400-407, 1951.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Lingxiao Wang. Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv: 0tt.1/9999...... 2019.

---

Published as a conference paper at ICLR 2024
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 8(3-4):229-2 256, 1992.
Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance- reduced policy gradient. In Uncertainty in Artificial Intelligence, pp. 541-551. PMLR, 2020.
Hsuan-Yo Yu Yao, Ping- Lin--Chun Hsieh, Kuo-Hao Ho, Kai-Chun Hu, Liang Chun Ouyang, and I-Chen Wu. Hinge policy optimization: Rethinking policy improvement and reinterpreting PPO, 2022. URL /t..hcpps///geeeeeveee.... net.eeww..eeeeeeeeeeeeeeeeeeeeee forum? 1d=gex 2G2bLch
Huizhuo Yuan, Xiangru Lian, Ji Liu, and Yuren Zhou. Stochastic recursive momentum for policy gradient methods. arXiv preprint arXiv:2003.0433 2020.
Junwei Zhang, Zhenghao Zhang, Shuai Han, and Shuai Lü. Proximal policy optimization via hanced exploration efficiency. Information Sciences, 609:750-765, 2022.
Junyu Zhang, Chengzhuo Ni, Csaba Szepesvari, Mengdi Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems, 34:2228-2222 22240, 2021.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6): 3586-3618 2020.
Zhenyu Zhang, Xiangfeng Luo, Tong Liu, Shaorong Xie, Jianshu Wang, Wei Wang, Yang Li, and Yan Peng. Proximal policy optimization with mixed distributed training. In International Con- ference on Tools with Artificial Intelligence, pp. 1452-1455 IEEE, 2019.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 AUXILIARY LEMMAS
Lemma A.1. If Assumption 3.7 holds, we have
[ S V2L min( 1 - TT(aas)) (As, a).
Proof. By applying Lemma 10 of Jnn et al. (202)), we get
[Tre(als]] - 22((Te
inf Tor (als)) (L d'ERd
(4)
Then we construct function ((e.a) (0) : - (a(as)) By its definition have ((a.a)
VTa(als). By Assumption 3.1 we have for any 01, 02. such that
( (01) (((a.)) (02) |o, (a|s) Te, [a|s)) < L 0211-
Similarly, obtain
( (2f((.0) inf f(())((0)) & 2L
d'eRa
that is
[T((| S 221(1- - Ta(a)))).
(5)
Combining Equation and Equation (5) yields the desired inequality.
Lemma A.2. If Assumption 3. holds, then for the sequence {0n, +oo. formed by PPO there is V n 0, 04
T,
V2L +60 60)
|10,K41 - 0n S Enorn
/To.. (a|s)
Proof. By the update process PPO in Equation
2 we have
Amen. (8,a) ))
Te (a|s)
|0 - 0n = En
Ven.k
Bsten.
clip
TT... (als)
Define events
TB.. (als)
Brnk 1 50
< 1 80
TB (a|s)
and its indicator function (( Then.
VTo.. (als)
A (osc1)
En,k Brow.
1 Leltn
To..: (a|s)
S en Bnon (ggng) [V (a|s)]
| (0.01))
Ten. (a|s)
By Lemma A.I
VTT... (a|s)] S 2T
This implies
122(1+ (1(11++00 00) |Ame., (
10 0n, S Cn,KEEToo.
v VToo, (als)
The lemma follows.
13

---

Published as a conference paper at ICLR 2024
A.2
PROOF OF LEMMA 3.1
We continue to use the events Blaa) and the indicator function 1 Ble defined in the proof Using these notations, we rewrite the gradient term in PPO as
Proof.
of Lemma
A.2
To., (a|s) Ton. (als)
Amen. (s.21) Erren.
VTon.((als) Amee. (s)a)
V
Bxe...
clip
1 B mik
Ton. (a|s)
B
V VTo.,, (c(o)
Amenn (s,a)) - -Bron- ('aynkiee
VTO.. (als)
AtOn, (8,
Ton.i (a(s)
Ton. (als)
VTOo,, (als) - ViTo., (a|s)
Exxn.
Amena ()
IT... (a|s)
Viee..: (als)
Amon. (s,a) + Xn. Yn.k.
= Ese...
To... (als)
Inspecting the norm of the term Xn in Equation
yields
||,,kKl =
Biee..
10 T (als)
ViTe..: (a|s)
Amon. (s.00)
(8)
Anye
VTT (a|s)]]
| (0,00)
To.. (a|s)
Now aim to estimate the indicator ln/nbtee
Through the mean value theorem and Lemma A.1
have
To. (a|s)
a (a[s) Nun. (a|s) = exp {
{~a
V (Vome
Ton, (a/s) (als)
T (0n, - omm)
6
(als)
Tin (a|s)
|V Dn.k Ton. (a/s)]]
IV Inn To.. (a|s)] (amneal)))]
110 0msll
E exp
110 0mill exp
exp
D0n, (a|s)
To (a(s)
V2T.
/T.
Bmall)
C
cxp
110m, emall exp
110n,k
V T8 (als)
T (als)
(9)
where 6y, is a point between On, and 8n,1 We construct another event
n, == =
{
(als) To..
(V2T+A))
k-1 Emn-1.
BTT..-. )
(22/1 ( + 60) T (a|s) A (0. (0000))
where À > is a coefficient to be determined. It is worth noting that the event (1.00) riele Fa-1, which is used in the subsequent proof. Then,
belongs to the n.
Tā.,, (als)
V mo.,, (a|s) + V T0 (als) V To.,, (a|s)
V To.. (als)
// To.,, (a|s)
To... (als)
By inspecting the function / (a(ss) we obtain
[[ao(a|))]]
V (a(s)
< V2T.
VTo(als)
That means
|V To. (als) - To,, (als) < 5/2L 1l0 0m.ill S V2L 1 0n.all-
14

---

Published as a conference paper at ICLR 2024
Therefore, whenever ((s.a) happens, through Lemma A.2 there
V " (als) 2 T0.,1 (als)
V2L. 110n,t On,1l
(V2L + 2)
Σ
k-1
Cn-18
E v TT (a|s)
V2I 0o)
A 1.1 (8, -V2 |6,,,1 8n-11
i=1
(V2L A) 110n,
1n.1ll
V2L 110
0n-
A - -|0,,--0,, 1-
we have that whenever Cla.e)
Substituting the above inequity into Equation
n, happens, there is
T8 (a|s)
E (e
.2
e
n. (a|s)
Now we choose
V2L
V2L
x=
min
1Inn1 - -00)
In(1 + 60)
which implies
To.. (als)
vaL
C Su; +00. e
To. (a|s)
which then implies that B (8)a) 0/ 1/C(e1)) nih,0o tny 1//o.um) Substituting into Equation
(s.) n,
(xx)) have Q2 1/Blyx
happens. As
8
yields
( n.kido
VTo,,, |Amen. (0.00)).
|& S Exo...
To... (als)
We will now calculate the conditional expectation of ||,, on Fn-1- Noting 1nolee) E F
have
E (I S EE
Erenn
Payckin
VToo., (a|s)]
Amenn (o.al) Fr-1
TB..1 TB...(a)1)) (als)
Brenn
Payosee
V To... (als)]
|A (0.00)
- vlAl A.AlRmaathe Rmaxthn.
T0n, (a|s)
Noting
I VTo... (a)s)]]
Esena
Payoluer nikino
A (s.a))
TT. (a|s)
22L Bronn
|mmen (s.a))
VToon
2L EsNsyon
1 Paaoters
V Ton, (als)
nitinn
1
50 R2 IA| $ V2L(V2L++ Ao)
Cnii Bron-ty
T (a|s)
we get
E E Bro..
| Vag.. (a|s)ll
A (s,00)
F Fo-
Ton, (als)
(10)
x V2T(V2L 100 55 R3 AAl
Σ Cni B T (a|s)
+ V A..maxx
15

---

Published as a conference paper at ICLR 2024
Now we inspect the norm of the term Y, in Equation
We have
Vi (a|s) V (als)
Anon. (s.01)
|n =
E Exop..
1 Blaee
1 (a|s)
V (a/s)
VTO... (a//))
S Exon.
| (s.aa))
TBn..1 (a|s)
< Emon.
Σ 110, 0mlll)
-lAkom. (sva)))
To (a|s)
1=1
k-1
V 12 + +00)
< R L
Σ
Cnit
E Exog-111
-Bron TBnm (a|s)
11.-1111188
1=1
Because
v 21 + td)
Exe,
Fn
VTO,. Ton, 11 (a|s)
have
E (1| Fa-1) <I.//111 21(1 66) Pi R |Al
Σ En.
Bren-na
VT (a|s)
(11)
+ v | Rmax
By the Lipschitz continuity of Vx, for any 01, O2 E R², there is
A Rmux L
(1 7) V2L
(VV - VV -VV())
-|0 B2ll.
7)-
713
Rmn For notational convenience, assign above coefficient as C = (1-2)"
Then
(1-7)*
the discrete difference V - V(O++1.1) (V' V(On,1)) 1(,,,,) can be expanded as
V V(On+1,1) - (V* - V(On,))) < (V. (0n+1,1 - 0n,1) + 2110m+1,1 - 0n,111
(0(0 - 0n.1) + (VV()..)) -0000...-VV000000 (O, (0n41.1- On.1)
(12)
+ +110n+1,11- - 11.1
We substitute Equation and Equation
into Equation (12) and obtain
T-1
ViTe.. (als) Ton,, (a|s)
((Ont1a) - ((On.)) - -
Σ
Emck
À A (8,a)
1=1
T-1 Σ
T-1
(13)
VV
Gn.kXnnk (V(
Cn
+ Cl0,,. - 6 1 1..11-11111..111 0n,2 || + -1100++11 - 1n,1
Then substituting Equation (10) and Equation
into Equation
(33 will acquire
V" ((6+411) (V* v(On,))) S
Co.k
(V(
k=1
4/1 + 50 R2
R VIALC
Cn.k
1
(0.,)).
(14)
T-1
(
2 C On +0
En,k
En,k
km
The lemma follows.
16

---

Published as a conference paper at ICLR 2024
A.3
PROOF OF THBOREM 3.2
Proof. Through Lemma 3.1 we know n > 1, there is
E(V ((0++11)) - E((V - v(On,1)))
Σ Cr.k
E
[V(
T-1
T-1
+ 4/ + 50 R
VAACC
Σ
Cn,k
Σ
(1
(15)
k=1
+ +2 O
Σ
n,
R=1
By inspecting the asymptotic order, conclude the existence of an No such that for n > No. have
Σ 1 BI 8 VIALC
Σ En
( en+ BIIV((M.)))
Note that here Nú depends only on the parameters 6o, Remax, |A], L and the learning rate 6nk--
Substituting the above inequity into (5). we know that when > ig max[No, f), there is
B(V* - v(On+1.1) - B((V* V(O,,))))
Σ Cn R=1
VV
(16)
+ +2 O
Σ
2
Summing Equation 16) from in I, and noticing the condition Σ 2k=1 2 +00, we obtain
B/ < +00, 6n,k
which implies
Engk IVV(O.,))1 400 a.s..
(17)
nsioo
The condition +oc T-1 (n)) too implies lim inminf (V = a.s., which
means that for any > O, there exists a subsequence {
+200 such that (o < 8. We
find all the boundary points in the sequence {4a +000 and denote them as {pm)+t) They satisfy
VV S 6, [VV((pa--t1..)
(0VV
> 6, (0 S 8. If
{pn +
) contains at most finitely many elements, there exists an no > 0, such that for any n > nop, (VVl < fp has infinitely many elements, we find
sup Spak
(V S
Σ IIVV SC
Σ
10m+1,1-0m.j
By substituting the expression PPO-Clip into the above inequality, and combining Lemma obtain
Pas
IV S 2/22/AL1(11 5o)
Σ
sup P2k <n<
Gn
R=Pak-
C. v 2| do)
Dak
Σ
> [VV(0.,)))
[Cnr kn
8
R=P3A-1
k=
17

---

Published as a conference paper at ICLR 2024
According to Equation [7p. there exists n1
O, such that for any P2k-1 > n1.
C 2| + 60)
Peu-
Σ
[VV((on.))1
<.
En
P=
Subsequently, for any n > max (io, ) there is
P2n 111 (O.)) & I (0
sup P2e
|VVV(0n..)) < <6,
S
which indicates that for any n max nu, n1), there IV < 26. By the arbitrariness of 6, we have
lim +-+
[VV = 0 a.s..
Therefore, for any 1,,1, have (O., )l < Re Reax 21 do). Through the Lebesgue's dominated convergence theorem, we obtain
lim E (V 1+0
as we desired.
18


Reviews:
[{"review_id": "kGLC1umaoj", "reviewer": "Reviewer_dSqC", "summary": "This paper presents theoretical analysis of convergence of PPO algorithm using clipping and version of two-time scale method, i.e., updating policy parameter with particular period. Analysis of PPO-clip is difficult due to non-smoothness of clipping operator and involves ratio of policies. Theorem 3.3 provides best iterate and averaged iterate convergence in terms of $||\\nabla V(\\theta_{n,1})||\\to 0$.", "questions": "- Above the paragraph of Step 1, what does it mean to have similar recursive inequality like policy gradient? Please provide more details.\n\n- In Step 1, how does the estimated clipped PPO loss have gradient? Should it be sub-gradient due to the non-smoothness of the clipping operator? Please provide more clarifications.\n\n- What is the intuitive meaning of $X_{n,k}$ and $Y_{n,k}$? The motivation of decomposition of error term  into $X_{n,k}$ and $Y_{n,k}$ is not really clear\n\n- The introduction of $C_{n,k}$ in Step 2 is quite abrupt. Please provide more details.\n\n- In Step 2, what is the meaning of bound of $\\mathbb{E}[||X_k|| \\mathcal{F} ]$? Depending on $\\frac{1}{\\sqrt{\\pi_{\\theta_{n-1,1}}(s,a)}}$ seems to be problematic. How is this term compensated? In deriving (14) where did $\\frac{1}{\\sqrt{\\pi_{\\theta_{n-1,1}(a\\mid s)}}}$ go?\n\n- I think there is typo in (13). $\\nabla V(\\theta_{n-1})$ should be $\\nabla V(\\theta_{n-1,1})$.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles important question regrading theoretical analysis of practical implementation of PPO algorithm. The aim of the paper is clear and simple. \n\n- The analysis seems to be novel. The authors use two-time scale method to overcome the difficulty to deal with ratio of the policy and constructing particular set of events enables to derive recursive inequalities to bound the norm of gradient of the clipped loss function.", "weaknesses": "- Even though the aim of this paper is to tackle theoretical analysis of practical implementation of PPO, still there is some gap. Using a inner and outer loop style update seems to be far from practical implementation.\n-  The estimate for advantage function, $\\phi_n$ can be estimated with parameterized value network (e.g. neural network). However, I believe extending the proof to Actor-Critic setting is non-trivial. Hence, I believe the proof is setting restricted to Monte-Carlo setting. \n- Assumption 3.4. restricts the generality of $T$ and the learning rate. Providing simple examples on the learning rate, and $T$ would be helpful to understand the conditions about Assumption 3.4. For example, can we use $\\frac{1}{k}$ or $\\frac{1}{\\sqrt{k}}$ as the step-size?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrSfVW8mB1", "reviewer": "Reviewer_wVXF", "summary": "This work takes a closer look at the theories behind the clipped surrogate objective of PPO (Proximal Policy Optimization). The authors provide a comprehensive analysis that proves the stationary point convergence of PPO-Clip and demonstrates the convergence rate.", "questions": "> the unbounded score function makes the ratio of two policies arbitrarily large, even in the late stages of the optimization process.\nDo the authors mean the ratio **could** become arbitrarily large?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The work is novel in that it investigates the theoretical convergence of PPO, whereas the field has overwhelmingly focused on the empirical performance and application of PPO (e.g., Dota 2 with PPO and ChatGPT, RLFH with PPO). This is partially because the clip operation is inherently non-smooth, thus posing challenges to empirical analysis. The authors' analysis seems sound and comprehensive; they also clearly listed out the necessary list of assumptions. The authors' Theorem 3.2. indicates \"PPO-Clip has the same convergence property as the best current results available for policy gradient\", which seems significant.", "weaknesses": "Please take this with a grain of salt, as I have primarily been using PPO under empirical settings. I struggle to understand how this work connects with the wider research community. What is the implication of this work? This work demonstrates PPO has stationary point convergence — can this property be used in some ways?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWVwGBYv5S", "reviewer": "Reviewer_v9Uz", "summary": "The paper provides a theoretical analysis of the convergence of a variant of the popular PPO algorithm where the objective function is clipped.", "questions": "Can you think of a possible impact your result can have on the field of representation learning?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides formal guarantees for the convergence of the clipping variant of PPO that has been used widely by reinforcement learning practitioners. This probably adds some reassurance that that this is a sound RL method to use.", "weaknesses": "The paper appears to be a good contribution to the filed of RL, but has little, if anything, to do with representation learning. ICLR'24 might not be the best venue for this paper.\n\nSome minor typos:\nLast paragraph on page 1: \"this analysis rely\" -> \"this analysis relies\"\nSame place: \"no longer involve\" -> \"no longer involves\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)