Partial Advantage Estimator for Proximal Policy Optimization
Authors: Xiulei Song
Yizhao Jin
Gregory Slabaugh
Simon Lucas
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
value estimator

Abstract: Estimation of value in policy gradient methods is a fundamental problem. Generalized Advantage Estimation (GAE) is an exponentially-weighted estimator of an advantage function similar to TD($\lambda$). It substantially reduces the variance of policy gradient estimates at the expense of bias. In practical applications, a truncated GAE is used due to the incompleteness of the trajectory, which results in a large bias during estimation. To address this challenge, instead of using the all truncated GAE, we propose to take a part of the calculated GAE for updates, which significantly reduces the bias due to the incomplete trajectory.  We perform experiments in MuJoCo and $\mu$RTS to investigate the effect of different partial coefficient and sampling lengths. We show that our partial GAE approach yields better empirical results in both environments.

Under review as a conference paper at ICLR 2023
PARTIAL ADVANTAGE ESTIMATOR FOR PROXIMAL POL- ICY OPTIMIZATION
Anonymous authors Paper under double-bling review
ABSTRACT
Estimation of value in policy gradient methods a fundamental problem. Gen- eralized Advantage Estimation (GAE) is exponentially an advantage function similar to X-retum. substantially reduces the variance of policy gradient estimates at the expense of bias. In practical applications, a truncated GAE is used due to the incompleteness of the trajectory, which results in large bias during estimation. To address this challenge, instead of using the entire truncated GAE, we propose to take part of it when calculating updates, which significantly reduces the bias resulting from the incomplete trajectory. We perform experiments in MuJoCo and μRTS to investigate the effect of different partial coefficient and sampling lengths. We show that our partial GAE approach yields better empirical results in both environments.
estimator of
INTRODUCTION
In reinforcement learning, agent judges its state according to its environment, then selects an ac- tion, and iterates these steps, constantly learning from the environment. Let S be the set of states and A the set of actions. The selection process is quantified by the transition probability P. When the agent makes the choice to execute an action, it will receive feedback rewards from the environment, that is, r S x A [Rmin: Rmar]] In this way, the model will produce trajectory sequence т, that is T (81, ...,..., 8T, ar). This trajectory will have a cumulative return, Σ 7' where is the discount factor and T is the number of steps performed. The goal of reinforcement learning is to find an optimal policy æ, so that an agent can obtain the maximum cumulative expected cumu- lative reward using this policy. Policy refers to the probability of specifying π (als) = p(A, = S, = s). Under this policy, the cumulative return follows a distribution, and the expected value of the cumulative return at state is defined state-value function:
action in each state,
Uar (s)
E
k=0
Accordingly, at state the expected value of cumulative return after executing action a is defined as state action value function:
gx (x) Ex
AretttlsSss s5 =a]
km
Often the temporal difference (TD) algorithm Monte Carlo (MC) method is used estimate the value function. However, each of these methods has its advantages and disadvantages. The TD algorithm value estimator has the characteristics of high bias and low variance. In contrast, the estimator of the MC algorithm has low bias and high variance. Kimura et al. (1998) put forward a method to skillfully find a balance between bias and variance, which is called X return. TD(A) proposed by Sutton 1988) is variant of the À return that provides a more balanced value estimation.
Generalized Advantage Estimation (GAE) is a method proposed by [Schulman et al. (2015b) to es- timate the advantage value function. In fact, it is method to apply the lambda return method to estimate the advantage function. As proposed in the PPO paper SShhulmnn et al. (017).. in prac- tical applications, due to the incompleteness of the trajectory, truncated GAE is used, which leads to large bias in the estimation process. For this, we propose partial GAE that uses partially cal- culated GAEs, rather than the entire truncated GAES, significantly reduce the bias that caused

---

Under review as a conference paper at ICLR 2023
by incomplete trajectories. In addition to experiments in common MuJoCo environments, also conduct experiments in the complex and challenging μRTS environment. Our methods have been empirically successful in these environments.
VALUE ESTIMATOR
The gradient in the policy gradient algorithm is generally written in the following form:
g ==
V V (aelgro(aelse)
(3)
where v, used to control the update amplitude of the the policy update in the gradient direction. In the basic policy gradient algorithm, the action value function Ox (s, a) is used as V t, and gx (s, a) is estimated by cumulative return Gr. One of the most direct methods of estimating the value function is the Monte Carlo (MC) method. MC starts directly from the definition of the value function, and takes the accumulation of return values as the estimator of the value function. The accumulation of the discounted reward En-o 7 Rr-n of reward sequence (rr, 114-1 1 ... Tt+NN is taken as the estimator of the state value under state 8t. The state value estimator of the MC algorithm is unbiased estimator. In addition, because the random variables in the estimator are all the returns after time and the dimensions are high, the estimator has the characteristic of high variance.
The TD algorithm uses r; + Ve (st+1) )) as the estimator of value function (=(s).. There a certain error, denoted as eg, between the approximate value function and the real value function. Equation can be obtained, and the bias Es [e (S++1)) can be obtained by using the TD algorithm. In addition, because there are fewer dimensions of random variables in the estimator, the variance of the estimator remains low.
S + (o(+1)) V5. + 7E (co(Se+1)]
Compared with the TD and MC methods, the X-return method seeks to find balance between bias and variance. In Equation 6 if X is O, it is the estimator of the TD method, and if A is 1, it is the estimator of the MC method.
G. (+tt())))
+
Σ
Are
(5)
1=0
N-1
= = (1 A)
Σ
66)
n=1
One disadvantage of using 9x (s, a) is that has large variance, which can be reduced by introducing the baseline b(t), = E Σ (s, Vologrooo)))))
Using the average incomes from the sample baseline offers some improvement. However, for the Markov process, the baseline should change according to the state, and the state baseline which has large value for all actions should be larger, and vice versa. While the A2C and A3C algorithms use Vi as the baseline, 9r (s, a) V2 called the advantage, Q(S, A) can be approximated as R +VV, and the advantage is A(s at) rı + >V(++))) V(sz). In the Generalized Advantage (GAE) paper (Schulman et al. (2015b). ). a TD(A)-like method is proposed, in which the weighted average of the estimated values of different lengths provides the estimated value. This method to calculate the advantage is adopted by powerful reinforcement learning algorithms such as TRPO Schulman et al. (2015a) and PPO Schulman et al. (2017).
8/41 T4I + (V(++4+11 V(S++1)
(8)
AGAE(T.))
= Em2))
(9)
1=0
2

---

Under review as a conference paper at ICLR 2023
When the parameter Y is introduced to estimate the policy gradient, g'is biased from g- Such policy gradient estimation problem actually aims at discounted cumulative reward. In the later part of this paper, the GAE is mainly discussed.
In addition to the above method for calculating Le, there are different methods for estimating the value function Bertsekas al. 2011]. In general, the value loss L,P is calculated with squared- error (Vo, V ) The GAE paper Schulman et al. (2015b) uses the trust region method to optimize the value function in each iteration of the batch optimization process.
minimize Σn Il Vo(sn) - Vn
(10)
subject N DN (VV((e))-Vo.ud (sw) c
2a
And, in most implementations, Vo, is clipped around the value estimates both sides Vo.., and Ve
clip(Vo, LVF = mar - eelipVo (Vo,
1 t,, 1o +e) Vetrgett))
(11)
The work of Tucker et al. (2018) proposes normalizing the advantage, which is shown to improve the performance of the policy gradient algorithm. After the GAE calculates advantages in a batch, the mean and standard deviation of are computed. Then for each advantage, one subtracts the mean and divides by the standard deviation. In Eq. 12 Aporm is normalized advantage, A; is advantage, A Amoon is mean of advantages, A is standard deviation of advantages
A mean
A ARorT
(12)
Asid
It is very important for the policy gradient algorithm to estimate more instructive value function. In the following section, we will mainly discuss the practical application of GAE and how it can be improved.
PARTIAL GAE
In the actual environment, task often has terminal states, which result in finite trajectory length. For the trajectory terminus at time D, one performs an iterative calculation from back to front to compute the GAE. Denote complete trajectory GAE A, (GAB(n.)D) Eq. generalized as A ()) when D - 20.
13 A AGAE(,,A.oo) can be
D-t
A AGAEGADD = (T
(13)
1=0
In practical applications, the length of one sample fixed in order to carry out parallel computing more efficiently. In another case, it takes a long time to sample complete trajectory. In order to make the training more efficient, only a part of the complete trajectory will be sampled at a time. As it is discussed in the PPO paper [Schulman et al. (217). truncated GAE is used for fixed-length trajectory segments, which shown in Figure Calculation of the GAE of an incomplete trajectory with length 7" is represented as:
AC CABLAAT) = 84 = +7 V(ST)
(14)
T = Σ (3)))))
A AGAE(TAT)
(15)
1=0
Following the GAE paper Schulman et al. 2015b). denote sum of i of 8 terms as A! Let A( be an estimator of the advantage function. When h A SY which has large bias and low variance. The bias becomes smaller as becomes larger.
k-1
k-1
Σ
7 sy = -V(s)) + +(s))+~~VV * +
Σ o ress
(16)
1=0
1=0
3

---

Under review as a conference paper at ICLR 2023
Ac AGABOAAT))
D(done)
Sampled Inaedtory
complete tmgeccory
partial training
A (AABGAAT)
ACABTAAT
AG AGAEGAA7
In practical application, considering parallel computing and avoiding episodes that are too long, a fixed sampling length is used. Since the last step of the sampling trajectory is not the termination of the complete trajectory, we have use the truncated GAE the estimator. For all of the calculated truncated GAF Aç AGAE(),A.I) we propose to use A (GAB(,,A.I) for updating in case
Figure 1:
I S and discard A ;CAE(,.A.I) in case > as it shown in red part of this figure.
Since (GAE(TATT) is the sum of exponentially-weigted A, T increases, the number of cumulative terms increases, and the bias decreases.
T-++1
ACABGAATT) = (1-
x)
Σ k=1
k
(17)
Consider two exceptional cases of A, AGAAE(TATTT T D and When T D, A AGAE(T,AA77) is the same as A, A. and its bias is the minimum, that is. bias of A, (.AA((:2.T) than bias of A AG For a specific trajectory, when and T # D, A AGAE(),A.7)) has the maximum bias.
will not be less
D-1+1
Aç GAA((TA,D)) = (1
Σ
x
A(h)
(18)
k=
AT (GAF(AAT) = +V - V(s+)
(19)
Truncated GAE, similar infinite )GAE(T.)) balances between bias and variance with A. AGAE(;,1.T) has low bias and large variance, while AGAB(,.0.7)) has low variance and large bias. When X < with cost of bias, the variance is substantially reduced. Compared with infinite GAE, truncated GAE has larger bias and lower variance. Relatively, it is more important to reduce bias, which means for specific trajectory as the step reduced, A GAE())) will become more instruc- tive.
Compared with GAE of complete trajectory AÇ (GAE(),A.)) denote the difference between trun- cated A, AAE((TT and GAE A GAE(()A.)) as B₁:
B, A GAEGGADD))
A AGAEGATT
Σ (7A)8 I-
(20)
Br is constant for a specific trajectory of length T. As shown in Equation 21 in the case of 0 <
7 1, B, decreases exponentially with step As B, reduced, the deviation between AGAE(7.A.1)
4

---

Under review as a conference paper at ICLR 2023
and A ACAB(O.A.D) is reduced. Since ACAE((TAA:T has larger bias than A, ACAB(TTA,D) reduce B, can
reduce the bias of A, (GAE(TAA.))
D-T B, = (1))
(1))----0 T41 (TA)T-1Br
(21)
1=0
In conclusion, practical applications, due to the fixed length sampling trajectory, the calculation of GAE is truncated, which will lead to large bias in the caleulated GAE when is the end of the trajectory. Instead, we propose to intercept a part of the GAE for use and drop the remainder of the trajectory with large bias. We propose a PPO algorithm with partial GAE described in Algorithm I with partial coefficient and sample length T. In each iteration, each sampler collects T samples and calculates T truncated GAE. We take part of a GAE & > 6, the advantage estimates time t in the trajectory are discarded in the training. Then we construct the surrogate loss as (Schulman al. (2017) on these N(T c) data, and optimize the policy with minibatch Adam for K epochs. For algorithm according to Equation 17 as T L increases, the bias of Aç (GAE(1.A.2)) decreases. As partial coefficient decreases, the bias of Å ... A, decreases.
Algorithm PPO with partial GAE for iteration=l, 2, do for actor=1, ..... do
Run policy Te.rd in environment, get samples (81, 87; Compute advantage estimates Âi,..., Ar
T is not done then
only use A,, A for updating
keep (s,, (r,,.., .... 8T, ar) as (81, (s.,,,..., 87- aT-
else if Tis done then Aq...., Ar for updating end if end for
for epoch K do
Optimize surrogate L wrt 0, with minibatch size 0 - Oold end for end for
EXPERIMENTS
conducted a set of experiments to verify the effect our proposed method, and conducted more detailed investigations:
As we discussed above, can smaller partial coefficients reduce the bias of GAE for im- proved training?
Can larger sampling length reduce the bias of GAE improve training?
We evaluated our method in MuJoCo (022) and μRTS Villar (2017). MuJoCo is physical simulation environment. We mainly conducted experiments Ant-v3 and complementary experi- ments in Halfcheetah-v3. Hopper-v3, Swimmer-v3, Walker2d-v3, jRTS is simplified RTS game environment. Unlike MuloCo, it has discrete states and discrete actions, and has a large number of game steps.
In the experiments with MuJoCo, we use Version 1.31 of MuJoCo (distributed with an MIT License). We use common values in applications to set Hyper-parameters The discount factor is 0.99, À for the GAE is 0.95, the clip coefficient of PPO is 0.2, the value coefficient is 1, the learning rate of the optimizer is 2.5e-4, the number of environments is 64, and the number of epochs is 2. To represent to policy, we use a three layer fully connected MLP Multi-layer Perceptron) with a 64 unit hidden layer, and there two additional noise layers after the MI.P for exploration during training. The
5

---

Under review as a conference paper at ICLR 2023
0000
3033
4003
2000
1393
1003
1002
(a) Halfcheetah-v3
(b) Hopper-v3
3002
120
saitaa c0
patial cent1-2
Issaline
4000
80-
60
2000
1032
(c) Swimmer-v3
(d) Walker2d-v3
Figure 2: training curve in different MuloCo environment
number steps in game of Ant-v3 is 1000, experiments were performed for sample length from 128 to 1024 and coefficient from 64 to 512.
In the experiments with ,RTS, we experimented 2020 μRTS competition. In this experiment, the discount factor y is 0.99, À for the GAE is 0.95,the clip coefficient of PPO is 0.2, the entropy regularization coefficient is 0.01, the value coefficient is 1, the learning rate of the optimizer is 2.5e-4, the number of environments is 64, and the number of epochs is 2. To represent to policy, we use a two layer convolutional neural network connected to three layer fully connected MILP with 512 unit hidden layer.
16x 16 map against CoacAI which won
In this paper, we use the training curve within certain training time to evaluate the algorithm, rather than fixed number of steps. This is because different algorithms and parameters will lead to differences in update time and training time. In practical applications, in addition to training effects, training time also should be considered. In the MuJoCo environment, we take the total reward of average 100 episodes episode after certain training time as the performance score. In μRTS, we take winning rate recent 100 episodes after a certain training time as the evaluation index. For each set of variables, we used 3 random seeds for the experiment. We use an Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz for our experiments.
4.1 WHAT IS THE EMPIRICAL EFFECT OF PARTIAL LENGTH AND SAMPLE LENGTH
As discussed above, the smaller the value in truncated GAE A (GAE(TA,TT) the smaller the bias of the estimate, and the larger the variance. However, the variance of A (GAF()ATT) is always smaller than that Aç G.) so theoretically, the partial coefficient should be as small as possible. In practical applications, small partial coefficients will lead to increase in the number of GAE calculations, while a larger sampling length will lead to a longer sequence to be processed single GAE calculation, which will increase the calculation time.

---

Under review as a conference paper at ICLR 2023
0003
5000
T-34
T-1824
1000
3032
Seadra
2200
1200
cwets
1200
1000
(b)
Figure 3: (a): training curve with different partial coefficient in Ant-v3, sample length 7 = 512. (b)
training curve with different sample length Ant-v3, partial coefficient 64.
303
Figure 4: The performance after hour training in Ant-v3. Since sample time greater than or equal to partial coefficient the white part has no data. T = means not to discard any GAE, which the baseline.
We compare common PPO as baseline with partial GAE PPO in Ant-v3, Halfcheetah-v3 Hopper- v3, Swimmer-v3, walker2d-v3, and we compare the effect of different partial coefficient and sample length Ant-v3. Our implementation does not require the tricks of Engstrom al. (2020) such as value clipping and advantage normal.
The experimental results is shown in Figure 2 and Figure 3 In Figure and Figure 3(a) using partial GAE can achieve better performance than the baseline, and improved performance can be obtained by using smaller partial coefficients. Figure 3(b) using larger sample length improves performance. Figure shows the performance after hour training as T and are varied. It can be seen that the highest performance score with the partial coefficient e e [384, 512] and sample length 7' (64, 128), in Ant-v3.

---

Under review as a conference paper at ICLR 2023
1.0
baseline
partial coefficient=64
0.8
winning rate
0.6
0.4
0.2
0.0
20000
40000
60000
80000
time(s)
Figure 5: Winning rate during training in μRTS 16x16 map against CoacAl.
Note that when the partial coefficient is small or the sampling length is large, the performance does not improve by continuing to increase the partial coefficient or reduce the sampling length. As shown in Equation 21 when (T t) is large enough, B, will become very small, and the change brought by continuing to increase (T' - t) is very small. Although GAE significantly reduces variance by sacrificing bias, and the variance of truncated GAE is smaller than that of the complete trajectory, this does not mean that bias should be minimized while ignoring the impact of variance. In practical applications, the median of partial coefficient and sample length T should be found to balance bias and variance. There is an intermediate value that makes the training effect the best, rather than maximizing (T (c-
We conducted additional experiments in the 16x16 map of pRTS. This is a sparsely rewarded en- vironment with long game steps. In this task, agents need learn how win real-time strategy (RTS) games: collect resources, build units, and destroy enemy units and bases. The sample length T 512 in the experiments.
4.2 TRUNCATED ADVANTAGE ESTIMATOR VARIANCE INVESTIGATION
In the discussion in the above experiments, we supposed that blindly reducing the partial coefficient e will lead to an increase in the variance of truncated GAE and make the training effect worse. In the MuloCo environment, we recorded the 2000 GAE Å, to calculate standard deviation under different t when the sampling length T 500, as shown in the Figure O red curve is the standard deviation of truncated GAE. As a whole, it can be seen from the figure that when is small, GAE has larger standard deviation (or variance). However, unlike the previously mentioned theory, the variance does not completely decrease with the increase of t, especially the end of sampling sequence, the variance increases with the increase of t.
A AAAEGATT = A + Ay
(22)
T-t
A Σ (72)'Te4
(23)
T-1-1
-== V(si) + + - -) A)
Σ
(22)
(24)
1=0
8

---

Under review as a conference paper at ICLR 2023
16
3.5
14
standard deviation
astrmate
3.0
eximaic
12
standard deviad
2.5
10
2.0
1.5
1.0
0.5
2
0.0
0
100
200
300
400
500
0
100
200
300
400
500
(a) Ant-v3
(b) Swimmer-v3
truncated OSC acourniisted rewords vitked cotemate part
standard devialion
250
standard deviation
80
200
60
150
40
100
50
20
Ieriettd DAE
accumm bited rewerrk wakus estimaate
0
100
200 300
400
500
0 100
200 300
400 500
t
t
(c) Hopper-v3
(d) HalfCheetah-v3
Figure 6: During Baseline training, the standard deviation under different sample time I. 2000 truncated GAE for sample time after one million time steps are recorded to calculate standard deviation.
Write the truncated GAE in the form of the Eq. 22 which can be divided into two partuuaccumllated reward part Až and value estimate part Ay. It can seen from the Figure 2 that the variance is more affected by the value estimate part Ay. And because of the uncertainty caused by the deviation of the value function fitting the actual value. the variance of the truncated GAE does not only decrease with the increase of f. As shown in the Figure 6, empirically, smaller will result in larger variance of truneated GAE. necessary to select an intermediate value of partial coefficient to balance the variance and bias of the truncated GAE to obtain better training effect.
CONCLUSION
How to estimate the value function is particularly important in policy gradient algorithms. GAE provides a method to balance the bias and variance of the estimation of the value function. However, practical applications, truncated GAE often used, and will lead to excessive bias of value estimation. We propose to use partial GAE in training to discard truncated GAE with excessive bias, which can reduce the bias and make value estimation more instructive.
We conducted experiments in MuloCo environment. The experimental results show that using par- tial GAE will always achieve improved training results. For partial GAE related parameters, sam- pling length 7' and partial coefficient& although theoretically increasing (T e) can reduce bias, there is an intermediate value for the best training effect due to the influence of comprehensive vari- ance. How to adjust sampling length T and partial coefficient adaptively may be direction for future research. We conducted additional experiments in pRTS. In this sparse environment with coefficient rewards and long game steps, partial GAE also performed well.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We note that our method does not introduce new potential societal harms as it is an improvement of value estimation; however inherits any potential societal harms of deep reinforcement learning methods, which are well documented in Whittlestone et al. (202I). Note that the policy of an agent trained by deep reinforcement learning is highly dependent on its explored state and training envi- ronment, which causes the agent to perform unexpected actions when it encounters state it has not seen before. It is not reliable rely on the generalization neural network to solve problems. is necessary to consider how to deal with unexpected actions of agents, especially when DRL agents are applied to the real world.
REPRODUCIBILITY STATEMENT
Our experiments were repeated three times with different random seeds. We will upload our code in the supplemental materials for verification.
REFERENCES
Dimiuri P Bertsekas et al. Dynamic programming and optimal control 3rd edition, volume ii. Bel- mont, MA: Athena Scientific, 2011.
Logan Engstrom, Andrew Iyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. International Conference on Learning Representations, 2020.
Hajime Kimura, Shigenobu Kobayashi, et al. An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value function. In ICMI., volume 98, 1998.
OpenAl. Gym-mujoco. 2022. Accessed: 2022.
joco/
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In nternational conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02455 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1777v:7707.06377, 2017.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines reinforcement learning. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5015-5024. PMLR, 10-15 Jul 2018.
Santi Ontañón Villar. microrts. httppp///gthhaoocco////attttttta Accessed: 2017-07-13.
2017.
Jess Whittlestone, Kai Arulkumaran, and Matthew Crosby. The societal implications of deep rein- forcement learning. Journal of Artificial Intelligence Research, 2021.
10


Reviews:
[{"review_id": "x2xbEyuEQ2", "reviewer": "Reviewer_VSUL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to abandon the samples near the end of the trajectory due to an estimation error in the GAE estimator. Experiments show the effectiveness of the method.", "summary_of_the_review": "\n\nGenerally, the paper is poorly written and contains a lot of typos and errors. The contribution of the paper is limited and it's hard to assess the effectiveness of the experimental results reported in the paper. \n\n", "strength_and_weaknesses": "\n\nStrength:\n\nThe proposed method is very simple and easy to implement.\n\nIt's also interesting to see that abandoning parts of the data near the end the trajectory can lead to better performance.\n\n\n\nWeakness:\n\nThe writing of the paper is poor and casual. There are a lot of typos and errors. Many notations are undefined.\n\nThe paper seems to be a technical report of detail of the practical implementation. The contribution to the community is limited.\n\nThe experiments are done with only 3 seeds. It's hard to asses the results .\n\n\n\nSeveral Questions:\n\nI don't understand why  you assume that $V(s_t)=0$?\n\nBy Fig.3, It seems that the method is very sensitive to the hyperparameter of the partial coefficient $\\epsilon$. How do you determine the value of this hyperparameter in practice?\n\n\n\n\n\nSeveral typos or notations undefined:\n\neq.2: missing action $a$, $q_\\pi(s,a)$, and please use big bracket.\n\nSection 3 terminus: do you mean terminate?\n\neq. 11: why there is $V_{\\theta_{t-1}}$ and $V_{\\theta_{t+1}}$ ? By the way, do you define $V_{theta}$?\n\neq.12: Do you define $A_i$, $A_{mean}$, $A_{std}$\n\nFigure 2: environment=>environments", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "cyvabwE8wxX", "reviewer": "Reviewer_zF4x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The standard GAE estimator uses a lambda weighted average of the k-step returns to estimate the advantage, i.e.,\nit uses $\\lambda(r_0 + \\gamma V(s_1)) + \\lambda^2 (r_0 + \\gamma r_1 + \\gamma^2 V(s_2) + ...)$ (in a normalized fashion).\nIn this standard method, the sequence continues until the terminal state or until $\\infty$ so the sequence has the same bias-variance properties at any time-step. (The bias here comes from using a function approximator critic of $V$ that is not exact.) However, in practice we often use truncated rollouts of some short horizon T, shorter than the whole episode, so that the policy can be updated multiple times during a single episode. This allows learning faster as the policy is updated more often.\n\nThe current work points out that if we use truncated rollouts, then the bias-variance properties are different for small $t$ compared to large $t$ where $t<T$ is the time-step in the rollout. For example, when $t=T-1$ the advantage estimate is just $r_{T-1} + \\gamma V(s_T) - V(s_{T-1})$, which relies more on the accuracy of the value critic compared to small $t$. In particular they mention that the bias might be large for large $t$, and the authors wish to reduce the bias.\n\nTo reduce the bias, they propose to do a partial GAE update, where they compute the advantage estimates for all $t<T$, but only update the policy for $t < H < T$. Then, they keep the data for $H \\leq t \\leq T$, gather more data, compute new GAE estimates, and update the policy parameters with the new data. This approach seems sensible as the gather data is not discarded. However, it seems there is a tradeoff of needing more memory. (The computation of the GAE estimates itself should be quite fast)\n\nExperiments were performed on 4 MuJoCo tasks with 3 seeds each, where there was a trend towards the authors' partial GAE estimator improving the performance. They also performed experiments on $\\mu RTS$ where the results showed that the new method quickly solves the task whereas the regular GAE does not.", "summary_of_the_review": "I recommend rejecting the paper as I found it unclear and incomplete.\n\nThe idea of using a partial GAE estimate as presented in the paper appears like a sensible implementation trick, and I could see this being published if the experimental results are convincing and the paper is written well.\n\nHowever, currently the quality of the paper seems far below the level I would expect to be published at ICLR (the length is only 8 pages, the writing is unclear, the experiments only look at reward values). I believe it requires much work on extending the experiments, and on improving the writing.", "strength_and_weaknesses": "Strengths\n---------------\n- The method appeared reasonable and simple to implement from a technical point of view.\n- GAE is a very common technique, so the idea is relevant to many works.\n\nWeaknesses\n---------------\n(See next section for justifications.)\n- I found the clarity to be insufficient. \n- It was not clear to me that the methods were well implemented as there did not appear to be a comparison with work or results produced by other researchers.\n- The experimental results only looked at reward values. It would be good to find additional experimental evidence to support the claims, e.g. computing the variance of the advantage estimates (and the bias if possible).\n- The work seems incomplete and not sufficiently thorough.\n- The literature review is very short. For example \"Asynchronous Methods for Deep Reinforcement Learning\" Mnih 2016 that popularized the truncated rollout methodology was not cited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BjYr6LwyUr", "reviewer": "Reviewer_QPtY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents partial advantage estimation for PPO instead of using all the (truncated) advantage functions. PPO uses the lambda return method to estimate the advantage function. The proposed algorithm takes a partial coefficient \\epsilon as input and discards trajectory for time steps greater than \\epsilon, the advantage estimates are discarded in the training. Then the loss is computed on the remaining data.", "summary_of_the_review": "Incremental improvement over PPO.\n", "strength_and_weaknesses": "The work is very incremental and numerical results to demonstrate the performance are also very limited. From the paper, it seems like the performance is improved without paying any price which doesnt seem right to me", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject