A Strong On-Policy Competitor To PPO
Authors: Xiangxiang Chu
Venue:  (2021)
Keywords: proximal policy optimization
deep reinforcement learning

Abstract: As a recognized variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely used with several advantages: efficient data utilization, easy implementation and good parallelism. In this paper, a first-order gradient on-policy learning algorithm called Policy Optimization with Penalized Point Probability Distance (POP3D), which is a lower bound to the square of total variance divergence is proposed as another powerful variant. The penalty item has dual effects, prohibiting policy updates from overshooting and encouraging more explorations.  Carefully controlled experiments on both discrete and continuous benchmarks verify our approach is highly competitive to PPO.

Under review as a conference paper at ICLR 2021
A STRONG ON-POLICY COMPETITOR TO PPO
Anonymous authors Paper under double-blind review
ABSTRACT
As recognized variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely used with several advantages: efficient data utilization, casy implementation. and good parallelism. In this paper, first-order gradient reinforcement learning algorithm called Policy Optimization with Penalized Point Probability Distance (POP3D), which is a lower bound to the square of total variance divergence, is proposed as another powerful variant. The penalty item has dual effects, prohibiting policy updates from overshooting and encouraging more explorations. By carefully controlled experiments on both discrete and continuous benchmarks, our approach is proved highly competitive to PPO.
INTRODUCTION
With the development of deep reinforcement learning. lots of impressive results have been produced in wide range of fields such playing Atari game (Mnih et al., 2015; Hessel et al., 2018), controlling robotics (Lillicrap et al., 2015), Go (Silver et al., 2017), neural architecture search (Tan et al., 2019; Pham et al., 2018).
The basis of a reinforcement learning algorithm is generalized policy iteration (Sutton & Barto, 2018), which states two essential iterative steps: policy evaluation and improvement. Among various algorithms, policy gradient is an active branch of reinforcement learning whose foundations are Policy Gradient Theorem and the most classical algorithm REINFORCEMENT (Sutton & Barto, 2018). Since then, handfuls of policy gradient variants have been proposed, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap al., 2015), Asynchronous Advantage Actor- -CtttcCritic (A3C) (Mnih et al., 2016), Actor-Critic using Kronecker-factored, Trust Region (ACKTR) (Wu et al., 2017), and Proximal Policy Optimization (PPO) (Schulman al., 2017).
Improving the strategy monotonically had been nontrivial until Schulman et al. (2015) proposed Trust Region Policy Optimization (TRPO), in which Fisher vector product is utilized to cut down the computing burden. Specifically, Kullback-Leibler divergence (KLD) acts as hard constraint in place of objective, because its corresponding coefficient difficult to set for different problems. However, TRPO still has several drawbacks: too complicated, inefficient data usage. Quite lot of efforts have been devoted to improving TRPO since then and the most commonly used one is PPO.
PPO can be regarded first-order variant of TRPO and have obvious improvements in several facets. In particular, a pessimistic clipped surrogate objective is proposed where TRPO's ' hard constraint is replaced by the clipped action probability ratio. In such a way, it constructs unconstrained optimization problem so that any first-order stochastic gradient optimizer can be directly applied. Besides, it's easier to be implemented and more robust against various problems, achieving an impressive result on Atari games (Brockman et al., 2016). However, the cost of data sampling is not always cheap. Haarnoja et al. (2018) design an off-policy algorithm called Soft Actor- Actor-Critic and achieves the state of the art result by encouraging better exploration using maximum entropy.
In this paper, we focus on the on-policy improvement improve PPO and answer the question: how to successfully leverage penalized optimization to solve the constrained one which is formulated by Schulman al. (2015).
1. It proposes a simple variant of TRPO called POP3D along with a new surrogate objective containing a point probability penalty item, which is symmetric lower bound to the square of the total variance divergence of policy distributions. Specifically, it helps to stabilize

---

Under review as a conference paper at ICLR 2021
the learning process and encourage exploration. Furthermore, it escapes from penalty item setting headache along with penalized version TRPO, where is arduous to select one fixed value for various environments.
2. It achieves state-of-the-ar results among on-policy algorithms with a clear margin on 49 Atari games within 40 million frame steps based achieves competitive results compared with PPO in the continuous domain. It dives into the mechanism of PPO's improvement over TRPO from the perspective of solution manifold, which also plays an important role in our method.
shared metries. Moreover, also
3. It enjoys almost all PPO's advantages such as easy implementation, fast learning ability.
We provide the code and training logs to make our work reproducible.
PRELIMINARY KNOWLEDGE AND RELATED WORK
2.1 POLICY GRADIENT
Agents interact with the environment and receive rewards which are used to adjust their policy in turn. At state 9r, one agent takes strategy л and transfers to a new state 8 rewarded r, by the environment. Maximizing discounted return (accumulated rewards) RG is its objective. In particular, given a policy n, R2 defined as
R4 = Σ + TH41 + + T
+ +... +7"T
(1)
y is the discounted coefficient control future rewards, which lies in the range (0, 1). Regarding neural network with parameter 0, the policy (o(a)s) can be learned by maximizing Equation I using the back-pre proppprrpagttion algorithm. Particularly, given (s, a) which represents the agent's return in state after taking action a, the objective function can be written as
max
E,,a log To (a)sa))s (s)(s.))..
6
Equation lays the foundation for handfuls of policy gradient based algorithms. Another variant can be deduced by using
A(s, a) = Q(s, a) - V(s)
(3)
to replace (ss, a) in Equation 2 equivalently, (ss) can be any function so long as V depends on but not In most cases, state value function is used for V,, which not only helps to reduce variations but has clear physical meaning. Formally, it can be written as
max Eg,0 log no (als)
2.2 ADVANTAGE ESTIMATE
A commonly used method for advantage calculation is one-step estimation, which follows
A(s,, a+) = Q(,,, a.) V(s)) = = + (s+4)) - Vss)).
(5)
However, more accurate method called generalized advantage estimation is proposed in Schulman et al. (2016), where all time steps of estimation are combined and summarized using X-based weights, The generalized advantage estimator A. (GA(())) is defined by Schulman al. (2016) as
20
A GAB(TA)) = (1 A) * (A + xAA(2) A +...) Σ (7
8V 110 ritl + V (5 - V($++1).
(6)
1
VV Op0!
Vss:)) + M74 + + kl 1 + Ceoi-i+-*VVCe *V
1=0
The parameter À meets 0 < A < 1, which controls the trade-off between bias and variance. All methods in this paper utilize A (AA((,.) to estimate the advantage.
2

---

Under review as a conference paper at ICLR 2021
2.3 TRUST REGION POLICY OPTIMIZATION
Schulman et al. (2015) propose TRPO to update the policy monotonically. In particular, its mathematical form is
max Ee To (ar((8) As] C.E ([ To (-][6)]]
Tond (1t)88))
(7)
= max Fanxo(alo)l (ooooo))/AAAA AToos ata (8, a))
where C' is the penalty coefficient, C
26y (1-7)2-
In practice, the policy update steps would be too small if C is valued as Equation 7. In fact, it's intractable to calculate beforehand since it requires traversing all states to reach the maximum. Moreover, inevitable bias and variance will be introduced by estimating the advantages of old policy while training. Instead, a surrogate objective is maximized based on the KLD constraint between the old and new policy, which can be written below,
E, Ta(ac (s)) As]
Toar (ael (8t)
(8)
s.f. E. I (-18)), (.())) 8
where 8 is the KLD upper limitation. In addition, the conjugate gradient algorithm is applied to solve Equation 8 more efficiently. Two major problems have yet to be addressed: one is its complexity using the conjugate gradient approach, another compatibility with architectures that involve noise or parameter sharing tricks (Schulman et al., 2017).
2.4 PROXIMAL POLICY OPTIMIZATION
To overcome the shortcomings of TRPO, PPO replaces the original constrained problem with pessimistic clipped surrogate objective where KL. constraint is implicitly imposed. The loss function can be written as
CLIP (0) E2 (m) clip(r:))) c)
To(a; (s))
(+)
Tooia (aclst)
where is a hyper- apper-parameter to control the clipping ratio. Except for the clipped PPO version, KL penalty versions including fixed and adaptive KLD. Besides, their simulation results convince that clipped PPO performs best with an obvious margin across various domains.
3 POLICY OPTIMIZATION WITH PENALIZED POINT PROBABILITY DISTANCE
Before diving into the details of POP3D, we review some drawbacks of several methods, which partly motivate us.
3.1 DISADVANTAGES OF KULLBACK LEIBLER DIVERGENCE
TRPO (Schulman et al., 2015) induced the following inequality",
26y
T)<T)) S Limaaaa (TO) +
(1 (1-))2a'
(10)
a = D (To TT)
D TV (Nowaa To) Dry (Tossall (T))
TRPO replaces the square of total variation divergence Dm (Nomes To) by Dmax (Tomur T0) max, D& Pxil(0oaallag (T=o)
Note that 7) means loss instead of the ICML. version (Schulman al., 2015).
3

---

In fact, PPO is called pessimistic proximal policy optimization" in the meaning of its objective construction style. Without loss of generality, supposing A, > 0 for given state s, and action de and the optimal choice is a;. When a, = a; good update policy is increase the probability of action to a relatively high value az by adjusting 0. However, the clipped item clip(r:(0), will fully contribute to the loss function by the minimum operation, which ignores further reward by Under review as a conference paper at ICLR 2021
Given discrete distribution and q, their total variation divergence (pll)) is defined
Drv(pllg) == 2 Σ pp.
(11)
in TRPO (Schulman et al., 2015). Obviously, Drv is symmetric by definition, while KLD is asymmetric. Formally, given state s, KLD of To for (+(+ss) can be written as
(+s) = Σ Toara (a|s) Teaa (als) ma(a(s)
(12)
Similarly, KLD in the continuous domain can be defined simply by replacing summation with integration. The consequence of KLD's asymmetry leads to a non-negligible difference of whether choose (Tz solutions. Robert compared the forward and reverse KL on a distribution, one solution matches only one of the modes, and another covers both modes (Murphy, 2012). Therefore, KL.D is not ideal bound or approximation for the expected discounted cost.
D ( Te,.a). Sometimes, those two choices result in quite different
3.2 DISCUSSION ABOUT PESSIMISTIC PROXIMAL POLICY
e, 1 + e)
zero gradients even though it's the optimal action. Other situation with A, < can be analyzed in the same manner.
However, if the pessimistic limitation is removed, PPO's performance decreases dramatically (Schul- man et al., 2017), which is again confirmed by our preliminary experiments. In word, the pessimistic mechanism plays a very critical role for PPO in that it has a relatively weak preference for a good action decision at given state, which in turn affects its learning efficiency.
3.3 RESTRICTED SOLUTION MANIFOLD FOR EXACT DISTRIBUTION MATCHING
To be simple, we don't 't take the model identifiability issues along with deep neural network into account here because they don't affect the following discussion much (LeCun al., 2015). Suppose To. is the optimal solution for a given environment, in most cases, more than one parameter set for can generate the ideal policy, especially when Ta. learned by a deep neural network. other words, the relationship between 0 and ma is many to one. On the other hand, when agents interact with the environment using policy represented by neural networks, they prefer to takes the action with the highest probability. Although some strategies of enhancing exploration are applied, they don't affect the policy much in the meaning of expectation.
RL methods can help agents learn useful policies after fully interacting with the environment. Take Atari-Pong game for example, when an agent sees a Pong ball coming close to the right (state s1), its optimal policy is moving the racket to the right position (for example, the "RIGHT" action) with <1 a distribution po; = [0.05, 0.05, 0, 0.7, 0.05, 0.05)². The probability of selecting "RIGHT" is relatively high value such 0.7. It's almost impossible to push it be 1.0 exactly since it's produced by softmax operation on several discrete actions. In fact, we hardly obtain the optimal solution accurately. Instead, our goal to find a good enough policy. ((RIGHT)s)) above a threshold is sufficient to be a good one. In other words, paying attention to the most critical actions sufficient, and we don't 't care much the probability value of the other non-critical actions. For example, good policy at is I?.?, 2 0.7, ?,?.?J. Note that n (a(as)) is represented by a neural network parameterized using and a good policy for the whole game means
this case, the policy of pushing
"The word "pessimistic" is used by the PPO paper.
'The action space is described ('NOOP", 'FIRE', 'RIGHT', 'LEFT', 'RIGHTFIRE', "LEFTFIRE").
4

---

Under review as a conference paper at ICLR 2021
that the network can perform well across the whole state space. Focusing those critical actions each state* and ignoring non-critical ones can help the network learn better and more easily.
Using penalty such as KLD cannot utilize this good property, because it involves all of the actions' probabilities. Moreover, doesn't I stop penalizing unless two distributions become exactly indifferent the advantage item is large enough compensate for the KLD cost. Therefore, even if 9 outputs Odd the same high probability for the right action, the penalization still exists. Suppose that two parameters for 05: 02 and 03. where Po; [0.01, 0.15, 0.05, 0.7, 0.01, 0.08] and pi [0.01, 0.01, 0.01, 0.7, 0.26, 0.01]. When the agent already chooses RIGHT at Si, the loss item from good penalized distance should be small. However, ( (-[s1)]l DKI(Ta, (-Is1) Te, ==339, However, it's not necessary to require the distribution of other actions ("NOOP,, 'FIRE, 'LEFT", 'RIGHTFIRE', 'LEFTFIRE") of Po; near to P Instead, it's better to relax this requirement enlarge the freedom degree of the network and focus on learning important actions. Doing this brings another advantage, the agent can explore more for non critical actions. From the perspective of the manifold, optimal parameters constitute solution manifold. The KLD penalty will act until / exactly locates the solution if possible, akin to mapping a point onto curve. Instead, if the agent concentrates only critical actions like human does, it's much easier to approach the manifold in higher dimension. This comparable to expanding the solution manifold by at least one dimension, e.g. from curves surfaces or from surfaces to spheres.
(-)ss) (-1s1))=0.15 and
3.4 EXPLORATION
One shared highlight in reinforcement learning is the balance between exploitation and exploration. For policy-gradient algorithm, entropy added the total loss to encourage exploration in most cases. When included in the loss function, KLD penalizes the old and new policy probability mismatch for all possible actions as Equation 12 given a state s. This strict punishment for every action's 's probability mismatch, which discourages exploration.
3.5 POINT PROBABILITY DISTANCE
To overcome the above-mentioned shortcomings, we propose a surrogate objective with the point probability distance penalty, which symmetric and more optimistic than PPO. the discrete domain, when the agent takes action a, the point probability distance between Ta (-|s) and (-(+ss) defined by
Do
(To (-18), pp
mo(-ss)) (al)) Ta(as)))
(13)
Attention should be paid to the penalty definition item, the distance measured by the point probability, which emphasizes its mismatch for the sampled actions for state. Unless it would lead to confusion, we omit a for simplicity in the following sections. Undoubtedly, Dpp is symmetric by definition. Furthermore, divergence Drv. As special case, can be easily proved that for binary distribution, (pll)) D (p)g))
be proved that Dpp is indeed a lower bound for the total variance
Theorem 3.1. For two discrete probability distributions and q with K values, then D 2 Do (p) (plgg) holds for any action a and E, Dr (pl (q) a lower bound for (pllg))
'Note that some states don't have critical action. Taking the Pong for example, when the hall is just shot back, the agent choose any action.
5

---

Under review as a conference paper at ICLR 2021
Proof. Let Pt a,
3 for the I-th action a, and suppose 2 b without loss of generalization. So,
K
Div(plla)
2
Pi
q:1)²
( Σ
pi
qi
+
Pt -q11)²
i=1
1=1,i71
K
2
Σ
8il 1 - B))2 = 1 1-11-0 a (1- 8) + 7 -)))
1
2
2
1=,17
((a- 3) - 1
1
-5(a - -3)) = (pp())
2
Eg DeD
Σ p(a) (pp() Ep([ p(a) D(r (tll)))))
Since 0 S Ta(als) < holds for discrete action space, D pp has lower and upper boundary: S Ppp S 1. Moreover, py less sensitive to action space dimension than KLD, which has similar effect PPO's clipped ratio to increase robustness and enhance stability. Equation 13 stays unchanged for the continuous domain, and the only difference is To (a(as) represents point probability density instead of probability.
3.6 POP3D
After we have defined the point probability distance, we use a new surrogate objective fo for POP3D, which can be written
E, To(a (1) A. BDD BD (To (-1s)) (o (-)+)))))]
(14)
To.i (0e(8) (86)
where 3 is the penalized coefficient. These combined advantages lead to considerable performance improvement, which escapes from the dilemma of choosing preferable penalty coefficient. Besides, we use generalized advantage estimates calculate Åg. Algorithm shows the complete iteration process of POP3D. Moreover, it possesses the same computing cost and data efficiency as PPO.
Algorithm 1 POP3D
1: Input: max iterations L, actors N, epochs to L do 3: for actor to N do
2: for iteration
4: Run policy Te..a for T time steps
5: Compute advantage estimations Å ÂI,..., ÄT
6: end for
7: for epoch = 1 to K do
8: Optimized loss objective f(0) w.r.t 0 with mini-batch size M < NT,, then update lold + O.
9: end for
10:
end for
3.7 WORKING MECHANISM OF POP3D
As for the toy example in Section 3.3, D DRIGHT (mo, (-Is) Te, (-Is)=DDRRG RIGHT (TT, [e, Therefore, it can help the agent to focus on the important action. When updating from dota as Equation 14, the gradient (0) w.r.t. 0 can be written
pp

---

Under review as a conference paper at ICLR 2021
Vof(0) = VoTo(ar(8t)
A, 2.3(mo((s(ss)) - -ooa (on
Tondd (aelse)
A.
Vono(as)s4)
To.a (ae(se) (3)(T)(()))))) - To (ase(()))))))
(15)
As
VoTTo(as(5s)
No.ra (aelss) 2.3((a)(:)))
To.c where (())(s)) == To(a1(1:) -o.c (ax)() (e) Suppose the agent selects dz at 8, using To.i and obtains a positive advantage Å, if To (a(a)())) is larger than To (a))s))) then 2/35(ag(s;) will play damping role avoid too greedy preference for a (i.e. too large probability), which turn leaves more space for other actions to be explored. Other cases such as negative A, can analyzed similarly. The hyper-parameter controls the damping force.
In the early stage of learning, Te..s (arlst) is near 1/K (taking discrete spaces for example) and the magnitude of Å, is large, while the damping force a bit weak. Therefore, the agent learns fast. Then / shows relative stronger force to avoid overshooting for action selection and encourage more exploration. As for the final stage, the policy changes slowly because the learning rate is low, where 5(a (s)) is small and therefore it converges.
3.8 RELATIONSHIP WITH PPO
To conclude this section, take time to why PPO works by taking the above viewpoints into account. When we pour more attention to Equation 9, the ratio r, r0) only involves the probability for given action o, which is chosen by policy In other words, all other actions' probabilities except a are not activated, which no longer contribute to pack-propagation and allow probability mismatch, which encourage exploration. This procedure behaves similarly to POP3D, which helps the network to learn more easily. Above all, POP3D is designed to conform with the regulations for overcoming above mentioned problems, and in the next section experiments from commonly used benchmarks will evaluate its performance.
EXPERIMENTS
4.1 CONTROLLED EXPERIMENTS SETUP
OpenAl Gym is a well-known simulation environment to test and evaluate various reinforcement algorithms,, which is composed of both discrete (Atari) and continuous (Mujoco) domains (Brockman et al., 2016). Most recent deep reinforcement learning methods such as DQN variants (Van Hasselt et al., 2016; Wang et al., 2016; Schaul et al., 2015; Bellemare et al., 2017: Hessel et al., 2018), A3C, ACKTR. PPO evaluated using only one set of tyyper-prrmmetee POP3D's performance on 49 Atari gam discrete action space and 7 Mujoco (v2, continuous).
Therefore, we evaluate
Since PPO is distinguished RL algorithm which defeats various methods such A3C, A2C ACKTR, we focus on a detailed quantitative comparison with fine-tuned PPO. And we don't consider large scale distributed algorithms Apex-DQN (Horgan al., 2018) and IMPALA (Espeholt al., 2018), because we concentrate on comparable and fair evaluation, while the latter is designed to apply with large scale parallelism. Nevertheless, some orthogonal improvements from those methods have the potentials to improve our method further. Furthermore, we include TRPO to acts baseline method. Engstrom et al. (2020) carefully study the underlying factor that helps PPO outperform TPRO. To avoid unfair comparisons, we carefully control the settings. In addition, quantitative comparisons between KLD and point probability penalty helps to convince the critical role of the latter, where the former strategy is named fixed KLD in Schulman et al. (2017) and can act as another good baseline in this context, named by BASELINE below.
*DQN variants are evaluated in Atari environment since they designed to solve problems about discrete action space. However, policy gradient-basee algorithms can handle both continuous and discrete problems.
7

---

Under review as a conference paper at ICLR 2021
In partieular, retrained one agent for each game with fine-tuned peper--arameters.. To avoid the problems of reproduction about reinforcement algorithms mentioned in Henderson et al. (2018), we take the following measures:
Use the same training steps and make use of the same amount of game frames(40M for Atari game and 10M for Mujoco).
Use the same neural network structures, which is the CNN model with one action head and one value head for the Atari game, and a fully-connected model with one value head and one action head which produces the mean and standard deviation of diagonal Gaussian distribution as PPO.
Initialize parameters using the same strategy PPO.
Keep Gym wrappers from Deepmind such reward clipping and frame stacking unchanged for Atari domain, and enable 30 no-ops at the beginning of each episode.
Use Adam optimizer (Kingma & Ba, 2014) and decrease a linearly from for Atari domain as PPO.
To facilitate further comparisons with other approaches, release the seeds and detailed re- sults' (across the entire training process for different trials). In addition, we randomly select three seeds from (0, 10, 100, 1000, 10000) for two domains, (10,100,1000 for Atari and 10,10,100) for Mujoco order to decrease unfavorable subjective bias stated in Henderson et al. (2018).
sults"(across
4.2 EVALUATION METRICS
PPO utilizes two score metrics for evaluating agents' performance using various RI. algorithms. One is the mean score of the last 100 episodes Score100- which measures how high strategy can hit eventually. Another is the average score across all episodes Scoreau, which evaluates how fast an agent learns. In this paper, conform to this routine and calculate individual metric by averaging three seeds in the same way.
4.3 DISCRETE DOMAIN COMPARISONS
Hyper-parameters We search hyper- pper-paraeeter four times for the penalty coefficient based on four
Atari games while keeping other typer-parament unchanged as PPO and fix B = 5.0 to train all Atari games. For BASELINE, we also search hyper-parameter four times on penalty coefficient and choose 3 10.0. To save space, detailed hyper-parameter setting can be found in Table 6 and 7.
This process is not beneficial for POP3D owing to missing optimization for all hyper-parameters. There are two reasons to make this choice. On the one hand, it's the simplest way to make relatively fair comparison group such as keeping the same itcrations and epochs within one loop to our knowledge. On the other hand, this process imposes low search requirements for time and resources. That's ' to say, we can draw a conclusion that our method is at least competitive to PPO if performs better benchmarks.
Comparisons The final score of each game is averaged by three different seeds and the highest is in bold. As Table shows, POP3D outperforms 32 across 49 Atari games given the final score, followed by PPO with 11, BASELINE with 5, and TRPO with 1, Interestingly, for games that POP3D score highest, BASELINE score worse than PPO more often than the other way round, which means that POP3D is not just an approximate version of BASELINE.
For another metric, POP3D wins 20 out of 49 Atari games which matches PPO with 18, followed by BASELINE with 6, and last ranked by TRPO with 5. If we measure the stability of an algorithm by the score variance of different trials, POP3D scores high with good stability across various seeds. And PPO behaves worse in Game Kangaroo and UpNDown. Interestingly, BASELINE shows large variance for different seeds for several games such as BattleZone, Freeway, Pitfall, and Seaquest. POP3D reveals its better capacity to score high and similar fast learning ability in this domain. The detailed metric for each game is listed in Table 3 and 4.
We use OpenAl's PPO and TRPO code: ht / // thub. com http//ppeaaaa// haaaaei/baaeleeee ins..git 'https://drive, google. eeogle..oo com/ // c7 971gg//77777// wwwp://gwwnnn.....am. http://www.d//o/o/oo//o/////o/oa
/vi
8

---

Under review as a conference paper at ICLR 2021
4.4 CONTINUOUS DOMAIN COMPARISONS
Sypeer-prrameters For PPO, we use the same hyper- parameter configuration as Schulman et al. (2017). Re- garding POP3D, we search on two games three times and select 5.0 the penalty coefficient. More details for PPO and POP3D are listed in Table Unlike the Atari domain, we utilize the con- stant learning rate strategy as Schulman et al. (2017) the continuous domain instead the linear decrease strategy.
Table I: Top: The number of games "won" by each algorithm for Atari games. Bot- tom: The number of games algorithm for Mujoco games. Each experi- ment is averaged across three seeds.
by each
about
Metric
PPO POP3D BASELINE TRPO
Score
11
Scorcall
18
20
6
5
Comparison Results The scores are also averaged on three trials and summarized in Table 1. POP3D occu- pies 6 out of 7 games on Score100- Evaluation metrics
Metrie
PPO POP3D
Score100
1
Scoreau
4
of both across different games illustrated in Table 2 and 5. In summary, both metrics indicate that POP3D is competitive to PPO the continuous domain.
5 CONCLUSION
In this paper, we introduce a new reinforcement learn- ing algorithm called POP3D (Policy Optimization with Penalized Point Probability Distance), which acts as TRPO variant like PPO. Compared with KLD that is upper bound for the square of total variance divergence between two distributions, the penalized point proba- bility distance is a symmetric lower bound. Besides, equivalently expands the optimal solution manifold effectively while encouraging exploration, which is similar mechanism implicitly possessed by PPO. The proposed method not only possesses several critical improvements from PPO but outperforms with clear margin 49 Atari games from the respective of fi- nal scores and meets PPO's match as for fast learning ability.
Table 2:
Mean final scores (last 100 episodes) of PPO, POP3D on Mujoco games after 10M frames. The results are averaged by three trials.
Game
PPO
POP3D
HalfCheetah
2726.03
3184.54
Hopper
2027.21
1452.09
InvertedDDi/enndulmm
4455.03
4907.64
InvertedPendulum
544.02
741.94
Reacher
-5.00
-4.29
Swimmer
111.88
112.08
Walker2d
1112.25
3966.01
More interestingly, not only suffers less from the penalty item setting headache along with TRPO, where arduous select one fixed value for various environments but outperforms fixed KLD baseline from PPO. In summary, POP3D is highly competitive and an alternative to PPO.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning, arXiv preprint www.iv:707.068800 2017.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint S.5.v::6666011500 2016.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2020. URL https: hatp::s//ppereeviie net Forum rid
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. pp. 1861-1870. 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference Artificial Intelli- gence, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arxiv:1509.02977 00977 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et Human-level control through deep reinforcement learning. nature, 51 (18((550):559-555 -333, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095-4106 2018.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952. 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
10

---

Under review as a conference paper at ICLR 2021
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676:334-3599 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 2820-2828, 2019.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Thirtieth AAAI conference on artificial intelligence, 2016.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003, 2016.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker- factorre approximation. Advances in neural information processing systems, pp. 5279-5288, 2017.

---

Under review as a conference paper at ICLR 2021
A SCORE TABLES AND CURVES
Mean scores of various methods for Atari domain are listed in Table 3 and 4.
Table 3: Mean final scores (last 100 episodes) of PPO, POP3D, BASELINE and TRPO on Atari games after 40M frames. The results are averaged on three trials.
game
POP3D
PPO
BASELINE
TPRO
Alien
1510.80
1431.17
1311.23
1110.40
Amidar
729.15
790.75
655.10
200.56
Assault
5400.13
4438.82
1846.75
1363.46
Asterix
4310.67
3483.17
3657.67
2651.35
Asteroids
2488.10
1605.33
1615.37
2205.70
Atlantis
2193605.67
2140536.33
1515993.33
1419104.67
BankHeist
1212.23
1206.67
1124.43
1125.17
BattleZone
15466.67
14766.67
14690.00
15123.33
BeamRider
4549.00
2624.19
6898.09
5073.75
Bowling
38.99
47.27
30.48
31.24
Boxing
97.23
93.70
65.33
50.07
Breakout
458.41
281.93
67.70
40.65
Centipede
3315.44
3565.18
3393.93
3353.14
Command
6308.33
4872.67
2676.00
2286.67
CrazyClimber
120247.33
105940.00
98219.67
87522.33
DemonAttack
61147.33
26740.57
57476.65
21525.08
DoubleDunk
-7.89
-11.22
-8.61
-10.04
Enduro
459.85
698.46
518.41
365.95
FishingDerby
28.99
17.72
-64.27
69.64
Freeway
21.21
21.11
18.37
20.89
Frostbite
316.87
280.30
280.30
291.77
Gopher
6207.00
1791.00
940.87
938.27
Gravitar
557.17
753.50
449.00
495.17
lecHockey
-4.12
-4.83
-3.61
-4.61
Jamesbond
527.17
488.17
685.17
901.67
Kangaroo
3891.67
6845.00
1850.00
1214.67
Krull
7715.68
8329.08
7204.95
4881.65
KungFuMaster
33728.00
29958.67
29843.67
26808.00
Montezuma- Contezuma-RRevene
0.00
10.67
0.67
0.00
MsPacman
1683.87
1981,50
1170,70
1133.57
NameThisGame
6065.63
5397.47
5672.60
5604.10
Pitfall
0.00
-2.32
-17.26
-43,60
Pong
20.50
20.80
20.79
19.63
PrivateEye
79.67
36.50
99.67
99.33
Qbert
15396.67
14556,83
4114.00
3781.88
Riverraid
8052.23
7360.40
7722.00
6773.67
RoadRunner
44679.67
36289.33
43626.33
2.061.33
Robotank
4.60
14.15
24.60
24.18
Seaquest
1807.47
1470.60
1501.47
926.40
SpaceInvaders
1216.15
944.63
814,53
634.07
StarGunner
48984.00
33862.00
47738.00
33442.67
Tennis
-8.32
-13.74
-19.13
-18.40
TimePilot
3770.33
$321.33
6278.33
5701.00
Tutankham
241.21
177.58
135.80
136.21
UpNDown
242701.51
153160.66
11815.87
10949.53
Venture
36.33
0.00
4,00
0.00
VidcoPinball
37780.70
31577.24
21438.64
25095.20
WizardOfWor
4704.00
4886.67
3533.67
3103.00
Zaxxon
9472.00
5728.67
1179.67
4796.67
12

---

Under review as a conference paper at ICLR 2021
Table 4: All episodes
of PPO, POP3D, BASELINE and TRPO Atari games after
40M frames.
The results are averaged by three trials.
game
POP3D
PPO
BASELINE
TRPO
Alien
1147.29
1115.94
851.13
841.08
Amidar
299.55
413.46
295.91
169.12
Assault
2139.15
2168.93
1159.50
971.78
Asterix
2004.43
2102.10
1884.68
1342.83
Asteroids
1652.48
1470.46
1477.71
1760.73
Atlantis
488134.03
596807.27
192798.74
174394.94
BankHeist
662.26
643.94
859.25
831.95
BattleZone
11131.44
9387.77
11674.30
12918.39
BeamRider
1965.27
1460.59
3321.25
2431.63
Bowling
37.97
39.41
33.90
30.99
Boxing
83.12
78.61
27.92
23.07
Breakout
143.60
124.98
29.99
26.56
Centipede
3056.81
3344.63
3042.48
3142.22
Chopper-
Command
3269.47
3106.14
1780.38
1595.82
CrazyClimber
97257.52
90169.60
69258.31
63189.78
DemonAttack
7611.27
7180.43
9814.42
6204.68
DoubleDunk
-13.70
-15.45
-15.93
-14,57
Enduro
107.84
321.20
92.59
140.67
FishingDerby
-21.00
-27.51
81.90
-81.77
Freeway
17.76
15.87
15.93
17.33
Frostbite
276.47
267.73
270.42
270.57
Gopher
1556.29
1196.20
900.74
875.93
Gravitar
413.20
509.81
342.74
317.86
IceHockey
-4.67
-5.50
.4.11
-5.21
Jamesbond
358.54
394.45
380.91
519.01
Kangaroo
1614.63
2199.74
937.98
566.85
Krull
6538.16
7195.24
4760.66
3861.87
KungFuMaster
23253.96
23283.31
19637.58
18293.12
Montezuma-
Revenge
0.14
0.74
0.22
0.12
MsPacman
1214.09
1482.77
860.63
864.84
NameThisGame
5353.14
5199.37
4562.32
4504.67
Pitfall
-2.41
-31.27
-33.93
Pong
13.24
12.83
7.20
-2.91
PrivateEye
87.37
52.76
56.70
98.79
Qbert
5852.10
6744.13
1760.92
1679.03
Riverraid
5260.89
5487.17
5220.64
4549.22
RoadRunner
25456.31
24688.07
20385.91
16269.40
Robotank
3.08
8.65
13.89
14.57
Seaquest
1487.84
1120.15
1112.51
848.47
SpaceInvaders
693.26
632.17
552.50
483.48
StarGunner
14734.11
13643.80
16288.35
13341.23
Tennis
-19.86
-21.80
-2.84
-21.04
TimePilot
3396.61
4410.87
4718.46
4544.68
Tutankham
179.96
152.72
103.95
109.18
UpNDown
38728.48
43208.99
5430.22
7085.02
Venture
15.89
0.57
0.03
VideoPinball
27346.44
27549.55
23998.09
23705.39
WizardO/Wor
2340.60
2743.40
2409.94
2045.17
Zaxxon
3739.56
1813.90
256.78
1521.28
13

---

Under review as a conference paper at ICLR 2021
Table 5: All episodes mean scores of PPO, POP3D on Mujoco games after 10M frames. The results are averaged by three trials.
game
PPO
POP3D
HalfCheetah
3250.22
2373.30
Hopper
1767.14
1257.72
InvericdDDoulleeenduuum
3684.92
2561.77
InvertedPendulum
531.77
552.98
Reacher
-5.94
-8.05
Swimmer
94.01
108.27
Walker2d
1770.37
2439.54
B EXPERIMENTS
B. HYPER-PARAMETER
B.1.1 ATARI
PPO's and POP3D's paramepassmeters for Mujoco games are respectively listed in Table 6.
Table 6: Left: PPO's hyper-parameters for Atari games. Right:POP3D's parametrrs for Atari
games.
Value
Hyper-parameter
Value
Horizon (T)
128
Horizon (T)
128
Adam step-size
2.5 x x10 x a
Adam step-size
2.5 x10 x a
Num epochs
3
Num epochs
Mini-batch size
32x8
Mini-batch size
32x8
Discount (7)
0.99
Discount (7)
0.99
GAE parameter (A)
0.95
GAE parameter (A)
0.95
Number of actors
8
Number of actors
Clipping parameter VF coeff.
0.1 x
VF coeff.
Entropy coeff.
0.01
Entropy coeff.
0.01
KL penalty coeff.
5.0
Table 7: BASELINE's hyper-parameters for Atari games.
Hyper-parameter
Value
Horizon (T)
128
Adam step-size
2.5 x10
Num epochs
3
Mini-batch size
32x8
Discount (7)
0.99
GAE parameter (A)
0.95
Number of actors
8
VF coeff.
1
Entropy coeff.
0.01
KI. penalty coeff.
10.0
B.1.2 MUJOCO
PPO's and POP3D's hyper-parameters for Mujoco games respectively listed in Table 8.
14

---

Under review as a conference paper at ICLR 2021
Table 8: Left: PPO's hyper-parameters for Mujoco games. RiPhDss hyper-parameters for
Mujoco games.
Hyper-parameter
Value
Hyper-parameter
Value
Horizon (T)
2048
Horizon (T)
2048
Adam step-size
3
x10-1
Adam step-size
3 x00-4
Num epochs
10
Num epochs
10
Mini-bater size
64
Mini-batch size
64
Discount (r)
0.99
Discount (7)
0.99
GAE parameter (A)
0.95
GAE parameter (A)
0.95
Clipping parameter
0.2
KL penalty coeff.
5.0
15

---

Under review as a conference paper at ICLR 2021
Amtider
Astafic
2000
4c00
1000
3000
1000
1100
Adanta
Sandtiaat
Bonding
2330000
2003
22000
130000
1000
Ending
Erealiet
Contieds
CheppnCCrmmand
CranyClmverr
335300
4000
60-
4C00
2000
25300
Coublbbnkc
Fuch Deringbertyy
Fresuey
40000
400
-25
-23
-300
Freebike
-00-
6003
500
2003
Yangaraa
Kangl
Melacman
12500
8003
2300
0003
23000
5000
4003
4000
200
0.0
-353
-00
2000
Rabwrark
Sangurd
3000
50803
1250
6000
12
1080
12003
2000
250
50000
200
1000
10000
Venture
Plance
ccoo3
10000
30000
ocea
S5c IP
Pranes
Figure 1: Score curves of three methods on Atari games within 40 million frame steps.
16

---

Under review as a conference paper at ICLR 2021
Figure 2: Score curves Mujoco games within 10 million frame steps.
17


Reviews:
[{"review_id": "cww45XcaZrp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discusses different methods in robust reinforcement learning including TRPO and PPO then propose a variant of TRPO using a new type of regularizer to quantify the difference between two distributions.\n\nStrength: \n- The paper is easy to understand and provide informative discussion on related methods.\n- The point probability distance is simple but appears to work well.\n- The numerical experiments are extensive which include discrete and continuous control tasks.\n\nWeakness:\n- There is lack of justification why the proposed algorithm converges.\n- Although POP3D appears to be better than PPO and others using the mean final scores metric, it seems to be comparable with PPO when using the all episodes mean scores which raise the concern about the clear improvement of POP3D.\n\nI have the following questions and comments:\nQ1. As the point probability distance is bounded, there is an issue with the scale effect where different environment can produce different range of the cumulative reward (the first term in the objective). In this situation, choosing a common \\beta that works for most environment appears to be challenging to me? How do you deal with this?\n\nQ2. As from Q1, is there a systematic way to select \\beta?\n\nQ3. As the KL divergence is the upper bound of the total variation distance while the point probability distance is the lower bound, POP3D does not inherit from the theoretical result. What is your opinion about this?\n\nC1. Equation (10) appears to be different from the one in the TRPO paper? Are you using a different definition of \\eta(.) function, it is better to state its definition for clarity.\n\nC2. It would be better to have figures illustrating the performance of 4 algorithms in some continuous tasks over the number of episodes collected to see their learning process.\n\n\nSmall comments: \n- The paper needs to be proofread again for some small typos.\n- If the hyper-parameter for POP3D is tuned, it is better if the hyper-parameter for other methods are tuned as well for fair comparison in continuous domain.\n- The point probability distance actually depends on the action taken at that iteration. Do you think the formulation needs to be changed to account for that?\n\nOverall, I suggest a weak-reject decision based on the following reason:\n- The paper lack theoretical discussion on the convergence of the new method.\n- The numerical results do not show significant improvement over existing methods especially PPO as the authors claim.\n- The authors claim that the new method removes the need of choosing preferable penalty coefficient while I still believe the parameter \\beta needs to be suitably chosen for particular environment.\n\nHowever, I can adjust my decision once I see the authors’ response.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "asFHH66gokI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces POP3D, an on-policy policy gradient algorithm that is a variant of TRPO and PPO. While TRPO uses a particular penalty function to keep the policy from being updated too aggressively, POP3D uses an alternative objective function that lower bounds the square of the total variance divergence between two policy distributions. The authors argue that this alternative formulation results in an algorithm that is sample-efficient, like PPO, but that is more effective at keeping policy updates from overshooting. The authors also argue that this new formulation helps users to avoid the arguably challenging process of selecting penalty constants, as required (for instance) by TRPO.\n\nOverall, this was an interesting paper. I believe that the idea of lower bounding the square of the total variance divergence is worth being investigated. The experimental results also seem promising, showing that the method may outperform POP and TRPO on a wide range of games. I do have a few questions about the statistical significance of the results, though: some of the score differences may not be significant, the results might be based on too few trials, and the authors only presented mean performances, but not standard deviation or standard error information. I liked the idea that the proposed lower bound solves possible issues arising from using the KL divergence, which is a measure that is not symmetric. On the other hand, after presenting the surrogate objective function's mathematical definition, the authors argued that this new formulation results in better exploration and that it \"expands the manifold [of] solutions\". I had a hard time following these arguments.\n\nFinally, in my opinion, writing could be improved. A few sentences are hard to read or appear to be presenting incomplete thoughts. For instance:\n\n- \"Hessian free strategy: Fisher vector product is utilized to cut down the computing burden.\" (page 1)\n- \"(...) the action is taken approximately strongly corrected with the highest probability value.\" (page 4)\n- \"Therefore, even if theta outputs theta_old the same high probability for the right action, it's still penalized owing to probabilities mismatch for other uncritical actions.\" (page 4)\n- \"From the perspective of the manifold, if the optimal parameters constitute a solution manifold.\" (page 4)\n\nI believe that this is an interesting work, though possibly still showing only preliminary results. I also think that the paper might be improved by clarifying a few of the authors' key mathematical arguments. As it is, and based on the thoughts presented above (and on the questions below), I would say that more work would benefit the paper and would argue for a weak reject. \n\nI have a few questions and comments for the authors:\n\n1) the authors state the \"[POP3D] dives into the mechanism of PPO's improvement over TRPO by the perspective of solution manifold\". Could you please clarify? What is the solution manifold perspective, and how is it used more effectively by POP3D, compared to previous algorithms? At the end of Section 3.3, you present an argument that I found hard to follow: \"From the perspective of the manifold, if the optimal parameters constitute a solution manifold.\". Later on, you also talked about \"[expanding] the solution manifold at least one dimension such as curves to surfaces\". I believe that these ideas may be central to the paper. Still, I had difficulty following the arguments in this section.\n\n2) a quick note on notation: \\hat{A}_t (Eq6) looks like a random variable denoting the action taken in time t. Perhaps the LHS of this equation is missing parameters specifying a (s,a) pair? In particular, note that the advantage function in Eq6 involves computing delta^V_{t+l}, a TD error for a specific experience; the current notation does not make it clear which experiences are being analyzed. Compare, for example, the definition of advantage used in Eq6 and the one used in Eq3.\n\n3) still regarding notation, I wonder if you could clarify which of s, a, s_t, a_t, are random variables and which ones are realizations of the corresponding random variables. In Eq6, for example, why do you write A(s_t, a) instead of A(s_t, a_t)?\n\n4) in Eq10, please formally define D_{TV}.\n\n5) immediately after Eq11, the authors argue that because the KL divergence is not symmetric, the choice of whether to use D_KL(p1, p2) or D_KL(p2, p1) might affect the algorithm's behavior, and that (for this reason) it might not be the best choice for a divergence measure. Could you please discuss the possible advantages of simply using the symmetrized version of this divergence (https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence)?\n\n6) in Eq12, where is the action \"a\", on the RHS, coming from? The inputs to D_pp are two distributions, but its definition seems to consider the difference between two particular probabilities. Could you please clarify the notation being used here?\n\n7) regarding your experiments, I wonder if you could discuss whether the score differences observed in Table 1 are statistically significant.\n\n8) your method does seem to perform better in some particular games, but it does not do so well in other games. A discussion/interpretation of these results would be nice: what are, for instance, the properties of the games that are hard for POP3D?\n\n9) regarding Table 3, could you please discuss whether you can draw strong conclusions regarding the performance difference between PPO and POP3D when analyzing just 3 trials? Also, do you have standard deviation information associated with the averages that you could present?\n\n10) one key point that the authors make is that POP3D encourages exploration by \"expanding the optimal solution manifold\". More efficient exploration is indeed advantageous, but I did not follow the argument being made here. Could you please clarify why POP3D results in better exploration?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PH6pFJEpp94", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors replace the divergence-based constraint in trust region policy optimization model with an alternate distance measure, which is added to the objective function with a multiplier (beta).  In fact, the parameter beta plays a role that is similar to a Lagrange multiplier, if the new distance measure is introduced as a constraint. The authors explain the shortcomings of KL-divergence and the solutions obtained with other methods but they do not provide a sufficient discussion how and why their simple approach overcomes those concerns. For instance, why would the new measure encourage exploration and what is the effect of large beta value on this?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper contains some interesting ideas, the reviewers felt that overall the paper is not theoretical well supported, and likewise the experiments are not fully convincing. Even after the rebuttal, these concerns still persist.
Decision: Reject