Year: 2020

Abstract:
As a recognized variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely used with several advantages: efficient data utilization, easy implementation, and good parallelism. In this paper, a first-order gradient reinforcement learning algorithm called Policy Optimization with Penalized Point Probability Distance (POP3D), which is a lower bound to the square of total variance divergence, is proposed as another powerful variant. The penalty item has dual effects, prohibiting policy updates from overshooting and encouraging more explorations. By carefully controlled experiments on both discrete and continuous benchmarks, our approach is proved highly competitive to PPO.

heading: 1 INTRODUCTION
text: With the development of deep reinforcement learning, lots of impressive results have been produced in a wide range of fields such as playing Atari game (Mnih et al., 2015; Hessel et al., 2018), controlling robotics (Lillicrap et al., 2015), Go (Silver et al., 2017), neural architecture search (Tan et al., 2019; Pham et al., 2018).
The basis of a reinforcement learning algorithm is generalized policy iteration (Sutton & Barto, 2018), which states two essential iterative steps: policy evaluation and improvement. Among various algorithms, policy gradient is an active branch of reinforcement learning whose foundations are Policy Gradient Theorem and the most classical algorithm REINFORCEMENT (Sutton & Barto, 2018). Since then, handfuls of policy gradient variants have been proposed, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., 2016), Actor-Critic using Kronecker-factored Trust Region (ACKTR) (Wu et al., 2017), and Proximal Policy Optimization (PPO) (Schulman et al., 2017).
Improving the strategy monotonically had been nontrivial until Schulman et al. (2015) proposed Trust Region Policy Optimization (TRPO), in which Fisher vector product is utilized to cut down the computing burden. Specifically, Kullback–Leibler divergence (KLD) acts as a hard constraint in place of objective, because its corresponding coefficient is difficult to set for different problems. However, TRPO still has several drawbacks: too complicated, inefficient data usage. Quite a lot of efforts have been devoted to improving TRPO since then and the most commonly used one is PPO.
PPO can be regarded as a first-order variant of TRPO and have obvious improvements in several facets. In particular, a pessimistic clipped surrogate objective is proposed where TRPO’s hard constraint is replaced by the clipped action probability ratio. In such a way, it constructs an unconstrained optimization problem so that any first-order stochastic gradient optimizer can be directly applied. Besides, it’s easier to be implemented and more robust against various problems, achieving an impressive result on Atari games (Brockman et al., 2016). However, the cost of data sampling is not always cheap. Haarnoja et al. (2018) design an off-policy algorithm called Soft Actor-Critic and achieves the state of the art result by encouraging better exploration using maximum entropy.
In this paper, we focus on the on-policy improvement to improve PPO and answer the question: how to successfully leverage penalized optimization to solve the constrained one which is formulated by Schulman et al. (2015).
1. It proposes a simple variant of TRPO called POP3D along with a new surrogate objective containing a point probability penalty item, which is symmetric lower bound to the square of the total variance divergence of policy distributions. Specifically, it helps to stabilize
the learning process and encourage exploration. Furthermore, it escapes from penalty item setting headache along with penalized version TRPO, where is arduous to select one fixed value for various environments.
2. It achieves state-of-the-art results among on-policy algorithms with a clear margin on 49 Atari games within 40 million frame steps based on two shared metrics. Moreover, it also achieves competitive results compared with PPO in the continuous domain. It dives into the mechanism of PPO’s improvement over TRPO from the perspective of solution manifold, which also plays an important role in our method.
3. It enjoys almost all PPO’s advantages such as easy implementation, fast learning ability.
We provide the code and training logs to make our work reproducible.

heading: 2 PRELIMINARY KNOWLEDGE AND RELATED WORK

heading: 2.1 POLICY GRADIENT
text: Agents interact with the environment and receive rewards which are used to adjust their policy in turn. At state st, one agent takes strategy π and transfers to a new state st+1, rewarded rt by the environment. Maximizing discounted return (accumulated rewards) Rt is its objective. In particular, given a policy π, Rt is defined as
Rt = ∞∑ n=0 (rt + γrt+1 + γ 2rt+2 + ...+ γ nrt+n). (1)
γ is the discounted coefficient to control future rewards, which lies in the range (0, 1). Regarding a neural network with parameter θ, the policy πθ(a|s) can be learned by maximizing Equation 1 using the back-propagation algorithm. Particularly, given Q(s, a) which represents the agent’s return in state s after taking action a, the objective function can be written as
max θ
Es,a log πθ(a|s)Q(s, a). (2)
Equation 2 lays the foundation for handfuls of policy gradient based algorithms. Another variant can be deduced by using A(s, a) = Q(s, a)− V (s) (3) to replace Q(s, a) in Equation 2 equivalently, V (s) can be any function so long as V depends on s but not a. In most cases, state value function is used for V , which not only helps to reduce variations but has clear physical meaning. Formally, it can be written as
max θ
Es,a log πθ(a|s)A(s, a). (4)

heading: 2.2 ADVANTAGE ESTIMATE
text: A commonly used method for advantage calculation is one-step estimation, which follows A(st, at) = Q(st, at)− V (st) = rt + γV (st+1)− V (st). (5)
However, a more accurate method called generalized advantage estimation is proposed in Schulman et al. (2016), where all time steps of estimation are combined and summarized using λ-based weights,. The generalized advantage estimator ÂGAE(γ,λ)t is defined by Schulman et al. (2016) as
Â GAE(γ,λ) t := (1− λ) ∗ (Â (1) t + λÂ (2) t + λ 2Â (3) t + . . .) = ∞∑ l=0 (γλ)lδVt+l
δVt+l = rt+l + γV (st+l+1)− V (st+l).
Â (k) t := k−1∑ l=0 γlδVt+l = −V (st) + rt + γrt+1 + · · ·+ γk−1rt+k−1 + γkV (st+k)
(6)
The parameter λ meets 0 ≤ λ ≤ 1, which controls the trade-off between bias and variance. All methods in this paper utilize ÂGAE(γ,λ)t to estimate the advantage.

heading: 2.3 TRUST REGION POLICY OPTIMIZATION
text: Schulman et al. (2015) propose TRPO to update the policy monotonically. In particular, its mathematical form is
max θ Et[ πθ(at|st) πθold(at|st) Ât]− CEt[KL[πθold(·|st), πθ(·|st)]]
= max s Ea∼πθ(a|s)[Aπθold (s, a)])
(7)
where C is the penalty coefficient, C = 2 γ(1−γ)2 .
In practice, the policy update steps would be too small if C is valued as Equation 7. In fact, it’s intractable to calculate C beforehand since it requires traversing all states to reach the maximum. Moreover, inevitable bias and variance will be introduced by estimating the advantages of old policy while training. Instead, a surrogate objective is maximized based on the KLD constraint between the old and new policy, which can be written as below,
max θ Et[ πθ(at|st) πθold(at|st) Ât]
s.t. Et[KL[πθold(·|st), πθ(·|st)]] ≤ δ (8)
where δ is the KLD upper limitation. In addition, the conjugate gradient algorithm is applied to solve Equation 8 more efficiently. Two major problems have yet to be addressed: one is its complexity even using the conjugate gradient approach, another is compatibility with architectures that involve noise or parameter sharing tricks (Schulman et al., 2017).

heading: 2.4 PROXIMAL POLICY OPTIMIZATION
text: To overcome the shortcomings of TRPO, PPO replaces the original constrained problem with a pessimistic clipped surrogate objective where KL constraint is implicitly imposed. The loss function can be written as
LCLIP (θ) = Et[min(rt(θ)Ât, clip(rt(θ), 1− , 1 + )Ât)]
rt(θ) = πθ(at|st) πθold(at|st) ,
(9)
where is a hyper-parameter to control the clipping ratio. Except for the clipped PPO version, KL penalty versions including fixed and adaptive KLD. Besides, their simulation results convince that clipped PPO performs best with an obvious margin across various domains.

heading: 3 POLICY OPTIMIZATION WITH PENALIZED POINT PROBABILITY DISTANCE
text: Before diving into the details of POP3D, we review some drawbacks of several methods, which partly motivate us.

heading: 3.1 DISADVANTAGES OF KULLBACK-LEIBLER DIVERGENCE
text: TRPO (Schulman et al., 2015) induced the following inequality1,
η(πθ) ≤ Lπθold (πθ) + 2 γ (1− γ)2 α2
α = DmaxTV (πθold , πθ)
DmaxTV (πθold , πθ) = maxs DTV (πθold ||πθ)
(10)
TRPO replaces the square of total variation divergence DmaxTV (πθold , πθ) by D max KL (πθold , πθ) = maxsDKL(πθold ||πθ). 1Note that η means loss instead of return as the ICML version (Schulman et al., 2015).
Given a discrete distribution p and q, their total variation divergence DTV (p||q) is defined as
DTV (p||q) := 1
2 ∑ i |pi − qi| (11)
in TRPO (Schulman et al., 2015). Obviously, DTV is symmetric by definition, while KLD is asymmetric. Formally, given state s, KLD of πθold(·|s) for πθ(·|s) can be written as
DKL(πθold(·|s)||πθ(·|s)) := ∑ a πθold(a|s) ln πθold(a|s) πθ(a|s) . (12)
Similarly, KLD in the continuous domain can be defined simply by replacing summation with integration. The consequence of KLD’s asymmetry leads to a non-negligible difference of whether choose DKL(πθold ||πθ) or DKL(πθ||πθold). Sometimes, those two choices result in quite different solutions. Robert compared the forward and reverse KL on a distribution, one solution matches only one of the modes, and another covers both modes (Murphy, 2012). Therefore, KLD is not an ideal bound or approximation for the expected discounted cost.

heading: 3.2 DISCUSSION ABOUT PESSIMISTIC PROXIMAL POLICY
text: In fact, PPO is called pessimistic proximal policy optimization2 in the meaning of its objective construction style. Without loss of generality, supposing At > 0 for given state st and action at, and the optimal choice is a?t . When at = a ? t , a good update policy is to increase the probability of action to a relatively high value a?t by adjusting θ. However, the clipped item clip(rt(θ), 1− , 1 + )Ât will fully contribute to the loss function by the minimum operation, which ignores further reward by zero gradients even though it’s the optimal action. Other situation with At < 0 can be analyzed in the same manner.
However, if the pessimistic limitation is removed, PPO’s performance decreases dramatically (Schulman et al., 2017), which is again confirmed by our preliminary experiments. In a word, the pessimistic mechanism plays a very critical role for PPO in that it has a relatively weak preference for a good action decision at a given state, which in turn affects its learning efficiency.

heading: 3.3 RESTRICTED SOLUTION MANIFOLD FOR EXACT DISTRIBUTION MATCHING
text: To be simple, we don’t take the model identifiability issues along with deep neural network into account here because they don’t affect the following discussion much (LeCun et al., 2015). Suppose πθ? is the optimal solution for a given environment, in most cases, more than one parameter set for θ can generate the ideal policy, especially when πθ? is learned by a deep neural network. In other words, the relationship between θ and πθ? is many to one. On the other hand, when agents interact with the environment using policy represented by neural networks, they prefer to takes the action with the highest probability. Although some strategies of enhancing exploration are applied, they don’t affect the policy much in the meaning of expectation.
RL methods can help agents learn useful policies after fully interacting with the environment. Take Atari-Pong game for example, when an agent sees a Pong ball coming close to the right (state s1), its optimal policy is moving the racket to the right position (for example, the "RIGHT" action) with a distribution ps1θ1 = [0.05, 0.05, 0.1, 0.7, 0.05, 0.05]
3. The probability of selecting "RIGHT" is a relatively high value such as 0.7. It’s almost impossible to push it to be 1.0 exactly since it’s produced by a softmax operation on several discrete actions. In fact, we hardly obtain the optimal solution accurately. Instead, our goal is to find a good enough policy. In this case, the policy of pushing p(RIGHT|s1) above a threshold is sufficient to be a good one. In other words, paying attention to the most critical actions is sufficient, and we don’t care much the probability value of the other non-critical actions. For example, a good policy at s1 is [?,?, ≥ 0.7, ?,?,?]. Note that πθ(a|s) is represented by a neural network parameterized using θ and a good policy for the whole game means
2The word “pessimistic” is used by the PPO paper. 3The action space is described as [‘NOOP’, ‘FIRE’, ‘RIGHT‘, ‘LEFT’, ‘RIGHTFIRE’, ‘LEFTFIRE’].
that the network can perform well across the whole state space. Focusing on those critical actions at each state4 and ignoring non-critical ones can help the network learn better and more easily.
Using a penalty such as KLD cannot utilize this good property, because it involves all of the actions’ probabilities. Moreover, it doesn’t stop penalizing unless two distributions become exactly indifferent or the advantage item is large enough to compensate for the KLD cost. Therefore, even if θ outputs θold the same high probability for the right action, the penalization still exists. Suppose that two parameters for θ1: θ2 and θ3, where ps1θ2 = [0.01, 0.15, 0.05, 0.7, 0.01, 0.08] and p s1 θ3
= [0.01, 0.01, 0.01, 0.7, 0.26, 0.01]. When the agent already chooses RIGHT at S1, the loss item from a good penalized distance should be small. However, DKL(πθ1(·|s1)||πθ2(·|s1))=0.15 and DKL(πθ1(·|s1)||πθ3(·|s1))=0.39. However, it’s not necessary to require the distribution of other actions (‘NOOP’, ‘FIRE’, ‘LEFT’, ‘RIGHTFIRE’, ‘LEFTFIRE’) of ps1θ2 near to p s1 θ1
. Instead, it’s better to relax this requirement to enlarge the freedom degree of the network and focus on learning important actions. Doing this brings another advantage, the agent can explore more for non critical actions. From the perspective of the manifold, optimal parameters constitute a solution manifold. The KLD penalty will act until θ exactly locates in the solution if possible, akin to mapping a point onto a curve. Instead, if the agent concentrates only on critical actions like a human does, it’s much easier to approach the manifold in a higher dimension. This is comparable to expanding the solution manifold by at least one dimension, e.g. from curves to surfaces or from surfaces to spheres.

heading: 3.4 EXPLORATION
text: One shared highlight in reinforcement learning is the balance between exploitation and exploration. For a policy-gradient algorithm, entropy is added in the total loss to encourage exploration in most cases. When included in the loss function, KLD penalizes the old and new policy probability mismatch for all possible actions as Equation 12 given a state s. This strict punishment for every action’s probability mismatch, which discourages exploration.

heading: 3.5 POINT PROBABILITY DISTANCE
text: To overcome the above-mentioned shortcomings, we propose a surrogate objective with the point probability distance penalty, which is symmetric and more optimistic than PPO. In the discrete domain, when the agent takes action a, the point probability distance between πθold(·|s) and πθ(·|s) is defined by
Dapp(πθold(·|s), πθ(·|s)) = (πθold(a|s)− πθ(a|s))2. (13)
Attention should be paid to the penalty definition item, the distance is measured by the point probability, which emphasizes its mismatch for the sampled actions for a state. Unless it would lead to confusion, we omit a for simplicity in the following sections. Undoubtedly, Dpp is symmetric by definition. Furthermore, it can be proved that Dpp is indeed a lower bound for the total variance divergence DTV . As a special case, it can be easily proved that for binary distribution, D2TV (p||q) = Dpp(p||q).
Theorem 3.1. For two discrete probability distributions p and q with K values, then D2TV (p||q) ≥ Dapp(p||q) holds for any action a and EaDapp(p||q) is a lower bound for D2TV (p||q).
4Note that some states don’t have critical action. Taking the Pong for example, when the ball is just shot back, the agent can choose any action.
Proof. Let pl = α, ql = β for the l-th action a, and suppose a ≥ b without loss of generalization. So,
D2TV (p||q) = ( 1
2 K∑ i=1 |pi − qi|)2 = ( 1 2 K∑ i=1,i6=l |pi − qi|+ 1 2 |pl − ql|)2
≥ (1 2 | K∑ i=1,i6=l pi − qi|+ 1 2 (α− β))2 = (1 2 |1− α− (1− β)|+ 1 2 (α− β))2
= ( 1 2 (α− β) + 1 2 (α− β))2 = Dapp(p||q)
EaDapp(p||q) = ∑ a p(a)Dapp(p||q) ≤ ∑ a p(a)D2TV (p||q) = D2TV (p||q)
Since 0 ≤ πθ(a|s) ≤ 1 holds for discrete action space, Dpp has a lower and upper boundary: 0 ≤ Dpp ≤ 1. Moreover, Dpp is less sensitive to action space dimension than KLD, which has a similar effect as PPO’s clipped ratio to increase robustness and enhance stability. Equation 13 stays unchanged for the continuous domain, and the only difference is πθ(a|s) represents point probability density instead of probability.

heading: 3.6 POP3D
text: After we have defined the point probability distance, we use a new surrogate objective fθ for POP3D, which can be written as
max θ Et[ πθ(at|st) πθold(at|st) Ât − βDatpp(πθold(·|st), πθ(·|st))], (14)
where β is the penalized coefficient. These combined advantages lead to considerable performance improvement, which escapes from the dilemma of choosing preferable penalty coefficient. Besides, we use generalized advantage estimates to calculate Ât. Algorithm 1 shows the complete iteration process of POP3D. Moreover, it possesses the same computing cost and data efficiency as PPO.
Algorithm 1 POP3D 1: Input: max iterations L , actors N , epochs K 2: for iteration = 1 to L do 3: for actor = 1 to N do 4: Run policy πθold for T time steps 5: Compute advantage estimations Â1, ..., ÂT 6: end for 7: for epoch = 1 to K do 8: Optimized loss objective f (θ) w.r.t θ with mini-batch size M ≤ NT , then update θold ← θ. 9: end for 10: end for

heading: 3.7 WORKING MECHANISM OF POP3D
text: As for the toy example in Section 3.3, DRIGHTpp (πθ1(·|s) ||πθ2(·|s)=DRIGHTpp (πθ1(·|s) ||πθ3(·|s)=0. Therefore, it can help the agent to focus on the important action. When updating θ from θold as Equation 14, the gradient f(θ) w.r.t. θ can be written as
∇θf(θ) = ∇θπθ(at|st) πθold(at|st) Ât − 2β[πθ(at|st)− πθold(at|st)]∇θπθ(at|st)
= ∇θπθ(at|st)[ Ât
πθold(at|st) − 2β(πθ(at|st)− πθold(at|st))]
= ∇θπθ(at|st)[ Ât
πθold(at|st) − 2βδ(at|st)]
(15)
where δ(at|st) := πθ(at|st)−πθold(at|st). Suppose the agent selects at at st using πθold and obtains a positive advantage Ât , if πθ(at|st) is larger than πθ(at|st), then 2βδ(at|st) will play a damping role to avoid too greedy preference for at (i.e. too large probability), which in turn leaves more space for other actions to be explored. Other cases such as negative Ât can be analyzed similarly. The hyper-parameter β controls the damping force.
In the early stage of learning, πθold(at|st) is near 1/K (taking K discrete spaces for example) and the magnitude of Ât is large, while the damping force is a bit weak. Therefore, the agent learns fast. Then β shows a relative stronger force to avoid overshooting for action selection and encourage more exploration. As for the final stage, the policy changes slowly because the learning rate is low, where δ(at|st) is small and therefore it converges.

heading: 3.8 RELATIONSHIP WITH PPO
text: To conclude this section, we take some time to see why PPO works by taking the above viewpoints into account. When we pour more attention to Equation 9, the ratio rt(θ) only involves the probability for given action a, which is chosen by policy π. In other words, all other actions’ probabilities except a are not activated, which no longer contribute to back-propagation and allow probability mismatch, which encourage exploration. This procedure behaves similarly to POP3D, which helps the network to learn more easily. Above all, POP3D is designed to conform with the regulations for overcoming above mentioned problems, and in the next section experiments from commonly used benchmarks will evaluate its performance.

heading: 4 EXPERIMENTS

heading: 4.1 CONTROLLED EXPERIMENTS SETUP
text: OpenAI Gym is a well-known simulation environment to test and evaluate various reinforcement algorithms, which is composed of both discrete (Atari) and continuous (Mujoco) domains (Brockman et al., 2016). Most recent deep reinforcement learning methods such as DQN variants (Van Hasselt et al., 2016; Wang et al., 2016; Schaul et al., 2015; Bellemare et al., 2017; Hessel et al., 2018), A3C, ACKTR, PPO are evaluated using only one set of hyper-parameters5. Therefore, we evaluate POP3D’s performance on 49 Atari games(v4, discrete action space ) and 7 Mujoco (v2, continuous).
Since PPO is a distinguished RL algorithm which defeats various methods such as A3C, A2C ACKTR, we focus on a detailed quantitative comparison with fine-tuned PPO. And we don’t consider large scale distributed algorithms Apex-DQN (Horgan et al., 2018) and IMPALA (Espeholt et al., 2018), because we concentrate on comparable and fair evaluation, while the latter is designed to apply with large scale parallelism. Nevertheless, some orthogonal improvements from those methods have the potentials to improve our method further. Furthermore, we include TRPO to acts as a baseline method. Engstrom et al. (2020) carefully study the underlying factor that helps PPO outperform TPRO. To avoid unfair comparisons, we carefully control the settings. In addition, quantitative comparisons between KLD and point probability penalty helps to convince the critical role of the latter, where the former strategy is named fixed KLD in Schulman et al. (2017) and can act as another good baseline in this context, named by BASELINE below.
5DQN variants are evaluated in Atari environment since they are designed to solve problems about discrete action space. However, policy gradient-based algorithms can handle both continuous and discrete problems.
In particular, we retrained one agent for each game with fine-tuned hyper-parameters6. To avoid the problems of reproduction about reinforcement algorithms mentioned in Henderson et al. (2018), we take the following measures:
• Use the same training steps and make use of the same amount of game frames(40M for Atari game and 10M for Mujoco).
• Use the same neural network structures, which is the CNN model with one action head and one value head for the Atari game, and a fully-connected model with one value head and one action head which produces the mean and standard deviation of diagonal Gaussian distribution as PPO.
• Initialize parameters using the same strategy as PPO.
• Keep Gym wrappers from Deepmind such as reward clipping and frame stacking unchanged for Atari domain, and enable 30 no-ops at the beginning of each episode.
• Use Adam optimizer (Kingma & Ba, 2014) and decrease α linearly from 1 to 0 for Atari domain as PPO.
To facilitate further comparisons with other approaches, we release the seeds and detailed results7(across the entire training process for different trials). In addition, we randomly select three seeds from {0, 10, 100, 1000, 10000} for two domains, {10,100,1000} for Atari and {0,10,100} for Mujoco in order to decrease unfavorable subjective bias stated in Henderson et al. (2018).

heading: 4.2 EVALUATION METRICS
text: PPO utilizes two score metrics for evaluating agents’ performance using various RL algorithms. One is the mean score of the last 100 episodes Score100, which measures how high a strategy can hit eventually. Another is the average score across all episodes Scoreall, which evaluates how fast an agent learns. In this paper, we conform to this routine and calculate individual metric by averaging three seeds in the same way.

heading: 4.3 DISCRETE DOMAIN COMPARISONS
text: Hyper-parameters We search hyper-parameter four times for the penalty coefficient β based on four Atari games while keeping other hyper-parameters unchanged as PPO and fix β = 5.0 to train all Atari games. For BASELINE, we also search hyper-parameter four times on penalty coefficient β and choose β = 10.0. To save space, detailed hyper-parameter setting can be found in Table 6 and 7.
This process is not beneficial for POP3D owing to missing optimization for all hyper-parameters. There are two reasons to make this choice. On the one hand, it’s the simplest way to make a relatively fair comparison group such as keeping the same iterations and epochs within one loop to our knowledge. On the other hand, this process imposes low search requirements for time and resources. That’s to say, we can draw a conclusion that our method is at least competitive to PPO if it performs better on benchmarks.
Comparisons The final score of each game is averaged by three different seeds and the highest is in bold. As Table 1 shows, POP3D outperforms 32 across 49 Atari games given the final score, followed by PPO with 11, BASELINE with 5, and TRPO with 1. Interestingly, for games that POP3D score highest, BASELINE score worse than PPO more often than the other way round, which means that POP3D is not just an approximate version of BASELINE.
For another metric, POP3D wins 20 out of 49 Atari games which matches PPO with 18, followed by BASELINE with 6, and last ranked by TRPO with 5. If we measure the stability of an algorithm by the score variance of different trials, POP3D scores high with good stability across various seeds. And PPO behaves worse in Game Kangaroo and UpNDown. Interestingly, BASELINE shows a large variance for different seeds for several games such as BattleZone, Freeway, Pitfall, and Seaquest. POP3D reveals its better capacity to score high and similar fast learning ability in this domain. The detailed metric for each game is listed in Table 3 and 4.
6We use OpenAI’s PPO and TRPO code: https://github.com/openai/baselines.git 7https://drive.google.com/file/d/1c79TqWn74mHXhLjoTWaBKfKaQOsfD2hg/view

heading: 4.4 CONTINUOUS DOMAIN COMPARISONS
text: Metric PPO POP3D
Score100 1 6 Scoreall 4 3
Hyper-parameters For PPO, we use the same hyperparameter configuration as Schulman et al. (2017). Regarding POP3D, we search on two games three times and select 5.0 as the penalty coefficient. More details about hyper-parameters for PPO and POP3D are listed in Table 8. Unlike the Atari domain, we utilize the constant learning rate strategy as Schulman et al. (2017) in the continuous domain instead of the linear decrease strategy.
Comparison Results The scores are also averaged on three trials and summarized in Table 1. POP3D occupies 6 out of 7 games on Score100. Evaluation metrics of both across different games are illustrated in Table 2 and 5. In summary, both metrics indicate that POP3D is competitive to PPO in the continuous domain.
5 CONCLUSION
In this paper, we introduce a new reinforcement learning algorithm called POP3D (Policy Optimization with Penalized Point Probability Distance), which acts as a TRPO variant like PPO. Compared with KLD that is an upper bound for the square of total variance divergence between two distributions, the penalized point probability distance is a symmetric lower bound. Besides, it equivalently expands the optimal solution manifold effectively while encouraging exploration, which is a similar mechanism implicitly possessed by PPO. The proposed method not only possesses several critical improvements from PPO but outperforms with a clear margin on 49 Atari games from the respective of final scores and meets PPO’s match as for fast learning ability.
More interestingly, it not only suffers less from the penalty item setting headache along with TRPO, where is arduous to select one fixed value for various environments but outperforms fixed KLD baseline from PPO. In summary, POP3D is highly competitive and an alternative to PPO.

heading: A SCORE TABLES AND CURVES
text: Mean scores of various methods for Atari domain are listed in Table 3 and 4.

heading: B EXPERIMENTS
text: B.1 HYPER-PARAMETERS
B.1.1 ATARI
PPO’s and POP3D’s hyper-parameters for Mujoco games are respectively listed in Table 6.
B.1.2 MUJOCO
PPO’s and POP3D’s hyper-parameters for Mujoco games are respectively listed in Table 8.

Review summaries:
The authors replace the divergence-based constraint in trust region policy optimization model with an alternate distance measure, which is added to the objective function with a multiplier (beta).  In fact, the parameter beta plays a role that is similar to a Lagrange multiplier, if the new distance measure is introduced as a constraint. The authors explain the shortcomings of KL-divergence and the solutions obtained with other methods but they do not provide a sufficient discussion how and why their simple approach overcomes those concerns. For instance, why would the new measure encourage exploration and what is the effect of large beta value on this?

This paper introduces POP3D, an on-policy policy gradient algorithm that is a variant of TRPO and PPO. While TRPO uses a particular penalty function to keep the policy from being updated too aggressively, POP3D uses an alternative objective function that lower bounds the square of the total variance divergence between two policy distributions. The authors argue that this alternative formulation results in an algorithm that is sample-efficient, like PPO, but that is more effective at keeping policy updates from overshooting. The authors also argue that this new formulation helps users to avoid the arguably challenging process of selecting penalty constants, as required (for instance) by TRPO.