Memory-Constrained Policy Optimization
Authors: Hung Le
Thommen Karimpanal George
Majid Abdolshah
Dung Nguyen
Kien Do
Sunil Gupta
Svetha Venkatesh
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
trust region policy optimization
on-policy
policy gradient
neural network

Abstract: We introduce a new constrained optimization method for policy gradient reinforcement learning, which uses two trust regions to regulate each policy update. In addition to using the proximity of one single old policy as the first trust region as done by prior works, we propose to form a second trust region through the construction of another virtual policy that represents a wide range of past policies. We then enforce the new policy to stay closer to the virtual policy, which is beneficial in case the old policy performs badly. More importantly, we propose a mechanism to automatically build the virtual policy from a memory buffer of past policies, providing a new capability for dynamically selecting appropriate trust regions during the optimization process. Our proposed method, dubbed as Memory-Constrained Policy Optimization (MCPO), is examined on a diverse suite of environments including robotic locomotion control, navigation with sparse rewards and Atari games, consistently demonstrating competitive performance against recent on-policy constrained policy gradient methods. 

Under review as a conference paper at ICLR 2022
MEMORY-CONSTRAINED POLICY OPTIMIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
We introduce new constrained optimization method for policy gradient rein- forcement learning, which uses (wo Irust regions to regulate each policy update. In addition to using the proximity of one single old policy as the first trust region as done by prior works, we propose to form a second trust region through the construction of another virtual policy that represents a wide range of past poli- cies. We then enforce the new policy to stay closer to the virtual policy, which is beneficial in case the old policy performs badly. More importantly, we propose a mechanism to automatically build the virtual policy from memory buffer of past policies, providing new capability for dynamically selecting appropriate trust re- gions during the optimization process. Our proposed method, dubbed as Memory- Constrained Policy Optimization (MCPO), is examined on a diverse suite of en- vironments including robotic locomotion control, navigation with sparse rewards and Atari games, consistently demonstrating competitive performance against re- cent on- policy constrained policy gradient methods.
INTRODUCTION
Reinforcement learning (RL) combined with neural networks is the current workhorse in machine learning. Using neural networks to approximate value and policy functions enables classical ap- proaches such as Q-learning and policy gradient to achieve promising results on many challenging
problems such as Go, Atari games and robotics (Silver et al. 2017 Mnih al.) 2013 Lillirrap et al. 2016: Mnih et al. 2016). Compared to Deep Q-learning, deep policy gradient (PG) methods are often more flexible and applicable to both discrete and continuous action problems. However, these methods tend to suffer from high sample complexity and training instability since the gradient may not accurately reflect the policy gain when the policy changes substantially (Kakade Lang- Ford 22022). This is exacerbated for deep policy networks where numerous parameters need to be optimized and small updates in parameter space can lead to huge changes in policy space.
To address this issue, one solution is to regularize each policy update by restricting the Kullback- Leibler (KL) divergence between the new policy and the previous one, which can guarantee mono- tonic policy improvement (Schulman et al., 2015a). However, jointly optimizing the approximate advantage function and the KL term does not work in practice. Therefore, SSchulmnn et al. (2013a) proposed Trust Region Policy Optimization (TRPO) to constrain the policy within KI. di- vergence radius, which requires second-order gradients. Alternatives such as Proximal Policy Op- timization (PPO) (Schulman et al. 2017) use simpler first-order optimization with adaptive KL or clipped surrogate objective while still maintaining the reliable performance of TRPO. Recent methods recast the problem through new lens using DnrectaaiionMaximizaztii Optimization. and this also results in first-order optimization with KL divergence term in the loss function (Abdolmaleki et al. 2018 Song et al. 2019 Yang et al., 2019 Tomar et al. 2020).
Mirror Descent
An issue with the above methods is that the previous policy used to restrict the new policy may be suboptimal and thus unreliable in practice. For example, due to stochasticity and imperfect ap- proximations, consider that the new policy may fall into local optimum even under trust-region optimizations. Then in the next update, this policy will become the "previous" policy, and will con- tinue pulling the next policy to stay in the local optimum, thus slowing down the training progress. For on-policy methods using mini- batch updates like PPO. the situation is more complicated as the "previous" policy is defined the old policy that was used to collect data, which can be either very far or close to the current policy. There is no guarantee that the old policy defines a reasonable trust region for regulating the new policy.

---

Under review as a conference paper at ICLR 2022
In this paper, we propose novel constrained policy iteration procedure, dubbed Memory- Constrained Policy Optimization (MCPO), wherein a virtual policy representing past policies takes part in regularizing each policy update. The virtual and the old policy together form two trust gions that attract the new policy by minimizing two KI. divergences. The virtual policy is designed complement the old policy, attracting the new policy more when the old policy performs badly. As such, we assign different coefficient weights to the two KL terms. The coefficient weights are computed dynamically based on the performance of the two policies (the higher performer yields higher coefficient weights). To create the virtual policy, we maintain a memory buffer of past poli- cies, from which we build a mixture of policies. particular, we use neural network-named the atlention network, which takes the context surrounding the current, the old and the virtual policy, to generate the attention weights to each policy in the memory. Then, we use the attention weights to perform a convex combination of the parameters of each policy, forming the virtual policy, The attention network is optimized to maximize the approximate expected advantages of the virtual pol- icy. To train our system, jointly optimize the policy and attention networks, alternating between sampling data from the policy and updating the networks in a mini-batch manner.
We verify our proposed MCPO through diverse set of experiments and compare with the per- formance of recent constrained policy optimization baselines. our experiment on classical control tasks, amongst tested models, MCPO shows minimal sensitivity to hyperparameter changes, con- sistently achieving good performance across tasks and hyperparameters. Our testbed on 6 Mujoco tasks shows that MCPO with big policy memory consistently outperforms others where the atten- tion network plays an important role. We also demonstrate MCPO's ' capability of learning efficiently on sparse reward and high-dimensional problems such as navigation and Atari games. Finally, our ablation study highlights the necessity of each component in MCPO.
BACKGROUND: POLICY OPTIMIZATION WITH TRUST REGION
In this section, we briefly review some fundamental constrained policy optimization approaches. A general idea is to force the new policy Ty to be close to a recent policy Taa.a- this paper, we refer to policy as its parameters (i.e. policy 0 means policy тT).
Conservative Policy Iteration (CPI) The method starts with a basic objective of policy gradient algorithms, which is to maximize the expected advantage Ag.
To (ae (sc) TToor (axl sx))
CPI (0) = E,
A
where the advantage A is function of returns collected from (st, ae) by using Tewra (see Appendix A and E, [-] indicates the empirical average over a finite batch of data. To constrain policy updates, the new policy is a mixture of the old and the greedy policy: Ø = argmax [CPT (0). That is, 0 = aboia + (1- - a) where a is the mixture hyperparameter (Kakade & Langford 2002). As the data is sampled from previous iteration's policy Oold, the objective needs importance sampling estimation. Hereafter, we denote =5(o)]5s) as rat, (0) for short.
(acls))
KL-Regularized Policy Optimization (with fixed adaptive KL coefficient) Another way to enforce the constraint is to jointly maximize the expected advantage and minimize KL divergence between the new and old policy, which ensures monotonie improvement (Schulman et al. 2015a).
KLPO (8) E, ratt (0) A. 3KL [ToLI (-1s:). (-1ss)]
where 33 is hyperparameter that controls the degree of update conservativeness, which can be fixed (KL Fixed) or changed (KL. Adaptive) during training (Schulman et al., 2017).
Trust Region Policy Optimization (TRPO) The method optimizes the expected advantage with hard constraint (Schulman et al.) 2015a). This claimed as a practical implementation that less conservative than the theoretically justified algorithm using KI. regularizer mentioned above.
2

---

Under review as a conference paper at ICLR 2022
LTRPO (0) = rats (0) As st 2 KL (aa (-1st) nh (-1st)]
where 8 the KL constraint radius.
Proximal Policy Optimization (PPO) PPO is family of constrained policy optimization, which uses first-order optimization and mini-batch updates including KI. Adaptive and clipped PPO. In this paper, we use PPO to refer the method that limits the change in policy by clipping the loss function (clipped PPO) (Schulman et al. (017).
L PPO (0) = min (rat, (0) A, clip (rat, (0), -, 1,,1-,1+00, 4))]
where is the clip hyperparameter.
In the above equations, is the currently optimized policy, which is also referred to as the current policy. fold represents past policy, which can be step before the current policy the last policy used to interact with the environment. In either case, the rule to decide Bold fixed throughout training. If for reason, Got is not optimal, it is unavoidable that the following updates will be negatively impacted. We will address this issue in the next section.
MEMORY-CONSTRAINE POLICY OPTIMIZATION (MCPO)
3.1 TWo KL DIVERGENCE CONSTRAINTS
In trust-region methods with mini-batch updates such PPO, the old policy Ootd is often chosen as the last policy that is used to collect observations from the environment. Before the next environment interaction, this old policy is fixed across policy updates, and be the current policy depending on the mini-batch size and the number of update loops. This can be detrimental to the optimization if this old policy is poor in quality, forcing the following updates be close to a poor solution. To tackle this issue, we propose to constrain the new policy not only to the policy Gotd. but also changeable policy that is representative of many past policies. Let ei denote the virtual policy that represents the history of policies. v is dynamically computed based on the past policies using attention mechanism (see Sec. 3.2.. In contrast to the fixed Bold. depending on the attention weights, v can represent a further or closer checkpoint to the current policy than Oatd- use both vp and Ostd to construct the objective function as follows,
many steps before
(0) Re ral, (0) Ar]
Li, [(1 ar (-18)) KL [ToWL (-1ss): mo (-1s))]
(1)
SE, [ar (-1s;) KL (T (-Is:) To (+|s:)]]
where aa (-1sx) is the coefficient weight resembling a forget gate, and 3 the scaling coefficient of the KL constraint terms. In this paper, the expectation is estimated by taking average over / in mini-batch of sampled data.
The forget gate determines how much the new policy should forget the virtual policy from the memory and focus on the lota. Intuitively, if the virtual policy is better than the old policy, the new R2ta)
policy should be kept close to the virtual policy and vice versa. Hence, a, "e where
R2 (1b) measures the performance of the policy th, which can be estimated by weighted importance sampling. That is R2 (v²) rat, (vb) A
Besides deciding which trust region the new policy should rely on, we dynamically weigh the whole KL terms via adjusting B. Using fixed threshold diaro to change (e.g in KL Adaptive, if KL [ (-1a;), To [-|s;)] > drargs increase 3 (Schulman et al., 2017)) showed limited perfor- mance since dearg should vary depending the current learning. We instead make use of da
3

---

Under review as a conference paper at ICLR 2022
E, [KL [Ta (-1st) mb (-1s))]l denote the "distance" between 2 policies Ta and Tà. we propose a switching- rule as follows,
reference for selecting B. Let d (a. b)
3 Imax
if (Oald, 0) > d (Dord, (6)
(2)
3
Sunn otherwise
The intuition is that we encourage the enforcement of the constraint when is too far from To.. using the distance between Mooia and Ty as reference.
Algorithm 1 Seamoory-Constrinee Policy Optimization (for 1 actor).
Require: A policy buffer M, an initial policy To.ia- T, K, B are the learning horizon, number of update epochs, and batch size, respectively.
1: Initialize Wold Oold. +- fold
2: for iteration = 1, do
3: Run policy 7p..4 in environment for T timesteps. Compute advantage estimates Å., ÅT
4: for epoch = 1,2, . do
5: for batch = 1, 2, B/B
6:
Compute (Eq. using doolds 0, Oolds then optimize L L wrt 0 (Eq.
7:
if d (0, vb) > d (Bold. v5) then add 0 to M
8:
if IM| N then remove the last item from M
9:
Bold < 3b
10: end for
11: end for
12: Bold <-8 0
13:
end for
3.2 LEARNING TO GENERATE THE VIRTUAL POLICY
critical compute suitable virtual policy. On one hand, if the virtual policy too far from the currently optimizing point, it will be irrelevant, pulling the new policy back and postponing the learning. On the other hand, if the virtual policy is too recent, it will not complement Aoi and cannot prevent major changes in the policy update. Also, it is reasonable to find a virtual policy that has good performance on current data, Otherwise, using trust regions near poor policies could destroy the learning. We will utilize these intuitions to build the virtual policy.
Policy Memory We first maintain a memory buffer M that stores past policy parameters M [0,))/I During optimization, we add policy's 's parameter to M if it is far enough from th. In particular, measure the distances (0.la, v)) and d (6, vb), then propose conditional writing:
Add to M d (0, > d (Oars, v))
(3)
The memory capacity is N. When the memory is full, discard the earliest policy in M.
Context Vector We hypothesize that the context surrounding Oold Potd and 6, where Pord is the last virtual policy, plays role in determining the next virtual policy. We build the context by extracting specific features: pair-wise distances between policies (d (Oold, Wota), d (0, Dota), d (Oma, 0)), the empirical returns of these policies (B, [Re (Wota)), E, [R, (0)). E, [Re (Pota)); policy entropy (B, lol (Teaa (+1s))) E, log (mo (-Is:))), E, |- log (TT (-1s:)))) and value losses (R,, (Veuea (sx) Vsarget (st)) R. (V, (st) Viarget (s+)) E. (Vosns (st) - Vearget (s+)) These features form context vector Vcontent that captures the properties of each policy and the lationship between them, which can represent the context that generates the attention weights.
Attention Mechanism We argue that the virtual policy should be determined based on the context. A simple strategy such as taking average of policies in M is likely sub-optimal as the quality of these policies vary and some be irrelevant the current learning context. Hence, propose to sum the policies in a weighted manner wherein the weights are generated by a neural network whose input is Uoottet- We compute v performing "attention" in the parameter space as follows,
4

---

Under review as a conference paper at ICLR 2022
Pendulum
LunarL
BWalker
Model
1M
1M
5M
KL Adaptive (deary = 0.003)
-407.74
238.30=34.07
206.9945.34
KI Adaptive (dearg 0.01)
-147.52+9.99
254.26= 19.44
247.70=11.16
KI Adaptive (dtarg = 0.03)
601.09+273.18
246.93=12.57
259.80=6.33
KL Fixed (3 0.01)
1051.144158.81
247.61=19.79
221.55+38,
KL Fixed (B 0.1)
464.29+426.27
256.75=20.53
263,56
Fixed (3 1)
-136.40+4.49
192.62+32.97
215.13=13.29
PPO (clip = 0.1)
-282.204243.42
242.98=13.50
105.07=19.13
PPO (clip = 0.2)
-514.284385.34
250.88=20.33
253.58=7.49
PPO (clip 0.3)
+22.33+229.32
259.93=22.52
160.51
MDPO (6g = 0.5)
-136.45+8.21
247.96#4.77
251.18+29.10
MDPO (3u 1)
139.14410.32
207. 207.96243.88
245.27=11.47
MDPO (āg = 2)
-135.5245.28
227.76=16.96
126.60
VMPO (ag 0.1)
-144.51+7.04
201.87+29.48
236.57=10.62
VMPO (og = 1)
-139.50+5.54
212.85=43.35
238.82#11.11
VMPO (og 5)
296.484213.06
222.13=35.55
164.402400.6
MCPO (N 5)
-133,42+4.53
262.23=12.44
265.80=5.55
MCPO (N 10)
-146.8843778
263.04+11.48
266.2648.87
MCPO (N 40)
135.57755.22
267.19=13.42
249.51=12.75
Table 1: Mean and std. over runs on classical control tasks (with number of training environment steps). Bold denotes the best mean. Underline denotes good results (if exist), statistically indifferent from the best in terms of Cohen effect size less than 0.5.
M
Σ Jop (Veonteaz); 0j
where d is the attention network- a feed-forward neural network parameterized by yp with softmax activation. The network outputs N-dimensional output vector establishing the attention weights. Here & a function of oo and we train to improve sp's performance.
Objective function Given the virtual policy v (p)), we find o to maximize its performance La (1)) E. [R (vb (1))).. We aim obtain the best representative of past policies without examining the per- formance of each individual policy in M since evaluating all policies is highly expensive, especially when M large. In addition, learning a "soft" attention is more flexible than searching for "hard" policy that performs best (in terms of L2) since the performance measurement itself can be noisy and not always reliable. To train the whole system, use gradient ascent maximize
MCPO L1 (0) L2
(5)
When optimizing L1, we fix yo and only update to avoid gradient pack-propggation via (similarly, when optimizing Lgg fix 0 and only update sp). Theoretical motivation for the design of L and Lg is given in Appendix C We implement MCPO using minibatch update procedure (Schulman
al. 2017). Algo. A Iillstraates high-level implementation of MCPO with actor.
4 EXPERIMENTAL RESULTS
In our experiments, the main baselines are recent on-policy constrained methods that use first-order optimization, in which most of them employ KL terms in the objective function. They are KL Adaptive, KL Fixed, PPO (Schulman et al. 2017). MDPO (Tomar et al. 2020) and VMPO (Song al. 2019). We also include second-order methods such TRPO (Schulman al. 2015a) and
ACKTR (Wu et al., 2017). Across experiments, for MCPO, we fix 3maz 10, 3min 0.01 and
only tune N. More details the baselines and tasks are given in Appendix B.I
5

---

Under review as a conference paper at ICLR 2022
Avg. Return
0.9
Unlock
0.9
UalactPickup
KI. Adaptive KL Fixed MCPO (Ours)
0.5
0.5
MDPO
PPO VMPO
0.0
0.0
25K
50K
75K
100K
250K
500K
750K
1M
Step
Step
Figure 1: Unlock (left) and UnlockPickup (right)'s learning curves (mean and std. over 10 runs).
4.1 CLASSICAL CONTROL: HYPERPARAMETER SENSITIVITY TEST
In this section, compare MCPO other first-order policy gradient methods (KL Adaptive, KL Fixed, PPO, MDPO and VMPO) on 3 classical control tasks: Pendulum, LunarLander and Bipedal- Walker, which are trained for one, and five million environment steps, respectively. Here, are curious to know whether tuning hyperparameters helps the baselines solve these simple tasks, and how their performances fluctuate the hyperparameters vary. For each model, we choose one hyperparameter that controls the conservativeness of policy update, and we try different values for the signature hyperparameter while keeping the others the same. For example, for PPO, we tune the clip value e; for KI. Fixed we tune 3 coefficient and these possible values are chosen following prior works. For our MCPO, we tune the size of the policy memory N (5, 10 and 40). We do not try bigger policy memory size to keep MCPO running efficiently (see Appendix B.2 for details of baselines, the choice of hyperparameters and running time analysis).
reports the results of MCPO and 5 baselines with different hyperparameters. For these simple tasks, tuning the hyperparameters often helps the model achieve least moderate performance. However, models like KL Adaptive and VMPO cannot reach good performance despite being tuned. PPO shows good results on LunarLander and BipedalWalker, yet underperfoms others on Pendulum. Interestingly, if tuned properly, the vanilla KL Fixed show competitive results compared to PPO and MDPO in BipedalWalker. Amongst all, our MCPO with suitable N achieves the best performance on all tasks. Remarkably, its performance does not fluctuate much as N changes from 5 to 40, often obtaining good and best results. On the contrary, other methods observe clear drop in performance the hyperparameters are set incorrectly (see Appendix B.2 for full learning curves).
4.2 NAVIGATION TASKS: SAMPLE EFFICIECCY TEST
Here, we validate our method on sparse reward environments using MiniGrid library (Chevalier- Boisvert et al., 2018). In particular, we test MCPO and other baselines (same as above) on Unlock and UnlockPickup tasks. In these tasks, the agent navigates through rooms and picks up objects to complete the episode. The agent only receives reward +1 if it can complete the episode successfully. For sample efficiency test, we train all models on Unlock (find key and open the door) and Unlock- Pickup (find key, open the door and pickup an object), for only 100,000 and million environment steps, respectively. The models use the best conservative hyperparameters found in the previous task (more in Appendix B.3).
Fig. shows the learning curves of examined models on these two tasks. For Unlock task, except for MCPO and VMPO, 100,000 steps seem not enough for the models to learn useful policies. When trained with million steps on UnlockPickup. the baselines can find better policies, yet still underpefrom MCPO. Here VMPO shows faster learning progress than MCPO at the beginning, however fails to converge to the best solution. Our MCPO the best performer, consistently ending up with average return of 0.9 (90% of episodes finished successfully).
To illustrate how the virtual policy supports building trust regions to boost MCPO's performance, we analyze the relationships amongst the old (0), the virtual policy (xb) and the policies stored in
M (0,) throughout Unlock training. Fig. 2 (a) plots the location of these policies over a truncated period of training (from update step 5160 to 7070). Due to conditional writing rule, the steps where policies are added to M can be uneven (first row-red lines), often distributed right after the locations of the old policy (second row- row-green lines). We query at 10-th step behind the old policy (fourth row- cyan lines) to find which policy in M has the highest attention (third row-yellow lines, linked by blue arrows). As shown in Fig. 2 (a) (second and third row), the attended policy, which mostly

---

Under review as a conference paper at ICLR 2022
Update timesteps corresponding different policy types
0
6old
-V
query
5160
7070
(b)
(c)
Average Return
Most Attrnded Slat
0.6
0.1
8se
20
0.2
15
Hopper
Humaneed
0.0
10
2000
4000
6000
2M 4M 6M
BM
10M
Policy Update Step
Stop
Figure 2: (a) Policy analysis on Unlock. First row (red lines): steps where policy is added to M, i.e. the steps f,. Second row (green lines): steps of old policies Bold. Third row (yellow lines): steps of mostly attended policy, approximating tb. Fourth row (cyan lines): 3 steps of interest where we want find their attended steps. Blue arrows link query step and the step that receives highest attention. (b) Quality of 1a vs. ood. Average return collected by &b and Ootd at different stages of training. (c) Mujoco tasks. The slot in M received the highest attention j = argmax] fup (Vcontent); over time.
resembles th, can be further or closer to the query step than the old policy depending on the training stage. Since let the attention network learn to attend the policy that maximizes the advantage of current mini-batch data, the attended one not necessarily the same as the old policy.
The choice of the chosen virtual policy being better than the old policy is shown in Fig, 2 (b) where we collect several checkpoints of virtual and old policies across training and evaluate each of them the
on 10 testing episodes. Here using to form the second KL constraint is beneficial policy generally pulled toward a better policy during training. That contributes to the excellent performance of MCPO compared to other single trust-region baselines, especially KL Fixed and Adaptive, which are very close to MCPO in term of objective function style.
4.3 MUJOCO CONTINUOUS CONTROL: EFFECTIVENESS OF LEARNED R
Next, we examine MCPO and some trust-region methods from the literature that are known to have good performance on continuous control problems: TRPO, PPO and MDPO. To understand the role of the attention network in MCPO, we design a variant of MCPO: Mean tb which simply constructs & by taking average over policy parameters M. We pick hard Mujoco tasks and train each model for 10 million environment steps. For each baseline, we again only tune the conservative hyperparameters and report the best configurations in Table (see Appendix B.4 for full results).
The results show that MCPO is the best performer on 5 out of 6 tasks, where clear outperformance gaps can be found HalfCheetah, Ant, Humanoid and HumanoidStandup. We note that this is only achieved as MCPO uses N 40, which indicates that bigger policy memory (more conservative- ness) is beneficial in this case. The variant Mean & demonstrates reasonable performance for the first 4 tasks, yet almost fails to learn the last two, which means using mean virtual policy is unsuitable in these tasks.
To understand the effectiiveness of the attention network, we visualize the attention pattern of MCPO on the last two tasks and on Hopper-a task that Mean & performs well. Fig. (c) illustrates that for the first two harder tasks, MCPO gradually leams favour older policies M (j" 35). which puts more restriction on the policy change as the model converges. This strategy seems critical for those tasks the difference in average return between learned & and Mean & is huge in these cases. On the other hand, on Hopper, the top attended slots are just above the middle policies in M (j* 25). which means this task prefers an average restriction.

---

Under review as a conference paper at ICLR 2022
Model
HalfCheetah
Walker2d
Hopper
Ant
Humanoid
HumanoidStandup
TRPO
1,111=114
3,966456
3,159=72
2,438=402
4,5764106
145,14343,702
PPO
1,75
5,278+594
2,968+1,000
3,421#534
3,375+1,,68
155,49446,663
MDPO
4,774x1,,99 ,598
4,957+330
3,153+956
3,553+696
1,620+2, 145
90,646+5,855
Mean d
4,94243,095
5,0564842
3,4304259
4.570=548
353+27
71.308#11.113
MCPO
6,173+595
5, 5,120+588
3,620+252
4,673:249
4,848+711
195,404432,801
Table 2: Mean and std. over runs on 6 Mujoco tasks at 10M environment steps.
ACKTR
MCPO
PPO
VMPO
163
Deum
162
Breakout
1e2
Auig Returrs
Copher
Seagueest
Specelnvaders
Betur
1.5
20M
10M
2M
10M
Stop
Stop
Stop
Figure 3: Mean and std. over 5 runs on 6 Atari games over 10M environment steps.
4.4 ATARI GAMES: SCALING TO HIGHER DIMENSIONS
As showcasing the robustness of our method to high- dimensinnnii inputs, we execute an experiment on a subset of Atari games wherein the states are screen images and the policy and value function ap- proximator uses deep convolutional neural networks. We choose 6 typical games (Mnih et al.) (013) and benchmark MCPO against PPO, ACKTR and VMPO, training all models for only TO million environment steps. In this experiment, MCPO uses N 10 and other basclines' hyperparameters are selected based on the original papers (see Appendix B.5).
Fig. visualizes the learning curves of the models. Regardless of our regular tuning, VMPO per- forms poorly, indicating that this method is unsuitable or needs extensive tuning to work for low- sample training regime. ACKTR, works extremely well on certain games (Breakout and Seaquest), but shows mediocre results on others (Enduro, BeamRider), overall underperfoming MCPO. PPO is always similar inferior to MCPO on this testhed. Our MCPO always demonstrates competitive results, outperforming all other models in 4 games, especially on Enduro and Gopher, and showing comparable results with that of the best model in the other 2 games.
4.5 ABLATION STUDY
Finally, we verify MCPO's 3 components; virtual policy (Eq. switching-B (Eq. 2 and con- ditional writing rule (Eq. 3). We also confirm the role of choosing the right memory size N and learning to attend to the virtual policy v. As such, we pick Bipedal Walkerradoo from OpenAl Gym and train MCPO with different configurations for 50M steps. First, we tune N (5,10 and 40) using the normal MCPO with all components and find out that N = 10 is the best, reaching about 169 return score. Keeping N = 10, for each component, we ablate or replace our component with an alternative and report the findings as follows.
Virtual policy To show the benefit of pushing the new policy toward the virtual policy, we imple- ment variant of MCPO that does not use é's KL term in Eq. [ (a.k.a O, 0). This variant underperfoms the normal MCPO by a margin of 100 return. Switching-3 The results show that compared to the annealed 3 strategy adopted from MDPO, our switching-3 3 achieves significantly better results with about 50 return score higher. Conditional writing We compare our proposal with the vanilla approach that adds new policy to M at every update step (frequent writing) and another
8

---

Under review as a conference paper at ICLR 2022
version that writes to M every interval of 10 update steps (uniform writing). Both frequent and uni- form writing show slow learning progress, and ends up with negative rewards. Perhaps, frequently adding policies to M makes the memory content similar, hastening the removal of older, yet maybe valuable policies. Uniform writing is better, yet it can still add similar policies to M and requires additional effort for tuning the writing interval. Learned d To benchmark, we try alternatives: (1) using Mean 1/ and (2) only using half of the features in Vcontent to generate &b (Eq. 4. The results confirm that the Mean & not a strong baseline for this task, only reaping moderate rewards. Using less features for the context causes information loss, hinder generations of useful 3 and thus, un- derperforms the full-feature version by a margin of 50 return. For completeness, we also compare our methods to heavily tuned baselines and confirm that the normal MCPO (N = 10) is the best performer. All the learning curves and details can found Appendix B.6
5 RELATED WORKS
A framework for model-free reinforcement learning with policy gradient approximate policy itera- tion (API), which alternates between estimating the advantage of the current policy, and updating the current policy's s parameters to obtain new policy by maximizing the expected advantage (Bertsekas)
& &Tsisiklis 1995: Sutton et al. (000). Theoretical studies have shown that constraining policy up-
dates is critical for API (Kakade & Langford 2002] Schulman et al.) 2015a Shaan et al. 2020
Vieillard et al.] 2020b). An early attempt to improve API is Conservative Policy Iteration (CPI).
which sets the new policy as a stochastic mixture of previous greedy policies (Kakade & Langford
2002). The mixture idea of CPI inspires other methods which explore different ways to estimate the
greedy policy (Pirotta et al. 2013) or employs neural networks as function approximators (Vieillard al. 2020a). Our paper difters from these works in three aspects: (I) we do not directly set the new policy to the mixture, rather, we use a mixture of previously found policies (the virtual policy) to define the trust region constraining the new policy via KL regularization; (2) our mixture can consist of more than 2 modes, and thus using multiple mixture weights (attention weights); (3) we use the attention network to learn these weights based on the training context.
Also motivated by Kakade & Langford (2002). TRPO extends the theory to general stochastic poli- cies, rather than just mixture polices, ensuring monotonic improvement by combining maximizing the approximate expected advantage with minimizing the KL divergence between two consecutive policies (Schulman et al., 2015a). Arguing that optimizing this way is too conservative and hard to tune, the authors reformulate the objective conjugate gradient and line search. To simplify the implementation of TRPO, Schhllmnn et al. (2017) introduces first-order optimization methods and code-level improvement, which results in PPO- an API method that optimizes a clipped surrogate objective using minibatch updates.
constrained optimization problem to solve it with
(chuulSn
Another line of works views constrained policy improvement as SispetatiionMMaxiizzznn rithm where minimizing KL-term corresponds to the Expectation step, which can be off-policy (Ab- dolmaleki al. 2018) on-policy (Song et al. 2019). From mirror descent perspective, several works also use KL divergence to regularize policy updates (Yang et al.] 2019 Tomar et al.] 2020)
algo-
Shani et al. 2020). A recent analysis also points out the advantages of using KL term as a regular- izer over hard constraint (Lazić et al. 202I). Some other works improve the standard trust-region with adaptive clip range (Wang et al.) 2019) or off off-policy data (Fakoor et al., 2020). Our approach shares similarities with them where we also jointly optimize the approximate expected advantage and KL constraint terms for multiple epochs of minibatch updates. However, we propose a novel dynamic virtual policy to construct the second trust region supplement to the traditional trust region defined by the old or previous policy.
DISCUSSION
We have presented Memory-Constrainee Policy Optimization, a new method to regularize each policy update with two-trust regions with respect one single old policy and another virtual policy representing history of past policies. The new policy is encouraged to stay closer to the region surrounding the policy that performs better. The virtual policy is determined online through learned attention to a memory of past policies. MCPO is applicable in various problems and settings, less sensitive to hyperparameter changes, and showing better performance in many environments.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. International Conference on Learn- ing Representations, 2018.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming: an overview. In Pro- ceedings of 1995 34th IEEE conference on decision and control, volume 1, pp. 560-564. IEEE, 1995.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://gitt wwt.tttwww..c....aaaa.aeaa cctp://.mmmcccc///c//ccccc mecb/ wwweecbb/mmmmmiiggiiii... 2018.
Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3o: Policy-on policy-off policy opti- mization. In Uncertainty in Artificial Intelligence, pp. 1017-1027. PMLR, 2020.
Kakade and J. Langford. Approximately optimal approximate reinforcement learning. ICMI,, 2002.
Nevena Lazié, Botao Hao, Yasin Abbasi- Yadkori, Dale Schuurmans, and Csaba Szepesvári. Opti- mization issues in kl-constrained approximate policy iteration. arXiv preprint arXiv:2102.06234, 2021.
Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization. arXiv preprint arXiv: arXiv:/901.000 1901000300.. 2019.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5187744)):52955555 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello. Safe policy iteration. In International Conference on Machine Learning, pp. 307-315. PMLR, 2013.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 020.041500004438. 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5668-5675, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 355:766)::336433355 2017.
10

---

Under review as a conference paper at ICLR 2022
H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posterioni policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2019.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural informa- tion processing systems, pp. 1057-1063, 2000.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint arXiv:2005.095: 049144 2020.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Deep conservative policy iteration. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6070-6077, 2020a.
Nino Vieillard. Bruno Scherrer, Olivier Pietquin, and Matthieu Geist. Momentum in reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 2529-2538. PMLR, 2020b.
Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proximal policy optimization. Advances in Neural Information Processing Systems, 32:626-636, 2019.
Yuhuai Wu, Elman Mansimov, Shun Liao, Roger Grosse, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using frontcker-factorred approximation. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 5285-5294, 2017.
Long Yang, Gang Zheng, Haotian Zhang, Yu Zhang, Qian Zheng, Jun Wen, and Gang Pan. Policy optimization with stochastic mirror descent, arXiv preprint arXiv: 1906. 10462, 2019.

---

Under review as a conference paper at ICLR 2022
APPENDIX
METHOD DETAILS
A.1 THE ATTENTION NETWORK
The attention network implemented feedforward neural network with one hidden layer:
Input layer: 12 units
Hidden layer: N units coupled with a dropout layer p = 0.5
Output layer: N units, softmax activation function
N is the capacity of policy memory. The 12 features of the input Vcontezt is listed Table 3
A.2 THE ADVANTAGE FUNCTION
In this paper, we use GAE (Schulman al. 2015b) as the advantage function for all models and experiments
N T-t-1
A,
Σ
Σ k=
(*a)* (re4k + vV (5t+241) (5(4*)
Nactor
where y is the discounted factor and Nactor the number of actors. The term ri+s + yV (s/+*+1) also known Viarget in computing the value loss. Note that Algo. Tillustrates the procedure for actor. In practice, we use Nactor depending on the tasks.
A.3 THE OBJECTIVE FUNCTION
Following Schulman et al. 2017), our objective function also includes value loss and entropy terms. This is applied to all of the baselines. For example, the complete objective function for MCPO reads
L = I, MCPO e, E, (Vo (s,) - Viorget (st)) + c2 log (Twars (-1s)))]
where 61 and C2 are value and entropy coefficient hyperparameters, respectively. Ve is the value network, also parameterized with 6.
EXPERIMENTAL DETAILS
B. BASELINES AND TASKS
All baselines in this paper share the same setting of policy and value networks. Except for TRPO, all other baselines minibatch training. The only difference is the objective function, which revolves around KL and advantage terms. We train all models with Adam optimizer. We summarize the policy and value network architecture in Table
The baselines ACKTR, P TRPO available public code. They are Pytorch reimplementation of OpenAl's stable baselines, which reproduce the original performance relatively well. For MDPO, we refer to the authors' source code to reimplement the method. For VMPO, we refer this open source code") reimplement the method. We implement KL Fixed and KL Adaptive, using objective function defined in Sec. 2
We use enviromments from Open Al gyms which are public and using The MIT License. Mujoco environments use Mujoco softward (our license is academic lab). Table lists all the environments.
httass//attabb..om//////////
httoss//qm..gppma..commmmm
http:://ww.rrboti..e..
12

---

Under review as a conference paper at ICLR 2022
Dimension
Feature
Meaning
d (0, Vold)
"Distance" between 0 and Vold "Distance" between Oold and Wold "Distance" between Ootd and 0 Approximate expected advantage of Vota Approximate expected advantage of Ooid Approximate expected advantage of Approximate entropy of Pold Approximate entropy of ood Approximate entropy of 0
(Ontay Vota)
3
d (Dota; 0)
E, [R (waad)]
E, RR (Oald)
E, [R, (0)
È, - log (Traad (+(s:))]
E, log (Towls (-1s²))
9
1 [- lolog (To (-1st))]
10
Be (V Vora (8t) Viarget (8t))
Value loss of that Value loss of Oold
11
E, (Vours (5t) Viarget (st))
12
E2 (Vo (8t)
Veorget (8+))
Value loss of 0
Table 3: Features of the context vector.
Input type
Policy/Vai Value networks
2-layer feedforward net (tanh, h=64)
Vector
3-layer ReLU CNN with kernels (32/8/4, 64/4/2, 32/3/11 3 +22-leye feedforward net (ReLU, h=512)
Image
Table 4: Network architecture shared across baselines.
Continuous
Gym
Tasks
action
category
Pendulum-v0
Classical
LunarLander-v2
Box2d
BipedalWalker-Ve
Unlock-vO
MiniGrid
UnlockPickup-v0
MuloCo tasks (y2): HalfCheetah
Walker2d, Hopper, Ant
MuJoCo
Humanoid, HumanoidStandup
Atari games (NoFramskip-v4);
Beamrider. Breakout
x
Atari
Enduro, Gopher
Seaquest, SpaceInvaders
BBjpedalWalkerrHaadddoe--3
Box2d
Table 5: Tasks used in the paper.
13

---

Under review as a conference paper at ICLR 2022
Bipedal
Hyperparameter
Pendulum
LunarLander
Bipedal
MiniGrid
Hardcore
Horizon T
2048
2048
2048
2048
2048
Adam step size
10
10
10
3
10
3 10
Num. epochs K
10
10
10
10
10
Minibatch size
64
64
64
64
Discount
0.99
0.99
0.99
0.99
0.99
GAE À
0.95
0,95
0.95
0.95
0.95
Num. actors Nactor
4
4
32
4
128
Value coefficient C1
0.5
0.5
0.5
0.5
0.5
Entropy coefficient C2
0
0
0
0
Table 6: Network architecture shared across baselines on Pendulum, LunarLander, BipedalWalker, MiniGrid and BipedalWaker Hardcore
Model
Speed (env. steps/s)
MCPO (N=5)
1,170
MCPO (N=10)
927
MCPO (N=40)
560
PPO
1,250
Table 7: Computing cost of MCPO and PPO on Pendulum.
B.2 DETAILS CLASSICAL CONTROL
For these tasks, all models share hyperparameters listed in Table 6 Besides, each method has its own set of additional hyperparameters. For example, PPO, KI. Fixed and KI. Adaptive have e, B and dearg. respectively. These hyperparameters directly control the conservativeness of the policy update for each method. For MDPO, is automatically reduced overtime through an annealing process from 1 to 0 and thus should not be considered control the conservativeness if 3 is annealed from a different value 8o rather 1. We realize that tuning Bg helped MDPO [T. We quickly tried with several values 3n ranging from 0.01 to 10 on Pendulum, and realize that only 8o E {0.5, 1, gave reasonable results. Thus, we only tuned MDPO with these ig in other tasks. For VMPO there are many other hyperparameters such as ijo- oo. 6n and e. Due to limited compute, we do not tune all of them. Rather, we only tune -te initial value of the Lagrange multiplier that scale the KL term the objective function. We refer to the paper's and the code repository's default values of oo to determine possible values ag E {0.1, 5). For our MCPO, we can tune several hyperparameters such as N, 3mins and 8maz- However, for simplicity, only tune N {5, 10, 40} and fix 8.ui 0.01 and Biar 10.
hyperparameter. However,
can still
On our machines using 1 GPU Tesla V100-SXM2, we measure the running time of MCPO with different N compared to PPO Pendulum task, which is reported in Table 7 As N increases, the running speed of MCPO decreases. For this reason, we do not test with > 40. However, we realize that with N 5 or N 10, MCPO only slightly slower than PPO. We also realize that the speed gap is even reduced when we increase the number of actors Nactor as in other experiments. In terms of memory usage, maintaining policy memory will definitely cost more. However, as our policy, value and attention networks are very simple. The maximum storage even for N 40 is less than 5GB.
In addition to the configurations reported in Table I for KL Fixed and PPO, we also tested with extreme values / 10 and e E {0.5, 0.83. Figs. and g visualize the learning curves of all configurations for all models.
B.3 DETAILS MINIGRID NAVIGATION
Based on the results from the above tasks, pick the best signature hyperparameters for the models to use in this task as in Table g In particular, for each model, we rank the hyperparameters per task (higher rank is better), and choose the that has the maximum total rank. For hyperparameters
14

---

Under review as a conference paper at ICLR 2022
Model
Chosen hyperparameter
KL Adaptive
diary 0.01
KL Fixed
= 0.1
PPO
0.2
MDPO
3 0.5
VMPO
ag
МСРO
N 10
Table 8: Signature hyperparameters used in MiniGrid tasks.
Hyperparameter
Mujoco
Atari
Horizon T
2048
128
Adam step size
10
2.5
10
Num. epochs K
10
4
Minibatch size B
32
Discount
0.99
0.99
GAE X
0.95
0.95
Num. actors N Nactor
16
32
Value coefficient C1
0.5
1.0
Entropy coefficient C2
0
0.01
Table 9: Network architecture shared across baselines on Mujoco and Atari
that share the same total rank, we prefer the middle value. The other hyperparameters for this task listed in T
B.4 DETAILS ON MUJOCO
For shared hyperparameters, we use the values suggested in the PPO's paper, except for the number of actors, which we increase for faster training as our models are trained for 10M environment steps (see Table(9).
For the signature hyperparameter of each method, we select some of the reasonable values. For PPO, the authors already examined with E (0.1, 0.3) on the same task and found 0.2 the best. This is somehow backed up in our previous experiments where we did not see major difference in performance between these values. Hence, seeking for other rather than the optimal 0.2, ran our PPO implementation with c E {0.2, 0.5, 0.8}. For TRPO, the authors only used the KL radius threshold 8 0.01, which may be already the optimal hyperparameter. Hence, only tried E (0.005, 0.013. The results showed that = 0.005 always performed worse. For MCPO and Mean v/, only with extreme N (5, 40). For MDPO, still tested with 30 E (0.5, 2) Full learning curves with different hyperparameter are reported in Fig. 9 Learning curves including TRGPPC are reported in Fig. 10
B.5 DETAILS ON ATART
For shared hyperparameters, we use the values suggested in the PPO's paper, except for the number of actors, which we increase to 32 for faster training (see Table 9). For the signature hyperparam- eter of the baselines, we used the recommended value in the original papers. For MCPO, we use N to balance between running time and performance. Table 10 shows the values of these hyperparameters.
We also report the average normalized human score (mean and median) the models over games in Table [1] As seen, MCPO is significantly better than other baselines in terms of both mean and median normalized human score. We also report full learning curves of models and normalized human score including TRGPPO in 9 games in Fig. 4 and Table 12 respectively.
We use the authors' source code https: whu..b..ttuucccggggggg com hoom//angyuuuu x TRGPPO using default con- figuration. Training setting adjusted follow the common setting for other baselines (see Table
15

---

Under review as a conference paper at ICLR 2022
Model
Chosen hyperparameter
PPO
= 0.2
ACKTR
0.01
VMPO
Ong
MCPO
N = 10
Table 10: Signature hyperparameters used in Atari tasks.
Model
Mean
Median
PPO
134.63
48.36
ACKTR
266.73
21,99
VMPO
412.19
20.85
MCPO
300.25
100.28
Table 11: Average normalized human score over games. For each model, the performance each is measured by the best checkpoint during training 10 million frames, then take average over runs.
To verify whether MCPO can maintain its performance over longer training, we examine Atari training for 40 million frames. As shown in Fig. 5 MCPO is still the best performer in this training regime.
B.6 DETAILS ON ABLATION STUDY
In this section, we give more details on the ablated baselines.
* No df We only changed the objective to
L1 (0) = E, rate (0) A.)
-3 [KL [Toas (-1se), no (+1s))]
(6)
where 3 still determined by the B-switching rule.
Annealed B We determine the 3 in Eq. [ by MDPO's annealing rule, a.k.a, B; 1.0 Tisuu where Trotal is the total number of training policy update steps and i is the current update step. We did not test with other rules such fixed adaptive / as realize that MDPO is often better than KL Fixed and KI. Adaptive in our experiments, indicating that the annealed B is stronger baseline.
Frequent writing We add new policy to M at every policy update step.
Uniform writing Inspired by the uniform writing mechanism in Memory-Augmentt ral Networks (Le et al. 2019), we add new policy to MM at every interval of 10 update steps. The interval size could tuned to get better results but would require additional effort, so we preferred our conditional writing over this one.
Neu-
Mean vi The virtual policy is determined
Model
Mean
Median
PPO
131.19
52,85
ACKTR
195.52
25.30
VMPO
18.20
13.56
TRGPPO
116.80
43.24
MCPO
229.99
65.78
Table 12: Average normalized human score over 9 games. For each model, the performance of each run is measured by the best checkpoint during training over 10 million frames, then take average over runs.
16

---

Under review as a conference paper at ICLR 2022
ACKTR
MCPO
PPO
TRGPPO
VMPO
1e3
BeamRider
1e2
Breakout
1e2
Enduro
Avg. Reetrm
3
2
1
1e3
Gopher
1e3
Seaquest
1e2
Spacelnvaders
Avg. Return
1.5
7.5
3
2
5.0
0.5
2.5
Qbert
BankHeist
1e1
IceHockey
Avg. Retum
4
1.0
0.5
2
0.5
0
0.0
1.0
2M
5M Step
7M
10M
0
2M
5M Step
7M
10M
2M
5M
7M
10M
Step
Figure 4: Atari games: learning curves (mean and std. 5 runs) across 10M training steps.
ACKTR
MCPO
PPO
Roam Rider
lc2
Breakost
Endure
Avg Retura
1.c4
Gupher
1c3
Seaquext
lc3
SpaceInvaders
2.5
Ang. Retrr
1.5
2.0
1.5
1.0
1.0
0.5
0.5
Step
Step
Siep
Figure Atari games: learning curves (mean and std. over 5 runs) across 40M training steps.
17

---

Under review as a conference paper at ICLR 2022
Adaptive
KL.
(Ours)
-250
020
N-3t
500
d-0
Nei
10000
1250
MDPO
PPO
VMРО
250
A-111
dpai1
Amt
-300
dgr65
-750
dprba
-1000
-1250
200K
400K
600K
BOOK
1M
200K
400K
600K
BOOK
IM
200K
400K
600K
BO0K
1M
Figure 6: Pendulum-vO: learning curves (mean and std. runs) across 1M training steps.
KL Adaptive
Fixed
MCPO (Ours)
200
deo.00
owo..00 LC3
Ne20
Ni-0
200
N-5
A+03
1000
Opel
-100
-200
200K
400K
600K
B0OK
1M
200K
400K
600K
BOCK
1M
0
200K
400K
600K
800K
1M
Figure LumarLander-v2: learning curves (mean and std. over runs) across 5M training steps.
M
3
0
Half feature We only use features from 1 to 6 listed in Table 3
The other baselines including KL Adaptive, KL Fixed, MDPO, PPO, and VMPO are the same as in
B.2 The full learning curves of all models with different hyperparameters plotted in Fig. 11
C THEORETICAL ANALYSIS OF MCPO
In this section, we explain the design our objective function L, and L2. Similar to Schulman et al. (015a), we construct theoretically guaranteed version of our practical objective functions that ensures monotonic policy improvement.
First, explain the design of L, by recasting L
Liguee (0) Lg (0)
C1 Dman (Oald: 0)
-2D
Dona (1,
where Logi (0) 17 (TT E. Prons the expected discounted reward 1 (f), Dmne (a, b) max, KL (ma (-1s), Ta (-18)), Ci = maxx, maxx, (A, and C2 > 0.
(s) Ea To (a|s) Ameal (8, atee local approximation
(1-7)²
18

---

Under review as a conference paper at ICLR 2022
KL Adaptive
Pixed
MCPO (Ours)
300
0H0.02
200
OHILD3 23
100
Kof
МПPО
PPO
VMPО
300
Punt
<pett
-100
2M
2M
3M
Figure 8: Waleral learning curves (mean and std. over runs) across 1M training steps.
TRPO
Mean
MCPO (Ouns)
MDPO
PPO
2000
2500
4000
Ar Hagger
2000
Arg Belam
5000
2500
5000
200000
100000
Figure 9: Mujoco: learning curves (mean and std. over 5 runs) across 10M training steps.
19

---

Under review as a conference paper at ICLR 2022
MCPO (Ours) MDPO
Mean PPO
TRGPPO
TRPO
HaliChectah
Walker2d
Hopper
Avg. Rehur
163
1e3
Humanoid
1e5
HumanoldSttaddup
2,0
AVg. Netum
1.0
2M
4M
8M
10M
2M
4M
6M
8M
10M
0
4M
6M
10M
Step
Step
Step
Figure 10: Mujoco: learning curves (mean and std. over 5 runs) across 10M training steps.
KL. Adaptivn
KI. Fleed
MCPO (Oura)
150
Sed
polt
100
milngg
-50
-150
200
150
100
-00
150
10M
20M
80M
50M
10M
20M
30M
50M
30M
30M
40M
Figure 11: Bibedal Bileeda/Wakker/Harddoee333 learning curves (mean and std. over runs) across 50M training steps.
20

---

Under review as a conference paper at ICLR 2022
As the KL is non-negative. L10.14 (0) < Le.s (0) DD."t." (fold: 0). According to Schulman
et al. (2015a), the RHS a lower bound on 17 (0). so L1 is also a lower bound on 17 (0) and thus, reasonable to maximize the practical L1, which is approximation of Li0
Next, we show that by optimizing both L1 and L2. we can interpret our algorithm as a monotonic policy improvement procedure. As such, need reformulate L2 as
Le (+)) Lowi (1)) C₁ DT.. (Bota: 1))
Note that compared to the practical La (as defined in the main paper on page 5), we have introduced here an additional LT term, which means we need to find d that is close to Oota and maximizes the approximate advantage Teaes (v)). As we maximize I2e (16), the maximizer vb satisfies
L2g (V) 2 Libara (Dold) Loaia (Dold)
We also have
17 (0)
Libora (0)
(8)
17 (foia) Lo (fold) S L2gard (v)
Loas (1b) CDDRL (Oolds 1/)
Lib (15)
(9)
Subtracting both sides of Eq. from Eq. yields
n (0) 77 (Bols) Liaase (0) Lisute (v)
Thus by maximizing Liowa (0), guarantee that the true objective (0) is non-decreasing.
Although the theory suggests that the practical L2 could be L E, RR (v (p)) - KL Taata (-1s;) Te [-]s;)]]. it would require additional tuning of C₁. More importantly, optimizing an objective form Lद needs very small step size, and could converge slowly. Hence, we simply discard the KL term and only optimize L2 i,, [Re (v (+)) instead. Empirical results show that using this simplification, MCPO's learning curves still generally improve monotonically over training time.
21


Reviews:
[{"review_id": "tZ7D0eVa7BF", "reviewer": "Reviewer_HBno", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have several concerns with this paper. In its current form, the paper is somewhat confusing to the reader, and could use from a clearer rewrite. More importantly, the novelty of the paper appears to be minor and the the design decisions of the author's algorithm do not seem to be very well explained.\n\n- Why is there a need to have two trust regions? Following the author's argument for their  virtual policy, would it not be sufficient to use simply a properly designed trust region with respect to to it?\n\n- The design of the virtual policy seems to be completely ad-hoc and unmotivated. Why are the policies in the memory buffer mixed by a weighted average of their parameters? A big reason to use the KL divergence is due to the fact that similarity in the parameter space may not lead to similarity in policy output (specially for parametrization such as neural networks). ", "summary_of_the_paper": "In this work, the authors propose the use a trust-region method in the vein of Proximal Policy Optimization (PPO). In contrast to previous work, the author propose to use two trust regions instead of a single one. Instead of constraining the policy to stay near to the previous policy, the author's proposed algorithm includes an extra virtual policy constructed from a memory buffer of past policies.", "summary_of_the_review": "In its current form, I cannot recommend this paper for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TQt2IGAAREd", "reviewer": "Reviewer_RkGG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: The overall paper is easy to follow, and the motivation is also attractive. The suboptimality of the previous policy in the trust region indeed cases undesirable constraints for follow-up optimization. Experimental results also validate the effectiveness algorithm.\n\nCons: 1. The major concern I have is how MCPO addresses such suboptimality problems. Use the example in the paper, when the previous policy is at the local optimum, the new policy will also be constrained around the local optimum. But how MCPO policy can escape from this optimum? There might be some history policies that explore other regions of the landscape, how are these policies guaranteed to be chosen as the additional constraint? Otherwise, adding the worse history policies which lie in the convex hull of the local optimum will even harm the policy optimization.\\\n2. It would be helpful to provide some theorem support, maybe under ideal cases (e.g., when the memory is not constrained).\\\n3. In experiments Fig. 2 and 3, the curves are not converging. Could you run for more iterations?\n", "summary_of_the_paper": "The paper concerns the suboptimality of the \"previous\" policy in the trust-region policy optimization algorithm. And the authors propose MemoryConstrained Policy Optimization (MCPO) that uses an additional trust-region relying on the virtual policy, the one that represents the history of policies.", "summary_of_the_review": "The motivation of the suboptimality is sound, but the algorithm to solve this problem is not very convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LFRZOsESqz", "reviewer": "Reviewer_9u6G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I agree with the authors that enforcing policy updates in constrained on-policy methods considering only previous policy can be suboptimal and it is better to utilize the history of past policies to make the update more efficient and it can be a right direction to improve the performance of on-policy methods. That being said, there are some concerns in terms of novelty, related works, and experimental results: \n\n- While using attention weights to combine the past policies is interesting and shows some promising results and I consider this is the main novelty of this paper, the idea of improving trust region constraint in on-policy methods has been extensively studied in recent years ( which is not a weakness though). However, one of the main concerns with this paper is it fails to discuss and compares with two important related works [1,2] which are more relevant to this paper. Both [1,2] proposed ways to improve constrained on-policy methods by introducing new ways to impose trust regions constraint: [1] propose a way to adaptively adjusts the clipping range within the trust region and [2] introduced a way to adaptively enforce soft trust region between current policy and all previous ones. Comparing and discussing these works in this paper, can help to highlight the contribution of this work and to better situate this work against them.\n \n- Although this paper used various benchmarks to compare with their work, I found the experimental results are not conclusive. Specifically, it is not clear which method is the best performing method on Atari and major experiments ( e.g. this method only shows better results Gopher and Enduro not others). Another concern is that the choice of Atari games seems very arbitrary. What is the logic to select only these games? Does your method only show good performance on these environments, not the rest?  In addition, it is really hard to come to any conclusion just by looking at Table 2 without seeing the learning curves for these experiments that summarize these results.  I have to acknowledge that the authors did a good job in regard to the ablation studies. But unfortunately this has not been the case for comparison with other methods. It is better to show clear improvement on two benchmarks rather than 5 different ones without a clear conclusion. \n\n- The paper is written fairly well, except for the experimental section as it doesn't have a smooth flow as other sections. I highly suggest improving presentation in that section.\n\n- It is not clear to me how the choice of $\\beta$ affects the two trust regions. Author explained on page 4, but it is confusing. Can the authors explain in detail why Eq2 makes sense? In addition, you said \"The intuition is that we encourage the enforcement of the constraint when $\\pi_\\theta$ is too far from $\\theta_{old}$ using the distance between $\\theta_{old}$ and $\\theta_{\\phi}$ as a reference\" if that is the case, why shouldn't you use beta ($1-\\beta$) for previous policy and ($\\beta$) for virtual policy then?\n\n\n[1] Trust Region-Guided Proximal Policy Optimization\nhttps://arxiv.org/abs/1901.10314\n\n[2] P3O: Policy-on Policy-off Policy Optimization\nhttps://arxiv.org/abs/1905.01756 \n", "summary_of_the_paper": "This paper proposes a new approach to regularize on-policy methods utilizing two soft trust region terms. The first term encourages the new policy to stay close to old policy like PPO and the second term enforces the new policy to stay close to the combination of past policies called virtual policy. The virtual policy is built by a convex combination of past policies in the memory through attention weights. The motivation to use attention is to ensure the virtual policy has good performance on current data. To evaluate their method, this paper uses 3 control tasks (i.e. Pendulum, LunarLander and Bipedal- Walker), MiniGrid library for sparse reward environments evaluation, 6 Mujoco tasks, and 6 Atari games.", "summary_of_the_review": "In general, this paper's proposed method has merits. But unfortunately, it missed some important related works and experimental results are not conclusive enough yet.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6JfbRwxzWxr", "reviewer": "Reviewer_JAMC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is quite a well-written paper. While other works have explored the idea of enforcing trust region on all or several past policies, the use of trust region constraints along with an attention mechanism for past policies is to the best of my knowledge novel. My detailed comments are as follows:\n\n- The presentation of the paper is mostly clear and overall I found it quite easy to read.\n- The experiments are very well-run, and I think it is one of the main strength of this paper. The authors demonstrated results on a large set of environments. The ablation studies are most appreciated and included discussions on most of the key components of the algorithm.\n- Can you comment briefly on the intuition behind conditional writing?\n- How did you determine the context vector features? You also mentioned in your ablation studies that using fewer context features causes a loss of information. Did you find any features that were particularly important?\n- (Minor) I suggest the authors change the heading of Section 2, Constrained Policy Optimization (CPO) (Achiam et al. 2017) is a well known paper for constrained RL and the heading may cause some confusion.\n- (Minor typo) Last paragraph of page 3: ...dynamically 'weigh' the whole KL....", "summary_of_the_paper": "This paper proposes a new trust-region based policy optimization method with two trust regions constraints for policy updates which enforces the proximity of the current policy to (1) the old policy prior to the current update and (2) a virtual policy constructed using an attention mechanism and memory buffer which stores several past policies. In addition a new algorithm based the the aforementioned ideas + minibatch updates is also proposed, which is shown to perform well on a series of high-dimensional control tasks.", "summary_of_the_review": "Overall, I think this paper proposed some very novel and insightful ideas with well-run experiments that would be beneficial to the community. I think this paper passes the bar for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two trust region constrained optimization for policy gradient RL, where the second trust region is based on a virtual policy built from a memory buffer and using an attention mechanism to combine prior policies. The reviewers agree that the paper is well written, the idea is novel, and the paper is extensively evaluated. The authors are commended for running the additional baselines during the rebuttal period.

However, the paper still contains some shortcomings, specifically, the results are somewhat inconclusive even after the rebuttal. While it is not expected that the method wins across the board, it is important to provide an analysis of the limitations of the method. When is the algorithm appropriate to use, and when is it not? 

To make the paper stronger, in the next version of the paper should:
- move the theory in the main text (Appendix C).
- provide the analysis of the algorithm and its limitations.
Decision: Reject