Learning to Constrain Policy Optimization with Virtual Trust Region
Authors: Hung Le
Thommen Karimpanal George
Majid Abdolshah
Dung Nguyen
Kien Do
Sunil Gupta
Svetha Venkatesh
Venue: NeurIPS 2022 Accept (2022)
Keywords: trust region policy optimization
attention mechanism
policy memory

Abstract: We introduce a constrained optimization method for policy gradient reinforcement learning, which uses two trust regions to regulate each policy update. In addition to using the proximity of one single old policy as the first trust region as done by prior works, we propose forming a second trust region by constructing another virtual policy that represents a wide range of past policies. We then enforce the new policy to stay closer to the virtual policy, which is beneficial if the old policy performs poorly. We propose a mechanism to automatically build the virtual policy from a memory buffer of past policies, providing a new capability for dynamically selecting appropriate trust regions during the optimization process. Our proposed method, dubbed Memory-Constrained Policy Optimization (MCPO), is examined in diverse environments, including robotic locomotion control, navigation with sparse rewards and Atari games, consistently demonstrating competitive performance against recent on-policy constrained policy gradient methods.

Learning to Constrain Policy Optimization with Virtual Trust Region
Hung Le, Thommen Karimpanal George, Majid Abdolshah, Dung Nguyen, Kien Do, Sunil Gupta, Svetha Venkatesh Applied Al Institute, Deakin University, Geelong, Australia thai le@deakin. edu.au
Abstract
We introduce constrained optimization method for policy gradient reinforcement learning, which uses a virtual trust region regulate each policy update. In addi- tion to using the proximity of one single old policy as the normal trust region, we propose forming a second trust region through another virtual policy representing a wide range of past policies. We then enforce the new policy to stay closer to the virtual policy, which is beneficial the old policy performs poorly. More im- portantly, we propose mechanism to automatically build the virtual policy from a memory of past policies, providing new capability for dynamically learning appropriate virtual trust regions during the optimization process. Our proposed method, dubbed Comooy-Consttrinee Policy Optimization (MCPO), examined in diverse environments, including robotic locomotion control, navigation with sparse rewards and Atari games, consistently demonstrating competitive perfor- mance against recent on-policy constrained policy gradient methods.
Introduction
Deep reinforcement learning (RL) is the current workhorse in machine learning. Using neural net- works to approximate value and policy functions enables classical approaches such as Q-learning 29 and policy gradient [23 to achieve promising results on many challenging problems such as Go, Atari games and robotics 221 15 12 B3. Compared to Deep Q-learning, deep policy gradient (PG) methods
often more flexible and applicable to discrete and continuous action problems. However, these methods tend to suffer from high sample complexity and training instability since the gradient may not accurately reflect the policy gain when the policy changes substantially (6) This is exacerbated for deep policy networks where numerous parameters need to be optimized, and minor updates in parameter space lead to considerable changes in policy space.
To address this issue, one solution is to regularize each policy update by restricting the Kullback- Leibler (KL) divergence between the new policy and the previous one, which guarantee mono- tonic policy improvement [17]. However, jointly optimizing the approximate advantage function and the KL term does not work in practice [17]. Therefore, Schulman et al. (2015) proposed Trust Region Policy Optimization (TRPO) to constrain the new policy within a KL divergence radius, which requires second-order gradients. Alternatives such as Proximal Policy Optimization (PPO) [9 use a simpler first-order optimization with adaptive KL or clipped surrogate objective while still maintaining the reliable performance of TRPO. Recent methods recast the problem through or Mirror Descent Optimization, and this also results in
new lens using Expectation-Maxiine first-order optimization with KL divergence term in the loss function
An issue with the above methods that the previous policy used to restrict the new policy may be suboptimal and thus unreliable in practice. For example, due to stochasticity and approximations, the new policy may fall into a local optimum even under trust-region optimizations. Then in the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

next update, this policy will become the "previous" policy, and will continue pulling the next policy stay in the local optimum. thus slowing down the training progress. For on-policy methods using mini-batch updates like PPO, the situation is more problematic as the "previous" policy is defined as the old policy to collect data, which can be either very far or close to the current policy. There no guarantee that the old policy defines reasonable trust region for regulating the new policy.
In this paper, we propose novel constrained policy iteration procedure, dubbed Memory- Constrained Policy Optimization (MCPO), wherein a virtual policy representing memory of past policies regularizes each policy update. The virtual policy forms a virtual trust region, attracting the new policy more when the old policy performs badly, which prevents the optimization from falling into local optimum caused by the old policy of poor quality. As such, we measure KL divergences corresponding to the virtual and old policy, and assign different weights to the two KL terms in building the objective function. The weights are computed dynamically based on the performance of the two policies (the higher performer yields higher weights).
In contrast prior works using heuristics (e.g. running average mean of past policies) to form additional trust regions (28), we argue that the virtual policy should be determined dynamically to maximize the performance on current training data. Thus we store policies in policy memory, and learn to extract the most relevant ones to the current context. We sum the past policies in a weighted the attention network. which
manner wherein the weights are generated by a neural network-namen takes the information of the current, the old and the last virtual policy as the input. The attention network is optimized to maximize the approximate expected advantages of the virtual policy. We jointly optimize the policy and attention networks to train our system, alternating between sampling data from the policy and updating the networks in a mini-batch manner.
We verify our proposed MCPO through diverse set of experiments and compare our performance with that of recent constrained policy optimization baselines. In our experiment on classical control tasks, amongst tested models, MCPO consistently achieves better performance across tasks and hyprrparameters. Our testbed on 6 Mujoco tasks shows that MCPO with a big policy memory is performant where the attention network plays an important role. We also demonstrate MCPO's capability of learning efficiently on sparse reward and high-dimensional problems such as navigation and Atari games, Finally, our ablation study highlights the necessity of MCPO's components such as the virtual policy and the attention network.
Background: Policy Optimization with Trust Region
In this section, we briefly review some fundamental constrained policy optimization approaches. A general idea is to force the new policy me to be close to a recent policy Tew. In this paper, we usually refer a policy via its parameters (i.e. policy 0 means policy πØ). We also use finite-horizon estimators for the advantage with discount factor y E (0, 1) and horizon T.
Conservative Policy Iteration (CPI) This method starts with basic objective of policy gradient algorithms, which is to maximize the expected advantage As-
To (at (8t)
LPII (0) = E;
Ag
Toora (at) 8x
where the advantage A, is a function of returns collected from (8t, at) by using Towss (see Appendix A.2 and É, [-] indicates the empirical average over a finite batch of data. To constrain policy updates, the new policy is a mixture of the old and the greedy policy: Ø = argmax CCII (0). That is, 0 = at + (1 a where a is the mixture hyperparameter (6]. As the data is sampled from the previous iteration's 's policy Bold, the objective needs importance sampling estimation. Hereafter, we denote (T)a))))) as Te (0) for short. Hear [s)
KL-Regularized Policy Optimization To enforce the constraint, one can jointly maximize the ad- vantage and minimize KL divergence between the new and old policy, which ensures monotonic improvement [17]
L KL (0) = E, Te (0) A, -KKL [ToSL (-1s)), To [-180)]

---

where 3 is hyperparameter that controls the update conservativeness, which can be fixed (KL Fixed) or changed (KL Adaptive) during training [9]
Trust Region Policy Optimization (TRPO) The method optimizes the expected advantage with hard constraint [17]. This is claimed practical implementation, less conservative than the theo- retically justified algorithm using KL regularizer mentioned above.
TRTPO (0) Rg (0) As st 8 2 KL (Trae (-18e), Th (-1st)
where 8 is the KL constraint radius.
Proximal Policy Optimization (PPO) PPO is a family of constrained policy optimization, which uses first-order optimization and mini-batch updates including KI. Adaptive and clipped PPO. In this paper, we use PPO to refer the method that limits the change policy by clipping the loss function (clipped PPO) [19]. The objective I, POO is defined as
E, min (0) A, clip (T. -,),1,,,1+ A.)]
where is the clip hyperparameter.
In all the above methods, is the currently optimized policy, which is also referred to the current policy. Ootd represents past policy, which can be one or many update steps before the current policy. In either case, the rule decide Ootd is fixed throughout training. If Oold is suboptimal, is unavoidable that the following updates will be negatively impacted. We will address this issue in the next section.
3 Menory-Constrained Policy Optimization
In trust-region policy gradient methods with mini-batch updates such as PPO, the old policy Ootd is often the last "sampling" " policy for collecting observations from the environment. This policy is, by design, fixed during updates of the main policy until the next interaction with the environment. This means aod may immediately precede the current policy but may be from many (update) steps before. Since Ootd is not up-to-date and could possibly be not good, using it to constrain the current policy 0 can cause the suboptimal update of 0, Similarly, the immediately preceding policy could also be poor in quality and thus, using it as fotd could be detrimental to the optimization.
To tackle this issue, we propose to learn to constrain 0 towards weighted combination multi- ple past policies besides Ootd- We argue that we can optimize the attention network to ensure the combined policy is optimal. Below formal description of method.
3.1 Virtual Policy
Computing the virtual policy via past policies The virtual policy should be determined based on the past policies and their contexts such quality, distance entropy. A simple strategy such as taking average of past policies is likely suboptimal as the quality of these policies vary and some can be irrelevant the current learning context. Let z be the weighted combination of M past policies [0.) which we refer as "virtual" policy for naming convenience, is computed as follows:
Σ fo(1)
where fop is a neural network parameterized by yp which outputs softmax attention weights over the M past policies, v a "context" vector capturing different relations among the current policy the last sampling policy fold- and the last virtual policy Vold. We build the context by extracting specific features: pair-wise distances between policies, empirical returns of these policies, policy entropy and value losses (details in Appendix Table 4. Intuitively, these features suggest which virtual policy will yield high performance (e.g., virtual policy that is closer to the policy that obtained high return and low value loss). The details of fo training will be given in Sec. 3.2

---

Algorithm Conntrr--C Policy Optimization.
Require: A policy buffer M, an initial policy No... T, K, B are the learning horizon, number of update epochs, and batch size, respectively.
1: Initialize vold <- Oold. +- oold
2: for iteration = 1, 2,... do
3: Run policy 20... in environment for timesteps. Compute advantage estimates d Ar
4: for epoch = 1, 2, ...K do
5: for batch = 1, T.2....TR T do
6:
Compute (Eq. I using Wold, 0, Oold, optimize 0 and yp by maximizing L MCCP (Eq.
7:
If D (0, v) > D (Oota: v) then add 0 to M
8:
if M N then remove the oldest item in M Vold < V
9:
10:
end for
end for
12: oold <- 0
end for
Storing past policies with diversity-promotiin writing We use memory buffer M store past policies. We treat M as queue with maximum capacity N, which means a new policy 0 will be added the end of M and M full, the oldest policy will be discarded. However, we do not add any new policy 0 to M unconditionally but only when 0 satisfies our "diversity-rromoting" condition. Let D (a, b) É, [KL (Ta (-1s;), me (-[s:)] denote the "distance" between policies and Th, the diversity diversity-promoting writing is defined as:
Adding to M if D(0, +)) D(Ootas vb)
(2)
where D(Ootd, v)) serves as a threshold. This condition makes sure that the policy to be added is far enough from v. It is reasonable because if 0 similar (in regard of Oata), the advantage of storing multiples past policies in order to find good one will disappear. We will elaborate more on this in Sec.
3.2 Policy Optimization with Two Trust Regions
During policy optimization, we make use of both Oold and constrain 0. Hence, Oold and v form trust regions, and aim to enforce the new policy be closer to the better one. To this end, we propose to learn a new policy 0 by maximizing the following objective function:
Optimizing the policy parameters
(0) EE, [T.
B (1 ar (-1s:) K (To (-1st) (-1s:)]
+ or (-1as) KL (TT (-1se) (-1s:]]
where / the weight balancing between the main objective and the policy constraints, a₂ is the weight balancing between constraining 0 towards sa and @ towards Oold The expectation is estimated by taking average over t in a mini-batch of sampled data. We note that in the early stage of learning, the attention network is not trained well, and thus the quality of the virtual policy s may be worse than Osta- In the long-term, & will get better and tend to provide a better trust-region. Hence, we need use both trust regions to ensure maximal performance any learning stage. We can also prove that using our proposed two trust regions guarantees monotonic policy improvement (see Appendix. C.. Below we introduce mechanism automatically determine the contribution of the two trust regions through computing a and B coefficients.
Intuitively, if the virtual policy is better than the old policy, the new policy should be kept close to the virtual policy and vice versa. Hence, a should be proportional to the contribution t to the exp(Ra(u))
final performance with regard to that of 0ddd Thus, we define: 0z (+1st) exp

---

Pendulum
LunarLander
BWalkrr
Model
IM
1M
5M
KL Adaptive (dtarg 0.01)
-147.5249 90
254.26 19.43
247.70
KL Fixed (8 = 0.1)
-464.29+426.27
256. 256.75=20.53
263.56# 10.04
PPO (clip e =0.3)
-591.31+229.22
259.93+22.52
260.51 17.86
MDPO (.o 2)
-135.52+.5.28
227.76=16.96
226.80+15.67
VMPO (Ag 1)
-139.50+5.54
212.85a43.35
238.82411.11
MCPO (N 5)
-133.4244.53
262.23+12.447
265.80
MCPO (N = 10)
-146.88+3.78
263.04+ 11.48
266.26+8.87
MCPO (N 40)
-135.57
267.19#13.42
249.51#12.75
Table I: Mean and std. over runs classical control tasks (with number of training steps). Bold denotes the best mean. Underline denotes good results (if exist), statistically indifferent from the best in terms of Cohen effect size less than 0.5. The baselines reported with best hyperparameters.
where Re(W), (fols) are the estimated returns corresponding to vp and oold. respectively. We estimate Re(-) via weighted importance sampling, that is, R-(-) = T-)A),, where both are computed using the same St. We also dynamically adjust by switching between values 3min and Bmaz
(0 < Amin Smax) as follows:
3 Bmar
if D(Oota 0) D(Ootd, v/)
(4)
Bmin
otherwise
where D(-,)) is again the KL "distance". The reason behind this update of is to encourage stronger enforcement of the constraint when is too far from Oold- Unlike using fixed threshold dearg to change 3 (e.g in KL Adaptive, if D (Ooza, 0) dtargs increase 3 [9),, we make use of & as reference for selecting This allows dynamic threshold that varies depending the current learning. We name this mechanism as switching-9 rule.
Optimizing the attention network parameters ip To encourage the attention network fp (Eq. to produce the optimal attention weights over past policies, we maximize the following objective:
L2(0) = E, [Re
(5)
which is the approximate expected return w.r.t. a. We write vi to emphasize that s is a function of yp (as formulated in Eq. [ We use gradient ascent to update by backpropagating the gradient aCe through the attention network. ap
By learning for we can guarantee that the virtual policy to the best combination of past policies in terms of return. Note that for on-policy learning setting, optimizing "soft" attention weights is usually more robust than searching for the best past policy in M because the estimation of the expected return using current data samples be noisy and is not always reliable. Also, searching is computationally more expensive than using the attention.
Final Objective
We train the whole system by maximizing the following objective:
L MCPO L, (0) L2((p)
(6)
where MCPO stands for Memory-Come Conssrrainnntttiieeede Policy Optimization. We optimize 9 and & alternately
by fixing one and learning the other. We implement MCPO using minibatch update procedure [19]. MCPO Pseudocode given in A For notational simplification, the algorithm uses 1 actor.
4 Experimental results
In our experiments, we show our optimization scheme is superior in different aspects: hyperpa- rameter sensitivity, sample efficiency and consistent performance in continuous and discrete action spaces. The main baselines are recent on-policy constrained methods that use first-order optimiza- tion, in which most of them employ KI. terms in the objective function. They are KI. Adaptive, VMPO (2 and TRGPPO [27] We also include second-order
KL Fixed, PPO 19), MDPO 24
5

---

Update timesteps corresponding to different policy types
8
-0
query
5160
(a)
7070
0.8
(b)
(c)
Most Attended Stol 1
:35
Return
0.6
30
25
0oi
20
0.2
Elumanood
0.0
10
Hummoood Stanndd
2000
4000
6000
2M Step
10M
Policy Update Step
Figure 1: (a) Policy analysis on Unlock. First row (red lines): steps where a policy is added to M, i.e. the steps of Oj. Second row (green lines): steps of old policies Onta Third row (yellow lines): steps of mostly attended policy, approximating ds. Fourth row (cyan lines): 3 steps of interest where we want to find their attended steps. Blue arrows link query step and the step that receives highest attention. (b) Quality of ta vs. Oold. Average return collected by &b and lold at different stages of training. (c) Mujoco tasks. The slot in M received the highest attention j argmaxj fop (Veontenz) over time.
methods such as TRPO 17 and ACKTR 30
Across experiments, for MCPO, we 8mas. 10,
8min 0.01 and only tune N. More details on the baselines and tasks are given in Appendix B.1 Our code is available httptt:::/// / thub com wtt.lccooccaaeeeeeeeeeeeeeeeeeee thaihungle/HCCP
4.1 Classical Control
In this section, we compare MCPO to other first-order policy gradient methods (KL Adaptive, KL Fixed, PPO, MDPO and VMPO) on 3 classical control tasks: Pendulum, LunarLander and Bipedal- Walker, which are trained for one, one and five million environment steps, respectively. Here, we are curious to know how the model performance fluctuates as the hyperparameters vary. For each model, we choose one hyperparameter that controls the conservativeness of the policy update, and we try different values for the signature hyperparameter while keeping the others the same. For example, for PPO, we tune the clip value e; for KL Fixed we tune 3 coefficient and these possible values are chosen following prior works. For our MCPO, we tune the size the policy memory N (5, 10 and 40). We do not try bigger policy memory size to keep MCPO running efficiently (see Appendix B.2 for details).
Table I reports the results of MCPO and baselines with the best hyperparameters. For these sim- ple tasks, tuning the hyperparameters often helps the model achieve at least moderate performance. However, models like KL Adaptive and VMPO cannot reach good performance despite being tuned. PPO shows good results on LunarLander and Bipedal lum. Interestingly, if tuned properly, the vanilla KL Fixed can show competitive results compared to PPO and MDPO in BilkealWalker. Amongst all, our MCPO with suitable N achieves the best performance on all tasks. Remarkably, its performance does not fluctuate much as N changes from to 40, often obtaining good and best results. We speculate that simple tasks, either with small or big M, the policy attention performance of MCPO. On the contrary, other methods observe a clear drop in performance as hyperparameters change (see Appendix B.2 for learning curves and the full table).
yet underperforms others on Pendu-
always find optimal virtual policy vù and thus, ensures stable
4.2 Navigation
Here, we validate our method on sparse reward environments using MiniGrid library 4 In partieu- lar, we test MCPO and other baselines (same as above) on Unlock and UnlockPickupp tasks. In these tasks, the agent navigates through rooms and picks up objects complete the episode. The agent only receives reward +1 if it can complete the episode successfully. For sample efficiency test, we

---

Model
HaliChectah
Walker2d
Hopper
Ant
Humanoid
HumanoidStandup
TRPO
2,811=114
3,966456
3,159=72
2,438+402
4,576#
145, 14343,702
PPO
1,75
5,278+594
2,968+1,002
3,421+534
3,375+1,684
155,49426,66
MDPO
4,774x1,598
4,957+330
3,153=956
3,553+696
1,620+2,14 145
90,646+5,855
TRGPPO
2,811=114
5,009+391
3,713+275
4,7962837
6,242=1192
162, 162,155+3755
Mean 1
4,942+3,095
5,056+842
3,430+259
4,570+5548
353427
71,308&11,113
MCPO
6,173+595
5,120+588
3,620+252
4673+249
4,848+711
195,404432,801
Table 2: Mean and std. over runs on 6 Mujoco tasks at 10M environment steps.
train all models on Unlock (find key and open the door) and UnlockPickup (find key, open the door and pickup an object), for only 100,000 and I million environment steps, respectively. The models use the best conservative hyperparameters found in the previous task (more in Appendix
Appendix's Fig. B shows the learning curves of examined models on these two tasks. For Unlock task, except for MCPO and VMPO, 100,000 steps seem insufficient for other models to learn useful policies. When trained with 1 million steps on UnlockPickup. the baselines can find better poli- cies, yet still underperform MCPO. Here VMPO shows faster learning progress than MCPO at the beginning, however it fails to converge to the best solution. Our MCPO is the best performer, con- sistently ending up with average return of 0.9 (90% of episodes finished successfully). To achieve sample-efficiency. MCPO needs to store and search for good policies (rarely found in sparse reward problems), and adhere to it during optimization. MCPO's success may be attributed to utilizing the virtual policy that has the highest return.
To illustrate how the virtual policy supports MCPO's performance, we analyze the relationships between the old (fota), the virtual policy (25) and the policies stored in M (8z) throughout Unlock training. Fig. T (a) plots the location of these policies over a truncated period of training (from update step 5160 to 7070). Due to diversity- promoting rule, the steps where policies are added to M can be uneven (first row-red lines), often distributed right after the locations of the old policy (second row-green lines). We query at 10-th step behind the old policy (fourth row-cyan lines) to find which policy in .M has the highest attention (third row-yellow lines, linked by blue arrows). As shown in Fig. (a) (second and third row), the attended policy, which mostly resembles vb, can be further or closer to the query step than the old policy depending on the training stage. Since we let the attention network learn to attend to the policy that maximizes the advantage of current mini- batch data, the attended one is not necessarily the old policy.
diversity-roomoting
The choice of the chosen virtual policy being better than the old policy is shown in Fig. (b) where we collect several checkpoints of virtual and old policies across training and evaluate each of them on 10 testing episodes. Here using v to form the second KI. constraint is beneficial as the new policy generally pulled toward a better policy during training. That contributes to the excellent performance of MCPO compared to other single trust-region baselines, especially KI. Fixed and Adaptive, which are very close to MCPO in term of objective function style.
4.3 Mujoco
Next, we examine MCPO and some trust-region methods from the literature that are known to have good performance on continuous control problems: TRPO, PPO, MDPO and TRGPPO (an improved version of PPO). To understand the role of the attention network in MCPO, we design a variant of MCPO (N- = 40): Mean v/: which simply constructs v by taking average over policy parameters in M. This baseline represents prior heuristic ways of building trust-region from past policies (28). We pick 6 hard Mujoco tasks and train each model for 10 million environment steps. We report the results of best tuned models in Table 2 (see Appendix B.4 for full results).
The results show that MCPO consistently achieves good performance across 6 tasks, where it outper- forms others significantly in HalfCheetah, Hopper, and HumanoidStandup. In other tasks, MCPO is the second best, only slightly earning less score than the best one in Walker2d and Ant. The variant Mean ae shows reasonable performance for the first 4 tasks, yet almost fails to learn on the last two. Thus, mean virtual policy can perform badly.
To understand the effectiveness of the attention network, we visualize the attention pattern of MCPO on the last two tasks and on Hopper-a task that Mean & performs well on. Fig. (c) illustrates that
7

---

Model
PPO
ACKTR
VMPO
TRGPPO
MCPO
Mean
131.19
195.52
18.20
116.80
229.99
Median
52.85
25.30
13.56
43.24
65.78
Table 3: Average normalized human score over 9 games. The performance of each run is measured
by the best checkpoint during training over 10 million frames, averaged over 5 runs.
for the first two harder tasks, MCPO gradually learns to favor older policies in M (j* > 35). which puts more restriction the policy change the model converges. This strategy seems critical for those tasks as the difference in average return between learned v and Mean tà is huge in these cases. On the other hand, Hopper, the top attended slots are just above the middle policies in M (j* ~ 25). which means this task prefers an average restriction. As in most cases, MCPO with big policy memory (NV 40, more conservativeness) is beneficial. For those tasks where MCPO does not show clear advantage, we speculate that conservative updates are less important.
We also visualize aa for HalfCheetah task, and observe an increasing trend across training steps (see Appendix Fig. 4. As the attention network gets trained, the virtual policy becomes better and can complement the old policy when the latter performs worse. Hence, on average, the weight a, tends to be bigger overtime.
4.4 Atari Games
As showcasing the robustness of our method to nigh-dimensional inputs, we execute an experiment on a subset of Atari games wherein the states are screen images and the policy and value function approximator uses deep convolutional neural networks. We choose 9 typical games (6 were intro- duced in [1] and 3 randomly chosen) and benchmark MCPO against PPO, ACKTR, VMPO and PRGPPO, training all models for only 10 million environment steps. In this experiment, MCPO uses N = 10 and other baselines' hyperparameters are selected based on the original papers (see Appendix B.5).
As seen in Table 3. MCPO is significantly better than other baselines in terms of both mean and median normalized human score. The learning curves of all models are given in Appendix Fig. 5) To confirm MCPO maintains the leading performance with more training iterations. we train competitive models for 40 million frames for the first 6 games (Appendix Fig. 6 The results show that MCPO still outperforms other basclines after 40M frames.
4.5 Ablation Study
Finally, we verify MCPO's 3 components: virtual policy (in Eq. switching-3 3 (Eq. and diversity-promoting rule (Eq. ( We also confirm the role of choosing the right memory size N and learning to attend to the virtual policy tb. In the task Bipedal (OpenAl Gym), we train MCPO with different configurations for 50M steps. First, we tune N (5,10 and 40) using the normal MCPO with all components on and find that N = 10 is the best. Keeping N = 10, we ablate or replace our component with an alternative and report the findings as follows.
Virtual policy: To show the benefit of pushing the new policy toward the virtual policy, we imple- ment 3 variants of MCPO (N=10) that (i) does not use vs KL. term in Eq. 3 (ar 0), (ii) use fixed o, = 0.5 and (iii) only use w's KI. (a, = 1.0). All variants underperforms the normal MCPO by margin of 100 or 50 return. Switching-S:: The results show that compared to the annealed strategy adopted from MDPO and adaptive 3 from PPO-KL, our switching-$ achieves significantly writing:
better results with about 50 and 200 return score higher, respectively.
We compare our proposal with the vanilla approach that adds a new policy to M at every update step (frequent writing) and other versions that write to M every interval of 10 and 100 update steps (uniform and sparse writing). Frequent, uniform and sparse writing all show slow learning progress, and ends up with low rewards. Perhaps, frequently adding policies to M makes the memory content similar, hastening the removal of older, yet maybe valuable policies. Uniform and sparse writing are better, yet it can still add similar policies to M and requires additional effort for tuning the writing interval. Learned thr To benchmark, we try alternatives: (1) using Mean & and (2) only using half
8

---

Adaptive
KI. Fixed
(Cur))
N=
0.000
N-
annolled
d=0.01
8- 0.1
N-5
Fised a 0.5
g- 1
100
d= 0.03
mean 4
ua uaiform
-30
-150
Proquent
MDPO
PPO
clip-0.1
0g-0.1
150
B-1
clip- 0.2
Qg-
100
8:
clip- 0.3
ao
clip= 0.5
clip=0.3
0
-50
-150
1889
10M
20M
30M
40M
50M
50M
Figure 2: Ablation study Biledal/WalkerHaaddeeees33 learning curves (mean and std.
runs) across 50M training steps.
of the features in Coontent to generate 1 (Eq. D The results confirm that the Mean yb is not a strong baseline for this task, only reaping moderate rewards. Using less features for the context causes in- formation loss, hinders generations of useful v and thus, underperforms the full-feature version by margin of 50 return. For completeness, also compare methods to heavily tuned baselines and confirm that the normal MCPO (N = 10) is the best performer (see the details in Appendix B.6).
Related work
framework for model-free reinforcement learning with policy gradient approximate policy iter- ation (API), which alternates between estimating the advantage of the current policy, and updating current policy's parameters to obtain a new policy by maximizing the expected advantage 22 23 Theoretical studies have shown that constraining policy updates is critical for API [6 17 20 26
An early attempt to improve API is Conservative Policy Iteration (CPI), which sets the new policy stochastic mixture of previous greedy policies 6 16 25). Our paper differs from these works three aspects: (1) we do not directly set the new policy to the mixture, rather, we use mixture of previously found policies (the virtual policy) to define the trust region constraining the new policy via KL regularization; (2) our mixture can consist of more than modes, and thus using multiple mixture weights (attention weights); (3) use the attention network learn these weights.
Also motivated by Kakade et (2002), TRPO extends theory to general stochastic policies, rather than just mixture polices, ensuring monotonic improvement by combining maximizing the approximate expected advantage with minimizing the KL divergence between two consccutive poli- cies [17]. Arguing that optimizing this way is too conservative and hard to tune, the authors formulate the objective as a constrained optimization problem to solve with conjugate gradient and line search. To simplify the implementation of TRPO, Schulman et al. (2017) introduces first- order optimization methods and code-level improvement, which results in PPO-an an API method that optimizes clipped surrogate objective using minibatch updates.
Constrained policy improvement can be seen as Expectation-Maxinn imizing the KL-term corresponds to the Expectation step, which can be off-policy [I or on on-policy (22. From the mirror descent perspective, several works also use KI. divergence to regularize policy updates (31 24 20). A recent analysis also points out the advantages of using KI. term as a regu-
algorithms where min-

---

larizer over a hard constraint 77 Some other works improve PPO with adaptive clip range [27] or off-policy data [5]. We instead advocate using 2 trust regions and apply our idea weaker back- bone: PPO-penalty. Our method is also different from S as we do not use off- off-policy data/gradient to update the policy. Overall, our approach shares similarities with them where we also jointly opti- mize the approximate expected advantage and KI. constraint terms for multiple epochs of minibatch updates. However, we propose a novel dynamic virtual policy to construct the second trust region as supplement to the traditional trust region defined by the old or previous policy.
Prior works have promoted different utilization memory assist reinforcement learning. Unlike episodic memory concepts that stores observations across agent's 's life [ g) the memory buffer used this paper stores the weight parameters of the policy network and attention mechanism is used to query this memory. The proposed memory instead can be viewed as an instance of program memory T0 11. That said, our memory different from these prior works since our attention is learned through an auxiliary objective that optimizes the quality of the attended policy (program) with novel diversity-pon ppenmitt- writing mechanisms.
Discussion
We have presented Mentry-Constrained Policy Optimization, new method to regularize each pol- icy update with two-trust regions with respect one single old policy and another virtual policy representing multiple past policies. The new policy is encouraged to stay closer to the region sur- rounding the policy that performs better. The virtual policy is determined online through learned attention to memory of past policies. Compared to other trust-region optimizations, MCPO shows better performance in many environments without much hyperparameter tuning.
Limitations Our method introduces several new components and hyperparameters such as N and 3. Due to compute limit, we have tuned |3 extensively and thus the reported results may be the best performance that our model can achieve. Although we find the default values 8maz = 10, 8min 0.01 work well across all experiments this paper, we recommend adjustments if users apply our method to novel domains.
Negative Societal Impacts Our work aims to improve optimization in RL to reduce the sample complexity of RL algorithms. This aim is genuine, and we do not think there are immediate harmful consequences. However, we are aware potential problems such as unsafe exploration committed by the agent (e.g. causing accidents) in real-world environments (e.g. self-driving cars). Finally, malicious users can misuse our method for unethical purposes, such as training harmful RL agents and robots. This issue is typical for any machine learning algorithm, and will do best to prevent it from our end.
Acknowledgments
This research was partially funded by the Australian Government through the Australian Re- search Council (ARC). Prof Venkatesh is the recipient of ARC Australian Laureate Fellowship (FL170100006).
References
[1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018.
[2] Dimitri P Bertsekas and John Tsitsiklis. Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE conference on decision and control, volume 1, pages 560-564. IEEE, 1995.
[3] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv: 1606 16066664666 2016.
10

---

[4] Maxime Chevalier-Boiste Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym. https://githi:i www.maxwiwcb//aaeecccocc http://wwiiemmmgggmmmmmmm sym-minigrid 2018.
[5] Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3o: Policy-on policy-off policy optimization. In Uncertainty Artificial Intelligence, pages 1017-1027. PMLR, 2020.
[6] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In ICML, 2002.
[7] Nevena Lazic, Botao Hao, Yasin Abbasi- Yadkori, Dale Schuurmans, and Csaba Szepes- vari. Optimization issues in kl-constrained approximate policy iteration. arXiv preprint arXiv:2102.06234, 2021.
[8] Hung Le, Thommen Karimpanal George, Majid Abdolshah, Truyen Tran, and Svetha Venkatesh. Model-based episodic memory induces dynamic hybrid controls. In A. Beygelz- imer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[9] Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memo- rization. arXiv preprint 1911:1901.013477 2019.
[10] Hung Le. Truyen Tran, and Svetha Venkatesh. Neural stored stored-program program memory. Interna- tional Conference Learning Representations, 2019.
[11] Hung Le and Svetha Venkatesh. Neurocoder: General-purpose computation using stored neu- ral programs. In Proceedings of the 39th International Conference Machine Learning, 2022.
[12] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[13] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[14] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
[15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
[16] Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello. Safe policy iteration. International Conference on Machine Learning, pages 307-315. PMLR, 2013.
[17] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889- 1897. PMLR, 2015.
[18] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:/506.000 024388 2015.
[19] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (10.::/777.33444 2017.
[20] Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. In Proceedings of the AAAI Confer- ence Artificial Intelligence, volume 34, pages 5668-5675 2020.
11

---

[21] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. nature, 550(7676):354 -59, 2017.
[22] H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2019.
[23] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradi- methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057-1063, 2000.
[24] Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint 20Xiv:2005.09814.. 2020.
[25] Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Deep conservative policy iteration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 6070-6077, 2020.
[26] Nino Vieillard, Bruno Scherrer, Olivier Pictquin, and Matthieu Geist. Momentum in rein- forcement learning. In International Conference Artificial Intelligence and Statistics, pages $529-2538. PMLR, 2020.
[27] Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proximal policy optimization. Advances in Neural Information Processing Systems, 32:626-636 2019.
[28] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv: 1.01224, 2016.
[29] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 833-4):77-2292, 1992.
[30] Yuhuai Wu, Elman Mansimov, Shun Liao, Roger Grosse, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-Fae cronecke--acctored approximation. In Proceed- ings of the 31st International Conference Neural Information Processing Systems, pages 5285-5294, 2017.
[31] Long Yang, Gang Zheng, Haotian Zhang, Yu Zhang, Qian Zheng, Jun Wen, and Gang Pan. Policy optimization with stochastic mirror descent. arXiv preprint arXiv: 1906. 10462, 2019.
12


Reviews:
[{"review_id": "fvjnS9pi3gD", "reviewer": "Reviewer_gPBo", "summary": "Most on-policy methods used a constraint only on the distance to the most current previous policy, but the constraint can deteriorate the performance of the current policy if the previous policy has worse performance than before. To tackle this issue, this paper proposes a new on-policy method called MCPO that uses two trust region constraints: constraint on 1) distance to the most current previous policy, and constraint on 2) distance to a good policy, called virtual policy, which is computed using previous policies stored in a policy memory. The parameters of the virtual policy are a weighted expectation of those of previous policies in the policy memory. The weights in the expectation are parameterized by a neural network called the attention network, whose input is a 12-dim feature vector, and the neural network is updated to maximize the estimated performance of the virtual policy. Empirical results show that the proposed MCPO outperforms other on-policy methods in most environments and show that all components of MCPO are essential to have the best performance through ablation studies.", "questions": "1. In the ablation study, are there more results of more variants of MCPO for each component?\n\n2. Is there any reason for selecting BipedalWalkerHardcore for the experiments in the ablation study? Is there any result performed on other environments for ablation study? ", "limitations": "While the authors give numerous experimental results to show the effectiveness of the proposed method, as aforementioned in weakness, it is still seen that it is a combination of simple and heuristic methods. In this case, I recommend two ways to the authors. The first is adding more results with various heuristic methods. In the ablation study, the authors considered only one simplest variant of the proposed one to compare with MCPO. To give reasons for why the chosen methods are reasonable, I think that the authors should measure the performance of at least 3 or more variants of MCPO for each component, and compare them with MCPO. The second way is to find the explanation of what the chosen methods mean through deriving equations. For example, considering the meaning of the term $(1-\\alpha_t) KL[\\pi_{old}, \\pi_\\theta] + \\alpha_t KL[\\pi_\\psi, \\pi_\\theta]$ when $\\alpha_t$ is the softmax of the true return of $\\pi_\\psi$ and $\\pi_\\theta$ can give a reasonable explanation for choosing the proposed components. If one of these is given, it helps readers understand the effectiveness of the chosen methods.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The main ideas are 1) constraining both the previous policy and the virtual policy, 2) defining and optimizing the virtual policy using the attention network, and 3) constructing the 12-dim feature vector for input to the attention network, and these are simple but interesting.\n\n2. Numerous empirical results show the proposed MCPO outperforms other on-policy algorithms in most environments and show the performance of variants of MCPO to ensure that all components of the method are essential to achieve the performance. \n\n\nWeakness\n1. Most of the components of the proposed method are designed heuristically based on some intuitions. For example, designing $\\alpha_t$, switching $\\beta$-rule, and the rule of storing a policy in the policy memory are all heuristically chosen methods. Although the authors showed the performance of MCPO with $\\alpha_t = 0$, with annealing $\\beta$, or with frequent writing and uniform writing, it is not enough to give reasons for why these methods improve the performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dju5bURgXIJ", "reviewer": "Reviewer_ECcM", "summary": "This work addresses the problem of policy optimization in policy gradient methods (PG) in deep RL. Due to the off-policy nature of using mini-batch data to make updates to the policy, the policy updates can become very far from the areas where data has been collected. In this work, the authors propose to construct a so-called virtual policy that is a weighted mixture of several past policies, where the weights are learned via an attention network. The policy updates are then regularized to stay within a KL-distance region around the virtual policy and the previous behaviour or “old policy”. In this way, if the old policy performs poorly or is stuck in a local optimum, the new policy will be encouraged towards the virtual policy. The authors provide empirical evidence of their proposed method having better performance than some previous works that also form trust regions or constraints of similar kinds (i.e. TRPO, PPO, and the like) on a variety of domains.", "questions": "- One potential issue I see with the way Eqn (2) chooses which new policy to include in its memory is that if the list of policies becomes very uniform $D(\\theta_{old}, \\psi)$ would become very small and many policies would get added, potentially making the usage of the memory useless. Have the authors observed something like this in their experimentation and how do they address it?\n- L253-254: Regarding Hopper “prefers an average restriction”. Not sure I understand this point. What does it mean that it prefers an average restriction? To my understanding, since MCPO favours better-performing policies, this means that the ~middle policy performed the best, but on average all the policies were not very bad since \"Mean $\\psi$\" also performed similarly. Are the authors saying this is related to the environment? If so, what kind of qualities of the environment might lead to this? Or is this more related to the optimization procedure? \n- How do we know if the regularization is working, i.e. the new policy is closer to $\\psi$ when the old policy becomes stuck or bad, and that this is the reason for improved results? I think some hand-designed experiments that demonstrate or visualize the effect of having the virtual policy would be interesting (similar to Figure 1 but more comprehensive, i.e. including situations where $\\theta_{old}$ is forced to be a bad or locally optimum policy)\n", "limitations": "The authors discuss some reasonable limitations in their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method of keeping a virtual policy that keeps track of past policies in order to avoid the pitfalls of simply keeping close to a potentially bad or suboptimal old policy is straightforward and sensible. I also think the use of a learned attention network to weigh these policies is a good idea. The empirical evaluation also shows good results on a wide variety of domains.\nI think the work lacks in a theoretical understanding of the impact of the proposed method. Does the virtual policy really help escape local optima? Are the improvements observed in performance really due to the virtual policy or are they due to for example, an improved optimization with the new objective? \n\nStrengths:\n- Good idea, simple to implement and use, asymptotic results, and learning curves (in the appendix) appear to consistently show improved performance over baselines.\n\nWeaknesses:\n- In-depth understanding of the impact of virtual policy as stated above.\n\nWriting:\n- Overall, the writing quality is good, and the paper is easy to follow.\n- Definitions of setting is missing, for example I did not realize we were in a finite-horizon setting until I read Alg 1. \n- L168: typo: optimize $\\theta$ and *$\\phi$* alternately\n- L178: N or M (L113-115) for number of past policies?\n- Section 4.2 results are in the Appendix but talked about in the main body. I understand space constraints but I would recommend only talking in-depth about results that can be shown in the main body for improved readability.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1nVP8hv0DfS", "reviewer": "Reviewer_HVYg", "summary": "The paper considers the trust-region-based policy optimization methods in deep reinforcement learning for which the policy update (from an old policy to a new one) at each iteration should be constrained by a trust region. This paper argues that for this family of methods the trust region constraint could impede the policy update since the old policy might not necessarily be a good one to define the trust region. The paper then proposes an improvement by incorporating into the TRPO objective a second type of trust region, which is defined as the KL divergence between the target policy and a linear combination of some history policies, i.e., *virtual trust region*. In order to effectively find such linear combination, the paper presents an attention-based mechanism to learn the combination coefficients based on some contextual features of the policy. The paper also empirically evaluates the augmented trust region method, Memory-Constrained Policy Optimization (MCPO) on both discrete and continuous control tasks.  ", "questions": "Q1: After digging into Eq. (3), I found the virtual trust region could be replaced by the cross entropy between $\\pi_\\psi$ and $\\pi_\\theta$ without affecting the maximization. In this sense, having the “virtual trust region” could be effectively imposing an entropy regularization. Could it be possible that having an entropy regularization (i.e., maximum entropy) for TRPO and all other trust-region based methods would yield a similar empirical performance as MCPO? We can try decaying entropy coefficient or tuning the coefficient based on the learning progress. \n\n\nQ2: (line 151) How to estimate $R_t(\\psi)$ from $A_t(\\theta_{old})$ via importance sampling given that their state distribution are also different? \n\nQ3: (algo 1 line 6) How to compute $\\psi$ via Eq. 1 using $\\psi_{old}$, $\\theta$, and $\\theta_{old}$? \n\nQ4: How the parameter $\\varphi$ is updated when optimizing Eq. (6)? \n\nQ5: (line 149) Why the $\\alpha_t$ is defined as the ratio of return exponents? This $\\alpha_t$ is different from what’s being used in Eq. 3 as it does not condition on $s_t$. \n\nQ6: (line 255 to 256) the paper states that “conservative updates are less important in some tasks”. Does it mean the virtual trust region constraint could impede training in some environments? This would be contradictory to the underlying motivation of this paper. ", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n\n[clarity] The paper presents an interesting idea of “extrapolating” the trust region by introducing a virtual one. The way of learning an attention-based mechanism to determine the trust region is also inspiring. \n\n\n**Weaknesses**: \n\n[originality] Lack of convincing evidence to motivate the work and method: The paper argues that the trust region defined according to the old policy could lead to “suboptimal policy update” (line 34 to 36, line 103 to 104). This argument is interesting. It would be more convincing if the paper could provide any empirical evidence to support such argument. Also, it is unclear how the “new policy may fall into a local optimum”. The trust region constraint itself is to ensure that updating policy could result in a performance improvement. That is, as long as the policy can be improved, regardless how much the improvement is, the trust region is then effective and the policy update is valid. Thus, it remains unclear to me how such local optimum would really impact the training. \n\n[clarity & significance] Lack of important technical details: One of major contributions of this paper is introducing “virtual trust region”, which is defined as the KL divergence between the psi policy and the new policy. The psi policy is constructed based on the context vector (Tab 4). However, it remains largely unknown what the distance between $\\theta$ and $\\psi$ is (feature dim 1, 2 & 3), how the return of a new policy (before it is updated) is computed (feature dim 6, 10, 11 & 12). Furthermore, it is also unclear how the psi weights are optimized. It is not straightforward how Eq. (5) is maximized w.r.t. $\\varphi$. The lack of these important details could downplay the significance of this work. \n\n[experimental results] Lack of insightful results to support how virtual trust region works: The paper presents many empirical results of MCPO. Most of them are from the lens of a whole policy training process, e.g., empirical returns. It is, however, largely unclear what role the virtual trust region plays at each policy iteration. Given that the main idea of the paper is about how a “bad” old policy would affect the trust region constraint, it would make more sense to dig into the trust region analysis (e.g., empirical KL estimates) at each iteration, rather than the empirical returns (one such analysis can be found in PPO paper, Figure 2). Moreover, the ablation studies presented in the current version are very inconclusive. No detailed training curves or analytical statistics are available. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept