V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
Authors: H. Francis Song
Abbas Abdolmaleki
Jost Tobias Springenberg
Aidan Clark
Hubert Soyer
Jack W. Rae
Seb Noury
Arun Ahuja
Siqi Liu
Dhruva Tirumala
Nicolas Heess
Dan Belov
Martin Riedmiller
Matthew M. Botvinick
Venue:  (2020)
Keywords: reinforcement learning
policy iteration
multi-task learning
continuous control

Abstract: Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned state-value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.

Published as a conference paper at ICLR 2020
V-MPO: ON-POLICY MAXIMUM A POSTERIORI POLICY OPTIMIZATION FOR DISCRETE AND CONTINUOUS CONTROL
H. Francis Song, Abbas Abdolmaleki; Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Riedmiller, Matthew M. Botvinick DeepMind, London, UK
{songf, aabdolma leki, htw.:b/mmrmiii..ggggggnnneeegrr soyer, jwrae, snoury, arahuja, liusiqi, dhruvat, heess, danbelov, riedmi botvinick com
ABSTRACT
Some of the most successful applications of deep reinforcement learning to chal- lenging domains discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algo- rithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based learned state- value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-base based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAl Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.
INTRODUCTION
Deep reinforcement learning (RL) with neural network function approximators has achieved superhu- man performance in several challenging domains (Mnih al.) 2015] Silver al. 2016 2018). Some of the most successful recent applications of deep RL to difficult environments such as Dota 2 (Ope nAI 2018a), Capture the Flag (Jaderberg al. 2019), Starcraft II (Vinyals al. 2019), and dexterous object manipulation (OpenAI]20195) 2018b) have used policy gradient-based methods such as Proximal Policy Optimization (PPO) (Schulman al. 2017) and the Importance- Weighted Actor-Leamer Architecture (IMPALA) (Espeholt et al. 2018) both in the approximately on-policy setting.
Policy gradients, however, can suffer from large variance that may limit performance, especially for high-dimensional action spaces (Wu et al., 2018). In practice, moreover, policy gradient methods typically employ carefully tuned entropy regularization in order to prevent policy collapse. As an alternative to policy traddent-bbsed algorithms, in this work we introduce approximate policy iteration algorithm that adapts Maximum Posteriori Policy Optimization (MPO) (Abdolmaleki et al.) 2018a b) to the on-policy setting. The modified algorithm, V-MPO, relies on learned state-value function (s) instead of the state-action value function used in MPO. Like MPO, rather than directly updating the parameters in the direction of the policy gradient, V-MPO first constructs target distribution for the policy update subject to a sample-based KL. constraint, then calculates the gradient that partially moves the parameters toward that target, again subject KL constraint.
Equull contribution

---

Published as a conference paper at ICLR 2020
As are partieularly interested in scalable RL algorithms that can be applied to multi-task settings where a single agent must perform a wide variety of tasks, we show for the case of discrete actions that the proposed algorithm surpasses previously reported performance in the multi-task setting for both the Atari-57 (Bellemare et al., 2012) and DML.ab-30 (Beattie et al., (016) benchmark suites, and does so reliably without population-base For few individual levels in DMLab and Atari we also show that V-MPO can achieve scores that are substantially higher than has previously been reported in the single-task setting, especially in the challenging Ms. Pacman.
tuning of hyperparameters (Jaderberg al.) (017a).
V-MPO is also applicable to problems with nigh-dimensional, continuous action spaces. We demon- strate this in the context of learning to control both a 22-dimensional simulated humanoid from full state observations- where V-MPÓ reliably achieves higher asymptotic performance than previous algorithms- and 36-dimensional simulated humanoid from pixel observations (Tassa et al. 20188) Merel et al. 2019).. In addition, for several OpenAl Gym tasks (Brockman et al. (2016) we show that V-MPO achieves higher asymptotic performance than has previously been reported.
BACKGROUND AND SETTING
We consider the discounted RL setting, where we seek to optimize a policy π for a Markov Decision
Process described by states s, actions initial state distribution p6 (so), transition probabilities (s-( [8t, (t), reward function r(st, at), and discount factor y E (0, 1). deep RI., the policy To(a; [(),](s),, which specifies the probability that the agent takes action a, in state 8, at time t, is described by a neural network with parameters 6. We consider problems where both the states and actions may be discrete or continuous. Two functions play central role in RL: the state-value function V (st)) = Eo Pes. 16 Ex=o r dx+x)] and the state-action value function Q" (*,,,, at) = Exy,a Ei ( ac+x)] = r(8), az) + BB (s++1)). where 80 ~ P0 "000 (8o). at ~ (a)))))) and 8141 ~ "pen (st+1 az).
In the usual formulation of the RL problem, the goal is to find a policy π that maximizes the expected Eee 7"r(st, In policy gradient algorithms (Williams
return given by J(TT) = Egg.e..s ......... 1992, Sutton et al. 2000; Mnih et al. 2016). for example, this objective is directly optimized by estimating the gradient of the expected return. An alternative approach to finding optimal policies derives from research that treats RI. as a problem in probabilistic inference, including Maximum a Posteriori Policy Optimization (MPO) (Levine, 2018; Abdolmaleki et al. [01 8a b). Here our objective is subtly different, namely, given suitable criterion for what are good actions take in a certain state, how do we find a policy that achieves this goal?
As was the case for the original MPO algorithm, the following derivation is valid for any such criterion. However, the policy improvement theorem (Sutton & Barto 1998) tells us that policy update performed by exact policy iteration, (s) = arg max, (² a) V (s)]. can improve the
policy if there is at least one state-action pair with a positive advantage and nonzero probability of
visiting the state. Motivated by this classic result, this work we specifically choose an exponential
function of the advantages AT (s, a) Q7 a) (+))).
Notation. the following we use Σ., to indicate both discrete and continuous sums (i.e., integrals) over states & and actions a depending on the setting. A sum with indices only, such as Σ.,> denotes a sum over all possible states and actions, while E.a-D. for example, denotes a sum over sample states and actions from a batch of trajectories (the "dataset") D.
3 RELATED WORK
V-MPO shares many similarities, and thus relevant related work, with the original MPO algorithm (Ab dolmaleki et al. 2018a b). In particular, the general idea of using KL constraints to limit the size of policy updates is present in both Trust Region Policy Optimization (TRPO; Schulman al. 2013) and Proximal Policy Optimization (PPO) (Schulman et al. 2017); we note, however, that this corresponds to the E-step constraint in V-MPO.
is worth noting here the following main differences with MPO, which is conceptually quite similar to V-MPO. MPO is primarily designed to be a sample-efficient off- off-policy algorithm in which the
2

---

Published as a conference paper at ICLR 2020
(a)
(b)
m
1
ACTOR
MLP
MLP
Target π
Trajectories
POLICY
VALUE
SHARED
LSTM or TrXL
Prev. action
Prev. neward
Target π
Every Targ stops
Online , V
LEARNER
ResNet
Language LSTM (DMLab only)
(a) Actor-learner architecture with a target network, which is used to generate agent experience in the environment and is updated every Turger learning steps from the online network. (b) Schematic of the agents, with the policy (0) and value (Ø) networks sharing most of their parameters through a shared input encoder and LSTM [or Transformer- XL (TrXL) for single Atari levels]. The agent also receives the action and reward from the previous step an input to the LSTM. For DMLab an additional LSTM used to process simple language instructions.
Figure 1:
E-step constructs conditional target distribution {(ass), which requires state-action value function Q(s, a) that can evaluate multiple sampled actions for a given state. In contrast, V-MPO is primarily (though not exclusively) designed joint distribution (s, a), and in the absence of a learned Q-function only one action per state is used. In this regard V-MPO can also be compared to Fitted Q-iteration by Advantage Weighted Regression (Neumann & Peters 2009), which learns a Q-function but uses only one action per state.
be on-policy algorithm in which the E-step constructs
V-MPO also be related Relative Entropy Policy Search (REPS) (Peters al. 2008). Two distinguishing features of V-MPO from REPS are the introduction of the M-step KI. constraint and the use of top-k advantages. Morcover, in REPS the value function is linear function of learned feature representation whose parameters are trained by matching the feature distributions under the policy's 's stationary state distribution. In V-MPO, the nonlinear neural network value function is instead learned directly from n-step returns. Interestingly, previous attempts to use REPS with neural network function approximators reported very poor performance, being particularly prone to local optima (Duan et al. 2016). In contrast, we find that the principles of EM-style policy optimization, when combined with this Iearned value function and appropriate constraints, can reliably train powerful neural networks, including transformers, for RL tasks.
Like V-MPO, Supervised Policy Update (SPU) (Vuong et al., 2019) seeks to exactly solve an optimization problem and fit the parametric policy to this solution. As we argue in Appendix D however, SPU uses this nonparametric distribution quite differently from V-MPO; as a result, the final algorithm is closer to policy gradient algorithm such PPO.
4 METHOD
V-MPO is approximate policy iteration (Sutton & Barto 1998) algorithm with specific prescrip- tion for the policy improvement step. In general, policy iteration uses the fact that the true state-value function = corresponding policy can be used obtain an improved policy n². Thus we
1. Generate trajectories from old policy Te.. (a]s) whose parameters loi fixed. To control the amount of data generated by a particular policy, we use a target network which is fixed for T Target learning steps (Fig. TT).
3

---

Published as a conference paper at ICLR 2020
2. Evaluate the policy Te (als) by learning the value function V (s) from empirical returns and estimating the corresponding advantages A a (s, a) for the actions that were taken (Section 4.1
3. Based on A (s, a), estimate an improved policy xa (als) which we call the "online" policy to distinguish it from the fixed target network (Section 4.2p.
The first two steps standard, and describing V-MPO's approach to step is the essential contribu- tion of this work. At a high level, our strategy is to first construct a nonparametric target distribution for the policy update. then partially move the parametric policy towards this distribution subject to a KL constraint. We first review policy evaluation (step 2) in Section 4.1 then derive the V-MPO policy improvement (step 3) in Section|4.2 Ultimately,
gradient descent to optimize single, relatively simple loss, which is given in Eq. I0 following the derivation. A summary of the full algorithm is also presented in Algorithm D
4.1 POLICY EVALUATION
In the present setting, policy evaluation means learning an approximate state-value function V* (s) given a policy x (a]s), which we keep fixed for T Turget learning steps (i.e., batches of trajectories). We note that the value function corresponding to the target policy is instantiated in the "online" network receiving gradient updates; bootstrapping uses the online value function, as it is the best available estimate of the value function for the target policy. Thus in this section refers to ae while the value function update is performed on the current o, which may share parameters with the current A.
We fit parametric value function V (s) with parameters ø by minimizing the squared loss
Σ (V3(s)) 2 SIND
(v( 2|D
where ((n)) is the standard n-step target for the value function at state 8 time (Sutton & Barto 1998). This return uses the actual rewards in the trajectory and bootstraps from the value function for the rest: for each f = =t.... t + 1 in an unroll, c(a)) -itrr " - + +f fftn-(/ (s+4)) (st4n)
The advantages, which are the key quantity of interest for the policy improvement step in V-MPO,
are then given by A (s 0z) = Gla) VJ (st) for each st, 0g in the batch of trajectories.
PopArt normalization. As are interested in the multi-task setting where single agent must learn large number of tasks with differing reward scales, we used PopArt (van Hasselt et al., 2016 Hessel al. 2018) for the value function, even when training a single task. We observed benefits in using PopArt even in the single-task setting, partly due to the fact that we do not tune the relative weighting of the policy evaluation and policy improvement losses despite sharing most parameters for the policy and value networks. Specifically, the value function outputs a separate value for each task in normalized space, which is converted to actual returns by shift and scaling operation, the statistics of which are learned during training. We used a scale lower bound of 10 10-2, scale upper bound of 10 and learning rate 10 4 for the statistics. The lower bound guards against numerical issues when rewards are extremely sparse.
importance-weighingg for off-policy data. is possible to importance-weight the samples using V-trace to correct for off-policy data (Espeholt et al. 2018), for example when data is taken from a replay buffer. For simplicity, however, no importance-weighting was used for the experiments presented in this work, which were mostly on-policy.
4.2 POLICY IMPROVEMENT IN V-MPO
In this section we show how, given the advantage function A (s, a) for the state-action distribution Pa (s. a) = To (a|s) (als)((s) (s) induced by the old policy no (als),
estimate improved policy Te(a|s). More formally, let I denote the binary event that the new policy is an improvement (in sense be defined below) over the previous policy: I if the policy is successfully improved and 0 otherwise. Then we would like to find the mode of the posterior distribution over parameters conditioned on this event, i.e., we seek the maximum a posteriori (MAP) estimate
8* arg max
log (I = 1) + log (
(2)
8
4

---

Published as a conference paper at ICLR 2020
where we have written p(I = 1 Po(I 1) to emphasize the parametric nature of the dependence
on A. We use the well-known identity log p( = Evyz) [lo e(2)
p(X D (zz))))))(( for
divergence between (Z.) and p( x with respect to Z, and the first term a lower bound because the KI. divergence
any latent distribution 2b (Z)) where DKL(wi [) (] is the Kullback-Leii
is always non-negative. Then considering 8, as latent variables.
log pa (I = 1) = Σ a) log Po = 1,8,a) + DRa a)llooss 8,a
(3)
((8,a)
Policy improvement in V-MPO consists of the following two steps which have direct correspondences the expectation maximization (EM) algorithm (Neal Hinton 1998): the expectation (E) step, we choose the variational distribution (0s, a) such that the lower bound on Po = 1) is as tight as possible, by minimizing the KL term. In the maximization (M) step we then find parameters 0 that maximize the corresponding lower bound, together with the prior term in Eq. 2
4.2.1 E-STEP
In the E-step, our goal is to choose the variational distribution (b(s, a) such that the lower bound on log po = 1) as tight as possible, which is the case when the KL. term in Eq. 3s zero. Given the old parameters low, this simply leads to 2)(s a) Po.a alI = 1), or
(s, pe (s, a)po a (I = 1/8, a)
Pal =
(I= 1)= Σ Pose (s, a)Po (I= = 1|s,a), a).
(4)
Pe II 1)
8,a
Intuitively, this solution weights the probability of each state-action pair with its relative improvement probability Pe.a (I = 1|s, a). We now choose a distribution Pe... (I = 1 (ss a) that leads to our desired
outcome. As prefer actions that lead higher advantage in each state, suppose that this probability is given by
Pols (I 1|s, a) x exp ( (ATROu (17 (8,
(S)
77
for some temperature 1) > O, from which we obtain the equation the right in Eq. 12 This probability depends on the old parameters Oota and not on the new parameters 0. Meanwhile, the value of 1) allows to control the diversity of actions that contribute to the weighting, but the moment is arbitrary. It turns out, however, that we can tune as part of the optimization. which is desirable since the optimal value of 77 changes across iterations. The convex loss that achieves this, Eq. is derived in Appendix A by minimizing the KL term in Eq. 3 subject to a hard constraint on (s)
Top-k advantages. We found that learning improves substantially if we take only the samples corresponding the highest 50% of advantages each batch for the E-step, corresponding to the use of DD rather than DD in Eqs. 12 maximum likelihood weights in Eq. 12 and the temperature loss in Eq. 13 since, mathematically, this corresponds to specific choice of the policy improvement probability in Eq. 5 to only use the top half of the advantages. This is similar to the technique used in the Cross Entropy Method (CEM) (Mannor et al. 2003) and Covariance Matrix Adaptation Evolutionary Strategy (CMA- ES) (Hansen al.. 1997| Abdolmaleki et al. 2017). and is special case of the more general feature that any rank-preserving transformation is allowed under this formalism. For example, in Fig. [ of the Appendix we show example of an agent trained with uniform weights given the top-k samples, instead of optimizing the temperature. Other choices are possible, and in future work we will investigate the suitability of different choices for specific applications.
13
Importantly, these must be consistent between the
Importance weighting for off-policy corrections. As for the value function, importance weights can be used in the policy improvement step to correct for off-policy data. While not used for the experiments presented in this work, details for how to carry out this correction are given in Appendix
4.2.2 M-STEP: CONSTRAINED SUPERVISED LEARNING OF THE PARAMETRIC POLICY
In the E-step we found the nonparametric variational state-action distribution (bss a). Eq. that gives the tightest lower bound (I 1) in Eq. In the M-step maximize this lower bound
5

---

Published as a conference paper at ICLR 2020
together with the prior term log p(0) with respect to the parameters 8, which effectively leads to constrained weighted maximum likelihood problem. Thus the introduction of the nonparametric distribution in Eq. separates the RL procedure from the neural network fitting.
We would like to find new parameters that minimize
[(0) - -.-E ass log Po Po(I=1,,,00) log p(0).
(ss
Note, however, that far have worked with the joint state-action distribution vb(s, a) while we are in fact optimizing for the policy, which is the conditional distribution To(als). Writing pa(s, a) = he (a is (p(s) since only the policy is parametrized by and dropping terms that are not parametrized by 0, the first term of Eq. is seen to be the weighted maximum likelihood policy loss
Ca (0)
Σ (s, a) log (a(ss).
In the sample- cample-based computation of this loss, we assume that any state-action pairs not in the batch of trajectories have zero weight, leading to the normalization in Eq, [2
As in the original MPO algorithm, a useful prior is to keep the new policy (a)ss) close to the old policy (a (a) :: log p(0) x -aBx--(ss) DKL (Te (Te..
(a|s)] T (a) [als))]] While intuitive, we motivate
this more formally Appendix [ It again more convenient specify bound on the KL divergence instead of tuning a directly, so we solve the constrained optimization problem
0 = arg min
Σ (s, a) log (als) s.l. 8,a
E srop(s)
DKL (aT) (als)]
< Ea.
(8)
0
Intuitively, the constraint in the E-step expressed by Eq. 18 Appendix A for tuning the temperature only constrains the nonparametric distribution; it is the constraint in Eq. 8 that directly limits the change in the parametric policy, particular for states and actions that were not in the batch of samples and which rely on the generalization capabilities of the neural network function approximator.
To make the constrained optimization problem amenable to gradient descent, we use Lagrangian relaxation to write the unconstrained objective as
=( = +
E srpp(e)
DKL (Tew (a)s] (Toss\([((aas
which we can optimize by following coordinate- strategy, alternating between the opti- mization over and a. Since and a are Lagrange multipliers that must be positive, after each gradient update project the resulting y and a to small positive value which we choose to be 7min = mmin = 10 throughout the results presented below.
KL constraints in both the E-step and M-step are generally well satisfied, especially for the E-step since the temperature optimization is convex. Fig. / in the Appendix shows an example of how the KL constraints behave in the Atari Seaquest experiment presented below. We note, in partieular, that it is desirable for the bounds to not just be satisfied but saturated.
4.3 FULL LOSS FUNCTION
In this section we provide the full loss function used to implement V-MPO, which perhaps simpler than is suggested by the derivation. Consider a batch of data D consisting of number of trajectories, with [D] total state-action samples. Each trajectory consists of an unroll of length n of the form T (st, az, (141)..... ($t+n-1- ac+n-1, Tttn), St+n] including the bootstrapped state sten, where r++1 r(st, The total loss is the sum of policy evaluation loss and policy improvement loss,
L( f, a. = ( + LvMMP (f, 17, a),
(10)
where & are the parameters of the value network, 0 the parameters of the policy network, and and are Lagrange multipliers. In practice, the policy and value networks share most of their parameters in the form of shared convolutional network (a ResNet) and recurrent LSTM core, and are optimized together (Fig. (T)) (Mnih et al.) 2016). We note, however, that the value network parameters & are considered fixed for the policy improvement loss, and gradients are not propagated.

---

Published as a conference paper at ICLR 2020
The policy evaluation loss for the value function, Ly (6), is the standard regression n-step returns and is given by Eq. [ above. The policy improvement loss LvMrr((e 77, a) is given by
(0, 77, ar) L, (0) (n) La ())
(11)
Here the policy loss is the weighted maximum likelihood loss
A A
exp
(0) -- Σ
vis a) log (o(as))
()) = E s. exp Aume((a.))
(12)
2,00D
where the advantages A (s, a) for the target network policy Thomp (als) are estimated according the standard method described above. The tilde over the dataset, D, indicates that we take samples corresponding to the top half advantages in the batch of data. The 7) "temperature", loss is
A (s,
a) a)
]
(n( ney + log
exp
(13)
2,0N
We perform the alternating optimization over O and while keeping single loss function by alternately applying a "stop-gradient" to the Lagrange multiplier and KI. term. Then the KI. constraint, which can be viewed form of trust-region loss, is given by
(0, =
1 D
Σ SED
a((aa sg (Touma (ds) + sg[[a]] DKL (T (a|s)][=((a|s))
(14)
where sg[[-]] indicates stop gradient, i.e., that the enclosed term is assumed constant with respect to all variables. Note that here the full batch D, not D.
For continuous action spaces parametrized by Gaussian distributions, we use decoupled KL constraints for the M-step in Eq. 14 as in Abdolmaleki et al. (2018b); the precise form is given in Appendix C
We used the Adam optimizer (Kingma & Ba 2015) with default TensorFlow hyperparameters to optimize the total loss in Eq. 10 In particular, the learning rate fixed at 10-4 for all experiments.
Algorithm 1 V-MPO
given Batch size B, unroll length n. Turget. KL bounds Cy. initialize Network parameters oonlines Pontines Lagrange multipliers 7), a.
repeat
Orarget f- Ounline
inil=..... Tunger do
Use policy Tanm to act the environment and collect trajectories 7 of length n. Update Oonline. Poulines 77, using Adam to minimize the total loss in Eq. I0 17 < max (7, Mmin)
max Ommin)
end for until Fixed number of steps.
5 EXPERIMENTS
Details on the network architecture and hyperparameters used for each task are given in Appendix
5.1 DISCRETE ACTIONS: DMLAB, ATARI
DMLab. DMLab-30 30 Beattie et al. 2016) is a collection of visually rich, partially observable 3D environments played from the first- first-person point of view. Like IMPALA, for DMLab used pixel control as an auxiliary loss for representation learning (Jaderberg et al. (2017)] Hessel et al. 2018). However, did not employ the optimistic asymmetric reward scaling used by previous IM IMPALA
7

---

Published as a conference paper at ICLR 2020
100
160
Modian human normallzd
140
120
90
100
80
80-
100- ain-capped
V-MPO (24)
INPALA (3x8 PST)
V-HPT (24)
R20 108
IMPALA
10
12
Envirenment (4 (4xagent) steps (B)
Environment (4xagent) steps (B)
(a) Multi-task DMLab-30.
(b) Multi-task Atari-57.
Figure 2: (a) Multi-task DMLab-30. IMPALA results show 3 runs of agents each; within run hyperparameters were evolved via PBT. For V-MPO each line represents a set of hyperparameters that are fixed throughout training. The final result of R2D2+ trained for 10B environment steps on individual levels (Kapturowski et al. 2019) is also shown for comparison (orange line). (b) Multi-task Atari-57. In the IMPALA experiment, hyperparameters were evolved with PBT. For V-MPO each of the 24 lines represents set of hyperparameters that were fixed throughout training, and all runs achieved a higher score than the best IMPALA run. Data for IMPALA ("Pixel-PopAre IMPALA for DMI aab330 and "PopArt-IMPALA" for Atari-57) obtained from the authors of Each agent step corresponds to 4 environment frames due to the action repeat.
Hessel et al. 20118)
www.tncceeeeooooooooo gwws,
ore, sijectt rwwooddeeemm
Poatt N0000 Homaer
102
300
VAPO 221
VHED PPILLAPET
Iaadoo
23
7
En 14 sageett stepe 181
Figure 3: V-MPO trained on single example levels from DMLab-30, compared to IMPALA and more recent results from R2D2+, the larger, DMLab-specific version of R2D2 (Kapturossii et al. 2019). The IMPALA results include hyperparameter evolution with PBT.
experiments to aid exploration subset of the DMLab levels, by weighting positive rewards more than negative rewards (Espeholt et al. 20118] Hessel et al. 2018 Kapturowski et al. 2019). Unlike in Hessel et al. (2018) we also did not use population-based training (PBT) (Jaderberg et al., 2017a).
Additional details for the settings used in DMLab can be found in Table 5 of the Appendix.
Fig. 2b shows the results for multi-task DMLab-30, comparing the V-MPO learning curves to data obtained from Hessel et al. (2018) for the PopArt IMPALA agent with pixel control. We note that the result for V-MPO at TOB environment frames across all levels matches the result for the Recurrent Replay Distributed DQN (R2D2) agent (Kapturowski et al., 2019) trained on individual levels for 10B environment steps per level. Fig. 3gg.Bssows example individual levels in DMLab where V-MPO achieves scores that are substantially higher than has previously been reported, for both R2D2 and IMPALA. The pixel-control IMPALA agents shown here were carefully tuned for DMI and are similar to the "experts" used in Schmiit et al. (2018); in all cases these results match or exceed previously published results for IMPALA (Espeholt et al. 2018 Kapturowski et al. 2019).
Atari. The Atari Learning Environment (ALE) (Bellemare et al. 2012) is collection of 57 Atari 2600 games that has served as an important benchmark for recent deep RL methods. We used the standard preprocessing scheme and maximum episode length of 30 minutes (108,000 frames), T in the Appendix. For the multi-task setting we followed Hessel et al. (2018) in setting the discount to zero on loss of life; for the example single tasks we did not employ this trick, since it
8

---

Published as a conference paper at ICLR 2020
133 SUTeeT
Soagastt
08040
8000
1500
19.07
13000
20003
Vieeo 19.1144
Crasonmart (404ggr) wicpa
Figure 4: Example levels from Atari. In Breakout, V-MPO achieves the maximum score of 864 in every episode. No reward clipping was applied, and the maximum length of an episode was 30 minutes (108,000 frames). Supplementary video for Ms. Pacman: https: htps:///bit.... 1y/ WQBB5
fumorald
10000
Edkedd REATTS
Epbodd
4010
4000
Opo Wilkobild
Waro 03)
200
6. Envireneentt stoss
Envromment 251
Enviurment siopu
Exerceement
(a)
Figure 5: (a) Humanoid "run" from full state Tassa et al. 2018 and (b) humanoid "gaps" from pixel observations (Merel et al. 2019). Purple curves are the same runs but without parametric KL constraints. Det. eval.: deterministic evaluation. Supplementary video for humanoid gaps: https://bit. http::///bit.y//29KKK2. 1y..1122222 (c)-(d) Example OpenAl Gym tasks. See also Fig. II in the Appendix for Gym Humanoid-Vi.
can prevent the agent from achieving the highest score possible by sacrificing lives. Similarly, while in the multi-task setting we followed previous work in clipping the maximum reward to 1.0, no such clipping was applied in the single-task setting in order to preserve the original reward structure. Additional details for the settings used in Atari can be found in Table i the Appendix.
Fig. 2 shows the results for multi-task Atari-57, demonstrating that it is possible for single agent to achieve "superhuman" median performance Atari-57 in approximately 4 billion (~70 million per level) environment frames. Again, while we did not employ PBT in order to demonstrate that individual V-MPO runs can exceed the performance of a population of IMPALA agents, Fig. 6 shows that with population-basee tuning of hyperparameters even higher performance is possible.
We also compare the performance of V-MPO on few individual Atari levels R2D2 (Kapturosski et al. 2019), which previously achieved some of the highest scores reported for Atari. Again, V. MPO can match or exceed previously reported scores while requiring fewer interactions with the environment. In Ms. Pacman, the final performance approaches 300,000 with 30-minute timeout (and the maximum 1M without). Inspired by the argument Kapturowski et al. (2019) that in a fully observable environment LSTMs enable the agent to utilize more useful representations than available in the immediate observation, for the single-task setting we used a Transformer-A XL (TrXL) (Dai et al. 2019) to replace the LSTM core. Unlike previous work for single Atari levels, we did not employ any reward clipping MMii et al. 2015 Espeholt et al. (018)) or nonlinear value function rescaling (Kapturowski et al. 2019).
5.2 CONTINUOUS CONTROL.
To demonstrate V-MPO's effectiveness in high-dimensional, continuous action spaces, here present examples of learning to control both a simulated humanoid with 22 degrees of freedom from full state observations and with 56 degrees of freedom from pixel observations (Tassa et al. 2018 Merel et al., 2019). As shown in Fig. 5h, for the 22-dimensional humanoid V-MPO reliably achieves higher asymptotic returns than has previously been reported, including for Deep Deterministic Policy

---

Published as a conference paper at ICLR 2020
Gradients (DDPG) (Lillierap al. 2013). Stochastic Value Gradients (SVG) (Heess al. 2013). and MPO. These algorithms are far more sample-efficiem but reach a lower final performance.
In the "gaps" task the 56-dimensional humanoid must rum forward to match target velocity of 4 m/s and jump over the gaps between platforms by learning to actuate joints with position-control (Merel et al., 2019). Previously, only agent operating in the space of pre-learned motor primitives was able to solve the task from pixel observations (Merel et al. 2018 2019); here we show that V-MPO can learn challenging visuomotor task from scratch (Fig. 5p. For this task we also demonstrate the importance of the parametric KL constraint, without which the agent learns poorly.
In Figs. 5c-d we also show that V-MPO achieves the highest asymptotic performance reported for two OpenAI Gym tasks (Brockman et al., 2016). Again, MPO and Stochastic Actor-Critic (Haarnjja al. 2018 are far more sample-efficiet but reach a lower final performance.
These experiments are presented to demonstrate the existence of higher-return solutions than have previously been reported, and an algorithm, V-MPO, that can reliably converge these solutions. However, in the future we desire algorithms that can do so while using fewer interactions with the environment.
CONCLUSION
In this work we have introduced scalable on-policy deep reinforcement learning algorithm, V-MPO, that is applicable to both discrete and continuous control domains. For the results presented in this work neither importance weighting nor entropy regularization was used; moreover, since the size of neural network parameter updates is limited by KL constraints, we were also able use the same learning rate for all experiments. This suggests that a scalable, performant RL algorithm may not require some of the tricks that have been developed over the past several years. Interestingly, both the original MPO algorithm for replay-basee off- seflpolicy learning (Abdolmaleki et al.) [ [2018a] (a and V-MPO for on-policy learning derived from similar principles, providing evidence for the benefits of this approach as an alternative to popular policy gradient-based methods.
ACKNOWLEDGMENTS
We thank Lorenzo Blanco, Trevor Cai, Greg Wayne, Chloc Hillier, and Vicky Langston for their assistance and support.
REFERENCES
Abbas Abdolmaleki, Bob Price, Nuno Lau, Luis P Reis, and Gerhard Neumann. Deriving and Improving CMA-ES with Information Geometric Trust Regions. Proceedings of the Genetic and Evolutionary Computation Conference, 2017.
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative Entropy Regularized Policy Iteration. arXiv preprint, 2018a. URL http::tttts/aaaatt
www.:org/pppoo wttgpp/f///l// 02256. pddf
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum Posteriori Policy Optimisation. Int. Conf. Learn. Represent. 2018b. URL
httts:://arxii hwtp://gggppppppeeeeee 0690000000000000.8.00000000000.0 pdf
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefraneq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind Lab. arXiv preprint arXiv:1612.03801, 2016.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47, 2012.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint, 2016. URL http: /ttp//xxxvvvvvv wrg..ooo...oooo. org/ /bs/ T606. 01540
10

---

Published as a conference paper at ICLR 2020
Peter Buchlovsky, David Budden, Dominik Grewe, Chris Jones, John Aslanides, Frederic Besse, Andy Brock, Aidan Clark, Sergio Gomez Colmenarejo, Aedan Pope, Fabio Viola, and Dan Belov. TF-Replicator: Distributed Machine Learning for Researchers. arXiv preprint, 2019. URL
http:///rxen .rg. abs///b///// 902 00465
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-XL Attentive Language Models Beyond Fixed-Length Context. arXiv preprint, 1901. C2860
2019. htt.lttppp/ccm http///awvvv..rrvaaaaa /
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. arXiv preprint, 2016. URL httt:://ta/aa
org/abs/1604: 06778
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable Distributed Deep-RL. RL with Importance Weighted Actor-Learner Architectures. arXiv preprint, 2018. URL http:////a......or/a//a///////// httpo//gg///////..o.ao 01561
Google. Cloud TPU, 2018. URL URLLittpps///looo.gggle...o//
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critics Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. arXiv preprint, 2018. URL http:::://aaxiw.orrg/aaaas//////
01290
Nikolaus Hansen, Andreas Ostermeier, and Andreas Ostermeier. Convergence Properties of Evolution Strategies with the Derandomized Covariance Matrix Adaptation: CMA-ES. 1997. URL http:
// //wwww www..mmapppmmmmmmmmmm..... po ytechn que www.ff//nnoaaaaae aus www.hhaannsnsssS. pddf
Nicolas Heess, Greg Wayne, David Silver, Timothy P. Lillicrap, Yuval Tassa, and Tom Erez. Learning continuous control policies by stochastic value gradients. arXiv preprint, 2015. URL http:
w/axxiv.org//bs////
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task Deep Reinforcement Learning with PopArt. arXiv preprint, 2018. URL
https:// /atp::///rxv...
4444.... pdf
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks. arXiv preprint. 2017a. URL
http://arxiv. wbtp://s///////..l.////// 09846
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement Learning with Unsupervised Auxiliary Tasks. Int. Conf. Learn. Represent., 2017b. httppp//wwpppgee..eeeeeeeeeeeeap
Max Jaderberg, Wojeiech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364:859 865, 2019. URL https wttpc//nnnnnt///4n.3333
//science www.nnceaggg.nce..
Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent Experience Replay in Distributed Reinforcement Learning. Int. Conf. Learn. Represent. 2019.
URL https: https:://pppeeeeeeeeeteee wttp///wwwwweeeeeeeedddd/dee
JA
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. Int. Conf.
Learn. Represent., 2015. https://a///aa http//awwwwvorras////a///// 2980
Sergey Levine. Reinforcement Learning and Control as InobabilisticcInfeennnee Inference: Tutorial and Review. arXiv preprint, 2018. URL htt.:tttpp//rrxi.orr///aa//

---

Published as a conference paper at ICLR 2020
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous control with deep reinforcement learning. arXiv preprint, 2015. URL http::/aaxiv...oo/m/ http:///aww....ooaa/////////////
02971
Shie Mannor, Reuven Y Rubinstein, and Yohai Gat. The cross entropy method for fast policy search. Proceedings of the 20th International Conference on Machine Learning, 2003. URL
https: htttps://www... aaai wwwpaaai.rrg/aaaaarraaeeeee ITCPs//CCMMM2O0220 1000003//CM003000 pd.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabillistic motor primitives for humanoid con- trol. arXiv preprint, 2018. URL
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hicrarchical Visuomotor Control of Humanoids. Int. Conf. Learn. Represent, 2019. URL htt.pp::///peneeveew.... http://wwww.e.eeeeaddaiaeee
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-Level Control through Deep Reinforcement Learning. Nature, 518:529 1333,, 2015. UtLhttt: doi oro. 10388 0ttp://10..00//mmtmee1333333 naturee423s
Volodymyr Mnih, Adria Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. arXiv:1602.01783, 2016. URL /Rt.ttt:://aaw....cc////aaaa /tt1///.aaa/12001183333
Radford M. Neal and Geoffrey E. Hinton. A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants. M.I. Jordan (ed.), Learn. Graph. Model. NATO ASI Ser vol. 89. Springer, Dordrecht, 1998.
Gerhard Neumann and Jan R. Peters. Fitted Q-iteration by Advantage Reigsttedeeggrsssion Advances in Neural Information Processing Systems, 2009, URL http//////ttpp/pppeest.toooooooa
OpenAl. OpenAl Five, 2018a. URL http:ppp.//eeeaa..cccom///////gp
OpenAL. Learning Dexterity, 2018b. dexterity/
URL
Jan Peters, M Katharina, and Yasemin Altün. Relative Entropy Policy Search. Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, pp. 1607-1612, 2008.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language Models are Unsupervised Multitask Learners 2019. URL https
//ddmccfpksyww..lluuu www.yywow.woodoooo..too.o htt.//ottteer-aaggaga---oooo haalss/slagaaggeee wttp::/wweeeeetttttttaaaaeeeeeae
models are
Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M.
Czarecki, Joel Z. Leibo, Heinrich Küittler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting Deep Reinforcement Learning. arXiv preprint, 2018. URL htttttt/paaxiiii
org/abs/ /103. 03835
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-Policy Actor-Critic with Shared Experience Replay. arXiv preprint, 2019. /RL/htts::/axxi...oo
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust Region Policy Optimization. arXiv preprint, 2015. http:/tttp///arrrrr..rro///////0
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proxiallll timization algorithms. arXiv preprint, 2017. URL http:://aarti...rrr//a////111100
12

---

Published as a conference paper at ICLR 2020
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, 1lya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484 489, 2016. URLhttp: /www nature. wwt.coi wwt.doiiinndeee 10, 1ww.110/n/ttueee99...
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self- self-play. Science, 362:1140-1144 2018. URL httpt::/ss/stt.. /sciince scceencmag..o org / httpcc/aaaa.org/ootteeee/e/e/e/e 03tp///////419114000
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
Richard Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. A. Solla, T. K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems 12, pp. 1057- 1063. MIT Press, 2000. URL http::/papep:: insappers.iips.... wttpppppppppp.....ec 111111111pppppyppp policy gracient - methode for reinfcrcement learning with function- agproximation.pal
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. DeepMind Control Suite. arXiv preprint, 2018. htL:http://att wwtp/iiiworg/.s/./o.//a
1801. 00690
Hado van Hasselt, Arthur Guez, Matteo Hessel, and David Silver. Learning functions across many orders of magnitudes. arXiv preprint, 2016. URL
httprr///aaab111..7777.0. 0771
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft using multi-agent reinforcement learning. Nature, 2019. URL http:://oi.
Quan Vuong, Keith Ross, and Yiming Zhang. Supervised Policy Update for Deep Reinforcement Learning. arXiv preprint, 2019. URL htttt:://rxxv..rrg 1105/ 11706
Ronald J. Williams.
Simple statistical gradient-following methods for connectionist reinforce- ment learning. Mach. Learn., 8:229-256, 1992. URL http::///dx.ddi.ogg/1000.00
BFC0992696
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M. Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. arXiv preprint, 2018. UtL/ttpp:/ //ap:i/////ii.///aa/aaa/eaa/aaaa wtwxii.org/abb//80...o
A DERIVATION OF THE V-MPO TEMPERATURE LOSS
In this section we derive the E-step temperature loss Eq. 22 To this end, we explicitly commit to the more specific improvement criterion in Eq. 5 by plugging into the original objective in Eq. We seck (s, a) that minimizes
I (1b(8, a)) = DKL. (v-(,, a))lpo)lla... (8, alI =0=]]] =1)
(15)
() Ev(s, log (s,
α Σ a(s A (s, +
+ + (3,
(16)
Pu (8, 8.a
5,a
8.4
13

---

Published as a conference paper at ICLR 2020
where À 17 loggoee Pesu (I 1) after multiplying through by 1/, which up to this point in the deriva- tion is given. We wish to automatically tune 7) so as to enforce a bound 6, on the KL term DeL (1+s, (e., a)) multiplying in Eq. 16 in which case the temperature optimization can also be viewed nonparametric trust region for the variational distribution with respect the old distribution. We therefore consider the constrained optimization problem
vb(s,a) = arg max ((e0))
Σ
((s,a)
(17)
Σ (s, a) a log 118 a))
< En and Σ (s),a)=
(18)
Peas (8, a)
We now use Lagrangian relaxation transform the constrained optimization problem into one that maximizes the unconstrained objective
Evlls,a - log (((, pane (s, )
(()(s.)).n.A)-
Σ (s. ((s..))AT... (s, 8,a
6
+1 1 1 (s,
(19)
with 2 0. (Note we are re-using the variables 7) and À for the new optimization problem.) Differentiating I with respect (8s, a) and setting equal to zero, we obtain
(s,
mm 17
(ss a) ao (8, a) exp
exp
(20)
Normalizing over s, a (using the freedom given by X) then gives
A Thae (s.a)
Poaa (8, a) exp
(5s a)
(21)
Esa Pena (s, a) exp
Tean (s.a)
which reproduces the general solution Eq. for our specific choice of policy improvement in Eq. However, the value of 1) can now be found by optimizing the corresponding dual function. Plugging Eq. into the unconstrained objective in Eq. 19 gives rise the d7-depedentt term
Tou (s,
(n) = 176n, + 1 log
Σ pa (8, a) exp
(22)
7)
Replacing the expectation with samples from po.. (s, a) in the batch of trajectories D leads to the loss in Eq.
B M-STEP KL CONSTRAINT
Here we give somewhat more formal motivation for the prior log) Consider a normal prior N( p. Σ) with mean μ and covariance E. We choose Σ- aF(aaa)) where is scal- ing parameter and F(Bo))) is the Fisher information for Ter (a)s) evaluated at Ø = fold Then log p(0) x x (0 FFB.uaae ( {term independent of Ø). where the first term is precisely the second-order approximation to the KI. divergence DKL. (foaal)))))) We now follow TRPO (Schulman al. 2015) in heuristically approximating this the state-averaged expression, Es-p(s) (t) [Di.. T (a () Re (a(aa))) [.]s))] We note that the KL divergence in either direction has the same second-order expansion, so choice of KL is an empirical one (Abdolmaleki al. 2018a).
C DECOUPLED KL CONSTRAINTS FOR CONTINUOUS CONTROL
As in Abdolmmalei et al. 2018b), for continuous action spaces parametrized by Gaussian distributions we use decoupled KL constraints for the M-step. This uses the fact that the KL divergence between two d-dimensional multivariate normal distributions with means μn, f12 and covariances Σ,, Eg can be written
DKL E,((,,,))) En) E2)) = (12 - (1-m)) (22 m4) TT d log
(23)
14

---

Published as a conference paper at ICLR 2020
where | is the matrix determinant. Since the first distribution and hence Σn in the KL divergence of Eq. depends on the old target network parameters, we see that we can separate the overall KL divergence into mean component and covariance component:
Dii. (T [TTA)
Lu
pl ) - (wo - paaa))
(24)
[>g
D (Tew
Tr<='' d a++oo log |
(25)
With the replacement DKt. (TR (FFA) + DeL (TT for C = 14, and corresponding Ogs ax, we obtain the total loss
(f, 17, ap. ap; = (x + Ln(()) + La, az) Lay (0, ax). where C, (0) and (T) are the same as before. Note, however, that unlike in Abdolmaleki al. (018a) we do not decouple the policy loss.
(26)
We generally set to be much smaller than En (see Table 7 Intuitively, this allows the policy to learn quickly in action space while preventing premature collapse of the policy, and, conversely, increasing "exploration" without moving in action space.
D RELATION TO SUPERVISED POLICY UPDATE
Like V-MPO, Supervised Policy Update (SPU) (Vuong et al. 2019) adopts the strategy of first solving nonparametric constrained optimization problem exactly, then fitting a neural network to resulting solution via supervised loss function. There is, however, an important difference from V-MPO, which we describe here.
In SPU, the KL. loss, which is the sole loss in SPU, leads to a parametric optimization problem that equivalent the nonparametric optimization problem posed initially. To see this, observe that the SPU loss seeks parameters (note the direction the KL divergence)
0" = arg min Σ
dh (s) (To(a)s)) (als))
(27)
To(als) Noo (als) exp (A (8, a)
arg min
Σ
To (s)
Σ To(als) log
(28)
To (a/s)
arg min
Σ
To (s)
Σ
To (als) log
TT(a|s)A** (s, a) + constant terms) }.
no, (a|s)
(29)
Multiplying by À since it be treated constant up to this point, we then see that this corresponds exactly to the (Lagrangian form) of the problem
0 arg max
To (a (s)
(30)
s.t.
To (s) DKL (T( (als) <e,
(31)
which is the original nonparametric problem posed in Vuong al. 2019)
E
FOR OFF-POLICY CORRECTIONS
The network that generates the data may lag behind the target network in common distributed, asynchronous implementations (Espeholt et al.) 2018). We can compensate for this by multiplying the exponentiated advantages by importance weights p(s,a):
TT (s.a)
p(s, a a)poo (8, a) exp
(s) Eaa p(s, a)p (s, a) exp
"1
(32)
500 (s.a)
T0 (s,
(7) = n7,, + 1lg
Σ
(33)
17
8.8
15

---

Published as a conference paper at ICLR 2020
where 0D are the parameters of the behavior policy that generated D and which may be different from Burges. The clipped importance weights p(s, a) are given by
p a) = min (1. Tim (a|s) Tep (a|s)
(34)
As was the case with V-trace for the value function, we did not find it necessary to use importance weighting and all experiments presented in this work did not use them for the sake of simplicity.
F NETWORK ARCHITECTURE AND HYPERPARAMETERS
For DMLab the visual observations were x2x96 RGB images, while for Atari the observations were 4 stacked frames of x4 grayscale images. The ResNet used to process visual observations is similar to the 3-section ResNet used in Hessel al. (2018), except the number of channels was multiplied by 4 in each section, so that the number of channels were (64, 128, 128) (Schmitt et al. 2019). For individual DMLab levels we used the same number of channels as Hessel et al. (2018), 8), i.e. (16, 32, 32). Each section consisted of a convolution and 3 x 3 max-pooling operation (stride 2), followed by residual blocks of size 2, i.e., a convolution followed by a ReLU nonlinearity, repeated twice, and skip connection from the imput residual block input to the output. The entire stack was passed through one more ReLU nonlinearity. All convolutions had a kernel size of 3 and a stride of 1. For the humanoid control tasks from vision, the number of channels in each section were (16, 32, 32).
Since some of the levels in DMLab require simple language processing, for DMLab the agents contained an additional 256-unit LSTM receiving an embedding of hashed words as input. The output of the language LSTM was then concatenated with the output of the visual processing pathway as well as the previous reward and action, then fed to the main LSTM.
For multi-task DMLab we used a 3-layer LSTM, each with 256 units, and an unroll length of 95 with batch size 128. For the single-task setting used 2-layer LSTM. For multi-task Atari and the 56-dimensional humanoid -gps control task a single 256-unit LSTM was used, while for the 22-dimensional humanoid-run task the consisted only of 2-layer MLP with 512 and 256 units (no LSTM). For single-task Atari a Transformer-Xi was used in place of the LSTM. Note that we followed Radford al. (2019) in placing the layer normalization on only the inputs to each sub-block. For Atari the unroll length was 63 with a batch size of 128. For both humanoid control tasks the batch size was 64, but the unroll length was 40 for the 22-dimensional humanoid and 63 for the 56-dimensional humanoid.
In all cases the policy logits (for discrete actions) and Gaussian distribution parameters (for continuous actions) consisted of 256-unit MLP followed by a linear readout, and similarly for the value function. For discrete actions we initialized the linear policy layer with zero weights and biases to ensure a uniform policy at the start of training.
The initial values for the Lagrange multipliers in the V-MPO loss are given in Table
Implementation note. We implemented V-MPO in an actor-learner framework (Espeholt et al., 2018) that utilizes TF-Replicator (Buchlovsky et al. 2019) for distributed training on TPU 8-core and 16-core configurations (Google][018) One practical consequence of this is that a full batch of data D was in fact split into or minibatches, one per core/replica, and the overall result obtained by averaging the computations performed for each minibatch. More specifically, the determination of the highest advantages and the normalization of the nonparametric distribution, Eq. [2] is performed within minibatches. While it is possible to perform the full-batch computation by utilizing cross- replica communication, we found this to be unnecessary.
DMLab action set. Ignoring the "jump" and "crouch" actions which we do not use, action in the native DMLah .a action space consists of 5 integers whose meaning and allowed values are given in Table 2 Following previous work on DMLab (Hessel al. 2018), used the reduced action set given in Table 3 with an action repeat of 4.
16

---

Published as a conference paper at ICLR 2020
HYPERPARAMETER
VALUE
DMLab
Atari
Continuous control
Initial 7)
1.0
1.0
1.0
Initial a
5.0
5.0
Initial Cy
1.0
Initial ax:
Table 1: Values for common V-MPO parameters.
ACTION NAME
RANGE
LOOK LEFT RIGHT PIXELS PER FRAME
[-512, 512]
LOOK DOWN.UP PIXELS PER FRAME
[-512, 512]
STRAFE LEFT RIGHT
[-1, 1]
MOVE_BACE B.TE.BACK.FBB FORWARD FIRE
[-1, 1]
[].
Table 2: Native action space for DMLab. See htghttss///gthhteee
master docs users/ invers/actio ons md for more details.
ACTION
NATIVE DMLAB ACTION
Forward (FW)
0,
C,
0]
Backward (BW)
0,
0,
C, -1, 0]
Strafe left
O, 1, 0, 01 0, 0, 1, 0, 01
0,
Strafe right
Small look left (LL)
[-10, O, C, 0, 01
Small look right (LR)
10,,
O, C, 6, 01
Large look left (LL)
[-60,
0,
0, 0]
Large look right (LR)
60,
0,
Look down
0, 10,
C, 6, 01
Look up
0, -10,
01
FW + small LL
[-10,
0, C,
1, 01
FW small LR
10,,
0, C, 0]
FW large LL
[-60,
+ large LR
60,
0, C, 1, 01
Fire
0,
11
Table 3: Reduced action set for DMLab from Hessel al. 2018
17

---

Published as a conference paper at ICLR 2020
LEVEL NAME
EPISODE REWARD
HUMAN--NRMMALIZED
IMPALA
V-MPO
IMPALA
V-MPO
alien
1163.00 + 148.43
2332.00 1 290.16
13.55 = 2.15
30.50 = 4.21
amidar
192.50 1 9.16
423.60 1 20.53
10.89 0.53
24.38 = 1.20
assault
4215.30 + 294.51
1225.90 60.64
768.46 56.68
193.13 11.67
asterix
4180.00 303.91
9955.00 + 2043.48
47.87 = 3.66
117.50 24.64
asteroids
3473.00 = 388.30
2982.00 164.35
4.85 = 0.35
atlantis
997530.00 = =3552.89
940310.00 = 6085.96
6086,50 21.96
5732.81 37.62
bank_heist
1329.00 + 2.21
1563.00 + 15.81
177.94 + 0.30
209.61 + 2.14
battle
43900.00 = 4738.04
61400.00 + 5958.52
119.27 13.60
169.52 17.11
rider
4598.00 = 618.09
3868.20 + 666.55
25.56 = 3.73
21.16 = 4.02
berzerk
1018.00 = 72.63
1424.00 + 150.93
35.68 = 2.90
51.87 6.02
bowling
63.60 = 0.84
27.60 + 0.62
29,43 = 0.61
3.27 0.45
boxing
93.10 = 0.94
100.00 0.00
775.00 7.86
832.50 0.00
breakout
484.30 = 57.24
400.70 18.82
1675.69 = 198.77
1385.42 65.36
centipede
6037.90 = 994.99
3015.00 404.97
39.76 = 10.02
9.31 +
chopper eoopper.command
4250.00 = 417.91
4340.00 + 714.45
52.29 = 6,35
53.66 + 10.86
crazy climber
100440.00 1 9922155
116760.00 11: 5312.12
357.94 37.61
423.09 21.21
defender
41585.00 4194.42
98395.00 1. 17552.17
244.78 26.52
604.01 1 110.99
-attack
77880.00 + 8798.44
20243.00 5434.41
4273.35 = 483.72
1104.56 298.77
doublc_dum
-0.80 0.31
12.60 + 1.94
809.09 14.08
1418.18 88.19
enduro
1187.90 = 76.10
1453,80 + 104.37
138.05 + 8.84
168.95 = 12.13
fishing-derby _debby
21.60 3.46
33.80 + 2.10
213.77 6.54
236.79 3.96
freeway
32.10 = 0.17
33.20 + 0.28
108.45 0.58
112.16 0.93
frosthite
250.00 + 0.00
260.00 0.00
4.33
4.56 +
gopher
11720.00 + 1687.71
7576.00 + 973.13
531.92 = 78.32
339.62 45.16
gravitar
1095.00 = 232.75
3125.00 + 191.87
29.01 = 7.32
92.88 6,04
hero
13159.50 68.90
29196.50 =: 752.06
40.71 0.23
94.53 2.52
ice_hockey
4.80 1.31
10.60 2.00
132.23 = 10.83
180.17 16.50
jamesbond
1015.00 = 91.39
+ 595.92
360.12 33.38
1379.11 217.65
kangaroo
1780.00 + 18.97
12790.00 = 629.52
57.93 = 0.64
427.02 21.10
krull
9738.00 = 360.95
7359.00 + 1064.84
762.53 33.81
539.67 99.75
kung fu_master
44340.00 2898.70
38620.00 2346.48
196.11 = 12.90
170.66 = 10.44
montezuma reveeng
0.00
0.00 0.00
1
0.00 1
mms .o-ppamman
1953.00 + 227.12
2856.00 + 324.54
24.77 = 3.42
38.36 = 4.88
name this game
5708.00 = 354.92
9295.00 + 679,83
59.33 = 6.17
121.64 11.81
phoenix
37030.00 6415.95
19560.00 = 1843.44
559.60 98.99
290.05 28.44
pitfall
-4.90 = 2.34
-2.80 1.40
3.35 0.04
3.39 = 0.02
pong
20.80 + 0.19
0.00
117.56 0.54
118.13 0.00
privatc_eye
100.00 + 0.00
100.00 + 0.00
0.11 0.00
0.11 0.00
qbert
5512.50 + 741.08
15297.50 1244.47
00.44=5.5
113.86 9.36
riverraid
8237.00 97.09
11160.00 + 733.06
43.72 0.62
62.24 4.65
road runner
28440.00 = 1215.99
51060.00 1 1560.72
362.91 15.52
651.67 = 19.92
robotank
29.60 + 2.15
46.80 1 3.42
282,47 = 22.22
459.79 = 35.29
seaquest
1888.00 1 63.26
9953.00 1 993.02
4.33 0.15
23.54 - 2.32
skiing
16624..00 = 592.28
15538. + 1573.39
- 6.69
13.01 = 12.33
solaris
1794.00 279.04
2194.00 417.91
5.03
8.64 = 3.77
space_invaders
793.50 = 90.61
1771.50 201.95
42,45 5.96 5,96
106.76 13.28
star bur_ggnner
44860.00 + 5157.74
60120.00 + 1953.60
461.05 = 53.80
620.24 20.38
surround
2.50 + 1.04
4.00 0.62
75.76 6.31
84.85 3.74
tennis
-0.10 + 0.09
23.10 0.26 0.26
152.90 0.61
302.58 1.69
time pilot
10890.00 787.46
22330.00 2443.11
440.77 = 47.40
1129.42 = 147.07
tutankham
218.50 = 13.53
254.60 9.99
132.59 8.66
155.70 6.40
up_n_down
175083.00 = 16341.05
82913.00 + 12142.08
1564.09 146.43
738.18 = 108.80
venture
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 - 0.00
video pinball
59898.40 23875.14
198845.20 98768.54
339.02 + 135.13
1125.46 559.03
wizard wor
6960.00 1730.97
7890.00 + 1595.77
152.55 - 41.28
174.73 = 38.06
yars revenge
12825.70 70 1 2065.90
41271.70 1 4726.72
18.90 - 4.01
74.16 9.18
zaxxon
11520.00 1 646.81
18820.00 1 754.69
125.67 1 7.08
205.53 1. 8.26
Median
117.56
155.70
Table 4: Multi-task Atari-57 scores by level after 11.4B total (200M per level) environment frames. All entries show mean + standard deviation. Data for IMPALA the authors of Hessel et al. (0188). Human-normallize scores are calculated as (E- -R) x 100, where E is the episode reward, R the episode reward obtained by random agent, and H is the episode reward obtained by a human.
obtained from
18

---

Published as a conference paper at ICLR 2020
SETTING
SINGLE-TASK MULKI-TASK
Agent discount
0.99
Image height
72
Image width
96
Number of action repeats
4
Number of LSTM layers
2
Pixel-control cost
x 10 10-3
T Turget
10
0.1
0.5
log-uniform)
[0.001, 0.01)
[0.01, 0.1)
Table Settings for DMLab.
SETTING
SINGLE-TASK MULTI-TASK
Environment discount end of life
Agent discount
0.997
0.99
Clipped reward range
no clipping
[-1, 1]
Max episode length
30 mins 1000.00000000000000000000000 frames)
Image height
84
Image width
84
Grayscale
Truc
Number of stacked frames
4
Number of action repeats
4
TrXL: Key/Value size
32
TrXL: Number heads
TrXL: Number of layers
TrXL: MLP size
512
T Targe
1000
100
fn
x 10
(log-uniform)
[0.005, 0.01) 10.001,0.01)
Table 6: Settings for Atari. TrXL: Transformer-XL.
SETTING
HUMANOID-STATE HUMANSID-PIXELS
OPENAI GYM
Agent discount
0.99
Unroll length
63
63
39
Image height
64
Image width
Target update period
100
Ci
0.1
0.01
Eap (log-uniform)
[0.01, 1.0)
10.05. 0.5
[0.005, 0.01]
Fag (log-uniform)
5 x 10 5 x 10 9)
[10
x 10
[5 x 10 x 10
Table 7: Settings for continuous control. For the humanoid gaps task from pixels the physics time step was 5 ms and the control time step 30 ms.
19

---

Published as a conference paper at ICLR 2020
180
Median hwman-nomnanooo nornnolmmadddd
140
120
60
40
V-MPO-PBT (24)
IMPALA PET (24)
Environment (4xagent) steps (B)
Figure 6: Multi-task Atari-57 with population population-based based training (PBT) (Jaderberg et al. 2017a) All settings of the PBT experiment were the same as without except the learning rates were also sampled log- unifurifyyyly from [8 x 10-5, 3 x 10=4) and E, from [0.05, 0.5). Along with 6, sampled log- uniformly from [0.001, 0.01) as in the original experiment, hyperparameters were evolved copy and mutation operators roughly once every 4 x 10 environment frames.
Seaquest
Episode roward
300000
030003
Entropy
2.3
2.00
400000
1.0
E-step KL
0.038
M-step
0.04
0002
Learner step
Figure 7: KL constraints during optimization for the Seaquest example in Fig. Values are subsampled but smoothed to show the variability.
20

---

Published as a conference paper at ICLR 2020
1000000
seaquest
Oct
Episode reward
600000
400000
V-MPO-TIXL (10)
#202@-108
0.0
2.5
3.0
7.5
10.0
12.5
15.0
17.5
20.0
Environment (4xagent) steps (B)
Figure Same as Fig. advantages.
(Atari Seaquest), but trained with uniform weights on the top 50% of
100-capped human-normalized
100
60
40
V-MPO, all-K 124)
IMPALA (3x8 PBT)
R202+8(30 steps)
Environment (4xagent) steps (B)
Figure 9: Same as Fig. 2 (multi-task DMLab-30), but trained without top-k, i.c., all advantages are used in the E-step. Note the small dip in the middle is due pause in the experiment and resetting of the human-normalized scores.
Figure 10: Example frame from the humanoid gaps task, with the agent's 64x64 first-person view on the right. The proprioceptive information provided to the agent in addition to the primary pixel observation consisted of joint angles and velocities, moo-tto-end-fffector vectors, root-frame velocity, rotational velocity, root-frame acceleration, and the 3D orientation relative to the z-axis.
21

---

Published as a conference paper at ICLR 2020
18000
18000
10000
Episode
8000
6000
4000
VIMPO (10)
MPOB25M
SAC@10M
Environment steps (B)
Figure 11: 17-dimensional Humanoid- V1 task in OpenAl Gym.
22


Reviews:
[{"review_id": "r1guwy_2FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents the V-MPO algorithm for on-policy reinforcement learning that can handle both continuous/discrete control, single/multi-task learning and use both low dimensional states and pixels. \n\nV-MPO adapts MPO, a recent off-policy deep reinforcement learning algorithm, to the on-policy setting with the following changes: (1) In policy evaluation step, instead of using state-action value Q estimated from off-policy replay data, the authors use on-policy data to estimate the state value V; (2) In E-step of policy learning, construct the target distribution q(a|s) with estimated advantages; and (3) in M-step, rewrite the KL divergence constraints with Lagrangian relaxation and alternate between optimising policy and the multiplier alpha. \n\nThe experiments are two-fold: (1) For discrete control, a multi-task problem setting is considered (DMLab-30 and Atari-57). V-MPO shows improved performance compared to IMPALA and R2D2, in terms of asymptotic scores. The result also suggest good stability for different hyper-parameters. (2) In continuous problems, the experiments follow the single-task problem setting, including Humanoids tasks and OpenAI Gym tasks. V-MPO achieves higher asymptotic returns compared with standard Deep RL algorithms (MPO, SAC, PPO). However, it has a lower sample-efficiency, especially compared to off-policy algorithms (MPO and SAC). \n\nPros:\n+ This paper demonstrates successful adaption of MPO to the on-policy problem setting and achieves better asymptotic score comparing to baseline methods. \n++ Results are achieved in a relatively hyper-parameter insensitive manner. And some commonly used tricks like entropy regularisation are not necessary.\n+ Along with the previous results of MPO, this demonstrates an alternative framework of policy gradient-based RL: first construct a nonparametric target behavioural distribution and then move the parametric policy towards this distribution. \n\nCons and Questions:\n1. The comparison between V-MPO with other single-task deep RL methods is non-standard because: (1) V-MPO is trained with far more samples, and (2) only asymptotic score is reported for baselines. It would be more informative to add the learning curve of baselines methods to show sample-efficiency and convergence properties. One may wonder if the baselines would provide better performance than reported if provided with a comparable number of steps in an appropriate way (E.g., seed search, hyper parameter sweeps). \n2. The proposed adaption of MPO from off-policy to on-policy applies to both discrete and continuous problem settings. However, the designed experiments include only the discrete multi-task setting and continuous single-task setting. How does V-MPO compare to other methods in single-task discrete problems? This may enable comparison to a wider set of prior work.\n3. Only top 50% advantages samples are used for generating target distribution in E-step, and all samples are used for policy updates in M-step. Does this mismatch between E-step and M-step sample pool make a difference in optimisation?\n4. Some small tricks we used like PopArt, Top-K advantage. It’s not clear if these are things that (were/could-also-be used) in competitor methods, and how important these are to achieve the good results shown here. At least an ablation study on their impact for V-MPO would be nice. \n\nMinor: \n- The mixture use of subscripts “old” and “target” is rather confusing, for example, in equation (5) and (15).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyeDKDGJcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an online variant of MPO, V-MPO. Compared to MPO, the main difference seems to be in the E-step. Instead of optimizing the non-parametric distribution towards a parameterized Q-function, V-MPO learns the V-function and updates the non-parametric distribution towards the advantages, which can be estimated on the samples of the last roll-outs based on the empirical returns and the learned V-function. Of course, updating towards (exponentiated and normalized) Q- or A-function does not make a difference. There are also some minor changes (which might still be crucial) such as an entropy constraint in the M-step and using only the top-k advantages during the E-Step (an option that was discussed in Abdolmaleki, et al. 2018a).\nV-MPO is evaluated on DLM, ALE and two humanoid tasks from the DeepMind Control Suite. In most of these tasks V-MPO achieves returns that are to the best of my knowledge higher than any previously reported ones. However, the experiments also use a very large number of system interactions (in the order of billions).\n\nContribution / Significance:\nI think that there will be relatively high interest in the paper due to the reported performances.\nThe technical contribution seems a bit incremental compared to MPO. Also, by learning a value function V-MPO gets closer to REPS. The submission lists the use of top-k samples and the M-step KL bound as the main differences to REPS. However, the former is not evaluated in the submission and the latter, albeit crucial, seems to be a relatively small modification. I do think that there are more differences to REPS, most important probably in the way of learning the value function and the corresponding differences in the derivations. However, I think that the differences abd similarities to MPO and REPS need to be discussed more thoroughly.\n\nSoundness:\nThe derivation of V-MPO is relatively sound. The optimization of the KL constraints seems very approximate, although it seems to work well in practice.\n\nClarity:\nI do not like the way the algorithm is presented. The submission specifies the complete loss function already at the beginning of the \"Method\"-Section and derives/motivates the individual terms in hindsight. The spaghetti-code like structure unnecessarily forces the reader to jump between pages or keeps the reader in the dark. I also do not like the \"stop-gradient\" notation which in my opinion puts the focus on low-level implementation details at the cost of not properly explaining what the optimization actually does. I think that paper is well-written in general, but the structure needs to be improved.\n\nExperiments:\nThe evaluation clearly focuses on achieving the best performance, and does a good job in that regard. \nHowever, a good evaluation should also help in understanding the mechanics of V-MPO. How does k (in top-k) affect the performance? How well are the constraints met during optimization? How does V-MPO compare to related on-policy methods (e.g. TRPO/PPO) on slightly more computational constrainted settings (eg rllab/mujoco with < 1e7 steps)?\n\n\nQuestions:\n- Did you experiment with controlled entropy reduction akin to MORE, instead of using fixed \nepsilon eta?\n\n- Can you give a rough estimate of the computational time required to perform these experiments on a standard desktop pc? It really is difficult for me to even roughly estimate it.\n\n\nAssessment:\nCurrently, I am leaning to accept because I think that V-MPO is overall a nice work. However I do think that submission needs to be revised. I mainly think that the structure needs to be improved and that V-MPO needs to better related to closely related work. I listed some additional experiments that would significantly improve the submission in my opinion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg9i7Sl5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approximate policy iteration algorithm that is based on the previous method, called MPO, which formulates policy optimization (PO) as a probabilistic inference problem. The adaptation makes MPO become an on-policy method. One key modification made to MPO is to use advantage functions instead of Q-value function. \n\nOverall, the paper follows an interesting topic in policy optimization as inference. It follows MPO to formulate PO as an inference problem using a similar principle. It proposes some extensions to MPO. The experiments show a lot of promising results. I have some concerns as follows.\n\n- As V-MPO is developed based on MPO, however, the background on MPO is missing. This makes the reader hard to judge the novelty and difference of V-MPO against MPO. The discussion on the difference is very little. \n\n- V-MPO extends MPO to on-policy setting, so the policy evaluation is key for this modification. Besides this modification, the E-step and M-step's derivations and objectives look quite similar to those of MPO. Can the authors comment on this?\n\n- In addition, V-MPO is said to work for both discrete and continuous domains? It would be clearer if the authors discuss which parts in their algorithm enable this ability?\n\n- It would also be great if the algorithmic description is included. The overall algorithm contains multiple optimization steps and hyperparameters, e.g. number of updates, how Lagrangian multiplier adapted etc..\n\n- Continuous tasks: The comparisons with low sample-efficient approaches look unfair. Are there ablations that show performance comparisons of all when set with an equal level of samples?\n\n- Can the author elaborate on the comment \"These must be consistent between the maximum likelihood weights in Eq. 3 and the temperature loss in Eq. 4\".\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension of MPO for on-policy reinforcement learning. The proposed method achieved promising results in a relatively hyper-parameter insensitive manner.

One concern of the reviewers is the lack of comparison with previous works, such as original MPO, which has been partially addressed by the authors in rebuttal. In addition, Blind Review #3 has some concerns with the fairness of the experimental comparison, though other reviews accept the comparison using standardized benchmark.

Overall, the paper proposes a promising extension of MPO; thus, I recommend it for acceptance.

Decision: Accept (Poster)