Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
Authors: Xue Bin Peng
Aviral Kumar
Grace Zhang
Sergey Levine
Venue:  (2021)
Keywords: reinforcement learning
policy search
offline RL
control

Abstract: In this work, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

Under review as a conference paper at ICLR 2021
ADVANTAGE- -EAATTGGE-WEIHTED REGRESSION: SIMPLE AND SCALABLE OFF-POLICY REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weiighte regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and gen- eral, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the tort-r---ttt-ttttte RL algorithms. AWR is also able acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, demonstrate algorithm on challenging continuous control tasks with highly complex simulated characters. (Vided")
INTRODUCTION
Model-free reinforcement learning can be general and effective methodology for training agents to acquire sophisticated behaviors with minimal assumptions on the underlying task. However, RL algorithms can be substantially more complex to implement and tune than standard supervised learning methods. Arguably the simplest reinforcement learning methods are policy gradient algo- rithms (Sutton et al. 2000), which directly differentiate the expected return and perform gradient ascent. Unfortunately, these methods can be notoriously unstable and are typically on-policy, often requiring a substantial number of samples to learn effective behaviors. Our goal is to develop an RL algorithm that is simple, easy to implement, and can readily incorporate off- off-policy data.
In this work, we propose advantage-weighter regression (AWR), simple off-policy algorithm for model-free RL. Each iteration of the AWR algorithm simply consists of two supervised regression steps: one for training a value function baseline via regression onto cumulative rewards, and another for training the policy via weighted regression. The complete algorithm is shown in Algorithm [ AWR can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. Despite its simplicity, we find that AWR achieves competitive results when compared to commonly used on-policy and off- off-pooicy RL algorithms, and can effectively incorporate fully off- off-policy data, which has been a challenge for other RL algorithms. Our derivation presents an interpretation of AWR as a constrained policy optimization procedure, and provides a theoretical analysis of the use of off-policy data with experience replay.
We first revisit the original formulation of 2007), an on-policy RL method that utilizes supervised learning to perform policy updates, and then propose a number of new design decisions that significantly improve performance on suite of standard control benchmark tasks. We then provide a theoretical analysis of AWR, including the capability to incorporate off-policy data with experience replay. Although the design of AWR involves only few simple design decisions, we show experimentally that these additions provide for large improvement over previous methods for regression-basee policy search, such RWR,
regression (RWR) (Peters & Schaal
'Supplementary video: sites.googee.com/vieww/wwaw

---

Under review as a conference paper at ICLR 2021
while also being substantially simpler than more modern methods, such MPO (Abdolmaleki et al. 2018b). We show that AWR achieves competitive performance when compared to several wstalestablished fafte-of-the-amre on-policy and off off-policy algorithms.
PRELIMINARIES
In reinforcement learning, the objective is learn a policy that maximizes an agent's expected return. At each time step I, the agent observes the state of the environment 8,, and samples an action from a policy a, ~ x (ar]s)). The agent then applies that action, which results in a new state 8141 and scalar reward rt r(s,, az). The goal is to learn policy that maximizes the expected return (T),,
J(x) = (r
= Exnde (s).amr(a)s) [)]
(1)
where (T( represents the likelihood of a trajectory = {(80, ro) (8), a1, ri), under a policy n, and 7 [0, 1) is the discount factor. ds) Er= 7 7'p(s, s| represents the unnormalized discounted state distribution induced by the policy (Sttoon & Barto 1998). and p(s, s| the likelihood of the agent being in state s after following ग for timesteps.
Our proposed AWR algorithm builds on ideas from reward-weighted regression (RWR) (Peters & Schaal 2007). , policy search algorithm based on an expectation maximmiiiommniiiziiiinn framework. At each iteration, the E-step constructs an estimate of the optimal policy according to π (a(ss) oc T& eexp (R),/3), where Ty represents the policy at the kth iteration, Rx,a = E 7'r, is the return, and B >> O is temperature parameter. Then the M-step projects x- onto the space of parameterized policies by solving supervised regression problem:
log (Taas)) exp ( (1 (e.)
k = arg max Egndan (s) Eawrs
The RWR update can be interpreted as fitting policy Tk+1 samples from the current policy Tke where the likelihood of each action is weighted by the exponentiated return for that action.
3 ADVANTAGE- WEIGHTED REGRESSION
In this work, we present ad edvantage-weighted regression (AWR). a simple off-policy RL algorithm based on reward-weighted regression. We first provide an overview of the AWR algorithm, and then describe its theoretical motivation and analyze its properties. The AWR algorithm is summarized in Algorithm [ Each iteration & of AWR consists of the following simple steps. First, the current policy Ta (a|s) is used to sample a batch of trajectories {T;} that are then stored in the replay buffer D. which is structured as a first-in first-out (FIFO) queue (Mnih et al. 2013). Then, a value function vP((s) (s) is fitted to all trajectories in the replay buffer D, which can be done with simple Monte Carlo return estimates R.a = E 7'r, Finally, the same buffer is used to fit a new policy using regression, where each state-action pair in the buffer is weighted according to the exponentiated advantage exp(? AP a)), with the advantage given by A a) = R.Pa - VP(ss) and B is a hyperparameter. In the following subsections, we first motivate AWR as a constrained policy search problem, and then extend our analysis to incorporate experience replay.
3.1 DERIVATION
In this section, derive the AWR algorithm policy search problem. Our goal is to find a policy that maximizes the expected improvement ((T) = ((r) (J4) over a sampling policy (aals). We first derive AWR for the setting where the sampling policy is a single Markovian policy. Then, in the next section, we extend our result to data from multiple policies,
approximate optimization of constrained
in the case of experience replay. The expected improvement 17(xT) can be expressed in terms of the advantage AM a) = R (P(s) with respect to (Kakade &
Langford 2002 Schulman et al. 2015):
<) ) = E Rg (a) Ba
[R
V*())
where Rg. denotes the return obtained by performing action a in state s and following μ for the following timesteps, and (*(s) = S. ((als)R da corresponds to the value function of μ. This objective differs from the ones used in the derivations of related algorithms, such RWR and
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Advantage-Weighted Regression
I:
random policy
2: +
3: for iteration k 1,.... kmax do
4: add trajectories {T:} sampled via The to D
5: V arg miny Bs
_ V(s)
6: Tk+1 + arg max, Ea log ((ass)exp () (RPa- -V((s)))
7: end for
REPS (Peters & Schaal 2007 Peters al. 2010; Abdolmaleki al. 2018b), which maximize the expected return J(x) instead of the expected improvement. The expected improvement directly gives rise an objective that involves the advantage. We will see later that this yields policy update that differ in a subtle but important way from standard RWR. As we show in our experiments, this difference results in large empirical improvement.
The objective in Equation 3 can be difficult to optimize due to the dependency between d, (s) and æ, as well as the need collect samples from æ. Following SShulman et (2015). we can instead optimize an approximation (T)) of (7()) using the state distribution of μr:
f)(=) = [x) (s) R V*(s)]
(4)
Here, (TT)) matches 17(T) to first order (Kakade & Langford 2002). and provides good estimate of if and μ are close in terms of the Kl.-divergence (Schulman et al. 2015). Using this objective, we can formulate the following constrained policy search problem:
[ d1((s)
/ r(als) [R - V*)) da ds
arg max
(5)
x
s.t. [ (s) (T(()))])({s)) ds s.
The constraint in Equation 6 ensures that the new policy и is close to the data distribution of μu, and therefore the surrogate objective (TT)) remains reasonable approximation to 17(T). We refer the reader to Schulman et al. (2015) for a detailed derivation and an error bound.
We derive AWR approximate solution to this constrained optimization. This derivation follows a similar procedure as Peters et al. (2010), and begins by forming the Lagrangian of the optimization problem presented above,
((T,3) [ ds)(s)
T (als)) [Ria V*)) da ds + (e- dd((s) ( (T((())llu((ss)) ds
(7)
where 3 is Lagrange multiplier. Differentiating L(x, 3) with respect to T (als) and solving for the optimal policy "²" results the following expression for the optimal policy
(s)) (aass) exp (} (R.. - V*()))
(a (s)
(8)
with Z(s) being the partition function. A detailed derivation is available in Appendix A If i is represented by function approximator (c.g., neural network), new policy can be obtained by projecting x* onto the manifold of parameterized policies,
arg min
EgDD DKL (T (-Is) (-)ss)))
arg max x
E Exdd (s) (ana(()s) log (als)exxp
V*())) -V((s)
(R
(10)
While this derivation for AWR largely follows the derivations used prior work (Peters et al. 2010
Abdolmaleki et al., 2018b), our expected improvement objective introduces a bascline / (7)(s) to the policy update, which as we show in our experiments, a erucial component for an effective algorithm. A similar advantage-weightingg scheme has been previously used for fitted Q-iteration (Neumann Peters 2009), where the policy is given by <(als) z exp ((R V*)(s) /3) In this defimttin, the likelihood of an action does not depend on the sampling distribution, and therefore does not enforce trust region with respect to μ.
3

---

Under review as a conference paper at ICLR 2021
3.2 EXPERIENCE REPLAY AND OFF-POLICY LEARNING
A crucial design decision of AWR is the choice of sampling policy ju(als). Standard implementations of RWR are typically on-policy, where the sampling policy is selected to be the current policy u(als) = (a(s) at iteration k. This can be sample inefficient, as data collected at each iteration are discarded after a single update iteration. Importance sampling can be incorporated into RWR to reuse data from previous iterations, but at the cost of larger variance (Kober & Peters 2209). Instead, we can improve sample efficiency of AWR by incorporating experience replay and explicitly accounting for training data from a mixture of multiple past policies. As described in Algorithm [ at each iteration, AWR collects a batch of data using the latest policy Ta. and then stores this data in a replay buffer D, which also contains data collected from previous policies { ; ). The value function and policy are then updated using samples drawn from D. This replay strategy is analogous to modeling the sampling policy as mixture of policies from previous iterations μa (T) = Σ/= (T),(T), where (T) = p(T(T,) (T.)) represents the likelihood of a trajectory under policy n, from the ith iteration, and the weight w; specify the probability of selecting TT.
We now extend the derivation from the previous section to the off- off-policy setting with experience replay, and show that Algorithm|| iindeed optimizes the expected improvement over a sampling policy modeled by the replay buffer. Given a replay buffer consisting of trajectories from past policies, the joint state-action distribution of μ is given by plss, a) = Ei=1 w;dx, (alss, and similarly for the marginal state distribution d (s) = Σ i== w,dm, (s). The expected improvement can now be expressed with respect to the set of sampling policies in the replay buffer.
inddeed
(T)) ((T) Σ (m,) Eg-d, (s) Bava((as)))
Σ [u:A (s.a)
(11)
where (s, = R (s) is the advantage with respect to each sampling policy. In Ap- pendix B we show that the update procedure in Algorithm [ optimizee the following objective:
arg max
Σ w/ (s) (s) Baw
(AT a)))
(12)
1=1
s.t. E (x)d, [DI. (T(+ sc.
(13)
m(s,a) Σ, wide, (als)
where u(als)) = the replay buffer. This objective can be solved via the Lagrangian to yield the following update:
(s) == Σj (5dr,(5) represents the conditional action distribution defined by
( 1 3
Ej E, (s)V*s E3 w.ddd (s)
arg max
Σ wi Eynd,, (s) (ano,()))))))) log m(als)exp
R
(14)
where the expectations can be approximated by simply sampling from D following Line of A detailed derivation is available in Appendix B Note, the baseline in the exponent now consists of an average of the value functions of the different policies. This mean value function (s) can be fitted by simply sampling from the replay buffer following Line 5 of Algorithm |
Algorithm[I [
V arg min
Σ w/ U,,.B,,d,, (s))ss
[[|R]
V(s)
(15)
V
The optimal solution ((s)= = E, v.,d, (s)
is exactly the baseline in Equation 14
Σ, wydd, (x)
3.3 IMPLEMENTATION DETAILS
Finally, we discuss several important design decisions for practical implementation of AWR. Monte Carlo estimates can be used to approximate the expected return R.a. but this can result in high- variance estimate. Instead, we approximate R.P using TD(A) to obtain a lower-variance estimate (Sutton & Barto 1998). TD(A) is applied by bootstrapping with the value function v/2, (s) from the previous ierattionn To set the value of the Lagrange multiplier B, we found that a simple adaptive heuristic of setting 33 to the standard deviation of all advantage values a in the replay buffer works well in practice. This is akin to the advantage normalization technique commonly used in imple- mentations of algorithms such PPO (Dhariwal et al. 2017). Details are available in Appendix
4

---

Under review as a conference paper at ICLR 2021
(a) HalfChcctah-v2
(b) Walker2d-v2
(c) Humanoid: Spinkick
(d) Dog: Canter
Figure I: Snapshots of AWR policies trained on OpenAl Gym and motion imitation tasks. Our simple algorithm learns effective policies for diverse suite of control tasks.
The weights ag. exp (R2,, (P())) used update the policy can occasionally assume excessively large values, which causes gradients to explode. Therefore, we apply weight clipping "g.a min (we,a) Wmax) with a threshold Wmax to prevent exploding weights.
RELATED WORK
Existing RL methods can be broadly categorized into on-policy and off off-policy algorithms (Sutton Barto (998). On-policy algorithms generally update the policy using data collected from the same policy. A popular class of on-policy algorithms is policy gradient methods (Williams 1992
Sutton et al., 2000), which can effective for a diverse array of complex tasks (Heess al. 2017
Pathak et al., 2017; Peng et al. 2018 Rajeswaran et al., 2018). However, on-policy algorithms typically data inefficient. Off-policy algorithms improve sample efficiency by enabling training using data from other sources, such data from different agents or data from previous iterations of the algorithm. Importance sampling is a simple strategy for off-policy learning (Sutton & Barto 1998) Meuleau et al.) 2000 Hacchyy et al., 2009), but introduce optimization instabilities due to the large variance of the importance sampling estimator. Dynamic programming methods based on Q-function learning can also leverage off-polic data (Precup al. 2001 Mnih et al.) 2015) Lillicrap
et al. 2016 Gu et al.) 2016; Haarnoja et al. 2018b). But these methods can be notoriously unstable. and in practice, require a variety of stabilization techniques (Hasselt et al., [016] Waag al.) [016]] Munos et al.) 2016 Hessel et al.) 2017 Fujimoto et 2018 Fu et al. 2019). Furthermore, it can be difficult to apply these methods to fully off off-policy data, where an agent unable collect additional
environmental interactions (Fujimoto et al. 2019 Kumar et al. 2019).
Policy search can also be formulated under an expectation-maxiiizztioo maximization framework (Peters et al. 2010 Neumann, 2011] Abdolmaleki et al. 2018b), early example of which is rewire-weighted regression (RWR) (Peters& Schaal| 2007). RWR presents a simple on-policy RL algorithm that casts policy search supervised regression problem. A similar algorithm, relative entropy policy search (REPS) (Peters et al.] 2010), can also be derived from the dual formulation of a constrained policy search problem. RWR has number appealing properties: it has very simple update rule, and since each iteration corresponds to supervised learning, it can be more stable and easier to implement than many of the previously mentioned RL methods. Despite these advantages. RWR has not been shown to be an effective when combined with neural networks (Schulman et al. 2015 Duan et al.) 2016). In this work, propose number of modifications the formulation of RWR produce an effective off-policy deep RL algorithm, while still retaining much of the simplicity of RWR.
The optimization problem being solved in AWR is similar to REPS (Peters et al.) 2010), but REPS optimizes the expected return instead of the expected improvement. The weights in REPS also contains a Bellman error term that resembles advantages, but are computed using a linear value function derived from a feature matching constraint. Learning the REPS value function involves minimization of a dual function, which is a complex function of the Bellman error, while the value function in AWR can be learned with simple supervised regression. More recently, Abdolmaleki et al. (2018b) proposed MPO, a deep RL variant of REPS, which applies a partial EM algorithm for policy optimization. The method first fits Q-function of the current policy via bootstrapping, and then performs& a policy improvement step with respect to this Q-function. MPO uses off-policy data for training Q-function and employs Retrace(A) for off-policy correction (Munos al. 2016). In
5

---

Under review as a conference paper at ICLR 2021
TellChrldahez2
Hoppori22 +2
La Landland
Walcer20e22
1750C
-00-
5300
1/44
1500
2300
4300
5000
2000
2500
-000
0.5
25
3.0
2.0
3.0
0.2
0.5
3.0
2.0
0.2
0.4
0.6
0.3
2.0
9.0
0.5
TRPO
PPO
DDPG
TD3
SAC
LAWER
RWR
AWR (Ours)
Figure 2: Learning curves of the various algorithms when applied to OpenAl Gym tasks. Results
averaged across 10 random seeds. AWR is generally competitive with the best current methods.
Task
TRPO
PPO
DDPG
TD3
SAC
LAWER
RWR
AWR (Ours)
Ant-v2
2901 + 85
4884 + 1249
#251
1907 + 765
7600 2 1683
2240 + 407
183 = 176
5372 + 163
HallCocttab-42
3302 1 125
7017 185
10503 2 382
12321 1 1529
16223 = 964
4596 2331
2015 = 370
9192 +: 157
Hopper-v2
1880 337
2514=
835 282
2794 +
2767 638
1830 + 563
605
3498 167
Humanoid-v2
552 +
4688 + 1153
4382 + 423
4738 + 03
6296 1: 332
108 386
508 + 18
6150 + 274
Lnderw)
185
220
Walker2d-v2
2765- 108
5096 934
401 4-470
4779 803
6210 + 511
2502 + 388
408 + 64
5813 + 483
Table 1: Final returns for different algorithms on the OpenAI Gym tasks, with = corresponding to one standard deviation of the average return across 10 random seeds. In terms of final performance, AWR generally competitive with prior methods.
contrast, AWR is simpler, as can simply fit a value function to the observed returns in a replay buffer, and performs weighted supervised regression the actions to fit the policy. Oh et al. (2018) proposed self-imitation learning (SIL), which augments policy gradient algorithms with an auxiliary behaviour cloning loss reuse samples from past experiences. Unlike SIL., AWR is standalone algorithm, and does not need to be combined with an auxiliary RL algorithm. NNumann & Peters (2009) proposed LAWER, kernel-based fitted Q-iteration algorithm where the Bellman error is weighted by the normalized advantage of each state-action pair. This was then followed by a soft-policy improvement step. Similar to Neumann & Peters (2009). AWR also uses exponentiated advantages, but LAWER's definition of the policy is different from the one in AWR and does not enforce a trust region constraint. Furthermore, AWR does not perform fitted Q-iteration, and instead utilizes off- off-policy data in simple constrained policy search procedure. [Wang et al. (2018) applied a similar advantage-weij weightgg scheme for imitation learning, but the method was not demonstrated for the RL setting. In this work, we propose several design decisions that are vital for effective RL algorithm. We also provide theoretical analysis of AWR when combined with experience replay, and show that the algorithm optimizes the expected improvement with respect to a mixture of policies modeled by replay buffer.
EXPERIMENTS
Our experiments aim to comparatively evaluate the performance of AWR with commonly used on-policy and off-policy deep RI. algorithms. We evaluate our method on the OpenAl Gym bench- marks (Brockman et al., 2016), consisting of discrete and continuous control tasks. We also evaluate our method complex motion imitation tasks with high-dimensional simulated characters. We then demonstrate the effectiveness of AWR on fully off off-policy learning, by training on static datasets of demonstrations from demo policies. Behaviors learned by the policies are best seen in the supplemen tary videc Code for our implementation of AWR is available at ditessooogle..comiie/////a Detailed hyperparameter settings are provided in Appendix C
5.1 BENCHMARKS
We compare AWR to a number of state-of-ther art RL algorithms, including on-policy algorithms, such as TRPO (Schulman et al. 2013) and PPO (Schulman et al.) (017). off-policy algorithms, such as DDPG (Lillicrap et al. 2016). TD3 (Fujimoto et al. 2018). and SAC (Haarnoja al. 2018a), as well as RWR (Peters & Schaal 2007) and LAWER (Neumann & Peters 2009 TRPO, PPO, and DDPG use the implementations from OpenAl baselines (Dhariwal et al.
TD3 and SAC use the implementations from Fujimoto et al. (2018) and Haarnoja et al. (201 1a). RWR and LAWER are implemented following the deseriptions Peters & Schaal (2007) and Neumann & Peters 2009)).
2017
but neural networks are used instead of kernel-based approximators.
Whiie we attempted to compare to MPO Abdolmaleki et al. 2018b). we were unable to find or implement version of the algorithm that reproduces the results comparable those reported by Ahdolmalcki al. (0188).

---

Under review as a conference paper at ICLR 2021
Aldeliore 2
Ableikars Walbar2o22
Heplay Suffers MallCheriarr
Paplky Buller Wafeas
6030
0300
5300
13
4230
6300
0001
-000
3000
2002
2230
200-
-200
1.0
167
Figure 3: Left: Learning curves comparing AWR with various components removed. Each com- ponent contributes to performance improvements. Right: Learning curves comparing AWR with different capacity replay buffers. AWR remains stable with large buffers containing primarily off- off-policy data from past iterations.
Snapshots of the AWR policies are shown in Figure I Figure shows learning curves comparing the different algorithms, and Table I summarizes the average returns of the final policies across 10 training runs initialized with different random seeds. Due to the slow wall-clock times of TD3 and SAC, some training runs did not have sufficient time to collect many samples as other algorithms. Overall, AWR shows competitive performance with the state-of-the-ar deep RI. algorithms. It is competitive with on-policy methods, such TRPO and PPO, in both sample efficiency and asymptotic performance. While it is not yet as sample efficient as current state-of-the-aee off-policy methods, such SAC and TD3, is able to achieve comparable asymptotic performance on most tasks. RWR tends to perform poorly on these tasks, which suggests that, the particular modifications from AWR are critical. AWR also significantly outperforms LAWER across the various tasks. Though both methods use similar wwwaanntdd-wigghiiie scheme, our design decisions for AWR produce a simpler and more effective algorithm.
5.2 ABLATION EXPERIMENTS
To determine the effects of various design decisions, we evaluate the performance of AWR when key components have been removed. The experiments include: an on-policy version of AWR (On- (On-Policy), where updates use only data from the latest policy, a version of AWR without the baseline (ss) (No Baseline), and version that uses Monte Carlo return estimates instead of TD(A) (No TD()). The effects of these components are illustrated in Figure [3] Overall, these design decisions appear to be vital for an effective algorithm, with the most crucial components being the use of experience replay and baseline. Updates using only on-policy data can lead to instabilities and noticeable degradation in performance, which may be due to overfitting on smaller dataset. Removing the baseline also noticeably hampers performance. Using simple Monte Carlo return estimates instead of TD(A) seems to be viable alternative, and the algorithm still achieves competitive performance on some tasks. When combined, these different components yield substantial performance gains over standard RWR.
To further evaluate the effect of experience replay, we compare policies trained using replay buffer with different capacities. Figure 3 illustrates the learning curves for buffers of size 5k, 20k, 50k, 100k, and 500k, with 50k being the default buffer size in our experiments. The size of the replay buffer appears to have a significant impact on overall performance. Smaller buffer sizes can result in instabilities during training, which again may be an effect of overfitting to a smaller dataset. As the buffer size increases, AWR remains stable even when the dataset is dominated by off-policy data from previous iterations. In fact, AWR appears more stable with larger replay buffers, but progress can also become slower. Since the sampling policy μ(aas)) is modeled by the replay buffer, larger buffer can limit the rate at which μı changes by maintaining older data for more iterations.
5.3 MOTION IMITATION
In this section, we show that AWR can also solve figh-dimensional tasks with complex simulated characters, including 34 DoF humanoid and 64 DoF dog. The objective of the tasks is to imitate reference motion clips recorded using mocap. The experimental setup follows the framework proposed by Peng et al. (2018) The motions include walking and running (e.g. canter), as well as acrobatic skills, such as cartwheels and spinkicks. Figure I shows snapshots of the behaviors learned by the AWR. Table 2 and Figure 4 compare the performance of AWR to RWR and PPO. AWR performs well across the set of challenging skills, consistently achieving comparable or better performance than PPO. RWR struggles with controlling the humanoid, but exhibits stronger performance on the dog. This difference may be due to the more dynamic and acrobatic skills of the humanoid.
7

---

Under review as a conference paper at ICLR 2021
Humanoki: Spinkick
Cantee
Task
РPO
RWR
AWR (Ours)
ITD
Humamoid:
Cartwheel
0.76 + 0.02
0.03 + 0.01
0.78 + 0.07
03
WiR--Gaal
Humanoid:
0.00
0.05 0.03
0.77 0.04
Spinkick
Dog: Canter
0.76 0.00
0.78 + 0.04
0.86 0.01
0.2
Dog: Trot
0.86 + 0.01
0.86 + 0.01
0.86 + 0.03
Dog: Turn
0.75 + 0.02
0.75 + 0.03
0.82 + 0.03
00
05
1/5
1.5
2.5
3.0
0.5
3.0
05
3.0
15
24
25
30
Table 2: Performance of algorithms the motion imitation tasks. Returns are normalized between tasks. On these challenging tasks, AWR generally learns faster than PPO and RWR.
Figure 4: Learning curves on motion imitation
the minimum and maximum possible returns.
Antio2
laPCChetthhu
liapperu22
Whlter25-2
3750
4003
22:00
4000
2003
1000
80
100
6PC
664
AMH
4003
K.C
BAG
AMR
60
190
150
80
PMD
113
MC
860
Figure 5: Performance of various algorithms on off-policy learning tasks with static datasets. AWR is able to learn policies that are comparable or better than the original demo policies.
5.4 OFF-POLICY LEARNING WITH STATIC DATASETS
Next, we evaluate AWR in a fully off-policc setting, where the algorithm is provided with a static dataset of experiences, and then tasked with learning the best possible policy without collecting any additional data. To evaluate our method, we use the off-policy tasks proposed by Kumar et al. (2019). The dataset consists of trajectories Unlike standard imitation learning tasks, which only observes the states and actions from the demo policy, the dataset also records the reward at each step. The demo policies are trained using SAC on various OpenAl Gym tasks. A dataset of million timesteps is collected for each task.
{(s. ag ro), ($1, a1,71) 71) ....) from rollouts of demo policy.
For AWR, we simply treat the dataset as the replay buffer D and directly apply the algorithm without any modifications. Figure comparres AWR to the original demo policy (Demo) and a behavioral cloning policy (BC). We also include comparisons to recent off-policy methods: batch-constrained O-learning (BCQ) (Fujimoto et al., 2019) and bootstrapping error accumulation reduction (BEAR) (Kumar et al. 2019), which have shown strong performance on off off-policy learning with static datasets. Note that both of these prior methods as TD3 and SAC, which are already quite complex. In contrast, AWR is simple and requires no modifications for the fully off- off-policy setting. Despite not collecting any additional data, AWR is able to learn effective policies from these fully off-policy datasets, achieving comparable or better performance than the original demo policies. On-policy methods, such as PPO performs poorly in this off- off- policy setting. Q-function based methods, such as TD3 and SAC, can in principle handle off-policy data but tend to struggle in practice (Fujimoto al. 2019 Kumar et al. 2019). Unlike Q-function based methods, AWR is less susceptible to issues from out- -fisttribttitioo actions as the policy always trained on observed actions from the behaviour data (Kumar et al.) 2019). AWR also shows comparable performance to BEAR and BCQ, which are specifically designed for this off- off-policy setting and introduce considerable algorithmic overhead.
modifications to existing off- off-policy RL methods, such
6 DISCUSSION AND FUTURE WORK
We presented advantage-weighted regression, simple off-policy reinforcement learning algorithm, where policy updates are performed using standard supervised learning methods. Despite its sim- plicity, our algorithm is able to solve challenging control tasks with complex simulated agents, and achieve competitive performance on standard benchmarks compared to number of well-established RL algorithms. Our derivation introduces several new design decisions, and our experiments verify the importance of these components. AWR is also able to learn from fully off-policy datasets, demon- strating comparable performance to state-of-the-arr off-policy methods. While AWR is effective for a diverse suite of tasks, it is not yet as sample efficient as the most efficient off-policy algorithms. We believe that exploring techniques for improving sample efficiency and performance on fully off ofoicy learning can open opportunities to deploy these methods in real world domains. A better theoretical understanding of the convergence properties of these algorithms, especially when combined with experience replay, could also be valuable for the development of future algorithms.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Manfred Otto Heess, and Martin A. Riedmiller. Relative entropy regularized policy iteration. ArXiv, abs/ 0ass1882.00256, 2018a.
Abbas Abdolmalcki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posterion policy optimisation. In International Conference on Learning Representations, 2018b. URL
ew htt.://wwwweeeeeeeeeeeeeeeeeeeee wtw. form?? d=S1 ANxQW0b
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https 2017.
//github. http://cooooooee/aaaaaaeaee
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, pp. 1329-1338, 2016. URL http: //proceedings..o mlr www.ar.reeeevvvvv /V4/////////// @uuan16.htmm
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2021- 2030, Long Beach, California, USA, 09- 15 Jun 2019. PMLR. URL http::/pproceedii
mlr. press /v9 / ful 9a. html
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR. URL httpp://proceecingg. mlr. press/v800 Eu mim htm.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2062-2062, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URLhttp 7/proceedings mlr press /ttp////fff//oo//o// 9a html
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings Machine Learning Research, pp. 2829 2829-2838, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http::/proceeddnngg .l www...pres///e. wtwps///vvv//uu...ooo html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018a. PMLR. URL httt:::/proceed.nnggge/////////e mlr.mmmmr... press/ w880haarnojjall htm.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Leurning Research, pp. 1861-1870 Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018b. PMLR. URL http: http:t:::/prrcceenggs...e press// 188/hharro jalbb html
Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiayma, and Jan Peters. Adaptive importance sampling for value function approximation in off off-policy reinforcement learning. Neural Netw., 22(10): 1410, December 2009. ISSN 0893-6080. doi: 10.1016/f neumet.2009.01 URL
http://dxxccm doi org 10. 101 6/ neunet. 0a...00000 2009.01 00..000.002

---

Under review as a conference paper at ICLR 2021
Danijar Hafner, James Davidson, and Vincent Vanhoucke. Tensorflow agents: Efficient batched reinforcement learning in tensorflow. CoRR, abs/1709.02878, 2017. URL http://arxiv
arq/abs/11s 1709. 02878
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learnnn with double q-learning. In Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, AAAT 16, pp. 2094- 2100, AAAI Press, 2016. URL http:///a....cc.o.ttttttttttoo.t 3016191
cfm 2 3016100
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. S. M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1700.00286, 2017. URL http:
//arxiv. www.xii..rg/abs///.... 02286
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney,
Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining URL http:
improvements in deep reinforcement learning. CoRR, abs/1710.02298,, 2017.
waaxxvv.rg//asaaaooooooo abs/ /1700. 02298
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp. 267-274, San Francisco. CA. USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1-55860-873-7,
URL http: //11 acm. org/ cetttiiaiiioooooo cfm71c-6455111 656005
Jens Kober and Jan R. Peters. Policy search for motor primitives in roboties. D. Koller, Schuur- mans, Y. Bengio, and I.. Bottou (eds.). Advances in Neural Information Processing Systems 21, pp. 149-856. Curran Associates, Inc., 2009.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off q-learning via bootstrapping error reduction. CoRR, abs/1906.00949, 2019. URL 1906. 00949
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
Nicolas Meuleau, Leonid Peshkin, Leslie P. Kaelbling, and Kee cung Kim. Off-policy policy search. Technical report, 2000.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature. 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx. doi. http://xx..iio./////.../// www.orgg/1... 1038 http:/nuuuree2222
Rémi Munos, Thomas Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Proceedings the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. 1054-1062, USA, 2016. Curran Associates Inc ISBN 978-1-5108-3881-4 URL http: wtw.:aaammrrr//ttaaeeeeeeeeeeeee wttp://ot//oaiiioooooooo wfw. wfw.:ddd
acm
3152214
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, pp. 807-814, USA, 2010. Omnipress. ISBN 978-1-60558-90777 URL
hittp://at acm. wr..orgg..o ci tat ion cfm? : http:oooccccmm.mmm11310313311110 001.43222.1114.44 3104425
Gerhard Neumann. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference International Conference on Machine Learning. ICML'11, pp. 817-824, USA, 201 Omnipress. ISBN 978-1-4503-0619-5 URL http: : ///dl acm. /rg/
citat cfm? 31 04482 10044442.310040 3104585
10

---

Under review as a conference paper at ICLR 2021
Gerhard Neumann and Jan R. Peters. Fitted q-iteration by advantage weighted regression. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 1177-1184, Curran Associates, Inc., 2009.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Muchine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 0178-38888888 Stockholmsmässan. p
Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: http::/proceeednnggeeeeeooo mir press/ w80/ohT Bb. ht.ml
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Xue Bin Peng, Picter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37 URL
(4):14311-143:14 July 2018. ISSN 0730-0301. doi: 10.11453197517:220311
//doi acm. wttp:o/gg./..../..........
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighter regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML '07, pp. 745-750, New York, NY, USA, 2007. ACM. ISBN 978-1-555933-73--33 doi: 10.1 1455 1273496. 011544961173355 1273590. URLhttpp://ool acm. 10. 11e1111452222 httg//..1111///2734455353311 1273590
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI' pp. 1607-1612. AAAI
http://.t..aa.../.o/.ooooo Press, 2010. http:p//ww.......ooo.oaaaaaaaaaa
2898886
Vitchyr Pong. Rikit. htt http:::/ggtccub..mm/m/ttttt////
2019.
Doina Precup, Richard S. Sutton, and Sanjoy Dasgupta. Off-policy temporal difference learning with function approximation. In Proceedings of the Eighteenth International Conference on Machine Learning. ICML '01, pp. 417-424, San Francisco, CA, USA, 2001. Morgan Kaufmaan Publishers Inc. ISBN 1-55860-778-1. URL http::/dl. org/ wttpg////.iii.....////o./.o.ottt fmmidd
015530.65588
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. URL hatt:://proceeedinge mlr press/ v 1man15. html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347,, 2017. URL http: wtwpaaaxvvvor../... abs/ 1707. 06347
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Richard S Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. A. Solla, T. K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems 12, pp. 1057-1063. MIT Press, 2000.
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Sesa-Bianchi and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 6288 -288-6297. Curran Associates, Inc., 2018

---

Under review as a conference paper at ICLR 2021
Ziyu Wang, Victor Bapst, Nicolas Heess. Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. CoRR, abs/ 1.000224 2016. http:///////ww..../aaaaaaaaa///a 111.1611111111...111111111111111
Ronald J. Williams, Simple statistical gradient-folloowing algorithms for connectionist reinforcement learning. Mach. Learn., 823-))29666 May 1992. ISSN 0885-6125. doi: 10.1007/BF0099266. URL httpp://ooal...oo//0000000////99
12


Reviews:
[{"review_id": "hwG96AKnLS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on developing an RL learning algorithm that is simple and can significantly improve performance over existing algorithms. The paper presents the algorithm AWR, which is an extension of the algorithm reward weighted regression. The primary extensions of RWR are using the advantage function instead of the q function and the ability to use experience replay. Experiments on common environments are conducted to evaluate the performance of AWR and compare it to other algorithms. There are ablation experiments to justify the choice of some of the extensions. \n\nThe development of simple, effective off-policy algorithms for reinforcement learning is still an open problem, and this paper tries to take a step towards addressing it. The paper gives a detailed derivation of the proposed algorithm, and the ablation studies provide some information as to what components in the algorithm make it useful. However, I do not believe this paper is ready for publication because the extensions are minor, and experiments lack scientific rigor, making it unclear what is to be learned from the paper. There are also some errors in the paper. \n\nThere are some essential questions the paper should address but it did not.\n\nWhat is the benefit of using the baseline in RWR? Does it change the weighting of the update, or is it just a variance reduction technique?\n\nThe beta term in RWR is adapted during learning. How does the choice of beta in AWR affect the weighting of rewards during learning?\n\nThe paper says there is a theoretical analysis of the algorithm, but I only see the algorithm's derivation. What analysis is this statement referring to? \n\nThe extension of RWR to using experience replay is formulated as using a mixture policy of past policies using weights w_i for each policy pi_i. How are weights w_i chosen? \n\nThe algorithm, as defined, performs a global maximization at each step. However, neural networks are used, and this maximization cannot be guaranteed. How are the updates performed? There are other changes to make the algorithm used in the experiments, but they are not connected to the algorithm's design, e.g., the normalized advantage function. \n\nExperiments:\nThe primary motivation and claim of this paper is the design of a more effective algorithm. Therefore, one should expect the presented algorithm to be a significant increase in performance. However, this cannot be concluded based on the experimental methodology used. The main issues are with how hyperparameters are selected and the lack of statistical analysis of the results. \n\nSome undefined processes set the hyperparameters for AWR, and the parameters for the other algorithms were left unspecified. Based on this experimental setup, it is impossible to tell if AWR is better or worse than the other algorithms or if it is just due to the specific setting of hyperparameters. \n \nThe performance results are given along with standard deviations over ten trials. However, this does not directly quantify how likely these results are achieved. Furthermore, it has been pointed out in several past works that the performance of RL algorithms are highly stochastic, and more trials and proper statistical analysis is needed to gain confidence in the outcomes (Colas et al., 2018, Henderson et al., 2017).  \n\nThere exist more rigorous evaluation procedures that may be useful in improving this paper (Dodge et al., 2019, Jordan et al. 2020, Sivaprasad et al. 2020).\n\nMathematical errors:\nThe jump from (9) to (10) is not correct. Sampling from a fixed-sized dataset is not equivalent to sampling from the state distribution of a policy. \n\nIn the algorithm, samples of states and actions are drawn according to the data distribution. However, this does not correspond to the empirical samples of discounted state distribution; i.e., there needs to be a gamma^t term in the expectation (Thomas, 2014). \n\n\nReferences\n\nColas, C., Sigaud, O., & Oudeyer, P. Y. (2018). How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv:1806.08295.\n\nDodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. arXiv preprint arXiv:1909.03004.\n\nHenderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2017). Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560.\n\nJordan, S. M., Chandak, Y., Cohen, D., Zhang, M., & Thomas, P. S. (2020). Evaluating the Performance of Reinforcement Learning Algorithms. In Proceedings of the 37th International Conference on Machine Learning.\n\nSivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., & Fleuret, F. (2020). Optimizer benchmarking needs to account for hyperparameter tuning. In Proceedings of the 37th International Conference on Machine Learning.\n\nThomas, P. (2014, January). Bias in natural actor-critic algorithms. In International conference on machine learning (pp. 441-448).\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_8xPsZXPYA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a reinforcement learning algorithm that applies advantage-weighted regression. In each iteration, it samples trajectories from a mixture of previous policies, estimates the value function and then computes the advantage value to estimate the policy. The idea is very similar to the work published in “Neumann, Gerhard and Peters, Jan R, Fitted Q-iteration by advantage weighted regression, Advances in neural information processing systems, 2009”, starting from reward-weighted regression and further developing to advantage weighted regression.  The difference in this paper is to add a constraint on the policy search, requiring the policy to be similar to the sampling policy. However, this constraint has also been studied in the paper \"Christian Wirth and Johannes Furnkranz and Gerhard Neumann, Model-Free Preference-based Reinforcement Learning, AAAI 2017\" (It seems not in reference). Overall, it may enhance this paper if it has more technical novelty when developing a new algorithm.\n \nMy major concern is that this paper has been submitted last year and resubmitted this year without new additions. Moreover, this paper has been available to the public since last year with information on authors and affiliations, which may violate the double-blind review policy (e.g. the version updated on Oct 7, 2019 at https://arxiv.org/abs/1910.00177).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGRoKS96dfa", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This study presents a deep reinforcement learning method, Advantage-Weighted Regression (AWR). The policy update of AWR is constrained as in a similar manner as REPS (Peters et al., 2010). Although the benefit of AWR is not clear in the reinforcement learning tasks, AWR exhibits its advantages in the context of imitation learning and off-policy learning with static datasets. \n\nThe paper is well-organized and easy to follow. However, there are some unclear points. I would like to ask the authors to clarify the following points. \n\nMy main concern is the novelty of the method because the difference between MPO and AWR is not clear to me. In my understanding, both MPO and AWR solve the same optimization problem to satisfy the KL divergence constraint, and both MPO and AWR update the policy by maximizing the weighted log-likelihood. The difference I’m aware of is \n1)\tMPO uses the Q-function, while AWR uses the advantage function\n2)\tMPO uses Retrace, while AWR uses TD(\\lambda)\n\nThe use of Retrace may not be important because Retrace can also be used for AWR. If we remove the baseline from AWR and uses Retrace instead of TD(\\lambda), is it equivalent to MPO? Please correct me if I misunderstand anything.\n\nOther questions:\n- I do not clearly understand why AWR is suitable for off-policy learning with static datasets. Please provide the rationale. \n\n- Why SAC and TD3 are not compared in the imitation learning tasks? I do not understand why only PPO and RWR were chosen as baselines. In addition, please describe the reason why the AWR is suitable for the imitation learning setting. \n\nMinor points\n- Although the “simplicity” of AWR is claimed several times, I’m not sure whether AWR is really simple because the difference between AWR and MPO is not big. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "L6rsK5gL1Zz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Advantage weighted regression, which relies on two sub-routines : (a) train a value function baseline using regression, and, (b) policy learning, which is advantage weighted.  The paper is well written, and has experiments on both on-policy and offline tasks, with ablation studies on various algorithm design choices.\n\nSome questions:\n\n[1] Contributions: it appears the objective is a minor modification off prior work, where, the returns are replaced with advantages. While this makes intuitive sense, and it does indicate improved results, the actual contribution is rather limited. While the paper does present results in off-policy settings, these derivations etc. are standard. Can the authors describe precisely what design decisions they propose in this work (on top of RWR/other prior work) that are novel? \n\n[2] The authors mention they present a theoretical analysis (e.g. in page 1) of AWR. While I see a derivation of the AWR update, this isn’t a theory analysis of any of the proposed algorithm's behavior. In particular, can the authors make a rigorous claim as to why this is a better algorithm than current approaches? This could be through bounding the variance of the updates and indicating this improves over prior work, or, alternatively, showing rates of convergence and how it compares against that of standard NPG. Alternatively, one can view the exponentiation of the advantage as reward transformations which tend to impact the convergence, if one utilizes a policy gradient method for obtaining the new policy (see for e.g. Ghosh et al. 2020 (An operator view of policy gradient methods)). However, these need to be precisely sketched out. Without such results, the paper indicates its merit through its experiments.\n\n[3] Experiments: Looking at table 1: I see that AWR appears to be significantly worse than SAC in several tasks - it is again unclear as to why this is considered “competitive in terms of final performance compared to prior works” as written in table 1’s captions. Furthermore, in terms of performance of TRPO, I find that the results appear to be very much lower compared to what I believe they obtain (for e.g. see Rajeswaran, Lowrey, Todorov and Kakade (Neurips 2017)) - could you mention how these experiments were conducted and why there are these discrepancies in terms of final performance? \n\n[4] Ablation: How is the \\beta parameter of AWR chosen? How sensitive is the behavior of the algorithm to this hyper-parameter?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to develop a simple yet efficient deep RL algorithm for off-policy RL. The proposed method uses advantages to as weight in regression, which is an extension of the known method of reward-weighted regression. The paper is in general nicely written, and it comes with a set of theoretical analyses and experiments. While all reviewers admit that the approach is interesting and the work makes an attempt to solve an important yet open problem, there are several aspects of the paper that make it not ready for publication in its current form:

- Novelty: As pointed out by reviewers, the proposed method appears to be a minor modification of existing off-policy solvers. Although the use of advantages as weights makes intuitive sense, it is unclear why and how the new method significantly differs from and outperforms existing methods. Going forward, it would be helpful if the authors could present more convincing arguments/experiments to demonstrate the power of ARW, relative to similar existing methods.

- Experiments provide some insights into the difference between several algorithms, but the results are not strong enough to support the claim of the paper. Please see reviewers' comments for more details. We strongly recommend the authors to take these comments into consideration and develop more rigorous experiments to demonstrate advantages of AWR.

- Theoretical analysis is limited. As R#2, R#3 mentioned, the theory analysis in the paper seems to not match the algorithm, and there remain bugs, this it doesn't add to the paper. Although theory might not be the focus of the paper, if the authors decide into include theoretical analysis, the analysis would hopefully provide insights into why and by how much the approach is better.


Decision: Reject