Advantage Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
Authors: Xue Bin Peng
Aviral Kumar
Grace Zhang
Sergey Levine
Venue:  (2020)
Keywords: reinforcement learning
policy search
control

Abstract: In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

Under review as a conference paper at ICLR 2020
ADVANTAGE-WEIGHTE REGRESSION: SIMPLE AND SCALABLE OFF-POLICY REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we aim to develop a simple and scalable reinforcement learning al- gorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off off-policy data. Our proposed ap- proach, which we refer to as advantage-weighhe weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy, The method simple and general, can accommodate continuous and discrete ac- tions, and can be implemented in just a few lines of code top of standard super- vised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We eval- uate AWR on suite of standard OpenAl Gym benchmark tasks, and show that it achieves competitive performance compared to number of well-established state-of-the art RL algorithms. AWR is also able to acquire more effective poli- cies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters. (Videq")
INTRODUCTION
Model-free reinforcement learning can a general and effective methodology for training agents to acquire sophisticated behaviors with minimal assumptions on the underlying task (Mnih et al. 2013 Heess et al.] 2017; Pathak et al., 2017). However, reinforcement learning algorithms can be substantially more complex implement and tune than standard supervised learning methods. Arguably the simplest reinforcement learning methods are policy gradient algorithms (Sutton et al.) 2000), which directly differentiate the expected return and perform gradient ascent. Unfortunately, these methods can be notoriously unstable and are typically on-policy (or nearly on-policy), often requiring a substantial number of samples learn effective behaviors. Our goal is develop reinforcement learning algorithm that is simple, easy to implement, and can readily incorporate off-policy experience data.
In this work, we propose advintage-weighted regression (AWR). simple off-policy algorithm for model-free RL. Each iteration of the AWR algorithm simply consists of two supervised regression steps: one for training a value function baseline via regression onto cumulative rewards, and another for training the policy via weighted regression. The complete algorithm shown in Algorithm
Figure 1: Complex simulated character trained using advantage- weeigtte regression. Left: Hus manoid performing a spinkick. Right: Dog performing canter.
'Supplementary video: fites.goooe..com/viee//vvruuuu

---

Under review as a conference paper at ICLR 2020
AWR can accommodate continuous and discrete actions, and can be implemented in just few lines of code on top of standard supervised learning methods. Despite its simplicity, we find that AWR achieves competitive results when compared to commonly used on-policy and off- off-poliey RL algo- rithms, and can effectively incorporate fully off-policy data, which has been a challenge for other RI. algorithms. Our derivation presents an interpretation of AWR constrained policy optimization procedure, and provides a theoretical analysis of the use of off-policy data with experience replay.
We first revisit the original formulation of reward-weighted regression, an on-policy RL method that utilizes supervised learning to perform policy updates, and then propose a number of new de- sign decisions that significantly improve performance on suite of standard continuous control benchmark tasks. We then provide theoretical analysis of AWR, including the capability to in- corporate off-policy data with experience replay. Although the design of AWR involves only a few simple design decisions, we show experimentally that these additions provide for large improve- ment over previous methods for regression- policy search, such as weiire-weighted regression (RWR) (Peters& & Schaal 2007). while also being substantially simpler than more modern methods, such as MPO (Abdolmaleki et al. 2018). We show that AWR achieves competitive performance when compared to several estaleestallihhed of-te-o-thhe-e art on-poliey and off- off-policy algorithms. We further demonstrate our algorithm on challenging control tasks with complex simulated characters.
2 PRELIMINARIES
In reinforcement learning, the objective is to learn a control policy that enables an agent to maxi- mize its expected return for given task. At each time step the agent observes the state of the environment 8t E S, and samples an action a, E A from a policy a, ~ <aay(s)) The agent then applies that action, which results in new state St+1 and a scalar reward T; = r(s,, at). The goal is to learn an optimal policy that maximizes the agent's expected discounted return ((T),
8 Σ
((z) = E Exppr(r)
r'r
= (s) (r(ss
(I)
where Pa (r) represents the likelihood of trajectory (S0, an. ro) (51. a1, r1). ...} under a pol- icy π, and E [0, 1) is the discount factor. da (s) = = p'p(s = s/ represents the unnormal- ized discounted state distribution induced by the policy π (Sutton & Barto 1998), and p(s; s is the likelihood of the agent being in state after following π for timesteps. A popular class of algorithms for solving this problem is policy gradient (PG) methods, which directly estimates the gradient of the expected return with respect to the policy parameters (=)( and then updates the policy with gradient ascent. Basic PG algorithms are generally on-policy methods, which require the data to be sampled from the same policy that is being optimized. This can result in poor sample efficiency, but PG algorithms can be modified to utilize off-policy data.
An alternative class of RL methods is expectation-maximizatiio algorithms. Instead of estimating the gradient of the expected return, EM algorithms first construct an estimate of the optimal policy using dataset of experiences (B-step), and then projects this estimate onto the space of param- eterized policies (M-step). An early example of an EM-based RL algorithm is reward-weighted regression (RWR) (Peters et al. 2010). At each iteration, the E-step constructs optimal policy according to π "T(ass) x (a/ss))x (Rga/3).. where ną represents the policy at the ktth iteration of the algorithm, Re.a De 7're the return, and 0 is temperature param- eter. Then the M-step projects xt onto the space of parameterized policies by solving a supervised regression problem:
estimate of the
( (3
1 Re,a
Tk+1 arg max Egwdr, (s) (a) (als) log raals) exp
The RWR update can be interpreted as fitting a new policy T to samples collected under the cur- rent policy Ty. where the likelihood of each action is weighted by the exponentiated return received for that action. Since EM algorithms do not directly estimate the gradient of the expected return with respect to the current policy, they can be more amenable learning from off-policy data.
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Sawaantage-Weigheed Regression
I:
random policy
2: D -
3: for iteration k I,..., kmax do
4: add trajectories {T:} sampled via Th to D
5: V arg miny B..arD
_ V(s)
6: Tk+11 + arg max, Es log T(ass) exp ( - -V2 v2((s))
7: end for
3 DDVANTAGEEWEIGGHEED REGRESSION
In this work, we present wetinntage-weigheed regression (AWR), a simple off- off-policy RL algorithm based on geward-weighter regression. We first provide an overview of the complete advantage- weighted regression algorithm, and then describe its theoretical motivation and analyze its proper- ties. The complete AWR algorithm is summarized Algorithm Each iteration of AWR consists of the following simple steps. First, the current policy T& (a|s) used to sample a batch of trajecto- ries { that are then stored in the replay buffer D, which is structured as a first-in first-out (FIFO) queue, is common for off-policy reinforcement learning algorithms (Mnih et al. 2015 Lillicrap et al. 2016). Then, the entire buffer is used to fit a value function (i((s) to the trajectories in
the replay buffer, which can be done with simple Monte Carlo return estimates R.P. -Ei 7'r.- Finally, the same buffer used to fit a new policy using advantage-weighted regression, where each state-action pair in the buffer is weighted according the exponentiated advantage exp( APr as),
with the advantage given by (s, a) R..a (s) and 3 is a hyperparameter. AWR uses only supervised regression learning subroutines, making the algorithm very simple implement. In the following subsections, we first motivate the algorithm as an approximation to constrained pol- icy search problem, and then extend our analysis to incorporate experience replay.
3.1 DERIVATION
In this section, derive the AWR algorithm policy search problem. Our goal is to find a policy that maximizes the expected improvement ([))= = ((r) J(s) over sampling policy u(als). We first derive AWR for the setting where the sampling policy is a single Markovian policy. Then, in the next section, we extend our result to the setting where the data is collected from multiple policies, as in the case of experience replay that we use in practice. The expected improvement ()()) can be expressed in terms of the advan- tage A (s. a) Rd (*(s) with respect the sampling policy y (Kakade & Langford 2002
approximate optimization of constrained
Schulman et al. 2015
(T) = Bgnd, (x) (s, a)] = (s) Baws((ls) R V*(s)]
(3)
where Rh denotes the return obtained by performing action in state s and following μ for the following timesteps, and (*(s) = S. (als))R da corresponds to the value function of p. This objective differs from the ones used in the derivations of related algorithms, such as RWR and REPS (Peters& & Schaal 2007: Peters et al. 2010 Abdolmaleki et al. 2018), which maximize the expected return J(T) instead of the expected improvement. The expected improvement directly gives rise to an objective that involves the advantage. We will see later that this yields weights for the policy update that differ in a subtle but important way from standard reward- weighted regression. As we show in our experiments, this difference results in a large empirical improvement.
The objective in Equation 3 can be difficult to optimize due the dependency between (s) and π, as well as the need to collect samples from æ. Following Schulman et al. (2015), we can instead optimize an approximation (T)) of 17(T)) using the state distribution of μr:
( )= F ( ( Rawn(s)s) [R V*(s)]
Here, (T)) matches (7)) to first order [Kakade & Langford 2002), and provides good estimate of n if n and y are close in terms of the Kl.ddivergence (Schulman al. 2015). Using this objective,
3

---

Under review as a conference paper at ICLR 2020
can formulate the following constrained policy search problem:
[ dd((s) /. i(als) [R".. (S))s] da ds
arg max
(5)
s.t. [ dn(s)DKL (1(()s)l ds <.
(6)
The constraint in Equation ensures that the policy is close to the data distribution of ju, and therefore the surrogate objective (i(T)) remains a reasonable approximation to v)(T). We refer the reader to Schulman al. (2013) for a detailed derivation and an error bound.
We can derive AWR as an approximate solution to this constrained optimization. This derivation follows a similar procedure as Peters et al. (2010), and begins by forming the Langrangian of the constrained optimization problem presented above,
[(T, 33 { du(s)
[RK, (s)] da ds + 3 { (M(-1s) mc
iTaas)
(s)
where / is Lagrange multiplier. Differentiating C(n, B) with respect to m(ass) and solving for the optimal policy n² results in the following expression for the optimal policy
π (a|s) = (ss) u(als) exp ( (RR,s - V*()s
(8)
with Z(s) being the partition function. A detailed derivation is available in Appendix A If i is represented by a function approximator (e.g., neural network), new policy can be obtained by projecting x² onto the manifold of parameterized policies,
arg min
EgwD [DKL (T (+|s)[[[(-[ss)]
(9)
(F (8) B Panlls)s) log (als)) exp (REA - VV((s)
arg max
(10)
While this derivation for AWR largely follows the derivations used in prior work (Peters et al. 2010 Abdolmaleki et al.) 2018), our expected improvement objective introduces a baseline (P(s) to the policy update, which as we show in our experiments, is crucial component for an effective algorithm. A similar selaantage-wiighiing scheme has been previously used for fitted Q-iteration Neumann & Peters 2009). where the policy is given by rTals)
z exp ((R V*(s)) /B). In this defiiitin,, the likelihood of an action does not depend on its likelihood under the sampling distribution, and therefore does not enforce a trust region with respect to the sampling distribution. Next, we extend AWR to incorporate experience replay for off- off-policy training, where the sampling policy is no longer a single policy, but rather a mixture of policies from past iterations.
3.2 EXPERIENCE REPLAY AND OFF-POLICY LEARNING
crucial design decision of AWR is the choice of sampling policy p(als)) Standard implementa- tions of RWR typically follow on-policy approach, where the sampling policy is selected to be the current policy ((ass) = Ty (a|s) at iteration k. This can be sample inefficient, as data collected at each iteration of the algorithms
discarded after single update iteration. Importance sam- pling can be incorporated into RWR to reuse data from previous iterations, but at the cost of larger variance from the importance sampling estimator (Kober & Peters 2009). Instead, we can improve sample efficiency of AWR by incorporating experience replay and explicitly accounting for training data from mixture of multiple prior policies. As described in Algorithm at each iteration, AWR collects a batch of data using the latest policy and then stores this data in replay buffer D, which also contains data collected from previous policies (. } policy are then updated using samples drawn from D. This replay strategy is analogous to modeling
The value function and
the sampling policy mixture of policies from previous iterations (T) >k WeF;(T (T). where ni (T) = p(T (T;)) represents the likelihood of a trajectory T under a policy n, from the ith iteration, and the weights Σ, w; = 1 specify the probabilities of selecting each policy π;
We now extend the derivation from the previous section to the off- off-policy setting with experience re- play, and show that A Aggorith i indeed optimizes the expected improvement over sampling policy
4

---

Under review as a conference paper at ICLR 2020
modeled by the replay buffer. Given replay buffer consisting of trajectories from past policies, the joint state-action distribution of μ is given by u((s, a) 1 wyds, (s) and similarly for the marginal state distribution di = Σi=1 w.dx, (s). The expected improvement can now be ex- pressed with respect to the set of sampling policies in the replay buffer: 17(TT) = J(T) - (T.) Similar to Equation 3 (7())) can be expressed in terms of the advantage (s, a) = Rg V*i of each sampling poticies,
Σ Ew (s)
TT(T) J(T) Σ w;:((:)) Bend (s)
(s) (ale(a)))
(11)
As before, we can optimize an approximation (i(T) of (T(T) using the state distribution of j,,
(TT) = da (awr(())))) [A a)] = Σ wi (Eawdd, (s))))) [ATi (s,a)]
(12)
1=1
In Appendix B we show that the update procedure Algorithm Dotiiiizes the following objective:
arg max
Σ Wi (s) (a) Bant(al)))) [sT (s, ())))))))
(13)
s.t. E (s) [DKL (T((s))lu(+ss)) e,
(14)
where p(aas)
represents the conditional action distribution defined by the replay buffer. This objective can be solved via the Lagrangian to yield the following update:
ml Σ, wide, (als)
dr, (8)
En wydn, (s)
( 1
Σ, w,dd,, (s)
arg max Wi Bgnde, (s) (anl)))) log ग (als)) exp
Σ 1=1
R
E, wydr (s)
T7
(15)
where the expectations can be approximated by simply sampling from D following Line 6 of Algo- rithm [ detailed derivation is available Appendix B Note, the baseline in the exponent now consists of an average of the value functions of the different policies. One approach for estimating this quantity would be to fit separate value functions V for each policy. However, if only a small amount of data is available from each policy, then V could be highly inaccurate (Fu et al. 2019). Therefore, instead of learning separate value functions, we fit single mean value function V(s) that directly estimates the weighted average of Vs
arg min Σ W; Eg,,de, (a))s))
|| - (s)112)
(16)
This loss can also be approximated by simply sampling from the replay buffer following Line 5 of Algorithm The optimal solution (s)) Σ, widr, (>) (s)) is exactly the baseline in Equation 15
Σ, myyd,, (8)
3.3 IMPLEMENTATION DETAILS
Finally, we discuss several design decisions that are important for a practical implementation of AWR. An overview of AWR is provided in Algorithm [ The policy update in Equation 10 requires sampling states from the discounted state distribution d (s). However, we found that simply sam- pling states uniformly from D was also effective, and simpler to implement. This is common strategy used in standard implementations ing the value function and policy, Monte Carlo estimates be used to approximate the expected return R.d of samples in D, but this can result in a approximate R.D using TD(A) to obtain lower-varianee estimate (Sutton & Barto 1998). TD(A) is applied by bootstrapping with the value function (s_(s) from the previous iteration. A simple Monte Carlo return estimator can also be used though, as shown in our experiments, but this pro- duces somewhat worse results. To further simplify the algorithm, instead of adaptively updating the Lagrange multiplier 3, as is done in previous methods (Peters & Schaal, [007 Peters et al.) 2010)) Abdolmaleki al. 2018), we find that simply using fixed constant for 3 is also effective. The
RL algorithms (Dhariwal et al. 2017). When updat-
estimate. Instead, we opt to
weights wh exp (R.,. VD (s)) used to update the policy occasionally assume cessively large values, which can cause gradients to explode. We therefore apply weight clipping Wg.a min (w..: Wmax) with threshold Wmax to mitigate issues due to exploding weights
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Existing RL methods can be broadly categorized into on-policy and off-policy algorithms (Sutton & Barto 1998). On-policy algorithms generally update the policy using data collected from the same
policy. A popular class of on on-policy algorithms is policy gradient methods (Williams) 1992 Sutton
al. 2000), which have been shown to be effective for diverse array of complex tasks (Heess al.)
2017 Pathak et al. 2017: Peng et al. 2018 Rajeswaran et al. 2018). However, on-policy algorithms are typically data inefficient, requiring large number of interactions with the environment. Off- policy algorithms improve sample efficiency by enabling a policy to be trained using data from other sources, such as data collected from different agents or data from previous iterations of the algorithm. Importance sampling is simple strategy for incorporating off-policy data (Suttinn & Barto 1998: Meuleau al. 22000 |Hachiya et al. 2200)) but can introduce optimization instabilities due to the potentially large variance of the importance sampling estimator. Dynamic programming methods based on Q-function learning can also leverage off- off-policy data (Precup et al. 2001] Mnih et al., 2015 Liliccap et al., 2016, Gu et al. 2016 HHarmoja al., 2018b). But these methods can be notoriously unstable, and in practice, require a variety of stabilization techniques to ensure more consistent performance (Hasselt al. 2016)] Wang al.
2016 Munos et al. (016 Hessel al.) Fujimoto et al., 2018) Nachum et al. 2018 Fu et al.
2019). Furthermore, it can be difficult to apply these methods to learn from fully off- policy data, where an agent is unable collect additional
environmental interactions (Fujimoto et al. 2019 Kumar et al. 2019).
Alternatively, policy search can also be formulated under an expectation-maxxiizaatie
framework.
This approach has lead to a variety of EM-based RL algorithms (Peters et al.) 2010: Neumann
2011 Abdolmaleki al., 2018), an early example of which is weight- regression (RWR)
(Peters& & Schaal 2007). RWR presents a simple on-policy RL algorithm that casts policy search as a supervised regression problem. A similar algorithm, relative entropy policy search (REPS) (Peters et al. 2010), can also be derived from the dual formulation of constrained policy search problem. RWR has a number appealing properties: it has a very simple update rule, and since each iteration corresponds to supervised learning, it can be more stable and easier to implement than many of the previously mentioned RL methods. Despite these advantages, RWR has not been shown to be an effective RL algorithm when combined with neural network function approximators, as demonstrated in prior work and our own experiments (Schulman et al.) 2015 Duan et al.) (0166). In this work, we propose a number of modifications to the formulation of RWR produce an effective off-policy deep RL algorithm, while still retaining much of the simplicity of previous methods.
Policy updates using supervised regression have been used in number of prior work. The opti- mization problem being solved in REPS is similar to AWR (Peters et 2010), but REPS optimizes the expected return instead of the expected improvement. The weights in REPS also contains Bellman error term that superficially resembles advantages, but are computed using linear value function derived from a feature matching constraint. Learning the REPS value function requires minimization of dual function, which is complex function of the Bellman error, while the value function AWR can be learned with simple supervised regression. More recently, AAbdoimaleki et al. (2018) proposed MPO, deep RI. variant of REPS, which applies partial EM algorithm for policy optimization. The method first fits a Q-function of the current policy via bootstrapping, and then performs a policy improvement step with respect to this Q-function under trust region constraint that penalizes large policy changes. MPO uses off-policy data for training a O-function critic via bootstrapping and employs Retrace(A) for off- off-policy correction (Munos et al.) (016)) In contrast, AWR is substantially simpler, as it can simply fit a value function to the observed returns in a replay buffer, and performs weighted supervised regression on the actions to fit the policy. Oh et al. (2018) proposed self-imitation learning (SIL), which augments policy gradient algorithms with an auxiliary behaviour cloning loss to reuse samples from past experiences. In contrast to SIL, AWR is a standalone algorithm, and does not need to be combined with an auxiliary RL algorithm. Neumann & Peters| (2009) proposed LAWER, a kernel-based fitted Q-iteration algorithm where the Bellman error is weighted by the normalized advantage of each state-action pair. This was then followed by a soft-policy improvement step. Similar to Neumann & Peters 2009). our method also uses exponentiated advantages during policy updates, but their definition of the policy is different from the one in AWR and does not enforce a trust region constraint. Furthermore, AWR does not perform fitted Q-iteration, and instead utilizes off- policy data in simple constrained policy search scheme for imitation learning,
procedure. /Wan et al. (2018) applied a similar advantage-weight but the method was not demonstrated for the RL setting. In this work, we propose several design
6

---

Under review as a conference paper at ICLR 2020
(a) HallCheetahh-2
(b) Hopper-v2
(c) LunarLander-v2
(d) Walker2d-22
Figure 2: Snapshots of AWR policies trained on OpenAl Gym tasks. Our simple algorithm learns effective policies for diverse set of discrete and continuous control tasks.
decisions that are vital for effective RL algorithm. We also provide theoretical analysis of AWR when combined with experience replay, and show that the algorithm indeed optimizes the expected improvement with respect to trajectory-level mixture of past policies modeled by replay buffer.
EXPERIMENTS
Our experiments aim comparatively evaluate the performance of AWR to commonly used on- policy and off-policy deep RL algorithms. We evaluate our method on the OpenAl Gym bench- marks (Brockman al., 2016), consisting of discrete and continuous control tasks. We also evaluate our method on complex motion imitation tasks with high-dimensional simulated characters, includ- ing a 34 DoF humanoid and 64 DoF dog (Peng et al.) 2018). We then demonstrate the effectiveness of AWR on fully off- oof-colicy learning, by training on static datasets of demonstrations collected from demo policies. Behaviors learned by the policies are best seen in the supplementary vided Code for our implementation of AWR is available at www.s/ogol...o.veewaeeaee
At each iteration, the agent collects a batch of approximately 2000 samples, which are stored in the replay buffer 'D along with samples from previous iterations. The replay buffer stores 50k of the most recent sam- ples. Updates to the value function and policy are performed by uniformly sampling minibatches of 256 samples from D. The value function is updated with 200 gradient steps per iteration, and the policy is updated with 1000 steps. Detailed hyperparameter settings are provided in Appendix
5.1 BENCHMARKS
We compare AWR number of state-of-the-art RL algorithms, including on-policy algorithms, such as TRPO (Schulman et al. 2015) and PPO (Schulman et al. (017). off-policy algorithms, such as DDPG (Lillicrap et al. 2016). TD3 (Fujimoto et al.) 2018). and SAC (Haarnoja al.) (0088)).
as well as RWR (Peters& & Schaal 2007). which we include for comparison due to its similarity to AWR TRPO and PPO use the implementations from OpenAl baselines (Dhariwal et al. 2017).
DDPG, TD3, and SAC uses the implementations from RLkit (Pong) 2019). RWR is a custom im- plementation following the algorithm described by Peters & Schaal (007))
Snapshots of the AWR policies are shown in Figure Learning curves comparing the different algo- rithms on the OpenAl Gym benchmarks are shown in Figure 3 and Table summarizes the average returns of the final policies across 5 training runs initialized with different random seeds. Overall, AWR shows competitive performance with the state-of-the-arr deep RL algorithms. It significantly outperforms on-policy methods such PPO and TRPO in both sample efficiency and asymptotic performance. While it is not yet as sample efficient as current state-of-the-art off-policy methods, such SAC and TD3, it is generally able to achieve a similar asymptotic performance, despite us- ing only simple supervised regression for both policy and value function updates. The complex Humanoid-V V2 task proved to be the most challenging case for AWR, and its performance still lags well behind SAC. Note that RWR generally does not perform well on any of these tasks. This sug-
Whiile we attempted to compare to MPO (Abolmalekii et al. 2018 we were unable to find source code for an implementation that reproduces results comparable to those reported by Abdolmaleki al. (2018), and could not implement the algorithm such that achieves similar performance those reported by the authors.

---

Under review as a conference paper at ICLR 2020
Hopperie22
Lomertandee-222
Welt
4393
10005
200
6000
0000
4000
2300
-100
3000
3000
0.3
0.5
25
2.5
0.0
02
0.4
0.0
0.3
0.2
0.4
066
3.0
055
2.0
TRPO
PPO
DDPG
TD3
SAC
RWR
AWR (Ours)
Figure 3: Learning curves of the various algorithms when applied to OpenAl Gym tasks. Results
are averaged across 5 random seeds. AWR is generally competitive with the best current methods.
Task
TRPO
PPO
DDPG
TD3
SAC
RWR
AWR (Ours)
Ant-v2
2901 85
1161 + 389
1550
4285
5909 ::
181 + 19
5067 256
HuliCheetah-v2
3302 428
4020 + 429
10563 1 382
4309 1238
92977 1206
1400 +: 370
9136 +: 184
Hopper-v2 v2
1880 + 837
1391 +: 304
855 +: 282
935 1: 489
2769 = 552
605 + 114
3405 + 121
Humanoid-v2
552 9
695 + 59
4382 + 423
81 + 17
8048 +: 700
509 + 18
4996 + 697
LunarLander v2
94
121 1
185 : 23
229 +
Walker2d-v2
2765 168
2617 + 362
401 + 470
4212 427
5805 + 587
406 + 64
5813 + 483
Table I: Final returns for different algorithms the OpenAI Gym tasks, with + corresponding to one standard deviation of the average return across 5 random seeds. In terms of final performance, AWR generally performs comparably or better than prior methods.
gests that, although AWR is simple and easy to implement, the particular modifications it makes compared to standard RWR are critical for effective performance. To illustrate AWR's generality on tasks with discrete actions, we compare AWR to TRPO, PPO, and RWR on LunarLander-v2. DDPG, TD3, and SAC are not easily applicable to discrete action spaces due to their need to back- propagate from the Q-function to the policy. On this discrete control task, AWR also shows strong performance compared to the other algorithms.
5.2 ABLATION EXPERIMENTS
To determine the effects of various design decisions, we evaluate the performance of AWR when key components of the algorithm have been removed. The experiments include an on-policy version of AWR (On-Policy), where only data collected from the latest policy is used to perform updates. We also compare with a version of AWR without the baseline V(s) (No Baseline), which corresponds to using the standard RWR weights We.a exp( 3 Re.a). and another version that uses Monte Carlo return estimates instead of TD(A) (No TD(A).. The effects of these components are illustrated in Figure 4 Overall, these design decisions appear to be vital for an effective algorithm, with the most crucial components being the use of experience replay and a baseline. Updates using only on-policy data can lead to instabilities and result in noticeable degradation in performance, which may be due to overfitting on smaller dataset. This issue might be mitigated by collecting a larger batch of on-policy data per iteration, but this can also negatively impact sample efficiency. Removing the baseline also noticeably hampers performance. Using simple Monte Carlo return estimates instead of TD(A) seems to be a viable alternative, and the algorithm still achieves competitive performance on some tasks. When combined, these different components yield substantial performance gains over standard RWR.
To better evaluate the effect of experience replay on AWR, we compare the performance of policies trained with different capacities for the replay buffer. Figure 4 illustrates the learning curves for buffers of size 5k, 20k, 50k, 100k, and 500k, with 50k being the default buffer size in our exper- iments. The size of the replay buffer appears to have significant impact on overall performance. Smaller buffer sizes can result in instabilities during training. which again may be an effect of over- fitting to a smaller dataset. As the buffer size increases, AWR remains stable even when the dataset is dominated by off-policy data from previous iterations. In fact, performance over the course of training appears more stable with larger replay buffers, but progress can also become slower. Since the sampling policy μ(aass) is modeled by the replay buffer, a larger buffer can limit the rate at which changes by maintaining older data for more iterations. Due to the trust region penalty in Equa- tion 7 a slower changing μ also prevents the policy ग from changing quickly. The replay buffer therefore provides simple mechanism to trade-off between stability and learning speed.
8

---

Under review as a conference paper at ICLR 2020
Ablation:: Foatara
Alfalion: Waler2e-22
Replay Buffer PalfC weldh-e2
Puplay Buller 2elee2
1000
72:3
CC03
$390
3300
13031
4009
2000
2300
2300
1000
3.0
1.5 2.0
0.0
0.
20
15
20 2.0 0.3
2.0
0.0
0.5
2.0
Figure 4: Left: Learning curves comparing AWR with various components removed. Each com- ponent appears contribute improvements in performance, with the best performance achieved when all components are combined. Right: Learning curves comparing AWR with different ca- pacity replay buffers. AWR remains stable with large replay buffers containing primarily off off-policy data from previous iterations of the algorithm.
(a) Humanoid: Cartwheel
(b) Humanoid: Spinkick
Dog: Trot
Dog:
Turn
Figure 5: Snapshots of 34 DoF humanoid and 64 DoF dog trained with AWR to imitate reference motion recorded from real world subjects. AWR is able learn sophisticated skills with characters with large numbers of degrees of freedom.
5.3 MOTION IMITATION
The Gym benchmarks present relatively low-dimensional tasks. In this section, we study how AWR can solve higher-dimensional tasks with complex simulated characters, including 34 DoF hu- manoid and 64 DoF dog. The objective of the tasks is to imitate reference motion clips recorded using motion capture from real world subjects. The experimental setup follows the motion imita- tion framework proposed by |Png et al. (2018). Motion clips are collected from publicly available datasets (CMU) SFU, Zhang et al. | 2018). The skills include highly dynamics motions, such as spinkicks and canters (i.e. running), and motions that requires more coordinated movements of the character's 's body, such as a cartwheel. Snapshots of the behaviors learned by the AWR policies are available in Figure Table
the performance of AWR RWR and the highly-tuned PPO implementation from Peng et al. (2018). Learning curves for the different algorithms are shown in Figure 66 AWR performs well across the of challenging skills, consistently achieving compa- rable or better performance than PPO. RWR struggles with controlling the humanoid, but exhibits stronger performance the dog. This performance difference may be due to the more dynamic and acrobatic skills of the humanoid, compared to the more standard locomotion skills of the dog.
5.4 OFF-POLICY LEARNING WITH STATIC DATASETS
Since AWR an off-policy RL algorithm, has the advantage of being able to leverage data from other sources. This not only accelerates the learning process on standard tasks. as dis- cussed above, but also allows us to apply AWR in fully off-policy setting. where the algorithm provided with static dataset of transitions, and then tasked with learning the best possible pol- icy. To evaluate our method in this setting, we use the off-policy tasks proposed by Kumar et al.) (2019). The objective of these tasks is to learn policies solely from static datasets, without collect- ing any additional data from the policy that is being trained. The dataset consists of trajectories T = {(S, ag. ru), , (s1, r1) ....} from rollouts of a demo policy. Unlike standard imitation learn- ing tasks, which only observes the states and actions from the demo policy, the dataset also provides the reward received by the demo policy at each step. The demo policies are trained using SAC on various OpenAl Gym tasks. A dataset of million timesteps is collected for each task.

---

Under review as a conference paper at ICLR 2020
Task
РРO
RWR
AWR (Ours)
Humanoki:
Certeer
Humamoid:
Cartwheel
0.76 + 0.02
0.03 + 0.01
0.78 + 0.07
033
JAL
PED
Humanoid:
Spinkick
0.00
0.05 0.03
0.77 0.04
Dog: Canter
0.76 0.03
0.78 + 0.04
0.86 0.01
Dog: Trot
0.86 + 0.01
0.86 + 0.01
0.86 + 0.03
0.2
Dog: Turn
0.75 + 0.02
0.75 + 0.03
0.82 + 0.03
00
05
1.0
1.5 3.0
0.5
8.0
3.0
85
3.0
20
20
Table 2: Performance statistics of algorithms Figure 6: Learning curves on motion imitation the motion imitation tasks. Returns are normal- tasks. On these challenging tasks, AWR generally ized between the minimum and maximum possi- learns faster than PPO and RWR. ble returns per episode.
Hooperuu?
3750
4008-
2500
4000
2003
3390
3000
2000
500
1003
Figure 7: Performance of various algorithms on off- learning tasks with static datasets. AWR is able to learn policies that are comparable or better than the original demo policies.
For AWR, simply treat the dataset the replay buffer D and directly apply the algorithm with- out additional modifications. Figure 7 compares AWR with other algorithms when applied to the datasets. We include comparisons to the performance of the original demo policy used to gener- ate the dataset (Demo) and a behavioral cloning policy (BC). The comparisons also include recent off-policy methods: batch-constrained Q-learning (BCQ) (Fujimoto et al. 2019) and bootstrapping error accumulation reduction (BEAR) (Kumar et al. 2019), which have shown strong performance on off- -fflcyl learning with static datasets. Note that both of these prior methods are modifications to existing off- off-pollcy RL methods, such as TD3 and SAC, which are already quite complex. In contrast, AWR simple and requires no modifications for the fully off-policy setting. Despite not collecting any additional data, AWR is able to learn effective policies from these fully off-policy datasets, achieving comparable or better performance than the original demo policies. On-policy methods, such as PPO performs poorly in this off-policy setting. Q-function based methods, such as TD3 and SAC, can in principle handle off-policy data but, as discussed in prior work, tend to strug- gle in this setting in practice (Fujimoto et al. 2019 Kumar et al. 2019). Indeed, standard behavioral cloning (BC) often outperforms these standard RL methods. In this fully off-policy setting, AWR can be interpreted an haluantage-wiighier form of behavioral cloning, which assigns higher like- lihoods to demonstration actions that receive higher advantages. Unlike Q-function based methods,
AWR less susceptible issues from dutto--distriuttinn actions as the policy is always trained on observed actions from the behaviour data (Kumar et al.) 2019). AWR also shows comparable perfor- mance to BEAR and BCQ, which are specifically designed for this off-policy setting and introduce considerable algorithmic overhead.
DISCUSSION AND FUTURE WORK
We presented advantage- advantage-weighted regression, simple off-policy reinforcement learning algorithm, where policy updates are performed using standard supervised learning methods. Despite its sim- plicity, our algorithm is able to solve challenging control tasks with complex simulated agents, and achieve competitive performance on standard benchmarks compared to a number of well-established RL algorithms. Our derivation introduces several new design decisions, and our experiments ver- ify the importance of these components. AWR is also able to learn from fully off- datasets, demonstrating comparable performance to state-of-the-ar off-policy methods. While AWR is effec- tive for diverse suite of tasks, it is not yet as sample efficient as the most efficient off-policy algo- rithms. We believe that exploring techniques for improving sample efficiency and performance on fully off-policy learning are also interested in exploring applications that are particularly suitable for these regression- bassed RL algorithms, as compared to other classes of RL techniques. A better theoretical understanding of the convergence properties these algorithms, especially when combined with experience replay, could also be valuable for the development of future algorithms.
open opportunities deploy these methods in real world domains. We
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. International Conference on Learn- ing Representations, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
CMU. Cmu graphics lab motion capture database. http://moccap.es.cm.cdd//
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https:
2017.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, pp. 1329- 1338, 2016. URL http:
//proceedings , ml www.sreesssee. / v48/ /duanllnn html
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2021 2030, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL, ngs ml press wttp///1//11/// html
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: httt:://proceedings mll. www.s/eesssssee. http:/ss//o///uiimooooooooo ht.m
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2052 2062, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URI. U //proceedings mlr Sttreas/y97/Euummmttooo.
html
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. Maria Florina Balean and Kilian Q. Weinberger (eds.), Pro- ceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2829-2838, New York, New York, USA, 20-22 Jun 2016. PMLR.
ml w.w..aa....p.................m.e press 488 /ul//gule html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmssaa, Stockholm Sweden, 10-15 Jul 2018a. PMLR. /RU/httppp/ppoeeenngs .mlr press/ V88C/haarno/ ja 8b. html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmssan, Stockholm Sweden, 10-15 15 Jul 2018b. PMLR. URL/itpp//ppoceeeingg..... v8C 2:00/haanooja j 8b. html
Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiayma, and Jan Peters. Adaptive importance sampling for value function approximation in off off-policy reinforcement learning. Neural Netw., 22(10): 1410, December 2009. ISSN 0893-6080. doi: 10.101 n0.10166).eumeet2000 neunet.2009.000 URL http::/dx...o doi 10g/100.0 101 6/16/3100 neunet. 2009.0000 01.0001.02

---

Under review as a conference paper at ICLR 2020
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAT' 16, pp. 2094-22 2100. AAAI Press, 2016. URL http://cl.ace.
wttp:/iiaaiion....c cfm?id=
301 6100000 D0. 3016191
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. S. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.022886 2017. URL http:
//arza wawxiv.org/aae... 1707. 02286
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining URL http://
improvements in deep reinforcement learning. CoRR, abs/1710.022988, 2017.
arxiv. arg/ass 710. 02298
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp- 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1-55860- 873-7. http://///dl.
org/ citation wfwmid
656005
Jens Kober and Jan R. Peters. Policy search for motor primitives in robotics. In D. Koller, D. Schu- urmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21. pp. 849 856. Curran Associates, Inc., 2009. URL/bttp://paperssa nips. ccc paper/ 3545 powpcylllsllppeeopeap search for-foooooo primitives in robotics.pdf
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. CoRR, abs/1906.00949, 2019. URL
abs/1906. 00949
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. ICLR, 2016.
Nicolas Meuleau, Leonid Peshkin, Leslie P. Kaelbling, and Kee eung Kim. Off-policy policy search. Technical report, 2000.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http::/dxx doi ort/ 100. www111008/nureeeee http//nuuure2222 42366
Rémi Munos, Thomas Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off- policy reinforcement learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 1054-1062, USA, 2016. Curran Associates Inc.
ISBN 978-1-5108-3881-9 URL htttt:p//e/ //ll acm. www.am/mwoooiiiiiiiinoooooooo wfm. wtt.:ooomm..mmmmmmmmmmm.mm 3157214
Ofir Nachum, Mohammad Norouzi, George Tucker, and Dale Schuurmans. Learning gaussian policies from smoothed action value functions, 2018. URL
net/
Corum?id=Blnin 0Z.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference Machine Learning, ICML' 10, pp. 807-814, USA, 2010. Omnipress. ISBN 978-1-60555-997777 URL
http://dl acm. owg/ eeteetittieee aatiintio... cfm?id= cfmmidd=31042222....... 3104425
Gerhard Neumann. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference on International Conference Machine Learning, ICML'II
pp. 817-824, USA, 2011. Omnipress. ISBN 17881-4503-061-55 URL http:// //dp: acm.
org/citat: on www.: 045885
12

---

Under review as a conference paper at ICLR 2020
Gerhard Neumann and Jan R. Peters. Fitted q-iteration by advantage weighted
gression. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Ad-
vances in Neural Information Processing Systems 21, pp. 1177-1184. Cur- ran Associates, Inc., 2009.
URL. http:
- -iwtt-------eeeeeeeeeeee-------- - hww.:a/wow..bbaaaaaao-eeeeeaaaaa
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learn- ing, volume 80 of Proceedings of Machine Learning Research, pp. 3878-38888888 3887, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL htt.httppp//rrreeeiing...oo vBC/ ht..18..tmm....
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37
(4):143:1-143:14, July 2018, ISSN 0730-0301. doi: 10.1145/3195517.2200 //doi 1ttpr///1.1111/1111111 .201311
http:
Jan Peters and Stefan Schaal. Reinforcement learning by reward- reward-weighted regression for operational space control. In Proceedings of the 24th International Conference Machine Learning. ICML '07, pp. 745-750, New York, NY, USA, 2007. ACM. ISBN 978-1-595933-93333, doi: 10. 144/ 1273496. 1273990. URLhttp::/ acm. http:///.0..aa..oooo001111111111 0ttpg//111111//22345555500 12 1273590
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of
the Twenty-Fourth AAAI Conference on Artificial Intelligence, AA pp. 1607-1612. AAAI
Press, 2010. URL http:://dl..cc..rrr//aaaaaatt.od
(222288867772288663 2898863,
Vitchyr Pong. Rlkit. http:///gthbbb.cm///ttccy//y///
2019.
Doina Precup. Richard S. Sutton, and Sanjoy Dasgupta. Off-policy temporal difference learning with function approximation. In Proceedings of the Eighteenth International Conference on Machine Learning. ICML '01, pp. 417-424, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN --55860-778-1, URL
ott.://a...aaor////aa.o///// http://a////iiiiii/ii/iiii cfm?id=645530.
655817
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, mir..
pp. 11888-88880 Lille, France, 07-09 Jul 2015. PMLR. URL press v 7 /ttess/w3///chaaaaam155
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 076447777777 2017. URL http::/arxiv/ orgg abs/
1 1170.00 06347
SFU. Sfu motion capture database. http://mocapp.cs.stuu
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Richard Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. A. Solla, T. K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems 12, pp. 1057-1063, MIT Press, 2000.
13

---

Under review as a conference paper at ICLR 2020
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6288-6297. Curran Associates, Inc., 2018.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. CoRR, abs/1611.01222 2016.
Ronald J. Williams. Simple statistical gradient- algorithms for connectionist reinforce- ment learning. Mach. Leurn., 8(3-4):229-256, May 1992. ISSN 0885-6125. BF00992696. URLhttps. /tt:::/doooooooo wttp:oo..rg.....0 0tt.00100700000966555500
doi: 10.1007/
He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Trans. Graph,, 37(4):45:1-145:11, July 2018. ISSN
0730-0301. doi: 10. 10.114//197777.32211 0169517.3001365
URL http://dot: acm. acm.org/10.1145/
14

---

Under review as a conference paper at ICLR 2020
A AWR DERIVATION
In this section, we derive the AWR algorithm as an approximate optimization of a constrained policy search problem. Our goal is find a policy that maximize the expected improvement n(T)) ( (()) over a sampling policy p(aas). We start with a lemma from Kakade & Langford 2002). which shows that the expected improvement can be expressed in terms of the advantage (s, a) = R V"(s) with respect to the sampling policy μ. where R denotes the return obtained by performing action a in state s and following p for the following timesteps, and V*(s) = Sa ((a|s) R da corresponds the value function of μ.
E (T)
Σ
7a AM (s:, ar)
(17)
=Rrops(r)
? ar) + (
(18)
2
(r)
VS0))) + Σ> 2r((s,, an)
(19)
00
Eoo-p(so) [V*(s))] + (x
Σ
( ax)
(20)
[() (
(21)
We can rewrite Equation with an expectation over states instead of trajectories:
2C
(Tx) (ympx(r)
Σ AM a.)
(22)
p(s: s/ / T(als)>* A* a) da ds
(23)
''pss: = -|/T) / T(als) a) da ds
(24)
1=0
(s)
T(ass) RRia - V*()) da ds,
(25)
where (s) = Er pp(s
represents the unnormalized discounted state distribution induced by the policy (Sutton & Barto 1998), and p(s, = s/ is the likelihood of the agent being
in state s after following т for timesteps.
The objective in Equation can be difficuult to optimize due to the dependency between dx (s) and t, as well as the need to collect samples from æ. Following Schulman al. (2015), we can optimize an approximation (T)) of (T)) using the state distribution of μs,
(()) = (du(s)) /
(als) R - V*()) da ds.
(26)
(())) matches 7 to first order (Kakade & Langford 2002). and provides a reasonable estimate of if π and μ are similar. Using this objective, we can formulate the following constrained policy search problem:
((his)) / r(als) [RM, - VP(s)] VV'ss] da ds arg max
(27)
s.t. DKL (T((()))((((ss)) Sc, Vs
(28)
ralss) da 1, Vs.
(29)
Since enforcing the pointwise KL constraint in Equation 28 at all states is intractable, we relax the constraint by enforcing it only in expectation f d, (s) DRL. (=(()))](u(((s)) ds < To further
15

---

Under review as a conference paper at ICLR 2020
simplify the optimization problem, relax the hard KL constraint by converting it into soft constraint with coefficient B.
([dui())
/ r(als) [R VV(s)] da ds B (c-
+3((- dx(s)DKL (=(((s)llu((ss) ds
arg max
/ T(als) da 1, ss.
s.t.
(30)
Next we form the Lagrangian,
[(T,, 3, = {f d=(s) i(als) R -V((s) da ds + (c- du (s (T(-1s)) (-+ss) ds
ff000 as 1 -
daals)da)
ds,
(31)
with 3 and a = fas Vs E S) corresponding to the Lagrange multipliers. Differentiating L(TT, 3. with respect to (als)) results in
Oc (s) (R.. - V*(s)) - 3 ds) logn(als) - Bdy (s) - -ddp(s) Css.
(a(a))
(32)
Setting to zero and solving for (aas)) gives
1 (R V*(s)) + logu(als)) -1
1
loga(als)
(33)
d,(ss)
n (als)) exxp () ( VV(()) exp
1 0g -11
(34)
dy,(s)
3
Since fg (a|s)) da 1, the second exponential term is the partition function ((s) that normalizes the conditional action distribution.
((s) exp
Os +
+1) =
=/ ((a'(s)) exp (3) R vi((s) da'.
(35)
ds
8
The optimal policy is therefore given by,
1 ा Ta(ass) = Z(s) u(als) exp (1 (Rea- VV(s))
(36)
If is represented by function approximator, the optimal policy л² be projected onto the manifold of parameterized policies by solving the following supervised regression problem
arg min (g)dd [DrL. (x (==((|s[[=((|ss)))
(37)
arg min
Band,(s) DKL
1 Z(s) u(als) exp
3 (Rh. VV*(s))
1
T(-1s)
(38)
arg Egdd, (s) (anp(als)) (an log n (als) exp
3 (R! -V'())
(39)
π
B AWR DERIVATION WITH EXPERIENCE REPLAY
In this section, we extend the derivation presented Appendix to incorporate experience replay using replay buffer containing data from previous policies. To recap, the sampling distribution is mixture past policies (Iı,..., ) where the mixture performed at the trajectory level. First, we define the trajectory distribution (u(T), marginal state-action distribution jus, a), and marginal state distribution (s) of the replay buffer according to:
h
(AT) = Σ
wid (T),
((s, =
Σ widn (a)s), ds(ss) = Σ w.d., (s)
(40)
1=1
16

---

Under review as a conference paper at ICLR 2020
where the weights Σ, W/ specify the probabilities of selecting each policy ny. The conditional action distribution ())s)) induced by the replay buffer is given by:
u(s, a) d
E Li=1 w.d.,, iT
((aas)
(41)
wyddy (s)
Next, using Lemma 6.1 from Kakade & Langlord (0022) (also derived in Appendix A). the expected
improvement of æ over each policy n; satishie
J(x) (T.) + Bgwd, (s) [AT a)]
(42)
The expected improvement over the mixture can then expressed with respect to the individual policies.
<()) = J(z) - J(1)
(43)
&
= J(x) Σ 1=1
wiJ(mi)
(44)
wy (J(m) J(
(45)
Σ Wi (Rand [A* (s,
(46)
In order to ensure that the policy is similar to the past policies, we constrain against the condi- tional action distributions the replay buffer,
Es-n(s) [DKL (T(als) [(aals) (r(aass
S.
(47)
Note that constraining against p(a's) has a number of desirable properties. First, the con- straint prevents the policy π from choosing actions that are vastly different from all of the policies {=1.-.. ) Second, the mixture weight assigned each π. in the definition of μ depends on the marginal state density dx, (s) for the particular policy. This property is desirable as the policy π is now constrained be similar π. only at states that are likely be visited by ,;. This then yields the following constrained objective:
max a
Σ Ws (s) [R - (s)
(48)
1=1
s.t.
(g)d DRL (T( s))] Sc
(49)
π r(als) da = 1, Vs..
(50)
The Lagrangian of the above objective is given by:
L(T, 3, a)) =
Σ wi Bewdd, (8) R. Bana(al))
[ - [T](s]]
Σ widr. (8) (-1s)
+ 3
6 6 (e) DKL.
T(-1s)
(51)
1== wydr, (s)
+ { (xs 1 (1-
((als)da) ds.
Solving the Lagrangian following the same procedure as Appendix A leads to an optimal policy of the following form:
Ei w.d,, (s) (R V vs)
ा (a|s)
2(s) (aass) exp
(52)
3
E, Wyddr
17

---

Under review as a conference paper at ICLR 2020
Finally, if is represented by function approximator, the optimal policy 'T" can be projected onto the manifold of parameterized policies by solving the following supervised regression problem
arg min
Bg M, (5) [DEL (* (-|s)|==((ss))
(53)
1 E. widz (s) (R ( Σ, wydn, (s)
(5))
arg min
(g) DEL
(laas)) exp
<-|ss)
Z(s)
B
x
(54)
One of the challenges of optimizing the objective in Equation 54 is that computing the expected return in the exponent requires rolling out multiple policies starting from the same state, which would require the environment to be resettable any given state. Therefore, obtain a more practical objective, we approximate the expected return across policies using single rollout from the replay buffer,
E, widhr (s)R Ej wjd (s)
= R2 such that (s, a) E D
(55)
This single-sample estimator results in biased estimate of the exponentiated advantage, because the expectation with respect to the mixture weights appears in the exponent. But in practice, we find this biased estimator to be effective for our experiments. Therefore, the objective used in practice is given by:
1 3
E, wydn, (s) E, wjdr (s)
arg max
Σ W/ Bennd, (8) E. Earxx (als) log n (als) exp
RT
(56)
where the expectations can be approximated by simply sampling from D following Line 6 of Algo- rithm Note, the baseline in the exponent different policies. One approach for estimating this quantity would be to fit separate value functions * for each policy. However, if only a small amount of data is available from each policy, then x could be highly inaccurate. Therefore, instead of learning separate value functions, we fit a single mean value function (s) that directly estimates the weighted average of V*
consists of an average of the value functions of the
V = arg min Σ W/s Ex (5) (s) (alss [[[RT - V(s)
(57)
This loss can also be approximated by simply sampling from the replay buffer following Line 5 of Algorithm The optimal solution (s)) Σ, wide, (s)) is exactly the baseline in Equation 56
2, w,de,
C EXPERIMENTAL SETUP
In our experiments, the policy is represented by a fully-connected network with 2 hidden layers consisting of 128 and 64 ReLU units respectively (Nair & Hinton 2010). followed by a linear output layer. The value function is modeled by separate network with a similar architecture, but consists of a single linear output unit for the value. Stochastic gradient descent with momentum is used to update both the policy and value function. The stepsize of the policy and value function are 5x 10-5 and 10- respectively, and momentum 0.9 is used for both. The temperature is set to 8 = 0.05 for all experiments, and À 0.95 is used for TD(A). The weight clipping threshold Wmax is set to 20. At each iteration, the agent collects a batch of approximately 2000 samples, which stored in the replay buffer D along with samples from previous iterations. The replay buffer stores 50k of the most recent samples. Updates to the value function and policy are performed by uniformly sampling minibatches of 256 samples from D. The value function is updated with 200 gradient steps per iteration, and the policy is updated with 1000 gradient steps.
18

---

Under review as a conference paper at ICLR 2020
D SIMILARITIES TO POLICY GRADIENTS
On the surface, the AWR policy update bears striking similarities to a conventional policy gradient
(PG) update (Sutton al. 2000
Egndy (s) (nls)
lo rtaas) (R -V((s)]
(Policy Gradient)
V log m(ass) exp (Rha- -V*(s))
(s) Banulals)
(AWR)
However, there a number of subtle but important differences between the two. First, basic policy gradient algorithms are on-policy methods, which requires the data to be sampled from the same policy that is being optimized
(s) and a =aaas), whereas AWR can in principle learn ing data from any sampling distribution This requirement for policy gradient methods is because PG directly differentiates through the sampling distribution compute the gradient of the expected return with respect to the policy parameters. But with AWR and other EM algorithms, they first construct estimate of the optimal action distribution each state, and then projects that action distribution onto the space of parameterized policies. Therefore AWR does not need to differentiate through the sampling distribution, which is a critical feature for settings such batch RL, where the sampling distribution (e.g. demo policy) may not be available to the agent. In AWR, the log prob-
ability of an action lals(a)s) is weighted by the exponentiated advantage exp ( (Re,a - V(s)))
while PG the log probability weighted just by the advantage (Rs,a - V(s) without the expo- nential. Since the exponentiated advantage is non-negative, the objective used in the AWR update a maximum likelihood objective that tries to maximize the likelihood of all actions, but to vary- ing amounts depending both positive and negative, therefore PG updates decrease the likelihood of actions with negative advantages, and thus it is not conventional maximum likelihood objective. In practice, negative TD updates are often a source of instability when applying PG to off- oof-po data.
the exponentiated advantage. In the case of PG, the advantage can be
19

---

Under review as a conference paper at ICLR 2020
E ADDITIONAL EXPERIMENTS
A comprehensive comparison of AWR with prior methods on all of the tasks considered are available
Figure
HallCheetah--2
Hopper-u2
4000
.0000
6000
5000
8000
500n
4000
6000
2000
3300
4000
2000
1000
2000
0.0
0.5
1.0
2.5
0.0
0.5
1.0
2.0
0.0
0.2
0.4
0.6
0.3
1.0
Samples
Humanoid-v2
LunerLander-xe
Walker26-42
6000
100-
4000
1000
3000
2000
100
1000
0.0
0.5
2.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.5
2.0
187
187
167
TRPO
PPO
DDPG
TD3
SAC
RWR
AWR (Ours)
Figure 8: Learning
of the various algorithms when applied to OpenAl Gym tasks. Results
are averaged over 5 random seeds. AWR is generally competitive with the best current methods.
Humanoidi: Spinkick
Humanold: Cartwheel
Dog:
1.3
1.0
FPO
0.8
RWI
0.0
0.8
0.5
0.0
0.5
1.0
2.0
3.0
3.5
0.0
0.5
1.0
2.0
3.0
3.5
0.0
0.5
1.0
2.0
2.5
3.0
3.5
Samples
167
167
Dog: Trot
Dog: Turn
1.0
0.8
0.8
0.6
0.6
0.4
04
0.0
1.0
1.5
2.0
2.5
3.0
3.3
4.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
4.0
lc7
107
Figure 9: Learning curves motion imitation tasks. On these challenging tasks, AWR generally learns faster than PPO and RWR.
E. WEIGHT CLIPPING
To analyze the effects of weight clipping on the stability of AWR, we compare learning curves of policies trained with weight clipping using threshold of Wmax 20, and policies trained without
20

---

Under review as a conference paper at ICLR 2020
weight clipping. Figure 10 compares the learning curves with and without clipping. 5 separate AWR runs with different random seeds are visualized separately. With weight clipping, performance re- mains stable throughout training. Policies trained without weight clipping substantially more unstable, exhibiting drastic fluctuations performance as a result of exploding gradients from ex- cessively large weights. Some training runs without clipping terminated early due exploding gradients causing the networks to output NaNs. These experiments suggest that weight clipping is vital for ensuring stable training with AWR.
Ho "Chestah-22
Walker2div2
7000
8000
6000
4000
2000
2000
2000
Cllpping
AWN Clipping
1.0
Figure 10: Learning curves comparing AWR policies trained with and without weight clipping. Weight clipping is vital for ensuring stable training with AWR. Policies trained without weight clipping are susceptible to exploding gradients due to excessively large weights.
21


Reviews:
[{"review_id": "rylVAPJCFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThe paper shows that good old reward weighted regression (RWR) with value-function baseline is still state-of-the-art algorithm.\n\n# Decision\nThe paper is well-written and provides many evaluations. The contribution should be articulated more carefully, though, taking into account that most algorithmic ideas are present in prior work (https://openreview.net/forum?id=H1gdF34FvS&noteId=Bkxi11nsdr). Perhaps, the experience replay part is somewhat novel. It seems emphasizing more that the aim is to show that simple methods are competitive rather than focusing on novelty could be a good idea.\n\nProvided that the authors incorporate the feedback of the other reviewers and update the paper accordingly, it will make a good contribution.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxTjQo6FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an off-policy reinforcement learning method in which the model parameters have been updated using the regression style loss function. Specifically, this method uses two regression update steps: one update value function and another one update policy using weighted regression. To compare the proposed method with others [main comprasion], 6 MuJoCo tasks are used for continuous control and LunarLander-v2 for discrete space. \n\n-- Even though this paper has done a good job in terms of running different experiments, the selection of some of the benchmarks seems arbitrary. For example, for discrete action space, this paper uses LunarLander which is rarely used in any papers so it makes very difficult to draw a conclusion based on these results. Common 49 Atari-2600 games should have been used for comparison. The same thing about experiments in section 5.3 is true too as those tasks are not that well-known. \n\n-- The proposed method doesn't outperform previous off-policy methods on Mujoco task (Table 1). Since the main claim of this paper is a new off-policy method, outperforming the previous off-policy methods is a fair game. The current results are not convincing enough.\n\n-- There are significant overlaps between this paper, \"Fitted Q-iteration by Advantage Weighted Regression\", \"Model-Free Preference-Based Reinforcement Learning \", and \"Reinforcement learning by reward-weighted regression for operational space control\" which makes the contribution of this paper very incremental.\n\n-- The authors used only 5 seeds to run Mujoco experiments. Given the sensitivity of Mujoco for different starting points, the experiments should have been run at least with 10 different seeds.\n\nQuestions:\n1) Shouldn't be an importance sampling ratio between \\pi and \\mu in the equations? starting from eq.5. \n2) Does the algorithm optimize the respect to $w$ as well? (eq. 15) if yes, why it is not mentioned in algorithm 1? Plus, since $d(s)$ is uniform dist. (at least this is assumed for implementation), eq. 14,15,and 16 (wherever there is d(s)), those can be simplified, e.g. \\hat{V} = \\sum(w_i V_i), wouldn't be better just introduced simplified version rather than current ones? (referring to the only equation above section 3.3)\n3) Is this the same code used to report results in this paper? if yes, I didn't see any seed assignment in the code?! and what is \"action_std\" in the code?\n\nThere are a couple of recent works in merging on-policy with off-policy updates which you might want to cite them. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl2SGsuur", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Note: I wrote this review after John Schulman's first comment, before any reply, and before Gehrard Neumann's comment]\n\nThe authors propose an actor-critic algorithm based mostly on regression. Being off-policy, the algorithm can learn from multiple policies. It can also be applied to continuous as well as to discrete actions, and it can be trained in a batch RL setting. They compare it to a set of state-of-the-art algorithms in standard openAI gym continuous action benchmarks and show competitive performance despite a much simpler implementation of the algorithm.\n\nI basically subscribe to John Schulman's comment below, both about empirical results and about citing Self-Imitation Learning, but I have a stronger point against the paper, which is insufficient positionning with respect to the relevant literature.\n\nThe paper does not cite or discuss the one below, though it looks VERY close:\n\n@inproceedings{neumann2009fitted,\n  title={Fitted Q-iteration by advantage weighted regression},\n  author={Neumann, Gerhard and Peters, Jan R},\n  booktitle={Advances in neural information processing systems},\n  pages={1177--1184},\n  year={2009}\n}\n\nThis paper also starts from RWR and performs weighted regression based on the advantage rather than on the return. So to me it is exactly the same idea, and it is mandatory that the authors clearly establish what is the novelty of their work with respect to this previous paper.\n\nLess importantly, the authors may also want to have a look at :\n\n@article{zimmer2019exploiting,\n  title={Exploiting the sign of the advantage function to learn deterministic policies in continuous domains},\n  author={Zimmer, Matthieu and Weng, Paul},\n  journal={arXiv preprint arXiv:1906.04556},\n  year={2019}\n}\n\nwhich also uses ideas along the same line.\n\nTo me, a good way to improve the novelty of this work would be to perform a detailed empirical study of the inner mechanisms of the algorithm on very simple benchmarks where the value function and policy could be visualized. In particular, how stable is the estimation of the value function? This is known to be an issue, as most algorithms avoid approximating it and prefer estimating the Q-function.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper caused a lot of discussions before and after the rebuttal. The concerns are related to the novelty of this paper, which seems to be relatively limited. Since we do not have a champion among positive reviewers, and the overall score is not high enough, I cannot recommend its acceptance at this stage.

Decision: Reject