Q-Value Weighted Regression: Reinforcement Learning with Limited Data
Authors: Piotr Kozakowski
Lukasz Kaiser
Henryk Michalewski
Afroz Mohiuddin
Katarzyna Kańska
Venue:  (2021)
Keywords: reinforcement learning
rl
offline rl
continuous control
atari
sample efficiency

Abstract: Sample efficiency and performance in the offline setting have emerged as among the main
challenges of deep reinforcement learning. We introduce Q-Value Weighted Regression (QWR),
a simple RL algorithm that excels in these aspects.

QWR is an extension of Advantage Weighted Regression (AWR), an off-policy actor-critic algorithm
that performs very well on continuous control tasks, also in the offline setting, but struggles
on tasks with discrete actions and in sample efficiency. We perform a theoretical analysis
of AWR that explains its shortcomings and use the insights to motivate QWR theoretically.

We show experimentally that QWR matches state-of-the-art algorithms both on tasks with
continuous and discrete actions. We study the main hyperparameters of QWR
and find that it is stable in a wide range of their choices and on different tasks.
In particular, QWR yields results on par with SAC on the MuJoCo suite and - with
the same set of hyperparameters -- yields results on par with a highly tuned Rainbow
implementation on a set of Atari games. We also verify that QWR performs well in the
offline RL setting, making it a compelling choice for reinforcement learning in domains
with limited data.

Under review as a conference paper at ICLR 2021
Q- VALUE WEIGHTED REGRESSION: REINFORCEMENT LEARNING WITH LIMITED DATA
Anonymous authors Paper under double-bling review
ABSTRACT
Sample efficiency and performance in the offline setting have emerged as among the main challenges of deep reinforcement learning. We introduce Q-Value Weighted Regression (QWR), a simple RL algorithm that excels in these aspects. QWR is extension of Advantage Weighted Regression (AWR), an off-policy actor-critic algorithm that performs very well on continuous control tasks, also in the offline setting, but struggles on tasks with discrete actions and in sample efficiency. We perform theoretical analysis of AWR that explains its shortcomings and use the insights to motivate QWR theoretically. We show experimentally that QWR matches wrate-of-the-ad algorithms both on tasks with continuous and discrete actions. We study the main hyperparameters of QWR and find that it is stable in a wide range of their choices and different tasks. In particular, QWR yields results on par with SAC on the MuJoCo suite and with the same set of hyperparameters yields results on par with highly tuned Rainbow implementation games. We also verify that QWR performs well in the offine RL setting, making it compelling choice for reinforcement learning in domains with limited data.
a set of Atari
INTRODUCTION
Deep reinforcement learning has been applied to large number of challenging tasks, from games
(iiverr et al.) 2017 OpenAI 20188 Vinyals al.) (017)) robotic control (Sadeghi & Levine 2016] OpenAl et al.) 2018 Rusu et al. (016). Since RI. makes minimal assumptions on the underlying task, holds the promise of automating a wide range of applications. However, its widespread adoption has been hampered by a number of challenges. Reinforcement learning algorithms can be substantially more complex to implement and tune than standard supervised learning methods and can have fair number of hyper-parameters and be brittle with respect to their choices, and may require a large number of interactions with the environment.
These issues well- knownn and there has been significant progress in addressing them. The policy gradient algorithm REINFORCE (Williams (1992)) is simple to understand and implement, but both brittle and requires on-policy data. Proximal Policy Optimization (PPO, /Schulman al. (2017) is a more stable on-policy algorithm that has seen number of successful applications despite requiring large number of interactions with the environment. Soft Actor-Critic (SAC, HHarnoja et al. (2018)) is a much more sample-efficient off-policy algorithm, but it is defined only for continuous action spaces and does not work well in the offline setting, known as batch reinforcement learning, where all samples are provided from earlier interactions with the environment, and the agent cannot collect more samples. Advantage Weighted Regression (AWR, Peng et al. (2019) is recent off- policy actor-critic algorithm that works well in the offline setting and is built using only simple and convergent maximum likelihood loss functions, making it easier to tune and debug. It competitive with SAC given enough time to train, but is less sample-efficient and has not been demonstrated to succeed in settings with discrete actions.
We replace the value function critic of AWR with Q-value function. Next, we add action sampling to the actor training loop. Finally, we introduce a custom backup to the Q-value training. The resulting algorithm, which we call Value Weighted Regression (QWR) inherits the advantages of AWR but is more sample-efficien and works well with discrete actions and in visual domains, e.g., on Atari games.

---

Under review as a conference paper at ICLR 2021
To better understand QWR perform number of ablations, checking different number of samples in actor training, different advantage estimators, and aggregation functions. These choices affect the performance of QWR only limited extent and it remains stable with each of the choices across the tasks we experiment with.
We experiments with QWR the MuloCo environments and subset of the Atari Learning Environment. Since sample efficiency is our main concern, we focus on the difficult case when the number of interactions with the environment is limited in most our experiments we limit 100K interactions. The experiments demonstrate that QWR is indeed more sample-efficient than AWR. On MuJoCo, it performs on par with Soft aetor-Critic (SAC), the current state-of-the-ar algorithm for continuous domains. On Atari, QWR performs par with OTRainbow, a variant of Rainbow highly tuned for sample efficiency. Notably, we use the same set of hyperparameters (except for the network architecture) for both our final MuloCo and Atari experiments.
2
Q VALUE WEIGHTED REGRESSION
ADVANTAGE WEIGHTED REGRESSION
Peng et al. (2019) recently proposed Advantage Weighted Regression (AWR), off-policy, actor- critic algorithm notable for its simplicity and stability, achieving competitive results across range of continuous control tasks. It can be expressed interleaving data collection and two regression tasks performed on the replay buffer, as shown in Algorithm
Algorithm Advantage Weighted Regression.
<- random actor parameters
2: Φ <- random critic parameters
3: D <- 0
4: for in O..n _in__itrattions
do 5: add trajectories {r.} sampled by Ta to D
6: for i in O..n O..n_critic stepssseppsssp do
7: sample (s,a) D
8: à - ay V [|| [=
9: end for
10: for in 0..n_actor sttrrrstepp_t do
11: sample (s, a) ~D
12:
0 Vo ex (R Vo(s))
13:
end for
14:
end for
AWR optimizes expected improvement an actor policy π ([ass) over sampling policy p(a s) by regression towards the well-performing actions in the collected experience. Improvement is achieved by weighting the actor loss by exponentiated advantage (s, a) an action, skewing the regression towards the better-perfonee Rza achieved by performing action a in state s and then following the sampling policy μ. To calculate the advantage, one first estimates the value, (s), using a learned critic and then computes Ay a) Ra.a (s). This results in the following formula for the actor:
actions. The advantage is calculated based on the expected return
arg max Exwd, (s) 1s))(s))) log m (als) exp
(R v2(ss)
where d,, De t= 1 p(s. s/m) denotes the unnormalized, discounted state visitation distri- bution of the policy (u, and temperature hyperparameter.
The critic is trained to estimate the future returns of the sampling policy μn:
arg min (a) is) [|] (ss)
To achieve off oof-policy learning, the actor and the critic are trained on data collected from a mixture of policies from different training iterations, stored in the replay buffer D.
2

---

Under review as a conference paper at ICLR 2021
2.2 ANALYSIS OF AWR WITH LIMITED DATA
While AWR achieves very good results after longer training, it is not very sample efficient, as noted in the future work section of (Pegg al. 2019). To understand this problem, we analyze a single loop of actor training in AWR under a special assumption.
The assumption we introduce, called state-determines-aaaee aactionn concerns the content of the replay buffer D of an off- off-py RL algorithm. The replay buffer contains all state-action pairs that the algorithm has visited far during its interactions with the environment. We say that replay buffer 'D satisfies the state siite-determnns-accee
assumption when for each state in the buffer, there is unique action that was taken from it, formally:
for all (s, a). (s',a') ED::::
a a=d.
This assumption may seem very limiting and indeed is not true in many artificial experiments with RL algorithms,, such as Atari games. Even random policy starting from the same state could violate the assumption the second time it collects a trajectory. But note that this assumption is almost always satisfied in real-world experiments with high-dimensional state spaces as any amount of noise added to a high-dimensional space will make repeating the exact same state highly improbable. For example, consider robot observing 32x32 pixel images. To repeat state, each of the 1024 pixels would have to have exactly the same value, which is highly improbable even with a small amount of noise. This assumption also holds in cases with limited data. When only few trajectories collected in large state space, it is unlikely a state will be repeated in the replay buffer, which makes the assumption relevant to the study of sample efficiency.
How does AWR perform under the state-determines-actioo assumption? In Theorem T formulated and proven in Appendix [A.]] we show that the AWR update rule under this assumption will converge to a policy that assigns probability to the actions already present in the replay buffer, thus cloning the previous behaviors. This is not the desired behavior, as an agent should consider various actions from each state, to ensure exploration.
The pate-determines-action assumption is the main motivating point behind QWR, whose theoretical properties are proven in Theorem 2 in Appendix [A.3] We now illustrate the importance of this assumption by creating a simple environment in which holds with high probability. We verify experimentally that AWR fails on this simple environment, while QWR is capable of solving it.
The environment, which we call BitFlip, is parameterized by an integer N. The state of the environ- ment consists of N bits and a step counter. The action space consists of N actions. When an action i chosen, the i-th bit is flipped and the step counter incremented. A game BitFlip starts in a random state with the step counter set to 0, and proceeds for steps. The initial state is randomized in such a way to always leave at least bits set to At each step, the reward is +1 a bit was flipped from 0 to 1 and the reward is -1 in the opposite case.
Since BitFlip starts in one random state out of 2N, at large enough N it is highly unlikely that the starting state will ever be repeated in the replay buffer. As the initial policy is random and BitFlip maintains a step counter to prevent returning to a state, the same holds for subsequent states.
BitFlip is simple game with very simple strategy, but the initial replay buffer will satisfy the state- determines-acteo assumption with high probability. As we will see, this is enough to break AWR.
CNNR
Number
Figure 1: AWR and QWR the BitFlip environment. The maximum possible return is 5.
3

---

Under review as a conference paper at ICLR 2021
We ran both AWR and QWR BitFlip for different values of N, for 10 iterations per experiment. In each iteration we collected 1000 interactions with the environment and trained both the actor and the critic for 300 steps. All shared hyperparameters of AWR and QWR were set to the same values, and the backup operator in QWR was set to mean. We report the mean out of 10 episodes played by the trained agent. The results are shown in Figure
As we can see, the performance of AWR starts deteriorating at a relatively small value of N = 8, which corresponds to state space with 5 5-2 1280 states, while QWR maintains high performance even at N 30, so around 5 10 states. Notice how the returns of AWR drop with N at higher values: 20 30, the agent struggles to flip even a single zero bit. This problem with AWR and large state spaces motivates us to introduce QWR next.
2.3 V-VALEE WEIGHTED REGRESSION
To remedy the issue indicated by Theorem T we introduce a mechanism to consider multiple different actions that can be taken from single state. We calculate the advantage of the sampling policy μ based learned Q-function: A₂ (s, a) (s a) (s), where (s) is the expected return of the policy μa, expressed using 2, by expectation over actions: (s) En-p(-1s) Qq. a). We
substitute our advantage estimator into the AWR actor formula (Equation 1) to obtain the QWR actor:
log (aass) exp ( -
arg max
(3)
Similar to AWR, we implement the expectation over states in Equation by sampling from the replay buffer. However, to estimate the expectation over actions, we average over multiple actions sampled from j during training. Because the replay buffer contains data from multiple different sampling policies, we store the parameters of the sampling policy μ(aas) conditioned on the current state in the replay buffer and restore in each training step to compute the loss. This allows to consider multiple different possible actions for a single state when training the actor, not only the one performed in the collected experience.
The use of a Q-network as a critic provides us with an additional benefit. Instead of regressing it towards the returns of our sampling policy μr, we can train estimate the returns of an improved policy μ², in a manner similar to Q-learning. This allows us to optimize expected improvement over μ², providing better baseline as long as Ea (c]s) Q (s, a) Ea-pp(1s) Qu (s, a), the policy improvement theorem for stochastic policies (Sutton & Barto 2018 Section 4.2) implies that the policy μ² achieves higher returns than the sampling policy μc:
Bany" (-1a) (s, 2 (s) V (s) 2 Vs((s)
(4)
4r* need not be parametric in fact, it not materialized in any way over the course of the algorithm. The only requirement is that we can estimate the Q backup Ea-ys( Q(s, a). This allows great flexibility in choosing the form of n² Since we want our method to work also continuous action spaces, we cannot compute the backup exactly. Instead, we estimate it based on several samples from the sampling policy μ. Our backup has the form E,, -un(-... F( ([{((s, a,), ..., Q(s, ax))). }). In this work, we extend the term Q-learning to mean training a Q-value using such generalized backup. To make training of the Q-network more efficient, we use multi-step targets, described in detail in Appendix A.4 The critic optimization objective using single-step targets is:
arg min E. Egnd, (s) Ea Ba-nials))))))) En Bo ( [lQ -
Q
where Q* = r(s,a) - (s. Q₂ (s. )
(5)
and T(s' as) is the environment's 's transition operator.
In this work, we investigate three choices of F: average, yielding p μi max, where μ² approx- imates the greedy policy; and log-sum-exp. F(X) log 지 Exex exp(x/r) interpolating between average and max with the temperature parameter т. This leads to three versions of the QWR algorithm: QWR-AVG, QWR-MAX, and QWR-I-SE. The last operator, log-sum-exp, is similar the
4

---

Under review as a conference paper at ICLR 2021
backup operator used in cestimum--ntropp reinforcement learning (see e.g. Haarnoja et al. (2018) and can be thought of as a soft goftgg backup, rewarding both high returns and uncertainty of the policy. It is our default choice and the final algorithm is shown in Algorithm 2
Algorithm 2 Q-Value Weighted Regression.
1: 0 <- random actor parameters
2: Ø + random critic parameters
3: D -
4: for k in _..__iteraiion - do
5: add trajectories { sampled by my D
6:
ti 4
7:
for in O..m sem_erilie_stee
do if i mod pudale_rrqqueney = 0 then
8:
9:
Φ: 6 end if
10:
11:
sample (s, 11 a,,) D
12:
sample a'o, .... a' n-1 u(-(s')
13:
Q' t r+ +f (s', an, o),... 2s a'',)'nn1)))
14:
a t -- av V [1Q - (s,
end for
16:
for i in o..n_actor _cctrrsttesssssss_______s 1 do
sample (s, ,1 ...). ~
18:
sample ao. .... An-1 (u-ls)
19:
V V Do (o(s, a,)
0 a Vo Σド=d log=o(ay)s)) cxpp m)]
20:
21: end for
22:
end for
3 RELATED WORK
Reinforcement learning algorithms. Recent years have seen great advances in the field of rein- forcement learning due to the use of deep neural networks function approximators. Mnih al.) (2013b) introduced DQN, an off -olicy algorithm learning a parametrized Q Q-value function through updates based on the Bellman equation. The DQN algorithm only computes the Q-value function, does not learn explicit policy. In contrast, policy-based methods such as REINFORCE (Williams) 1992) learn a parameterized policy, typically by following the policy gradient (Sutton et al., 1999) estimated through Monte Carlo approximation of future returns. Such methods suffer from high variance, causing low sample efficiency. Actor-critic algorithms, such as A2C and A3C (Sutton et al. 2000 Mnih et al. 2016), decrease the variance of the estimate by jointly learning policy and value functions, and using the latter as an action- indien-eddeeedennt baseline for calculation the policy gradient. The PPO algorithm (Schulman al. (017)) optimizes clipped surrogate objective in order allow multiple updates using the same sampled data.
Continuous control. Lillicrap et al. (2015) adapted Q-learning continuous action spaces. In ad- dition to Q-value function, they learn a deterministic policy function optimized by backpropagating the gradient through the Q-value function. HHarnoja et al. (2018) introduce Soft Actor- Aitor-Critic (SAC): a method learning in a similar way, but with stochastic policy optimizing the Maximum Entropy RL (Levine, 2018) objective. Similarly to our method, SAC also samples from the policy during training.
Advantage-weighted regression. The QWR algorithm is a successor of AWR proposed by Peng ctt al. 2011),, which in turn is based on Reward- Weighted Regression (RWR, Peters & Schaal (2007) and AC-RÉPS proposed by Wirth et al. (2016). Mathematical and algorithmical foundations of ad advantage-vu wantage-weeihttdd regression were developed by Neumann & Peters (2099). The algorithms share the same good theoretical properties: RWR, AC-REPS, AWR, and QWR losses be mathematically reformulated in terms KL-divergence with respect to the optimal policy (see formulas (7)-(10) in Peng et al. (2019)). QWR is different from AWR in the following key aspects: instead of empirical
5

---

Under review as a conference paper at ICLR 2021
returns in the advantage estimation we train Q function (see formulas T and 3 below for precise definition) and use sampling for the actor. QWR is different from AC-REPS as it uses deep learning
for function approximation and Q-learning for fitting the critic, see Section [
Several recent works have developed algorithms similar to QWR. We provide a brief overview and ways of obtaining them from the QWR pseudocode (Algorithm 2p. AWR can be recovered by learning a value function ((s) as a critic (Line 14) and sampling actions from the replay buffer (lines 12
and 18 in Algorithm 2). AWAC (Nair et al. 2020) modifies AWR by learning a Q-function for the critic. We get it from QWR by sampling actions from the replay buffer (lines 12 and (8). Note that compared AWAC. by sampling multiple actions for each state, QWR is able to take advantage of Q-learning to improve the critic. CRR (Wang et al.] (020) augments AWAC with training a distributional Q-function in Line 14 and substituting different functions for computing advantage weights in Line Again, compared to CRR, QWR samples multiple actions for each state, and a way similar to QWR, MPO (Abdolmaleki et al. 2018) samples actions during actor training to improve generalization. Compared to QWR, it introduces dual function for dynamically tuning 3 in Line 20 adds a prior regularization for policy training and trains the critic using Retrace (Munos et al. 2016) targets in line 13 QWR can be thought of as significant simplification of MPO, with addition of Q-learning to provide a better baseline for the actor. Additionally, the classical DQN (Mnih et al. (2033) algorithm for discrete action spaces be recovered from QWR by removing the actor training loop (lines [6|21 computing a maximum over all actions in Q-network training (line] for data collection.
so can take advantage of Q-learning.
13 and using an epsilon-greedy policy w.r.t. the Q-network
Offline reinforcement learning. Offline RL is the main topic of the survey Levine et al. (020). The authors state that "offline reinforcement learning methods equipped with powerful function approximation may enable data to be turned into generalizable and powerful decision making engines". We see this as one of the major challenges of modern RL and this work contributes to this challenge. Many current algorithms perform to some degree in offline RL, e.g., variants of DDPG and DQN developed by Fujimoto et al. (2018); [Agarwal et al. (2019), as well as the MPO algorithm by Abdolmaleki et al. (2018) promising alternatives to AWR and QWR analyzed in this work.
ABM (Siegel et al. 2020) is a method of extending RL algorithms based on policy networks to offline settings. It first learns prior policy network on the offline dataset using a loss similar to Equation I and then learns the final policy network using any algorithm, adding an auxiliary term penalizing KL-divergence from the prior policy. CQL (Kumar et al. 2020) is method of extending RL algorithms based on Q-networks to offline settings by introducing an auxiliary loss. To compute the loss. CQL samples actions on-line during training of the Q-network, similar Line 14 in QWR. EMaQ (Ghasemipour et al. (020)) learns an ensemble of Q-functions using an Expected-Men -Ma backup operator and uses during evaluation to pick the best action. The Q-network training part is similar to QWR with F = max in line in Algorithm 2
The imitation learning algorithm MARWIL by Wang et al. (018) contirms that the advantage- weighted regression performs well in the context of complex games.
EXPERIMENTS
Neural architectures. In all MuJoCo experiments, for both value and policy networks, we use multi-layer perceptrons with two layers 256 neurons each, and ReLU activations. In all Atari experiments, for both value and policy networks, we use the same convolutional architectures as in Mnih et al.) (2013a). To feed actions to the network, we embed them using one linear layer. connected to the rest of the network using the formula o tanh((a where o is the processed observation and a is the embedded action. This is followed by the value or policy head. For the policy, we parameterize either the log- probabilitiee of actions in case of discrete action spaces, or the mean of a Gaussian distribution in case of continuous action spaces, while keeping the standard deviation constant, as 0.4.
!CRR sets the advantage weight function f to be hyperparameter in log - V) (Line 20 In QWR, (x) = exp(x/8).
6

---

Under review as a conference paper at ICLR 2021
Algorithm
Half-Cheetah
Walker
Hopper
Humanoid
QWR-LSE
2323 + 332
1301 + 445
1758 + 735
511 + 57
QWR-MAX
2250 + 254
1019 1185
1187 345
503 49
QWR-AVG
1691 682
1052 231
420 - 65
455 - 41
AWR
0
67
110 =
500 4
SAC
5492
493 6
1197 175
645 + 27
PPO
51 + 41
-14 + 98
15 1 75
72 + 18
Table Comparison of variants of QWR with AWR Peng et al. 2019), SAC (Haarnoja et al. and PPO (Schulman et al. 2017 4 MuJoCo environments 100K samples.
2018)
Algorithm
Boxing
Breakout
Freeway
Gopher
Pong
Seaquest
QWR-LSE
4.6
8
21.2
776
-7.6
308
QWR-MAX
-1.8
0.8
16.8
580
-2
252
QWR-AVG
-0.8
1.4
19.2
548
-9
296
PPO
-3.9
5.9
246
-20.5
370
OTRainbow
2.5
1.9
27.9
349.5
-19.3
354.1
MPR
16.1
14.2
23.1
341.5
10.5
361.8
MPR-aug
30.5
15.6
24.6
593.4
-3.8
603.8
SimPLe
9.1
16.4
20.3
845.6
12.8
683.3
Random
0.1
1.7
257.6
-20.7
68.4
Table Comparison of variants of QWR with the sample-efficient variant of Rainbow (Hessel et al. 2017 van Hasselt el al. (2019), MPR (Schwarzer et al.) (2020) SimPLe /Kaiser et al.) 2019) and random scores on Atari games at 100K samples. We report results of the the augmented and on-augmented version of the MPR algorithm. Since MPR and SimPLe are based learning a model of the environment, we do not consider them when choosing the best scores.
4.1 SAMPLE EFFICIENCY
Since we are concerned with sample efficiency, we focus our first experiments on the case when the number of interactions with the environment is limited. To use a single number that allows comparisons with previous work both on MuloCo and Atari, we decided to restrict the number of interactions to 100K. This number is high enough, that the state-of-thh-ar algorithms such as SAC reach good performance.
We run experiments on 4 MuloCo environments and 6 Atari games, evaluating three versions of QWR with the 3 backup operators introduced in Section 2.3 QWR-LSE (using log-sum-cxp), QWR-MAX (using maximum) and QWR-AV AVG (using average). For all those experiments, truncation horizon to and the number of action samples & to 8. We discuss the choice these values and show ablations below, while more experimental details are given in Appendix A.
set the Q target
In Tables D and 2 we present the final numbers at 100K samples for the considered algorithms and environments. To put them within context, we also provide numbers for SAC, PPO, OTRainbow variant of Rainbow tuned for sample efficiency, MPR and SimPLe.
On all considered MuJoCo tasks, QWR exceeds the performance of AWR and PPO. The better sample efficiency is particularly well visible in the case of Walker, where each variant of QWR performs better than any baseline considered. On Hopper, QWR-LSE the best variant outpaces all baselines by a large margin. On Humanoid, it comes close to SAC the state of the art on MuJoCo.
QWR surpasses PPO and Rainbow out of Atari games. In Gopher and Pong QWR wins even against the augmented and non non-augmented versions of the model-based MPR algorithm.
4.2 ABLATIONS
In Figure 2
provide ablation of QWR with respect to the backup method F, multistep target horizon T ("margin") and the number of action samples to consider when training the actor and the critic. As can see, the algorithm is fairly robust the choice of these hyperparameters.
7

---

Under review as a conference paper at ICLR 2021
2200
5004
4000
3000
3500
300
Average return
2500
3000
Average
2000
Average
2000
2000
2000
nusamples 4
samples
n_somples 4
2,sampies 2
8. sammples =
0,5amples -
samples =
p. 0
samples 8
samples
sasmpiess
samples
-2000
200 Thousands of samples
150
250
-1300
150 Thousands of samples
100
X0
-500
100
250
Thousands of samples
(a) QWR-LSE, margin 1.
(b) QWR-QWR--RR LSE, margin 3.
(c) QWR LSE, margin 7.
8000
3500
3000
Averagee retumn
4000
2500
3000
2000
Average
2000
Average
1500
2000
1000
n. n,samples 22
n_samples 4
nasapess =
n. pamppes =
n. n_samples -
n_samples 4
samples
samppies =
samppes
n_samples
-1800
-500
Thousands samples
Thousands samples
Thousands samples
(d) QWR-MAX, morgin 1.
(e) QWR-MAX, margin 3.
(f) QWR-MAX, margin 7.
Figure 2: Ablation of QWR with respect to the margin, the number of action samples and the method of training the critic. The results are shown on the Calf-Cheetah environment. The plots show the median of 5 runs with the shaded area denoting the interquartile range.
In total, the log-sum-exp backup (LSEE) achieves the best results compare Figure 2b and Figure 2e Max backup performs well with margin 1, but is more sensitive to higher numbers of samples compare Figure 2d and Figure 2c] The log-sum-exp backup is less vulnerable to this effect compare Figure 2ajand Figure 2d. Higher margins decrease performance see Figuue 2c and Figure 2b We conjecture this be due to stale action sequences in the replay buffer biasing the multi-step targets. Again, the log-sum-exp backup is less prone to this issue compare Figure 2c to Figure 2i
4.3 OFFLINE RL
Both QWR and AWR are capable of handling expert data. AWR was shown to behave in a stable way when provided only with a number of expert trajectories (see Figure 7 in Peng al. (2019) without additional data collection. In this respect, the performance of AWR is much more robust than the performance of PPO and SAC. In Figure 3 we show the same result for QWR in terms of re-using the expert trajectories, it matches or exceeds AWR. The QWR trainings based on offline data were remarkably stable and worked well across all environments we have tried.
For the offline RL experiments, we have trained each algorithm for 30 iterations, without additional data collection. The training trajectories contained only states, actions and rewards, without any Algorithm-specifii data. In QWR, we have set the per-step sampling policies μ to be Gaussians with mean at the performed action and standard deviation set to 0.4, as usual in MuJoCo experiments.
DISCUSSION AND FUTURE WORK
We present Q-value Weighted Regression (QWR), an off-policy actor-critic algorithm that extends Advantage Weighted Regression with action sampling and Q-learning. is significantly more sample- efficient than AWR and works well with discrete actions and in visual domains, e.g., on Atari games. QWR consists of two interleaved steps of supervised training: the critic learning the Q function using predefined backup operator, and the actor learning the policy with weighted regression based on multiple sampled actions. Thanks to this clear structure, QWR is simple to implement and debug. It is also stable in wide range of hyperparameter choices and works well in the offline setting.
8

---

Under review as a conference paper at ICLR 2021
2009
1750
1300
1250
1000
1000
1900
790
(a) HalfCheetah
(b) Hopper
(c) Walker2d
(d) HalfCheetah
(e) Hopper
i Walker2d
Figure 3: Figures 3a 3b and 3c show offline trainings based on 50 trajectories of length 1000 collected by diverse policies. The horizontal lines mark the average return of a policy from the dataset. The bars denote median returns out of 4 runs, and the vertical lines denote the interquartile runge. Data for figures 3d Benad] 3 is borrowed from Peng el al. (2019) to cover a broader family of algorithms and show that offinn training fails for many RL algorithms.
Importantly, we designed QWR thanks a theoretical analysis that revealed why AWR may not work when there are limits on data collection in the environment. Our analysis for the limited data regime is based on the wewedddeeeee-eaatee assumption that allows to fully solve AWR analytically while still being realistic and indicative of the performance of this algorithm with few samples. We believe that using the state-determines-aaciie assumption can yield important insights into other RI. algorithms well.
QWR already achieves state-of-the-ard results in settings with limited data and we believe that it can be further improved in the future. The critic training could benefit from the advances in Q-learning methods such as double Q-networks (van Hasselt et al., 2013) or Polyak averaging (Polyak) 1990). already used in SAC. Distributional Q-learning Bellemare et al. (2017) and the use of ensembles like REM Agarwal et al. 2020) could yield further improvements.
Notably, the QWR results present achieved with the same of hyperparameters (except for the network architecture) both for MuJoCo environments and for Atari games. This rare among deep reinforcement learning algorithms, especially among ones that strive for sample-efficieng: Combined with its stability and good performance in offline settings, this makes QWR compelling choice for reinforcement learning in domains with limited data.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. CoRR, abs/ 806.06920, 2018 URL http::t/p///a/v..rrg http://aaxx....o///////00.000. 06920
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off-cy deep reinforcement learning. CoRR, abs/1907.04544 2019. URL htt.:t::///aaxww.oo///////
1907.0455 04543
Rishabh Agarwal, Dale Schuuraas,, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning, 2020.
Mare G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning, 2017.

---

Under review as a conference paper at ICLR 2021
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. CoRR, abs/1812.000 12.00900, 2018. URL WRRLaatpp///axx.cee http///axwv..rraaaaaaaaaaaeeeeee 02900 02900
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.02298,, 2017. URL http:
//axxiv.org/abs//1110 02298
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. CoRR, abs/1903.00374, 2019, htttttt/aaxxiii www.aawwwww...aaaaaaaaaaaarreree abt///ss/19999 00374
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: review, and perspectives open problems, 2020.
Tutorial,
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013a.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013b.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, 2016.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and efficient off- policy reinforcement learning. CoRR, abs/ 1add16606.02477 2016. URUntip:/aaxxi..r/// arbbb// 1606. 02647]
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets, 2020.
Gerhard Neumann and Jan R. Peters. Fitted q-iteration by advantage weighted regression. In D. Koller,
D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 1177-1184 Curran Associates, Inc., 2009.
cc/
pdf
OpenAl. Openai five. |ttpe///baoo..genaa.ccc///eeeee
2018.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex
Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. CoRR, abs/1808.00177, 2018. URLhttp!
//arxiv.orge wwt.ooogg/abaaaaaee http://aaa11111100 00177
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression:
Simple and scalable off- off-policy reinforcement learning, 2019.
10

---

Under review as a conference paper at ICLR 2021
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weightt regression for opera- tional space control. Proceedings of the 24th International Conference on Machine Learning, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 598199993793.. doi: 10. 10.1145/1273496. 1273590. URL.http:://dol http//oi..rg/00.111111 1273496. 12685927339999
ICML '07, pp- 745-750,
Boris T. Polyak. New method of stochastic approximation type. 1990.
Andrei A. Rusu, Matej Vecerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. CoRR, abs/1610.04286, 2016. URL
http://arxiv. org abs 140.b//288800
Fereshteh Sadeghi and Sergey Levine. (cad)S-28rl: Real single-image flight without a single real /
image. CoRR, abs//611 1.0420,, 2016. URL
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations, 2020.
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017. URL /ttp::/arxiv..rm
abs/1712.015 01815
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 02620)3249.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference Neural Information Processing Systems, NIPS'99, pp. 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
Richard S Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. A. Solla, K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems 12, pp. 1057-1063. MIT Press, 2000.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning, 2015.
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to parametric models in rein- forcement learning? CoRR, abs//196.00243,, 2019. URLLattpp//arxivv www.wwoorrreeee wwt.abs////aeooooooeo.....eaaaea abs/1906
05243
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy P. Lillicrap. Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft II: A new challenge for reinforcement learning. CoRR, abs/1 708.04787. 2017. URL httphhhtp//aae.te /ttp::///axiii org/ abs 708. 04782
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In S. Bengio, H. Wallach, H. Larochelle, N. -asa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 6288 6288-6297. Curran Associates, Inc., 2018.
K. Grauman,

---

Under review as a conference paper at ICLR 2021
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2020.
Ronald Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Leam... 8(3-4) May 1992. ISSN 0885-6125, doi: 10. 00.1007/BF00999666 URL /ttp:::/ooi.oog/ 10t.00000000000006555000
Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann, Model-free preference-ban based reinforce- ment learning. In Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirlieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp.
2222-2228. AAAI Press, 2016. URL http: / www. aaai orq/ ocs iidex.de...///// php/AAAI/
SAAT1s/papez//iew/1222
12


Reviews:
[{"review_id": "s3u2_kmsbw6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on offline policy learning with a limited dataset and proposes a sample efficient algorithm called Q-Value Weighted Regression. Based on the Advantage Weighted Regression algorithm, this algorithm calculates the advantage of the sampling policy \\mu by estimated Q-value function. Experiment results show that the QWR algorithm has better performance than the AWR algorithm with limited data.\nThis paper is well-written and easy to follow.  The main contribution is delivered:  New sample efficient algorithm. However, I still have some concerns about this paper.\nFirst, in the experiment part, the authors compared the performance between three kinds of QWR algorithms and the AWR algorithm. Though QWR has better performance than the AWR algorithm, it seems that the QWR algorithms' output is more unstable than other algorithms.\nSecond, as the authors mention in related work, several recent works have developed algorithms similar to the QWR algorithm. It is better to show the performance of those algorithms in the experiment. If those developed algorithms have similar performance to the QWR algorithm, the QWR algorithm's importance may be affected.\nThird, in the critic part of the algorithm (line 7 to line 15), the Q^*-value function is estimated by sample a’ and the Q_{\\phi} value function uses samples (independent sample for two value function). However, in the actor part of the algorithm (line 16 to line 20), both the value function Q_{\\phi} and \\hat{V} is estimated by samples (same sample for two value function). So, I was wondering why there is a difference between the critic-step and actor-step?\nFinally, in the formula (5), it seems strange that there is an expectation for random variable a’ and variable s', but none of them appear in the formula.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fol77HnePlM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a variant of AWR with an added Q-function. It is motivated by the *state-determines-action* assumption, which is used to argue that AWR should always return the data-collecting policy if optimized to convergence. Experimental evaluation of QWR focuses on online sample efficiency and offline performance.\n\nI have reviewed this paper for a previous conference. It seems that the issues pointed out during that review process have not been addressed. Most importantly, the main motivating theorem (Theorem 1) for QWR appears to be incorrect: it relies on $\\log \\pi(a \\mid s)$ being nonpositive, but this is not true in general in continuous action spaces. Instead of fixing the theorem, it has been moved to the appendix.\n\nThe algorithm presented here is largely the same as that in two recent papers: [advantage-weighted actor critic](https://arxiv.org/abs/2006.09359) and [critic regularized regression](https://arxiv.org/abs/2006.15134). It would be unfair to fault this paper for its similarity to others written in the same timeframe, but it does probably mean that the paper should be judged more on its experimental evaluation and analysis than on its novelty. By the metric of experimental evaluation, it falls short. The offline setting has recently seen more standardization of benchmarks (see, eg, [D4RL](https://arxiv.org/abs/2004.07219) and the [DQN Replay Dataset](https://offline-rl.github.io/)), and evaluating on these would allow for cleaner comparison to other works. (To the paper's credit, it does discuss these prior works and their relation to QWR.)\n\nIt is possible this comparison was not performed because the offline setting studied is different than the usual offline setting: the data-collecting sampling policy $\\pi(a \\mid s)$ is required for the optimization in Equation 3, unlike most offline RL works which use only logged trajectories. It is also not necessarily an issue to study a different setting, but it requires some amount of justification for the modified problem setting. It also makes the evaluation more important, because it becomes unclear whether any gains are due to improving the algorithm or relaxing the problem statement.\n\nIf I am mistaken about the main theorem being incorrect, I will happily increase my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "coRiV0czfki", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a Q-value weighted regression (QWR) on top of the advantage weighted regression (AWR) to improve the sample efficiency for offline RL settings. Through the analysis to the AWR, the authors claim that it performs poorly in scenarios with discrete actions, which motivates the development of QWR. Empirically, the authors show that QWR is comparable to SAC in continuous tasks and a variant of Rainbow in discrete tasks. \n\nI think the topic investigated in this work is critical. Particularly in many real-world applications, which are offline, the sample efficiency stands to be the most important problem. The paper is also easy to follow and includes thorough investigation on the literature survey. However, the authors need to pay attention to a few major issues that discourage me to give a decent score. \n\n1. Though this work proposes a new offline RL method, the novelty is marginal. As QWR is completely on top of AWR, which seems quite straightforward. This just looks like an incremental step. The authors mention in the paper they theoretically analyze the AWR to motivate the QWR, but after checking the Appendix, no formal theoretical results have been reported, particularly for the proposed QWR. It would be better to see formally how QWR improves on top of AWR, in terms of returns and some constants. \n\n2. The state-determines-action assumption is problematic. In the paper, the authors say “But in the case of limited data, when only a few trajectories were collected, this assumption may hold, at least for a large subset of the replay buffer, which makes it relevant to the study of sample efficiency.” Such an assumption is quite strong in the paper. Since the authors leverage this assumption to give Theorem 1 for AWR, which reveals shortcomings for AWR. This sets a nice motivation for QWR. However, without this assumption, the motivation becomes weak? How to justify this assumption? I would suggest the authors to show in-depth analysis for AWR and QWR to theoretically show their fundamental distinctions.\n\n3. The experimental results are not convincing. In Table 1, though QWR outperforms both AWR and PPO, but not SAC. Similarly, in Table 2, still QWR cannot outperform completely baselines. Though in the abstract, the authors have mentioned, in the result section, they failed to give any further discussion. For offline RL, AWR completely outperforms the proposed QWR, though they are close. Still, the authors just directly described the results without discussion.\n\n*****************************\nAfter reading carefully the rebuttal from the authors, I raise the score a bit as it somewhat clarifies some confusion. However, the paper still needs improvement, particularly in terms of analytical results. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "F-u7Li3kGqR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes an off-policy actor-critic QWR algorithm that extends the AWR from (Peng et al. (2019)). The QWR algorithm estimates the Q-function using parameterized Q-network whereas AWR estimates the V-function. \n\nI think the paper is not novel enough to guarantee acceptance. The contribution of the paper of the paper is limited given the existing work on AWR. \n\nThe Section 2.2 is very confusing. They prove that AWR convergences to a policy that takes the actions appeared in the replay buffer in state-determines-action assumption holds. I think no-algorithm can learn if the state-determines-action assumption holds. Many existing off-policy papers make assumption on sampling policy (strong ignorability (1)). \n\n1. Shalit, Uri, Fredrik D. Johansson, and David Sontag. \"Estimating individual treatment effect: generalization bounds and algorithms.\" International Conference on Machine Learning. PMLR, 2017.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed Q-value-weighted regression approach for improving the sample efficiency of DRL. It is related to recent papers on advantage-weighted regression methods for RL. The approach is interesting, intuitive, and bears merits. Developing a simple yet sample-efficient algorithm using weighted regression would be a critical contribution to the field. The work has the potential to make an impact, if it has all the necessary ingredients of a strong paper.

However, reviewers raised a few issues that have to be addressed before the paper can be accepted. As some reviewers pointed out, there seem to be unaddressed major issues from previous submissions. Novelty appears limited, especially because the proposed approach is very similar to recent works (e.g., AWR). The experiment section lacks comparison to recent similar algorithms, and the available comparisons appear to be not strong enough to justify merits of the proposed algorithm. Theorem 1 requires an unrealistic state-determines-action assumption for the replay buffer. Although the authors made an effort to justify this assumption, it remains very problematic and rules out most randomized/exploration algorithms.
Decision: Reject