Offline Reinforcement Learning with Implicit Q-Learning
Authors: Ilya Kostrikov
Ashvin Nair
Sergey Levine
Venue: ICLR 2022 Poster (2022)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning
Batch Reinforcement Learning
Continuous Control

Abstract: Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This tradeoff is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose a new offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function, without any explicit policy. Then, we extract the policy via advantage-weighted behavioral cloning, which also avoids querying out-of-sample actions. We dub our method Implicit Q-learning (IQL).  IQL is easy to implement, computationally efficient, and only requires fitting an additional critic with an asymmetric L2 loss. IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

Published as a conference paper at ICLR 2022
OFFLINE REINFORCEMENT LEARNING WITH IMPLICIT Q-LEARNING
Ilya Kostrikov, Ashvin Nair & Sergey Levine
Department of Electrical Engineering and Computer Science
University of California, Berkeley (kostrikov, anairl aanir117ebeereeeee 7eeberkeley.. edu, svlevine www.:/eeee..bbbeeee.ee.....ee.
ABSTRACT
Offline reinforcement learning requires reconciling two conflicting aims: learning policy that improves over the behavior policy that collected the dataset, while the same time minimizing the deviation from the behavior policy so as to avoid errors due distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training improve the policy, and therefore need either constrain these actions to be in-distribution, or else regularize their values. We propose a new offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random vari- able, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing up into Q-function, without any explicit policy. Then, we extract the policy via advantage-weightee behavioral cloning, which also avoids querying out-of-sample actions. We dub our method implicit Q-learning (IQL). IQL is easy to implement, computationally efficient, and only requires fitting an additional critic with an asymmetric L.2 loss. IQL demonstrates the state-of-the-art performance on D4RL, standard bench- mark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.
INTRODUCTION
Offline reinforcement learning (RL) addresses the problem of learning effective policies entirely from previously collected data, without online interaction (Fujimoto el al.) 2019 Lange et al.] 2012) This is very appealing in range of real-world domains, from robotics logistics and operations research, where real-world exploration with untrained policies is costly or dangerous, but prior data available. However, this also carries with major challenges: improving the policy beyond the level of the behavior policy that collected the data requires estimating values for actions other than those that were seen the dataset, and this, turn, requires trading off policy improvement against distributional shift, since the values of actions that are too different from those in the data are unlikely to be estimated accurately. Prior methods generally address this by either constraining the policy to limit how far it deviates from the behavior policy (Fujimoto et al. 2019 Wu et al. 2019: Fujimoto
& Gu 2021 I Kumar et al. 2019 Nair et al. 2020 Wang et al. 2020). or by eegullrizin the learned value functions to assign low values of distribbtiiion actions (umar et al.
2020 Kostrikov
al. 202T). Nevertheless, this imposes a trade-ofi between how much the policy improves and how
vulnerable it is misestimation due to distributional shift. Can we devise an offline RI. method that

---

Published as a conference paper at ICLR 2022
avoids this issue by never needing to directly query or estimate values for actions that were not seen in the data?
In this work, start from an observation that in-distributtin constraints widely used in prior work might not be sufficient to avoid value function extrapolation, and we ask whether it is possible to learn optimal policy with in-sample learning. without ever querying the values of any unseen actions. The key idea in our method is to approximate an upper expectile of the distribution over values with respect to the distribution of dataset actions for each state. We alternate between fitting this value function with expectile regression, and then using it to compute Bellman backups for training the Q-function. We show that we can do this simply by modifying the loss function in a SARSA-style TD backup, without ever using out-of-samplee actions in the target value. Once this Q- function has converged, we extract the corresponding policy using advantage- adyantage-wiighted behavioral cloning. This approach does not require explicit constraints or explicit regularization of out-of- distribution actions during value function training. though our policy extraction step does implicitly enforce constraint, discussed in prior work on advantage-weigee 2007
regression (Peters& & Schaal
Peng et al. 2019: Nair et al. 2020 Wang et al. 20200.
Our main contribution is implicit Q-learning (IQL), a new offline RL algorithm that avoids ever querying values of unseen actions while still being able to perform multi-step dynamic program- ming updates. Our method is easy to implement by making small change to the loss function in a simple SARSA-like TD update and is computationally very efficient. Furthermore, our approach demonstrates the state-of-the-art performance on D4RL., popular benchmark for offline reinforce- ment learning. In particular, our approach significantly improves over the prior state-of-the-ar on challenging Ant Maze tasks that require to "stitch" several sub-optimal trajectories. Finally, we demonstrate that our approach is suitable for finetuning; after initialization from offline RL, IQL is capable of improving policy performance utilizing additional interactions.
2 RELATED WORK
A significant portion of recently proposed offline RL methods are based on either constrained or reg- ularized approximate dynamic programming (e.g.. Q-learning actor-critic methods), with the con- straint or regularizer serving to limit deviation from the behavior policy. We will refer to these meth- ods as "multi-step dynamic programming" algorithms, since they perform true dynamic program- ming for multiple iterations, and therefore can in principle recover the optimal policy if provided with high-coverage data. The constraints can be implemented via an explicit density model (Wu
et al. 2019 Fujimoto et al.) 2019) Kumar et al. 2019 Ghasemipour et al.) (02TT) implicit divergence
constraints Nair et al. 2020, Wang et al.) 2020 Peters & Schaal 2007: Peng et al. 2019 Siegel
et al., 2020). or by adding a supervised learning term to the policy improvement objective Fujimoto
Gu 202T). Several works have also proposed to directly regularize the Q-function to produce
low values for out-of-distribution actions (Kostrikov et al.) [221]] Kumar et al. 2020 Fakoor et al.) 202T]. Our method is also a multi-step dynamic programming algorithm. However, in contrast to prior works, our method completely avoids directly querying the learned Q-function with unseen actions during training, removing the need for any constraint during this stage, though the subse- quent policy extraction, which is based on advantage weighted regression (Peng al. 2019 Nair jet al., 2020), does apply an implicit constraint. However, this policy does not actually influence value function training.
In contrast to multi-step dynamic programming methods, several recent works have proposed meth- ods that rely either on single step of policy iteration, fitting the value function or Q-function of the behavior policy and then extracting the corresponding greedy policy (Peng et al., 2019 Brand- fonbrener et al. 202T; Gulcehre al. (021]. or else avoid value functions completely and utilize behavioral cloning-sty objectives (Chen et al. 202T). We collectively refer to these as "single-step" approaches. These methods avoid needing to query unseen actions as well, since they either use no value function at all, or learn the value function of the behavior policy. Although these methods are simple to implement and effective on the MuJoCo locomotion tasks in D4RL, we show that such single-step methods perform very poorly more complex datasets in D4RL, which require combin- ing parts of suboptimal trajectories ("stitching"). Prior multi-step dynamic programming methods perform much better in such settings. does method. We discuss this distinction in more detail in Section 5.T] Our method also shares the simplicity and computational efficiency of single-step approaches, providing an appealing combination of the strengths of both types of methods.
2

---

Published as a conference paper at ICLR 2022
Our method is based on estimating the characteristies of a random variable. Several recent works involve approximating statistical quantities of the value function distribution. In particular, quan- tile regression (Koenker & Hallock, 2001) has been previously used in reinforcement learning to estimate the quantile function of a state-action value function (Dabney et al.) 2018b a Kuznetsov et al. 2020). Although our method is related, in that we perform expectile regression, not to estimate the distribution of values that results from stochastic transitions, but rather estimate expectiles of the state value function with respect to random actions. This is very different statistic: our aim is not to determine how the Q-value can vary with different future outcomes, but how the Q-value can vary with different actions while averaging together future outcomes due stochastic dynamics. While prior work distributional RL can also be used for offline RL, it would suffer from the same action extrapolation issues as other methods, and would require similar constraints or regularization, while our method does not.
aim is
3 PRELIMINARIES
The RL problem is formulated in the context of a Markov decision process (MDP) (S, A. (((s),((s|ss) a), rss, a), 7). where is a state space, A is an action space, po(s) is a dis- tribution of initial states, p(s'|s,a) is the environment dynamics, r r(s, a) is reward function, and a discount factor. The agent interacts with the MDP according to policy r[als). The goal is to obtain a policy that maximizes the cumulative discounted returns:
(r((s,, ar)]so ~ po()) ~ x(118)). S441 ~ p(. a.) a.).
π = arg max ET
Off Off-policy RL methods based on approximate dynamic programming typically utilize a state-action value function (Q-function), referred to as Q(s, a), which corresponds to the discounted returns obtained by starting from the state and action a, and then following the policy m.
Offline reinforcement learning. In contrast to online (on-policy or off-policy) RL methods, of- fline RL uses previously collected data without any additional data collection. Like many recent offline RL methods, our work builds approximate dynamic programming methods that minimize temporal difference error, according to the following loss:
(f) B(s.a.")==((((s...
+ y max Qg(s', a') (Q(ssaa))),
where D is the dataset, (s a) is a parameterized Q-function, Qg(s, a) is target network (e.g.. with soft parameters updates defined via Polyak averaging), and the policy is defined as T(s) = arg max, Qo a). Most recent offline RL methods modify either the value function loss (above) to regularize the value function in way that keeps the resulting policy close to the data, or constrain the arg max policy directly. This is important because out-of -iitt--distribttionn actions a' can produce erroneous values for Qá(s', aa)) in the above objective, often leading to overestimation the policy defined to maximize the (estimated) Q-value.
4 IMPLICIT Q-LEARNING
In this work, we aim to entirely avoid querying out-of-sample (unseen) actions in our TD loss. Although the goal of this work is approximate the optimal Q-function, we start by considering fitted Q evaluation with a SARSA-style objective which has been considered prior work on Offline Reinforcement Learning (Brandionbrener et al. 202T]] Gulcehre al. 2021). This objective aims to learn the value of the dataset policy Ty (also called the behavior policy):
(2) This objective never queries values for out-of-sample actions, in contrast to Eqn. One specific property of this objective that is important for this work is that it uses mean squared error (MSE) that fits Qo (s, a) predict the mean statistics of the TD targets. Thus, if we assume unlimited capacity and no sampling error, the optimal parameters should satisfy
(0)) = "'s.
a) + *Qg(s' a') Qo(s,a))))
Q8- (s. a) r(s,a) +
[Q ('), a' (-1s))(-)s))
Prior work (Brandionbrener et al. directly using this objective learn * and then train the policy अw. to maximize Q This avoids
2021 Gulcehre et al. 2021 Peng et al. 2019) has proposed
3

---

Published as a conference paper at ICLR 2022
1.0
0.01
1.5
0.8
fu O]
(.5(EEE))
0.3
1.0
0.6
0.9.99
0.2
Mass
0.5
ma.i (x)
0.4
ma.1
ma.
0.2
0.1
mas
ma.
0.0
ma.
ma.(0)
Ma.39
ma,s0(0)
0.0
0.0
00.5
1.0
0.5
0.0
0.5
1.0
-2
-1.0
0.5
0.0
0.5
1.0
Figure 1: Left: The asymmetric squared loss used for expectile regression. T = 0.5 corresponds to the standard mean squared error loss, while r 0.9 gives more weight to positives differences. Center: Expectiles of normal distribution. Right: an example of estimating state conditional ex- pectiles of two-dimensional random variable. Each æ corresponds to a distribution over y. We can approximate a maximum of this random variable with expectile regression: 7 0.5 correspond to the conditional mean statistics of the distribution, while approximates the maximum operator over in-support values of
any issues with out-of-din dittr---iitributionn actions, since the TD loss only uses dataset actions. However, while this procedure works well empirically simple MuJoCo locomotion tasks in D4RL., will show that performs very poorly on more complex tasks that benefit from multi-step dynamic programming. In our method, which we derive next, we retain the benefits of using this SARSA- objective, but modify it so that it allows us to perform multi-step dynamic programming and learn near-optimal Q-function.
Our method will perform a Q-function update similar to Eqn. 2 but we will aim to estimate the maximum Q-value over actions that are the support of the data distribution. Crucially, will show that it is possible to do this without ever querying the learned Q-function out-of-sample actions by utilizing expectile regression. Formally, the value function we aim to learn given by:
L(0)= B(eaae")=D((((ss)) +7
max d'E.A
Qo(d' Qo(s,a))7)
(4)
st. (a'()''')00
Our algorithm, implicit Q-Learning (IQL), aims estimate this objective while evaluating the Q- function only on the state-action pairs in the dataset. To this end, we propose to fit Qo s,, a) to estimate state-conditional expectiles of the target values, and show that specific expectiles approxi- mate the maximization defined above. In Section 4.4 we show that this approach performs multi-step dynamic programming in theory, and in Section we show that does so practice.
4.1 EXPECTILE REGRESSION
Practical methods for estimating various statistics of random variable have been thoroughly studies in applied statistics and econometries. The E 1) expectile of some random variable X is defined as a solution to the asymmetric least squares problem:
arg min Ex-xx [L](x (m))) where Lz(u) = = 1(u < 0)lu². vi
That is, for > 0.5, this asymmetric loss function downweights the contributions of æ values smaller than my while giving more weights larger values (see Fig. ! left). Expectile regression closely related to quantile regression (Koenker & Hallock 2001), which is a popular technique for estimating quantiles of a distribution widely used reinforcement learning (Dabney et al. 2018b The quantile regression loss is defined as asymmetric & loss.
OOu method could also be derived with quantiles, but since we are not interested in learning all of the expectiles/quantiles unlike prior work (Dabney et al.] 2018b ab. it is more convenient to estimate single expectile hecause this involves simple modification to the MSE loss that is already used in standard RI.
methods. We found work somewhat better than quantile regression with its corresponding loss.
4

---

Published as a conference paper at ICLR 2022
We can also use this formulation predict expectiles of a conditional distribution:
arg min ((t,,))>D(/0000 (.)))
m=(#)
Fig. | (right) illustrates conditional expectile regression on a simple two-dimensional distribution. Note that we can optimize this objective with stochastic gradient descent. It provides unbiased gradients and is easy to implement with standard machine learning libraries.
4.2 LEARNING THE VALUE FUNCTION WITH EXPECTILE REGRESSION
Expectile regression provides us with a powerful framework to estimate statistics of a random vari- able beyond mean regression. We can expectile regression modify the policy evaluation objective in Eqn. 2 to predict an upper expectile of the TD targets that approximates the maximum of r(ss a) + Qa (s a') over actions a' constrained to the dataset actions, in Eqn. 40. This leads to the following expectile regression objective:
L(Ø)= =
(r((), 2Qpls a)(s)
However, this formulation has a significant drawback. Instead of estimating expectiles just with respect the actions in the support of the data, it also incorporates stochasticity that comes from the environment dynamics s' ~ p(-|s, a). Therefore, a large target value might not necessarily reflect the existence of single action that achieves that value, but rather "lucky" sample that happened to have transitioned into a good state. We resolve this by introducing a separate value function that approximates an expectile only with respect to the action distribution, leading the following loss:
Ly (+) Bs,.)) >D (T (TZ (s, Vs()))]
(5)
We can then use this estimate to update the Q-functions with the MSE loss, which averages over the stochasticity from the transitions and avoids the "lucky" sample issue mentioned above:
(o) E(s,a."') ~D[(r(s, a) + (V as)
(6)
Note that these losses not use any explicit policy, and only utilize actions from the dataset for both objectives, similarly to SARSA-style policy evaluation. In Section 44 will show that this procedure recovers the optimal Q-function under some assumptions. Also, even though only one action is available for every state in the dataset for continuous action spaces, due to neural network generalization, the expectile regression does not result in SARSA-style policy evaluation as shown in Section 5
4.3 POLICY EXTRACTION AND ALGORITHM SUMMARY
While modified TD learning procedure learns approx- imation to the optimal Q-function, it does not explicitly rep- resent the corresponding policy, and therefore requires sep- arate policy extraction step. While one can consider any tech- nique for policy extraction that constrains the learned policy to stay close to the dataset actions, we aim for simple method for policy extraction. As before, we aim avoid using out- of-samples actions. Therefore, we extract the policy with eerranttge-weighhee regression (Peters & Schaal 2007 Peng al. 2019) previously successfully used for policy extraction in Offline RL [Wang et al. 2018 Nair et al. 2020:
Algorithm Implicit Q Q-learning
Initialize parameters vb 0, 0, 6. TD learning (IQL): for each gradient step do
+- vb Av Ly (1))
@ - 0 Xa V Lo i -- (1 a) + al
Brandfon
end for
brener et al. 20210:
Policy extraction (AWR): for each gradient step do
L- (6) = (x.o) ~D (oppeee((([
(s)(s)) lo n (a]s)), (7)
@ x A L end for
where E oo) is an inverse temperature. Note that this objective does not clone all actions from the dataset but, as shown in prior work, this objective learns policy that maximizes the Q-values subject to a distri-
bution constraint (Peters & Schaal 2007] Peng et al., 2019) Nair et al. 2020). This step can be seen selecting and cloning the most optimal actions in the dataset.
5

---

Published as a conference paper at ICLR 2022
Our final algorithm consists of two stages. First, we fit the value function and Q. performing num- ber of gradient updates alternating between Equ. 5 and (6). Second, we perform stochastic gradient descent Eqn. 7 For both steps, use version of clipped double Q-learning (Fujimoto al, 2018), taking a minimum of two Q-functions for V V-fuuccion and policy updates. We summarize our final method in Algorithm | Note that the policy does not influence the value function in any way, and therefore extraction could be performed either concurrently or after TD learning. Concurrent learning provides way IQL with online finetuning, as we discuss in Section 3.3
4.4 ANALYSIS
In this section, we will show that IQL. can recover the optimal value function under the dataset support constraints. First, we prove simple lemma that we will then use to show how our approach can enable learning the optimal value function.
Lemma 1. Let X be real-valued random variable with bounded support and supremum of the support is x." Then,
lim my x² T-1
Proof Sketch. One can show that expectiles of a random variable have the same supremum x². Moreover, for all and such that < 72, we get my, May. Therefore, the limit follows from the properties of bounded monotonically non-decreasing functions.
In the following theorems, we show that under certain assumptions, our method indeed approximates the optimal state-action value Q" and performs multi-step dynamical programming. We first prove technical lemma relating different expectiles of the Q-function, and then derive main result regarding the optimality of our method.
For the sake of simplicity, we introduce the following notation for analysis. Let E [2]] be 7th expectile of X (e.g.. E0.5 corresponds to the standard expectation). Then, we define V+(s) and 5 and 6 correspondingly, recursively as:
Qr (s, a). which correspond to optimal solutions of Equ.
(+) = Ba (Q, a)l.
Q+ a) r(s, a) 7. ~p- (s')) Lemma 2. For all and T2 such that Tι < we get V (s) S V (s).
Proof. The proof follows the policy improvement proof (utton Barto 2018). See Appendix
aeA Corollary 2.1. For any and we have V*(s) ma(as))0
Q" (8, where (&) is defined
as above and Q* (s,a) is an optimal state-action value function constrained the dataset and defined as
(*( = r(s, a) 7
max
Q'(s', a')
S.f. Ty (a'aals')30 1s'120
Proof. The proof follows from the observation that convex combination is smaller than maximum.
Theorem 3.
lim V+(s) max aEA (s,s..).
=a(a|s)>0
Proof. Follows from combining Lemma and Corollary 2.1
Therefore, for a larger value of < 1, we get a better approximation of the maximum. On the other hand, it also becomes a more challenging optimization problem. Thus, we treat hyperparam- eter. Due to the property discussed in Theorem g we dub our method implicit Q-learning (IQL). We also emphasize that our value learning method defines the entire spectrum of methods between SARSA (r = 0.5) and Q-Learning (7 - 1). Note that in contrast to other multi-step methods, IQL absorbs the policy improvement step into value learning. Therefore, fitting Q- Q-function corresponds to the policy evaluation step, while fitting the value function with IQL corresponds to implicit policy improvement.

---

Published as a conference paper at ICLR 2022
(a) toy maze MDP
(b) true optimal V
(c) One-step Policy Eval.
(d) IQL
toy umaze environment (a). When the static dataset heavily corrupted by suboptimal actions, one-step policy evaluation results in a value function that degrades to zero far from the rewarding states too quickly (c). Our algorithm aims to learn a near-optimal value function, combining the best properties of SARSA-style evaluation with the ability to perform multi-step dynamic programming. leading to value functions that are much closer to optimality (shown in (b)) and producing a much better policy (d).
Figure 2: Evaluation of our algorithm
5 EXPERIMENTAL EVALUATION
Our experiments aim evaluate our method comparatively, in contrast to prior offline RL meth- ods, and in particular to understand how our approach compares both to single-step methods and multi-step dynamic programming approaches. We will first demonstrate the benefits of multi-step dynamic programming methods, such as ours, in contrast to single-step methods, showing that on some problems this difference can be extremely large. We will then compare IQL with state-of-the art single-step and multi-step algorithms the degree to which we can learn effective policies using only the actions in the dataset. We examine domains that contain near-optimal trajectories, where single-step methods perform well, as well as domains with no optimal trajectories at all, which require multi-step dynamic programming. Finally, will study how IQL compares prior methods when finetuning with online RL starting from an offline RL initialization.
the D4RL (Fu et al. 2020) benchmark tasks, studying
5.1 THE DIFFERENCE BETWEEN ONE-STEP POLICY IMPROVEMENT AND IQL
We first use a simple maze environment to illustrate the importance of multi-step dynamic program- ming for offline RL. The maze has a u-shape, single start state, and a single goal state (see Fig. 2a The agent receives reward of 10 for entering the goal state and zero reward for all other transitions. With probability of 0.25, the agent transitions state. The dataset consists of optimal trajectory and 99 trajectories with uniform random actions. Due to a short horizon the problem, we use = 0.9.
random state, and otherwise to the commanded
Fig. 2 (c, d) illustrates the difference between single-step methods which fit Q* (x, a) via SARSA- style objective, in this ease represented by Onepstep RL (Brandfonbrener et al.] 2021; Wang et al. 2018 Gulcehre et al. 202I) and IQL with 0.95. Note that these methods represent special case of our method with values, these values decay much faster we move further away than they would for the optimal value function, and the resulting policy is highly suboptimal. Since IQL (d) performs iterative dynamic programming, it correctly propagates the signal, and the values no longer dominated by noise. The resulting value function closely matches the true optimal value function (b).
0.5. Although states closer to the high reward state will still have higher
5.2 COMPARISONS ON OFFLINE RL BENCHMARKS
Next, we evaluate our approach on the D4RL benchmark in comparison to prior methods (see Ta- ble D. The MuJoCo tasks D4RL consist of the Gym locomotion tasks, the Ant Maze tasks, and the Adroit and Kitchen robotic manipulation environments. Some prior works, particularly those proposing one-step methods, focus entirely on the Gym locomotion tasks. However, these tasks include significant fraction of near-optimal trajectories in the dataset. In contrast, the Ant Maze tasks, especially the medium and large ones, contain very few or no near-optimal trajectories, making them very challenging for one-step methods. These domains require "stitching" parts of suboptimal trajectories that travel between different states to find a path from the start to the goal of the maze (Fu et al. 2020 As we will show, multi-step dynamic programming is essential in these domains. The Adroit and Kitchen tasks are comparatively less discriminating, and we found that most RI methods perform similarly imitation learning in these domains (Florence et al. 2021

---

Published as a conference paper at ICLR 2022
Table I: Averaged normalized scores on MuJoCo locomotion and Ant Maze tasks. Our method outper- forms prior methods on the challenging Ant Maze tasks, which require dynamic programming, and is competitive with the best prior methods on the locomotion tasks.
Datnee
BC
105BC
BCQ
ARM
AWAC
Drestep
TD1-BC
CQI
IQI (las))
baifchheetthmmw22222
42.6
42.5
47.0
42.620.1
53.6
43.5
48.4=0.11
0..3=0.3
44.0=5A
47.4=0.22
hopger-m-x2
56.7
6..6++10
0.7
57.0
09.6+25
59.3424.2
58.4-211
66.2--.77
walker 264--22
75.3
72.6
7400:114
0.5
72.4
818:222
83.7221
72.5+088
78.3+ 8.7
36.6
40.6
2014
3666288
50.5
40.5
4.66
4.55
4..22:12
hopper-m-ex22 m-2e--
18.1
75.9
33.3
$2.77700
49.6
57.2
97,5++0.7
60.9 18.8
95.0+664
94.7*8A
walker 24-m--x22
666410
42.54:20
77.24:55
halfshhettub-m-e-422
868+13
93.44:66
90.724.3
91.6#288
hoppe--me--22
$2.5
110.9
8
107.6:11
0.7
55.8
10
98.0229.4
108.4668
91,5-14,3
2ulker24-mo--22
107.5
109.0
109.5
108.1
3.5
74.5
11300:0.44
110.1
108
1096: 1,0
466.7
666.2
231.4
661.6122277
31.
54.6
62.8
89.8
$9.2
59.9
$6.7
64.1
78.6
74.0
87.5 2.6
ummazz-----10
45.6
50.2
K3.0
53.0
48.7
49.3
60.7
71.4
84.0
62.2 13.8
antrsze-m----00
0.0
5.4
15.0
7.3
mnteaze-m---10
0.0
9.8
0.0
0.5
0.7
3.0
53.7
70.0
0.0
0.0
0.0
0.0
0.
0.0
0.0
0.2
15.8
30.6-25..8
untrove------0
0.0
6.0
0.0
0.0
0.0
0.0
0.0
14,9
4..5
112.2
109.1
Intal
566.9
800.
790.3
784.8
340.5
800.9
841.2
1002.1
1070.4:: 102.0
kilcoenn vü Intal
154.5
148.5
159.8-2244
adroit-D total
104.5
93.6
118.1 30.7
825.9
1210.3
138.3 155.3
900m
20m
80m
Note that challengiin to couspare stop malti-step methhods directly. Boundfoobroeere et and network size, that significantly differest from other methods. We repoct results for the original typesparmmeitss and runtime for a comparable set of hyperparameters
reports results set hyperprameters such batch
We therefore focus our analysis on the Gym locomotion and Ant Maze domains, but include full Adroit and Kitchen results in Appendix B for completeness.
Comparisons and baselines. We compare to methods that are representative of both multi- step dynamic programming and one-step ap- proaches. the former category, we compare to CQL (Kumar et al., 2020), TD3+BC (Fu- jumoto Gui 2021). and AWAC (Nair et al. 2020). In the latter category, we compare to Onestep RL (Brandronbrener et al. 2021
antmaaze-meduum-layy-ov
w/ CDQ
Epicods Retum
T- 0.5(MSE)
50
0.7
25
0.00
0.25
0.50
0.75 1.00
Gradient Staps (x10%)
Gradient Stops (x10f)
and Decision Transformers (Chen et al. 2021)
antmaze-large-plaayyv
Epicoda Retur
200
We obtained the Decision Transformers results on Ant Maze subsets of D4RI. tasks using the author-provided implementation and fol- lowing authors instructions communicated over email. We obtained results for TD3+BC and Onestep RL (Exp. Weight) directly from the au- thors. Note that Chen et al. (202T) and Brand- tonbrener et al. (221)) incorrectly report results for some prior methods, such as CQL, using the "-v0" environments. These generally produce lower scores than the "-v2" environments that these papers use for their own methods. We use the "-v2" environments for all methods to en- sure fair comparison, resulting in higher values for CQL. Because of this fix, our reported CQL scores are higher than all other prior methods. We obtained results for "-v2" datasets using an author-suggested implementation 3 On the Gym locomotion tasks (halfcheetah, hopper, walker2d), find that IQL performs comparably the best performing prior method, CQL. On the more challenging Ant Maze task, IQL outperforms CQL, and outperforms the one-step methods by very large margin.
40-
150
20
50
0.00 0.25 0.50 0.75 1.00
0.00
0.25 0.50 0.75 1.00
Gradient Steps (x10%)
Gracient Steps (x10*)
Estimating a larger expectile is crucial for animaze tasks that require dynamical programming ('stitching'). Right: Clipped dou- ble Q-Learning (CDQ) is crucial for learning val- ues for 7 = 0.9.
Figure 3: Left:
Runtime. Our approach is also computationally faster than the baselines (see Table Tp. For the baselines, we measure runtime for our reimplementations of the methods in JAX (Bradbury et al.) 2018) built on top of JAXRL (Kostrikov) 2021h, which are typically faster than the original imple-
8

---

Published as a conference paper at ICLR 2022
mentations. For example, the original implementation of CQL takes more than 4 hours to perform IM updates, while ours takes only 80 minutes. Even so, IQL still requires about 4x less time than reimplementation of CQL average, and is comparable to the fastest prior one-step methods. We did not reimplement Decision Transformers due to their complexity and report runtime of the original implementation.
Effect of r hyperparameter. We also demonstrate that it is crucial to compute a larger expectile on tasks that require "stitching" (see Fig. B We provide complete results in Appendix B With larger values of т, our method approximates Q-learning better, leading to better performance on the Ant Maze tasks. Moreover, due to neural network generalization, values learned with expectile regression increase with a larger and do not degrade to behavior policy values (T = 0.5). Finally, clipped double Q-Learning is crucial for estimating values for larger T 0.9.
5.3 ONLINE FINE-TUNING AFTER OFFLINE RL
The policies obtained by offline RL can often be improved with a small amount of online interac- tion. IQL. is well-suited for on- line fine-tuning for two reasons. First, IQL has strong offline per- formance, as shown in the pre- vious section, which provides good initialization. Second, IQL implements weighted behav- ioral cloning policy extraction step, which has previously been shown to allow for better on- line policy improvement com- pared to other types of offline constraints (Nair al.) 2020). To evaluate the finetuning capa- bility of various RL algorithms, we first run offline RL on each
Dataset
AWAC
CQL
IQL (Uur)
antmaze umaze vO antmaze- -iver--vv
56.7 - 59.0
70.1
99.4
88.0 - 96.3
49.3
49.0
99.4
67.0
49.0
untmazz-mcddum-ppee
0.0
+ 0.0
23.0
0.0
69.0
89.2
untmaze- mellmaze--mdium-eee mediim-iiverre----- diverse-vO
0.7
0.3
23.0
- 32.3
71.8
91.4
unimaze large play vD
0.0
0.0
1.0
0.00
36.8
+ 51.8
untmuze-large dirge-divers--O0
1.0 - 0.0
1.0
-+ 0.0
422
59.8
antmaze-s10 total
107.7 + 108.3
151.3
231.1
374.8 437.5
pen binary v0 door-binary-v0
44.6 70.3
31.2
-999
37,4 60.7
1.3 + 30.1
0.2
- 0.0
0.7 32,3
relocato-binary-v00
0.8 2.7
31.0
hand-vō total
46.7 - 103.1
31,5 9.9
38.1 124.0
total
211.4
182.8
241.0
412.9
Table 2: Online finetuning results showing the initial perfor- mance after offline RI., and performance after 1M steps of on- line RL. In all tasks, IQL is able to finetune to significantly higher performance than the offline initialization, with final per- formance that is comparable to or better than the best of either AWAC or CQL on all tasks except pen-binary-vls
dataset, then run 1M steps of online RL, and then report the final performance. We compare to AWAC (air et al. 2020). which has been proposed specifically for online finetuning, and CQL (Ku- mar et al. 2020g. which showed the best performance among prior methods in our experiments in the previous section. Exact experimental details are provided in Appendix C We use the challenging Ant Maze D4RL domains (Fu et al., 2020). as well as the high-dimension
dexterous manipulation environments from Rajeswaran al. (2118) which Nair et al. (2020) propose to use study online adaptation with AWAC. Results are shown in Table 5 On the Ant Maze domains, IQL significantly outperforms both prior methods after online finetuning. CQL attains the second best score, while AWAC performs comparatively worse due to much weaker offline initialization. On the dexterous hand tasks, IQL performs significantly better than AWAC on glocate-binary-vee comparably on door-binary-v0, and slightly worse pen-binary-v0, with the best overall score.
CONCLUSION
We presented implicit Q-Learning (IQL), a general algorithm for offline RL that completely avoids any queries to values of out-of-sample actions during training while still enabling multi-step dy- namic programming. To our knowledge, this is the first method that combines both of these fea- tures. This has a number of important benefits. First, our algorithm is computationally efficient: we can perform 1M updates on one GTX 1080 GPU in less than 20 minutes. Second, sim- ple to implement, requiring only minor modifications over a standard SARSA-like TD algorithm, and performing policy extraction with simple weighted behavioral cloning procedure resembling supervised learning. Finally, despite the simplicity and efficiency of this method, we show that it at- tains excellent performance across all of the tasks in the D4RL benchmark, matching the best prior methods on the MuJoCo locomotion tasks, and exceeding the -fate-of-the-ar performance challenging ant maze environments, where multi-step dynamic programming is essential for good performance.
the

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank Dibya Ghosh and the anonymous reviewers for helpful comments on earlier drafts of the paper. This research was supported by the Office Naval Research, C3.ai, and Intel, with compute support from Google.
REFERENCES
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- -Minn,, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//gi //github. com google/jax
David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. arXiv preprint arXiv:2106.08000 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01 10110000.13,,0220 2021.
Will Dabney, Georg Ostrovski. David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. International conference on machine learning. pp. 1096- 1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In -hcooy-Second AAAI Conference Artificial Intelligence, 2018b.
Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander Smola. Continuous doubly constrained batch reinforcement learning. arXiv preprint arXiv:2102.09225. 2021.
Pete Florence, Corey Lynch, Andy Zeng. Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, arXiv preprint arXiv:2109.001 00137, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.00 066600000 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062 PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. arXiv preprint XrXiv:2003.0955. 2021.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http:// thub. com qooq le/ fax
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXXiv:442..60 2014.
10

---

Published as a conference paper at ICLR 2022
Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4): 143-156, 2001.
Iyya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX., 10 2021.
URL https: /ttp::://gieee www.:thww.....eeeeeeeeeeeeeeeeee www.o//kksttrioo////00 jaxr
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum, Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policg q-learning via bootstrapping error reduction. arXiv preprint arXiv: /9000099066 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04777 2020.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture continuous distributional quantile critics. In International Conference Machine Learning, pp. 5556-5566. PMLR, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 1rXiv:1910.00177.. 2019 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward- reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745- 750, 2007.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learn- ing and Demonstrations. In Robotics: Science and Systems, 2018. URL lattps://arxiv.
10087.....08.....
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1988, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In NeurlPS, pp. 6291-6300, 2018
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006.151 2020.
Yifan Wu. George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint /rXiv: 1361, 2019.
11


Reviews:
[{"review_id": "a-7fLBPBPl", "reviewer": "Reviewer_eQvF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies how to avoid updating with out-of-sample or unseen actions during Q-updates which is one of the main challenges in offline RL as unseen actions can generate erroneous values for Q-functions and result in overestimation problems. While this paper looks at right place to address this issue, there are some concerns with novelty and design choices in this paper: \n\n- [1] originally proposed to use SARSA-like TD update during policy evaluation step that doesn't require querying out-distribution actions either. While [1] applies their method only in the discrete-action space, there is significant overlap between this work and [1]. Importantly, the authors fail to acknowledge and discuss this important related work which also weakens their argument about novelty with regard to the in-sample policy evaluation step.\n\n- There is lots of unknown about why expectile regression was selected for value function updates and it is not clear what is the motivation to go with expectile regression rather than mean regression. Particularly, what would happen if use mean regression, i.e. $L_V(\\phi) = E_{(s,a)} [(Q_\\hat{\\theta}(s,a) - V_\\phi(s))^2 ] $,  where it requires no $\\tau$? How will performance change?  Can you run experiments by using MSE loss instead? \n\n- One other idea is to use $V(s') = \\frac{1}{N} \\sum_{a' \\sim \\pi(.|s')} (Q_\\hat{\\theta}(s',a')] $ which is utilized in [4] ( see equation 1 in that paper). In addition, this work seems very similar to [4] as well. Can you discuss this and can you run experiments where $V(s)$ is estimated as mentioned? \n\n- What happens if $V_\\phi(s')$ is evaluated on s' in Eq 6 that is not trained on ( i.e s' is not shown during training of $V$ in Eq 5)?\n\n- Looking at table 1, it seems your proposed method performs very similarly to others in locomotion-v2 and kitchen-v0, but marginally better on antmaze-v0. Since these results are very close, can you do significance testing or include their learning curves to provide a better picture about these results? Finally, can you also add BCQ results to Table 1?\n\n- Can you explain how the learned policy is used during the deployment/test phase? Did you evaluate the learned policy with something like Algorithm 1 in [2]? \n\nRelated works:\n In the related work section, you only discussed offline methods where either policy or Q-function are regularized, not both. However, [3] shows that regularizing both policy and Q-function can be beneficial which is relevant to this paper as this paper also regularizes both Q and policy but implicitly. In addition, [2] also restricts the Q-function updates to [almost] state-action pairs that are in-distribution by learning a density function ( although there is still a chance for OOD action to be sampled by their density function). Both these works are missing and need to be discussed in the related work section.\n\n[1] Regularized Behavior Value Estimation https://arxiv.org/abs/2103.09575\n\n[2] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL https://arxiv.org/abs/2007.11091\n\n[3] Continuous Doubly Constrained Batch Reinforcement Learning https://arxiv.org/abs/2102.09225\n\n[4] Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning https://arxiv.org/abs/2002.08396\n\n", "summary_of_the_paper": "This paper proposes a new offline RL algorithm that uses in-sample policy evaluation and advantage-weighted regression during policy improvement. Particularly, it utilizes sarsa-like TD to update Q-function and avoids the query value of unseen actions during training. To evaluate their proposed method, they used the D4RL benchmark to compare with previous offline RL methods. In addition, they show results for fine-tuning of learned offline-policies during online deployment.  ", "summary_of_the_review": "While the general motivation of this paper about offline reinforcement learning is spot on, there are issues with this paper as mentioned above. Happy to hear authors' answers to raised issues.\n\n\n**update after rebuttal**\n\nConsidering authors' responses and comments, I've increased my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O5sgeoZoz_d", "reviewer": "Reviewer_DeQA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper overall is easy to follow and well written. I have two major concerns / questions:\n1. For a task with continuous state space, most state may only have one action in the logging data, which means $\\pi_\\beta(a|s)$ maybe a Dirac distribution. If this is the case, $E_a Q(s,a)$ would be the same with $\\max_{a\\in A,  \\pi_\\beta(a|s)>0} Q(s,a)$. If this is the case, does it mean the SARSA-style policy evaluation and optimal Q-function are not different? Then what's the role of expectile regression under this case?\n2. The results of IQL on Gym locomotion tasks are on-par or worse than those from CQL, it would be interesting to add some analysis of this. I found it a little bit counter-intuitive because IQL is more similar to behavior cloning than CQL, but underperforms on tasks with better quality data. \n\nMinor:\n\n1. Should it be $V_{\\tau}(s) = E_{a\\sim \\pi_\\beta (a|s)}[Q_{\\tau}(s,a)]$ above Lemma 2?\n2. It would be great to bold the best model on Table 1. ", "summary_of_the_paper": "This paper proposes in-sample Q-learning which composed of fitting value function with expectile regression, training Q-function and extracting the policy using advantage-weighted behavioral cloning. The method is validated on a set of D4RL examples and shows promising results on Ant Maze tasks. ", "summary_of_the_review": "In summary, I think the paper is well written and the idea is interesting. The empirical evaluation is significantly better than the existing baselines on challenging tasks like Ant Maze, and I thereby recommend accepting the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4PGOrl5QHF", "reviewer": "Reviewer_SYQw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized and easy to follow and read. The main contribution apart from prior work is the change in Q-function learning, which involves an additional value function learning and the expectile regression technique. After reading, I have the following question: \n1.\tThe one-step and multi-step concepts come from Peng et al., 2019. I think in their case, the multi-step means iterative optimization between policy improvement and policy evaluation, so that it is the dynamic programming problem. In this paper, the multi-step of IQL is for learning Q function and value function? \n2.\tThe learned optimal Q-function is for the learning of policy. Why advantage-weighted behavior cloning based on optimal Q-function could be better? I cannot see direct reasons here. \n3.\tFrom the proof, it is only when expectile tau goes to 1 in limit, so that the learned value function could be the optimal under the data. From the code, I observed that for MuJoCo tau=0.7, Adroit tau=0.7, and Ant-maze tau=0.9, which all do not goes limit to 1, like 0.99. I think there is a gap between the theory and the implementation. \n4.\tGiven your learned optimal Q-function is pretty good and accurate, why not just optimize a parameterized policy (deterministic or Gaussian policy) based on the learned Q-function, for example maximizing E_(a,s) [Q(s,a)]. Why must use behavior cloning? BC is hard to exceed the best policy provided in the dataset.  \n5.\tIs the value function really necessay in the framework? Do you try to remove the value function learning by just doing expectile regression over target Q values and Q values? The reason for asking this is that in continuous RL case, each state in the dataset is corresponding to one and only one action, which I think it is hard to train a good value function $V(s)$ with only one action $a$ by your eq(5). \n", "summary_of_the_paper": "This paper proposes an algorithm In-Sample Q-Learning (IQL), which learns a Q- and value function under behavior policy and applies advantage-weighted behavior cloning for policy learning. In one sentence, the main idea of the paper is to learn an approximately optimal Q/value function for better behavior cloning. The expectile regression instead of MSE is applied to find optimal Q-function of behavior policy.  ", "summary_of_the_review": "The idea is simple and straight forward, but more explanations are needed to provide strong insights. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "28YrEuJUx7", "reviewer": "Reviewer_Sf4r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that the main strength of this paper is the novel idea of using an expectile regression on dataset actions to learn a value function that is based on high-performing actions in the dataset. There have been many previous pieces of research on using quantile regression on RL, but most of them were to learn a distribution of value functions where the randomness comes from the environment, and the quantile regression is usually for the worst-case robustness. This paper, on the other hand, uses expectile regression on the value function where the randomness comes from the action and shows that it generalizes the Bellman expectation equation and Bellman optimality equation. As far as I know, this is the first suggestion of such an algorithm. The trick of using both $V$ and $Q$ functions to enable such learning seems to be very clever as well.\n\nTheoretical analysis also backs up the algorithm well, and empirical performance is also presented well, showing its advantage over other algorithms. It is especially interesting that IQL works well on hard domains (antmaze, dexterous manipulation, etc.). IQL will be a simple and robust but high-performing tool for an offline RL. I also believe that a similar approach can be applied to standard RL algorithms, increasing their robustness. \n\nIt will be more interesting if the paper contains more analysis and experiments, such as the learned value functions for different $\\tau$s and more state-of-the-art offline RL algorithms, but I am satisfied with the current form of the submission.", "summary_of_the_paper": "This paper proposes an offline RL algorithm named IQL, which generalizes between Bellman expectation equation and Bellman optimality equation with expectile regression. Expectile regression assigns low weights for low-performing samples and assigns high weights for high-performing samples to learn the expectile. By performing $\\tau$-expectile regression on the randomness of action in the dataset (from the sampling process of behavior policy), IQL is able to obtain a value function that is $\\tau$ optimal. IQL is generalizing SARSA and Q-learning in the sense that SARSA corresponds to IQL with $\\tau=0.5$ and Q-learning corresponds to IQL with $tau=1.0$. It offers very stable learning of a near-optimal value since it is only using in-distribution samples and the effective dataset size can be controlled through $\\tau$. In the experiments, the paper has shown that IQL can obtain state-of-the-art performance.", "summary_of_the_review": "The paper provides a good-performing robust offline algorithm based on a very simple but novel idea. I recommend acceptance of paper based on the novelty of the idea and the clean presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new paradigm --- called in-sample Q learning --- to tackle offline reinforcement learning. Based on the novel idea of using expectile regression, the proposed algorithm enjoys stable performance by focusing on in-sample actions and avoiding querying the values of unseen actions. The empirical performance of the proposed algorithm is appealing, outperforming existing baselines on several tasks. The paper is also well written.
Decision: Accept (Poster)