Year: 2021

Abstract:
Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose a new offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function, without any explicit policy. Then, we extract the policy via advantage-weighted behavioral cloning, which also avoids querying out-of-sample actions. We dub our method in-sample Q-learning (IQL). IQL is easy to implement, computationally efficient, and only requires fitting an additional critic with an asymmetric L2 loss. IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

text: Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose a new offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function, without any explicit policy. Then, we extract the policy via advantage-weighted behavioral cloning, which also avoids querying out-of-sample actions. We dub our method in-sample Q-learning (IQL). IQL is easy to implement, computationally efficient, and only requires fitting an additional critic with an asymmetric L2 loss. IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

heading: 1 INTRODUCTION
text: Offline reinforcement learning (RL) addresses the problem of learning effective policies entirely from previously collected data, without online interaction (Fujimoto et al., 2019; Lange et al., 2012). This is very appealing in a range of real-world domains, from robotics to logistics and operations research, where real-world exploration with untrained policies is costly or dangerous, but prior data is available. However, this also carries with it major challenges: improving the policy beyond the level of the behavior policy that collected the data requires estimating values for actions other than those that were seen in the dataset, and this, in turn, requires trading off policy improvement against distributional shift, since the values of actions that are too different from those in the data are unlikely to be estimated accurately. Prior methods generally address this by either constraining the policy to limit how far it deviates from the behavior policy (Fujimoto et al., 2019; Wu et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2019; Nair et al., 2020; Wang et al., 2020), or by regularizing the learned value functions to assign low values to out-of-distribution actions (Kumar et al., 2020; Kostrikov et al., 2021). Nevertheless, this imposes a trade-off between how much the policy improves and how vulnerable it is to misestimation due to distributional shift. Can we devise an offline RL method that avoids this issue by never needing to directly query or estimate values for actions that were not seen in the data?
In this work, we start from an observation that in-distribution constraints widely used in prior work might not be sufficient to avoid value function extrapolation, and we ask whether it is possible to learn an optimal policy with in-sample learning, without ever querying the values of any unseen actions. The key idea in our method is to approximate an upper expectile of the distribution over values with respect to the distribution of dataset actions for each state. We alternate between fitting this value function with expectile regression, and then using it to compute Bellman backups for training the Q-function. We show that we can do this simply by modifying the loss function in a SARSA-style TD backup, without ever using out-of-sample actions in the target value. Once thisQfunction has converged, we extract the corresponding policy using advantage-weighted behavioral cloning. This approach does not require explicit constraints or explicit regularization of out-ofdistribution actions during value function training, though our policy extraction step does implicitly enforce a constraint, as discussed in prior work on advantage-weighted regression (Peters & Schaal, 2007; Peng et al., 2019; Nair et al., 2020; Wang et al., 2020).
Our main contribution is in-sample Q-learning (IQL), a new offline RL algorithm that avoids ever querying values of unseen actions while still being able to perform multi-step dynamic programming updates. Our method is easy to implement by making a small change to the loss function in a simple SARSA-like TD update and is computationally very efficient. Furthermore, our approach demonstrates the state-of-the-art performance on D4RL, a popular benchmark for offline reinforcement learning. In particular, our approach significantly improves over the prior state-of-the-art on challenging Ant Maze tasks that require to “stitch” several sub-optimal trajectories. Finally, we demonstrate that our approach is suitable for finetuning; after initialization from offline RL, IQL is capable of improving policy performance utilizing additional interactions.

heading: 2 RELATED WORK
text: A significant portion of recently proposed offline RL methods are based on either constrained or regularized approximate dynamic programming (e.g., Q-learning or actor-critic methods), with the constraint or regularizer serving to limit deviation from the behavior policy. We will refer to these methods as “multi-step dynamic programming” algorithms, since they perform true dynamic programming for multiple iterations, and therefore can in principle recover the optimal policy if provided with high-coverage data. The constraints can be implemented via an explicit density model (Wu et al., 2019; Fujimoto et al., 2019; Kumar et al., 2019; Ghasemipour et al., 2021), implicit divergence constraints (Nair et al., 2020; Wang et al., 2020; Peters & Schaal, 2007; Peng et al., 2019; Siegel et al., 2020), or by adding a supervised learning term to the policy improvement objective (Fujimoto & Gu, 2021). Several works have also proposed to directly regularize the Q-function to produce low values for out-of-distribution actions (Kostrikov et al., 2021; Kumar et al., 2020; Fakoor et al., 2021). Our method is also a multi-step dynamic programming algorithm. However, in contrast to prior works, our method completely avoids directly querying the learned Q-function with unseen actions during training, removing the need for any constraint during this stage, though the subsequent policy extraction, which is based on advantage-weighted regression (Peng et al., 2019; Nair et al., 2020), does apply an implicit constraint. However, this policy does not actually influence value function training.
In contrast to multi-step dynamic programming methods, several recent works have proposed methods that rely either on a single step of policy iteration, fitting the value function or Q-function of the behavior policy and then extracting the corresponding greedy policy (Peng et al., 2019; Brandfonbrener et al., 2021; Gulcehre et al., 2021), or else avoid value functions completely and utilize behavioral cloning-style objectives (Chen et al., 2021). We collectively refer to these as “single-step” approaches. These methods avoid needing to query unseen actions as well, since they either use no value function at all, or learn the value function of the behavior policy. Although these methods are simple to implement and effective on the MuJoCo locomotion tasks in D4RL, we show that such single-step methods perform very poorly on more complex datasets in D4RL, which require combining parts of suboptimal trajectories (“stitching”). Prior multi-step dynamic programming methods perform much better in such settings, as does our method. We discuss this distinction in more detail in Section 5.1. Our method also shares the simplicity and computational efficiency of single-step approaches, providing an appealing combination of the strengths of both types of methods.
Our method is based on estimating the characteristics of a random variable. Several recent works involve approximating statistical quantities of the value function distribution. In particular, quan-
tile regression (Koenker & Hallock, 2001) has been previously used in reinforcement learning to estimate the quantile function of a state-action value function (Dabney et al., 2018b;a; Kuznetsov et al., 2020). Although our method is related, in that we perform expectile regression, our aim is not to estimate the distribution of values that results from stochastic transitions, but rather estimate expectiles of the state value function with respect to random actions. This is a very different statistic: our aim is not to determine how the Q-value can vary with different future outcomes, but how the Q-value can vary with different actions while averaging together future outcomes due to stochastic dynamics. While prior work on distributional RL can also be used for offline RL, it would suffer from the same action extrapolation issues as other methods, and would require similar constraints or regularization, while our method does not.

heading: 3 PRELIMINARIES
text: The RL problem is formulated in the context of a Markov decision process (MDP) (S,A, p0(s), p(s′|s, a), r(s, a), γ), where S is a state space, A is an action space, p0(s) is a distribution of initial states, p(s′|s, a) is the environment dynamics, r(s, a) is a reward function, and γ is a discount factor. The agent interacts with the MDP according to a policy π(a|s). The goal is to obtain a policy that maximizes the cumulative discounted returns:
π∗ = argmax π Eπ [ ∞∑ t=0 γtr(st, at)|s0 ∼ p0(·), at ∼ π(·|st), st+1 ∼ p(·|st, at) ] .
Off-policy RL methods based on approximate dynamic programming typically utilize a state-action value function (Q-function), referred to as Q(s, a), which corresponds to the discounted returns obtained by starting from the state s and action a, and then following the policy π.
Offline reinforcement learning. In contrast to online (on-policy or off-policy) RL methods, offline RL uses previously collected data without any additional data collection. Like many recent offline RL methods, our work builds on approximate dynamic programming methods that minimize temporal difference error, according to the following loss:
LTD(θ) = E(s,a,s′)∼D[(r(s, a) + γmax a′ Qθ̂(s ′, a′)−Qθ(s, a))2], (1)
where D is the dataset, Qθ(s, a) is a parameterized Q-function, Qθ̂(s, a) is a target network (e.g., with soft parameters updates defined via Polyak averaging), and the policy is defined as π(s) = argmaxaQθ(s, a). Most recent offline RL methods modify either the value function loss (above) to regularize the value function in a way that keeps the resulting policy close to the data, or constrain the argmax policy directly. This is important because out-of-distribution actions a′ can produce erroneous values for Qθ̂(s
′, a′) in the above objective, often leading to overestimation as the policy is defined to maximize the (estimated) Q-value.

heading: 4 IN-SAMPLE Q-LEARNING
text: In this work, we aim to entirely avoid querying out-of-sample (unseen) actions in our TD loss. Although the goal of this work is to approximate the optimal Q-function, we start by considering fittedQ evaluation with a SARSA-style objective which has been considered in prior work on Offline Reinforcement Learning (Brandfonbrener et al., 2021; Gulcehre et al., 2021). This objective aims to learn the value of the dataset policy πβ (also called the behavior policy):
L(θ) = E(s,a,s′,a′)∼D[(r(s, a) + γQθ̂(s ′, a′)−Qθ(s, a))2]. (2)
This objective never queries values for out-of-sample actions, in contrast to Eqn. (1). One specific property of this objective that is important for this work is that it uses mean squared error (MSE) that fits Qθ(s, a) to predict the mean statistics of the TD targets. Thus, if we assume unlimited capacity and no sampling error, the optimal parameters should satisfy
Qθ∗(s, a) ≈ r(s, a) + γEs′∼p(·|s,a) a′∼πβ(·|s) [Qθ̂(s ′, a′)]. (3)
Prior work (Brandfonbrener et al., 2021; Gulcehre et al., 2021; Peng et al., 2019) has proposed directly using this objective to learnQπβ , and then train the policy πψ to maximizeQπβ . This avoids any issues with out-of-distribution actions, since the TD loss only uses dataset actions. However,
while this procedure works well empirically on simple MuJoCo locomotion tasks in D4RL, we will show that it performs very poorly on more complex tasks that benefit from multi-step dynamic programming. In our method, which we derive next, we retain the benefits of using this SARSA-like objective, but modify it so that it allows us to perform multi-step dynamic programming and learn a near-optimal Q-function.
Our method will perform a Q-function update similar to Eqn. (2), but we will aim to estimate the maximum Q-value over actions that are in the support of the data distribution. Crucially, we will show that it is possible to do this without ever querying the learned Q-function on out-of-sample actions by utilizing expectile regression. Formally, the value function we aim to learn is given by:
L(θ) = E(s,a,s′)∼D[(r(s, a) + γ max a′∈A
s.t. πβ(a′|s′)>0
Qθ̂(s ′, a′)−Qθ(s, a))2]. (4)
Our algorithm, in-sample Q-Learning (IQL), aims to estimate this objective while evaluating the Q-function only on the state-action pairs in the dataset. To this end, we propose to fit Qθ(s, a) to estimate state-conditional expectiles of the target values, and show that specific expectiles approximate the maximization defined above. In Section 4.4 we show that this approach performs multi-step dynamic programming in theory, and in Section 5.1 we show that it does so in practice.

heading: 4.1 EXPECTILE REGRESSION
text: Practical methods for estimating various statistics of a random variable have been thoroughly studies in applied statistics and econometrics. The τ ∈ (0, 1) expectile of some random variable X is defined as a solution to the asymmetric least squares problem:
argmin mτ
Ex∼X [Lτ2(x−mτ )], where Lτ2(u) = |τ − 1(u < 0)|u2.
That is, for τ > 0.5, this asymmetric loss function downweights the contributions of x values smaller than mτ while giving more weights to larger values (see Fig. 1, left). Expectile regression is closely related to quantile regression (Koenker & Hallock, 2001), which is a popular technique for estimating quantiles of a distribution widely used in reinforcement learning (Dabney et al., 2018b;a) 1. The quantile regression loss is defined as an asymmetric `1 loss.
We can also use this formulation to predict expectiles of a conditional distribution: argmin mτ (x) E(x,y)∼D[Lτ2(y −mτ (x))].
1Our method could also be derived with quantiles, but since we are not interested in learning all of the expectiles/quantiles, unlike prior work (Dabney et al., 2018b;a), it is more convenient to estimate a single expectile because this involves a simple modification to the MSE loss that is already used in standard RL methods. We found it to work somewhat better than quantile regression with its corresponding `1 loss.
Fig. 1 (right) illustrates conditional expectile regression on a simple two-dimensional distribution. Note that we can optimize this objective with stochastic gradient descent. It provides unbiased gradients and is easy to implement with standard machine learning libraries.

heading: 4.2 LEARNING THE VALUE FUNCTION WITH EXPECTILE REGRESSION
text: Expectile regression provides us with a powerful framework to estimate statistics of a random variable beyond mean regression. We can use expectile regression to modify the policy evaluation objective in Eqn. (2) to predict an upper expectile of the TD targets that approximates the maximum of r(s, a) + γQθ̂(s
′, a′) over actions a′ constrained to the dataset actions, as in Eqn. (4). This leads to the following expectile regression objective:
L(θ) = E(s,a,s′,a′)∼D[Lτ2(r(s, a) + γQθ̂(s ′, a′)−Qθ(s, a))].
However, this formulation has a significant drawback. Instead of estimating expectiles just with respect to the actions in the support of the data, it also incorporates stochasticity that comes from the environment dynamics s′ ∼ p(·|s, a). Therefore, a large target value might not necessarily reflect the existence of a single action that achieves that value, but rather a “lucky” sample that happened to have transitioned into a good state. We resolve this by introducing a separate value function that approximates an expectile only with respect to the action distribution, leading to the following loss:
LV (ψ) = E(s,a) ∼D[Lτ2(Qθ̂(s, a)− Vψ(s))]. (5) We can then use this estimate to update the Q-functions with the MSE loss, which averages over the stochasticity from the transitions and avoids the “lucky” sample issue mentioned above: LQ(θ) = E(s,a,s′) ∼D[(r(s, a) + γVψ(s′)−Qθ(s, a))2]. (6) Note that these losses do not use any explicit policy, and only utilize actions from the dataset for both objectives, similarly to SARSA-style policy evaluation. In Section 4.4, we will show that this procedure recovers the optimal Q-function under some assumptions. Also, despite the fact that only one action is available for every state in the dataset for continuous action spaces, due to neural network generalization the expectile regression does not result in SARSA-style policy evaluation as shown in Section 5.2.

heading: 4.3 POLICY EXTRACTION AND ALGORITHM SUMMARY
text: Algorithm 1 In-Sample Qlearning
Initialize parameters ψ, θ, θ̂, φ. TD learning (IQL): for each gradient step do
ψ ← ψ − λV∇ψLV (ψ) θ ← θ − λQ∇θLQ(θ) θ̂ ← (1− α)θ̂ + αθ
end for Policy extraction (AWR): for each gradient step do
φ← φ− λπ∇φLπ(φ) end for
While our modified TD learning procedure learns an approximation to the optimal Q-function, it does not explicitly represent the corresponding policy, and therefore requires a separate policy extraction step. While one can consider any technique for policy extraction that constrains the learned policy to stay close to the dataset actions, we aim for a simple method for policy extraction. As before, we aim to avoid using out-ofsamples actions. Therefore, we extract the policy with advantage weighted regression (Peters & Schaal, 2007; Peng et al., 2019) previously successfully used for policy extraction in Offline RL (Wang et al., 2018; Nair et al., 2020; Brandfonbrener et al., 2021): Lπ(φ) = E(s,a) ∼D[exp(β(Qθ̂(s, a)− Vψ(s))) log πφ(a|s)], (7) where β ∈ [0,∞) is an inverse temperature. Note that this objective does not clone all actions from the dataset but, as shown in prior work, this objective learns a policy that maximizes the Q-values subject to a distribution constraint (Peters & Schaal, 2007; Peng et al., 2019; Nair et al., 2020). This step can be seen as selecting and cloning most optimal actions in the dataset.
Our final algorithm consists of two stages. First, we fit the value function and Q, performing a number of gradient updates alternating between Eqn. (5) and (6). Second, we perform stochastic gradient descent on Eqn. (7). For both steps, we use a version of clipped double Q-learning (Fujimoto et al., 2018), taking a minimum of two Q-functions for V -function and policy updates. We summarize our final method in Algorithm 1. Note that the policy does not influence the value function in any way, and therefore extraction could be performed either concurrently or after TD learning. Concurrent learning provides a way to use IQL with online finetuning, as we discuss in Section 5.3.

heading: 4.4 ANALYSIS
text: In this section, we will show that IQL can recover the optimal value function under the dataset support constraints. First, we prove a simple lemma that we will then use to show how our approach can enable learning the optimal value function. Lemma 1. Let X be a real-valued random variable with a bounded support and supremum of the support is x∗. Then,
lim τ→1
mτ = x ∗
Proof Sketch. One can show that expectiles of a random variable have the same supremum x∗. Moreover, for all τ1 and τ2 such that τ1 < τ2, we get mτ1 ≤ mτ2 . Therefore, the limit follows from the properties of bounded monotonically non-decreasing functions.
In the following theorems, we show that under certain assumptions, our method indeed approximates the optimal state-action value Q∗ and performs multi-step dynamical programming. We first prove a technical lemma relating different expectiles of the Q-function, and then derive our main result regarding the optimality of our method.
For the sake of simplicity, we introduce the following notation for our analysis. Let Eτx∼X [x] be a τ th expectile of X (e.g., E0.5 corresponds to the standard expectation). Then, we define Vτ (s) and Qτ (s, a), which correspond to optimal solutions of Eqn. 5 and 6 correspondingly, recursively as:
Vτ (s) = Eτa∼πβ(·|s)[Qτ (s, a)],
Qτ (s, a) = r(s, a) + γEs′∼p(·|s,a)[Vτ (s′)]. Lemma 2. For all s, τ1 and τ2 such that τ1 < τ2 we get
Vτ1(s) ≤ Vτ2(s).
Proof. The proof follows the policy improvement proof (Sutton & Barto, 2018). See Appendix A.
Corollary 2.1. For any τ and s we have
Vτ (s) ≤ max a∈A
s.t. πβ(a|s)>0
Q∗(s, a)
where Vτ (s) is defined as above and Q∗(s, a) is an optimal state-action value function constrained to the dataset and defined as
Q∗(s, a) = r(s, a) + γEs′∼p(·|s,a)  max a′∈A
s.t. πβ(a′|s′)>0
Q∗(s′, a′)  .
Proof. The proof follows from the observation that convex combination is smaller than maximum.
Theorem 3. lim τ→1 Vτ (s) = max a∈A
s.t. πβ(a|s)>0
Q∗(s, a).
Proof. Follows from combining Lemma 1 and Corollary 2.1.
Therefore, for a larger value of τ < 1, we get a better approximation of the maximum. On the other hand, it also becomes a more challenging optimization problem. Thus, we treat τ as a hyperparameter. Due to the property discussed in Theorem 3 we dub our method implicit Q-learning (IQL). We also emphasize that our value learning method defines the entire spectrum of methods between SARSA (τ = 0.5) and Q-Learning (τ → 1). Note that in contrast to other multi-step methods, IQL absorbs the policy improvement step into value learning. Therefore, fitting Q-function corresponds to the policy evaluation step, while fitting the value function with IQL corresponds to implicit policy improvement.

heading: 5 EXPERIMENTAL EVALUATION
text: Our experiments aim to evaluate our method comparatively, in contrast to prior offline RL methods, and in particular to understand how our approach compares both to single-step methods and multi-step dynamic programming approaches. We will first demonstrate the benefits of multi-step dynamic programming methods, such as ours, in contrast to single-step methods, showing that on some problems this difference can be extremely large. We will then compare IQL with state-of-theart single-step and multi-step algorithms on the D4RL (Fu et al., 2020) benchmark tasks, studying the degree to which we can learn effective policies using only the actions in the dataset. We examine domains that contain near-optimal trajectories, where single-step methods perform well, as well as domains with no optimal trajectories at all, which require multi-step dynamic programming. Finally, we will study how IQL compares to prior methods when finetuning with online RL starting from an offline RL initialization.

heading: 5.1 THE DIFFERENCE BETWEEN ONE-STEP POLICY IMPROVEMENT AND IQL
text: We first use a simple maze environment to illustrate the importance of multi-step dynamic programming for offline RL. The maze has a u-shape, a single start state, and a single goal state (see Fig. 2a). The agent receives a reward of 10 for entering the goal state and zero reward for all other transitions. With a probability of 0.25, the agent transitions to a random state, and otherwise to the commanded state. The dataset consists of 1 optimal trajectory and 99 trajectories with uniform random actions. Due to a short horizon of the problem, we use γ = 0.9.
Fig. 2 (c, d) illustrates the difference between single-step methods which fit Qπ(s, a) via SARSAstyle objective, in this case represented by Onepstep RL (Brandfonbrener et al., 2021; Wang et al., 2018; Gulcehre et al., 2021) and IQL with τ = 0.95. Note that these methods represent a special case of our method with τ = 0.5. Although states closer to the high reward state will still have higher values, these values decay much faster as we move further away than they would for the optimal value function, and the resulting policy is highly suboptimal. Since IQL (d) performs iterative dynamic programming, it correctly propagates the signal, and the values are no longer dominated by noise. The resulting value function closely matches the true optimal value function (b).

heading: 5.2 COMPARISONS ON OFFLINE RL BENCHMARKS
text: Next, we evaluate our approach on the D4RL benchmark in comparison to prior methods (see Table 1). The MuJoCo tasks in D4RL consist of the Gym locomotion tasks, the Ant Maze tasks, and the Adroit and Kitchen robotic manipulation environments. Some prior works, particularly those proposing one-step methods, focus entirely on the Gym locomotion tasks. However, these tasks include a significant fraction of near-optimal trajectories in the dataset. In contrast, the Ant Maze tasks, especially the medium and large ones, contain very few or no near-optimal trajectories, making them very challenging for one-step methods. These domains require “stitching” parts of suboptimal trajectories that travel between different states to find a path from the start to the goal of the maze (Fu et al., 2020). As we will show, multi-step dynamic programming is essential in these domains. The Adroit and Kitchen tasks are comparatively less discriminating, and we found that most RL methods perform similarly to imitation learning in these domains (Florence et al., 2021).
We therefore focus our analysis on the Gym locomotion and Ant Maze domains, but include full Adroit and Kitchen results in Appendix B for completeness.
Comparisons and baselines. We compare to methods that are representative of both multistep dynamic programming and one-step approaches. In the former category, we compare to CQL (Kumar et al., 2020), TD3+BC (Fujimoto & Gu, 2021), and AWAC (Nair et al., 2020). In the latter category, we compare to Onestep RL (Brandfonbrener et al., 2021) and Decision Transformers (Chen et al., 2021). We obtained the Decision Transformers results on Ant Maze subsets of D4RL tasks using the author-provided implementation2 and following authors instructions communicated over email. We obtained results for TD3+BC and Onestep RL (Exp. Weight) directly from the authors. Note that Chen et al. (2021) and Brandfonbrener et al. (2021) incorrectly report results for some prior methods, such as CQL, using the “-v0” environments. These generally produce lower scores than the “-v2” environments that these papers use for their own methods. We use the “-v2” environments for all methods to en-
sure a fair comparison, resulting in higher values for CQL. Because of this fix, our reported CQL scores are higher than all other prior methods. We obtained results for “-v2” datasets using an author-suggested implementation.3 On the Gym locomotion tasks (halfcheetah, hopper, walker2d), we find that IQL performs comparably to the best performing prior method, CQL. On the more challenging Ant Maze task, IQL outperforms CQL, and outperforms the one-step methods by a very large margin. Runtime. Our approach is also computationally faster than the baselines (see Table 1). For the baselines, we measure runtime for our reimplementations of the methods in JAX (Bradbury et al., 2018) built on top of JAXRL (Kostrikov, 2021), which are typically faster than the original imple-
2https://github.com/kzl/decision-transformer 3https://github.com/young-geng/CQL
mentations. For example, the original implementation of CQL takes more than 4 hours to perform 1M updates, while ours takes only 80 minutes. Even so, IQL still requires about 4x less time than our reimplementation of CQL on average, and is comparable to the fastest prior one-step methods. We did not reimplement Decision Transformers due to their complexity and report runtime of the original implementation.
Effect of τ hyperparameter. We also demonstrate that it is crucial to compute a larger expectile on tasks that require “stitching” (see Fig. 3). Complete results are provided in Appendix B. With larger values of τ , our method approximates Q-learning better, leading to better performance on the Ant Maze tasks. Moreover, due to neural network generalization, values learned with expectile regression increase with a larger τ and do not degrade to behavior policy values (τ = 0.5). Finally, clipped double Q-Learning is crucial for estimating values for a larger τ = 0.9.

heading: 5.3 ONLINE FINE-TUNING AFTER OFFLINE RL
text: The policies obtained by offline RL can often be improved with a small amount of online interaction. IQL is well-suited for online fine-tuning for two reasons. First, IQL has strong offline performance, as shown in the previous section, which provides a good initialization. Second, IQL implements a weighted behavioral cloning policy extraction step, which has previously been shown to allow for better online policy improvement compared to other types of offline constraints (Nair et al., 2020). To evaluate the finetuning capability of various RL algorithms, we first run offline RL on each
dataset, then run 1M steps of online RL, and then report the final performance. We compare to AWAC (Nair et al., 2020), which has been proposed specifically for online finetuning, and CQL (Kumar et al., 2020), which showed the best performance among prior methods in our experiments in the previous section. Exact experimental details are provided in Appendix C. We use the challenging Ant Maze D4RL domains (Fu et al., 2020), as well as the high-dimensional dexterous manipulation environments from Rajeswaran et al. (2018), which Nair et al. (2020) propose to use to study online adaptation with AWAC. Results are shown in Table 2. On the Ant Maze domains, IQL significantly outperforms both prior methods after online finetuning. CQL attains the second best score, while AWAC performs comparatively worse due to much weaker offline initialization. On the dexterous hand tasks, IQL performs significantly better than AWAC on relocate-binary-v0, comparably on door-binary-v0, and slightly worse on pen-binary-v0, with the best overall score.

heading: 6 CONCLUSION
text: We presented implicit Q-Learning (IQL), a general algorithm for offline RL that completely avoids any queries to values of out-of-sample actions during training while still enabling multi-step dynamic programming. To our knowledge, this is the first method that combines both of these features. This has a number of important benefits. First, our algorithm is computationally efficient: we can perform 1M updates on one GTX1080 GPU in less than 20 minutes. Second, it is simple to implement, requiring only minor modifications over a standard SARSA-like TD algorithm, and performing policy extraction with a simple weighted behavioral cloning procedure resembling supervised learning. Finally, despite the simplicity and efficiency of this method, we show that it attains excellent performance across all of the tasks in the D4RL benchmark, matching the best prior methods on the MuJoCo locomotion tasks, and exceeding the state-of-the-art performance on the challenging ant maze environments, where multi-step dynamic programming is essential for good performance.

heading: A PROOFS
text: A.1 PROOF OF LEMMA 2
Proof. We can rewrite Vτ1(s) as
Vτ1(s) = E τ1 a∼µ(·|s)[r(s, a) + γEs′∼p(·|s,a)[Vτ1(s ′)]]
≤ Eτ2a∼µ(·|s)[r(s, a) + γEs′∼p(·|s,a)[Vτ1(s ′)]]
= Eτ2a∼µ(·|s)[r(s, a) + γEs′∼p(·|s,a)E τ1 a′∼µ(·|s′)[r(s ′, a′) + γEs′′∼p(·|s′,a′)[Vτ1(s′′)]]
≤ Eτ2a∼µ(·|s)[r(s, a) + γEs′∼p(·|s,a)E τ2 a′∼µ(·|s′)[r(s ′, a′) + γEs′′∼p(·|s′,a′)[Vτ1(s′′)]]
= Eτ2a∼µ(·|s)[r(s, a) + γEs′∼p(·|s,a)E τ2 a′∼µ(·|s′)[r(s ′, a′) + γEs′′∼p(·|s′,a′)Eτ1a′′∼µ(·|s′′)[r(s ′′, a′′) + . . .]]
... ≤ Vτ2(s)

heading: B EXPERIMENTAL DETAILS
text: Experimental details. For the MuJoCo locomotion tasks, we average mean returns overs 10 evaluation trajectories and 10 random seeds. For the Ant Maze tasks, we average over 100 evaluation trajectories. We standardize MuJoCo locomotion task rewards by dividing by the difference of returns of the best and worst trajectories in each dataset. Following the suggestions of the authors of the dataset, we subtract 1 from rewards for the Ant Maze datasets. We use τ = 0.9 and β = 10.0 for Ant Maze tasks and τ = 0.7 and β = 3.0 for MuJoCo locomotion tasks. We use Adam optimizer (Kingma & Ba, 2014) with a learning rate 3 · 10−4 and 2 layer MLP with ReLU activations and 256 hidden units for all networks. We use cosine schedule for the actor learning rate. We parameterize the policy as a Gaussian distribution with a state-independent standard deviation. We update the target network with soft updates with parameter α = 0.005. And following Brandfonbrener et al. (2021) we clip exponentiated advantages to (−∞, 100]. We implemented our method in the JAX (Bradbury et al., 2018) framework using the Flax (Heek et al., 2020) neural networks library.
Extended results on Locomotion and Ant Maze tasks. We present learning curves for MuJoCo locomotion tasks in Fig. 4. We also present results on Locomotion and Ant Maze for different values of τ in Fig. 5 and Table 3. We want to emphasize that τ = 0.5 corresponds to using the mean squared error instead of expectile regression.
Results on Franca Kitchen and Adoit tasks. For Franca Kitchen and Adroit tasks we use τ = 0.7 and the inverse temperature β = 0.5. Due to the size of the dataset, we also apply Dropout (Srivastava et al., 2014) with dropout rate of 0.1 to regularize the policy network. See complete results in Table 4.

heading: C FINETUNING EXPERIMENTAL DETAILS
text: For finetuning experiments, we first run offline RL for 1M gradient steps. Then we continue training while collecting data actively in the environment and adding that data to the replay buffer, running 1 gradient update / environment step. All other training details are kept the same between the offline RL phase and the online RL phase. For dextrous manipulation environments (Rajeswaran et al., 2018), we use τ = 0.8 and β = 3.0, 25000 offline training steps, and add Gaussian noise with standard deviation σ = 0.03 to the policy for exploration.
For baselines we compare to the original implementations of AWAC (Nair et al., 2020) and CQL (Kumar et al., 2020). For AWAC we used https://github.com/rail-berkeley/ rlkit/tree/master/rlkit. We found AWAC to overfit heavily with too many offline gradient steps, and instead used 25000 offline gradient steps as in the original paper. For the dextrous manipulation results, we report average return normalized from 0 to 100 for consistency, instead of success rate at the final timestep, as reported in Nair et al. (2020). For CQL, we used https://github.com/aviralkumar2907/CQL. Our reproduced results offline are worse than the reported results, particularly on medium and large antmaze environments. We were not able to improve these results after checking for discrepancies with the CQL paper authors and running CQL with an alternative implementation (https://github.com/tensorflow/agents). Thus, although for offline experiments (Table 1) we report results from the original paper, for finetuning experiments we did not have this option and report our own results running CQL in Table 2.

heading: D CONNECTIONS TO PRIOR WORK
text: In this section, we discuss how our approach is related to prior work on offline reinforcement learning. In particular, we discuss connections to BCQ Fujimoto et al. (2019).
Our batch constrained optimization objective is similar to BCQ (Fujimoto et al., 2019). In particular, the authors of BCQ build on the Q-learning framework and define the policy as
π(s) = argmax a
s.t.(s,a)∈D
Q(s, a). (8)
Note that in contrast to the standard Q-learning, maximization in Eqn. (8) is performed only over the state-action pairs that appear in the dataset. In Fujimoto et al. (2019), these constraints are implemented via fitting a generative model µ(·|s) on the dataset, sampling several candidate actions from this generative model, and taking an argmax over these actions:
π(s) = argmax {ai|ai∼µ(·|s),i=1...N} Q(s, ai).
However, this generative model can still produce out-of-dataset actions that will lead to querying undefined Q-values. Thus, our work introduces an alternative way to optimize this objective without requiring an additional density model. Our approach avoids this issue by enforcing the hard constraints via estimating expectiles. Also, it is worth mentioning that a number of sampled actions N in BCQ has similar properties to choosing a particular expectile τ in our approach.
Note that our algorithm for optimal value approximation does not require an explicit policy, in contrast to other algorithms for offline reinforcement learning for continuous action spaces (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Wu et al., 2019; Kostrikov et al., 2021; Kumar et al., 2019; 2020). Thus, we do not need to alternate between actor and critic updates, though with continuous actions, we must still extract an actor at the end once the critic converges.
E DIFFERENT ESTIMATORS OF V (s)
We also evaluate different ways to estimate the value function V (s) (Table 5). We compare V (s) learned with expectile regression as in IQL with V (s) estimated with several samples from the learned policy as in ABM (Siegel et al., 2020). In particular, we use N = 20 to estimate the value function.

Review summaries:
This paper proposes a new offline RL algorithm that uses in-sample policy evaluation and advantage-weighted regression during policy improvement. Particularly, it utilizes sarsa-like TD to update Q-function and avoids the query value of unseen actions during training. To evaluate their proposed method, they used the D4RL benchmark to compare with previous offline RL methods. In addition, they show results for fine-tuning of learned offline-policies during online deployment.  

This paper proposes an algorithm In-Sample Q-Learning (IQL), which learns a Q- and value function under behavior policy and applies advantage-weighted behavior cloning for policy learning. In one sentence, the main idea of the paper is to learn an approximately optimal Q/value function for better behavior cloning. The expectile regression instead of MSE is applied to find optimal Q-function of behavior policy.  

This paper proposes an offline RL algorithm named IQL, which generalizes between Bellman expectation equation and Bellman optimality equation with expectile regression. Expectile regression assigns low weights for low-performing samples and assigns high weights for high-performing samples to learn the expectile. By performing $\tau$-expectile regression on the randomness of action in the dataset (from the sampling process of behavior policy), IQL is able to obtain a value function that is $\tau$ optimal. IQL is generalizing SARSA and Q-learning in the sense that SARSA corresponds to IQL with $\tau=0.5$ and Q-learning corresponds to IQL with $tau=1.0$. It offers very stable learning of a near-optimal value since it is only using in-distribution samples and the effective dataset size can be controlled through $\tau$. In the experiments, the paper has shown that IQL can obtain state-of-the-art performance.