Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Authors: Mitsuhiko Nakamoto
Yuexiang Zhai
Anikait Singh
Max Sobol Mark
Yi Ma
Chelsea Finn
Aviral Kumar
Sergey Levine
Venue: NeurIPS 2023 poster (2023)
Keywords: offline reinforcement learning
online fine-tuning
TL;DR: We propose a method for acquiring offline RL initializations that facilitate fast online fine-tuning

Abstract: A compelling use case of offline reinforcement learning (RL) is to obtain a policy initialization from existing datasets followed by fast online fine-tuning with limited interaction. However, existing offline RL methods tend to behave poorly during fine-tuning. In this paper, we devise an approach for learning an effective initialization from offline data that also enables fast online fine-tuning capabilities. Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being calibrated, in the sense that the learned Q-values are at a reasonable scale. We refer to this property as calibration, and define it formally as providing a lower bound on the true value function of the learned policy and an upper bound on the value of some other (suboptimal) reference policy, which may simply be the behavior policy. We show that offline RL algorithms that learn such calibrated value functions lead to effective online fine-tuning, enabling us to take the benefits of offline initializations in online fine-tuning. In practice, Cal-QL can be implemented on top of the conservative Q learning (CQL) for offline RL within a one-line code change. Empirically, Cal-QL outperforms state-of-the-art methods on 9/11 fine-tuning benchmark tasks that we study in this paper. Code and video are available at https://nakamotoo.github.io/Cal-QL

Cal-QL: Calibrated Offline RL Pre- Pre-Training for Efficient Online Fine-Tuning
Mitsuhiko Nakamoto
Yuexiang Zhai
Anikait Singh
Max Sobol Mark
Yi Ma¹ Chelsea Finn
Aviral Kumar"
Sergey Levine
UC Berkeley Stanford University
Abstract
A compelling use case of offline reinforcement learning (RL) is to obtain policy initialization from existing datasets followed by fast online fine-tuning with limited interaction. However, existing offline RL methods tend to behave poorly during fine- tuning. In this paper, we study the fine-tuni tin-tuning problem the context of conservative offline RL methods and we devise an approach for learning an effective initialization from offline data that also enables fast online tine-tuning capabilities. Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also ensuring that the learned Q-values are at reasonable scale. We refer to this property as calibration, and define formally as providing a lower bound on the true value function of the learned policy and an upper bound on the value of some other (suboptimal) reference policy, which may simply be the behavior policy. We show that conservative offline RL algorithm that also learns a calibrated value function leads to effective online fine-tunin,, enabling us to take the benefits of offline initializations in online fine-tuning: In practice, Cal-QL be implemented on top of the conservative Q lcarning (CQL) [32] for offline RL within one-line code change. Empirically, Cal-QL outperforms state-of-the-art methods on 9/11 fine-tuning benchmark tasks that we study in this paper. Code and video available https: : //nakamaoooooooooooo github. Tambb.o/Cal--p0
Introduction
Modern machine learning successes follow a common recipe: pre-training models on general-purpose, Internet-ccale data, followed by fine-tuning the pre-trained initialization limited amount of data for the task of interest [22, 7]. How can we translate such a recipe to sequential decision-making problems? A natural way to instantiate this paradigm is utilize offline reinforcement learning (RL) [37] for initializing value functions and policies from static datasets, followed by online fine- tuning improve this initialization with limited active interaction. If successful, such a recipe might enable efficient online RL with much fewer samples than current methods that learn from scratch.
Many algorithms for offline RL have been applied to online fine-tuning. Empirical results across such works suggest a counter- inntuer-it trend: policy initializations obtained from more effective offline RL methods tend exhibit worse online fine-tuning performance, even within the same task (see Table 2 of Kostrikov et al. [31] & Figure 4 of Xiao et al. [57]). On the other end, on- line RL methods training from scratch (or RL from demonstrations [53], where replay buffer is seeded with the offline data) seem to improve online at significantly faster rate. However,
"Equal contributions. Corresponding authors: Mitsuhiko Nakamoto, Yuexiang Zhai, and Aviral Kumar (Inakamoto, simonzhai) @berkeley.edu, wwiralku@andrew.cml
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

these online methods require actively collecting data by rolling out policies from scratch, which inherits similar limitations to naive online RL methods in problems where data collection ex- pensive or dangerous. Overall, these results suggest that it is challenging to devise an offline RI. algorithm that both acquires a good initialization from prior data and also enables efficient fine-tuning.
How can we devise a method to learn an effec- tive policy initialization that also improves dur- ing fine-tuning? Prior work [32. 6] shows that one can learn a good offline initialization by op- timizing the policy against a conservative value function obtained from an offline dataset. But, as we show in Section 4.1, conservatism alone is insufficient for efficient online fine-tuning, Con- servative methods often tend to "unlearn" the policy initialization learned from offline data and waste samples collected via online interac- tion in recovering this initialization. We find that the "unlearning" phenomenon is conse- quence of the fact that value estimates produced via conservative methods can be significantly lower than the ground -round-truth return of any valid policy. Having Q-value estimates that do not lie on a similar scale as the return of a valid policy is problematic. Because once fine-tuning begins, actions executed in the environment for exploration that are actually worse than the pol- icy learned from offline data could erroneously appear better, if their ground-truth return value is larger than the learned conservative value estimate. Hence, subsequent policy optimization will degrade the policy performance until the method recovers.
Fast Sample-Effiie Efficient Online Fine-Tuning
Calbrated (uan)) Pror Aeyotten Pror 3
Return
Idea
Offline Pre-training
Online Fine-Tuning
Figure 1: We study offline RL pre-training followed by online RL fine-tuning. Some prior offiine RL meth- ods tend exhihit slow performance improvement in this setting (yellow). resulting in worse asymptotic per- formance. Others suffer from initial performance degra- dation once online fine-tuning begins (red). resulting in a high cumulative regret. We develop an approach that "calibrates" the learned value function attain fast improvement with a smaller regret (bluc).
If we ensure that the conservative value estimates learned using the offline data are calibrated, meaning that these estimates are on a similar scale as the true return values, then we can avoid the unlearning phenomenon caused by conservative methods (see the formal definition in 4.1). Of course, we cannot enforce such a condition perfectly, since would require eliminating all errors in the value function. Instead, we devise method for ensuring that the learned values upper bound the true values of some reference policy whose values can be estimated more easily (e.g.. the behavior policy), while still lower bounding the values of the learned policy. Though this does not perfectly ensure that the learned values are correct, we show that it still leads to sample-efficient online fine-tuning. Thus, our practical method, calibrated Q-learning (Cal-QL.). learns conservative value functions that "callibrated" against the behavior policy, via a simple modification to existing conservative methods.
The main contribution of this paper is Cal-QL, a method for acquiring an offline initialization that facilitates online fine-tuning. Cal-QL aims to learn conservative value functions that are calibrated with respect to a reference policy (e.g., the behavior policy). Our analysis of Cal-QL shows that Cal-QL attains stronger guarantees on cumulative regret during fine-tuning. In practice, Cal-QL can be implemented on top of conservative Q-learning [32], prior offline RL method, without any additional hyperparameters. We evaluate Cal-QL across a range of benchmark tasks from [10], [51] and [44], including robotic manipulation and navigation. We show that Cal-QL. matches or outperforms the best methods on all tasks, in some cases by 30-40%.
2 Related Work
Several prior works suggest that online RL methods typically require a large number of samples [50, 54, 61, 26, 64, 18. 38] to learn from scratch. We can utilize offline data to accelerate online RL algorithms. Prior works do this in variety of ways: incorporating the offline data into the replay buffer of online RI. [48, 53, 23, 52], utilizing auxiliary behavioral cloning losses with policy gradients [46, 27, 67, 66]. or extracting high-level skill space for downstream online RI. [17, 1]. While these methods improve the sample efficiency of online RL from scratch, as we will also show in our results, they do not climinate the need to actively roll out poor policies for data collection. To address this issue, different line of work first runs offline RL for learning a good policy and value initialization from the offline data, followed by online fine-tuming [45, 30, 41, 3, 56, 36, 42]. These

---

approaches typically employ offline RL methods based on policy constraints or pessimism [12, 49, 16, 15, 30, 51, 36] on the offline data, then continue training with the same method on combination of offline and online data once fine-tuning begins [43, 28, 62, 32, 4]. Although pessimism is crucial for offline RI. (25, 6], using pessimism or constraints for fine-tuning [45, 30, 41] slows down fine-tuning or leads to initial unlearning, as we will show in Section 4.1. In effect, these prior methods either fail to improve as fast as online RL or lose the initialization from offline RL. We aim to address this limitation by understanding some conditions on the offline initialization that enable fast fine-tuning.
Our work is most related to methods that utilize a pessimistic RL algorithm for offline training but incorporate exploration in fine-tuning [36, 42, 56]. In contrast to these works, our method aims to learn a better offline initialization that enables standard online fine-tuning. Our approach fine-tunes naively without ensembles [36] exploration [42] and, we show in our experiments, this alone is enough to outperform approaches that employ explicit optimism during data collection.
3 Preliminaries and Background
The goal in RL is to learn the optimal policy for an MDP M = (S, the state and action spaces. P(s'|s, a) and r(s, a) are the dynamics and reward functions. p(s) denotes the initial state distribution. 7 E (0, 1) denotes the discount factor. Formally, the goal is to learn policy π : S + that maximizes cumulative discounted value function, denoted by (*) = 1-
S, denote
1- E, (a h"r(ss, 4s) s). The Q-function of given policy т is defined as
Ea, (s,, at) |000 s, ag = a), and we use Q7 to denote the estimate of the Q-function of a policy π as obtained via a neural network with parameters e.
Given access to an offline dataset D {(ss,, a, r, r,,') collected using a behavior policy TT. we aim to first train a good policy and value function using the offlin dataset D alone, followed by an online phase that utilizes online interaction in M. Our goal during fine-tuning is to obtain the optimal policy with the smallest number of online samples. This can be expressed as minimizing the
cumulative regret over rounds of online interaction: Reg(K) (K) := Eaup Σx=1
V*(s) V (s)
As we demonstrate in Section 7, existing methods face challenges in this setting.
Our approach will build on the conservative Q-learning (CQL.) [32] algorithm. CQL. imposes an additional regularizer that penalizes the learned Q-function on out-of -uitof-distriuution (OOD) actions while compensating for this pessimism on actions seen within the training dataset. Assuming that the value function is represented by a function, Qe. the training objective of CQL is given by
1 2 B [Qu(s,a) - B (().
min a (E (EawDaawa [Qa(s, a)] - Ex (Qu(s, a)) + 0
Conservative regularizer R()))
(3.1)
where Bddh ((s, a) is the backup operator applied to a delayed target Q-network, Q: B ((s =
r(s, a) + (a(")) a')]. The second term is the standard TD error [40, 13, 20J. The first term (() (in blue) is a conservative regularizer that aims to prevent overestimation in the Q-values for OOD actions by minimizing the Q-values under the policy <(a|s), and counterbalances by maximizing the Q-values of the actions in the dataset following the behavior policy TT..
4 When Can Offline RL Initializations Enable Fast Online Fine-Tuning?
A starting point for offline pre-training and online fine-tuning is to simply initialize the value function with one that is produced by an existing offline RL method and then perform fine-tuning. However, we empirically find that initializations learned by many offline RI. algorithms can perform poorly during fine-tuning. We will study the reasons for this poor performance for the subset of conservative methods to motivate and develop our approach for online fine-tuning, calibrated Q-learning.
4.1 Empirical Analysis
Offline RL followed by online fine-tuning typically poses non-trivial challenges for a variety of methods. While analysis in prior work [45] notes challenges for a subset of offline RL methods, in
Figure 2, we evaluate the fine-tuning performance of a variety of prior offline RL methods (CQL [32]. IQL [30], TD3+BC [11], AWAC [45]) on a particular diagnostic instance of visual pick-and-place task with a distractor object and sparse binary rewards [51], and find that all methods struggle to attain the best possible performance, quickly. More details about this task are in Appendix B.
3

---

1.00
Average Q-Value
Normalized Score
10
0.75
20-
0.50
30
0.25
40
0.00
0
100000
200000
100000
200000
Training Steps
Training Steps
Figure 3: The evolution of the average Q-value and the success rate of CQL over the course of offline pre-training and online fine-tuning. Fine-tuning begins 50K steps. The red-colored part denotes the period performance recovery which also coincides with the period of Q-value adjustment.
While the offline Q-function initialization obtained from all methods attains a similar (normalized) return of around 0.5, they suffer from difficulties during fine-tuning: TD3+BC. IQL. AWAC attain slow asymptotic performance and CQL unlearns the offline initialization, followed by spending a large amount of online interaction to recover the offline performance again, before any further improvement. This initial unlearning appears in multiple tasks as we show in Appendix F. In this work, we focus on developing effective fine-tuning strategies
10
Normalized Score
02-
0.4
CQUL
02
IQL. AWAC
top of conservative methods like CQL. To do so, we next aim to understand the potential reason behind the initial unlearning in CQL.
Environment
Steps
X104
Figure 2: Multiple prior offline RL al- gorithms suffer from difficulties during fine-tuning including poor asymptotic per- formance and initial unlearning.
Why does CQL unlearn initially? To understand why CQL unlearns initially, we inspect the learned Q-values aver- aged over the dataset in Figure 3. Observe that the Q-values learned by CQL in the offline phase much smaller than their ground-truti value (as expected), but these Q- Q-values drastically jump and adjust in scale when fine- one-tuning begins. In fact, we observe that performance recovery (red segment Figure 3) coincides with a period where the range of Q-values changes to match the true range. This is as expected: as a conservative Q-function experiences new online data, actions much worse than the offline policy on the rollout states appear to attain higher rewards compared to the highly underestimated offline Q-function, which in turn deceives the policy optimizer into unlcarning the initial policy. We illustrate this idea visually in Figure 4. Once the Q-function has adjusted and the range of Q-values closely matches the true range, then fine-tuning can proceed normally, after the dip.
To summarize, our empirical analysis indicates that methods existing fine-tuning methods suffer from difficulties such as initial unlearning or poor asymptotic performance. In particular, we observed that conservative methods can attain good asymptotic performance, but "waste" samples to correct the learned Q-function. Thus, this paper, we attempt to develop good fine-tuning method that builds on top of an existing conservative offline RL method, CQL, but aims to "calibrate" the Q-function so that the initial dip in performance can be avoided.
4.2 Conditions on the Offline Initialization that Enable Fast Fine-Tuning
Our observations from the preceding discussion motivate two conclusions in regard to the offline Q-initialization for fast fine-tuning: (a) methods that learn conservative Q-functions can attain good asymptotic performance, and (b) if the learned Q-values closely match the range of ground-truth Q-values the task, then online fine-tuning does not need to devote samples unlearn and then recover the offline initialization. One approach to formalize this intuition of Q-values lying on similar scale as the ground-truth Q-function is via the requirement that the conservative Q-values learned by the conservative offline RL method must be lower-bounded by the ground-truth Q-value of a sub-optimal reference policy. This will prevent conservatism from leaming overly small Q-values. We will refer to this property as "calibration" with respect to the reference policy.
Definition 4.1 (Calibration),, An estimated Q-function Q7 for given policy и is said to be calibrated with respect to a reference policy μ if E,,z (s, a)] 2 En-p [Q" a)] == (*(s), Vs E D.
If the learned Q-function Q is calibrated with respect to a policy μ that is worse than n, would prevent unlcarning during fine-tuning that observed in the case of CQL. This is because the

---

policy optimizer would not unlearn π in favor of a policy that is worse than the reference policy f upon observing new online data as the expected value of π is constrained be larger than V Be-= [Q% (s, a)] 2 (*()). Our practical approach Cal-QI. will enforce calibration with respect to a policy ls whose ground-truth value, V*(s), can be estimated reliably without bootstrapping error (e.g., the behavior policy induced by the dataset). This is the key idea behind our method (as we will discuss next) and is visually illustrated in Figure 4.
Cal-QL: Calibrated Q-Learning
Our approach, calibrated Q-learning (Cal-QL) aims learn conservative and calibrated value function initializations from an offline dataset. To this end, Cal-QL builds on CQL [32] and then constrains the learned Q-function to produce Q- Q-values larger than the Q-value of a reference policy j per Definition 4. 1. In principle, our approach can utilize many different choices of reference policies, but for developing practical method, we simply utilize the behavior policy as our reference policy.
Calibrating CQL. We can constrain the learned Q-function Q7 to be larger than V via a simple change to the CQL training objective shown Equation 3.1: masking out the push down of the learned valuue on out-of-distribution (OOD) actions in CQL if the Q-function is not calibrated, i.e. Baa [Q7 (s, a)] S (s)).. Cal-QL modifies the CQL regularizer, R(0) this manner:
Egg Daa- a
[max (Qa(s, V*() Es [Qo(s,a)]
(5.1)
where the changes from standard CQL are depicted in red. As long as a (in Equation 3.1) is large, for any state-action pair where the learned Q-value is smaller than Q" the Q-function in Equation 5.1 will upper bound Q" in a tabular setting. Of course, as with any practical RL method, with function approximators and gradient-based optimizers, we cannot guarantee that we can enforce this condition for every stte-acction pair, but in our experiments, we find that Equation 5.1 is sufficient to enforce the calibration in expectation
the states in the dataset.
Pseudo-code and implementation details. Our implementation of Cal-QL, directly builds on the implementation of CQL from Geng [14]. We present a pseudo-code for Cal-QL in Appendix A. Additionally, we list the hyperparameters a for the CQL algorithm and our baselines for each suite of tasks in Appendix C. Following the protocol in prior work [30, 52], the practical implementation of Cal-QL trains on a mixture of the offline data and the new online data, weighted in some proportion during fine-tuning. To get V*(s), we fit function approximator Q V/" to the return-to-go values via regression, but we observed that also simply utilizing the return-to-ge estimates for tasks that end in terminal was sufficient for our use case. We show Section 7, how this simple one-line change to the objective drastically improves over prior tine-tuning results.
Pre-trained Q-function
Online fine-tuned Q-function
bad action appears optimal under learned Q-function
CQL
update
datesot
action exploration
bad action doesn't appear optimal under learned Q-function
Cal-QL
online uodato
action ouring exploration
ground truth values
learned Q-function
Figure 4: Intuition behind policy unlearning with CQL. and the idea behind Cal-QL. The plot visualizes a slice of the learned Q Q-function and the ground-truth values for given state. Erroneous peaks on suboptimal actions (x-axis) arise when updating CQL Q-functions with online data. This in turn can lead the policy to deviate away from high-reward actions covered by the dataset in favor of erroneous new actions, resulting in deterioration the pre-trained policy. In contrast, Cal-QL corrects the scale of the learned Q-values by using a reference value function, such that actions with worse Q-values than the reference value function do not crroncously appear optimal in fine-tuning.
5

---

6 Theoretical Analysis of Cal-QL
We will now analyze the cumulative regret attained online fine-tuning, when the value function
is pre-trained with Cal-QL, and show that enforcing calibration (Defintion 4.1) 1) leads to a favorable regret bound during the online phase. Our analysis utilizes tools from Song al. [52], but studies the impact of calibration on fine-tuning. We also remark that we simplity the treatment of certain aspects (e.g., how to incorporate pessimism) as it allows us cleanly demonstrate the benefits of calibration.
Notation & terminology. In our analysis, we will consider an idcalized version of Cal-QL for simplicity. Specifically, following prior work [52] under the bilinear model [9], we will operate in a finite-horizon setting with a horizon H. We denote the learned Q-function at each learning iteration & for given (s,a) pair and time-step by Q (s, a). For any given policy म, let CN 2 1 En
denote the concentrability coefficient such that C := JC
VElLoo ". Ba,arry (T 1(s.a)- (s.a))² i.e., a coefficient that quantifies the distribution shift between the policy and the dataset D, in terms of the ratio of Bellman errors averaged under т and the dataset D. Note that C represents the Q-function class and we assume C has a bellman-bilinen rank [9] of d. We also use C# to denote the concentrability coefficient over a subset of calibrated Q-functions w.r.t. a reference policy Eezo' Be, (T (s,))) (=.0) (a.())
I.C. C# == m V VElid Ex,aron (T (s.a)- fx(*))))*
which provides C# C..
Similar to C, let d,, denote the bellman bilinear rank of Cyy the calibrated Q-function class w.r.t. the reference policy Intuitively, we have C₁ C C. which implies that d, S d. The formal definitions are provided in Appendix H.2. We will use n to denote the arg-max policy induced by Q
Intuition. We intuitively discuss how calibration and conservatism enable Cal-QL to attain smaller regret compared to not imposing calibration. Our goal is to bound the cumulative regret of online [V* (so) /=* (so)]. We can decompose this expression into two terms:
fine-tuning, >n Beg
K (s0) max Q7 o))
K
Reg(K)=
Bap=p
Expop
max (6 a) V (so)
(6.1)
miscalibration
overestimation
This decomposition of regret into terms (i) and (ii) is instructive. Term (ii) corresponds to the amount of over-estimation in the learned value function, which is expected to be small if a conservative RL algorithm is used for training. Term (i) is the difference between the ground-truth value of the optimal policy and the learned Q-function and is negative if the learned Q-function were calibrated against the optimal policy (per Definition 4)1.. Of course, this is not always possible because we do not know V* a priori. But note that when Cal-QL utilizes reference policy μ with a high value VH, close to V*, then the learned Q-function Qo is calibrated with respect to Q" per Condition 4.1 and term (i) can still be controlled. Therefore, controlling this regret requires striking balance between learning calibrated (term (i)) and conservative (term (ii) Q-function. We now formalize this intuition and defer the detailed proof to Appendix H.6.
Theorem 6.1 (Informal regret bound of Cal-QL), With high probability, Cal-QL obtains the following bound on total regret accumulated during online fine-tuning:
Reg () = ( min (CH, / dK log ((F)), KE,[V*(so) - v*(ss)] + H drk log (111))
where F is the functional class of the Q-function.
Comparison to Song et al. [52]. Song et al. [52] analyzes an online RL algorithm that utilizes offline data without imposing conservatism or calibration. We now compare Theorem 6.1 to Theorem 1 of Song et al. [52] to understand the impact of these conditions on the final regret guarantee. Theorem 1
of Song et al. [52) presents a regret bound: Reg(K) = Ō (C+ H / dK log ((FI) and we note some improvements our guarantee, that we also verify via experiments in Section 7.3: (a) for the setting where the reference policy μ contains near-optimal behavior, i.c.. -ー- V/ < O(H / log (IFI) /K), Cal-QL can enable a tighter regret guarantee compared to Song et al. [52]; (b) as we show in Appendix H.3, the concentrability coefficient CH' appearing our guarantee no larger than the one that appears in Theorem of Song et al. [52], providing another source of improvement; and (c) finally, in the case where the reference policy has broad coverage and is highly sub-optimal, Cal-QL reverts back to the guarantee from [52], meaning that Cal-QL improves upon this prior work.

---

Cal-qi (Ous)
COL
SAC offine
IQL
Hybri Rl.
AWAC
COL
actrrao diwemee
Neculaa Soree
Lavinntaral Staps
Eastreezeet
Hichesoorgpktot
Narradoon Soits
Suvivarren Styge
Brvinoneet Stege
Buiposeet Stye
Barinossann Stage
yea Ri
Enctronment Shaps
Lantroezent
Sage
Eustrrazannt Strpe
Figure 6: Online fine-tuning after offline initialization on the benchmark tasks. The plots show the online fine-tuning phase after pre-training for each method (except SAC-based approaches which are not pre-trained). Observe that Cal-QL consistently matches or exceeds the speed and final performance of the best prior method and is the only algorithm to do so across all tasks. (6 seeds)
7 Experimental Evaluation
The goal of our experimental evaluation is to study how well Cal-QL can facilitate sample-efficien online fine-tuning. To this end, we compare Cal-QL with several other state-of-the-ae art fine-tuning methods on variety of offline RL benchmark tasks from D4RL [10], Singh al. [51], and Nair et al. [45], evaluating performance before and after fine-tuning. We also study the effectiveness of Cal-QL. on higher-dimensional tasks, where the policy and value function must process raw image observations. Finally, we perform empirical studies to understand the efficacy of Cal-QI. with different dataset compositions and the impact of errors in the reference value function estimation.
Offline RL tasks and datasets. We evaluate Cal-QL. on number of benchmark tasks and datasets used by prior works [30, 45] evalu- ate fine-tuning performance: (1) The AntMaze tasks from D4RL [10] that require controlling an ant quadruped robot to navigate from a start- ing point desired goal location in maze. The reward is +1 if the agent reaches within pre-specified small radius around the goal and O otherwise. (2) The FrankaKitche tchen tasks from D4RL require controlling a 9-DoF Franka robot attain desired configuration of kitchen. To succeed, policy must complete four sub-tasks in the kitchen within a single rollout, and it receives a binary reward of +1/0 for every sub-task completes. (3) Three Adroit dexterous manipulation tasks [47, 30, 45] that require learning complex manipulation skills on 28-DoF five-fingered hand to (a) manipulate a pen in-hand to desired configuration open-binary: (b) open door by unlatching the handle (door-binary) and (c) relocating ball to a desired location welloaate-iinrryy + +1/0 reward if it succeeds
PEL. Figure 5: Tasks: We evaluate Cal-QL on diverse set
of benchmark problems: AntMaze and Frankakitchen domains from [10]. Adroit tasks from [45] a vision- based robotic manipulation task from [34].
The agent obtains sparse binary solving the task. Each of these tasks only provides a narrow offline dataset consisting of 25 demonstrations collected via human teleoperation and additional trajectories collected by a BC policy. Finally, to evaluate the efficacy Cal-QL on a task where we learn from raw visual observations, we study (4) a pick-and-place task from prior work [51, 34] that requires learning to pick ball and place it bowl, in the presence distractors. Additionally, we compared Cal-QL on D4RL locomotion tasks cheetah, hopper, walker) in Appendix D.
7

---

Comparisons, prior methods, and evaluation protocol. We compare Cal-QL to rumning online SAC [21] from scratch, as well as prior approaches that leverage offline data. This includes naively fine-tuning offline RI. methods such as CQL [32] and
as well as fine-tuning with AWAC [45], O3F [42] and online decision transformer (ODT) [65], methods specifically designed for offline RI. followed by online fine-tuning. In addition, we also compare to baseline that trains SAC [21] using both online data and offline data (denoted by "SAC + offline data") that mimics DDPGID [53] but utilizes SAC instead of DDPG, We also compare to Hybrid RL [52], recently proposed method that improves the sample efficiency of the "SAC + offline data" approach, and "CQL+SAC", which first pre-train with CQL and then fine-tuning with SAC on a mixture of offline and online data without conservatism. More details of each method can be found in Appendix C. We present learning curves for online fine-tuning and also quantitatively evaluate each method its ability to improve the initialization learned from offline data measured in terms of (i) final performance after a pre-defined number of steps per domain and (ii) the cumulative regret over the course of online fine-tuning. In Section 7.2, we run Cal-QL with a higher modate-to-date (UTD) ratio and compare it to RLPD [2], a more sample-effccient version of "SAC offline data".
7.1 Empirical Results
We first present a comparison of Cal-QL in terms of the normalized performance before and after fine-tuning in Table and the cumulative regret in a fixed number of online steps in Table 2. Following the protocol of [10], we normalize the average return values for each domain with respect to the highest possible return (+4 in FrankaKitchen; +1 in other tasks; see Appendix C.1 for more details).
Cal-QL, improves the offline initialization significantly. Observe in Table and Figure that while the performance of offline initialization acquired by Cal-QL. is comparable to that of other methods such as CQL and IQL., Cal-QI. is able to improve over its offline initialization the most by 106.9 % in aggregate and achieve the best fine-tuned performance in 9 out of 11 tasks.
Cal-QL. enables fast fine-tuning, Observe in Table 2 that Cal-QL. achieves the smallest regret on 8 out of 11 tasks, attaining an average regret of 0.22 which improves over the next best method (IQL) by 42 %. Intuitively, this means that Cal-QL does not require running highly sub-optimal policies. In tasks such as relocate- -inttyy..brrr...e. Cal-QL enjoys the fast online learning benefits associated with naïve online RL methods that incorporate the offline data in the replay buffer (SAC + offline data and Cal-QL are the only two methods attain a score of 2 90% on this task) unlike prior offline RL methods. As shown in Figure 6, in the kitchen and antmaze domains, Cal-QL brings the benefits of fast online learning together with a good offline initialization, improving drastically on the regret metric. Finally, observe that the initial unlearning at the beginning of fine-tuning with conservative methods observed in Section 4. is greatly alleviated in all tasks (see Appendix F for details).
Task
COL
IQL
AWAC
O3F
ODT
CQL-SAC
Hybmid SRL
SACeod
SAC
Cal-QL (Ouns)
large-di veras
25
59
19 -
10 -3
-300
->
95
Jarge-play
34 + 76
41 51 51
00-00 00
68 - 01
00
- 00
21
00
00
+000
+ -00
26 90
mediun diverse
65 98
70 -2
00
92
00 00
98
75 98
modiun ppaay
72 +
+
parsial
71 75
40 60
01 13
11 + 22
71 00
10
-007
+ 030
67
- 79
mixed
56
18-+48 48
12
06
38 80
complete
57 57 50
06
+
68
pen
55 13
88 92
88 92
91 + 89
48 10
+17
+ 11
79 99
door
22 88
+8
29 13
01
66
+39
35 92
relocate
45
35
-
ranipolation
97
49
70
42
-41
00
+1
-> 01
49 99
average
71
50
30-440 69
16 26-+-20
44 45
00 - OZ
42 29
24
- 23
-3 04
44 90
Improvemeent
+71.0%
37.7%
23.7%
3.0%%
N/A
300.5%
NIA
N/A
N/A
106.99
Table 1: Normalized score before & after online fine-tuning. Observe that Cal-QL AL-QL improves over the best prior fine- tuning method and attains a much larger performance improvement over the course of online fine-tuning. The numbers represent the normalized score out of 100 following the convention in [10].
7.2 Cal-QL. With High Update-to-Data (UTD) Ratio
We can further enhance the online sample efficiency of Cal-QL by increasing the number of gradient steps per environment step made by the algorithm. The number of updates per environment step is usually called the update-to- data (UTD) ratio. In standard online RL, running off policy Q-learning with a high UTD value (e.g., 20, compared to the typical value of 1) often results in challenges pertaining to overfitting [39, 5, 2, 8). As expected, we noticed that running Cal-QL with a high UTD value also leads these overfitting challenges. To address these challenges in high UTD settings, we combine Cal-QI. with the Q-function architecture in recent work, RLPD [2] (i.e., we utilized layer normalization in the Q-function and ensembles akin to Chen et al. [5]), that attempts to tackle
8

---

Task
CQL
IQL
AWAC
03F
ODT
CQL+SAC
Hybrid
SAC+od
SAC
Cal-QL (Ours)
large-diverse diverse
0.35
0.46
1,00
0.62
01.98
0.99
1.00
1.00
1.00
0.20
large-play
1,00
0.91
1,00
1.00
1.00
0.28
mediun-diverrs
0.06
0.08
0.99
0.03
0.95
0.06
0.98
0.77
1.00
0.05
mediun-play
0.09
0.10
0.99
0.04
0.96
0.06
0.90
0.47
1.00
0.07
partial
0.31
0.49
0.89
0.78
0.97
0.98
0.98
0.92
0.27
mixed
0.60
0.88
0.72
0.97
0.99
1.00
0.27
complete
0.70
0.53
0.97
0166
0.99
0.99
0.96
0.91
0.44
pen
0.36
0.11
0.12
0.13
0.90
0.56
0.75
0.87
0.11
door
0,36
0.25
0.81
0.82
0.23
0.35
0.60
0.94
0.23
relocate
0,71
0.74
0.95
0.71
0,30
0.89
1.00
0.43
manipulation
0.32
0.38
1.00
1.00
0.11
average
0.41
0.38
0.82
0.54
0.97
0.69
0.82
0.86
0.96
0.22
Table 2: Cumulative regret averaged over the steps of fine-tuning, The smaller the better and 1.00 is the
worst. Cal-QL attains the smallest overall regret, achicving the best performance among 8 11 tasks.
Cal QL (UTD-2))
Cal QL (UTD-1)
BLPD (UTD-20)
699 08
435 118
K10 05
Cha 07
Strgo
Steps
Steps
Stepe
Tee Mearyoo
dom Hearystt
Welostt Miarppot
Sension Souae
Figure 7: Cal-QL with UTD=20. Incorporating design choices from RLPD enables Cal-QL to achieve sample-efficient fine-tuning with UTD=200. Specifically, Cal-QL generally attains similar or higher asymptotic performance as RLPD, while also exhibiting smaller cumulative regret. (3 seeds)
overfitting challenges. Note that Cal-QL still first pre-trains on the offlin dataset using Equation 5.1 followed by online fine-tuning, unlike RLPD that runs online RL right from the start. In Figure we compare Cal-QL (UTD 20) with RLPD [2] (UTD 20) and also Cal-QL (UTD- 1) as baseline. Observe that Cal-QL (UTD- = 20) improves over Cal-QL (UTD 1) and training from scratch (RLPD).
7.3 Understanding the Behavior of Cal-QL
In this section, we aim understand the behavior of Cal-QL. by performing controlled experiments that modify the dataset composition, and by investigating various metrics to understand the properties of scenarios where utilizing Cal-QL is especially important for online fine-tuning.
Effeet of data composition. To understand the efficacy of Cal-QI. with different data compositions, it newly constructed fine-tuning task the medium-size AntMaze domain with low-coverage offline dataset, which is generated via a scripted controller that starts from a fixed initial position and navigates the Cal-QL and baseline CQL (for comparison) on this task, alongside the trend of average Q-values over the course of offline pre-training (to the left of the dashed vertical line, before 250 training epochs) and online fine-tuning (to the right of the vertical dashed line, after 250 training epochs), and the trend of bounding rate, i.e., the fraction of transitions in the data-buffer for which the constraint in Cal-QL actively lower-bound, the learned Q-function with the reference value. For comparison, we also plot these quantities for diverse dataset with high coverage on the task (we use the antmaze-medi di verse from Fu et al. [10] as a representative diverse dataset) in Figure 8.
fixed goal position.
Figure 8, we plot the performance of
Observe that for the diverse dataset, both naive CQL and Cal-QL perform similarly, and indeed, the learned Q-values behave similarly for both of these methods. In this setting, online learning doesn't tt spend samples to correct the Q-function when fine-tuning begins leading to a low bounding rate, almost always close to O. Instead, with the narrow dataset, observe that the Q-values learned by

---

Cal-QL. (narrow) Cal-QL. (diverse)
CQL (narrow)
CQL (diverse)
-200
88-
Avosuge: Q-saltes
-500
Dounzeed Date
Noemalizd
0.-
-500
0.4-
-500
200
400
620
-20
430
000
100
200
Epoch
Epoch
Epoch
Figure 8: Performance of Cal-QL with data compositions. Cal-QL most effective with narrow datasets, where Q-values need be corrected the beginning of fine-tuning.
naïve CQL are much smaller, and are corrected once fine-tuning begins. This correction co-occurs with a drop in performance (solid blue line on left), and naive CQL is unable to recover from this drop. Cal-QL which calibrates the scale of the Q-function for many more samples in the dataset, stably transitions to online fine-tuning with no unlearning (solid red line on left).
This suggests that in settings with narrow datasets (e.g., in the experiment above and in the adroit and domains from Figure 6), Q-values learned by naive conservative methods are more likely to be smaller than the ground-truth Q-function of the behavior policy due to function approximation errors. Hence utilizing Cal-QL to calibrate the Q-function against the behavior policy can be significantly helpful. On the other hand, with significantly high-coverage datasets, especially in problems where the behavior policy is also random and sub-optimal, Q-values learned by naive methods are likely to already be calibrated with respect to those of the behavior policy. Therefore no explicit calibration might be needed (and indeed, the bounding rate tends to be very close to shown in Figure 8). In this case, Cal-QL will revert back to standard CQL, case of the diverse dataset above. This intuition is also reflected in Theorem 6.1: when the reference policy j is close a narrow, expert policy, we would expect Cal-QL be especially effective in controlling the efficiency of online fine-tuning.
visual-mast mastimmmaiplllae
observe in the
Estimation errors in the reference value function do not affect performance significantly. In our experiments, we compute the ref- erence value functions using Monte-Carlo return estimates. However, this may not be available in all tasks. How does Cal-QQ allQQL behave when reference value functions must be estimated using the offline dataset itself? To answer this, we ran an experiment on the kitchen domain, where instead of using an estimate for Q" based on the Monte-Carlo return, we train neural network function approximator Q% to ap- proximate Q" via supervised regression on to Monte-Carlo return, which is then utilized by Cal-QL. Observe in Figure 9, that the per- formance Cal-QL largely remains unaltered. This implies as long as we can obtain a reasonable function approximator estimate the Q-function the reference policy (in this case, the behavior policy), errors in the reference Q-function do not affect the performance of Cal-QL significantly.
MN Appeozinstor
Moclee Carlee--rrroo aoturn
Korudize Soore
11.6-
11.2
Euvircamen: Stege
Figure 9: Using a neural net- work approximator for the ref- erence value function performs comparable using the Monte- Carlo return. This indicates that errors in the reference Q-function do not negatively impact the per- formance.
8 Discussion, Future Directions, and Limitations
In this work we developed Cal-QL, a method for acquiring conservative offline initializations that facilitate fast online fine-tuning. Cal-QI. learns conservative value functions that are constrained be larger than the value function of reference policy. This form of calibration allows us to avoid initial unlcarning when fine-tuning with conservative methods, while also retaining the effective asymptotic performance that these methods exhibit. Our theoretical and experimental results highlight the benefit of Cal-QL in enabling fast online fine-tuning. While Cal-QL outperforms prior methods, we believe that can develop even more effective methods by adjusting calibration and conservatism more carefully. A limitation of our work is that we do not consider fine-tuning setups where pre-training and fine-tuning tasks are different, but this is an interesting avenue for future work.
10

---

Acknowledgmentt
This research was partially supported by the Office of Naval Research N00014-21-1-2888 N00014- 22-1-2102, ARO 191----21-1 the joint Simons Foundation-NSF DMS grant #2031899, AFOSR FA9550-22-1-0273, and Tsinghua-Berkehe Shenzhen Institute (TBSI) Research Fund, as well support from Intel and C3.ai, the Savio computational cluster resource provided by the Berkeley Research Computing program, and computing support from Google. We thank Philip J. Ball, Laura Smith, and Ilya Kostrikov for sharing the experimental results of RLPD. AK is supported by the Apple Scholars in AI/M PhD Fellowship. MN is partially supported by the Nakajima Foundation Fellowship. YZ is partially supported by Siemens CITRIS and TBSI research fund. YZ would like to thank Prof. Song Mei for insightful suggestions on the presentation of Theorem 6.1.
References
[1] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611 2020.
[2] P. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02944 2023.
[3] A. Beeson and G. Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning arXiv preprint arXiv:2211.1180) 2022.
[4] Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
[5] X. Chen, C. Wang. 7. Zhou, and K. W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations, 2021. URL https: http///ppwreeeeeeeeeee/ee wwt. httt:/ffrummddddde ft
[6] C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.0244 2022.
[7] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.0400 2018.
[8] P. D'Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh Inter- national Conference Learning Representations, 2023. URL https: : /ttp:://opeeeiieeeeeeee net/ forum?id=0pC- 9aBBVJe. -eBBB---------.
[9] S. Du, S. Kakade, J. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization in In International Conference on Machine Learning, pages 2826 2836. PMLR. 2021.
[10] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[11] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2 06860, 2021.
[12] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint 1ate::222200000 2018.
[13] Fujimoto,, H. van Hoof, and Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), pages 1587-1596, 2018.
[14] X. Geng. Jaxeql: simple implementation of sac and eql in jax. 2022. https://:: com/young- wyyoongggngg////oeoo ggen/JJxXQLL. JaxCQL.
[15] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.

---

[16] Y. Guo, S. Feng, N. Le Roux, E. Chi, H. Lee, and M. Chen. Batch reinforcement learning through continuation method. In International Conference on Learning Representations, 2020.
[17] A. Cupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learnin. arXiv preprint arXiv:1910.11955 2019.
[18] A. Gupta, A. Pacchiano, Y. Zhai, S. M. Kakade, and S. Levine. Unpacking reward shap- ing: Understanding the benefits of reward engineering on sample complexity. arXiv preprint arXiv:2210.09579, 2022.
[19] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In arXiv, 2018. URL https: //ar //arxiv. hatg//df///t /1011. Tww1110010000..1111111
[20] T. Haarmoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. Technical report, 2018.
[21] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arvi: 1812.05905, 2018.
[22] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[23] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, Schaul. B. Piot, D. Horgan, Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[24) C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34: 13418, 2021.
[25] Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[26] S. Kakade and Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning (ICML), volume 2, 2002.
[27] B. Kang, Z. Jie, and J. Feng. Policy optimization with demonstrations. In International conference on machine learning, pages 2469-2478 PMLR, 2018
[28] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning, arXiv preprint arXiv:2005.0595.1 2020.
[29] I. Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms JAX, 10 2021. URL https: //gi htt///gtttttu thub. cww http://kosttttoooo jxxrl.
[30] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprini a /0,00666,, 2021.
[31] I. Kostrikov, J. Tompson, R. Fergus. and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050 2021.
[32] A. Kumma, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 2wXiv:2006.04779 2020.
[33] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine. Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials. arXiv e-prints, art. arXiv:2210.05178. Oct. 2022. doi: 20455500rr.i.22110055555
[34] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine. Pre-training for robots: Offline
rl enables learning new tasks from a handful of trials. arXiv preprint 2r..0:2210005778, 2022.
12

---

[35] T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[36] S. Lee, Y. Sco, K. Lee, P. Abbeel, and J. Shin. Offine-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
[37] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.006443, 2020.
[38] Q. Li, Y. Zhai, Y. Ma, and S. Levine. Understanding the complexity gains of single-task rl with a curriculum. arXiv preprint arKiv:2212. 12809, 2022.
[39] Q. Li, A. Kumar, Kostrikov, and S. Levine. Efficient deep reinforcement learning requires regulating overfitting. In The Eleventh International Conference Learning Representations, 2023. URL https: http://pppeevieeeeeeeee ewwew.rrveww..... net http://oromiddiiie -d-4--1r461-4-
[40] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 150..000000020
[41] J. Lyu, X. Ma, X. Li, and Z. Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.044 04745, 2022.
[42] M. S. Mark, A. Ghadirzadeh, X. Chen, and C. Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
[43] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: 191.:v:912..0072. 2019.
[44] A. Nair, M. Dalal, A. Gupta, and S. Levine, Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09959, 2020. URL tatttss://aattt org/ abs/a/as.......///////////ee
[45] A. Nair, M. Dalal, A. Gupta, and Levine. Accelerating online reinforcement learning with offline datasets, arXiv preprint arXiv:2006.0993 2020.
[46] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1oXiv:1709.10000000 2017.
[47] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
[48] Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
[49] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08330: 2020.
[50] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre. G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[51] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint S01i::2010.14500 2020.
[52] Y. Song, Y. Zhou, A. Sekhari, D. Bagnell, A. Krishnamurthy, and W. Sun. Hybrid RL: Using both offline and online data can make RI. efficient. In The Fleventh International Conference on Learning Representations, 2023. URL https:/ wew.eeew.eeeeeeeee net/ id=yyBis80iUuu.
[53] M. Vecerik, T. Hester, Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 1133::1177.08811111 2017.
13

---

[54] O. Vinyals. I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350-334 2019.
[55] A. Wagenmaker and A. Pacchiano. Leveraging offline data in online reinforcement learning. arXiv preprint
2022.
[56] Wu, H. Wu, Z. Qiu, J. Wang. and M. Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.062.3, 2022.
[57] C. Xiao, H. Wang. Y. Pan, A. White, and M. White. The in-sample softmax for offline reinforcement learning. In International Conference on Learning Representations, 2023, URL https://oppnzrevixx. wtwper/ewweeeeeeeeooi.....ii... d-t
[58] Xie and N. Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pages 550-559. PMLR, 2020.
[59] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. Bellman cooinannconssseett pessimism for offline reinforcement learning. Advances in neural information processing systems. 34, 2021.
[60] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27407, 2021.
[61] D. Ye, G. Chen, W. Zhang, S. Chen, Yuan, B. Liu, Chen, Liu, Qiu, H. Yu, Y. Yin, B. Shi, L. Wang, T. Shi, Q Fu, W. Yang, L. Huang, and W. Liu. Towards playing full moba games with deep reinforcement learning. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 621-632. Curran Associates, Inc., 2020. URL http:: httpt:pep//poooeedding wtwpeeeeeiiinnneriiiiieeeeeeeeee neurips cc/ .dff.
H. Larochelle, M. Ranzato, R. Hadsell, M.
[62] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 2020.
[63] A. Zanette, M. J. Wainwright, and E. Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34: 1 13626-13640, 2021.
[64] Y. Zhai, C. Back, Z. Zhou, J. Jiao, and Y. Ma. Computational benefits of intermediate rewards for goal-reaching policy learning. Journal of Artificial Intelligence Research, 73:847-896, 2022.
[65] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In International Conference on Machine Learning, pages 27042-27059. PMLR, 2022.
[66] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforeement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[67] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills, arXiv preprint arXiv: 09564, 2018.
14

---

Appendices
A
Implementation details of Cal-QL
Our algorithm, Cal-QL is illustrated in Algorithm 1. A python-style implementation is provided in Appendix A.2. The official code and experimental logs available https://gi com/ nakamotoo/ CCalQQL
A.1
Cal-QL Algorithm
We use Jo (0) to denote the calibrated conservative consevaatieeregulliizzz for the Q network update:
JQ = (EawD.owx [max (Q)(s.a).Q((saa)) Es [Q((s.))))
(A.1)
Calibrated conservative regularizer R(O)
E B ((ol(s.))B'Q((.000)0)
(A.2)
Algorithm Cal-QL. pseudo-code
1: Initialize Q-function, Qo. policy, T for step in (1...., Nj do
Train the Q-function using the conservative regularizer in Eq. A.I:
: 1/Q VoJo(
(A.3)
4:
Improve policy To with SAC-style update:
Φ: == 0 + 77 B. (Qo(s log & (a
(A.4)
5: end for
A.2 Python Implementation
Listing 1: Training Q networks given batch of data
q_data critic(batch) ('observationss) batch ['actions
next_diat. actor (batch [next_ pnett_oaaarattiinnss text.pi_actions next_log_pie next_dist... sample ()
target_qual target (batch P'basevvation *], pext.pi.actions)
target_qval batch ('reaaddss self gemns (1 batch ['dees target_qval
td_less mse_loss da_a, target__val)
sapples randon_actions uniform (aum_sanplle. batch_size, action_din), minm-1, max=1) randon_pi 0.5 batch l'actions '], shape [-1]
pi_actions, log-pis (batch (bbbervatooo)))
q_rand_ia critic ['observatiooee] ' aaddon.actions) randon,pi
q_pi_ie ( pi_sctions) log_pis
Cal-OL's modification batch batchl"nc_retura (mup_eaaeun q-pi_is max(q_pi_is, nc_retura)
cat_q concatenate ([q_rand_ia q-pi_ie), пеx aex_axie-True)
cat_q logeumexp (cat_g. axis=0) sue over nun, samples critic_loss td_loss ((cat_q q_data) m)an cql_alpha)
critic_optimizer... zero_grad critic_ (otte__oos.backsarddg backxard pritic_optimizer.etee
15

---

Listing 2: Training the policy (or the actor) given batch of data
distribation of actions
pi_actions, log-pis actor ("observations")
calculate value of actor actions qpi (aatiiebaach(oooorvusssttt qpi qpt.min(axia==0)
actions)
al.) actor_loss - (log-pis self alpha spi). mean
sane objective
optimize loss
actor octor_optinizer... 2eroogrradd
actor_loss. bacc__oos.bacxvard() octor_optimizerrstepp
B Environment Details
B.1
Antmaze
The Antmaze navigation tasks aim to control an 8-DoF ant quadruped robot to move from starting point to a desired goal in a maze. The agent will receive sparse +1/0 rewards depending on whether reaches the goal or not. We study each method on "medium" and "hard" (shown in Figure 5) mazes which are difficult to solve, using the following datasets from D4RL [10]: large-diverse, large-play, medium-divee diverse, and medium um-play. The difference between "diverse" and "play" datasets is the optimality of the trajectories they contain. The "diverse" datasets contain the trajectories commanded to a random goal from random starting points, while the "play" datasets contain the trajectories commanded to specific locations which not necessarily the goal. We used episode length of 1000 for each task. For Cal-QL, CQL, and IQL, we pre-trained the agent using the offline dataset for 1M steps. We then trained online fine-tuning for 1M environment steps for each method.
B.2 Franka Kitchen
The Franka Kitchen domain require controlling a 9-DoF Franka robot to arrange kitchen environment into desired configuration. The configuration is decomposed into 4 subtasks, and the agent will receive rewards 0f +1, +2, +3, or +4 depending on how many subtasks it has managed to solve. To solve the whole task and reach the desired configuration, it is important to learn not only how to solve each subtask, but also to figure out the correct order to solve. We study this domain using datasets with three different optimalities: kitchen- cocchenn-ompleeee lete, The "complete" dataset contains the trajectories the robot performing the whole task completely. The "partial" dataset partially contains some complete demonstrations, while others incomplete demonstrations solving the subtasks. The "mixed" dataset only contains incomplete data without any complete demonstrations, which is hard and requires the highest degree of stitching and generalization ability. We used an episode length of 1000 for each task. For Cal-QL, CQL, and IQL, we pre-trained the agent using the offline dataset for 500K steps. We then performed online fine-tuning for 1. 1.25M environment steps for each method.
and kitchen-mi xed.
B.3 Adroit
The Adroit domain involves controlling 24-DoF shadow hand robot. There are tasks we consider in this domain: pen-binary, relocate-binary, relocate-binars limited set of narrow human expert data distributions (~ 25) with additional trajectories collected by a behavior-cloned policy. We truncated each trajectory and used the positive segments (terminate when the positive reward signal is found) for all methods. This domain has a very narrow dataset distribution and a large action space. In addition, learning in this domain is difficult due to the sparse reward, which leads to exploration challenges. We utilized variant of the dataset used in prior work [44] to have a standard comparison with SOTA offline fine-tuning experiments that consider this domain. For the offline learning phase, we pre-trained online fine-tuning for 300K environment steps for the pen-binary task, and 1M environment steps for the door -iiarb and relocate-be binary tasks. The episode length 100, 200, and 200 for pen-binane door binary, and relocate- relocate-binary respectively.
These tasks comprise a
agent for 20K steps. We then performed
16

---

B.4 Visual Manipulation Domain
The Visual Manipulation domain consists of pick-and-place task. This task a multitask formulation explored in the work, Pre-training for Robots (PTR) [33]. Here each task is defined placing object in a bin, A distractor object was present the scene as an adversarial object which the agent had to avoid picking. There were 10 unique objects and no overlap between the task objects and the interfering/distracoor objects. The episode length is 40. For the offine phase, we pre-trained the policy with offline data for 50K steps. We then performed online fine-tuning for 100K environment steps for each method, taking 5 gradient steps per environment step.
C Experiment Details
C.1 Normalized Scores
The visual-manipulatt
adroit, and an antmaze domains are all goal-oriented, sparse reward tasks. In these domains, we computed the normalized metric as simply the goal achieved rate for each method. For example, in the visual manipulation environment, if the object was placed successfully in the bin, a +1 reward was given to the agent and the task completed. Similarly, for the door-binary task in the adroit tasks, we considered the success rate of opening the door. For the kitchen task, the task to solve a series of 4 sub-tasks that need to be solved an iterative manner. The normalized score is computed simply as
Hiusks solved
total
C.2 Mixing Ratio Hyperparameter
In this work, explore the mixing ratio parameter m. which is used during the online fine-tuning phase. The mixing ratio is either value in the range [0. 1] or the value -1. this mixing ratio is within [0, 1). it represents what percentage of offline and online data is in each batch when fine-tuning. For example, if the mixing ratio m = 0.25, that means for each batch we sample 25% from the oflline data and 75% from online data. Instead, if the mixing ratio is -1, the buffers are appended to each other and sampled uniformly. We present an ablation study over mixing ratio in Appendix G.
C.3 Details and Hyperparameters for CQL and Cal-QL
We list the hyperparameters for CQL and Cal-QL in Table We utilized variant of Bellman backup that computes the target value by performing maximization over target values computed for k actions sampled from the policy at the next state, where we used = 4 in visual pick and place domain and & = 10 in others. In the Antmaze domain, we used the dual version of CQL [32] and conducted ablations over the value the threshold of the CQL regularizer R(()) (target action gap) domain which is not presented in the original paper, swept over the alpha values of a 0.5, 1, 5, 10, and utilized separate a values for offline (a = 5) and online (a 0.5) phases for the final results. We built our code upon the CQL implementation from https : //github. com/ com/youn- geng/JaxCQL JaxCQL [14]. We used a single NVIDIA TITAN RTX chip to each of our experiments.
instead of a. In the visual- iisual-maipulattin
C.4 Details and Hyperparameters for IQL.
We list the hyperparameters for IQL in Table 4. To conduct experiments, used the of- ficial implementation of IQL provided by the authors [30], and primarily followed their recom- mended parameters, which they previously ablated over in their work. In the risual-manipulation domain which is not presented in the original paper, we performed a parameter sweep over expec- tile 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99 and temperature 3 1, 3, 10, 25, 50 and selected the best-performing values of T = 0.7 and B 10 for our final results. In addition, as the second pest-performing method in the Aisual-manipulation domain, we also attempted to use separate values for IQL, for a fair comparison with CQL and Cal-QL. However, we found that it has little to no effect, as shown in Figure 10.
C.5 Details and Hyperparameters for AWAC and ODT
We used the JAX implementation of AWAC from sttps://github. com/ htt///kkotiiko/jjzztz [29]. We primarily followed the author's recommended parameters, where used the Lagrange multiplier
17

---

= 1.0 for the Antmaze and Franka Kitchen domains, and À = 0.3 for the Adroit domain. In the misual-manipllation domain, we performed a parameter sweep over À = 0.1, 0.3, 1, 3,3, 10 and selected the best-performing value of À = 1 for our final results. For ODT, we used the author's official implementation from https: /ettps//giiii //gitt/b..b..... htt.:ttub.co/eeeeeee comm/faceboooreeerrchh... one//nnnine.d with the author's recommended parameters they used in the Antmaze domain. In addition, in support of our result of AWAC and ODT (as shown in Table 1), the poor performance of Decision Transformers and AWAC in the Antmaze domain can also be observed in Table and Table 2 of the IQL paper [30].
C.6 Details and Hyperparameters for SAC, SAC+ + Offline Data, Hybrid RL and CQL + SAC
We used the standard hyperparameters for SAC as derived from the original implementation in [19]. We used the same other hyperparameters as CQL and Cal-QL. We used automatic entropy tuning for the policy and critic entropy terms, with a target entropy of the negative action dimension. For SAC, the agent is only trained with the online explored data. For SAC + Offline Data, the offline data and online explored data combined together and sampled uniformly. For Hybrid RI., we use the same mixing ratio used for CQL and Cal-QL presented in Table 3. For CQL SAC, we first pre-train with CQL and then run online fine-tuning using both offline and online data, also using the same mixing ratio presented in Table 3.
Table 3: CQL, Cal-QL Hyperparameters
Hyperparameters
Adroit
Kitchen
Antmaze
Manipulation
& target action gap mixing ratio
5 (online: 0.5)
0.8
1, 0.0.25, 0.5
-1 0.25, 0.5
0.5
0.2, 0.5, 0.7, 0.9
Table 4: IQL Hyperparameters
Hyperparameters
Adroit
Kitchen
Antmaze
Manipulation
expectile T
0.8
0.7
0.9
0.7
temperature 3
0.5
10
10
mixing ratio
-1. 0.2. 0.5
-1, 0.25, 0.5
0.5
0.2, 0.5, 0.7
D D4RL locomotion benchmark
In this section, we further compare Cal-QL with previous methods on the D4RL locomotion bench- marks. Since the dataset for these tasks does not end in terminal, we estimate the reference value functions using a neural network by fitting SARSA Q-function to the offline dataset. We present the normalized scores before and after fine-tuning in Table 5, and the cumulative regret in Table 6. Overall, we observe that Cal-QL outperforms state-of of-e----theerrrr art methods such as IQL and AWAC, as well as fast online RL methods that do not employ pre-training (Hybrid RL), performing comparably to CQL.
Task
CQL
IQL
AWAC
Hybrid RL
Cal-QL (Ours)
belfcheetah wew.cheeb-eeeeer-r222
0.79 -3 1.02
0,95 U.54
0.69 - 0.65
NAA 0.84
0.85 - 1.00
halfcheetah-nod serraah-nddduu---n
0.59 - 1.00
0.86 + 0.57
0.72 - 0.77
NA + 0.86
0.54 - 0.99
0.43 +
0.47
N/
0.51 0.93
halfcheetab-nediue
0.53 - 0.97
0.47 0.43
0.49 + 0.57
N/A + 0.88
0.52 - 0.93
malfcheetah-randonn
0.36 1.01
0.11 0.54
- 0.37
N/A
0.33 - 1.04
hopper-export
1.00 0.73
1.02 + 0.52
1.12 - 1111
N/A 0.54
0.58 - 0.75
hoppar-ndduumexpprre
0.86 136--900 0.92
0.07 - 0.81
0.30 - 1.03
N/A - 1.00
0.69 - 0.76
hoppar-mediu--rellayy-2
0.69 1.11
0,76 0.86
1.08
N/A
0.76 1,10
hopper me2ium-2
0.78 - 0.99
0.57 + D.26
0.58 0.90
N/A + 1.06
0.89 - 0.98
hopper-randon--2
0.09 - 1.08
0.08 + 0.64
0.43
N/A +
0.10 - 0.79
Lllkerdd-exper---g
1.08 1.12
1.09 1.02
1.11 1.24
N/A 1.12
0.92 1.00
vnl walbaarat-neddu---pprr....
1.00 - 0.56
1.09 - 1.06
0.86 + 1.16
N/A - 0.95
0.96 0.77
0.76 0.92
0.63 +
0.94
N/A 1.03
0.52 0.99
velker2d
0.80 + 1.11
0.81 + 1.04
0.75 0.98
N/A + 0.86
0.75 - 1.03
valker24
0.05 + 0.85
0.06 + 0.09
0.04
N/A +
0.04 0.49
average
0.66 - 0.96
0.00 - 0.65
0.57 + 0.78
N/A + 0.89
0.60 - 0.90
Table 5: Normalized score hefore & after online fine-tuning on D4RI. locomotion tasks
18

---

Tusk
CQL
IQL
AWAC
Hybrid RL
Cal-QL (Ours)
hal
0.03
01.7
0.28
0.49
01.88
0.05
0.42
0.24
0.49
0.07
holfcheetah rediu--replay-v
0.14
0.65
0.56
0.29
0.16
0.48
0.31
0.17
salffhheetah-randdn-22
0.11
0.54
0.67
0.33
0.10
hopper-expertt-22
0.17
0.58
-0.07
0.37
0.22
hopper-nadiu-expprrss
0.09
0.32
-0.02
0.32
0.17
hogper-noddiu-rrpppaay-yy
0.05
0.12
hopper-mediim--2
0.04
0.73
0.08
0.26
0.05
hopper-randon-122
0.17
0.54
0.74
0.37
0.27
walker2d-expertt
-0.02
0.15
-0.18
0.11
0.15
0.02
-0.14
0.17
0.10
wmlker2ä--eeiim-rrplaag
0.06
0.10
0.19
0.30
0.13
walker2d-medium-Ve
0.11
0.12
0.07
0.29
0,22
randoon-2-
0.53
0.92
0.96
0.56
0.49
average
0.12
0.42
0.26
0.33
0.17
Table 6: Cumulative regret on D4RL, locomotion tasks. The smaller the better and 1.00 the worst.
E Extended Discussion on Limitations of Existing Fine-Tuning Methods
In this section, we aim to highlight some potential reasons behind the slow improvement of other methods our empirical analysis experiment Section 4.1, and specifically, we use IQL for the analysis. We first swept over the temperature / values used in the online fine-tuning phase for IQL, which controls the constraint on how closely the learned policy should match the behavior policy. As shown in Figure 10, the change in the temperature has little no effect the sample efficiency. Another natural hypothesis is that IQL improves slowly because we are not making enough updates per unit of data collected by the environment. To investigate this, ran IQL with (a) five times as many gradient steps per step of data collection (UTD = 5), and (b) with a more aggressive policy update. Observe in Figure 11 that (a) does not improve the asymptotic performance of IQL, although does improve CQL meaning that there is room for improvement on this task by making more gradient updates. Observe in Figure 11 that (b) often induces policy unlearning, similar to the failure mode in CQL. These two observations together indicate that policy constraint approach can slow down learning asymptotically, and we cannot increase the speed by making more aggressive updates as this causes the policy to find erroneously optimistic out-of-distribution actions, and unlear the policy learned from offline data.
0.6-
Normslord
8-1
0.4
0.2-
8-10
8m:50
0.0
25000
50000
750000
100000
136000 180000 179000
200000
Beviormmett Stoge
Figure 10: Abalation on IQL's online temperature values: The change in the temperature & used in online fine-tuning phase has little to effect the sample efficiency.
1.0
Nettribot Steee
0.6
0.4
(UID=S)
0.2
10E. [1
OgL (UTD-:1)
IEg Logar (500))
n2
CQC (UTDD.5)
IQL. (UTD=1]
0.0
20000
0000
100000
200000
20000
0000
Stege
SEEEEEAA Stepe
Steps
Figure 11: IQL and CQL: Step 0 on the x-axis the performance after offline pre-training. Observe while CQL suffers from initial policy unlearning, IQL improves slower throughout fine-tuning.
19

---

F Initial Unlearning of CQL on Multiple Tasks
Cal-QL (Ouss
CQL
Uitchee mi
Nonrebee Seore
05
Rechneree Stape
Festrnnnent Stepe
Earteneent Steps
Rormoorrrn Stape
yeu Herry 50
daon Manary vp
velorat st
Xemalont
Somnlen
10 40 Enstroarent Stope
Bastroment
Exstrasannn
Stape
Figure 12: While CQL experiences initial unlearning, Cal-QL effectively mitigates and quickly recovers its performance.
In this section, we show the learning curves of CQL and Cal-QL from Figure 6 and zoom in on the x-axis provide clearer visualization of CQL's initial unlearning in the Franka Kitchen, Adroit, and the "esuaalmanipulation domains. As depicted in Figure 12, it is evident across all tasks that while CQL experiences initial unlearning, Cal-QL can effectively mitigate it and quickly recovers its performance. Regarding the Antmaze domain, as we discussed in section 7.3, CQL does not exhibit initial unlearning since the default dataset has high coverage of data. However, we can observe similar phenomenon if we narrow down the dataset distribution (as shown in Figure 8).
G Ablation Study over Mixing Ratio Hyperparameter
Here we present an ablation study over mixing ratio on the adroit door-binary task for Cal-QL (left) and IQL (right). Perhaps as intuitively expected, we generally observe the trend that larger mixing ratio (i.e., more offline data) exhibits slower performance improvement during online fine-tuning.
Cal-QL
IQL
evalsaation/olllaccie wwchieeeeeaeeeeeeepeeeeeeee rate
gool www.aacchhedeeee rate raticc: 0.5 hid our hirh yomy tor
eixtrg_ratio: 0. - aixtnng ratto:: 6.3
stxing_ratte: 9.5
aistng_rattoo 7.7 mixing_ration 8.5
Min Myyg-r "-m" Mm
05
ma
022
ETU, Mttpa
200c
500c
Figure 13: Ablation study over mixing ratio on the adroit door-binary task for Cal-QL (left) and IQL (right).
H Regret Analysis of Cal-QL
We provide a theoretical version of Cal-QL in Algorithm 2. Policy fine-tuning has been studied in different settings [60, 52, 55J. Our analysis largely adopts the settings and results in Song et al. [52], with additional changes in Assumption H.1, Assumption H.3, and Definition H.4. Note that the goal of this proof is to demonstrate that a pessimistic functional class (Assumption /) allows one to utilize the offline data efficiently, rather than providing analytical technique for regret analysis.
20

---

See comparisons between Section H.3 and Section 1.1. Note that we use instead of Qo in the main text denote the estimated Q function for notation simplicity.
Algorithm 2 Theoretical version of Cal-QL
1: Input: Value function class F, total iterations K, offline dataset Dj of size moff for h 1-11.
2: Initialize (8, a) = 0, a(s
3: for k= 1=.....; K do
4: Let N² be the greedy policy w.r.t. fc
b Le., n (s arg max, (s a). D online data collection
5: For each h, collect mon online tuples D ~ di
6: Set pk+1
H (s, a) = 0, a)) 7: for h H
FQI with offline and online data 8: Estimate f using conservative least squares on the aggregated data: b Le., CQL
do
regularized class Cj
K Bog [[s..0) - mgs (2 )
1++1 arg min Sc0n
Eng [(t,a) -r -ax-max ( (
+
(H.I)
9:
f *+1 max pett1 -Qoe)
D Set a reference policy for calibration (Definition 4.1)
end for
11: end for
Output: π
H.1 Preliminaries
In this subsection, follow most of the notations and definitions in Song et al. [52]. In particular, we consider the finite horizon cases, where the value function and Q function are defined as:
(s) E
Tr sh.
(H.2)
H
@7(8, a) = E
Σ
rt|,, 8h = 8,
(H.3)
T=
We also define the Bellman operator T such that vf S A:
Tf(s, a) = E.,a [R(s, a)] + Ey-P((s.) max f(s a'), v(s,a)
(H.4)
o'
where R(s, a) E A[0, 1] represents a stochastic reward function.
H.2
Notations
Feature covariance matrix Mgyh:
En.b. Σ XAXT AX
(H.5)
Matrix Norm Zanette et al. [63]: for a matrix E, the matrix norm ||u||> is defined as:
|ulls = VuEu
(H.6)
Weighted & norm: for given distribution B A(S A) and function S:Sx x + R. we denote the weighted L² norm as:
I1fll = ((
(H.7)
A stochastic reward function R(s, a) E 1))
For each offline data distribution v {uD..... 1H-1), the offline data set time step
(2)) contains data samples (s, v, where (s, a) vh. r E R(s,a), ~ p(s.a).
21

---

Given policy т := ( where yy : + A(A), dz A(s,a) denotes the sate-action occupancy induced by π at step h.
We consider the value-based function approximation setting, where we are given a function class C = Co ... CH with Cy S x A [0, Vmax].
A policy xf is defined as the greedy policy w.r.t. f: (s) = arg max, (s, Specifi- cally, at iteration k, we use nh to denote the greedy policy w.r.t. ft.
H.3 Assumptions and Defintions
Assumption H.1 (Pessimistic Realizability and Completeness). For any policy x², say Cn. is pessimistic function class w.E.f n², if for any h, we have O7 E Cn. and additionally, for any fa+1 E CC++t, we have Th++ E Cn and fn fn(s,a) S Q (s,a), V(8, a) E x A.
Definition H.2 (Bilinear model Du et al. 19I, We say that the MDP together with the function class F is bilinear model of rand of for any h E [H 1}, there exist two (known) mappings XA, Wh FF Rd with max, (f)ll S Bx and max, [IW,(f)ll2 Bw such that
Vf.gEF:
E s.a (sn(s, - -onhti a)l = ((XA(O)
(H.8)
Assumption H.3 (Bilinear Rank of Reference Policies). Suppose Quee E Crel C. C. where Cret is the function class of our reference policy, assume the Bilinear rank of Cuet is dret and dret d.
Definition H.4 (Calibrated Bellman envor transfer coefficient). For any policy n. define the calibrated transfer coefficient w.r.. to reference policy ne as
Σn 11
E. Ex.add fh+ (s, fra(s,a)
Cref
max (e)
(H.9)
Eh= Bs (T Sh+1 a) 10000-aa8, (a()
where Qrei = Q
H.4 Discussions on the Assumptions
The pessimistic realizability and completeness assumption (Assumption H.1) is motivated by some theoretical studies of the pessimistic offline methods [59, 6] with regularizers:
min a (EonD,awz QQ((s a)] E. (8, + Bea.''D 2
Conservative regularizer (()))
(H.10)
Since the goal of the conservative regularizer R(0) intrinsically wants to enforce Qo(s, (s)) < (=(s.T((ss)
(H.11)
where x is the training policy and T" is the reference (behavior) policy. One can consider (H. as the Lagrange duality formulation of the following primal optimization problem:
min Bo (Qo(s, ((s subject to Es-D.ann (Qo(8, a)] S EuwD.arre [Qo(s, a)
(H.
where the constraint set is equivalent to Assumption H.I. Although Assumption H.I directly characterizes the constraint set of the primal form of (H) the exact theoretical connection between the pessimistic realizability and the regularized bellman consistency equation is beyond the scope of this work and would like to leave that for future studies.
Assumption H. allows us to restrict the functional class of interest to a smaller conservative function class C F, which leads smaller Bellman rank of the reference policy (d,ef d) suggested in Assumption H.3, and a smaller concentrability coefficient (Cref < C.) defined in Definition H.4, and 1.2. Assumption H.3 and Definition 1.2 provide the Bellman Bilinear rank and Bellman error transfer cocfficient of the pessimistic functional class C of interest.
22

---

H.S Proof Structure Overview
We provide an overview of the proof structure and its dependency on different assumptions below:
Theorem H.5: the total regret decomposed into offline regrets and online regrets.
Bounding offline regrets, requiring Definition H.4 and the following lemmas:
* Performance difference lemma w.r.t. a comparator policy (Lemma I.5).
* Least square generalization bound (Lemma I.4), requiring Assumption H.1.
Bounding online regrets, requiring Definition H.2
Performance difference lemma for the online enror (Lemma I6).
* Least square generalization bound (Lemma 1.4), requiring Assumption H.1.
*
Upper bounds with the bilinear model assumption (Lemma I.7).
* Applying Elliptical Potential Lemma [35] with bellman rank d and dref (Lemma 1.8), requiring Assumption H.3.
H.6 Our Results
Theorem H.5 (Formal Result of Theorem 6.1). Fix ő E (0, 1), Moft K, mon = 1, suppose and the function class follows Assumption H.1 w.r.t T", Suppose the underlying MDP admits Bilinear rank on function class C and deet on Cret, respectively, then with probability least 8, Algorithm
obtains the following bound on cumulative suboptimality w.r.t any comparator policy x²:
v = =m(min
(min {crs HK (IF1//), K (v- -Voe) H/o K log (1F1/8)
(H.13)
Note that Theorem H.5 provides a guarantee for any comparator policy x", which can be directly applied n² described in our informal result (Theorem 6. We also change the notation for the reference policy from in Theorem 6.1 to xe (similarly, dret. V C. correspond to du VP, C in Theorem 6.1) for notation consistency in the proof. Our proof of Theorem H.5 largely follows the proof of Theorem 1 of [52], and the major changes are caused by Assumption H.I, Assumption H.3, and Definition H.4.
Proof. Let E denote the event that (f a) I (r, a)} and Eg denote the event that {fo (s, a) > Qr, Let (s) = max. Qro (s, a), we start by noting that
K
K
Vx vx
Σ
E Eenu
1/ (s)
vot
(8)
k=1
A=1
K
K
Espp
[{Ex] ( Vo- (8) /rot (8)
+
E Exup
I {8E)
(vro((s) max (s.
1=1
=U, by the definition of Eg and line 9 of Algorithm 2
K
K
+
E Egnup 1 1{Ex}
max fo (s, vorl" (s)
E Egsp
[1Ex}
(Vo (s) max fo (s.a)
T
+
Esmp
1{E+}
max f a) 10""" (s)
(H.14)
Let Ki KK 1 f (s, > Qref (s,a)) and K₂ =k (1 s (st (or equiva- (Ex For To. we have lently K1 = Σk 1 (Ex). K₂ 1
ro = KyKenp (vT' (s) V (roc(s))
(H.15)
23

---

For 2, we have
K
1 [{E} (Vo' (s)
max (((xtss)))
22
Baup
k=1
{8x)
E s.ar (TTh+2(0,a) - f
k=1
h=0
(H.16)
(18)
H
Cref {Ex}
Σ Beaam ( - T -Thh
(384) S K1C / Doil.
where Aot is similarly defined as Song et al. [52] (See (1.3) of Lemma 1.4). Inequality (i) holds because Lemma 1.5, incquality (it) holds by the definition of C (Definition H.4), inequality (tii) holds by applying Lemma 1.4 with the function class satisfying Assumption H. and Definition H.4. Note that the telescoping decomposition technique in the above equation also appears [58, 24, 9]. Next, we will bound 11 + T3:
(EL) + + {Ex}) Beddo max (s, Vort (o)
k=1
H-1
(Ex) + +1 (E{x))
Σ
[fh((s))) -T (
h=0
(H.17)
H-1
(8)
(1{Ex) + 11 (E6))
Σ
hm0
(10)
R-1
(1 + 1 (Ex)
Σ
1|xx
M| V + AB
k=1
4=D
where Aon is similarly defined Song et al. [52] (See (L.4) of Lemma I.4). Incquality (t) holds by Lemma 1.6, equation (ti) holds by the definition of Bilinear model ((H.8) in Definition H.2),
inequality (dii) holds by Lemma 1.7 and Lemma 1.4 with the function class satisfying Assumption H.1. Using Lemma 1.8, we have that
Pi + Ts
11 (Ex) + (1xE)
Σ
(F
Vo + AB
x
h=0
( HI
K1 K2B
K2 K2B3
2d
1
(Aon ABGy) Ki + H 2dret log
1
(Bon xBby) Kg
Ad
Adhrof
(1)
2d008
1+
Ki d (Don + B3BBy) Ki 2dhor log log
1+
K₂ dret
) (Don + ByxBy K2
(H.18)
where the first part of inequality (i) holds by the assumption that the underlying MDPs have bellman rank d (Definition H.2) when & happens, and the second part of inequality (i) holds by the assumption that Cref has bilinear rank dret (Assumption H.3) Cref has bellman rank dref when & happens. Incquality (ii) holds by plugging in À Bs. Substituting (H.15), incquality H. and inequality (H.18) into (H.14), we have
Σ ya
varh
<Tu + P1 + Ia K2 (x*((s) - (vrc(s)) Ki Ki VH AoA
(8)-
1m1
1 Ki
K2 dset
+H
(v
2dlog
(Aon + B3.By K1 + 2drot log
1+
(Aon + By K₂
(H.19)
24

---

Plugging in the values of Don Aoft from (I.3) and (I.4), and using the subadditivity of the square root function, we have
K2
V=" (s) -v)((s)) + 16V666oo
Czt Kij
log
2HK1 FF]
molf
6
(H.20)
K1
16V V6Vnax
log
Bx Bw
H
2d log
1
Mon
K2 deet
16V
log
+ Bx Bw
H1
2d log
1+
Mon
Setting Moff
K, Mon
in the above equation completes the proof, we have
va ya
2ō (cpe HK1 log (1F1/8) +0 H HdK log (1F1/8)
+ K2 V
(s) (V*(s)
(voe (s) Ō dret log (1F1/6)
Vref
(H.21)
0 Crref H /dK log (1F1/5)
K1 > K2, otherwise.
0 K2 (K2 (V*'- - -rref) + dret K2 log (1F1/8)
Ō (inn C H H dK log (IF1/8), K (V=' Vreff
H dhet K log (1F1/8)}
where the last inequality holds because K1, K2 K, which completes the proof.
I Key Results of HyQ [52]
In this section, we restate the major theoretical results of Hy-Q [52] for completeness.
L1 Assumptions
Assumption I.1 (Realizability and Bellman completeness). For any h, we have Q7 E Fr. and additionally, for any fa+1 E FR+1, have T fh+1 E Fh. Definition I.2 (Bellman error transfer coefficient). For any policy we define the transfer coefficient as
h= Exaady [T 1 (8, Sn (n(
C₂ max
max feF
(L.1)
11 V > Es Eaaan (T fh+1 a) fn(s
1.2 Main Theorem of Hy-Q
Theorem 1.3 (Theorem Song et al. [52]). Fix 8 E (0, mott Moft that the underlying MDP admits Bilinear rank d (Definition I1.2), and the function class F satisfies Assumption 1.1. Then with probability at least suboptimality w.r.l. any comparator policy n²:
K, mon = and suppose
HyQ obtains the following bound on cumulative
Reg(K) δ (maxfCr*, 1)V Bw
(L.2)
1.3 Key Lemmas
L33.1
Least Squares Generalization and Applications
Lemma 1.4 (Lemma 7 of Song et al. [52], Online and Offline Bellman Error Bound for FQI). Let 8 (0, and Vh [H 1], k e [K]. let f be the estimated value function for time step h
25

---

computed via least square regression using samples in the dataset {Di, D) Dh...., p...... ............ in (H.1) in the iteration of Algoritim 2. Then with probability least 1 8. any [ 1] and &s [K], we have
11 1++
Ts
T
256V
2 =: Aor
(1.3)
2,
Moff
max
and
Ells - - 2.
1 256V (0g(2/11(71/00 Don:
(I.4)
Mi0n
where Vn denotes the offline data distribution time h, and the distribution F.j A(s, a) defined such that 8, a r di.
L3.. Bounding Offline Suboptimality via Performance Difference Lemma
Lemma I,5 (Lemma of Song al. [52], performance difference lemma of w.r.t. T²). Let T" (To.... TH - be a comparator policy and consider any value function f = (fo- fH where fn S A R. Then we have
H Σ
E Eawdo Vo (s) max So(o,
B Ba (T f)(,,a)),
(1.5)
where we define fa (s, a) = 0, ((s.).
1.3.3 Bounding Online Suboptimality via Performance Difference Lemma
Lemma I.6 (Lemma of Song et al. [52], performance difference lemma). For any function f where Sb. S A - IK. and h E [ 1}. we have
Eg-do max fo(s, a) Vo (s)
Σ
saa (f((s,a) Tsn+i
(I.6)
where we define [H(s, a) 0, Vs,
Lemma 1.7 (Lemma 8 of Song et al. [52], upper bounding bilinear class). For any 2 and E [ - 1], we have
(WA (|x(0)))
E 1-1sh
af [1A -TSh-shh -Thh
AB
(I.7)
where Eg-1ch defined as (H.5) and we use df," to denote dy.
Lemma 1.8 (Lemma 6 of Song et al. [52], bounding the inverse covariance norm). Let
"...") (n( E Rd be a sequence of vectors with [ & Bx < 00, vk: < K.
Then have
K
KB 1+
Σ
EEL.A S (
2dK log
)
(1.8)
Xd
1=1
where we define Ek:h = Di=
+ AI and we assume X 2 B& holds Vk E [K].
26


Reviews:
[{"review_id": "XZ8pA6ItAn", "reviewer": "Reviewer_vsVq", "summary": "Initially, this paper conducts an empirical investigation to understand why the offline policy derived from prior methods is inadequate as an initialization for online fine-tuning. Building upon these empirical findings and the derived hypothesis, the authors design Cal-QL, a method differing from CQL by only a single line, aiming to calibrate the Q function. The superiority of Cal-QL is demonstrated through both theoretical and empirical studies.", "questions": "Please see Weaknesses.", "limitations": "The authors have adequately addressed the limitations and the potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors present meticulous empirical and theoretical explorations justifying the proposed calibration.\n- The method, while simple, proves to be quite effective.\n- The paper boasts commendable writing and organization.", "weaknesses": "- In section 4.1, Figure 3 shows a clear increase in estimated Q values correlating with the increase in normalized scores. However, it does not clarify whether this increase stems from in-distribution data, out-distribution data, or both. Therefore, the connection between the results in Figure 3 and the conclusion in lines 162-164, which suggest that the rise in estimated Q values is due to the in-distribution data, may not be logically stringent.\n- Although the locomotion datasets in D4RL are extensively used as benchmarks, the authors did not test Cal-QL on these datasets. It would enhance the study if the authors could include comparisons with baselines on D4RL locomotion datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and positive feedback on the paper. To address your concerns, we have now added comparisons on D4RL locomotion tasks, where we find that Cal-QL outperforms prior methods IQL, AWAC, Hybrid RL, performing similarly to the best method, CQL. We have also clarified other concerns. \n\n**Please let us know if your concerns are addressed and if so, we would appreciate it if you could upgrade your score. We are happy to discuss further.**\n\n___\n\n## Results on D4RL locomotion datasets.\n\nWe have now added results on these tasks in **Figure B** in the one page PDF for Cal-QL, CQL, IQL, AWAC, and Hybrid RL. In general, we observe that Cal-QL attains the best performance on these tasks, tied with CQL (within the standard error margins). Our setup is as follows: following the scheme in prior work (Off2On, Lee et al. 2022) which finds non-conservative fine-tuning to perform best on top of the pre-trained CQL Q-function on the locomotion tasks (albeit it studies the deprecated -v0 versions of the locomotion tasks), we set the conservatism hyperparameter $\\alpha$ to 0 for both CQL and Cal-QL during fine-tuning. For fair comparisons, we also tune the value of the conservatism hyperparameter $\\beta$ for all other methods (IQL, AWAC). Clearly, the trends in the result demonstrate the superior performance of Cal-QL over other state-of-the-art fine-tuning (IQL, AWAC) and fast online RL methods (Hybrid RL).\n\n## Regarding Figure 3. \n\nWe apologize for the confusion. We have now plotted the average expected Q-value on states from the offline dataset (i.e., the in-distribution data) in **Figure A** in the PDF (we plot the placeholder value of 0 for Q-values on online data before fine-tuning begins at 50k steps). The Q-values on in-distribution data are labeled as `training/q_pi_offline_avg` and the Q-values on online data (i.e., out-of-distribution data) is labeled as `training/q_pi_online_avg`. \n\nObserve that the change in average Q-value plotted in Figure 3 of the submission stems from an increase in Q-values on both the in-distribution offline data and out-of-distribution data. Note crucially that the amount of increase on in-distribution Q-values is larger than the Q-values of online data. Therefore, the conclusion is that the CQL algorithm requires a few online samples to correct the heavily under-estimated in-distribution Q-values once it experiences online data logically follows from our experiment. \n"}, {"review_id": "iIslBr6aEd", "reviewer": "Reviewer_zucz", "summary": "The paper proposes a new offline to online algorithm, Cal-QL, that aims to mitigate the unlearning during the beginning of the online fine-tune stage. The paper first conducts empirical experiments to observe that the reason for the unlearning is due to overly pessimistic function values, and thus proposes the Cal-QL algorithm where lower bound the estimated value by the value of the reference policy. With the change to CQL, the proposed algorithm can outperform other Off2On algorithms on various tasks and shows consistent improvements, without the unlearning phenomena. The paper also proposes a theoretical algorithm and shows the regret bound under the bilinear model setting.  ", "questions": "See above.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper investigates empirically the reason for the initial unlearning phenomena at the beginning of the fine-tuning stage, shows good evidence for the issue of overly pessimistic value estimation, and thus well motivate the calibrated algorithm design. \n\n2. The paper compares with a good range of baselines in offline2online methods, and shows consistent improvements over the previous baselines, and indeed resolves the unlearning issue at the beginning of the fine-tuning stage, and thus provides good practical contribution. \n\n3. The overall algorithm is simple and a good amount of implementation details are provided in the paper. \n\n4. Some theory is also provided to make the paper more complete. ", "weaknesses": "1. Although it is good to provide many baselines to compare, the current baselines presentation is slightly confusing. I understand that RLPD and HyQ are reasonable baselines to compare with, but the current paper is not clearly comparing with RLPD or HyQ? One would guess that Hybrid RL means HyQ and SAC+offline means RLPD, but it is not clear that in the experiments the paper is indeed comparing with the exact algorithms. \n\n2. The theory algorithm seems to have some gap with the actual Cal-QL algorithm. \n\n3. The completeness assumption in the calibrated case seems strong, because the paper is focusing on a certain strange subset of function class. For example, can one show the completeness assumption holds even in the tabular case? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and a positive assessment of our work. We address your concerns below and will update the paper to clarify each of the questions. \n\n**Please let us know if your questions are resolved, and if so, we would be grateful if you are willing to upgrade your score. We are happy to discuss further if any questions are remaining.**\n\n___\n\n> **the current baselines presentation is slightly confusing; the current paper is not clearly comparing with RLPD or HyQ.** \n\nTo clarify, we do compare Cal-QL to the exact approaches of RLPD and HyQ in our experiments. The Hybrid RL approach in Figure 6 corresponds exactly to HyQ and we will update the legend of Figure 6 to clearly indicate this. \n\nSince RLPD has been specifically designed for high UTD RL, we compare to RLPD in all of our high UTD experiments in Figure 7. In our low UTD experiments (Figure 6), for fair comparison, we compare to ``SAC + offline data’’, a reduction of RLPD for the low UTD setting, which retains the 50:50 sampling between offline and online data, but does not utilize Q-function ensembles with layer normalization which is specifically required for addressing overestimation with high UTD. We will clarify these points in the updated version of the paper.\n\n___\n\n> **theory algorithm seems to have some gap with the actual Cal-QL algorithm.** \n\nWe will update the paper to discuss the limitation that our theoretical algorithm is a simplification of the practical algorithm. That said, we clarify that the goal of our theoretical analysis is to provide some intuition and motivation for our practical algorithm, rather than to provide a rigorous guarantee for our practical method. That said, we do note that this sort of a gap between theoretical algorithms and the practical algorithm is quite common in prior works: as one example, a closely related work Hybrid RL [3] (Song et al. ICLR 2023) also analyzes a version of FQI which minimizes TD-error exactly using an oracle and assumes access to all state-action pairs in theory, but the practical algorithm utilizes the DQN algorithm, which does not satisfy any of these conditions. \n\nAnother example is ATAC (ICML 2022) [2], where the authors provide a theoretical version (Alg. 1) and a practical version (Alg. 2) of their algorithm. The regret analysis corresponds to the theoretical algorithm assuming the existence of a no-regret policy optimization oracle, which is hard to practically achieve. As for the practical version (Algorithm 2), ATAC also adopts an actor-critic type algorithm (with a pessimistic regularizer on the critic network) for updating the Q network and policy network. Definitely, addressing the gap between theory and practice is important for the RL community, but we believe that our contribution is akin to other works that develop practical algorithms supported by theoretical results for motivation.\n\n___\n\n> **The completeness assumption in the calibrated case seems strong; For example, can one show the completeness assumption holds even in the tabular case?**\n\nThank you for your question! Yes, in the tabular case, the completeness assumption B.1 holds if we consider the subset of the function class where the values in the learned table are bounded by the Q-value of the reference policy. For instance, if the reference policy is the optimal policy, then in the tabular setting a pessimistic function class can still represent all possible value functions. \n\nThat said, we do agree with you that this sort of an assumption can be strong. Our rationale for this assumption was that it greatly simplified the exposition of our main insight (i.e., calibration during offline training can reduce cumulative regret incurred in online fine-tuning), without needing to handle the complexities with analyzing conservative value function training objectives like prior work (e.g., ATAC). We have already provided Appendix B.4 to elaborate on this connection and note that our analysis can be extended using the tools for handling conservatism in ATAC, which analyzes a pessimistic offline RL algorithm which utilizes a conservatism regularizer similar to CQL and Cal-QL, to avoid the need for this assumption. We have now noted this in the paper.    \n\n[1] Xie, Tengyang, et al. \"Bellman-consistent pessimism for offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 6683-6694.  \n[2] Cheng, Ching-An, et al. \"Adversarially trained actor critic for offline reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.  \n[3] Song, Yuda, et al. \"Hybrid RL: Using both offline and online data can make RL efficient.\" ICLR2023.\n"}, {"review_id": "dVuNkMncwd", "reviewer": "Reviewer_zhZZ", "summary": "The paper proposed a new approach to Offine-RL Online Fine-Tuning called calibrated Q-learning (Cal-QL). This approach aims to learn an effective initialization from offline data that enables fast online fine-tuning capabilities. Cal-QL accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data while ensuring that the learned Q-values are at a reasonable scale. The contributions of this paper include: 1) The paper theoretically analyzes the cumulative regret obtained by online fine-tuning, when the value function pre-training with Cal-QL. 2) Empirically, Cal-QL outperforms previous methods on 9/11 fine-tuning benchmark tasks.", "questions": "1.\tHow many random seeds are used for the experimental results of Table 1 and Table 2? Same as Figure 6, six random seeds? Moreover, how many random seeds are used in Figure 7 and 8?\n2.\tIn Appendix Table 3, for the visual-manipulation task, the paper utilized separate values for offline (\\alpha = 5) and online (\\alpha = 0.5) phases for the final results. Is the proposed algorithm sensitive to hyperparameters (\\alpha and mixing ratio)? To what extent do the final experimental results of different tasks depend on the hyperparameters?\n", "limitations": "1.\tIt would be better if there are more comparative experiments with important baseline methods [1,2].\n2.\tTo test the sensitivity of the method to hyperparameters, there should be more experimental comparisons and analyses of hyperparameters \\alpha and mixing ratio.\n\nReferences:\n\n[1] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702–1712. PMLR, 2022.\n\n[2] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.06239, 2022.\n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tWriting is clear and understandable.\n2.\tThe paper proposes a new definition (calibration) and gives the theoretical analysis of Cal-QL.\n3.\tThe method can be implemented on top of conservative Q-learning (CQL) for offline RL with few code changes.\n4.\tThe Cal-QL is better than the baseline methods in most tasks.\n", "weaknesses": "1.\tThe assumption of Cal-QL seems a bit too strong, that Cal-QL will be particularly effective in controlling the efficiency of online fine-tuning only when the reference policy μ is close to the narrow expert policy.\n2.\tCal-QL is also most related to methods that utilize a pessimistic RL algorithm for offline training but incorporate exploration in fine-tuning [1,2]. However, these two important baseline methods are missing from the experimental comparison results (Figure 6, Table 1 and 2).\n3.\tThe coefficient (\\alpha) of the conservative term (in equation 3.1) is an important parameter that directly affects the experimental results. However, the hyperparameter (\\alpha) ablation experiments are missing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for a positive assessment of our work. To address your concern regarding comparisons, we have added an experiment to compare Cal-QL to SPOT (Wu et al 2022), as you suggested. We found that Cal-QL outperforms this prior method (more details below). We also clarify that we already compare to O3F which performs pessimistic offline RL training followed by fine-tuning with an exploration bonus in **Figure 6** and **Tables 1 & 2** in the submission and find that Cal-QL outperforms this method as well. In Figure D, we also add an ablation study over $\\alpha$ for Cal-QL and find that Cal-QL is no more sensitive to $\\alpha$ than CQL or other methods such as TD3+BC. We address the remaining concerns below.\n\n\n**Please let us know if your concerns are addressed and if so, we would be grateful if you could update your score. We would be happy to discuss further if any concerns remain.**    \n___\n\n> **Additional comparisons to methods that use pessimistic offline RL + optimism during online fine-tuning.** \n\nWe have added a comparison to SPOT (Wu et al. 2022). We utilized the implementation of SPOT in the Clean Offline Reinforcement Learning (CORL) library (https://github.com/tinkoff-ai/CORL) and evaluated it on the 4 antmaze tasks. As shown below, we find that Cal-QL outperforms this approach, especially on the harder antmaze-large domains. \n|   env      |    Cal-QL (Normalized score)   |   SPOT (Normalized score)    |   Cal-QL (Regret)    | SPOT (Regret) | \n|------------|:--------------:|:---------------------------------------:|:-------------------------:|:------------------:|\n| large-diverse |33 - **95**|17.5 - 81.0| **0.20**  |0.23|\n|large-play|26 - **90**|31.5 - 87.0|**0.28**|0.29|\n| medium-diverse| 75 - **98**|  73.7 - 94.5 | 0.05  | 0.05  |\n| medium-play |54 - 97|67.2 - **97.2**  | 0.07  |  **0.05** |\n\nRegarding Off2On (Lee et al. 2022), the published code only contains code for online fine-tuning and does not provide the code for pre-training. In the limited time for this rebuttal, we tried our best to implement the offline pre-training part but were not able to get it running. Additionally, due to differences in the codebase infrastructure, we cannot directly utilize pre-trained checkpoints from our code directly in the Off2On codebase. Nonetheless, we agree this is a valuable baseline and will aim to get this method for the final version of the paper. \n\nFinally, also note that in Tables 1 & 2, Figure 6, we already compare to O3F (Mark et al. 2022), another approach that employs optimism during online training in conjunction with pessimistic offline RL pre-training and find that Cal-QL outperforms this approach as well.   \n___\n\n> **Ablation study for $\\alpha$, online & offline alpha** \n\nThis is an important point and we will add extensive ablation studies over these parameters for all methods in the final. During the rebuttal period, we ran an ablation on the Adroit door-binary task and found that setting $\\alpha$ to its offline value was generally pretty close to the best-tuned $\\alpha$ value, though other values of $\\alpha$ can also work during the online fine-tuning phase. Empirically, we recommend that a practitioner can perform a sweep in the neighborhood of the offline alpha value to tune their method efficiently.\n\nSpecifically, on the visual manipulation task, we present an ablation over alpha values for Cal-QL, CQL and TD3+BC in **Figure D** in the PDF. We observed that Cal-QL required the modification of the value of $\\alpha$ to attain best results. However, this isn’t any different from prior methods such as CQL or TD3+BC, which also required this. IQL’s performance was relatively more robust on the expectile parameter $\\beta$, but all of these $\\beta$ values performed worse than half of the $\\alpha$ values that we ran for Cal-QL. Thus, **Cal-QL is no more sensitive to $\\alpha$ values than other state-of-the-art algorithms on this task.**\n____\n\n> **Ablation study over mixing ratio.**\n\nWe ran an experiment to evaluate the performance of both IQL and Cal-QL with different mixing ratios on the Adroit door-binary task. The results are shown in **Figure C** in the PDF. Perhaps as expected, we find an intuitive trend that for both methods, a smaller mixing ratio (i.e., training more on online data than offline data) improves faster than larger mixing ratios which entail more updates on offline data. **In summary, note that this trend in Cal-QL’s performance as a function of mixing ratio is no different than other state-of-the-art methods, such as IQL.** We will run a more extensive ablation study for the final version of the paper.\n___\n\n> **Random seeds**\n\nWe utilize 6 seeds for our results in Table 1 and 2 like in Figure 6. For Figures 7 & 8, we use 3 seeds. The high UTD experiments in Figure 7 are time-consuming, hence we only ran 3 seeds for these experiments.  \n___\n\n>**The assumption of Cal-QL seems a bit too strong, that Cal-QL will be particularly effective in controlling the efficiency of online fine-tuning only when the reference policy μ is close to the narrow expert policy.** \n\nWe clarify that this is not an assumption, but rather a conclusion of our analysis for the special case when the reference policy is close to an expert policy. That is, we specifically stress that this is **not the __only__ case** where Cal-QL will be particularly effective, but rather one of the special cases that we chose to discuss in the paper. Theoretically, Cal-QL is effective whenever the bound on $\\text{Reg}(K)$ in Theorem 6.1 is smaller than the guarantee of Song et al. 2023, even in other cases because we show that $C^\\mu_{\\pi^*}$ is smaller than the concentrability coefficient in Song et al. 2023. \nTo corroborate the efficacy of Cal-QL with non-expert reference (behavior) policies, in our experiments in **Figure 6**, we demonstrate that Cal-QL is effective on antmaze-large and kitchen tasks, where the reference (behavior) policy is not expert.   \n"}, {"review_id": "XhwLCUzfNX", "reviewer": "Reviewer_J85q", "summary": "The paper introduces a method called Calibrated Q-learning (Cal-QL) for efficient online fine-tuning. This method learns an effective initialization from offline data, which enables fast online fine-tuning. It does so by underestimating the value of the learned policy from offline data while ensuring that the learned Q-values are at a reasonable scale.", "questions": "Why not conduct experiments on continuous control tasks similar to those performed by CQL and AWAC?", "limitations": "not applicable", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work focuses on the offline pre-training phase for better online fine-tuning and identifies why CQL unlearn initially, which is new to me. The paper is well-written and easy to follow.", "weaknesses": "My main concerns are listed below:\n\n1. Does the proposed approach increase the possibility of overestimation, given that it selects values larger than the behaviour policy? This question particularly arises when dealing with datasets that lack diversity or expert data.\n2. The baselines, such as CQL and AWAC, were tested on locomotion tasks, such as Hopper, Walker2d, and HalfCheetah. However, I did not see experiment results on these datasets. Does your approach perform poorly on these tasks?\n3. Some newly presented offline-to-online baselines, such as off2on[1], are not included.\n\n[1] [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and for a positive assessment of our work. To address your concerns, we have now run experiments evaluating Cal-QL on Gym locomotion tasks and found that Cal-QL outperforms prior methods on these tasks, tied with CQL. We also added a comparison to SPOT (Wu et al. 2023), a state-of-the-art offline pre-training and online fine-tuning method similar to Off2On. Regarding Off2On (Lee et al. 2022), we note that the author’s published code only contains the online fine-tuning portion of the method and does not provide the code for pre-training. In the limited time for this rebuttal, we tried our best to implement the offline pre-training part but were not able to get it running. But nonetheless, we agree this is a valuable comparison and we will aim to get this method for the final version of the paper. \n\n\nWe answer the rest of your questions in detail below. **Please let us know if your concerns are addressed, and if so, we would appreciate it if you are willing to upgrade your score. We are happy to discuss further and answer any remaining concerns.**\n___\n\n> **The baselines, such as CQL and AWAC, were tested on locomotion tasks, such as Hopper, Walker2d, and HalfCheetah. However, I did not see experiment results on these datasets. Does your approach perform poorly on these tasks?** \n\nTo address this concern, we have now added results on these tasks in **Figure B** in the one page PDF for Cal-QL, CQL, IQL, AWAC, and Hybrid RL. In general, we observe that Cal-QL attains the best performance on these tasks, tied with CQL (within the standard error margins). Our setup is as follows: following the scheme in prior work (Off2On, Lee et al. 2022) which finds non-conservative fine-tuning to perform best on top of the pre-trained CQL Q-function on the locomotion tasks (albeit it studies the deprecated -v0 versions of these tasks), we set the conservatism hyperparameter $\\alpha$ to 0 for both CQL and Cal-QL during fine-tuning. For fair comparisons, we also tune the value of the conservatism hyperparameter for all other methods (IQL, AWAC) and pick the best value. Clearly, the trends in the result demonstrate the superior performance of Cal-QL over other state-of-the-art fine-tuning (IQL, AWAC) as well as fast online RL methods (Hybrid RL), that do not employ separate pre-training.\n___\n> **Some newly presented offline-to-online baselines, such as off2on[1], are not included.** \n\nAs discussed earlier, we have now added an experiment comparing Cal-QL to a new method, SPOT (Wu et al. 2022) on the antmaze domains and found that Cal-QL attains better final performance and cumulative regret. Regarding Off2On (Lee et al. 2022), the published code only contains the online fine-tuning part and does not provide the code for pre-training. In the limited time for this rebuttal, we tried our best to implement the offline pre-training part but were not able to get it running. Additionally, due to differences in the codebase infrastructure, we cannot directly utilize pre-trained checkpoints from our code directly in the Off2On codebase. We will aim to get this method for the final version of the paper.  \n|   env      |    Cal-QL (Normalized score)   |   SPOT (Normalized score)    |   Cal-QL (Regret)    | SPOT (Regret) | \n|------------|:----------------------------------------:|:---------------------------------------:|:-------------------------:|:------------------:|\n| large-diverse | 33 - **95**   | 17.5 - 81.0  |  **0.20**  | 0.23  |\n| large-play | 26 - **90**   | 31.5 - 87.0  |  **0.28**  | 0.29  |\n| medium-diverse | 75 - **98**  |  73.7 - 94.5 | 0.05  | 0.05  |\n| medium-play |  54 - 97 | 67.2 - **97.2**  | 0.07  |  **0.05** |\n___\n> **Does the proposed approach increase the possibility of overestimation, given that it selects values larger than the behavior policy? This question particularly arises when dealing with datasets that lack diversity or expert data.** \n\nThis is a great question! We do observe that for a given value of the conservatism hyperparameter $\\alpha$, the Q-values learned by Cal-QL are higher than those learned by base CQL, which tend to be quite heavily underestimated in datasets that lack diversity. This does not mean that the Q-values are necessarily overestimated with respect to their ground-truth value (i.e., the Q-function of  the _learned policy_), even in the setting with narrow data, because the learned policy can still improve upon the behavior policy in a narrow dataset (for example, consider the D4RL locomotion medium datasets).\nIn the special case of expert data, such as on the Adroit tasks, we did observe that the Q-values learned by Cal-QL were larger than the ground-truth expert behavior policy return, but we believe this is not hurtful to performance: for attaining good offline performance, all we need is for the underlying method to create sufficient gap between Q-values at out-of-distribution and in-distribution actions. We always observe in our experiments that Cal-QL is able to induce a similar gap as CQL, thus preserving the relative ordering on actions. Moreover, a bit of overestimation may in fact be helpful during online fine-tuning, especially to keep the learned  policy close to the offline expert data. \nIndeed, we find that empirically, this over-estimation was not a problem: note that in all of our results, the performance of CQL and Cal-QL in the offline phase were quite similar to each other, and during fine-tuning, Cal-QL outperforms CQL by alleviating or reducing the performance dip. We have added this discussion in the paper now.\n"}]

Author rebuttal:
Dear Reviewers and the AC,

We thank all the reviewers for their detailed and constructive feedback. We are glad that Reviewer vsVq found our explanations to be meticulous, Reviewer zucz found our algorithm to be simple, and all reviewers found our paper to be well-written and clear.

To address the reviewers' concerns, we have added several new experiments in the rebuttal PDF attached to this response:

- In **Figure B**, we have added results comparing Cal-QL with other state-of-the-art fine-tuning methods (IQL, AWAC) and Hybrid RL on D4RL locomotion datasets and observe that outperforms these prior methods.

- In **Figures C and D**, we ran ablation experiments over $\alpha$ and mixing ratio to determine the sensitivity of Cal-QL to these parameters. We observed that Cal-QL is no more sensitive to the values of these parameters than existing state-of-the-art fine-tuning algorithms.

- In **Figure A**, we provided an additional plot to better understanding the unlearning phenomenon in CQL, supplementing Figure 3 in the main paper.

- In our response to **Reviewers zhZZ & J85q**, we also provide comparisons between Cal-QL and SPOT (Wu et al. 2022), a recently proposed online fine-tuning method on the antmaze tasks from D4RL. We observe that Cal-QL outperforms this prior method on the harder, antmaze-large tasks.

Additionally, we also provide several clarifications to address the reviewers' questions in our individual responses to each reviewer.

We believe that addressing the reviewers' feedback has improved the quality of the paper. We would be more than happy if the reviewers can let us know if their questions are answered. We will be happy to discuss further if there are any remaining questions.

Meta-review: This paper is concerned with addressing the issue of unlearning during the online fine-tuning stage in offline to online reinforcement learning. The authors propose Cal-QL, a new algorithm that mitigates unlearning by lower bounding the estimated value using the reference policy, providing superior performance compared to other offline-to-online algorithms in empirical and theoretical studies. The idea looks interesting and novel, and the proposed method is sound with theoretical and experimental support.
Decision: Accept (poster)