SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mitsuhiko Nakamoto $^{1*}$  Yuexiang Zhai $^{1*}$  Anikait Singh $^{1}$  Max Sobol Mark $^{2}$
Yi Ma<sup>1</sup> Chelsea Finn<sup>2</sup> Aviral Kumar<sup>1</sup> Sergey Levine<sup>1</sup>
$^{1}$ UC Berkeley 2Stanford University
A compelling use case of offline reinforcement learning (RL) is to obtain a policy initialization from existing datasets followed by fast online fine-tuning with limited interaction. However, existing offline RL methods tend to behave poorly during fine-tuning. In this paper, we study the fine-tuning problem in the context of conservative offline RL methods and we devise an approach for learning an effective initialization from offline data that also enables fast online fine-tuning capabilities. Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also ensuring that the learned Q-values are at a reasonable scale. We refer to this property as calibration, and define it formally as providing a lower bound on the true value function of the learned policy and an upper bound on the value of some other (suboptimal) reference policy, which may simply be the behavior policy. We show that a conservative offline RL algorithm that also learns a calibrated value function leads to effective online fine-tuning, enabling us to take the benefits of offline initializations in online fine-tuning. In practice, Cal-QL can be implemented on top of the conservative Q learning (CQL) [32] for offline RL within a one-line code change. Empirically, Cal-QL outperforms state-of-the-art methods on 9/11 fine-tuning benchmark tasks that we study in this paper. Code and video are available at https://nakamoto.github.io/Cal-QL
Modern machine learning successes follow a common recipe: pre-training models on general-purpose, Internet-scale data, followed by fine-tuning the pre-trained initialization on a limited amount of data for the task of interest [22, 7]. How can we translate such a recipe to sequential decision-making problems? A natural way to instantiate this paradigm is to utilize offline reinforcement learning (RL) [37] for initializing value functions and policies from static datasets, followed by online fine-tuning to improve this initialization with limited active interaction. If successful, such a recipe might enable efficient online RL with much fewer samples than current methods that learn from scratch.
Many algorithms for offline RL have been applied to online fine-tuning. Empirical results across such works suggest a counter-intuitive trend: policy initializations obtained from more effective offline RL methods tend to exhibit worse online fine-tuning performance, even within the same task (see Table 2 of Kostrikov et al. [31] & Figure 4 of Xiao et al. [57]). On the other end, online RL methods training from scratch (or RL from demonstrations [53], where the replay buffer is seeded with the offline data) seem to improve online at a significantly faster rate. However,
*Equal contributions. Corresponding authors: Mitsuhiko Nakamoto, Yuexiang Zhai, and Aviral Kumar (\{nakamoto, simonzhai\} @berkeley.edu, aviralku@andrew.cmu.edu)
these online methods require actively collecting data by rolling out policies from scratch, which inherits similar limitations to naïve online RL methods in problems where data collection is expensive or dangerous. Overall, these results suggest that it is challenging to devise an offline RL algorithm that both acquires a good initialization from prior data and also enables efficient fine-tuning.
How can we devise a method to learn an effective policy initialization that also improves during fine-tuning? Prior work [32, 6] shows that one can learn a good offline initialization by optimizing the policy against a conservative value function obtained from an offline dataset. But, as we show in Section 4.1, conservatism alone is insufficient for efficient online fine-tuning. Conservative methods often tend to "unlearn" the policy initialization learned from offline data and waste samples collected via online interaction in recovering this initialization. We find that the "unlearning" phenomenon is a consequence of the fact that value estimates produced via conservative methods can be significantly lower than the ground-truth return of any valid policy. Having Q-value estimates that do not lie on a similar scale as the return of a valid policy is problematic. Because once fine-tuning begins, actions executed in the environment for exploration that are actually worse than the policy learned from offline data could erroneously
Figure 1: We study offline RL pre-training followed by online RL fine-tuning. Some prior offline RL methods tend to exhibit slow performance improvement in this setting (yellow), resulting in worse asymptotic performance. Others suffer from initial performance degradation once online fine-tuning begins (red), resulting in a high cumulative regret. We develop an approach that "calibrates" the learned value function to attain a fast improvement with a smaller regret (blue).
appear better, if their ground-truth return value is larger than the learned conservative value estimate. Hence, subsequent policy optimization will degrade the policy performance until the method recovers.
If we can ensure that the conservative value estimates learned using the offline data are calibrated, meaning that these estimates are on a similar scale as the true return values, then we can avoid the unlearning phenomenon caused by conservative methods (see the formal definition in 4.1). Of course, we cannot enforce such a condition perfectly, since it would require eliminating all errors in the value function. Instead, we devise a method for ensuring that the learned values upper bound the true values of some reference policy whose values can be estimated more easily (e.g., the behavior policy), while still lower bounding the values of the learned policy. Though this does not perfectly ensure that the learned values are correct, we show that it still leads to sample-efficient online fine-tuning. Thus, our practical method, calibrated Q-learning (Cal-QL), learns conservative value functions that are "calibrated" against the behavior policy, via a simple modification to existing conservative methods.
The main contribution of this paper is Cal-QL, a method for acquiring an offline initialization that facilitates online fine-tuning. Cal-QL aims to learn conservative value functions that are calibrated with respect to a reference policy (e.g., the behavior policy). Our analysis of Cal-QL shows that Cal-QL attains stronger guarantees on cumulative regret during fine-tuning. In practice, Cal-QL can be implemented on top of conservative Q-learning [32], a prior offline RL method, without any additional hyperparameters. We evaluate Cal-QL across a range of benchmark tasks from [10], [51] and [44], including robotic manipulation and navigation. We show that Cal-QL matches or outperforms the best methods on all tasks, in some cases by  $30 - 40\%$ .
Several prior works suggest that online RL methods typically require a large number of samples [50, 54, 61, 26, 64, 18, 38] to learn from scratch. We can utilize offline data to accelerate online RL algorithms. Prior works do this in a variety of ways: incorporating the offline data into the replay buffer of online RL [48, 53, 23, 52], utilizing auxiliary behavioral cloning losses with policy gradients [46, 27, 67, 66], or extracting a high-level skill space for downstream online RL [17, 1]. While these methods improve the sample efficiency of online RL from scratch, as we will also show in our results, they do not eliminate the need to actively roll out poor policies for data collection.
To address this issue, a different line of work first runs offline RL for learning a good policy and value initialization from the offline data, followed by online fine-tuning [45, 30, 41, 3, 56, 36, 42]. These
approaches typically employ offline RL methods based on policy constraints or pessimism [12, 49, 16, 15, 30, 51, 36] on the offline data, then continue training with the same method on a combination of offline and online data once fine-tuning begins [43, 28, 62, 32, 4]. Although pessimism is crucial for offline RL [25, 6], using pessimism or constraints for fine-tuning [45, 30, 41] slows down fine-tuning or leads to initial unlearning, as we will show in Section 4.1. In effect, these prior methods either fail to improve as fast as online RL or lose the initialization from offline RL. We aim to address this limitation by understanding some conditions on the offline initialization that enable fast fine-tuning.
Our work is most related to methods that utilize a pessimistic RL algorithm for offline training but incorporate exploration in fine-tuning [36, 42, 56]. In contrast to these works, our method aims to learn a better offline initialization that enables standard online fine-tuning. Our approach fine-tunes naively without ensembles [36] or exploration [42] and, as we show in our experiments, this alone is enough to outperform approaches that employ explicit optimism during data collection.
The goal in RL is to learn the optimal policy for an MDP  $\mathcal{M} = (\mathcal{S},\mathcal{A},P,r,\rho ,\gamma)$ .  $\mathcal{S},\mathcal{A}$  denote the state and action spaces.  $P(s^{\prime}|s,a)$  and  $r(s,a)$  are the dynamics and reward functions.  $\rho (s)$  denotes the initial state distribution.  $\gamma \in (0,1)$  denotes the discount factor. Formally, the goal is to learn a policy  $\pi :S\mapsto \mathcal{A}$  that maximizes cumulative discounted value function, denoted by  $V^{\pi}(s) = \frac{1}{1 - \gamma}\sum_{t}\mathbb{E}_{a_{t}\sim \pi (s_{t})}[\gamma^{t}r(s_{t},a_{t})|s_{0} = s]$ . The Q-function of a given policy  $\pi$  is defined as  $Q^{\pi}(s,a) = \frac{1}{1 - \gamma}\sum_{t}\mathbb{E}_{a_{t}\sim \pi (s_{t})}[\gamma^{t}r(s_{t},a_{t})|s_{0} = s,a_{0} = a]$ , and we use  $Q_{\theta}^{\pi}$  to denote the estimate of the Q-function of a policy  $\pi$  as obtained via a neural network with parameters  $\theta$ .
Given access to an offline dataset  $\mathcal{D} = \{(s, a, r, s')\}$  collected using a behavior policy  $\pi_{\beta}$ , we aim to first train a good policy and value function using the offline dataset  $\mathcal{D}$  alone, followed by an online phase that utilizes online interaction in  $\mathcal{M}$ . Our goal during fine-tuning is to obtain the optimal policy with the smallest number of online samples. This can be expressed as minimizing the cumulative regret over rounds of online interaction:  $\operatorname{Reg}(K) := \mathbb{E}_{s \sim \rho} \sum_{k=1}^{K} \left[ V^{\star}(s) - V^{\pi^k}(s) \right]$ . As we demonstrate in Section 7, existing methods face challenges in this setting.
Our approach will build on the conservative Q-learning (CQL) [32] algorithm. CQL imposes an additional regularizer that penalizes the learned Q-function on out-of-distribution (OOD) actions while compensating for this pessimism on actions seen within the training dataset. Assuming that the value function is represented by a function,  $Q_{\theta}$ , the training objective of CQL is given by
$$
\min  _ {\theta} \alpha \underbrace {\left(\mathbb {E} _ {s \sim \mathcal {D} , a \sim \pi} \left[ Q _ {\theta} (s , a) \right] - \mathbb {E} _ {s , a \sim \mathcal {D}} \left[ Q _ {\theta} (s , a) \right]\right)} _ {\text {C o n s e r v a t i v e r e g u l a r i z e r} \mathcal {R} (\theta)} + \frac {1}{2} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q _ {\theta} (s, a) - \mathcal {B} ^ {\pi} \bar {Q} (s, a)\right) ^ {2} \right], \tag {3.1}
$$
where  $\mathcal{B}^{\pi}\bar{Q}(s,a)$  is the backup operator applied to a delayed target Q-network,  $\bar{Q}$ :  $\mathcal{B}^{\pi}\bar{Q}(s,a) := r(s,a) + \gamma \mathbb{E}_{a' \sim \pi(a'|s')}[\bar{Q}(s',a')]$ . The second term is the standard TD error [40, 13, 20]. The first term  $\mathcal{R}(\theta)$  (in blue) is a conservative regularizer that aims to prevent overestimation in the Q-values for OOD actions by minimizing the Q-values under the policy  $\pi(a|s)$ , and counterbalances by maximizing the Q-values of the actions in the dataset following the behavior policy  $\pi_{\beta}$ .
4 When Can Offline RL Initializations Enable Fast Online Fine-Tuning?
A starting point for offline pre-training and online fine-tuning is to simply initialize the value function with one that is produced by an existing offline RL method and then perform fine-tuning. However, we empirically find that initializations learned by many offline RL algorithms can perform poorly during fine-tuning. We will study the reasons for this poor performance for the subset of conservative methods to motivate and develop our approach for online fine-tuning, calibrated Q-learning.
4.1 Empirical Analysis
Offline RL followed by online fine-tuning typically poses non-trivial challenges for a variety of methods. While analysis in prior work [45] notes challenges for a subset of offline RL methods, in Figure 2, we evaluate the fine-tuning performance of a variety of prior offline RL methods (CQL [32], IQL [30], TD3+BC [11], AWAC [45]) on a particular diagnostic instance of a visual pick-and-place task with a distractor object and sparse binary rewards [51], and find that all methods struggle to attain the best possible performance, quickly. More details about this task are in Appendix B.
Figure 3: The evolution of the average Q-value and the success rate of CQL over the course of offline pre-training and online fine-tuning. Fine-tuning begins at 50K steps. The red-colored part denotes the period of performance recovery which also coincides with the period of Q-value adjustment.
While the offline Q-function initialization obtained from all methods attains a similar (normalized) return of around 0.5, they suffer from difficulties during fine-tuning: TD3+BC, IQL, AWAC attain slow asymptotic performance and CQL unlearns the offline initialization, followed by spending a large amount of online interaction to recover the offline performance again, before any further improvement. This initial unlearning appears in multiple tasks as we show in Appendix F. In this work, we focus on developing effective fine-tuning strategies on top of conservative methods like CQL. To do so, we next aim to understand the potential reason behind the initial unlearning in CQL.
Why does CQL unlearn initially? To understand why CQL unlearns initially, we inspect the learned Q-values averaged over the dataset in Figure 3. Observe that the Q-values
learned by CQL in the offline phase are much smaller than their ground-truth value (as expected), but these Q-values drastically jump and adjust in scale when fine-tuning begins. In fact, we observe that performance recovery (red segment in Figure 3) coincides with a period where the range of Q-values changes to match the true range. This is as expected: as a conservative Q-function experiences new online data, actions much worse than the offline policy on the rollout states appear to attain higher rewards compared to the highly underestimated offline Q-function, which in turn deceives the policy optimizer into unlearning the initial policy. We illustrate this idea visually in Figure 4. Once the Q-function has adjusted and the range of Q-values closely matches the true range, then fine-tuning can proceed normally, after the dip.
To summarize, our empirical analysis indicates that methods existing fine-tuning methods suffer from difficulties such as initial unlearning or poor asymptotic performance. In particular, we observed that conservative methods can attain good asymptotic performance, but "waste" samples to correct the learned Q-function. Thus, in this paper, we attempt to develop a good fine-tuning method that builds on top of an existing conservative offline RL method, CQL, but aims to "calibrate" the Q-function so that the initial dip in performance can be avoided.
Figure 2: Multiple prior offline RL algorithms suffer from difficulties during fine-tuning including poor asymptotic performance and initial unlearning.
4.2 Conditions on the Offline Initialization that Enable Fast Fine-Tuning
Our observations from the preceding discussion motivate two conclusions in regard to the offline Q-initialization for fast fine-tuning: (a) methods that learn conservative Q-functions can attain good asymptotic performance, and (b) if the learned Q-values closely match the range of ground-truth Q-values on the task, then online fine-tuning does not need to devote samples to unlearn and then recover the offline initialization. One approach to formalize this intuition of Q-values lying on a similar scale as the ground-truth Q-function is via the requirement that the conservative Q-values learned by the conservative offline RL method must be lower-bounded by the ground-truth Q-value of a sub-optimal reference policy. This will prevent conservatism from learning overly small Q-values. We will refer to this property as "calibration" with respect to the reference policy.
Definition 4.1 (Calibration). An estimated  $Q$ -function  $Q_{\theta}^{\pi}$  for a given policy  $\pi$  is said to be calibrated with respect to a reference policy  $\mu$  if  $\mathbb{E}_{a\sim \pi}[Q_{\theta}^{\pi}(s,a)]\geq \mathbb{E}_{a\sim \mu}[Q^{\mu}(s,a)]\coloneqq V^{\mu}(s),\forall s\in D$
If the learned Q-function  $Q_{\theta}^{\pi}$  is calibrated with respect to a policy  $\mu$  that is worse than  $\pi$ , it would prevent unlearning during fine-tuning that we observed in the case of CQL. This is because the
policy optimizer would not unlearn  $\pi$  in favor of a policy that is worse than the reference policy  $\mu$  upon observing new online data as the expected value of  $\pi$  is constrained to be larger than  $V^{\mu}$ :  $\mathbb{E}_{a\sim \pi}[Q_{\theta}^{\pi}(s,a)]\geq V^{\mu}(s)$ . Our practical approach Cal-QL will enforce calibration with respect to a policy  $\mu$  whose ground-truth value,  $V^{\mu}(s)$ , can be estimated reliably without bootstrapping error (e.g., the behavior policy induced by the dataset). This is the key idea behind our method (as we will discuss next) and is visually illustrated in Figure 4.
5 Cal-QL: Calibrated Q-Learning
Our approach, calibrated Q-learning (Cal-QL) aims to learn a conservative and calibrated value function initializations from an offline dataset. To this end, Cal-QL builds on CQL [32] and then constrains the learned Q-function to produce Q-values larger than the Q-value of a reference policy  $\mu$  per Definition 4.1. In principle, our approach can utilize many different choices of reference policies, but for developing a practical method, we simply utilize the behavior policy as our reference policy.
Calibrating CQL. We can constrain the learned Q-function  $Q_{\theta}^{\pi}$  to be larger than  $V^{\mu}$  via a simple change to the CQL training objective shown in Equation 3.1: masking out the push down of the learned Q-value on out-of-distribution (OOD) actions in CQL if the Q-function is not calibrated, i.e., if  $\mathbb{E}_{a\sim \pi}[Q_\theta^\pi (s,a)]\leq V^\mu (s)$ . Cal-QL modifies the CQL regularizer,  $\mathcal{R}(\theta)$  in this manner:
$$
\mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi} [ \max  \left(Q _ {\theta} (s, a), V ^ {\mu} (s)\right) ] - \mathbb {E} _ {s, a \sim \mathcal {D}} \left[ Q _ {\theta} (s, a) \right], \tag {5.1}
$$
where the changes from standard CQL are depicted in red. As long as  $\alpha$  (in Equation 3.1) is large, for any state-action pair where the learned Q-value is smaller than  $Q^{\mu}$ , the Q-function in Equation 5.1 will upper bound  $Q^{\mu}$  in a tabular setting. Of course, as with any practical RL method, with function approximators and gradient-based optimizers, we cannot guarantee that we can enforce this condition for every state-action pair, but in our experiments, we find that Equation 5.1 is sufficient to enforce the calibration in expectation over the states in the dataset.
Pseudo-code and implementation details. Our implementation of Cal-QL directly builds on the implementation of CQL from Geng [14]. We present a pseudo-code for Cal-QL in Appendix A. Additionally, we list the hyperparameters  $\alpha$  for the CQL algorithm and our baselines for each suite of tasks in Appendix C. Following the protocol in prior work [30, 52], the practical implementation of Cal-QL trains on a mixture of the offline data and the new online data, weighted in some proportion during fine-tuning. To get  $V^{\mu}(s)$ , we can fit a function approximator  $Q_{\theta}^{\mu}$  or  $V_{\theta}^{\mu}$  to the return-to-go values via regression, but we observed that also simply utilizing the return-to-go estimates for tasks that end in a terminal was sufficient for our use case. We show in Section 7, how this simple one-line change to the objective drastically improves over prior fine-tuning results.
Figure 4: Intuition behind policy unlearning with CQL and the idea behind Cal-QL. The plot visualizes a slice of the learned Q-function and the ground-truth values for a given state. Erroneous peaks on suboptimal actions (x-axis) arise when updating CQL Q-functions with online data. This in turn can lead the policy to deviate away from high-reward actions covered by the dataset in favor of erroneous new actions, resulting in deterioration of the pre-trained policy. In contrast, Cal-QL corrects the scale of the learned Q-values by using a reference value function, such that actions with worse Q-values than the reference value function do not erroneously appear optimal in fine-tuning.
6 Theoretical Analysis of Cal-QL
We will now analyze the cumulative regret attained over online fine-tuning, when the value function is pre-trained with Cal-QL, and show that enforcing calibration (Definition 4.1) leads to a favorable regret bound during the online phase. Our analysis utilizes tools from Song et al. [52], but studies the impact of calibration on fine-tuning. We also remark that we simplify the treatment of certain aspects (e.g., how to incorporate pessimism) as it allows us to cleanly demonstrate the benefits of calibration.
Notation & terminology. In our analysis, we will consider an idealized version of Cal-QL for simplicity. Specifically, following prior work [52] under the bilinear model [9], we will operate in a finite-horizon setting with a horizon  $H$ . We denote the learned Q-function at each learning iteration  $k$  for a given  $(s,a)$  pair and time-step  $h$  by  $Q_{\theta}^{k}(s,a)$ . For any given policy  $\pi$ , let  $C_{\pi} \geq 1$  denote the concentrability coefficient such that  $C_{\pi} := \max_{f \in \mathcal{C}} \frac{\sum_{h=0}^{H-1} \mathbb{E}_{s,a \sim d_h^{\pi}}[\mathcal{T}f_{h+1}(s,a) - f_h(s,a)]}{\sqrt{\sum_{h=0}^{H-1} \mathbb{E}_{s,a \sim \nu_h}(\mathcal{T}f_{h+1}(s,a) - f_h(s,a))^2}}$ , i.e., a coefficient that quantifies the distribution shift between the policy  $\pi$  and the dataset  $\mathcal{D}$ , in terms of the ratio of Bellman errors averaged under  $\pi$  and the dataset  $\mathcal{D}$ . Note that  $\mathcal{C}$  represents the Q-function class and we assume  $\mathcal{C}$  has a bellman-bilinear rank [9] of  $d$ . We also use  $C_{\pi}^{\mu}$  to denote the concentrability coefficient over a subset of calibrated Q-functions w.r.t. a reference policy  $\mu$ :  $C_{\pi}^{\mu} := \max_{f \in \mathcal{C}, f(s,a) \geq Q^{\mu}(s,a)} \frac{\sum_{h=0}^{H-1} \mathbb{E}_{s,a \sim d_h^{\pi}}[\mathcal{T}f_{h+1}(s,a) - f_h(s,a)]}{\sqrt{\sum_{h=0}^{H-1} \mathbb{E}_{s,a \sim \nu_h}(\mathcal{T}f_{h+1}(s,a) - f_h(s,a))^3}}$ , which provides  $C_{\pi}^{\mu} \leq C_{\pi}$ . Similar to  $\mathcal{C}$ , let  $d_{\mu}$  denote the bellman bilinear rank of  $C_{\mu}$  – the calibrated Q-function class w.r.t. the reference policy  $\mu$ . Intuitively, we have  $C_{\mu} \subset C$ , which implies that  $d_{\mu} \leq d$ . The formal definitions are provided in Appendix H.2. We will use  $\pi^k$  to denote the arg-max policy induced by  $Q_{\theta}^k$ .