SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuition. We intuitively discuss how calibration and conservatism enable Cal-QL to attain a smaller regret compared to not imposing calibration. Our goal is to bound the cumulative regret of online fine-tuning,  $\sum_{k}\mathbb{E}_{s_0\sim \rho}[V^{\pi^{\star}}(s_0) - V^{\pi^k}(s_0)]$ . We can decompose this expression into two terms:
$$
\operatorname {R e g} (K) = \underbrace {\sum_ {k = 1} ^ {K} \mathbb {E} _ {s _ {0} \sim \rho} \left[ V ^ {\star} \left(s _ {0}\right) - \max  _ {a} Q _ {\theta} ^ {k} \left(s _ {0} , a\right) \right]} _ {(i) := \text {m i s c a l i b r a t i o n}} + \underbrace {\sum_ {k = 1} ^ {K} \mathbb {E} _ {s _ {0} \sim \rho} \left[ \max  _ {a} Q _ {\theta} ^ {k} \left(s _ {0} , a\right) - V ^ {\pi^ {k}} \left(s _ {0}\right) \right]} _ {(i i) := \text {o v e r e s t i m a t i o n}}. \tag {6.1}
$$
This decomposition of regret into terms (i) and (ii) is instructive. Term (ii) corresponds to the amount of over-estimation in the learned value function, which is expected to be small if a conservative RL algorithm is used for training. Term (i) is the difference between the ground-truth value of the optimal policy and the learned Q-function and is negative if the learned Q-function were calibrated against the optimal policy (per Definition 4.1). Of course, this is not always possible because we do not know  $V^{\star}$  a priori. But note that when Cal-QL utilizes a reference policy  $\mu$  with a high value  $V^{\mu}$ , close to  $V^{\star}$ , then the learned Q-function  $Q_{\theta}$  is calibrated with respect to  $Q^{\mu}$  per Condition 4.1 and term (i) can still be controlled. Therefore, controlling this regret requires striking a balance between learning a calibrated (term (i)) and conservative (term (ii)) Q-function. We now formalize this intuition and defer the detailed proof to Appendix H.6.
Theorem 6.1 (Informal regret bound of Cal-QL). With high probability, Cal-QL obtains the following bound on total regret accumulated during online fine-tuning:
$$
\mathrm {R e g} (K) = \widetilde {O} \Big (\min \left\{C _ {\pi^ {\star}} ^ {\mu} H \sqrt {d K \log (| \mathcal {F} |)}, K \mathbb {E} _ {\rho} [ V ^ {\star} (s _ {0}) - V ^ {\mu} (s _ {0}) ] + H \sqrt {d _ {\mu} K \log (| \mathcal {F} |)} \right\} \Big),
$$
where  $\mathcal{F}$  is the functional class of the  $Q$ -function.
Comparison to Song et al. [52]. Song et al. [52] analyzes an online RL algorithm that utilizes offline data without imposing conservatism or calibration. We now compare Theorem 6.1 to Theorem 1 of Song et al. [52] to understand the impact of these conditions on the final regret guarantee. Theorem 1 of Song et al. [52] presents a regret bound:  $\mathrm{Reg}(K) = \widetilde{O}\left(C_{\pi^{\star}}H\sqrt{dK\log(|\mathcal{F}|)}\right)$  and we note some improvements in our guarantee, that we also verify via experiments in Section 7.3: (a) for the setting where the reference policy  $\mu$  contains near-optimal behavior, i.e.,  $V^{\star} - V^{\mu}\lesssim O(H\sqrt{d\log(|\mathcal{F}|) / K})$ , Cal-QL can enable a tighter regret guarantee compared to Song et al. [52]; (b) as we show in Appendix H.3, the concentrability coefficient  $C_{\pi^{\star}}^{\mu}$  appearing in our guarantee is no larger than the one that appears in Theorem 1 of Song et al. [52], providing another source of improvement; and (c) finally, in the case where the reference policy has broad coverage and is highly sub-optimal, Cal-QL reverts back to the guarantee from [52], meaning that Cal-QL improves upon this prior work.
Figure 6: Online fine-tuning after offline initialization on the benchmark tasks. The plots show the online fine-tuning phase after pre-training for each method (except SAC-based approaches which are not pre-trained). Observe that Cal-QL consistently matches or exceeds the speed and final performance of the best prior method and is the only algorithm to do so across all tasks. (6 seeds)
7 Experimental Evaluation
The goal of our experimental evaluation is to study how well Cal-QL can facilitate sample-efficient online fine-tuning. To this end, we compare Cal-QL with several other state-of-the-art fine-tuning methods on a variety of offline RL benchmark tasks from D4RL [10], Singh et al. [51], and Nair et al. [45], evaluating performance before and after fine-tuning. We also study the effectiveness of Cal-QL on higher-dimensional tasks, where the policy and value function must process raw image observations. Finally, we perform empirical studies to understand the efficacy of Cal-QL with different dataset compositions and the impact of errors in the reference value function estimation.
Offline RL tasks and datasets. We evaluate Cal-QL on a number of benchmark tasks and datasets used by prior works [30, 45] to evaluate fine-tuning performance: (1) The AntMaze tasks from D4RL [10] that require controlling an ant quadruped robot to navigate from a starting point to a desired goal location in a maze. The reward is  $+1$  if the agent reaches within a pre-specified small radius around the goal and
Figure 5: Tasks: We evaluate Cal-QL on a diverse set of benchmark problems: AntMaze and Frankakitchen domains from [10], Adroit tasks from [45] and a vision-based robotic manipulation task from [34].
0 otherwise. (2) The FrankaKitchen tasks from D4RL require controlling a 9-DoF Franka robot to attain a desired configuration of a kitchen. To succeed, a policy must complete four sub-tasks in the kitchen within a single rollout, and it receives a binary reward of  $+1/0$  for every sub-task it completes. (3) Three Adroit dexterous manipulation tasks [47, 30, 45] that require learning complex manipulation skills on a 28-DoF five-fingered hand to (a) manipulate a pen in-hand to a desired configuration (pen-binary), (b) open a door by unlatching the handle (door-binary), and (c) relocating a ball to a desired location (relocate-binary). The agent obtains a sparse binary  $+1/0$  reward if it succeeds in solving the task. Each of these tasks only provides a narrow offline dataset consisting of 25 demonstrations collected via human teleoperation and additional trajectories collected by a BC policy. Finally, to evaluate the efficacy of Cal-QL on a task where we learn from raw visual observations, we study (4) a pick-and-place task from prior work [51, 34] that requires learning to pick a ball and place it in a bowl, in the presence of distractors. Additionally, we compared Cal-QL on D4RL locomotion tasks (halfcheetah, hopper, walker) in Appendix D.
Comparisons, prior methods, and evaluation protocol. We compare Cal-QL to running online SAC [21] from scratch, as well as prior approaches that leverage offline data. This includes naively fine-tuning offline RL methods such as CQL [32] and IQL [30], as well as fine-tuning with AWAC [45], O3F [42] and online decision transformer (ODT) [65], methods specifically designed for offline RL followed by online fine-tuning. In addition, we also compare to a baseline that trains SAC [21] using both online data and offline data (denoted by "SAC + offline data") that mimics DDPGfD [53] but utilizes SAC instead of DDPG. We also compare to Hybrid RL [52], a recently proposed method that improves the sample efficiency of the "SAC + offline data" approach, and "CQL+SAC", which first pre-train with CQL and then run fine-tuning with SAC on a mixture of offline and online data without conservatism. More details of each method can be found in Appendix C. We present learning curves for online fine-tuning and also quantitatively evaluate each method on its ability to improve the initialization learned from offline data measured in terms of (i) final performance after a pre-defined number of steps per domain and (ii) the cumulative regret over the course of online fine-tuning. In Section 7.2, we run Cal-QL with a higher update-to-data (UTD) ratio and compare it to RLPD [2], a more sample-efficient version of "SAC + offline data".
7.1 Empirical Results
We first present a comparison of Cal-QL in terms of the normalized performance before and after fine-tuning in Table 1 and the cumulative regret in a fixed number of online steps in Table 2. Following the protocol of [10], we normalize the average return values for each domain with respect to the highest possible return (+4 in FrankaKitchen; +1 in other tasks; see Appendix C.1 for more details).
Cal-QL improves the offline initialization significantly. Observe in Table 1 and Figure 6 that while the performance of offline initialization acquired by Cal-QL is comparable to that of other methods such as CQL and IQL, Cal-QL is able to improve over its offline initialization the most by  $106.9\%$  in aggregate and achieve the best fine-tuned performance in 9 out of 11 tasks.
Cal-QL enables fast fine-tuning. Observe in Table 2 that Cal-QL achieves the smallest regret on 8 out of 11 tasks, attaining an average regret of 0.22 which improves over the next best method (IQL) by  $42\%$ . Intuitively, this means that Cal-QL does not require running highly sub-optimal policies. In tasks such as relocate-binary, Cal-QL enjoys the fast online learning benefits associated with naive online RL methods that incorporate the offline data in the replay buffer (SAC + offline data and Cal-QL are the only two methods to attain a score of  $\geq 90\%$  on this task) unlike prior offline RL methods. As shown in Figure 6, in the kitchen and antimaze domains, Cal-QL brings the benefits of fast online learning together with a good offline initialization, improving drastically on the regret metric. Finally, observe that the initial unlearning at the beginning of fine-tuning with conservative methods observed in Section 4.1 is greatly alleviated in all tasks (see Appendix F for details).
Task	CQL	IQL	AWAC	O3F	ODT	CQL+SAC	Hybrid SRL	SAC+od	SAC	Cal-QL (Ours)
large-diverse	25 → 87	40 → 59	00 → 00	59 → 28	00 → 01	36 → 00	→ 00	→ 00	→ 00	33 → 95
large-play	34 → 76	41 → 51	00 → 00	68 → 01	00 → 00	21 → 00	→ 00	→ 00	→ 00	26 → 90
medium-diverse	65 → 98	70 → 92	00 → 00	92 → 97	00 → 03	64 → 98	→ 02	→ 68	→ 00	75 → 98
medium-play	62 → 98	72 → 94	00 → 00	89 → 99	00 → 05	67 → 98	→ 25	→ 96	→ 00	54 → 97
partial	71 → 75	40 → 60	01 → 13	11 → 22	-	71 → 00	→ 00	→ 07	→ 03	67 → 79
mixed	56 → 50	48 → 48	02 → 12	06 → 33	-	59 → 01	→ 01	→ 00	→ 02	38 → 80
complete	13 → 34	57 → 50	01 → 08	17 → 41	-	21 → 06	→ 00	→ 05	→ 06	22 → 68
pen	55 → 13	88 → 92	88 → 92	91 → 89	-	48 → 10	→ 54	→ 17	→ 11	79 → 99
door	22 → 88	41 → 88	29 → 13	04 → 08	-	29 → 66	→ 88	→ 39	→ 17	35 → 92
relocate	06 → 69	06 → 45	06 → 08	03 → 35	-	01 → 00	→ 99	→ 16	→ 00	03 → 98
manipulation	50 → 97	49 → 81	50 → 73	-	-	42 → 41	→ 00	→ 01	→ 01	49 → 99
average	42 → 71	50 → 69	16 → 20	44 → 45	00 → 02	42 → 29	→ 24	→ 23	→ 04	44 → 90
improvement	+ 71.0%	+ 37.7%	+ 23.7%	+ 3.0%	N/A	- 30.3%	N/A	N/A	N/A	+ 106.9%
Table 1: Normalized score before & after online fine-tuning. Observe that Cal-QL improves over the best prior fine-tuning method and attains a much larger performance improvement over the course of online fine-tuning. The numbers represent the normalized score out of 100 following the convention in [10].
7.2 Cal-QL With High Update-to-Data (UTD) Ratio
We can further enhance the online sample efficiency of Cal-QL by increasing the number of gradient steps per environment step made by the algorithm. The number of updates per environment step is usually called the update-to-data (UTD) ratio. In standard online RL, running off-policy Q-learning with a high UTD value (e.g., 20, compared to the typical value of 1) often results in challenges pertaining to overfitting [39, 5, 2, 8]. As expected, we noticed that running Cal-QL with a high UTD value also leads these overfitting challenges. To address these challenges in high UTD settings, we combine Cal-QL with the Q-function architecture in recent work, RLPD [2] (i.e., we utilized layer normalization in the Q-function and ensembles akin to Chen et al. [5]), that attempts to tackle
Task	CQL	IQL	AWAC	O3F	ODT	CQL+SAC	Hybrid RL	SAC+od	SAC	Cal-QL (Ours)
large-diverse	0.35	0.46	1.00	0.62	0.98	0.99	1.00	1.00	1.00	0.20
large-play	0.32	0.52	1.00	0.91	1.00	0.99	1.00	1.00	1.00	0.28
medium-diverse	0.06	0.08	0.99	0.03	0.95	0.06	0.98	0.77	1.00	0.05
medium-play	0.09	0.10	0.99	0.04	0.96	0.06	0.90	0.47	1.00	0.07
partial	0.31	0.49	0.89	0.78	-	0.97	0.98	0.98	0.92	0.27
mixed	0.55	0.60	0.88	0.72	-	0.97	0.99	1.00	0.91	0.27
complete	0.70	0.53	0.97	0.66	-	0.99	0.99	0.96	0.91	0.44
pen	0.86	0.11	0.12	0.13	-	0.90	0.56	0.75	0.87	0.11
door	0.36	0.25	0.81	0.82	-	0.23	0.35	0.60	0.94	0.23
relocate	0.71	0.74	0.95	0.71	-	0.86	0.30	0.89	1.00	0.43
manipulation	0.15	0.32	0.38	-	-	0.61	1.00	1.00	0.99	0.11
average	0.41	0.38	0.82	0.54	0.97	0.69	0.82	0.86	0.96	0.22
Table 2: Cumulative regret averaged over the steps of fine-tuning. The smaller the better and 1.00 is the worst. Cal-QL attains the smallest overall regret, achieving the best performance among 8 / 11 tasks.
Figure 7: Cal-QL with UTD=20. Incorporating design choices from RLPD enables Cal-QL to achieve sample-efficient fine-tuning with UTD=20. Specifically, Cal-QL generally attains similar or higher asymptotic performance as RLPD, while also exhibiting a smaller cumulative regret. (3 seeds)
overfitting challenges. Note that Cal-QL still first pre-trains on the offline dataset using Equation 5.1 followed by online fine-tuning, unlike RLPD that runs online RL right from the start. In Figure 7, we compare Cal-QL (UTD = 20) with RLPD [2] (UTD = 20) and also Cal-QL (UTD = 1) as a baseline. Observe that Cal-QL (UTD = 20) improves over Cal-QL (UTD = 1) and training from scratch (RLPD).
7.3 Understanding the Behavior of Cal-QL
In this section, we aim to understand the behavior of Cal-QL by performing controlled experiments that modify the dataset composition, and by investigating various metrics to understand the properties of scenarios where utilizing Cal-QL is especially important for online fine-tuning.
Effect of data composition. To understand the efficacy of Cal-QL with different data compositions, we ran it on a newly constructed fine-tuning task on the medium-size AntMaze domain with a low-coverage offline dataset, which is generated via a scripted controller that starts from a fixed initial position and navigates the ant to a fixed goal position. In Figure 8, we plot the performance of Cal-QL and baseline CQL (for comparison) on this task, alongside the trend of average Q-values over the course of offline pre-training (to the left of the dashed vertical line, before 250 training epochs) and online fine-tuning (to the right of the vertical dashed line, after 250 training epochs), and the trend of bounding rate, i.e., the fraction of transitions in the data-buffer for which the constraint in Cal-QL actively lower-bounds the learned Q-function with the reference value. For comparison, we also plot these quantities for a diverse dataset with high coverage on the task (we use the antmaze-medium-diverse from Fu et al. [10] as a representative diverse dataset) in Figure 8.
Observe that for the diverse dataset, both naive CQL and Cal-QL perform similarly, and indeed, the learned Q-values behave similarly for both of these methods. In this setting, online learning doesn't spend samples to correct the Q-function when fine-tuning begins leading to a low bounding rate, almost always close to 0. Instead, with the narrow dataset, we observe that the Q-values learned by
Figure 8: Performance of Cal-QL with data compositions. Cal-QL is most effective with narrow datasets, where Q-values need to be corrected at the beginning of fine-tuning.
naive CQL are much smaller, and are corrected once fine-tuning begins. This correction co-occurs with a drop in performance (solid blue line on left), and naive CQL is unable to recover from this drop. Cal-QL which calibrates the scale of the Q-function for many more samples in the dataset, stably transitions to online fine-tuning with no unlearning (solid red line on left).
This suggests that in settings with narrow datasets (e.g., in the experiment above and in the adroit and visual-manipulation domains from Figure 6), Q-values learned by naive conservative methods are more likely to be smaller than the ground-truth Q-function of the behavior policy due to function approximation errors. Hence utilizing Cal-QL to calibrate the Q-function against the behavior policy can be significantly helpful. On the other hand, with significantly high-coverage datasets, especially in problems where the behavior policy is also random and sub-optimal, Q-values learned by naive methods are likely to already be calibrated with respect to those of the behavior policy. Therefore no explicit calibration might be needed (and indeed, the bounding rate tends to be very close to 0 as shown in Figure 8). In this case, Cal-QL will revert back to standard CQL, as we observe in the case of the diverse dataset above. This intuition is also reflected in Theorem 6.1: when the reference policy  $\mu$  is close to a narrow, expert policy, we would expect Cal-QL to be especially effective in controlling the efficiency of online fine-tuning.
Estimation errors in the reference value function do not affect performance significantly. In our experiments, we compute the reference value functions using Monte-Carlo return estimates. However, this may not be available in all tasks. How does Cal-QL behave when reference value functions must be estimated using the offline dataset itself? To answer this, we ran an experiment on the kitchen domain, where instead of using an estimate for  $Q^{\mu}$  based on the Monte-Carlo return, we train a neural network function approximator  $Q_{\theta}^{\mu}$  to approximate  $Q^{\mu}$  via supervised regression on to Monte-Carlo return, which is then utilized by Cal-QL. Observe in Figure 9, that the performance of Cal-QL largely remains unaltered. This implies as long as we can obtain a reasonable function approximator to estimate the Q-function of the reference policy (in this case, the behavior policy), errors in the reference Q-function do not affect the performance of Cal-QL significantly.
Figure 9: Using a neural network approximator for the reference value function performs comparable to using the Monte-Carlo return. This indicates that errors in the reference Q-function do not negatively impact the performance.
8 Discussion, Future Directions, and Limitations
In this work we developed Cal-QL a method for acquiring conservative offline initializations that facilitate fast online fine-tuning. Cal-QL learns conservative value functions that are constrained to be larger than the value function of a reference policy. This form of calibration allows us to avoid initial unlearning when fine-tuning with conservative methods, while also retaining the effective asymptotic performance that these methods exhibit. Our theoretical and experimental results highlight the benefit of Cal-QL in enabling fast online fine-tuning. While Cal-QL outperforms prior methods, we believe that we can develop even more effective methods by adjusting calibration and conservatism more carefully. A limitation of our work is that we do not consider fine-tuning setups where pre-training and fine-tuning tasks are different, but this is an interesting avenue for future work.
This research was partially supported by the Office of Naval Research N00014-21-1-2838, N00014-22-1-2102, ARO W911NF-21-1-0097, the joint Simons Foundation-NSF DMS grant #2031899, AFOSR FA9550-22-1-0273, and Tsinghua-Berkeley Shenzhen Institute (TBSI) Research Fund, as well as support from Intel and C3.ai, the Savio computational cluster resource provided by the Berkeley Research Computing program, and computing support from Google. We thank Philip J. Ball, Laura Smith, and Ilya Kostrikov for sharing the experimental results of RLPD. AK is supported by the Apple Scholars in AI/ML PhD Fellowship. MN is partially supported by the Nakajima Foundation Fellowship. YZ is partially supported by Siemens CITRIS and TBSI research fund. YZ would like to thank Prof. Song Mei for insightful suggestions on the presentation of Theorem 6.1.
[1] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020.
[2] P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948, 2023.
[3] A. Beeson and G. Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211.11802, 2022.
[4] J. Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799, 2020.
[5] X. Chen, C. Wang, Z. Zhou, and K. W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=AY8zfZmOtDd.
[6] C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal. Adversarily trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.02446, 2022.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[8] P. D'Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=OpC-9aBBVJe.
[9] S. Du, S. Kakade, J. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning, pages 2826-2836. PMLR, 2021.
[10] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[11] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06860, 2021.
[12] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900, 2018.
[13] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), pages 1587-1596, 2018.
[14] X. Geng. Jaxcql: a simple implementation of sac and cql in jax. 2022. URL https://github.com/young-geng/JaxCQL.
[15] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.