SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[54] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
[55] A. Wagenmaker and A. Pacchiano. Leveraging offline data in online reinforcement learning. arXiv preprint arXiv:2211.04974, 2022.
[56] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.06239, 2022.
[57] C. Xiao, H. Wang, Y. Pan, A. White, and M. White. The in-sample softmax for offline reinforcement learning. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=u-RuvyDYqCM.
[58] T. Xie and N. Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pages 550-559. PMLR, 2020.
[59] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[60] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27407, 2021.
[61] D. Ye, G. Chen, W. Zhang, S. Chen, B. Yuan, B. Liu, J. Chen, Z. Liu, F. Qiu, H. Yu, Y. Yin, B. Shi, L. Wang, T. Shi, Q. Fu, W. Yang, L. Huang, and W. Liu. Towards playing full moba games with deep reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 621-632. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/06d5ae105ea1bea4d800bc96491876e9-Paper.pdf.
[62] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.
[63] A. Zanette, M. J. Wainwright, and E. Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34: 13626-13640, 2021.
[64] Y. Zhai, C. Baek, Z. Zhou, J. Jiao, and Y. Ma. Computational benefits of intermediate rewards for goal-reaching policy learning. Journal of Artificial Intelligence Research, 73:847–896, 2022.
[65] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In International Conference on Machine Learning, pages 27042-27059. PMLR, 2022.
[66] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[67] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564, 2018.
A Implementation details of Cal-QL
Our algorithm, Cal-QL is illustrated in Algorithm 1. A python-style implementation is provided in Appendix A.2. The official code and experimental logs are available at https://github.com/nakamotoo/Cal-QL
A.1 Cal-QL Algorithm
We use  $J_{Q}(\theta)$  to denote the calibrated conservative regularizer for the Q network update:
$$
\begin{array}{l} J _ {Q} (\theta) := \alpha \underbrace {\left(\mathbb {E} _ {s \sim \mathcal {D} , a \sim \pi} \left[ \max  \left(Q _ {\theta} (s , a) , Q ^ {\mu} (s , a)\right) \right] - \mathbb {E} _ {s , a \sim \mathcal {D}} \left[ Q _ {\theta} (s , a) \right]\right)} _ {\text {C a l i b r a t e d c o n s e r v a t i v e r e g u l a r i z e r} \mathcal {R} (\theta)} (A.1) \\ + \frac {1}{2} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q _ {\theta} (s, a) - \mathcal {B} ^ {\pi} \bar {Q} (s, a)\right) ^ {2} \right]. (A.2) \\ \end{array}
$$
Algorithm 1 Cal-QL pseudo-code
1: Initialize Q-function,  $Q_{\theta}$ , a policy,  $\pi_{\phi}$
2: for step  $t$  in  $\{1, \dots, N\}$  do
3: Train the Q-function using the conservative regularizer in Eq. A.1:
$$
\theta_ {t} := \theta_ {t - 1} - \eta_ {Q} \nabla_ {\theta} J _ {Q} (\theta) \tag {A.3}
$$
4: Improve policy  $\pi_{\phi}$  with SAC-style update:
$$
\phi_ {t} := \phi_ {t - 1} + \eta_ {\pi} \mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi_ {\phi} (\cdot | s)} [ Q _ {\theta} (s, a) - \log \pi_ {\phi} (a | s) ] \tag {A.4}
$$
A.2 Python Implementation
Listing 1: Training Q networks given a batch of data
q_data = critic(batch['observations'], batch['actions'])  
next_dist = actor(batch['next�行ations'])  
next_pi ACTIONS, next_log_pis = next_dist.sample()  
target_qval = target_critic(batch['observations'], next_pi ACTIONS)  
target_qval = batch['rewards'] + self.gamma * (1 - batch['dones']) * target_qval  
td_loss = mse_loss(q_data, target_qval)  
num_samples = 4  
random ACTIONS = uniform((num_samples, batch_size, action_dim), min=-1, max=1)  
random_pi = 0.5 ** batch['actions'].shape[-1]  
pi ACTIONS, log_pis = actor(batch['observations']))  
q_rand_is = critic(batch['observations'], random ACTIONS) - random_pi  
q_pi_is = critic(batch['observations'], pi ACTIONS) - log_pis  
# Cal-QL's modification  
mc_return = batch['mc_return'].repeat(num_samples)  
q_pi_is = max(q_pi_is, mc_return)  
cat_q = concatenate([q_rand_is, q_pi_is], new_axis=True)  
cat_q = logsumexp(cat_q, axis=0) # sum over num_samples  
critic_loss = td_loss + ((cat_q - q_data).mean() * cql_alpha)  
critic_optimizer.zero_grad()  
critic_loss.backup()  
criticOptimizer.step()
Listing 2: Training the policy (or the actor) given a batch of data
# return distribution of actions  
pi_actions, log_pis = actor(batch['observations']))  
# calculate q value of actor actions  
qpi = critic(batch['observations'], actions)  
qpi = qpi.min(axis=0)  
# same objective as CQL (kumar et al.)  
actor_loss = (log_pis * self.alpha - qpi).mean()  
# optimize loss  
actor_OPTimizer.zero_grad()  
actor_loss.backup()  
actor_OPTimizer step()
B Environment Details
B.1 Antmaze
The Antmaze navigation tasks aim to control an 8-DoF ant quadruped robot to move from a starting point to a desired goal in a maze. The agent will receive sparse  $+1/0$  rewards depending on whether it reaches the goal or not. We study each method on "medium" and "hard" (shown in Figure 5) mazes which are difficult to solve, using the following datasets from D4RL [10]: large-diverse, large-play, medium-diverse, and medium-play. The difference between "diverse" and "play" datasets is the optimality of the trajectories they contain. The "diverse" datasets contain the trajectories commanded to a random goal from random starting points, while the "play" datasets contain the trajectories commanded to specific locations which are not necessarily the goal. We used an episode length of 1000 for each task. For Cal-QL, CQL, and IQL, we pre-trained the agent using the offline dataset for 1M steps. We then trained online fine-tuning for 1M environment steps for each method.
B.2 Franka Kitchen
The Franka Kitchen domain require controlling a 9-DoF Franka robot to arrange a kitchen environment into a desired configuration. The configuration is decomposed into 4 subtasks, and the agent will receive rewards of  $0, +1, +2, +3$ , or  $+4$  depending on how many subtasks it has managed to solve. To solve the whole task and reach the desired configuration, it is important to learn not only how to solve each subtask, but also to figure out the correct order to solve. We study this domain using datasets with three different optimialities: kitchen-complete, kitchen-partial, and kitchen-mixed. The "complete" dataset contains the trajectories of the robot performing the whole task completely. The "partial" dataset partially contains some complete demonstrations, while others are incomplete demonstrations solving the subtasks. The "mixed" dataset only contains incomplete data without any complete demonstrations, which is hard and requires the highest degree of stitching and generalization ability. We used an episode length of 1000 for each task. For Cal-QL, CQL, and IQL, we pre-trained the agent using the offline dataset for 500K steps. We then performed online fine-tuning for 1.25M environment steps for each method.
B.3 Adroit
The Adroit domain involves controlling a 24-DoF shadow hand robot. There are 3 tasks we consider in this domain: pen-binary, relocate-binary, relocate-binary. These tasks comprise a limited set of narrow human expert data distributions ( $\sim$  25) with additional trajectories collected by a behavior-cloned policy. We truncated each trajectory and used the positive segments (terminate when the positive reward signal is found) for all methods. This domain has a very narrow dataset distribution and a large action space. In addition, learning in this domain is difficult due to the sparse reward, which leads to exploration challenges. We utilized a variant of the dataset used in prior work [44] to have a standard comparison with SOTA offline fine-tuning experiments that consider this domain. For the offline learning phase, we pre-trained the agent for 20K steps. We then performed online fine-tuning for 300K environment steps for the pen-binary task, and 1M environment steps for the door-binary and relocate-binary tasks. The episode length is 100, 200, and 200 for pen-binary, door-binary, and relocate-binary respectively.
B.4 Visual Manipulation Domain
The Visual Manipulation domain consists of a pick-and-place task. This task is a multitask formulation explored in the work, Pre-training for Robots (PTR) [33]. Here each task is defined as placing an object in a bin. A distractor object was present in the scene as an adversarial object which the agent had to avoid picking. There were 10 unique objects and no overlap between the task objects and the interfering/distractor objects. The episode length is 40. For the offline phase, we pre-trained the policy with offline data for 50K steps. We then performed online fine-tuning for 100K environment steps for each method, taking 5 gradient steps per environment step.
C.1 Normalized Scores
The visual-manipulation, adroit, and antimaze domains are all goal-oriented, sparse reward tasks. In these domains, we computed the normalized metric as simply the goal achieved rate for each method. For example, in the visual manipulation environment, if the object was placed successfully in the bin, a  $+1$  reward was given to the agent and the task is completed. Similarly, for the door-binary task in the adroit tasks, we considered the success rate of opening the door. For the kitchen task, the task is to solve a series of 4 sub-tasks that need to be solved in an iterative manner. The normalized score is computed simply as  $\frac{\# \text{tasks solved}}{\text{total tasks}}$ .
C.2 Mixing Ratio Hyperparameter
In this work, we explore the mixing ratio parameter  $m$ , which is used during the online fine-tuning phase. The mixing ratio is either a value in the range [0, 1] or the value -1. If this mixing ratio is within [0, 1], it represents what percentage of offline and online data is seen in each batch when fine-tuning. For example, if the mixing ratio  $m = 0.25$ , that means for each batch we sample  $25\%$  from the offline data and  $75\%$  from online data. Instead, if the mixing ratio is -1, the buffers are appended to each other and sampled uniformly. We present an ablation study over mixing ratio in Appendix G.
C.3 Details and Hyperparameters for CQL and Cal-QL
We list the hyperparameters for CQL and Cal-QL in Table 3. We utilized a variant of Bellman backup that computes the target value by performing a maximization over target values computed for  $k$  actions sampled from the policy at the next state, where we used  $k = 4$  in visual pick and place domain and  $k = 10$  in others. In the Antmaze domain, we used the dual version of CQL [32] and conducted ablations over the value of the threshold of the CQL regularizer  $\mathcal{R}(\theta)$  (target action gap) instead of  $\alpha$ . In the visual-manipulation domain which is not presented in the original paper, we swept over the alpha values of  $\alpha = 0.5, 1, 5, 10$ , and utilized separate  $\alpha$  values for offline  $(\alpha = 5)$  and online  $(\alpha = 0.5)$  phases for the final results. We built our code upon the CQL implementation from https://github.com/young-geng/JaxCQL [14]. We used a single NVIDIA TITAN RTX chip to run each of our experiments.
C.4 Details and Hyperparameters for IQL
We list the hyperparameters for IQL in Table 4. To conduct our experiments, we used the official implementation of IQL provided by the authors [30], and primarily followed their recommended parameters, which they previously ablated over in their work. In the visual-manipulation domain which is not presented in the original paper, we performed a parameter sweep over expectile  $\tau = 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99$  and temperature  $\beta = 1, 3, 5, 10, 25, 50$  and selected the best-performing values of  $\tau = 0.7$  and  $\beta = 10$  for our final results. In addition, as the second best-performing method in the visual-manipulation domain, we also attempted to use separate  $\beta$  values for IQL, for a fair comparison with CQL and Cal-QL. However, we found that it has little to no effect, as shown in Figure 10.
C.5 Details and Hyperparameters for AWAC and ODT
We used the JAX implementation of AWAC from https://github.com/ikostrikov/jaxrl [29]. We primarily followed the author's recommended parameters, where we used the Lagrange multiplier
$\lambda = 1.0$  for the Antmaze and Franka Kitchen domains, and  $\lambda = 0.3$  for the Adroit domain. In the visual-manipulation domain, we performed a parameter sweep over  $\lambda = 0.1, 0.3, 1, 3, 10$  and selected the best-performing value of  $\lambda = 1$  for our final results. For ODT, we used the author's official implementation from https://github.com/facebookresearch/online-dt, with the author's recommended parameters they used in the Antmaze domain. In addition, in support of our result of AWAC and ODT (as shown in Table 1), the poor performance of Decision Transformers and AWAC in the Antmaze domain can also be observed in Table 1 and Table 2 of the IQL paper [30].
C.6 Details and Hyperparameters for SAC, SAC + Offline Data, Hybrid RL and CQL + SAC
We used the standard hyperparameters for SAC as derived from the original implementation in [19]. We used the same other hyperparameters as CQL and Cal-QL. We used automatic entropy tuning for the policy and critic entropy terms, with a target entropy of the negative action dimension. For SAC, the agent is only trained with the online explored data. For SAC + Offline Data, the offline data and online explored data is combined together and sampled uniformly. For Hybrid RL, we use the same mixing ratio used for CQL and Cal-QL presented in Table 3. For CQL + SAC, we first pre-train with CQL and then run online fine-tuning using both offline and online data, also using the same mixing ratio presented in Table 3.
Table 3: CQL, Cal-QL Hyperparameters
Hyperparameters	Adroit	Kitchen	Antmaze	Manipulation
α	1	5	-	5 (online: 0.5)
target action gap	-	-	0.8	-
mixing ratio	-1, 0.25, 0.5	-1, 0.25, 0.5	0.5	0.2, 0.5, 0.7, 0.9
Table 4: IQL Hyperparameters
Hyperparameters	Adroit	Kitchen	Antmaze	Manipulation
expectile τ	0.8	0.7	0.9	0.7
temperature β	3	0.5	10	10
mixing ratio	-1, 0.2, 0.5	-1, 0.25, 0.5	0.5	0.2, 0.5, 0.7, 0.9
D D4RL locomotion benchmark
In this section, we further compare Cal-QL with previous methods on the D4RL locomotion benchmarks. Since the dataset for these tasks does not end in a terminal, we estimate the reference value functions using a neural network by fitting a SARSA Q-function to the offline dataset. We present the normalized scores before and after fine-tuning in Table 5, and the cumulative regret in Table 6. Overall, we observe that Cal-QL outperforms state-of-the-art methods such as IQL and AWAC, as well as fast online RL methods that do not employ pre-training (Hybrid RL), performing comparably to CQL.
Task	CQL	IQL	AWAC	Hybrid RL	Cal-QL (Ours)
halfcheetah-expert-v2	0.79 → 1.02	0.95 → 0.54	0.69 → 0.65	N/A → 0.84	0.85 → 1.00
halfcheetah-medium-expert-v2	0.59 → 1.00	0.86 → 0.57	0.72 → 0.77	N/A → 0.86	0.54 → 0.99
halfcheetah-medium-replay-v2	0.51 → 0.95	0.43 → 0.36	0.43 → 0.47	N/A → 0.89	0.51 → 0.93
halfcheetah-medium-v2	0.53 → 0.97	0.47 → 0.43	0.49 → 0.57	N/A → 0.88	0.52 → 0.93
halfcheetah-random-v2	0.36 → 1.01	0.11 → 0.54	0.14 → 0.37	N/A → 0.80	0.33 → 1.04
hopper-expert-v2	1.00 → 0.73	1.02 → 0.52	1.12 → 1.11	N/A → 0.54	0.58 → 0.75
hopper-medium-expert-v2	0.86 → 0.92	0.07 → 0.81	0.30 → 1.03	N/A → 1.00	0.69 → 0.76
hopper-medium-replay-v2	0.69 → 1.11	0.76 → 0.86	0.70 → 1.08	N/A → 0.77	0.76 → 1.10
hopper-medium-v2	0.78 → 0.99	0.57 → 0.26	0.58 → 0.90	N/A → 1.06	0.89 → 0.98
hopper-random-v2	0.09 → 1.08	0.08 → 0.64	0.09 → 0.43	N/A → 0.80	0.10 → 0.79
walker2d-expert-v2	1.08 → 1.12	1.09 → 1.02	1.11 → 1.24	N/A → 1.12	0.92 → 1.00
walker2d-medium-expert-v2	1.00 → 0.56	1.09 → 1.06	0.86 → 1.16	N/A → 0.95	0.96 → 0.77
walker2d-medium-replay-v2	0.76 → 0.92	0.63 → 0.95	0.60 → 0.94	N/A → 1.03	0.52 → 0.99
walker2d-medium-v2	0.80 → 1.11	0.81 → 1.04	0.75 → 0.98	N/A → 0.86	0.75 → 1.03
walker2d-random-v2	0.05 → 0.85	0.06 → 0.09	0.04 → 0.04	N/A → 0.90	0.04 → 0.49
average	0.66 → 0.96	0.60 → 0.65	0.57 → 0.78	N/A → 0.89	0.60 → 0.90
Table 5: Normalized score before & after online fine-tuning on D4RL locomotion tasks.
Task	CQL	IQL	AWAC	Hybrid RL	Cal-QL (Ours)
halfcheetah-expert-v2	0.03	0.47	0.28	0.49	0.08
halfcheetah-medium-expert-v2	0.05	0.42	0.24	0.49	0.07
halfcheetah-medium-replay-v2	0.14	0.65	0.56	0.29	0.16
halfcheetah-medium-v2	0.15	0.61	0.48	0.31	0.17
halfcheetah-random-v2	0.11	0.54	0.67	0.33	0.10
hopper-expert-v2	0.17	0.58	-0.07	0.37	0.22
hopper-medium-expert-v2	0.09	0.32	-0.02	0.32	0.17
hopper-medium-replay-v2	0.09	0.09	0.05	0.27	0.12
hopper-medium-v2	0.04	0.73	0.08	0.26	0.05
hopper-random-v2	0.17	0.54	0.74	0.37	0.27
walker2d-expert-v2	-0.02	0.15	-0.18	0.11	0.15
walker2d-medium-expert-v2	0.03	0.02	-0.14	0.17	0.10
walker2d-medium-replay-v2	0.06	0.10	0.19	0.30	0.13
walker2d-medium-v2	0.11	0.12	0.07	0.29	0.22
walker2d-random-v2	0.53	0.92	0.96	0.56	0.49
average	0.12	0.42	0.26	0.33	0.17
Table 6: Cumulative regret on D4RL locomotion tasks. The smaller the better and 1.00 is the worst.
E Extended Discussion on Limitations of Existing Fine-Tuning Methods
In this section, we aim to highlight some potential reasons behind the slow improvement of other methods in our empirical analysis experiment in Section 4.1, and specifically, we use IQL for the analysis. We first swept over the temperature  $\beta$  values used in the online fine-tuning phase for IQL, which controls the constraint on how closely the learned policy should match the behavior policy. As shown in Figure 10, the change in the temperature  $\beta$  has little to no effect on the sample efficiency. Another natural hypothesis is that IQL improves slowly because we are not making enough updates per unit of data collected by the environment. To investigate this, we ran IQL with (a) five times as many gradient steps per step of data collection (UTD = 5), and (b) with a more aggressive policy update. Observe in Figure 11 that (a) does not improve the asymptotic performance of IQL, although it does improve CQL meaning that there is room for improvement on this task by making more gradient updates. Observe in Figure 11 that (b) often induces policy unlearning, similar to the failure mode in CQL. These two observations together indicate that a policy constraint approach can slow down learning asymptotically, and we cannot increase the speed by making more aggressive updates as this causes the policy to find erroneously optimistic out-of-distribution actions, and unlearn the policy learned from offline data.
Figure 10: Abalation on IQL's online temperature values: The change in the temperature  $\beta$  used in online fine-tuning phase has little to no effect on the sample efficiency.
Figure 11: IQL and CQL: Step 0 on the x-axis is the performance after offline pre-training. Observe while CQL suffers from initial policy unlearning, IQL improves slower throughout fine-tuning.
F Initial Unlearning of CQL on Multiple Tasks
Figure 12: While CQL experiences initial unlearning, Cal-QL effectively mitigates it and quickly recovers its performance.
In this section, we show the learning curves of CQL and Cal-QL from Figure 6 and zoom in on the x-axis to provide a clearer visualization of CQL's initial unlearning in the Franka Kitchen, Adroit, and the visual-manipulation domains. As depicted in Figure 12, it is evident across all tasks that while CQL experiences initial unlearning, Cal-QL can effectively mitigate it and quickly recovers its performance. Regarding the Antmaze domain, as we discussed in section 7.3, CQL does not exhibit initial unlearning since the default dataset has a high coverage of data. However, we can observe a similar phenomenon if we narrow down the dataset distribution (as shown in Figure 8).
G Ablation Study over Mixing Ratio Hyperparameter
Here we present an ablation study over mixing ratio on the adroit door-binary task for Cal-QL (left) and IQL (right). Perhaps as intuitively expected, we generally observe the trend that a larger mixing ratio (i.e., more offline data) exhibits slower performance improvement during online fine-tuning.
Cal-QL
Figure 13: Ablation study over mixing ratio on the adroit door-binary task for Cal-QL (left) and IQL (right).
IQL
H Regret Analysis of Cal-QL
We provide a theoretical version of Cal-QL in Algorithm 2. Policy fine-tuning has been studied in different settings [60, 52, 55]. Our analysis largely adopts the settings and results in Song et al. [52], with additional changes in Assumption H.1, Assumption H.3, and Definition H.4. Note that the goal of this proof is to demonstrate that a pessimistic functional class (Assumption H.1) allows one to utilize the offline data efficiently, rather than providing a new analytical technique for regret analysis.
See comparisons between Section H.3 and Section I.1. Note that we use  $f$  instead of  $Q_{\theta}$  in the main text to denote the estimated  $Q$  function for notation simplicity.
Algorithm 2 Theoretical version of Cal-QL
1: Input: Value function class  $\mathcal{F}$ , # total iterations  $K$ , offline dataset  $\mathcal{D}_h^\nu$  of size  $m_{\mathrm{off}}$  for  $h \in [H - 1]$ .
2: Initialize  $f_h^1(s, a) = 0, \forall (s, a)$ .
3: for  $k = 1, \ldots, K$  do
4: Let  $\pi^t$  be the greedy policy w.r.t.  $f^{k}$ $\triangleright$  I.e.,  $\pi_h^k (s) = \arg \max_a f_h^k (s,a)$
5: For each  $h$ , collect  $m_{\mathrm{on}}$  online tuples  $\mathcal{D}_h^k \sim d_h^{\pi^k}$ $\triangleright$  online data collection
6: Set  $f_{H}^{k + 1}(s,a) = 0, \forall (s,a)$ .
7: for  $h = H - 1,\ldots 0$  do  $\triangleright$  FQI with offline and online data
8: Estimate  $f_{h}^{k + 1}$  using conservative least squares on the aggregated data: I.e., CQL regularized class  $\mathcal{C}_h$
$$
f _ {h} ^ {k + 1} \leftarrow \underset {f \in \mathcal {C} _ {h}} {\arg \min } \left\{\widehat {\mathbb {E}} _ {\mathcal {D} _ {h} ^ {\nu}} \left[ f (s, a) - r - \underset {a ^ {\prime}} {\max } f _ {h + 1} ^ {k + 1} \left(s ^ {\prime}, a ^ {\prime}\right) \right] ^ {2} + \sum_ {\tau = 1} ^ {K} \widehat {\mathbb {E}} _ {\mathcal {D} _ {h} ^ {\tau}} \left[ f (s, a) - r - \underset {a ^ {\prime}} {\max } f _ {h + 1} ^ {k + 1} \left(s ^ {\prime}, a ^ {\prime}\right) \right] ^ {2} \right\} \tag {H.1}
$$
9:  $f_{h}^{k + 1} = \max \{f_{h}^{k + 1},Q_{h}^{\mathrm{ref}}\}$  Set a reference policy for calibration (Definition 4.1)
10: end for
11: end for
12: Output:  $\pi^K$
H.1 Preliminaries
In this subsection, we follow most of the notations and definitions in Song et al. [52]. In particular, we consider the finite horizon cases, where the value function and  $\mathbf{Q}$  function are defined as:
$$
V _ {h} ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {\tau = h} ^ {H - 1} r _ {\tau} | \pi , s _ {h} = s \right] \tag {H.2}
$$
$$
Q _ {h} ^ {\pi} (s, a) = \mathbb {E} \left[ \sum_ {\tau = h} ^ {H - 1} r _ {\tau} | \pi , s _ {h} = s, a _ {h} = a \right]. \tag {H.3}
$$