SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \epsilon = \int_ {\mathbf {s} \in \mathcal {S}} d _ {\pi_ {\beta}} (\mathbf {s}) \epsilon_ {\mathbf {s}} \mathrm {d} \mathbf {s} \\ = d _ {\pi_ {\beta}} (\mathbf {s} _ {0}) \epsilon_ {\mathbf {s} _ {0}} + \int_ {\mathbf {s} \in \mathcal {S}, \mathbf {s} \neq \mathbf {s} _ {0}} d _ {\pi_ {\beta}} (\mathbf {s}) \epsilon_ {\mathbf {s}} \mathrm {d} \mathbf {s} \\ <   d _ {\pi_ {\beta}} (\mathbf {s} _ {0}) D _ {\mathrm {K L}} (\pi (\cdot | \mathbf {s} _ {0}) \| \pi_ {\beta} (\cdot | \mathbf {s} _ {0})) + \int_ {\mathbf {s} \in \mathcal {S}, \mathbf {s} \neq \mathbf {s} _ {0}} d _ {\pi_ {\beta}} (\mathbf {s}) D _ {\mathrm {K L}} (\pi (\cdot | \mathbf {s}) \| \pi_ {\beta} (\cdot | \mathbf {s})) \mathrm {d} \mathbf {s} \\ = \int_ {\mathbf {s} \in \mathcal {S}} d _ {\pi_ {\beta}} (\mathbf {s}) D _ {\mathrm {K L}} \left(\pi (\cdot | \mathbf {s}) \| \pi_ {\beta} (\cdot | \mathbf {s})\right) \mathrm {d} \mathbf {s}. \\ \end{array}
$$
This stands in contradiction to condition 1, where  $\int_{\mathbf{s} \in S} d_{\pi_{\beta}(\mathbf{s})} D_{\mathrm{KL}}(\pi(\cdot|\mathbf{s}) \| \pi_{\beta}(\cdot|\mathbf{s})) \mathrm{d}\mathbf{s} \leq \epsilon$ . Thus, the assumption that Equation (5)  $\Rightarrow$  Equations (12) to (13) is proven false, which confirms that Equation (5)  $\Rightarrow$  Equations (12) to (13).
Denote the feasible region of the problem in Definition 3.2 as  $\mathcal{F}^k (\epsilon)$ , and the feasible region of the problem in Definition 3.1 as  $\mathcal{F}^k (\{\epsilon_{\mathbf{s}},\mathbf{s}\in S\})$ . According to Lemma C.1, we can infer that:
$$
\mathcal {F} ^ {k} (\epsilon) = \bigcup_ {\left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} \in C} \mathcal {F} ^ {k} \left(\left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\}\right). \tag {14}
$$
Considering that the problems in Definition 3.1 and Definition 3.2 share the same objective function, we have:
$$
\forall \epsilon \geq 0, \quad \exists \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} \in C, \quad \pi_ {*} ^ {k + 1} [ \epsilon ] \in \mathcal {F} ^ {k} (\left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\}), \tag {15}
$$
where  $\pi_{*}^{k + 1}[\epsilon ]$  is the optimal solution corresponding to the optimal value  $J_{*}^{k}[\epsilon ]$  for the problem in Definition 3.2.
Based on Equation (15), we can derive that
$$
\forall \epsilon \geq 0, \quad \exists \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} \in C, \quad \exists \pi \in \mathcal {F} ^ {k} \left(\left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\}\right), \quad J _ {\pi} ^ {k} \left[ \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} \right] \geq J _ {*} ^ {k} [ \epsilon ]. \tag {16}
$$
Here  $J_{\pi}^{k}[\{\epsilon_{\mathbf{s}}, \mathbf{s} \in \mathcal{S}\}]$  is the objective function value in 3.1 with the solution  $\pi$  and KL constraints  $\{\epsilon_{\mathbf{s}}, \mathbf{s} \in \mathcal{S}\}$ .
Under the KL constraints  $\{\epsilon_{\mathbf{s}},\mathbf{s}\in S\}$ , the optimal value  $J_{*}^{k}[\{\epsilon_{\mathbf{s}},\mathbf{s}\in S\} ]$  of the problem in Definition 3.1 is no less than  $J_{\pi}^{k}[\{\epsilon_{\mathbf{s}},\mathbf{s}\in S\} ]$ . Therefore
$$
\exists \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\}, \quad J _ {*} ^ {k} \left[ \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} \right] \geq J _ {\pi} ^ {k} \left[ \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} \right] \geq J _ {*} ^ {k} [ \epsilon ]. \tag {17}
$$
Proposition 3.4 is proven. Q.E.D.
C.2 Proof of Proposition 3.5
Because the state space  $S$  possibly contains an infinite number of states, the optimization problem in Definition 3.1 is probably a problem with infinite constraints, which is not easy to deal with. Therefore, we first start with a simplified version of the optimization problem, where the state space contains only a finite number of states, and further extend the conclusion on the simplified optimization problem to the original one in Definition 3.1.
The simplified optimization problem is stated as follows:
Definition C.2 (Simplified optimization problem with point-wise constraints). The simplified optimization problem with point-wise KL constraints on a state space with only a finite number of states  $S = \{\mathbf{s}_1, \mathbf{s}_2, \dots, \mathbf{s}_m\}$  is defined as
$$
\max  _ {\pi} \mathbb {E} _ {\mathbf {s} \sim d _ {\pi_ {\beta}} (\cdot), \mathbf {a} \sim \pi (\cdot | \mathbf {s})} \left[ Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s}) \right] \tag {18}
$$
$$
\mathrm {s . t .} D _ {\mathrm {K L}} \left(\pi (\cdot | \mathbf {s} _ {i}) \| \pi_ {\beta} (\cdot | \mathbf {s} _ {i})\right) \leq \epsilon_ {i}, \quad i = 1, 2, \dots , m \tag {19}
$$
$$
\int_ {\mathbf {a} \in \mathcal {A}} \pi (\mathbf {a} | \mathbf {s} _ {i}) \mathrm {d} \mathbf {a} = 1, \quad i = 1, 2, \dots , m. \tag {20}
$$
Here,  $\pi^k (k\in \mathbb{N})$  denotes the policy at the  $k$ -th iteration,  $\pi_{\beta}$  is a behavior policy representing the action selection way in the offline dataset or current replay buffer, and  $d_{\pi_{\beta}}(\mathbf{s})$  is the state distribution of  $\pi_{\beta}$ . we utilize the optimal solution for Definition C.2 as  $\pi^{k + 1}$ .
For this simplified optimization problem, we have a lemma below, whose derivation is related to AWR [37] and AWAC [35]:
Lemma C.3. Consider the optimization problem in Definition C.2. The optimal solution of  $\pi^{k+1}$ , denoted as  $\pi_{*}^{k+1}$ , satisfies that  $\forall \mathbf{s} \in \{\mathbf{s}_1, \mathbf{s}_2, \dots, \mathbf{s}_m\}, \mathbf{a} \in \mathcal{A}$
$$
\pi_ {*} ^ {k + 1} (\mathbf {a} | \mathbf {s}) = \frac {\pi_ {\beta} (\mathbf {a} | \mathbf {s})}{Z _ {\mathbf {s}}} \exp \left(\beta_ {\mathbf {s}} \left(Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s})\right)\right). \tag {21}
$$
Proof. The Lagrangian function  $L(\pi, \lambda, \mu)$  is given by:
$$
\begin{array}{l} L (\pi , \lambda , \mu) = - \mathbb {E} _ {\mathbf {s} \sim d _ {\pi_ {\beta}} (\cdot), \mathbf {a} \sim \pi (\cdot | \mathbf {s})} \left[ Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s}) \right] \\ + \sum_ {i = 1} ^ {m} \lambda_ {i} \left(\int_ {\mathbf {a}} \pi (\mathbf {a} | \mathbf {s} _ {i}) \mathrm {d} \mathbf {a} - 1\right) + \sum_ {i = 1} ^ {m} \mu_ {i} \left(D _ {\mathrm {K L}} \left(\pi (\cdot | \mathbf {s} _ {i}) \| \pi_ {\beta} (\cdot | \mathbf {s} _ {i})\right) - \epsilon_ {i}\right), \tag {22} \\ \end{array}
$$
where  $\lambda = (\lambda_1, \lambda_2, \dots, \lambda_m), \mu = (\mu_1, \mu_2, \dots, \mu_m)$  are Lagrange multipliers. According to the KKT necessary conditions [20], the optimal solution satisfies that  $\forall \mathbf{a} \in \mathcal{A}, i \in \{1, 2, \dots, m\}$ ,
$$
\frac {\partial L}{\partial \pi (\mathbf {a} | \mathbf {s} _ {i})} = - d _ {\pi_ {\beta}} (\mathbf {s} _ {i}) \left(Q ^ {\pi^ {k}} (\mathbf {s} _ {i}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s} _ {i})\right) + \lambda_ {i} + \mu_ {i} \left(\log \left(\frac {\pi (\mathbf {a} | \mathbf {s} _ {i})}{\pi_ {\beta} (\mathbf {a} | \mathbf {s} _ {i})}\right) + 1\right) = 0. \tag {23}
$$
Therefore, the optimal policy  $\pi_{*}^{k + 1}$  is:
$$
\pi_ {*} ^ {k + 1} (\mathbf {a} | \mathbf {s} _ {i}) = \frac {\pi_ {\beta} (\mathbf {a} | \mathbf {s} _ {i})}{Z _ {\mathbf {s} _ {i}}} \exp \left(\beta_ {\mathbf {s} _ {i}} \left(Q ^ {\pi^ {k}} \left(\mathbf {s} _ {i}, \mathbf {a}\right) - V ^ {\pi^ {k}} \left(\mathbf {s} _ {i}\right)\right)\right), \tag {24}
$$
where  $Z_{\mathbf{s}_i} = \exp \left(\frac{\lambda_i}{\mu_i} + 1\right)$ ,  $\beta_{\mathbf{s}_i} = \frac{d_{\pi_\beta}(\mathbf{s}_i)}{\mu_i}$  are state-dependent factors.
Furthermore, due to the constraint in Equation (20),  $Z_{\mathbf{s}_i}$  also equals to  $\int_{\mathbf{a} \in \mathcal{A}} \pi_\beta(\mathbf{a} | \mathbf{s}_i) \exp \left( \beta_{\mathbf{s}_i} \left( Q^{\pi^k}(\mathbf{s}_i, \mathbf{a}) - V^{\pi^k}(\mathbf{s}_i) \right) \right) \mathrm{d}\mathbf{a}.$
Now we consider extending the conclusion in Lemma C.3 to the more complex optimization problem in Definition 3.1, where the state space  $S = [s_{\min}, s_{\max}]^l$  is a compact set and contains an infinite number of states. Without loss of generality, suppose that  $S = [0,1]$ . The derivation below is easy to transfer to  $S = [s_{\min}, s_{\max}]^l$  with small modifications.
Specifically, we construct a sequence sets  $\{B_i, i = 0,1,2,\dots\}$  with each element  $B_{i} = \{j / 2^{i}, j = 0,1,\dots,2^{i}\}$ . We can observe that  $\{B_{i}, i = 0,1,2,\dots\}$  satisfies:
$$
\forall i \in \mathbb {N} _ {+}, \quad B _ {i} \subseteq S; \tag {25}
$$
$$
\forall i \in \mathbb {N} _ {+}, \quad B _ {i} \subseteq B _ {i + 1}; \tag {26}
$$
$$
\forall i \in \mathbb {N} _ {+}, \quad | B _ {i} | = 2 ^ {i} + 1 <   \infty , \quad i. e., \text {a l l} B _ {i} \text {a r e f i n i t e s e t s , a n d t h e r e f o r e a l l} B _ {i} \text {a r e c o m p a c t}; \tag {27}
$$
$$
\lim  _ {i \rightarrow \infty} \sup  _ {x \in S} \inf  _ {y \in B _ {i}} \| x - y \| _ {\infty} = \lim  _ {i \rightarrow \infty} \frac {1}{2 ^ {i + 1}} = 0. \tag {28}
$$
The qualities in Equations (25) to (28), together with the assumption in Proposition 3.5 that the feasible space constrained by Equations (2) to (4) is not empty for every  $\mathbf{s} \in S$ , meets the prerequisites of the discretization method proposed by [41]. Set  $\pi_{*}^{k+1}[A]$  as the optimal solution of the following optimization problem:
$$
\pi_ {*} ^ {k + 1} [ A ] = \arg \max  _ {\pi} \mathbb {E} _ {\mathbf {s} \sim d _ {\pi_ {\beta}} (\cdot), \mathbf {a} \sim \pi (\cdot | \mathbf {s})} \left[ Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s}) \right] \tag {29}
$$
$$
\text {s . t .} D _ {\mathrm {K L}} (\pi (\cdot | \mathbf {s}) \| \pi_ {\beta} (\cdot | \mathbf {s})) \leq \epsilon_ {\mathbf {s}}, \quad \mathbf {s} \in A \tag {30}
$$
$$
\int_ {\mathbf {a} \in \mathcal {A}} \pi (\mathbf {a} | \mathbf {s}) \mathrm {d} \mathbf {a} = 1, \quad \mathbf {s} \in A, \tag {31}
$$
where  $A$  is a subset of  $S$ . According to the Theorem 2.1 in [41],
$$
\pi_ {*} ^ {k + 1} [ B _ {i} ] \xrightarrow {i \rightarrow \infty} \pi_ {*} ^ {k + 1} [ \mathcal {S} ]. \tag {32}
$$
For any  $B_i$ , because  $|B_i| < \infty$ , Equation (21) holds for  $\pi_*^{k+1}[B_i]$ . By combining Equation (21) with Equation (32), we succeed in proving Equation (7) in Proposition 3.5.
Furthermore, we utilize a parameterized policy  $\pi_{\phi}$  to approximate the optimal policy  $\pi_{*}^{k + 1}$  in Equation (7), i.e.,
$$
\begin{array}{l} \phi = \underset {\phi} {\arg \min } \mathbb {E} _ {\mathbf {s} \sim d _ {\pi_ {\beta}} (\cdot)} \left[ D _ {\mathrm {K L}} \left(\pi_ {*} ^ {k + 1} (\cdot | \mathbf {s}) \| \pi_ {\phi} (\cdot | \mathbf {s})\right) \right] (33) \\ = \underset {\phi} {\arg \max } \mathbb {E} _ {\mathbf {s} \sim d _ {\pi_ {\beta}} (\cdot)} \left[ \right. \frac {1}{Z _ {\mathbf {s}}} \mathbb {E} _ {\mathbf {a} \sim \pi_ {\beta} (\mathbf {a} | \mathbf {s})} \left[ \right. \exp \left( \right.\beta_ {\mathbf {s}} \left(Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s}))\right) \log \pi_ {\phi} (\mathbf {a} | \mathbf {s}) \left. \right]\left. \right]. (34) \\ \end{array}
$$
In practice,  $Z_{\mathrm{s}}$  in Equation (34) is challenging to calculate. We follow the derivation of AWR [37] and AWAC [35], and omit the term  $\frac{1}{Z_{\mathrm{s}}}$ . Therefore Equation (34) can be rewritten as
$$
\begin{array}{l} \phi = \arg \max  _ {\phi} \mathbb {E} _ {\mathbf {s} \sim d _ {\pi_ {\beta}} (\cdot)} \left[ \mathbb {E} _ {\mathbf {a} \sim \pi_ {\beta} (\mathbf {a} | \mathbf {s})} \left[ \exp \left(\beta_ {\mathbf {s}} \left(Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s})\right)\right) \log \pi_ {\phi} (\mathbf {a} | \mathbf {s}) \right] \right] (35) \\ = \arg \max  _ {\phi} \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D}} \left[ \right. \exp \left( \right.\beta_ {\mathbf {s}} \left(Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s}))\right) \log \pi_ {\phi} (\mathbf {a} | \mathbf {s}) \left. \right]. (36) \\ \end{array}
$$
Thus Equation (8) in Proposition 3.5 is also proven. Q.E.D.
D FamO2O's Extension to Non-AWR Algorithms
In this section, we discuss extending FamO2O to non-AWR algorithms, specifically considering the Conservative Q-Learning (CQL) [27].
CQL's policy update rule is given in Equation (37):
$$
\pi^ {k + 1} = \arg \max  _ {\pi} \mathbb {E} _ {\mathbf {s} \sim \mathcal {D}, \mathbf {a} \sim \pi (\cdot | \mathbf {s})} \left[ \alpha \cdot Q ^ {k} (\mathbf {s}, \mathbf {a}) - \log \pi (\mathbf {a} | \mathbf {s}) \right], \tag {37}
$$
where  $\pi^k$  and  $Q^k$  represent the policy and Q function at iteration  $k$  respectively, and  $\alpha$  is a hyperparameter. CQL already incorporates a conservative estimation for out-of-distribution data in its Q function update rule, thus lacking a policy constraint for policy updates in Equation (37). However, Equation (37) presents a balance between exploitation (the blue part) and exploration (the red part), determined by  $\alpha$ . We thus deploy FamO2O to adjust this balance adaptively per state.
Under FamO2O, the policy update rule morphs to Equation (38):
$$
\pi_ {u} ^ {k + 1} = \arg \max  _ {\pi_ {u}} \mathbb {E} _ {\mathbf {s} \sim \mathcal {D}, \mathbf {a} \sim \pi_ {u} (\cdot | \mathbf {s}, \alpha_ {\mathbf {s}})} \left[ \alpha_ {\mathbf {s}} \cdot Q ^ {k} (\mathbf {s}, \mathbf {a}) - \log \pi_ {u} (\mathbf {a} | \mathbf {s}, \alpha_ {\mathbf {s}}) \right], \tag {38}
$$
$$
\text {w h e r e} \quad \alpha_ {\mathbf {s}} = \pi_ {b} ^ {k} (\mathbf {s}), \tag {39}
$$
where  $\pi_u$  is the universal model taking a state and balance coefficient as input to yield an action, and  $\pi_b$  is the balance model mapping a state to a balance coefficient. The changes, denoted in red and blue, depict FamO2O's additional input and the substitution of the adaptive balance coefficient  $\alpha_{\mathrm{s}}$  for the pre-defined one  $\alpha$ , respectively.
Additionally, the balance model update rule (Equation (40)) aims to maximize the corresponding Q value by finding an optimal  $\alpha_{\mathbf{s}}$ , akin to Equation (10).
$$
\pi_ {b} ^ {k + 1} = \underset {\pi_ {b}} {\arg \max } \mathbb {E} _ {\mathbf {s} \sim \mathcal {D}} \left[ Q ^ {k} \left(\mathbf {s}, \underbrace {\pi_ {u} ^ {k + 1} \left(\mathbf {s} , \overbrace {\pi_ {b} (\mathbf {s})}\right)} _ {\text {a c t i o n}}\right) \right]. \tag {40}
$$
As for Q function updates, by consolidating balance model  $\pi_b: S \mapsto \mathcal{B}$  and  $\pi_u: S \times \mathcal{B} \mapsto \mathcal{A}$  into a standard policy  $\pi_u(\cdot, \pi_b(\cdot)): S \mapsto \mathcal{A}$ , these updates remain identical to standard CQL.
The following section outlines the specifics of baseline (Appendix E.1) and FamO2O (Appendix E.2) implementations.
E.1 Baseline Implementation
We commence by detailing our baseline implementation. For IQL [24], AWAC [35], and Balanced Replay [28], which have been tested in an offline-to-online RL setting in their respective papers, we utilize the official codes<sup>34</sup>.
For CQL [27] and TD3+BC [10], we introduce an online fine-tuning process based on the recommended offline RL codes by the authors<sup>56</sup>. Specifically, our online fine-tuning for CQL and TD3+BC mirrors the implementation in rlkit, that is, appending samples acquired during online interactions to the offline dataset while maintaining the same training objective during offline pre-training and online fine-tuning.
For SAC [13] and BC, we leverage CQL's codebase, which has already incorporated these two algorithms.
Regarding training steps, we adhere to IQL's experiment setting for a fair comparison across all methods in Figure 5, which involves  $10^{6}$  gradient steps each for offline pre-training and online fine-tuning. It should be noted that due to AWAC's overfitting issue, as discussed by [24], we limit AWAC's offline pre-training to  $2.5 \times 10^{4}$  gradient steps, as recommended. For non-AWR algorithms comparison, namely CQL [27], BR [28], and CQL+FamO2O, we employ  $2 \times 10^{6}$  offline gradient steps, as suggested in CQL's implementation, and  $1 \times 10^{6}$  online gradient steps, aligning with IQL.
$^{3}$ IQL and AWAC: https://github.com/rail-berkeley/rlkit, commit ID c81509d982b4d52a6239e7bfe7d2540e3d3cd986.  
 $^{4}$ Balanced Replay: https://github.com/shlee94/Off20nRL, commit ID 6f298fa  
 $^{5}$ CQL: https://github.com/young-geng/JaxCQL, commit ID 80006e1a3161c0a7162295e7002aebb42cb8c5fa.  
 $^{6}$ TD3+BC: https://github.com/sfujim/TD3_BC, commit ID 8791ad7d7656cb8396f1b3ac8c37f170b2a2dd5f.
E.2 Implementation of FamO2O
Next, we provide FamO2O's implementation details. We implement our FamO2O on the official codes of AWAC [35], IQL [24], and CQL [27] discussed above. Our major modification is (i) adding a balance model and (ii) making the original policy conditioned on the balance coefficient (called "universal model" in our paper). Except for these two modifications, we do not change any training procedures and hyperparameters in the original codes.
For the first modification, we employ the stochastic model used by SAC in rlkit to construct our balance model and implement the training process of the balance model described in Section 4.2. For the second modification, we change the input dimension of the original policy from the state dimension  $l$  to  $l + l_b$ , where  $l_b$  is the dimension of balance coefficient encodings. We encode balance coefficients in the same way as the positional encoding proposed by [47]. This encoding design expands the dimension of the balance coefficients and avoids the balance coefficients being neglected by the universal model.
For IQL with FamO2O, adaptive balance coefficients are no less than 1.0 (Locomotion) / 8.0 (AntMaze) and no larger than 5.0 (Locomotion) / 14.0 (AntMaze). For AWAC with FamO2O, the adaptive balance coefficients are no less than 2.0 (Locomotion) / 9.0 (AntMaze) and no larger than 3.0 (Locomotion) / 11.0 (AntMaze). For CQL with FamO2O, the adaptive balance coefficients are no less than 0.5 and no larger than 1.5. To ensure a fair comparison, in Figure 9, the fixed balance coefficient ranges cover the adaptive ranges discussed above.
As for training steps, for IQL with FamO2O, we use  $10^{6}$  gradient steps in the offline pre-training phase and  $10^{6}$  gradient steps in the online fine-tuning phase. For AWAC with FamO2O, we use  $7.5 \times 10^{4}$  gradient steps in the offline pre-training phase and  $2 \times 10^{5}$  (Locomotion) /  $1 \times 10^{6}$  (AntMaze) gradient steps in the online fine-tuning phase. For CQL with FamO2O, we use  $2 \times 10^{6}$  gradient steps in the offline pre-training phase and  $1 \times 10^{6}$  in the online fine-tuning phase.
For more implementation details, please refer to Tables 4 to 6.
F.1 Sensitivity Analysis on Different Ranges of Balance Coefficients
To find out whether FamO2O's performance is sensitive to the range of balance coefficients, we implement FamO2O on IQL [24] with 3 different balance coefficient ranges (i.e., [1, 5], [1, 4], and [2, 4]) on 3 datasets of different qualities (i.e., medium-replay, medium, and medium-expert). The results can be found in Figure 12. We further implement FamO2O on IQL with 4 different balance coefficient ranges (i.e., [6, 12], [8, 12], [8, 14], and [9, 11]) on 3 datasets of different maze sizes (i.e., umaze, medium, large). The results can be viewed in Figure 13. It can be observed that on various datasets of different qualities or maze sizes, the performance of FamO2O does not vary significantly with different ranges of the balance coefficients. This indicates that within a reasonable scope, FamO2O is insensitive to the range of coefficient balances.
Figure 12: Sensitivity analysis on different ranges of balance coefficients. We choose 3 different ranges (i.e., [1, 5], [1, 4], and [2, 4]) and 3 different dataset qualities (i.e., medium-replay, medium, and medium-expert) in the D4RL [8] HalfCheetah environment. Results are evaluated over 6 random seeds. The shade around curves denotes  $95\%$  CIs of the policy performance. Each epoch contains  $5\mathrm{k}$  gradient steps. The dashed lines divide the pre-training and fine-tuning.
Figure 13: Sensitivity analysis on different ranges of balance coefficients. We choose 4 different ranges (i.e., [6, 12], [8, 12], [8, 14], and [9, 11]) and 3 different maze sizes (i.e., umaze, medium, and large) in the D4RL [8] AntMaze play environments. Results are evaluated over 6 random seeds. The shade around curves denotes  $95\%$  CIs of the policy performance. Each epoch contains  $5\mathrm{k}$  gradient steps. The dashed lines divide the pre-training and fine-tuning.
F.2 Comparing FamO2O and Balance Coefficient Annealing
To delve deeper into the efficacy of FamO2O's balance model more thoroughly, we set up a comparison against the balance coefficient annealing. The balance coefficient annealing is a process that gradually augments the balance coefficient value as the fine-tuning stage proceeds. This method is intuitively reasonable, as the initial conservativeness due to the balance coefficient annealing helps to curb extrapolation errors at the commencement of fine-tuning, and as the process continues, this conservativeness is gradually eased to facilitate policy improvement.
We incorporate balance coefficient annealing into IQL [24], named IQL+annealing, providing a basis for comparison with IQL+FamO2O. The comparison results are presented in Table 3. The results show that FamO2O exceeds the performance of balance coefficient annealing across most of the D4RL Locomotion datasets [8].
Table 3: Comparing FamO2O and balance coefficient annealing, FamO2O notably surpasses the latter on the majority of D4RL locomotion datasets [8], demonstrating statistically significant performance improvement.
Dataset	FamO2O	Annealing
hopper-mr-v2	97.64	87.82
hopper-m-v2	90.65	72.98
hopper-me-v2	87.28	72.26
halfcheetah-mr-v2	53.07	53.35
halfcheetah-m-v2	59.15	57.37
halfcheetah-me-v2	93.10	93.91
walker2d-mr-v2	92.85	87.29
walker2d-m-v2	85.50	85.89
walker2d-me-v2	112.72	112.72
total	771.96	723.61
95% CIs	753.52~788.51	704.21~744.43
In addition, we calculate the  $95\%$  CIs for both IQL+FamO2O and IQL+annealing in Table 3, following the suggestion by riable [3]. The lower CI for IQL+FamO2O is mush higher than the upper CI for IQL+annealing, signifying a statistically significant improvement of FamO2O over balance coefficient annealing. These findings establish that in comparison to annealing, FamO2O showcases a more sophisticated and effective approach in adaptively modulating the balance between improvement and constraint.
G Potential Negative Societal Impacts
This work proposes a plug-in framework, named FamO2O, which can be implemented on top of the existing offline-to-online RL algorithms to further improve policy performance. Therefore, the potential negative societal impacts of our method are similar to those of the offline-to-online RL. Generally, FamO2O would greatly improve policy performance. However, as with most RL algorithms, FamO2O cannot guarantee to take safe and effective actions in all kinds of scenarios. We advocate that the users of FamO2O should be aware of the potential consequences and utilize FamO2O safely, especially in online environments such as self-driving, robotics, and healthcare.