SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shenzhi Wang $^{1*}$ , Qisen Yang $^{1*}$ , Jiawei Gao $^{1}$ , Matthieu Lin $^{2}$ , Hao Chen $^{4}$ , Liwei Wu $^{1}$ , Ning Jia $^{3}$ , Shiji Song $^{1}$ , Gao Huang $^{1\dagger}$
$^{1}$  Department of Automation, BNRist, Tsinghua University
$^{2}$  Department of Computer Science, BNRist, Tsinghua University
$^{3}$  Beijing Academy of Artificial Intelligence (BAAI)  $^{4}$  Independent Researcher
Project Page: https://shenzhi-wang.github.io/NIPS_Fam020
Offline-to-online reinforcement learning (RL) is a training paradigm that combines pre-training on a pre-collected dataset with fine-tuning in an online environment. However, the incorporation of online fine-tuning can intensify the well-known distributional shift problem. Existing solutions tackle this problem by imposing a policy constraint on the policy improvement objective in both offline and online learning. They typically advocate a single balance between policy improvement and constraints across diverse data collections. This one-size-fits-all manner may not optimally leverage each collected sample due to the significant variation in data quality across different states. To this end, we introduce Family Offline-to-Online RL (FamO2O), a simple yet effective framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O utilizes a universal model to train a family of policies with different improvement/constraint intensities, and a balance model to select a suitable policy for each state. Theoretically, we prove that state-adaptive balances are necessary for achieving a higher policy performance upper bound. Empirically, extensive experiments show that FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark. Codes are available at https://github.com/LeapLabTHU/FamO20.
(a) General intuitions of our proposed FamO2O
(b) FamO2O's performance enhancement
Figure 1: FamO2O trains a policy family from datasets and selects policies state-adaptively using online feedback. Easily integrated, FamO2O statistically enhances existing algorithms' performance.
* Equal contribution. Email: {wsz21, yangqs19}@mails.tsinghua.edu.cn.
† Corresponding author. Email: gaohuang@tsinghua.edu.cn.
Offline reinforcement learning (RL) provides a pragmatic methodology for acquiring policies utilizing pre-existing datasets, circumventing the need for direct environment interaction [29]. Nonetheless, the attainable policy performance in offline RL is frequently constrained by the quality of the dataset [66]. The offline-to-online RL paradigm addresses this limitation by refining the offline RL policy through fine-tuning in an online setting [35].
While online fine-tuning can indeed elevate policy performance, it also potentially exacerbates the issue of distributional shift [29], where policy behavior diverges from the dataset distribution. Such shifts typically ensue from drastic policy improvements and are further amplified by state distribution changes when transitioning from offline learning to online fine-tuning [11, 25, 9, 26]. Prior works have attempted to counter this by imposing policy constraints on the policy improvement objective to deter excessive exploration of uncharted policy space [25, 54, 35]. However, this conservative approach can inadvertently stifle policy improvement [35]. In essence, offline-to-online RL necessitates an effective balance between policy improvement and policy constraint during policy optimization.
Regrettably, prior offline-to-online RL algorithms tend to adopt a monolithic approach towards this improvement-constraint trade-off, indiscriminately applying it to all data in a mini-batch [10, 66, 53] or the entire dataset [35, 24, 27, 30]. Given the inherent data quality variation across states (see Figure 2), we argue that this one-size-fits-all manner may fail to optimally exploit each sample. In fact, data yielding high trajectory returns should encourage more "conservative" policies, while data leading to poor returns should incite more "radical" policy improvement.
(b) Hopper
Figure 2: A t-SNE [46] visualization of randomly selected states from (a) HalfCheetah and (b) Hopper medium-expert datasets in D4RL [8]. The color coding represents the return of the trajectory associated with each state. This visualization underscores the significant variation in data quality across different states.
In this paper, we introduce a novel framework, Family Offline-to-Online RL (FamO2O), which can discern a state-adaptive improvement-constraint balance for each state. FamO2O's design is founded on two key insights delineated in Figure 1(a): (i) The collected dataset, abundant in environmental information, could facilitate the training of a diverse policy family ranging from conservative to radical, and (ii) feedback from online learning might assist in selecting an appropriate policy from this family for each state. As depicted in Figure 3, FamO2O incorporates a universal model and a balance model. The universal model, conditioned on a balance coefficient, determines the degree of policy conservatism or radicalism, while the balance model learns a mapping from states to balance coefficients, aiding the universal model in tailoring its behavior to each specific state.
FamO2O represents, to the best of our knowledge, the first offline-to-online RL approach harnessing the power of state-adaptive improvement-constraint balances. Theoretically, we establish that in policy optimization, point-wise KL constraints afford a superior performance upper bound compared to the distributional KL constraints adopted in prior works [37, 35, 24]. Importantly, these state-adaptive balances become indispensable when addressing point-wise KL constraints, thereby underlining the necessity of incorporating such balances in offline-to-online RL. Experimental results, as summarized in Figure 1(b), reveal that FamO2O is a simple yet effective framework that statistically significantly improves various offline-to-online RL algorithms and achieves state-of-the-art performance.
We introduce essential RL and offline-to-online RL concepts and notations here. For descriptive convenience and theoretical analysis, we use the advantaged-weight regression (AWR) algorithm framework [37, 35, 24], but FamO2O isn't limited to the AWR framework. We later demonstrate its integration with non-AWR algorithms in Section 5.3 and Appendix D.
2.1 Reinforcement learning formulation
RL is typically expressed as a Markov decision process (MDP) [43], denoted as  $(S, \mathcal{A}, P, d_0, R, \gamma)$ . Here,  $S$  and  $\mathcal{A}$  are the state $^1$  and action space;  $P(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a})$  is the environmental state transition probability;  $d_0(\mathbf{s}_0)$  represents initial state distribution;  $R(\mathbf{s}_t, \mathbf{a}_t, \mathbf{s}_{t+1})$  is the reward function; and  $\gamma \in (0, 1]$  is the discount factor.
2.2 Offline-to-online reinforcement learning
Offline-to-online RL is a training paradigm including two phases: (i) offline pre-training: pre-training a policy based on an offline dataset; (ii) online fine-tuning: fine-tuning the pre-trained policy by interacting with the environment. Note that in the offline pre-training phase, the policy cannot interact with the environment, but in the online fine-tuning phase, the policy has access to the offline dataset.
Similar to offline RL algorithms, offline-to-online RL algorithms' training objectives usually consist of two terms, either explicitly [35, 24] or implicitly [27, 28]: (i) policy improvement, which aims to optimize the policy according to current value functions; (ii) policy constraint, which keeps the policy around the distribution of the offline dataset or current replay buffer. Using the AWR algorithm framework [37, 35, 24], we demonstrate our method (also applicable to non-AWR algorithms) and define notations in Equation (1) for later use.
$$
L _ {\pi} = \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D}} \left[ \overbrace {\exp \left(\underbrace {\beta} _ {\text {b a l i n c e c o e f f i c i e n t}} \left(\underbrace {Q (\mathbf {s} , \mathbf {a}) - V (\mathbf {s})}\right)\right)} ^ {\text {i m i t a t i o n w e i g h t}} \cdot \underbrace {\log \pi (\mathbf {a} | \mathbf {s})} _ {\text {p o l i c y i m p r o v e m e n t}} \right]. \tag {1}
$$
$L_{\pi}$  is a maximization objective, while  $\mathcal{D}$  represents the collected dataset. Initially, during offline pretraining,  $\mathcal{D}$  starts with the pre-collected offline dataset  $\mathcal{D}_{\mathrm{offline}} = \{\left(\mathbf{s}_k,\mathbf{a}_k,\mathbf{s}_k',r_k\right)\mid k = 1,2,\dots ,N\}$ . As we move to the online fine-tuning phase, online interaction samples are continuously incorporated into  $\mathcal{D}$  [35, 24]. The balance coefficient  $\beta$  is a predefined hyperparameter moderating between the policy improvement and policy constraint terms, while the imitation weight sets the imitation intensity for the state-action pair  $(s,a)$ .
3 State-Adaptive Balance Coefficients
Our design of state-adaptive improvement-constraint balances is motivated by the observations that (i) the quality of the dataset's behavior, i.e., the trajectory returns, fluctuates greatly with different states, as shown in Figure 2; (ii) state-dependent balances are conducive to a higher performance upper bound. In this section, we will theoretically validate the latter point.
We first present the policy optimization problem with point-wise KL constraints in Definition 3.1, which is the focus of FamO2O:
Definition 3.1 (Point-wise KL Constrained Optimization Problem). We consider a policy optimization problem defined as follows:
$$
\max  _ {\pi} \mathbb {E} _ {\mathbf {s} \sim d \pi_ {\beta} (\cdot), \mathbf {a} \sim \pi (\cdot | \mathbf {s})} \left[ Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s}) \right] \tag {2}
$$
$$
\mathrm {s . t .} D _ {\mathrm {K L}} (\pi (\cdot | \mathbf {s}) | \pi_ {\beta} (\cdot | \mathbf {s})) \leq \epsilon_ {\mathbf {s}}, \quad \forall \mathbf {s} \in \mathcal {S} \tag {3}
$$
$$
\int_ {\mathbf {a} \in \mathcal {A}} \pi (\mathbf {a} | \mathbf {s}) \mathrm {d} \mathbf {a} = 1, \quad \forall \mathbf {s} \in \mathcal {S}. \tag {4}
$$
Here,  $\pi^k (k\in \mathbb{N})$  denotes the policy at iteration  $k$ ,  $\pi_{\beta}$  signifies a behavior policy representing the action selection way in the collected dataset  $\mathcal{D}$ ,  $d_{\pi_{\beta}}(\mathbf{s})$  refers to the state distribution of  $\pi_{\beta}$ , and  $\epsilon_{\mathbf{s}}$  is a state-related constant. The optimal policy derived from Equations (2) to (4) is designated as  $\pi^{k + 1}$ .
The optimization problem common in previous work [37, 35, 24] is shown in Definition 3.2:
Definition 3.2 (Optimization problem with distributional KL constraints). The definition of the policy optimization problem with distributional KL constraints is the same as Definition 3.1, except that
For simplicity, we use "state" and "observation" interchangeably in fully or partially observed environments.
Equation (3) in Definition 3.1 is substituted by Equation (5), where  $\epsilon$  is a constant:
$$
\int_ {\mathbf {s} \in \mathcal {S}} d _ {\pi_ {\beta}} (\mathbf {s}) D _ {\mathrm {K L}} \left(\pi (\cdot | \mathbf {s}) \| \pi_ {\beta} (\cdot | \mathbf {s})\right) \mathrm {d} \mathbf {s} \leq \epsilon . \tag {5}
$$
Remark 3.3. The update rule in Equation (1) is based on the optimization problem in Definition 3.2.
The point-wise constraints' superiority over distributional constraints is shown in Proposition 3.4:
Proposition 3.4 (Advantage of point-wise KL constraints). Denote the optimal value in Definition 3.1 as  $J_{*}^{k}[\{\epsilon_{\mathbf{s}}, \mathbf{s} \in S\}]$ , the optimal value in Definition 3.2 as  $J_{*}^{k}[\epsilon]$ . These optimal values satisfy:
$$
\forall \epsilon \geq 0, \quad \exists \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\}, \quad J _ {*} ^ {k} [ \left\{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \right\} ] \geq J _ {*} ^ {k} [ \epsilon ]. \tag {6}
$$
Proof. Please refer to Appendix C.1.
Proposition 3.4 indicates that the optimal value under the point-wise KL constraints, given suitable point-wise constraints, is no less than that under distributional KL constraints. This finding justifies our approach under point-wise constraints.
Proposition 3.5 shows the necessity of state-dependent balance coefficient design in solving the point-wise KL constraint optimization problem:
Proposition 3.5 (State-dependent balance coefficient). Consider the optimization problem in Definition 3.1. Assume that the state space  $\mathcal{S} = [s_{\min}, s_{\max}]^l$  ( $l$  is the state dimension), and the feasible space constrained by Equations (3) to (4) is not empty for every  $\mathbf{s} \in \mathcal{S}$ . Then the optimal solution of  $\pi^{k+1}$ , denoted as  $\pi_*^{k+1}$ , satisfies that  $\forall \mathbf{s} \in \mathcal{S}, \mathbf{a} \in \mathcal{A}$ ,
$$
\pi_ {*} ^ {k + 1} (\mathbf {a} | \mathbf {s}) = \frac {\pi_ {\beta} (\mathbf {a} | \mathbf {s})}{Z _ {\mathbf {s}}} \exp \left(\beta_ {\mathbf {s}} \left(Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s})\right)\right), \tag {7}
$$
where  $\beta_{\mathbf{s}}$  is a state-dependent balance coefficient, and  $Z_{\mathbf{s}}$  is a normalization term. When utilizing a parameterized policy  $\pi_{\phi}$  to approximate the optimal policy  $\pi_{*}^{k + 1}$ , the training objective can be formulated as:
$$
\phi = \underset {\phi} {\arg \max } \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D}} \left[ \exp \left(\beta_ {\mathbf {s}} \left(Q ^ {\pi^ {k}} (\mathbf {s}, \mathbf {a}) - V ^ {\pi^ {k}} (\mathbf {s})\right)\right) \log \pi_ {\phi} (\mathbf {a} | \mathbf {s}) \right]. \tag {8}
$$
Proof. The proof is deferred to Appendix C.2.
In contrast to AWR [37] and AWAC [35], Proposition 3.5 highlights state-dependent (marked in blue) balance coefficients in Equations (7) to (8), as opposed to a pre-defined hyperparameter in Equation (1). This state-adaptiveness is due to Proposition 3.5 considering the finer-grained constraints in Definition 3.1. Together, Proposition 3.4 and Proposition 3.5 indicate state-adaptive balance coefficients contribute to a higher performance upper bound.
4 Family Offline-to-Online RL
Section 3 theoretically proves that finer-grained policy constraints enhance performance upper bounds, necessitating state-adaptive balance coefficients. Accordingly, we introduce FamO2O, a framework adaptively assigning a balance coefficient to each state, easily implemented over various offline-to-online algorithms like [35, 24, 27], hereafter called the "base algorithm".
Essentially, FamO2O trains a policy family with varying balance coefficients during offline pretraining. During online fine-tuning, FamO2O
identifies the appropriate policy, corresponding to the suitable balance coefficient, for each state from this policy family. In this section, we present FamO2O using the AWR algorithm framework (Equation (1)). FamO2O's compatibility with non-AWR algorithms is discussed in Appendix D.
Figure 3: FamO2O's inference process. For each state s, the balance model  $\pi_{b}$  computes a state-adaptive balance coefficient  $\beta_{\mathbf{s}}$ . Based on s and  $\beta_{\mathbf{s}}$  the universal model  $\pi_{u}$  outputs an action a.
As shown in Figure 3, FamO2O's policy consists of two components: a universal model  $\pi_u$  and a balance model  $\pi_b$ . Denote the space of balance coefficients as  $\mathcal{B}$ . For every state  $\mathbf{s}$ , the balance model $^2$ $\pi_b : S \mapsto \mathcal{B}$  figures out a suitable balance coefficient  $\beta_{\mathbf{s}}$ ; based on the state  $\mathbf{s}$  and state-related balance coefficient  $\beta_{\mathbf{s}}$ , the universal model  $\pi_u : S \times \mathcal{B} \mapsto \mathcal{A}$  outputs an action. The balance coefficient  $\beta_{\mathbf{s}}$  is to control the conservative/radical degree of the universal model  $\pi_u$  in dealing with the state  $\mathbf{s}$ .
4.1 Learning universal model
We initially address training the universal model  $\pi_u$ , aimed at learning a policy family with varying balances between policy improvement and constraint. The formal optimization target of  $\pi_u$  is:
$$
\pi_ {u} ^ {k + 1} = \underset {\pi_ {u}} {\arg \max } \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D}} \left[ \exp \left(\beta_ {\mathbf {s}} \left(Q ^ {k} (\mathbf {s}, \mathbf {a}) - V ^ {k} (\mathbf {s})\right)\right) \log \pi_ {u} (\mathbf {a} | \mathbf {s}, \boldsymbol {\beta} _ {\mathbf {s}}) \right]. \tag {9}
$$
$Q^{k}$  and  $V^{k}$ , detailed in Section 4.3, represent  $Q$  and  $V$  functions at iteration  $k$ . Equation (9) echoes Equation (8), but the policy also takes a balance coefficient  $\beta_{\mathrm{s}}$  as input (highlighted in blue). In the offline pre-training phase,  $\beta_{\mathrm{s}}$  is randomly sampled from balance coefficient space  $\mathcal{B}$ . This encourages  $\pi_{u}$  to learn varied strategies. During online fine-tuning,  $\beta_{\mathrm{s}}$  is set by balance model  $\pi_{b}$  before input to universal model  $\pi_{u}$ , which prompts cooperation between  $\pi_{u}$  and  $\pi_{b}$ .
4.2 Learning balance model
Next, we outline how the balance model  $\pi_b$  chooses an appropriate policy for each state from the policy family trained by the universal model  $\pi_u$ . As indicated in Section 4.2, every  $\beta_{\mathbf{s}} \in \mathcal{B}$  corresponds to a unique policy. Consequently, to select the optimal policy,  $\pi_b$  needs to determine the appropriate balance coefficient  $\beta_{\mathbf{s}}$  for each state  $\mathbf{s}$ . Given this rationale, the update rule for  $\pi_b$  is:
$$
\pi_ {b} ^ {k + 1} = \underset {\pi_ {b}} {\arg \max } \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D}} \left[ Q ^ {k} (\mathbf {s}, \underbrace {\pi_ {u} ^ {k + 1} (\mathbf {s} , \widetilde {\pi_ {b} (\mathbf {s})})} _ {\text {a c t i o n}}) \right]. \tag {10}
$$
Here,  $\pi_u^{k + 1}$  represents the updated universal model in Equation (9). Intuitively, Equation (10) aims to find a  $\pi_b$  that maximizes  $Q^k$  value by translating balance coefficients into actions with  $\pi_u^{k + 1}$ . This design is grounded in the understanding that the  $Q$  value serves as an estimate of future return, which is our ultimate goal of striking a balance between policy improvement and constraint. Concerns may arise about  $Q^k$ 's extrapolation error in Equation (10) potentially misleading  $\pi_b$ 's update. Empirical evidence suggests this is less of an issue if we avoid extremely radical values in the balance coefficient space  $\mathcal{B}$ . Following the update rule in Equation (10),  $\pi_b$  effectively assigns balance coefficients to states, demonstrated in Section 6.1.
4.3 Learning value functions
Furthermore, we explain the value functions update. As per Equations (9) to (10), a single set of  $Q$  and  $V$  functions evaluate both  $\pi_u$  and  $\pi_b$ . This is due to  $\pi_b: S \mapsto \mathcal{B}$  and  $\pi_u: S \times \mathcal{B} \mapsto \mathcal{A}$  collectively forming a standard RL policy  $\pi_u(\cdot, \pi_b(\cdot)): S \mapsto \mathcal{A}$ . Hence, the value functions update mirrors that in the base algorithm, simply replacing the original policy with  $\pi_u(\cdot, \pi_b(\cdot))$ .
Finally, we offer a pseudo-code of FamO2O's training process in Appendix B.
In this section, we substantiate the efficacy of FamO2O through empirical validation. We commence by showcasing its state-of-the-art performance on the D4RL benchmark [8] with IQL [24] in Section 5.1. We then evidence its performance improvement's generalizability and statistical significance in Section 5.2. Moreover, Section 5.3 reveals FamO2O's compatibility with non-AWR-style algorithms like CQL [27], yielding significant performance enhancement. Lastly, we reserve detailed FamO2O analyses for Section 6 and Appendix F. For more information on implementation details, please refer to Appendix E.2.
2Despite  $\pi_u$  and  $\pi_b$  being stochastic models, we note that them as functions using “ $\mapsto$ ” hereafter for brevity.
D4RL Normalized Score (Normalized to [0, 1])
Figure 5: Comparisons between our FamO2O against various competitors on D4RL normalized scores [8]. All methods are tested on D4RL Locomotion and AntMaze for 6 random seeds. FamO2O achieves state-of-the-art performance by a statistically significant margin among all the competitors in offline-to-online RL (i.e. IQL [24], Balanced Replay (BR) [28], CQL [27], AWAC [35], and TD3+BC [10]), online RL (i.e. SAC [13]), and behavior cloning (BC).
Table 1: Enhanced performance achieved by FamO2O after online fine-tuning. We evaluate the D4RL normalized score [8] of standard base algorithms (including AWAC [35] and IQL [24], denoted as "Base") in comparison to the base algorithms augmented with FamO2O (referred to as "Ours"). All results are assessed across 6 random seeds. The superior offline-to-online scores are highlighted in blue. FamO2O consistently delivers statistically significant performance enhancements across different algorithms and task sets.
Dataset1	AWAC [35]	IQL [24]	Avg.
Base	Ours	Base	Ours	Base	Ours
hopper-mr-v2	56.0	86.8	91.0	97.6	73.5	92.2
hopper-m-v2	54.1	75.0	65.4	90.7	59.7	82.8
hopper-me-v2	97.7	92.9	76.5	87.3	87.1	90.1
halfcheetah-mr-v2	43.9	49.0	53.7	53.1	48.8	51.0
halfcheetah-m-v2	44.8	47.6	52.5	59.2	48.7	53.4
halfcheetah-me-v2	91.0	90.6	92.8	93.1	91.9	91.8
walker2d-mr-v2	72.8	84.4	90.1	92.9	81.5	88.6
walker2d-m-v2	79.0	80.0	83.8	85.5	81.4	82.8
walker2d-me-v2	109.3	108.5	112.6	112.7	110.9	110.6
locomotion total	648.4	714.9	718.3	772.0	683.4	743.4
95% CIs	640.5~656.8	667.3~761.4	702.5~733.5	753.5~788.5	674.6~692.0	732.1~754.2
umaze-v0	64.0	96.9	96.5	96.7	80.4	96.8
umaze-diverse-v0	60.4	90.5	37.8	70.8	66.2	80.6
medium-diverse-v0	0.2	22.2	92.8	93.0	45.2	57.6
medium-play-v0	0.0	34.2	91.5	93.0	45.2	63.6
large-diverse-v0	0.0	0.0	57.5	64.2	24.7	32.1
large-play-v0	0.0	0.0	52.5	60.7	21.4	30.3
antmaze total	124.7	243.7	428.7	478.3	283.1	361.1
95% CIs	116.5~132.6	226.2~259.9	406.7~452.7	456.7~498.7	274.1~291.1	347.2~374.3
total	773.0	958.6	1146.9	1250.3	960.0	1104.5
95% CIs	761.5~784.6	936.8~979.6	1119.5~1175.1	1221.9~1277.0	911.3~1008.8	1063.5~1145.4
$^{1}$  mr: medium-replay, m: medium, me: medium-expert.
Datasets Our method is validated on two D4RL [8] benchmarks: Locomotion and AntMaze. Locomotion includes diverse environment datasets collected by varying quality policies. We utilize IQL [24] settings, assessing algorithms on hopper, halfcheetah, and walker2d environment datasets, each with three quality levels. AntMaze tasks involve guiding an ant-like robot in mazes of three sizes (umaze, medium, large), each with two different goal location datasets. The evaluation environments are listed in Table 1's first column.
Figure 4: FamO2O's improvement over base algorithms [35, 24]. For D4RL Locomotion, AntMaze [8], and overall, FamO2O shows significant and meaningful performance gains, meeting Neyman-Pearson criteria [3].
Figure 6: IQM scores of FamO2O's implementation on top of CQL [27]. Demonstrating a statistically significant superiority over vanilla CQL [27] and CQL+BR [28], FamO2O affirms its adaptability to non-AWR algorithms.
Figure 7: Comparing FamO2O with the base algorithm [24] on (a) imitation weights and (b) action distances.  $\Delta$  indicates the difference between FamO2O's metrics and the base's. Generally, FamO2O emphasizes high-quality data imitation and aligns more closely with high-quality trajectories compared to the base algorithm.
Metrics Considering RL's inherent variability, we adopt robust evaluation methods per riable [3]. Besides conventional Medium and Mean, we integrate IQM and Optimality Gap metrics for broader assessment. We also employ riable's probability of improvement metric for gauging the likelihood of our method outperforming others. We confirm our performance enhancement's statistical significance using  $95\%$  Confidence Intervals (CIs).
5.1 Benchmark Comparison
FamO2O's state-of-the-art performance is demonstrated by implementing it over IQL [24] and comparing with baseline methods.