SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines We benchmark FamO2O against: (i) offline-to-online RL, including IQL [24], Balanced Replay (BR) [28], CQL [27], AWAC [35], and TD3+BC [10]. For IQL, BR, and AWAC, official implementations are used. In the case of CQL and TD3+BC, we implement online fine-tuning based on the author-provided offline pre-training codes, following the procedures in IQL and AWAC; (ii) online RL method, SAC [13], to highlight offline pre-training's efficacy; (iii) behavior cloning (BC), which is implemented by maximizing the log-likelihood of the samples in the offline dataset. For SAC and BC, we utilize the implementations of CQL. Further details are in Appendix E.1.
Comparison As shown in Figure 5, FamO2O outperforms competitors across all metrics (IQM, Medium, Optimality Gap, and Mean). Specifically, for IQM, Optimality Gap, and Mean, FamO2O's  $95\%$  CIs don't overlap with the competitors'. Even for Medium, all baseline expectations fall below the lower limit of FamO2O's  $95\%$  CIs. The results underscore the significant edge of our state-adaptive policy constraint mechanism over competing methods.
5.2 Analyzing FamO2O's Performance Enhancement
Though FamO2O demonstrated superior performance on the D4RL benchmark in Section 5.1, it's vital to discern the actual contributions of FamO2O from its base policy, IQL. Therefore, we address two key questions: (i) Does FamO2O consistently enhance other offline-to-online RL algorithms? (ii) Is the performance boost by FamO2O statistically significant given RL's inherent variability?
Setup We apply FamO2O to AWAC and IQL. AWAC [35] is one of the most famous offline-to-online algorithms, and IQL [24] is a recently proposed method that achieves great performance on D4RL [8]. We use the authors' codes, maintaining the same hyperparameters for a fair comparison. Further details are in Appendix E.2.
Comparison Table 1 shows AWAC's and IQL's performances w/ and w/o FamO2O. FamO2O generally enhances performance by a statistically significant margin across most datasets, regardless of the base algorithm, highlighting its versatility. Even on complex datasets where AWAC barely succeeds, e.g., AntMaze medium-diverse and medium-play, FamO2O still achieves commendable performance.
Pursuing riable's recommendation [3], we evaluated FamO2O's statistical significance by calculating average probabilities of improvement against base policies (Figure 4). In all three cases (Locomotion,
AntMaze, and Overall), the lower CI bounds exceed  $50\%$ , denoting the statistical significance of FamO2O's improvement. Specifically, the upper CI on Locomotion surpasses  $75\%$ , demonstrating statistical meaning as per the Neyman-Pearson criterion.
5.3 Versatility of FamO2O with Non-AWR Algorithms
To demonstrate FamO2O's versatility beyond AWR-based algorithms, we extended it to CQL [27] in addition to AWAC [35] and IQL [24]. The implementation specifics are in Appendix D. As Figure 6 reveals, FamO2O significantly outperforms CQL. Even when compared to Balance Replay (BR) [28], an offline-to-online method designed specifically for CQL, FamO2O still shows statistically significant superior performance. These results highlight FamO2O's adaptability to non-AWR algorithms.
In this section, we further provide some in-depth studies on FamO2O, including visualization (Section 6.1) and quantitative analyses (Sections 6.2 to 6.6). More analyses are deferred to Appendix F.
6.1 Does FamO2O really have state-wise adaptivity?
Here, we design a simple maze environment to visualize the state-wise adaptivity of FamO2O. As shown in Figure 8(a), the agent starts at a random cell in the top row and is encouraged to reach the goal at the bottom right corner through two crossing points. During data collection, guidance is provided when the agent passes through the lower crossing point, but no guidance for the upper crossing point. To elaborate, the guidance refers to compelling the agent to adhere to the route and direction that yields the shortest path to the goal. Without it, the agent moves randomly. It can be observed in Figure 8(b) that the agent generally outputs lower balance coefficients for the states with high-quality samples (i.e., those derived from the agent's movement with guidance) while generating higher
balance coefficients for the states with low-quality data (i.e., data gathered when the agent moves without guidance). This result shows FamO2O's state-wise adaptivity in choosing proper balance coefficients according to the data quality related to the current state.
Figure 8: State-wise adaptivity visualization in a simple maze environment. (a) Higher data quality at the crossing point in the 5th row compared to the 2nd row. (b) Colors denote different balance coefficient values at traversed cells during inference. FamO2O typically displays conservative (or radical) behavior at cells with high-quality (or low-quality) data.
6.2 What is the effect of state-adaptive balances?
In this section, we explore the impact of state-adaptive improvement-constraint balance. In order to encompass data of varying quality levels, we assess FamO2O using the medium-replay datasets of D4RL Locomotion [8]. Our analysis focuses on two metrics: imitation weights and action distances.
Imitation weights are defined in Equation (1), with larger (or smaller) values prompting the agent to align more closely (or less) with the replay buffer  $\mathcal{D}$ 's behavior. Action distance, delineated in Equation (11), quantifies the discrepancy in action selection between a policy  $\pi$  and a trajectory  $\tau$ :
$$
d _ {\text {a c t i o n}} ^ {\pi , \tau} = \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \tau} \left[ \| \underset {\mathbf {a} ^ {\prime}} {\arg \max } \pi \left(\mathbf {a} ^ {\prime} | \mathbf {s}\right) - \mathbf {a} \| _ {2} ^ {2} \right]. \tag {11}
$$
Here, a lower (or higher) action distance  $d_{\mathrm{action}}^{\pi,\tau}$  signifies a greater (or lesser) behavioral similarity between the policy  $\pi$  and the trajectory  $\tau$ .
We evaluate IQL with and without FamO2O ('Base' and 'FamO2O') regarding imitation weights and action distances, depicted in Figure 7. Figure 7(a) computes the average imitation weight difference (AIWD) per trajectory in the offline dataset. AIWD indicates the mean imitation weight difference between FamO2O and the base algorithm for each  $(s,a)$  pair within a trajectory. Figure 7(b) likewise determines an average action distance difference per offline dataset trajectory.
(a) Ablation on AWAC
(b) Ablation on IQL
Figure 9: Comparing adaptive and fixed balance coefficients on (a) AWAC [35] and (b) IQL [24] using D4RL Locomotion datasets [8]. Adaptive coefficients consistently outperform fixed ones with distinct  $95\%$  CIs.
Figure 10: FamO2O alleviates performance drop due to distributional shift during the shift from offline pretraining to online fine-tuning.
Figure 7(a) reveals that FamO2O typically shows higher imitation weights than the base algorithm for high-return trajectories. Figure 7(b) indicates that FamO2O aligns more with high-quality trajectories and less with low-quality ones than the base algorithm. These results highlight the state-adaptive balance's role in promoting emulation of high-quality behavior and avoidance of low-quality behavior.
6.3 State-adaptive balances vs. fixed balances?
To prove that our adaptive balance coefficients are better than traditional fixed balance coefficients, we compare FamO2O against the base algorithms with different balance coefficients as hyperparameters. As shown in Figure 9, on both AWAC [35] and IQL [24], our adaptive balance coefficients outperform all the fixed balance coefficients. Significantly, the  $95\%$  CIs of adaptive and fixed balance coefficients have no overlap. These comparison results indicate that our adaptive balance coefficient approach surpasses the fixed balance coefficient method by a statistically significant margin.
6.4 Does FamO2O's efficacy rely on varied data qualities?
It's worth noting that our method's efficacy doesn't rely on varied data qualities. Table 1 clearly demonstrates that FamO2O surpasses the base algorithms in performance across all the medium datasets, namely hopper-medium-v2, halfcheetah-medium-v2, and walker2d-medium-v2, which all maintain a relatively consistent data quality. We claim that FamO2O can determine the suitable conservative/radical balance for each state in online scenarios based on the data quality in the collected dataset. If the dataset is diverse in quality, the balances will be diverse; if the quality is consistent, the balances will be correspondingly consistent. The above claim is supported by Figure 11, which indi
Figure 11: Balance coefficients' mean and std for IQL w/FamO2O on D4RL HalfCheetah.
cates that (i) in datasets with more (or less) diverse data qualities, i.e., medium-expert (or medium), the balance coefficients are more (or less) diverse, with corresponding larger (or smaller) standard deviations; (ii) with higher (or lower) quality datasets, the balance coefficients are averagely lower (or higher), leading to a more conservative (or radical) policy.
6.5 Does FamO2O mitigate the performance drop stemming from the distributional shift?
Despite FamO2O's primary objective not being direct distributional shift handling, its state-adaptive improvement-constraint balances prove beneficial in mitigating performance degradation during the
offline pre-training to online fine-tuning transition, attributed to the distributional shift. Existing offline-to-online RL algorithms [24, 35] already incorporate mechanisms to counter distributional shift, hence significant performance drops are infrequent. Figure 10 illustrates the training curves of IQL and IQL+FamO2O across both offline pre-training (negative steps) and online fine-tuning (positive steps) on antmaze-umaze-diverse, on which IQL exhibits the most significant decline in performance when transitioning from offline pre-training to online fine-tuning. As evidenced, while IQL+FamO2O initially experiences a performance drop akin to IQL during online fine-tuning, it recovers rapidly and attains high performance, in stark contrast to IQL's sustained performance decline throughout the fine-tuning stage.
6.6 Balance model vs. random selector?
To validate the effect of the balance model  $\pi_{b}$  in choosing balance coefficients, we present a FamO2O variant, denoted as random-FamO2O, with the balance model replaced with a random balance coefficient selector. Other training settings keep the same for FamO2O and random-FamO2O. Table 2 shows the improvement percentages and probability of improvement of FamO2O against random-FamO2O. As we can observe, FamO2O outperforms random-FamO2O on almost all the datasets of D4RL Locomotion [8]. Furthermore, the lower CI of the probability of improvement is much higher than  $50\%$ , and the upper CI exceeds  $75\%$ . This indicates that the effect of the balance model is not only statistically significant but also statistically meaningful as per the Neyman-Person statistical testing criterion.
Table 2: Ablation study on the balance model. The absence of the balance model results in decreased performance of FamO2O on nearly all D4RL Locomotion datasets [8]. The model's impact is statistically significant and meaningful as per the Neyman-Person criterion.
Dataset	FamO2O	random-FamO2O
hopper-mr-v2	97.64	80.87
hopper-m-v2	90.65	86.37
hopper-me-v2	87.28	77.08
halfcheetah-mr-v2	53.07	53.75
halfcheetah-m-v2	59.15	53.15
halfcheetah-me-v2	93.10	92.72
walker2d-mr-v2	92.85	91.38
walker2d-m-v2	85.50	84.84
walker2d-me-v2	112.72	110.54
total	771.96	730.70
prob. of improvement	0.70 (95% CIs: 0.59~0.79)
This work underscores the significance of state-adaptive improvement-constraint balances in offline-to-online RL. We establish a theoretical framework demonstrating the advantages of these state-adaptive balance coefficients for enhancing policy performance. Leveraging this analysis, we put forth Family Offline-to-Online RL (FamO2O), a versatile framework that equips existing offline-to-online RL algorithms with the ability to discern appropriate balance coefficients for each state.
Our experimental results, garnered from a variety of offline-to-online RL algorithms, offer substantial evidence of FamO2O's ability to significantly improve performance, attaining leading scores on the D4RL benchmark. In addition, we shed light on FamO2O's adaptive computation of state-adaptive improvement-constraint balances and their consequential effects through comprehensive analyses. Ablation studies on the adaptive balances and balance model further corroborate FamO2O's efficacy.
The limitation of our work is that FamO2O has been evaluated on just a handful of representative offline-to-online RL algorithms, leaving a vast array of such algorithms unexamined. Additionally, FamO2O's utility is somewhat limited, as it is applicable exclusively to offline-to-online algorithms. In future work, we aim to expand the applicability of FamO2O by integrating it with a broader spectrum of offline-to-online algorithms. Additionally, we intend to investigate the feasibility of incorporating state-adaptive improvement-constraint balances in offline RL settings, where online feedback is intrinsically absent, or even applying our adaptive design to the LLM-as-agent domain [49, 50].
This work was supported in part by the Key-Area Research and Development Program of Guangdong Province under Grant 2020B1111500002, the National Natural Science Foundation of China under Grants 62022048 and 62276150, and the Guoqiang Institute of Tsinghua University.
[65] Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072, 2020.
[66] Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210.13846, 2022.
A Related Work 15
B Training Process of FamO2O 16
C Theoretical Proofs 17
C.1 Proof of Proposition 3.4 17
C.2 Proof of Proposition 3.5 18
D FamO2O's Extension to Non-AWR Algorithms 19
E Implementation Details 20
E.1 Baseline Implementation 20
E.2 Implementation of FamO2O 21
F More Experimental Results 21
F.1 Sensitivity Analysis on Different Ranges of Balance Coefficients 21
F.2 Comparing FamO2O and Balance Coefficient Annealing 22
G Potential Negative Societal Impacts 22
Offline RL Reinforcement learning (RL) usually improves a policy through continuous online interactions with the environment [43, 32, 62]. To reduce the huge demand for online interactions, especially when they are costly or risky, researchers proposed offline RL that utilizes pre-collected datasets to improve the policy without interacting with the environment [29]. Directly applying off-policy algorithms in offline RL usually leads to poor performance. This phenomenon is due to distributional shift, where the agent may learn inaccurate value estimates of out-of-distribution (OOD) state-action pairs [29, 61, 60]. Existing algorithms address this issue via policy constraints [11, 25, 54, 23, 37, 35], importance sampling [19, 14, 65], regularization [33, 27], uncertainty estimation [2], and imitation learning [4, 42, 52].
Offline-to-online RL Offline-to-online RL adds online fine-tuning to offline RL to enhance policy performance. Similar to the distributional shift problem in offline RL, offline-to-online (O2O) RL also suffers from off-policy bootstrapping error accumulation caused by OOD data, which causes a large "dip" in the initial performance of online fine-tuning [35]. Moreover, O2O RL algorithms tend to be excessively conservative and result in plateauing performance improvement [35]. This emphasizes the importance of finding a balance between being optimistic about improving the policy during the online phase and still being constrained to the conservative offline policy. However, as previously discussed, existing O2O RL algorithms generally adopt a "one-size-fits-all" trade-off, either in the process of Q-learning [27, 30, 24, 31], along with the policy improvement [35], via a revised replay buffer [28], or through some alignment approaches [53, 59]. It's worth noting that, [66] also proposes an adaptive weight in online fine-tuning. However, the adaptive weight is a single parameter for all the samples. In addition, its update requires an extra human-defined target score, which is absent in the setting considered by our competitors and us. Furthermore, [63] presents a policy expansion strategy that adaptively selects a policy from a set, which includes both offline pre-trained policies and online learned policies. Nevertheless, this policy set comprises only two policies, in contrast to the infinite number of policies contained in FamO2O's policy family.
Conditioned RL policy Algorithms in this scheme can be divided into two stages: (i) a set of policies is trained; (ii) the policy with the highest performance is selected. Generally, these methods learn a family of policies that exhibit different behaviors and search for the best one that successfully executes the desired task [36, 57]. A similar method to our FamO2O is UP-OSI [58], where a Universal Policy (UP) within a parameterized space of dynamic models and an Online System Identification (OSI) function that predicts dynamic model parameters based on recent state and action history are proposed. Unlike these prior methods utilizing models to identify the environmental parameters, the balance model in our method aims to determine a proper balance between policy improvement and constraints for each state. It's worth noting that [18, 45, 12] also exhibit conditioned designs. However, [18] focuses on conditioned value functions and pure offline RL; [45] relies on human interactions; [12] concentrates on adjusting offline RL policies according to the estimated probabilities of various MDPs in uncovered areas, but uncoverage might not be a primary concern in offline-to-online RL due to the accessibility to the online environment. Moreover, [56, 55] both exhibit a comparable understanding of how to adjust policy improvement and policy constraint across samples of varying quality. Nevertheless, the focus of [56, 55] is on offline RL. Specifically, [56] requires a small number of expert demonstrations, while [55] necessitates annotated preferences to appropriately calibrate the balance for each individual sample. Additionally, in areas such as computer vision, there are existing works that condition their training and/or inference on various input samples [15, 16, 40, 38, 39, 51]. These studies align with the adaptive approach in our work.
Hierarchical RL Hierarchical Reinforcement Learning (HRL) decomposes complex tasks into manageable subtasks, enhancing sample efficiency, interpretability, and generalization in RL. HRL approaches bifurcate into predefined and learned options (or skills). Predefined options, guided by domain knowledge or human demonstrations, shape high-level policy or constrain action space, exemplified by MAXQ [6], options framework [44], skill chaining [22], and feudal RL [5]. Learned options, optimizing objectives like entropy or reward, are seen in option-critic, skill-discovering [22], feudal networks [48], HEXQ [17], DIAYN [7], HIRO [34], and HIDIO [64]. While FamO2O shares some design elements with HRL, such as hierarchical policy decision-making, the objectives diverge: FamO2O focuses on balancing policy improvement and constraints per state, while HRL seeks to simplify complex tasks via goals or options.
B Training Process of FamO2O
Here we provide the pseudo-codes of FamO2O (see Algorithm 1) to demonstrate its training process.
Algorithm 1 FamO2O Algorithm
Require: replay buffer  $\mathcal{D}$  offline dataset  $\mathcal{D}_{\mathrm{offline}}$  offline and online training steps  $N_{\mathrm{off}},N_{\mathrm{on}}$  1: Initialize  $\pi_b^0,\pi_u^0,Q^0,V^0$  2: Initialize  $\mathcal{D}$  with  $\mathcal{D}_{\mathrm{offline}}$  3: for  $k = 0\rightarrow N_{\mathrm{off}} - 1$  do > offline pre-training phase 4: Sample a mini-batch  $M_{k}$  from  $\mathcal{D}$  5: Assign a random balance coefficient for each sample in  $M_{k}$  , denoting the balance coefficient set as  $B_{M_k}$  6: Update  $\pi_u^k$  to  $\pi_u^{k + 1}$  with  $M_{k},B_{M_{k}}$  by Equation (9) 7: Update  $\pi_b^k$  to  $\pi_b^{k + 1}$  with  $M_{k}$  by Equation (10) 8: Update  $Q^{k},V^{k}$  to  $Q^{k + 1},V^{k + 1}$  respectively with  $M_{k}$  according to the base algorithm 9: end for 10: for  $k = N_{\mathrm{off}}\to N_{\mathrm{on}} - 1$  do >online fine-tuning phase 11: Collect samples with  $\pi_u^k,\pi_b^k$  and add samples to  $\mathcal{D}$  12: Sample a mini-batch  $M_{k}$  from  $\mathcal{D}$  13:  $\pi_b^k$  computes a balance coefficient for each sample in  $M_{k}$  . Denote the balance coefficient set as  $B_{M_k}$  14: Update  $\pi_u^k$  to  $\pi_u^{k + 1}$  with  $M_{k},B_{M_{k}}$  by Equation (9) 15: Update  $\pi_b^k$  to  $\pi_b^{k + 1}$  with  $M_{k}$  by Equation (10) 16: Update  $Q^{k},V^{k}$  to  $Q^{k + 1},V^{k + 1}$  respectively with  $M_{k}$  according to the base algorithm 17: end for
C Theoretical Proofs
C.1 Proof of Proposition 3.4
Lemma C.1. Equation (5) is equivalent to:
$$
\exists \{\epsilon_ {\mathbf {s}}, \mathbf {s} \in \mathcal {S} \} \in C, \quad \forall \mathbf {s} \in \mathcal {S}, \quad D _ {\mathrm {K L}} (\pi (\cdot | \mathbf {s}) \| \pi_ {\beta} (\cdot | \mathbf {s})) \leq \epsilon_ {\mathbf {s}}, \tag {12}
$$
$$
w h e r e C = \left\{\left\{\epsilon_ {\mathbf {s}} ^ {\prime}, \mathbf {s} \in \mathcal {S} \right\} \mid \int_ {\mathbf {s} \in \mathcal {S}} d _ {\pi_ {\beta}} (\mathbf {s}) \epsilon_ {\mathbf {s}} ^ {\prime} \mathrm {d} \mathbf {s} = \epsilon , \epsilon_ {\mathbf {s}} ^ {\prime} \geq 0 \right\}. \tag {13}
$$
Proof. We aim to prove Equation (5)  $\Rightarrow$  Equations (12) to (13) in our manuscript through a contradiction approach. Let's assume that Equation (5)  $\nRightarrow$  Equations (12) to (13), signifying the following:
1. Condition 1 According to Equation (5),  $\int_{\mathbf{s} \in \mathcal{S}} d_{\pi_\beta(\mathbf{s})} D_{\mathrm{KL}}(\pi(\cdot|\mathbf{s}) \| \pi_\beta(\cdot|\mathbf{s})) \, \mathrm{d}\mathbf{s} \leq \epsilon$ ;
2. Condition 2 The converse proposition of Equations (12) to (13) is:  $\forall \epsilon_{\mathbf{s}} \geq 0$  ( $\mathbf{s} \in S$ ) which satisfy  $\int_{\mathbf{s} \in S} d_{\pi_{\beta}}(\mathbf{s}) \epsilon_{\mathbf{s}} \mathrm{d}\mathbf{s} = \epsilon$ , there exists  $\mathbf{s} \in S$ ,  $D_{\mathrm{KL}}(\pi(\cdot|\mathbf{s}) \| \pi_{\beta}(\cdot|\mathbf{s})) > \epsilon_{\mathbf{s}}$ .
Next, we form a special set  $\{\epsilon_{\mathbf{s}}, \mathbf{s} \in \mathcal{S}\}$ . Selecting an arbitrary  $\mathbf{s}_0 \in \mathcal{S}$ , the set fulfills:
1. For all  $\mathbf{s} \neq \mathbf{s}_0$ ,  $\epsilon_{\mathbf{s}} = D_{\mathrm{KL}}(\pi (\cdot |\mathbf{s})\| \pi_{\beta}(\cdot |\mathbf{s}))$
2.  $\epsilon_{\mathbf{s}_0} = \epsilon -\int_{\mathbf{s}\in S,\mathbf{s}\neq \mathbf{s}_0}d_{\pi_\beta}(\mathbf{s})D_{\mathrm{KL}}(\pi (\cdot |\mathbf{s}_0)\| \pi_\beta (\cdot |\mathbf{s}_0))\mathrm{d}\mathbf{s}.$
It can be clearly seen that the set adheres to  $\int_{\mathbf{s} \in S} d_{\pi_{\beta}}(\mathbf{s}) \epsilon_{\mathbf{s}} \mathrm{d}\mathbf{s} = \epsilon$ . Based on Condition 2, we can deduce that  $D_{\mathrm{KL}}(\pi(\cdot|\mathbf{s}_0) \| \pi_{\beta}(\cdot|\mathbf{s}_0)) > \epsilon_{\mathbf{s}_0}$ . Hence,