Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
Authors: Shenzhi Wang
Qisen Yang
Jiawei Gao
Matthieu Gaetan Lin
HAO CHEN
Liwei Wu
Ning Jia
Shiji Song
Gao Huang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: reinforcement learning
offline-to-online reinforcement learning
offline reinforcement learning
policy improvement
policy constraint
TL;DR: FamO2O, a novel plug-in framework for offline-to-online reinforcement learning, effectively determines state-adaptive improvement-constraint balances, resulting in significant performance improvements on the D4RL benchmark.

Abstract: Offline-to-online reinforcement learning (RL) is a training paradigm that combines pre-training on a pre-collected dataset with fine-tuning in an online environment. However, the incorporation of online fine-tuning can intensify the well-known distributional shift problem. Existing solutions tackle this problem by imposing a policy constraint on the policy improvement objective in both offline and online learning. They typically advocate a single balance between policy improvement and constraints across diverse data collections. This one-size-fits-all manner may not optimally leverage each collected sample due to the significant variation in data quality across different states. To this end, we introduce Family Offline-to-Online RL (FamO2O), a simple yet effective framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O utilizes a universal model to train a family of policies with different improvement/constraint intensities, and a balance model to select a suitable policy for each state. Theoretically, we prove that state-adaptive balances are necessary for achieving a higher policy performance upper bound. Empirically, extensive experiments show that FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark. Codes are available at https://github.com/LeapLabTHU/FamO2O.

Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
Shenzhi Wang Qisen Yang'*, Jiawei Gao', Matthieu Lin?, Hao Chen', Liwei Wu
Ning Jia Shiji Song' Gao Huang
Department of Automation, BNRist, Tsinghua University
Department of Computer Science, BNRist, Tsinghua University
3 Beijing Academy of Artificial Intelligence (BAAI) Independent Researcher
Project Page: sstps://shenzhi-wee gi thub.
Abstract
Offline-to-on to-lnne reinforcement learning (RL) is training paradigm that combines pre-training on a pre-collected dataset with fine-tuning in an online environment. However, the incorporation of online fine-tuning can intensify the well-known distributional shift problem. Existing solutions tackle this problem by imposing a policy constraint on the policy improvement objective in both offline and online learning. They typically advocate single balance between policy improvement and constraints across diverse data collections. This see-siiefiittaal manner may not optimally leverage each collected sample due to the significant variation in data quality across different states. To this end, we introduce Family Offline-tee Coiiee-to-Oniine to-Online RL (FamO20), simple yet effective framework that empowers existing algorithms to determine adaptid improvement-constre constraint balances. FamO20 utilizes a universal model train family of policies with different improvement/constraann intensities, and a balance model to select a suitable policy for each state. The- oretically, prove that state-adaptive balances necessary for achieving higher policy performance upper bound. Empirically, extensive experiments show that FamO20 offers statistically significant improvement over various existing methods, achieving state-of- performance on the D4RL benchmark. Codes are available at https: tettthuu.ccom hatpa//c/a/aaaaaaa/////aaa02222
Train policy family based on the collected dataset
larn policy selection from the online environment feedbacks
vanilla
FamO20
AWAC
IQL
COL
collected dataset
online environment
feedback
65
85
105
100
80
train
haried-balance policy family
policy selector
action
57
75
conservative
redical
state-wise select
65
(a) General intuitions of our proposed Fam020
(b) Fam020's performance enhancement
Figure 1: FamO20 trains policy family from datasets and selects policies state-adaptively using
online feedback. Easily integrated, FamO20 statistically enhances existing algorithms' performance.
Eqqul contribution. Email: [wsz21, yanggs19) tsinghua. edw.huauaeedd..eeee en.
1 Corresponding author. Email: gaohuang@tsinghus eereeegguuuuuuuueeee cn.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Offline reinforcement learning (RL) provides pragmatic methodology for acquiring policies utillizing
pre-existing datasets, cireumventing the need for direct environment interaction [29]. Nonetheless, the attainable policy performance in offline RL is frequently constrained by the quality of the dataset [66]. The offline-to-onllee RL paradigm addresses this limitation by refining the offline RL policy through
fine-tuning in an online setting [35]
While online fine-tuning can indeed elevate policy performance, it also potentially exacerbates the issue of distributional shift [29], where policy behavior diverges from the dataset distribution. Such shifts typically ensue from drastic policy improvements and are further amplified by state distribution changes when transitioning from offline learning to online fine-tuning [T 25 2 26]. Prior works have attempted to counter this by imposing policy constraints
the policy improvement objective to deter excessive exploration of uncharted policy space [25 54 35]. However, this conservative approach can inadvertently stifle policy improvement (35). In essence, offline-to-olline RL necessitates effective balance between policy improvement and policy constraint during policy optimization.
Regrettably, prior offline-to dffline-to-online to-online RL algorithms tend to adopt a monolithic approach towards this improvemment-constrainn trade-off, indiseriminately applying it to all data in a mini-batch (T0, 66 53 or
high-quality
the entire dataset [35 24 27 30).
Given the inherent data quality vari-
ation across states (see Figure 2).
low-quality
we argue that this one-size- fits-all manner may fail to optimally exploit each sample. In fact, data yielding high trajectory returns should courage more "conservative" poli- cies, while data leading poor turns should incite more "radical" policy improvement.
(@) HalfChectah
(b) Hopper
Figure t-SNE 46 visualization of randomly selected states from (a) HalfCheetah and (b) Hopper medium-expert datasets D4RL. R The color coding represents the return of the trajectory associated with each state. This visualization underscores the significant variation in data quality across different states.
In this paper, we introduce a novel framework, Family Offline-to-Online RL (FamO20), which can discern state-adaptive improvement-consse on two key insights delineated in Figure [a): (i) The collected dataset, abundant in environmental information, could facilitate the training of diverse policy family ranging from conservative to radical, and (ii) feedback from online learning might assist in selecting an appropriate policy from this family for each state. As depicted in Figure 3, FamO20 incorporates universal model and balance model. The universal model, conditioned on a balance coefficient, determines the degree of policy conservatism radicalism, while the balance model learns mapping from states balance coefficients, aiding the universal model in tailoring its behavior to each specific state.
balance for each state. FamO2O's design is founded
FamO20 represents, to the best of our knowledge, the first offline-toonlliee RL approach harnessing the power of state-adaptive improvement-constraint balances. Theoretically. we establish that in policy optimization, point-wise KL constraints afford superior performance upper bound compared to the distributional KL constraints adopted in prior works [37 35] 24]. Importantly, these adaptie balances become indispensable when addressing point- wise KL constraints, thereby underlining the necessity of incorporating such balances in offline-to-online RI. Experimental results, as summarized in Figure bb)) reveal that FamO20 is simple yet effective framework that statistically significantly improves various offline-to-online RI. algorithms and achieves state-of-the-art performance.
2 Preliminaries
We introduce essential RL and oflline-t-onnlie RL concepts and notations here. For descriptive convenience and theoretical analysis. we use the eevantaged-weigee regression (AWR) algorithm framework (37 35 24), but Fam020 isn't limited to the AWR framework. We later demonstrate its integration with non-AWR www.oiiiitttaaoaa. algorithms in Section 5.3 and Appendix

---

2.1 Reinforcement learning formulation
RL is typically expressed as a Markov decision process (MDP) 43 denoted as (S,A,P,de R, 7). Here. 8 and A are the state" and action space; P(s SS, a) is the environmental state transition probability; do (8p) represents initial state distribution; R(s ag 8441) is the reward function; and y (0, 1] is the discount factor.
2.2 Offineeto-online reinforcement learning
Offline-to-online RL a training paradigm including two phases: (i) offtine pre-training: pre-training policy based offline dataset; (ii) online fine-tuning: fine-tuning the pre-trained policy by interacting with the environment. Note that in the offiine pre-training phase, the policy cannot interact with the environment, but in the online fine-tuning phase, the policy has access the offline dataset.
Similar to offline RL algorithms, offlnee-o-online RL algorithms' training objectives usually consist of two terms, either explicitly [35] 24] or implicitly [27 [28]: (i) policy improvement, which aims to optimize the policy according to current value functions; (ii) policy constraint, which keeps the policy around the distribution of the offline dataset or current replay buffer. Using the AWR algorithm we demonstrate our method (also applicable to non-AWR algorithms) and (T) for later use.
framework [37 35 24),
define notations in Equation
initation weight
L = E (R.aNDD
exp
3 (Q(s,)) V(())
log π (a|s)
balance coefficient
policy improvement
policy constraint
Ln is maximization objective, while D represents the collected dataset. Initially, during offline pre- training. D starts with the pre-collected offline dataset Doftlne {(s,, a rk) 1, N.. As move to the online fine-tuning phase, online interaction samples continuously incorporated into D [35 24).. The balance coefficient 3 is predefined hyperparameter moderating between the policy improvement and policy constraint terms, while the imitation weight sets the imitation intensity for the state-action pair (s, a).
3 State-Adaptive Balance Coefficients
Our design of state-adaptive improvemeent-onntrriint balances is motivated by the observations that (i) the quality of the dataset's behavior, i.e., the trajectory returns, fluctuates greatly with different states, as shown in Figure 2 (ii) state-dependent balances are conducive to a higher performance upper bound. In this section, we will theoretically validate the latter point.
We first present the policy optimization problem with point-wise KL constraints Definition 3.1 which is the focus of FamO20:
Definition 3.1 (Point-wise KI. Constrained Optimization Problem). We consider policy optimization problem defined follows:
max Egwds (.).ds(()),,x((11e) (Q= (s. -aa)-**(p (s)
(2)
x
s.t. DKL (T)((()) co. Vs E
(3)
r(als) V. aEA
(4)
Here, (* N) denotes the policy at iteration k, ng signifies behavior policy representing the action selection way in the collected dataset D, d (s) refers to the state distribution of Tg, and 6g is state-related constant. The optimal policy derived from Equations to 44 is designated as k+1
The optimization problem common in previous work 37 35
is shown Definition 3.2
Definition 3.2 (Optimization problem with distributional KL constraints). The definition of the policy optimization problem with distributional KL constraints is the same as Definition 3.1 except that
For simplicity, "state" and "observation" interchangeably fully partially observed environments.

---

Equation 3 in Definition 3.1 is substituted by Equation 5 where is constant:
/ Jess dng (T))))))(((((((s)))))((s) ds S
(5)
Remark 3.3. The update rule in Equation ① is based the optimization problem in Definition 3.2
The point-wise constraints' superiority over distributional constraints is shown in Proposition 3.4 Proposition 3.4 (Advantage of point-wise KL constraints). Denote the optimal value in Definition as J [(xx S) the optimal value in Definition 3.2 as [%(e). These optimal values satisfy:
3.7
VE 2 0, E SS, He SH 2 J.(e).
(6)
Proof. Please refer to Appendix C.I
Proposition 3.4 indicates that the optimal value under the point-wise KL constraints, given suitable point-wise constraints, is no less than that under distributional KL. constraints. This finding justifies approach under point-wise constraints.
Proposition 3.5 shows the necessity of dependent balance coefficient design in solving the point-wise KI. constraint optimization problem:
Proposition 3,5 (State-dependent balance coefficient). Consider the optimization problem in Defini- tion 3.1 Assume that the state space S [smin: Smax (1 is the state dimension), and the feasible space constrained by Equations (3) to (4) is not empty for every s S. Then the optimal solution of ++1 denoted x+1, satissies that vs E S, A.
(als) (a(a)s) Z, cxp
(s,
-s.a)
where Bg is a state-dependent balance coefficient, and Zg is normalization term. When utilizing a parameterized policy Te approximate the optimal policy xh+1 the training objective can be formulated as:
Φ arg ma exp(3s(Q"* (s. V= (s)) log) (
(8)
Proof. The proof deferred Appendix C.2
In contrast to AWR 37 and AWAC 35 Proposition 3.5 highlights state-dependent (marked in blue) balance coefficients in Equations
(8) as opposed
pre-defined hyperparameter
in Equation (I) This adte-addativeenss is due to Proposition [35] considering the finer finer-grained
constraints in Definition 3.1 Together, Proposition 3.44 and Proposition 3.5 indicate state-adaptive
balance coefficients contribute to higher performance upper bound.
a(s'Is,
4 Family Offline-to-nnline RL
5 state
action
Section IattionBtthewee theoretically proves that finer- aner-grained policy constraints enhance performance upper bounds, necessitating adaptie balance co- efficients. Accordingly, we introduce FamO2O, a framework adaptively assigning balance co- efficient to each state, casily implemented over various offline-to-onling algorithms like [35] 27 hereafter called the "base algorithm".
environment
balance model
Ty (a]s, P.)
Hy (B.
Bg
state adaptive balance coefficient
universal model
Figure 3: FamO20's inference process. For each
state S, the balance model Wa computes a state- adaptive balance coefficient (3.. Based on s and Bg. the universal model n. outputs an action a.
Essentially, FamO20 trains policy family with varying balance coefficients during offline pre- training. During online fine-tuning, FamO20 identifies the appropriate policy, corresponding to the suitable balance coefficient, for each state from this policy family. this section, we present FamO20 using the AWR algorithm framework (Equation (T). FamO20's compatibility with AWR algorithms is discussed in Appendix D

---

As shown in Figure 3, FamO200s policy consists of two components: universal model Tu and balance model n. Denote the space of balance coefficients B. For every state the balance mode T6 : S + B figures out suitable balance coefficient (4 based on the state s and state-related balance coefficient &s. the universal model тu x ト A outputs an action. The balance coefficient 3x to control the conservative/radica degree of the universal model u in dealing with the state s.
4.1 Learning universal model
We initially address training the universal model Tus aimed learning policy family with varying balances between policy improvement and constraint. The formal optimization target of nu is:
R++1 arg maxB (s.a)-D
cxp(s (Q" (s. - (s))) log Tu (als,, 8.)]
(9)
Q and Vk, detailed in Section 4.3 represent Q and V functions at iteration kt Equation (9) echocs Equation (8). but the policy also takes a balance coefficient 8g as input (highlighted in blue). In the offline pre-training phase, Ag is randomly sampled from balance coefficient space B. This encourages Ta to learn varied strategies. During online fine-tuning, Ag is set by balance model TT before input to universal model uu, which prompts cooperation between Ta and TT.
4.2 Learning balance model
Next, we outline how the balance model тr chooses an appropriate policy for each state from the policy family trained by the universal model Ty- As indicated in Section 4.2] every B3 E B corresponds to unique policy. Consequently, to select the optimal policy, Tb needs to determine the appropriate balance coefficient 8g for each state s. Given this rationale, the update rule for Ty is:
balance coefticient 9
TT k++ (s, (5) k++ arg (s,) ( "b
(10)
action
Here, Th represents the updated universal model in Equation Intuitively, Equation (10) aims to find a my that maximizes Q* value by translating balance coefficients into actions with n.. x+1, This design is grounded in the understanding that the Q value serves an estimate of future return, which our ultimate goal of striking balance between policy improvement and constraint. Concerns may arise about Q* ' extrapolation error in Equation evidence suggests this is less an issue if we avoid extremely radical values in the balance coefficient space B. Following the update rule in Equation (10), Te effectively assigns balance coefficients to states, demonstrated Section 6.1
(10) potentially misguiding 's update. Empirical
4.3 Learning value functions
Furthermore, we explain the value functions update, As per Equations 9 to 10 a single set of Q and V functions evaluate both Tu and My. This is due to TT collectively forming standard RL policy Tu(: mb(-)) S + A. Hence, the value functions update mirrors that the base algorithm, simply replacing the original policy with =u(-.m()))
B and nu B + A
Finally, we offer a pseudo-code of FamO2O's training process in Appendix
Experimental Evaluation
In this section, we substantiate the efficacy of FamO20 through empirical validation. We commence by showcasing its oft performance on the D4RL benchmark [8] with IQL 24 in Sec- 5.1 We then evidence its performance improvement's 's generalizability and statistical significance FamO20's 's compatibility with Aon--WWsstyle algo- rithms like CQL [27]. yielding significant performance enhancement. Lastly, we reserve detailed FamO20 analyses for Section 6 and Appendix F For more information on implementation details,
in Section 5.2, Moreover,
(5.3) (3.3)onn533]rvveee
please refer to Appendix E.2
Despite m. and being stochastic models, notate them functions using "+>" hereafter for brevity.
5

---

IQM
Median
Optimality Gap
Mean
Ours
IQL
BR CQL AWAC
TD3+BC
SAC
BC
0.45 0.60 0.75
0.90
0.45
0.60
0.75
0.90
0.30
0.45
0.60
0.45
0.60
0.75
DARL Normalized Score (Normalized
Figure 5: Comparisons between our FamO20 against various competitors on D4RL normalized
scores 8 All methods are tested D4RL Locomotion and AntMaze for 6 random seeds. FamO20 achieves ofae-offthhe-rtt performance by a statistically significant margin among all competitors
in offline-to-online RI. (i.e. IQL [24], Balaned Replay (BR) [28] CQL [27]. AWAC [35, and
TD3+BC [10). online RL (i.e. SAC 13 D, and behavior cloning (BC).
Table 1: Enhanced performance achieved by FamO20 after online fine-tuning, We evaluate the D4RL normalized score [8 of standard base algorithms (including AWAC (35 and IQL [24], denoted as "Base") in comparison to the base algorithms augmented with FamO20 (referred to as "Ours"), "). All results are assessed across random seeds. The superior dffline-to-nnlin scores are highlighted in blue. FamO20 consistently delivers statistically significant performance enhancements across different algorithms and task sets.
AWAC 35
IQL
24
Avg.
Dataset
Base
Ours
Base
Ours
Base
Ours
hopper-mr-v2
56.0
86.8
91.0
97.6
73.5
92.2
hopper-m-v2
54.1
75.0
65.4
90.7
59.7
82.8
hopper-me-v2
97.7
92.9
76.5
87.3
87.1
90.1
halfcheetah-mr--22
43.9
49.0
53.7
53.1
48.8
51.0
halfcheetah-m--V
44.8
47.6
52.5
59.2
48.7
53.4
halfcheetah-mee-22
91.0
90.6
92.8
93.1
91.9
91.8
walker2d-mm-v2
72.8
84.4
81.5
88.6
walker2d-m-v2
79.0
80.0
83.8
85.5
81.4
82.8
walker2d-me-v2
109.3
108.5
112.6
112.7
110.9
110.6
locomotion total
648.4
714.9
718.3
772.0
683.4
743.4
95% Cls
640.5-656.8
667.3-761.4
702.5-733.5
753.5-788.5
674.6-692.0
732.1-754.2
umaze-v0
64.0
96.9
96.5
96.7
80.4
96.8
umaze-diverse-vi
60.4
90.5
37.8
70.8
66.2
80.6
medium-diverse--
0.2
22.2
92.8
93.0
45.2
57.6
medium-play-vo
0.0
34.2
91.5
93.0
45.2
63.6
large-diverse-v0
0.0
57.5
64.2
24.7
32.1
large-play-vD
0.0
0.0
52.5
60.7
21,4
30.3
antmaze total
124.7
243.7
428.7
478.3
283.1
361.1
95% Cls
116.5-132.6
226.2-259..
406.7-452.7
456.7-498.7
274.1-291.1
347.2-374.3
total
773.0
958.6
1146.9
1250.3
960.0
1104.5
95% Cls
761.5-784.6
936.8-979.6
1119.5-1175.1
1221.9-1277.0
911.3-1008.8
1063.5-1145.4
mr: medium-replay. medium, medium-expor
Datasets Our method is validated on
P(Fam020 > Base)
two D4RL g benchmarks: Locomo- tion and AntMaze. Locomotion includes diverse environment datasets collected by varying quality policies. We utilize IQI. [24] settings, assessing algorithms on hopper, halfcheetah, and walker2d en- vironment datasets, each with three qual- ity levels. AntMaze tasks involve guid- ing ant-like robot in mazes of three sizes (umaze, medium, large), each with two different goal location datasets. The evaluation environments are listed Ta- ble Is first column.
Locomation
Antmaze
Overall
0.50
Figure FamO20's improvement over base algo- rithms [35] 24]. For D4RL Locomotion, AntMaze [8].
and overall, FamO20 shows significant and meaningful
performance gains, meeting Neyman-Pearson criteria [3]

---

CQL-Famm120
(a) largerinitation weights
(b) closer to trajectories
Base
FamO20
FamO20
DARL Normellen Scores
1
normalized trajectory return
100
105-
80
80
100-
60
40
20
20
modium-reppay medium expert
<0
>0
<0 D
>0
Types
iimitation weights
distances
Figure 6: IQM scores of FamO20's
Figure 7: Comparing FamO20 with the base algo- rithm (24] on (a) imitation weights and (b) action dis- tances. A indicates the difference between FamO2O's metrics and the base's s. Generally, FamO20 emphasizes high-quality data imitation and aligns more closely with high-quality trajectories compared to the base algorithm.
implementation on top of CQL (27]. Demonstrating statistically significant superiority over vanilla CQL [27] and
CQL+BR (28). FamO20 affirms its
adaptability to non-AWR algorithms.
Metries Considering RL's inherent variability, we adopt robust evaluation methods per rliable 3 Besides conventional Medium and Mean, we integrate IQM and Optimality Gap metrics for broader assessment. We also employ rliable's probability of improvement metric for gauging the likelihood of method outperforming others. We confirm performance enhancement's s statistical significance using 95% Confidence Intervals (CIs).
5.1 Benchmark Comparison
FamO2O's state-of-the-art performance is demonstrated by implementing it over IQL. comparing with baseline methods.
24 and
Baselines We benchmark FamO20 against: (i) offline-to-online RL, including IQL [24] Balanced Replay (BR) (28), CQL [27], AWAC (35), and TD3+BC [[O]. For IQL, BR, and AWAC, official implementations are used. In the case of CQL and TD3+BC, we implement online fine-tuning based on the author-provided offline pre-training codes, following the procedures IQL and AWAC; (ii) online RL method, SAC [13]. highlight offline pre-training's efficacy; (iii) behavior cloning (BC), which is implemented by maximizing the lig-liielhood of the samples in the offline dataset. For SAC and BC, we utilize the implementations of CQL. Further details are in Appendix E.
Comparison As shown in Figure FamO20 outperforms competitors across all metrics (IQM, Medium, Optimality Gap, and Mean). Specifically, for IQM, Optimality Gap, and Mean, FamO20's 95% Cls don't overlap with the competitors'. Even for Medium, all baseline expectations fall below the lower limit of FamO2O's 95% Cls. The results underscore the significant edge of our state-adaptive policy constraint mechanism over competing methods.
5.2 Analyzing Fam020's Performance Enhancement
Though FamO20 demonstrated superior performance on the D4RL benchmark in Section 5.1 it's vital to discern the actual contributions of FamO20 from its base policy, IQL. Therefore, we address two key questions: (i) Does FamO20 consistently enhance other offline-to-nline RL algorithms? (ii) Is the performance boost by FamO20 statistically significant given RL's inherent variability?
Setup We apply FamO20 to AWAC and IQL. AWAC [35] is one of the most famous offline-to- online algorithms, and IQL 2 is recently proposed method that achieves great performance on D4RL [8) We use the authors' codes, maintaining the same hyperparameters for a fair comparison. Further details are in Appendix E.2
Comparison Table shows Tshows AWAC's and IQL's performances w/ and w/o FamO20. FamO20 gener- ally enhances performance by statistically significant margin across most datasets, regardless of the base algorithm, highlighting its versatility. Even on complex datasets where AWAC barely succeeds, egg. AntMaze medium-divenss and medium-play, FamO20 still achieves commendable performance.
Pursuing rliable's recommendation we evaluated FamO20's statistical significance by calculating average probabilities of improvement against base policies (Figure 4). In all three cases (Locomotion,
7

---

AntMaze, and Overall), the lower CI bounds exceed 50%, denoting the statistical significance of FamO2O's 's improvement. Specifically, the upper CI on Locomotion surpasses 75%, demonstrating statistical meaning as per the Neyman-Pearson criterion.
5.3 Versatility of FamO20 with Non-AWR Algorithms
To demonstrate FamO20's versatility beyond AWR- WWR-based algorithms, we extended it CQL 27 in
addition to AWAC (35) and IQL [24]. The implementation specifics are in Appendix D As Figure reveals, FamO20 significantly outperforms CQL. Even when compared to Balance Replay (BR) an offline-to-online method designed specifically for CQL, FamO20 still shows statistically significant superior performance. These results highlight FamO2O's adaptability to non-AWR algorithms.
28
6 Discussion
In this section, we further provide some in-depth studies on FamO2O, including visualization (Sec- tion 6.1) and quantitative analyses (Sections 6.2 to 6.6). More analyses deferred to Appendix F
6.1 Does FamO20 really have state-wise adaptivity?
Here, we design simple maze environ- ment to visualize the state-wise adaptivity of FamO2O. As shown in Figure 8 a). the agent starts at random cell in the top row and is en- couraged to reach the goal at the bottom right corner through two erossing points. During data collection, guidance is provided when the agent passes through the lower crossing point, but guidance for the upper crossing point. To elaborate, the guidance refers to compelling the agent to adhere to the route and direction that yields the shortest path to the goal. Without it, the agent moves ran- domly. It can be observed in Figure b(b) that the agent generally outputs lower balance efficients for the states with high-quality sam- ples (i.e.. those derived from the agent's move- ment with guidance) while generating higher balance coefficients for the states with low-quality data (i.e., data gathered when the agent moves without guidance). This result shows FamO20's state-wise adaptivity in choosing proper balance coefficients according to the data quality related the current state.
high
(a)
(b)
low
pessiblestartingggaai goal w/ guidance @ w/ guidence
Figure 8: State-wise adaptivity visualization in simple maze environment. (a) Higher data quality at the crossing point in the 5th row compared to the 2nd row. (b) Colors denote different balance coefficient values at traversed cells during inference. FamO20 typically displays conservative (or radical) behavior at cells with high-quality (or low-quality) data.
6.2
What is the effect of state-adaptive balances?
In this section, we explore the impact of state- adae-ad improvement-consteee balance. In order to encompass data of varying quality levels, we assess FamO20 using the medium-replay datasets of D4RL Locomotion g Our analysis focuses on two metrics: imitation weights and action distances.
Imitation weights are defined in Equation D. with larger (or smaller) values prompting the agent to align more closely (or less) with the replay buffer D's behavior. Action distance, delineated in Equation quantifies the discrepancy action selection between policy æ and trajectory T:
d = E (s [|larg max π (Taa) ()) al 2]
(11)
Here, lower (or higher) action distance di signifies greater (or lesser) behavioral similarity between the policy x and the trajectory т.
We evaluate IQL with and without FamO20 ('Base' and 'FamO20") regarding imitation weights and action distances, depicted Figure Figure a computes the average imitation weight difference (AIWD) per trajectory in the offline dataset. AIWD indicates the mean imitation weight difference between FamO20 and the base algorithm for each (s, a) pair within a trajectory. Figure ( likewise determines average action distance difference per offline dataset trajectory.

---

adaptive balance
fixed balance
95% Cls
B ase
FamO20
Nommliize DARL. Soore
Normalized D4RL Score
80
60
76
81
40
69
75
20
low
middle
high
middle
high
Balance Coefficient
0
-1.0
-0.5
1.0 le6
(a) A Allttion AWAWAC
(b) Ablation on IQL
training steps
Comparing adaptive and fixed balance coefficients on (a) AWAC (35 and (b) IQL (24] using D4RL Locomotion datasets (8]. Adaptive coefficients consistently outperform fixed ones with distinct 95% Cls.
Figure 9:
Figure 10: FamO20 alleviates per- formance drop due to distributional shift during the shift from offline pre- training to online fine-tuning.
Figure 7[a) reveals that FamO20 typically shows higher imitation weights than the base algorithm for high-return trajectories. Figure ( indicates that FamO20 aligns more with high-quality trajectories and less with low-quality ones than the base algorithm. These results highlight the state-adaptive balance's role in promoting emulation of high-quality behavior and avoidance of low-quality behavior.
6.3 State-adaptive balances fixed balances?
To prove that adaptive balance coefficients are better than traditional fixed balance coefficients, compare FamO20 against the base algorithms with different balance coefficients as byperparameters. As shown in Figure on both AWAC (35] and IQL 244, adaptive balance coefficients outperform all the fixed balance coefficients. Significantly, the 95% CIs of adaptive and fixed balance coefficients have no overlap. These comparison results indicate that adaptive balance coefficient approach surpasses the fixed balance coefficient method by statistically significant margin.
6.4 Does FamO20's efficacy rely on varied data qualities?
It's worth noting that our method's ef- ficacy doesn't 't rely on varied data qual- ities. Table clearly demonstrates that FamO20 surpasses the base algorithms in performance across all the medium datasets, namely hopper-medium-v2 halfcheetah--meiin--22 and walker2d- medium-v2, which all maintain rela- tively consistent data quality. We claim that FamO20 determine the suitable conservative/radicaa balance for each state in online scenarios based the data quality in the collected dataset. If the dataset is diverse in quality, the bal- ances will be diverse; if the quality is consistent, the balances will be corre- spondingly consistent. The above claim supported by Figure [T] which indi- cates that (i) in datasets with more (or less) diverse data qualities, i.e., medium- experrt (or medium), the balance coefficients are more (or less) diverse, with corresponding larger (or smaller) standard deviations; (ii) with higher (or lower) quality datasets, the balance coefficients are averagely lower (or higher), leading to a more conservative (or radical) policy.
medium
5.0
medium_exper _eppert
Balance Coefficient
4.5
4.0
3.5
2.5
2.0
1.5
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
Step
1e6
Figure 11: Balance coefficients' mean and std for IQL w/ FamO20 D4RL HalfCheetah.
6.5 Does FamO20 mitigate the performance drop stemming from the distributional shift?
Despite FamO20's primary objective not being direct distributional shift handling, state-adaptive improvement-connstrin balances prove beneficial in mitigating performance degradation during the

---

offline pre-training to online fine-tuning transition, attributed to the distributional shift. Existing offline-to-onling RL algorithms [24 35 already incorporate mechanisms to counter distributional shift, hence significant performance drops are infrequent. Figure TQ illustrates the training curves of IQL and IQL++amO20 across both offlin pre-training (negative steps) and online fine-tuning (positive steps) on antmaze-uaazeddieee performance when transitioning from offline pre-training to online fine-tuning, As evidenced, while IQL+FamO20 initially experiences performance drop akin to IQL during online fine-tuning, it recovers rapidly and attains high performance, in stark contrast to IQL's sustained performance decline throughout the fine-tuning stage.
on which IQL. exhibits the most significant decline in
6.6 Balance model vs. random selector?
Table 2: Ablation study on the balance model. The absence of the balance model results in de- creased performance of FamO20 D4RL Locomotion datasets g The model's im- pact is statistically significant and meaningful as per the Neyman-Person criterion.
To validate the effect of the balance model TT& in choosing balance coefficients, we present a FamO20 variant, denoted as candom-Fam02) with the balance model replaced with random balance coefficient selector. Other training settings keep the same for FamO20 and random-FamO20. Ta- ble | shows the improvement percentages and probability of improvement of FamO20 against
nearly all
Dataset
FamO20
Candom-FamO22:
hopper-mr-v2
97.64
80.87
hopper-m-v2
90.65
86.37
hopper-me-v2
87.28
77.08
As we can observe, al-
halfcheetah-mr-v2
53.07
53.75
FamO20 outperforms random-Famm22 most all the datasets of D4RL Locomotion [8] Furthermore, the lower CI of the probability of improvement is much higher than 50%, and the upper CI exceeds 75%. This indicates that the effect of the balance model is not only sta- tistically significant but also statistically mean- ingful as per the Neyman-Person statistical testing criterion.
halfcheetah-m--2
59.15
$3.15
balfcheetah-mm-v2
93.10
92.72
walker2d-mr-v2
92.85
91.38
walker2d-m-v2
85.50
84.84
walker2d-me-v2
112.72
110.54
total
771.96
730.70
prob. improvement
0.70 (95% Cls: 0.59~-0.79)
7 Conclusion
This work underscores the significance of state-adaptive improvement-conssrainn balances in offline-to- online RL. We establish a theoretical framework demonstrating the advantages of these mate-adaptive balance coefficients for enhancing policy performance. Leveraging this analysis, we put forth Family Eefline-to-Oningg RL (FamO20), versatile framework that equips existing algorithms with the ability discern appropriate balance coefficients for each state.
oonline RL
Our experimental results, garnered from a variety of offline-to-online RL algorithms, offer substantial evidence of FamO2O's ability to significantly improve performance, attaining leading scores on the D4RL henchmark. In addition, we shed light on FamO20's adaptive computation of state-adaptive wewpovemmen--oonttriiie balances and their consequential effects through comprehensive analyses. Ablation studies on the adaptive balances and balance model further corroborate FamO20's ' efficacy,
The limitation of our work is that FamO20 has been evaluated on just a handful of representative offline-to- to-online RL algorithms, leaving a vast array of such algorithms unexamined. Additionally, FamO20's O'' utility is somewhat limited, as is applicable exclusively offline-to-onling algorithms. In future work, we aim to expand the applicability of FamO20 by integrating it with a broader spectrum of -nfiine-t--onliee algorithms. Additionally, intend to investigate the feasibility of incorporating state-adaptive improvement-constrain balances in offline RI. settings, where online feedback is intrinsically absent, or even applying our adaptive design to the M-a-M-as-aeen domain 49 50
Acknowledgement
This work was supported in part by the Key-Area Research and Development Program of Guangdong Province under Grant 2020B11 11881111500002. the National Natural Science Foundation of China under Grants 62022048 and 62276150, and the Guoqiang Institute of Tsinghua University.
10

---

References
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint urXiv:1803.08375
2018.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
[3] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
[4] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang. Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances Neural Information Processing Systems, 2020.
[5] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 1992.
[6] Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Artificial Intelligence Research, 2000.
[7] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 060000 2018.
[8] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[9] Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning, 2019.
[10] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement Icarning. In Advances in Neural Information Processing Systems, 2021.
[11] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, 2019.
[12] Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. International Conference on Machine Learning. 2022.
[13] Tuomas Haarojj,, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 2018
[14] Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation. In International Conference on Machine Learning, 2017.
[15] Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song. Junfeng Cao. Wenhui Huang, Chao Deng, and Gao Huang, Learning weight samples for dynamic early-exiting networks. European Conference on Computer Vision, 2022.
[16] Yizeng Han, Zhihang Yuan, Yifan Pu, Chenhao Xue, Shiji Song, Guangyu Sun, and Gao Huang. Latency- aware spatial-wist dynamic networks. In Advances in Neural Information Processing Systems, 2022.
[17] Bernhard Hengst. Discovering hierarchy in reinforcement learning with hexq. In International Conference Machine Learning, 2002.
[18] Jocy Hong, Aviral Kumar, and Sergey Levine. Confidence-condittonee value functions for offline rein- forcement learning. In International Conference on Learning Representations, 2023.
[19] Nan Jiang and Lihong Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, 2016.
[20] William Karush. Minima of Functions of Several Variables with Inequalities as Side Conditions. Springer Basel, 2014.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.

---

[22] George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in nearal information processing xystems, 2009.
[23] Ilya Kostrikov, Rob Fergus. Jonathan Tompson, and Ofir Nachum. Offline reinforeement learning with fisher divergence critic regularization. In International Conference on Machine Learning, 2021.
[24] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. In International Conference on Learning Representations, 2022.
[25] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
[26] Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback reinforcement learning via distribution correction, In Advonces in Neural Information Processing Systems, 2020.
[27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
[28] Seunghyun Lee, Younggyo Sco, Kimin Lee, Picter Abbecl, and Jinwoo Shin. Offline-to-onling rein- forcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, 2022.
[29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint (iti::20000016443 2020.
[30] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q q-learnin for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[31] Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS, 2022.
[32] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness. Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-555 2015.
[33] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algacdice: Policy gradient from arbitrary experience. arXiv preprint arXiv: 1912,0007, 2019.
[34] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hicrarchical reinforce- ment learning. Advances in neural information processing systems, 2018.
[35] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.094 09359, 2020.
[36] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-We Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 2020.
[37] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-wcighted regression: Simple and scalable off policy reinforeement learning. arXiv preprint 1rXiv:1910.00177,, 2019.
[38] Yifan Pu, Yizeng Han, Yulin Wang, Junlan Feng, Chao Deng, and Gao Huang. Fine-grained recognition with learnable semantic data augmentation. arXiv preprint urXiv:2309.0033
2023.
[39] Yifan Pu, Weicong Liang. Yiduo Hao, Yuhui Yuan, Yukang Yang. Chao Zhang, Han Hu, and Gao Huang. Renk-detr for high quality object detection. In Advances in Neural Information Processing Systems, 2023.
[40] Yifan Pu, Yiru Wang, Zhuofan Xia, Yizeng Han, Yulin Wang, Weihao Gan, Zidong Wang. Shiji Song, and Gao Huang. Adaptive rotated convolution for rotated object detection. In International Conference on Computer Vision 2023.
[41] Rembert Reemtsen. Discretization methods for the solution of semi-infinite programming problems. Journal of Optimization Theory and Applications, 1991.
[42] Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin A. Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[43] Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning. MIT press Cambridge. 1998.
12

---

[44] Richard S. Sutton, Doina Precup. and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.
[45]
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. User-interactive offline reinforcement learning. In The Eleventh International Conference on Learning Representations. 2023.
[46] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
[48] Alexander Sasha Vezhnevets. Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver. and Koray Kavukcuoglu. Feudal networks for hicrarchical reinforcement learning. In International Conference on Machine Learning, 2017.
[49] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Chen, Yankai Lin, et al. A survey on large language model hased autonomous agents. arXiv preprint arXiv:2.308. / 1432, 2023.
[50] Shenzhi Wang. Chang Liu, Zilong Zheng, Siyuan Qi. Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang. Shiji Song, and Gao Huang. Avalon's s game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320. 2023.
[51] Shenzhi Wang. Liwei Wu, Lei Cui, and Yujun Shen. Glancing the patch: Anomaly localization with global and local fcature comparison. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[52] Ziyu Wang. Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenherg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. In Advances in Neural Information Processing Systems, 2020.
[53] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforeement learning. In Advances in Neural Information Processing Systems, 2022.
[54] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: I. 1361, 2019.
[55] Qisen Yang. Shenzhi Wang. Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforce- ment learning with action preference query. arXiv preprint arXiv:2306.033: 2023.
[56] Qisen Yang, Shenzhi Wang. Qihang Zhang, Gao Huang, and Shiji Song. Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance. arXiv preprint arXiv:2309.014 2023.
[57] Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In International Conference on Learning Representations, 2019.
[58] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. preparing for the unknown: Learning a universal policy with online system identification. In Robotics: Science and Systems, 2017.
[59] Zishun Yu and Xinhua Zhang, Actor-critic aligmment for oflline-tooollne reinforcement learning. In International Conference on Muchine Learning, 2023.
[60] Yang Yue, Bingyi Kang, Xiao Ma, Gao Huang. Shiji Song, and Shuicheng Yan. Offline prioritized experience replay. arXiv preprint 20l.:2306.05412.. 2023.
[61] Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint 2210022210.09241, 2022.
[62] Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Value-consistent representation learning for data-efficient reinforcement learning. In AAAI, 2023.
[63] Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging ofline-to-online reinforvement learning. In International Conference on Learning Representations (ICLR), 2023.
[64] Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference Learning Representations, 2021.
13

---

[65] Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09077 2020.
[66] Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to on-one reinforcement learning. arXiv preprint arXiv:2210.13846, 2022.
14

---

Appendices
Contents
Related Work
15
Training Process of FamO20
16
Theoretical Proofs
17
Proof of Proposition 3.4
17
Proof of Proposition 3.5
18
D FamO2O's Extension to Non-AWR Algorithms
19
Implementation Details
Baseline Implementation
20
Implementation of FamO20
More Experimental Results
21
Sensitivity Analysis on Difterent Ranges of Balance Coefficients
21
F2
Comparing FamO20 and Balance Coefficient Annealing
22
Potential Negative Societal Impacts
22
A
Related Work
Offiine RL
Reinforcement learning (RL) usually improves policy through continuous online
interactions with the environment 44 32 62 To reduce the huge demand for online interactions,
especially when they costly risky, researchers proposed offline RL that utilizes pre-collected
datasets to improve the policy without interacting with the environment 29 Directly applying off-policy algorithms in offline RL usually leads poor performance. This phenomenon due to distributional shift, where the agent may learn inaccurate value estimates of out-of- distriibitinnno (OOD)
state-action pairs (2 61 60). Existing algorithms address this issue via policy constraints || 25
54 23 37 35), importance sampling [19 14 65
regularization
33 27
uncertainty estimation
and imitation learning
52).
Offline-to-nllne RL Offline-to-online RL adds online fine-tuning to offline RL enhance policy performance. Similar to the distributional shift problem offline RL, offline-to-onlinn (020) RL also suffers from off-policy bootstrapping error accumulation caused by OOD data, which causes a large "dip" in the initial performance of online fine-tuning [35]. Moreover, 020 RL algorithms tend be excessively conservative and result in plateauing performance improvement 35 This emphasizes the importance of finding balance between being optimistic about improving the policy during the online phase and still being constrained to the conservative offline policy. However, as previously discussed, existing O20 RL algorithms generally adopt sine-size-fits-alll trade-off, either in the process of Q-learning (27 30 24 3T) along with the policy improvement [35], via
revised replay buffer [28]. or through some alignment approaches [53] [9]. It's worth noting that, (66) also proposes an adaptive weight online fine-tuning. However, the adaptive weight single parameter for all the samples. In addition, its update requires an extra human-defined target score, which absent in the setting considered by our competitors and us. Furthermore, 633 presents policy expansion strategy that adaptively selects a policy from a set, which includes both offline pre-trained policies and online learned policies. Nevertheless, this policy set comprises only two policies, in contrast to the infinite number of policies contained in FamO20's policy family.
15

---

Conditioned RL policy Algorithms in this scheme can be divided into two stages: (i) set of policies is trained; (ii) the policy with the highest performance is selected. Generally, these methods learn family of policies that exhibit different behaviors and search for the best one that successfully executes the desired task [6 (57. A similar method our FamO20 UP-OSI 58 where Universal Policy (UP) within parameterized space of dynamic models and an Online System Identification (OSI) function that predicts dynamic model parameters based on recent state and action history are proposed. Unlike these prior methods utilizing models to identify the environmental parameters, the balance model in our method aims to determine proper balance between policy improvement and constraints for each state. It's worth noting that [88 45 12 also exhibit conditioned designs. However, [8] focuses on conditioned value functions and pure offline RL; 45 relies on
human interactions; [22 concentrates
adjusting offline RL policies according to the estimated probabilities of various MDPs in uncovered areas, but uncoverage might not be primary concern in offline-to-onling RI. due to the accessibility to the online environment. Moreover, [56 55 both
exhibit a comparable understanding of how to adjust policy improvement and policy constraint across samples of varying quality. Nevertheless, the focus of [56] (55) is on offline RL. Specifically, [56]
requires a small number of expert demonstrations. while (55) necessitates annotated preferences
to appropriately calibrate the balance for each individual sample. Additionally, in areas such as computer vision, there are existing works that condition their training and/or inference on various These studies align with the adaptive approach our work.
input samples [5 16
38 39 51.
Hierarchical RL Hierarchical Reinforcement Learning (HRL) decomposes complex tasks into manageable subtasks, enhancing sample efficiency, interpretability. and generalization in RL. HRL approaches bifurcate into predefined and learned options (or skills). Predefined options, guided by domain knowledge or human demonstrations, shape high-level policy or constrain action space, exemplified by MAXQ (6) options framework (44), skill chaining (22), and feudal RL (5). Learned options, optimizing objectives like entropy or reward, are seen in option-critic, skill-discovering (22), feudal networks [48). HEXQ [77, DIAYN 77,, HIRO (34,, and HIDIO [64]. While FamO20 shares some design elements with HRL, such as hierarchical policy decision- making, the objectives diverge: FamO2O focuses on balancing policy improvement and constraints per state, while HRL seeks to simplify complex tasks via goals or options.
B Training Process of FamO20
Here we provide the pseudo-codes of FamO20 (see Algorithm T to demonstrate its training process.
Algorithm 1 FamO20 Algorithm
Require: replay buffer D. offline dataset Dottines offline and online training steps Nott Non 0 1: Initialize Th. = Q": 170
2: Initialize with Domine
do 4: Sample a mini-batch M₁ from D
3: for 0= Non
D offline pre-training phase
5: Assign a random balance coefficient for each sample in Mp. denoting the balance coefficient
set as BM,
6: Update b n. with Mg, BM, by Equation
7: Update = to Te s+1 with Mj by Equation (10)
8: Update Q* Vk to Qi+1, yk+1 respectively with Mf according to the base algorithm
9:
end for
10: for k == Noft + Non - do
online fine-tuning phase
11:
Collect samples with ~f m and add samples to D Sample a mini-batch M, from D
12:
13:
o computes balance coefficient for each sample in Mg. Denote the balance coefficient set Ba
as
14:
Update k to n *+1 with MR, BM, by Equation
15:
Update v to n, 6+1 with Mg by Equation 10)
16:
Update Q V to Q++1, yh+1 respectively with Mf according to the base algorithm
17:
end for
16

---

C Theoretical Proofs
C.1 Proof of Proposition 3.4
Lemma C.I. Equation 5 is equivalent to:
[ VsES, (RI(((((S][TT(()))) S cx. / SES
(12)
= (6.8
where C=
dnp (s)e = e, 2 0
(13)
Proof. We aim to prove Equation (5) Equations 12 to 13 in our manuscript through a contra-
diction approach. Let's assume that Equation * Equations 12)
(3, signifying the following:
1. Condition
According to Equation 5 Jaes d= Dr. (T( mm (-Is))ds S;
2. Condition 2
The converse proposition of Equations 12 to 13
is: Ve, 2 ( E S)
which satisfy
faes drp (s)c,ds there exists
S, D
a (-(s)) > cs.
Next, we form special {Es, 8). Selecting an arbitrary so S, the fulfills::
1. For all s + 80, 6x DRL (-1ss (s).
2. Exg = -S drp DRI (T so (s))
can be clearly seen that the set adheres to saes drs (s)edds = e. Based on Condition 2, we can deduce that (xL(T(-|so)]
> 6 Hence, [ dx (s)c,ds
==
d cs0 + J (s)
dr (80) (T [s)) + JES dr (1)6)))))
d (T) (ss
SES
This stands in contradiction to condition 1, where SES ( (a) (x ( S <. Thus, the assumption that Equation (5) /> Equations
(12) (13)
proven false, which confirms that
Equation Equations
112 to
Denote the feasible region of the problem Definition 3.2 as F+((), and the feasible region of the problem in Definition 3.1 as F& (fcz. E S}). According to Lemma C.1 we can infer that:
F*(())
U
(1e,,
(14)
(s,,ES)ec
Considering that the problems in Definition 3.1 and Definition 3.2 shares the same objective function, we have:
2 0, 3(6, E S) C, *
[ F ((es, s E (S),
(15)
where , is the optimal solution corresponding to the optimal value [ for the problem in
Definition 3.2
Based on Equation
15). we can derive that
Ve 2 0,
3f6. 8}&C. C, 3r e F S}), Hille SH J. (16)
17

---

Here 1,,, S SH is the objective function value in 3.1 with the solution and KL constraints {Gx, S).
Under the KL constraints ( 8), the optimal value s S}] of the problem in Defini-
tion 3.1 is no less than c E SH. Therefore
H(,, E S}: E I ES] J
(17)
Proposition
3.4 is proven. Q.E.D.
C.2 Proof of Proposition 3.5
Because the state space S possibly contains an infinite number of states, the optimization problem in Definition 3.1 is probably problem with infinite constraints, which is not easy to deal with. Therefore, we first start with a simplified version of the optimization problem, where the state space contains only a finite number of states, and further extend the conclusion on the simplified optimization problem the original one Definition 3.1
The simplified optimization problem stated as follows:
Definition C.2 (Simplified optimization problem with point-wise constraints). The simplified opti- mization problem with point-wise KL constraints on a state space with only finite number of states (Si, S2, "*, Sm is defined
max Bender ()a()))))))) [Qr (s, V= (s)
(18)
s.t. DRL (8:)][T[(((ss))) S 62 i= 1,2,"...
(19)
[ da i == 1,2, m.
(20)
Here, (k E N) denotes the policy at the k-th iteration, Ty behavior policy representing the action selection way in the offline dataset or current replay buffer, and dæg (s) is the state distribution of Ty. utilize the optimal solution for Definition C.2 as x*+1
For this simplified optimization problem, we have lemma below, whose derivation is related to AWR (37) and AWAC (35):
Lemma C.3. Consider the optimization problem in Definition C.2
The optimal solution of k+1
denoted as nk+1, ssatisfies that V/s E {81,82, 8m), E A.
(als)
(als)
exp (s,
- - (s)))
(21)
Ze
Proof. The Lagrangian function L(T, A. pi) is given by:
L(T, p. = Bandey (a)a)(())))) (s. v** (s)
m
m
(22)
(alss) da -1) Σ Mh (T
ei):
where (A1, A2,....
Am), 14 (M), u2, ... (m)) are Lagrange multipliers. According to
KKT necessary conditions (20] the optimal solution satisfies that Va E A, E 21, ..................... ,
ØL.
dry (8:) (Q" (st, v-" v=*(ss) (s:)) + de + u4
log
(als:) T Ta(a)))))))))))))))){{)))
+ 0. (23)
(als))
Therefore, the optimal policy xt+ 1 is:
TT (als:) exp (Bx, (2,(0 (st, y (si)))
π k+ ++1 (a(ss))
(24)
Zei
mi + +1) , (x))
where Zo, exp Furthermore,
ue are state-dependent factors.
due to the constraint in Equation SaE (a)s)) exp (Be, (Q (8, a) V** (s:)) da.
20 Za.
also
equals
to
18

---

Now we consider extending the conclusion in Lemma C33
to the more complex optimization problem is a compact set and contains an infinite
in Definition 3.1 where the state space S miin: Smax
number of states. Without loss of generality. suppose that S = [0, 1.. The derivation below is easy to
transfer to S = [sminn smax] 1 with small modifications.
Specifically, we construct a sequence sets {B; i 0, 2..,... with each element B; (j/2',j 0, 1, } We can observe that {B, i = 0, 2. ... satisfies:
J
WE W-N. B S;
(25)
MEN..
B; - B
(26)
VL.N.
[B,] 2' +1< 00, i.e., all B, are finite sets, and therefore all B; are compact;
(27)
lim sup inf || y|1.0 lim 1-000 ZES VEB;
(28)
2-000 2
The qualities in Equations 25 to 28
together with the assumption in Proposition 3.5 that the
feasible space constrained by Equations (2
(4) is not empty for every s E S, meets the prerequisites
of the discretization method proposed by 41 Set n m [A] as the optimal solution of the following optimization problem:
I [A] =arg maxli (111)) (Q (s, V (s)
(29)
DKL (I ( (-18) cs. SEA
(30)
/ T(als) da SEA,
(31)
aEA
where A is a subset of S. According to the Theorem 2.1 in
[B]]
k+1 [S].
(32)
For any B,, because [B,| < oc, Equation 21 holds for 7. [B,]. By combining Equation 21 with
Equation we succeed in proving Equation 7
Proposition 3.5
Furthermore, we utilize a parameterized policy T to approximate the optimal policy x k+1 in Equation 77. i.e.
Φ arg min Eg ( [DKI. (= (1))
(33)
1 E Ba-ss(als)
arg maxl Bndd !
(x (s, - (s)) (aga))))))
(34)
Za
In practice, Ze in Equation 34 is challenging to calculate. We follow the derivation of AWR
37
and AWAC (35), and omit the term z Therefore Equation (34 can rewritten as
arg max
Badd (.) (awas(a)a) (exp(3((Q (s v= (s))) log (alss)
(35)
arg max (8.a)~D
(xxp(3((Q**) (s, - y* (s)) log (a
(36)
Thus Equation 8 in Proposition
3.5 is also proven. Q.E.D.
D FamO20's Extension to Non-AWR Algorithms
In this section, we discuss extending FamO20 to non-AWR algorithms, specifically considering the Conservative Q-Learning (CQL) (27).
CQL's policy update rule given Equation
k+1 arg maxE, D [a (5, a) log T (als))
(37)
19

---

where π and Q" represent the policy and Q function at iteration respectively, and a is hyper- paramter. CQL already incorporates conservative estimation for out-of-distribution data in Q function update rule, thus lacking policy constraint for policy updates in Equation (37). However, Equation (37) presents a balance between exploitation (the blue part) and exploration (the red part), determined by a. We thus deploy FamO20 to adjust this balance adaptively per state.
Under FamO2O, the policy update rule morphs Equation (38
π arg maxlggD a.w. [as Q* a) log n. (a|s, as)]
(38)
where
ag = a==,((s)
(39)
where Ta is the universal model taking a state and balance coefficient as input to yield an action, and Ty is the balance model mapping state to balance coefficient. The changes, denoted in red and blue, depict FamO2O's additional input and the substitution of the adaptive balance coefficient Og for the pre-defined one a, respectively.
Additionally, the balance model update rule (Equation (40) value by finding optimal ag, akin to Equation 10
aims maximize the corresponding Q
balance coefficient 8,
arg maxl Th
Q*(s,
T k+1 (s. my (s(
(40)
actiion
As for function updates, by consolidating balance model n. : S + B and T.. SSx B + A into standard policy (: mx(-)) S + A. these updates remain identical to standard CQL.
E
Implementation Details
The following section outlines the specifics of baseline (Appendix E.1 and FamO20 (Appendix E.2 implementations.
E.1 Baseline Implementation
We commence by detailing our baseline implementation. For IQL (24), AWAC [35. and Balanced Replay (28). which have been tested in an offline-to-onlin RL setting in their respective papers, we
utilize the official codes
For CQL (27 and TD3+BC [10], introduce an online fine-tuning process based on the recom- mended offline RL codes by the authors
f Specifically, our online fine-tuning for CQL and TD3+BC mirrors the implementation in rikit,, that is, appending samples acquired during online interactions to the offline dataset while maintaining the same training objective during offline pre- pre-training and online fine-tuning.
For SAC [13 and BC, we leverage CQL's codebase, which has already incorporated these two algorithms.
Regarding training steps, we adhere IQL's experiment setting for fair comparison across all methods in Figure 5 which involves 10th gradient steps each for offline pre-training and online fine-tuning. should be noted that due to AWAC's overfitting issue, as discussed by (24), we limit AWAC's offline pre-training to 2.5 x 10 gradient steps, as recommended. For non-AWR algorithms comparison, namely CQL 227, BR 28 , and CQL+FamO20, we employ 2 x 10t offline gradient steps, suggested in CQL's implementation, and 1 x 10 online gradient steps, aligning with IQL.
c81 e11.9iiihhhi22220egeeereaaaaaass httpss//giihuub.cmmraallbbbbe
'IQL and AWAC:
commit
ID
'Balanced httppp////ggghhhi.oooo///aaaaa/a commit 110.1111611110000000000022222222
commit ID 6f298fa
STD3+BC http: httpe/tthh.ccccf.u..eea.aaaeaeee www.scowww/m./....... commit ID 8791 3771a777777555533311131311771000
20

---

E.2
Implementation of FamO20
Next, we provide FamO20's ' implementation details. We implement our FamO20 on the official codes of AWAC [35], IQL (24), and CQL [27] discussed above. Our major modification is (i) adding balance model and (ii) making the original policy conditioned on the balance coefficient (called "universal model" in our paper). Except for these two modifications, we do not change any training
procedures and hyperparameters in the original codes.
For the first modification, we employ the stochastic model used by SAC in rlkit to construct our balance model and implement the training process of the balance model described in Section 4.2 For the second modification, we change the input dimension of the original policy from the state dimension / 1 + lb, where la is the dimension of balance coefficient encodings. We encode balance coefficients in the same way as the positional encoding proposed by [47]. This encoding design expands the dimension of the balance coefficients and avoids the balance coefficients being neglected by the universal model.
For IQL with FamO20, adaptive balance coefficients (AntMaze) and no larger than 5.0 (Locomotion) 14.0 (AntMaze). For AWAC with FamO20, the adaptive balance coefficients 3.0 (Locomotion) / 11.0 (AntMaze). For CQL with FamO2O, the adaptive balance coefficients are no less than 0.5 and no larger than 1.5, To ensure fair comparison, in Figure 9 the fixed balance coefficient ranges cover the adaptive ranges discussed above.
less than 1.0 (Locomotion) 8.0
no less than 2.0 (Locomotion) / 9.0 (AntMaze) and no larger than
As for training steps, for IQL with FamO2O, phase and 10th gradient steps in the online fine-tuning phase. For AWAC with FamO20, we use 7.5 x 10 gradient steps in the offline pre-training phase and 2 x 10 (Locomotion) / x 10 (AntMaze) gradient steps in the online fine-tuning phase. For CQL with FamO2O, we use 2 x 106 gradient steps in the offline pre-training phase and 10 in the online fine-tuning phase.
10t gradient steps in the offline pre-training
For more implementation details, please refer to Tables to 6
F
More Experimental Results
F. Sensitivity Analysis on Different Ranges of Balance Coefficients
To find out whether FamO20's performance is sensitive the range of balance coefficients, implement FamO20 on IQI. (24 with 3 different balance coefficient ranges (i.e., [1,5). [1,4], and [2, 42) on datasets of different qualities (i.e., medium-replay, medium, and medium-cxpert), The results can be found in Figure I2 We further implement FamO20 on IQL with 4 different balance coefficient ranges (i.e., [6, 12). 18, 12). [8, 14], and [9, 11)) on 3 datasets of different maze sizes (i.e., umaze, medium, large). The results can be viewed in Figure [3] It can be observed that on various datasets of different qualities or maze sizes, the performance of FamO20 does not vary significantly with different ranges of the balance coefficients. This indicates that within reasonable scope, FamO20 is insensitive to the range of coefficient balances.
medium-replas replay
medium
medlum-cxper
1.0
50-
normllzed score
pormalized score
80
40-
40
20
30
40
20
10
[1. 4)
4]
0
100
200
300
400
100
200
300
400
100
200
300
400
training epochs
training opachs
training opochs
Figure 12: Sensitivity analysis different ranges of balance coefficients. We choose 3 different ranges (i.e., [1, 5). [1,4], and [2, 4)) and 3 different dataset qualities (i.e., medium-replay, medium, and medium-expert) in the D4RL 8 HalfCheetah environment. Results are evaluated over 6 random seeds. The shade around curves denotes 95% CIs of the policy performance. Each epoch contains 5k gradient steps. The dashed lines divide the pre-training and fine-tuning.
21

---

umaze
medium
large
Pormlzzss score
namalized scora
normatzed score
14] [8. 12]
[9, 131
400
200 300
100
200
300
training epachs
traliving epochs
training epochs
Figure 13: Sensitivity analysis on different ranges of balance coefficients. We choose 4 different ranges (i.e., [6, 12]. 12). 88 14], and [9, 11]) and different maze sizes (i.e., umaze, medium, and large) in the D4RL 8 AntMaze play environments. Results are evaluated over random seeds. The shade around curves denotes 95% CIs of the policy performance. Each epoch contains 5k gradient steps. The dashed lines divide the pre- vre-training and fine-tuning.
F.2 Comparing FamO20 and Balance Coefficient Annealing
To delve deeper into the efficacy of FamO20's balance model more thoroughly, we set up a comparison against the balance coefficient annealing. The balance coef- ficient annealing is a process that gradu- ally augments the balance coefficient value as the fine-tune teeettnnng stage proceeds. This method is intuitively reasonable, as the ini- tial conservativeness due to the balance co- efficient annealing helps curb extrapola- tion errors at the commencement of fine- tuning, and the process continues, this conservativeness gradually eased to fa- cilitate policy improvement.
Table 3: Comparing FamO20 and balance coefficient annealing, FamO20 notably surpasses the latter the majority of D4RL locomotion datasets (8) demonstrat- ing statistically significant performance improvement.
Dataset
FamO20
Annealing
hopper-mr-v2
97.64
87.82
hopper-m-v2
90.65
72.98
hopper-me-v2
87.28
72.26
halfchectahhmrr- -2
53.07
53.35
halfcheetah-m-v22
59.15
57.37
halfcheetah-me-v2
93.10
93.91
incorporate balance coefficient anneal- ing into IQL (24,, named IQL+annealing, providing a basis for comparison with IQL+FamO20. The comparison results are presented Table 3 The results show that FamO20 exceeds the performance of bal- ance coefficient annealing across most the D4RL Locomotion datasets (8].
walker2d-mr-v2
92.85
87.29
walker2d-m-v2
85,50
85.89
walker2d-me-v2
112.72
112.72
total
771.96
723.61
95% Cls
753.52-788.51
704.21-744.43
In addition, we calculate the 95% CIs for both IQL+FamO20 and IQL+annealing in Table 3 following the suggestion by rliable [3. The lower CI for IQL+FamO20 is mush higher than the upper for IQL+annealing, signifying a statistically significant improvement of FamO20 over balance coefficient annealing. These findings establish that in comparison to annealing, FamO20 showcases a more sophisticated and effective approach in adaptively modulating the balance between improvement and constraint.
G Potential Negative Societal Impacts
This work proposes a plug-in framework, named FamO2O, which can be implemented on top of the existing offline-to-nnine RL algorithms further improve policy performance. Therefore, the potential negative societal impacts of our method are similar to those of the offline-to-online RL. Generally, FamO20 would greatly improve policy performance. However, as with most RL algorithms, FamO20 cannot guarantee to take safe and effective actions in all kinds of scenarios. We advocate that the users FamO20 should be aware of the potential consequences and utilize FamO20 safely, especially in online environments such as self-driving, robotics, and healthcare.
22

---

Table Details of FamO20 implemented IQL [24] for the D4RL Locomotion and AntMaze
benchmark 8 Except for our newly added balance model, the hyperparameters of our method keep the those of the vanilla IQL..
Name
Value
optimizer
Adam [2]
learning rate
3 x 10 44
Balance model
update frequency
5 (Locomotion) / (AntMaze)
hidden units
[256, 256
activation
ReLU
optimizer
Adam
learning rate
3 x 10-4
Universal model
update frequency
hidden units activation
256, 256
ReLU
optimizer
Adam [2]
learning rate
3 x 10-4
update frequency
Q function model
hidden units
[256, 256
activation
ReLU
target Q soft update rate
5 x 10-3
optimizer
Adam 21
learning rate
3 x 10 -4
update frequency
function model
hidden units
[256, 256]
activation
ReLU [I]
target V soft update rate
5 x 10
quantile
0.7 (Locomotion) 0.9 (AntMaze)
batch size replay buffer size discount
256
Other training parameters
2 x 106
0.99
23

---

Table 5: Details of FamO20 implemented on AWAC [35] for the D4RL Locomotion and AntMaze
benchmark 88. Except for our newly added balance model, the hyperparameters of our method keep the same as those of the vanilla AWAC.
Name
Value
optimizer
Adam PI
learning rate update frequency
x 10-1
Balance model
hidden units
[256. 256]
activation
ReLU [
optimizer
Adam [21] 3 x 10
learning rate
update frequency
Universal model
hidden units activation
[256, 256, 256, 256] ReLU [1]
weight decay
10
optimizer
Adam 21 10
learning rate
update frequency
Q function model
hidden units
256.. 256]
activation
ReLU [[]
target Q soft update rate
x 10
batch size
1024
Other training parameters
replay buffer size discount
10
0.99
Table Details of FamO20 implemented on CQL 27 for the D4RL Locomotion benchmark g Except for newly added balance model, the hyperparameters of our method keep the same those of the vanilla AWAC.
Name
Value
optimizer
Adam 21
learning rate update frequency
3 x 10-4 -4
Balance model
hidden units activation
[256, 256] ReLU [I]
optimizer
Adam 21
learning rate
3 x 10-4
update frequency
Universal model
hidden units activation
2556, 256 ReLU
optimizer
Adam 21
learning rate
3 x 10
update frequency
Q function model
hidden units
[256, 256
activation
ReLUU [I
target Q soft update rate
5 x 10-3
batch size
256
Other training parameters
replay buffer size
2 x 106
discount
0.99
24


Reviews:
[{"review_id": "zyz1KBH89E", "reviewer": "Reviewer_pp8g", "summary": "The paper introduces a new method to mitigate the distribution shift problem in the offline to online RL problem. The paper states the intuition that the policy should behave differently on states with different values, that is, the policy should be more conservative on high return states and exploratory on the low return states. With this intuition, the paper proposed to train a family of policies in the offline to online setting, specifically, train another \"policy\" to parameterize the rollout policy via the state-adaptive balance coefficient. The experiments show that FamO2O outperforms previous O2O baselines, and show via a toy experiment that FamO2O indeed leans the state-wise adaptivity, and various ablations show the importance of each design choices. ", "questions": "See above. ", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experiment result is solid as it evaluates on extensive D4RL dataset with different data quality and both locomotion and maze tasks. \n\n2. The discussion sections validate the algorithm design choices, and section 6.1 verify that the algorithm indeed learns a state-wise adaptive policy, which support the intuition and motivation of the algorithm.", "weaknesses": "1. The exploitation vs. exploration intuition is not brand new in the offline to online setting, there is also some work with similar intuition [1]. I believe proper comparison is required given the similarity of the intuition, although I believe training a family of policies (or conditionally parametrized the policy) seems like a slight generalization.\n\n2. It is confusing that in the universal model training (eq. (9)), the exponential of the advantages are weighted by the balance parameter $\\beta$, but when training the balance model, the Q-function (as in the loss) is not weighted by the balance parameter. There seems to be some consistency. I can tell that the unweighted objective (the current form of eq (10)) would be more computationally friendly, but theoretically, it seems more natural to optimize over a weighted version where eq. 10 is also weighted by $\\beta, which sample from the balance model.\n\n3. The action distance in eq. (11) may not be the best metric to measure the discrepancy between a policy and a trajectory. For example, if the policy that induces the trajectory only takes $a_1$ in $s$, and the evaluated policy takes $a_1$ with $p=0.51$ and $a_2$ with $p=0.49$, which may induce a very negative reward, or cause great trajectory derailment (which is not recorded in the offline trajectory), the proposed metric will still be 0 but in reality the evaluated policy is not that close to the trajectory. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions, which have helped us improve our manuscript.\n\n> Q1: The exploitation vs. exploration intuition is not brand new in the offline to online setting, there is also some work with similar intuition [1]. I believe proper comparison is required given the similarity of the intuition, although I believe training a family of policies (or conditionally parametrized the policy) seems like a slight generalization.\n\nThank you for bringing up the important issue of exploitation vs. exploration intuition and the comparison to previous work. **Regrettably, the reference [1] was not provided in your comments, so we are unable to make a direct comparison to the specific work you are referring to.** If you could kindly provide the missing reference, we will be more than happy to provide a detailed response and comparison.\n\nIn addition, we would like to highlight that Reviewer APCr has pointed out four references that bear similarity to the ideas in our paper. We have conducted a thorough comparison with those references and addressed the similarities and differences in [our response to Reviewer APCr's Q3](https://openreview.net/forum?id=vtoY8qJjTR&noteId=EQ1sNpef2S).\n\n---\n> Q2: It is confusing that in the universal model training (eq. (9)), the exponential of the advantages are weighted by the balance parameter $\\beta$, but when training the balance model, the Q-function (as in the loss) is not weighted by the balance parameter. There seems to be some consistency. I can tell that the unweighted objective (the current form of eq (10)) would be more computationally friendly, but theoretically, it seems more natural to optimize over a weighted version where eq. 10 is also weighted by $\\beta$, which sample from the balance model.\n\nThank you for your observations on Equations (9) and (10). We acknowledge your suggestion to weight Q with the balance coefficient in Equation (10).\n\nWe will explore the effects of this weighting in Equation (10) in two cases:\n\n- If Q were weighted by $\\beta_{\\mathbf{s}} \\sim \\pi_b$ **without stopping the gradient** (see the equation below), training $\\pi_b$ would aim to find the $\\beta_{\\mathbf{s}}$ that maximizes the Q-value and also increase $\\beta_{\\mathbf{s}}$, the output of $\\pi_b$. The latter is undesired, as FamO2O's objective is to find the proper $\\beta_{\\mathbf{s}}$ for each state $\\mathbf{s}$, not to pursue larger $\\beta_{\\mathbf{s}}$ leading to an aggressive policy.\n\n$\\qquad\\pi_b^{k+1}=\\underset{\\pi_b}{\\arg \\max}$ \n$\\mathbb{E}_{(s, a)\\sim\\mathcal{D}}[$ \n\n$\\qquad\\quad{\\color{red}\\beta_{s}} Q^k(\\mathbf{s}, \\pi_u^{k+1}(\\mathbf{s}, \\beta_{\\mathbf{s}}))], \\quad\\text{where}\\quad \\beta_{\\mathbf{s}}\\sim\\pi_b(\\mathbf{s}).$\n\n- If **the gradient were stopped** (see the equation below), the more aggressive balance could result in larger $\\beta_{\\mathbf{s}}$, skewing the Q-value. This would focus $\\pi_b$ on radical policies, leading to possible extrapolation error.\n\n$\\qquad\\pi_b^{k+1}=\\underset{\\pi_b}{\\arg \\max}$ \n$\\mathbb{E}_{(s, a)\\sim\\mathcal{D}}[$ \n\n$\\qquad\\quad{\\color{red}\\operatorname{stopgrad}(\\beta_{s})} Q^k(\\mathbf{s}, \\pi_u^{k+1}(\\mathbf{s}, \\beta_{\\mathbf{s}}))], \\quad\\text{where}\\quad \\beta_{\\mathbf{s}}\\sim\\pi_b(\\mathbf{s}).$\n\nIn summary, the unweighted formulation in Equation (10) is intentionally chosen to let $\\pi_b$ find the proper balance between improvement and constraint for maximum returns. We hope this clarifies our design decision.\n\n---\n> Q3: The action distance in eq. (11) may not be the best metric to measure the discrepancy between a policy and a trajectory. For example, if the policy that induces the trajectory only takes $a_1$ in $s$, and the evaluated policy takes $a_1$ with $p=0.51$ and $a_2$ with $p=0.49$, which may induce a very negative reward, or cause great trajectory derailment (which is not recorded in the offline trajectory), the proposed metric will still be 0 but in reality the evaluated policy is not that close to the trajectory.\n\nThank you for your insightful observation regarding the action distance in Equation (11). We understand the scenario you described. However, our specific formulation for action distance is defined as \n\n$d_{\\text{action}}^{\\pi,\\tau} = \\mathbb{E}_{(s, a)\\sim\\tau}[|| \\underset{a'}{\\arg\\max}\\pi(a'|s)-a||_2^2]$. \n\nSince the environments in D4RL are continuous action spaces, **$\\pi$ outputs a normal distribution**, and thus \n\n$\\arg\\max_{a'}\\pi(a'|s)=\\mathbb{E}_{a'}[\\pi(a'|s)]$.\n\nWith this premise, **the action distance can provide an accurate measure of the distance between the center of the distribution and the actions within the dataset**, and it should adequately reflect the discrepancy between a policy and a trajectory in the context of our work.\n\n---\n\nOnce again, we express our gratitude for your expertise and careful consideration. We remain open to further dialogue and are eager to address any more questions or concerns you may have."}, {"review_id": "j51bnUnC6u", "reviewer": "Reviewer_APCr", "summary": "The paper proposes a new algorithm to perform offline-to-online reinforcement learning. The core idea is to consider a state-adaptive balance parameter, which aims to encourage imitation of dataset behavior only if the corresponding advantages / values are high, while prior works have mostly assumed fixed balances. The authors provide detailed experimental evaluation of their approach and show superiority over relevant baselines.", "questions": "Could you please define how you use the term data quality (I'm assuming it has something to do with the data's return, but I'm not sure)?\n\nPlease elaborate why the random datasets in the D4RL datasets were not considered.\n\nThere exist offline-to-online methods that already consider something like state-adaptive policy changes, like confidence-conditioned value functions [1], as well as other adaptive offline-to-online policy concepts [2,3] that could easily be extended to your case - why were they not considered? (I realise that some are rather recent, but I think they deserve at least brief discussion)\n\nI do not understand section 6.1 / figure 8 at all - what is meant by guidance? what is meant by high / low quality data? Starting positions are denoted as triangles, suggesting a moving direction but I don't think that's intended. The text says high balance coefficients are found where data quality is also high (quality=reward / return?) & that higher quality data is found at the lower crossing point in the 5th row, however from the color encoding it seems that the balance coefficient there is actually the lowest...\n\nI think the ratio between offline and online training steps as well as collected interaction steps is a crucial parameter for reproducibility / future comparisons, however that information is only conveyed in the figure 10 axis & the appendix - I think it should be explicitly stated somewhere in the experimental section text.\n\nWhen I look at figure 10, it seems that after all online training steps, the attained performance is almost identical to that at the end of offline training - isn't that a very disappointing result and couldn't you just throw away the whole online part altogether then? Since FamO2O outperforms prior offline-to-online baselines, does that mean that these baselines only get worse with online training?\n\n[1] Hong, J., Kumar, A., & Levine, S. (2022). Confidence-Conditioned Value Functions for Offline Reinforcement Learning. ICLR 2023\n[2] Swazinna, P., Udluft, S., & Runkler, T. (2022). User-Interactive Offline Reinforcement Learning. ICLR 2023\n[3] Ghosh, D., Ajay, A., Agrawal, P., & Levine, S. (2022, June). Offline rl policies should be trained to be adaptive. ICML 2022\n[4] Emmons, S., Eysenbach, B., Kostrikov, I., & Levine, S. (2021). RvS: What is Essential for Offline RL via Supervised Learning?. arXiv preprint arXiv:2112.10751.", "limitations": "no, limitations are not discussed - perhaps looking at the not improved performance after online fine-tuning in figure 10, as well as the often only small improvements over standard IQL, a brief discussion would be good.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I think offline-to-online RL is a very relevant and promising direction for research since if reliable it would enable much more practical applications of RL in real-world tasks. The core idea of the paper, training a collection of policies that adapts on a per state level, appears logical and powerful (however please see [2,3,4], which I think put forward similar ideas) and will in the future probably significantly influence the way offline-to-online RL is performed and thought about. The paper is overall very well written and understandable & offers a strong statistical analysis of the proposed algorithms performance compared to relevant baselines.", "weaknesses": "The term data quality is frequently used but not really introduced (could mean accuracy / truth of the information in the data, but I think it means something like return).\n\nNot all D4RL locomotion datasets were used - its unclear whether that's an issue since the selection is not justified.\n\nThe considered baselines appear relevant (TD3+BC & CQL less so since its already shown in the AWAC paper that using the same conservative formulations in the online setting does not work well), however I think the closest existing methods are missing: E.g. Confidence-conditioned value functions [1] automatically adjusts its policy confidence level (as far as I understand on a per state level since every state goes into the considered history). Also methods like RvS [4], which condition on return, could easily be extended to this setting (i.e. always try to maximise conditioned return during online data collection). Generally I have the feeling that some very relevant related work regarding offline adaptive policies was not considered, i.e. also:\n[2] User-Interactive Offline RL (which considers policies conditioned on a balance between conservative and very liberal)\n[3] offline policies should be trained to be adaptive (which adapts policies based on the online collected history)\nI'm not sure the state-adaptive balance between conservative and radical idea for offline-to-online is thus entirely novel.\n\nI see some issues regarding clarity that could easily be fixed (see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your thoughtful comments and critiques are sincerely appreciated and have been instrumental in refining our study. **Due to the character limit, the references in this rebuttal are at the global rebuttal.** Thanks for your time and effort.\n\n> Q1: Definition of the term data quality?\n\nThank you for pointing this out. Your assumption that **the \"data quality\" refers to the return** is correct. Although we have indeed implicitly mentioned this in Figure 2 and Lines 44-48 of our manuscript, we will make sure to explicitly define the term in the next version of our paper.\n\n---\n\n> Q2: Elaborate on why the random datasets in the D4RL datasets were not considered.\n\nThank you for your question. Our approach mainly **follows the practice in IQL paper**, and many other works (e.g., [1,2]) also exclude tests on random datasets. **The primary reason is the RL agent's primacy bias** [3], leading to overfitting on poor-quality random datasets, resulting in performance that often does not exceed that of directly training online for the same number of gradient steps. To confirm this, we tested IQL with offline-to-online learning on `hopper-random-v2` and compared it with SAC's online learning performance. The results, as shown in the table below, demonstrate that IQL's performance on random datasets is significantly lower than SAC's. Hence, using random datasets for offline-to-online RL is considered to be of limited practical value in our context.\n|                                                            | Normalized Score Mean | Normalized Score Std |\n| ---------------------------------------------------------- | --------------------- | -------------------- |\n| IQL (offline-to-online, pre-trained on `hopper-random-v2`) | 40.02667              | 38.79444             |\n| SAC (online learning)                                      | 86.86755              | 21.07426             |\n\n---\n> Q3: Other related works [5,6,7,8]\n\nThank you for pointing out these related works. Our FamO2O method emphasizes two main features: 1) Utilizing various data, similar to Balanced Replay [4], and 2) employing a conditioned policy, related to the papers mentioned by you [5,6,7,8] and by Reviewer urL3 [9]. Having discussed Balanced Replay extensively, we will now focus on the conditioned policy's related works:\n\n1. **[5]:** They focus on diverse conservative **Q-values**, while we prioritize varying conservative **policies**. Their method **isn't suited for continuous action settings** like D4RL, but ours is versatile for both continuous and discrete actions.\n2. **[6]:** Their setting relies on **user interaction** for online adjustments, whereas ours is **automatic**.\n3. **[7]:** This paper emphasizes offline RL, estimating probabilities of various MDPs in uncovered areas with Bayesian posterior, and adjusting policies accordingly. However, in the offline-to-online RL context, where agents explore during online fine-tuning, **uncoverage may not be a primary concern**. Their focus is on **adapting to different MDPs**, while ours targets **utilizing data of varying quality**.\n4. **[8, 9]:** Despite sharing conditioned policy use (as discussed in our related work section), they aim to liberalize policy learning from merely copying offline behavior, **differing significantly from our motivation**, and their strategies are **not adaptively adjustable**.\n\nWe value your insight and will include this discussion in the related work section.\n\n---\n> Q4: Queries on guidance, high/low-quality data, starting positions & triangles, and texts in L258-262\n\nThank you for your comments and questions regarding Section 6.1 and Figure 8. We apologize for any confusion caused and provide explanations here:\n\n1. **Guidance**: Guidance refers to directing the agent to the shortest route across a crossing point during offline data collection. Without it, the agent moves randomly.\n2. **High/Low-Quality Data**: High-quality data results from the agent moving w/ guidance, while low-quality data is collected when the agent moves w/o guidance.\n3. **Starting Positions & Triangles**: The triangles don't represent moving direction. We've revised the figure to avoid confusion (see Figure 2 in the global rebuttal's PDF).\n4. **Texts in L258-262**: There were typos in lines L258-262. The corrected statement is: \"Figure 8(b) shows the agent typically outputs **lower** balance coefficients for **high-quality** samples and **higher** ones for **low-quality** data.\" This aligns with the FamO2O motivation, as lower/higher balances reflect a more conservative/radical policy.\n\nWe will add the above clarifications and modifications to the next version.\n\n---\n> Q5: Explicitly state crucial parameters in the experimental section.\n\nThanks for your constructive suggestions. We will state the crucial parameters, e.g., offline and online training steps, and collected interaction steps, in the experimental section in the next version.\n\n---\n> Q6: The almost identical performances after online fine-finetuning and offline pre-training in Figure 10.\n\nThank you for your question. Figure 10 specifically highlights **an extreme case** on a single dataset, where IQL shows **the most significant performance drop**. It demonstrates that even in this situation, IQL+FamO2O can alleviate the drop and attain good performance. However, this is an isolated instance, and generally, **as shown in the table below, both IQL and IQL+FamO2O achieve better performances after online fine-tuning compared to after offline pre-training**.\n\n|            | Offline Performance Sum | Online Performance Sum | Fine-tuning Improvement |\n| ---------- | ---------- | ------ | ----------------------- |\n| IQL        | 581.7| 718.3| +136.6|\n| FamO2O+IQL | 584.4| 772.0| +187.6|\n\n---\n> Q7: Limitations are not discussed.\n\nThanks for your advice. We will discuss limitations in the revised version.\n\n---\nThank you once again for your valuable feedback and advice. We look forward to further discussion to refine our work."}, {"review_id": "87GMp1jHJU", "reviewer": "Reviewer_urL3", "summary": "This paper approaches offline-to-online RL from the intuition that at a particular state, if the dataset already contains good actions, then the subsequent online tuning should be more conservative to retain the good actions in the dataset; but if the dataset's actions is poor, then more radical policy improvement is needed.\nTo this end, this paper introduces a framework, Family Offline-to-Online RL (FamO2O), which aims at a state-adaptive improvement-constraint balance for each state.\nSpecifically, from the collected dataset, the authors train a diverse policy family ranging from conservative to radical and use the environmental feedback to select an appropriate policy from this family at each state.\nPractically, this is achieved by a universal model, which determines the degree of policy conservatism; and a balance model, which learns the balance coefficients at each state.\nExperimental results show that FamO2O improves on various offline-to-online RL methods and achieves competitive performance.", "questions": "1. Could you explain more on how the updating rule Eqn. (10) train the balance model $\\pi_b$ to control policy improvement and constraint? Why does maximizing the $Q^k$ values achieve this balance? \n2. The significant variation in data quality across different states depicted by Figure 2 may come from the nature of the medium-expert datasets, which by construction contains medium and expert trajectories. Does this variation in data quality exist in other type of datasets, say expert, medium, or medium replay? More generally, is this phenomenon of significant data-quality variation ubiquitous or specific?\n3. Could you explain more on the benefit of \"state-adaptive improvement-constraint balances\"? Even if we do not make such a balance and simply just online finetune an offline-pretrained policy, the environmental feedback should be able to tell us which actions are good and should be retained and which actions are poor and should be improved. \n4. Could you explain more why you consider the term $\\log \\pi (a|s)$ in Eqn. (1) as a policy constraint? What is the target of this constraint, especially when $(s,a)$ is an online interaction samples? \n5. Could you explain more on how you get Eqn. (12) and (13) from Eqn. (5)? It would be better if you can expand Appendix C.1 to include more details and explanations. The current version is a bit hard to follow.\n6. nit: in Eqn. (6), are you missing a \"$\\forall \\epsilon > 0$\" before $\\exists \\\\{...\\\\}$?\n7. [L161] Could you explain more on the cooperation between $\\pi_u$ and $\\pi_b$? Why couldn't we still randomly sample $\\beta_s$ during online fine-tuning?\n8. [L159] How many $\\beta_s$ vectors are required to learn the universal model? How does this number scale with the number of states in the dataset? And how to select/design the balance coefficient space $\\mathcal{B}$?", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and generally easy-to-follow.\n2. The proposed method is well theoretically justified.\n3. The empirical discussion and ablation study are thorough.", "weaknesses": "1. The proposed method seems require abundant diverse data, which may not be feasible on harder settings, e.g., the Adroit domain in the D4RL benchmark where the data is limited and the data distribution is narrow and lack of diversity. \n2. Maybe I misunderstand something. I think the purpose of the $\\pi_b(s)$ model is to find the $\\beta_s$ that corresponds to the optimal sequence of constraints $\\\\{\\epsilon_s, s \\in \\mathcal{S}\\\\}$  in Eqn. (6), which relates to $\\beta_s$ by the Language multipliers $\\mu(s)$ (Line 537, Appendix C.2). In this regard, the paper's story of balancing policy improvement and constraint can be slightly over-complicated and somewhat confusing.\n3. The proposed method is in spirit similar to decision-transformer-style methods, and is therefore less a surprise.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our thanks to you for the detailed feedback and constructive criticism that guided our revisions.\n\n> Q1: The proposed method seems to require abundant diverse data.\n\nThanks for pointing this out. The confusion might stem from an unclear explanation in our manuscript, and we'll clarify it in the paper's next version:\n\nIt's worth noting that **our method doesn't necessarily need varied data qualities**. We claim that it can determine the best conservative/radical balance for each state in online scenarios based on the data quality in the collected dataset. **If the dataset is diverse in quality, the balances will be diverse; if the quality is consistent, the balances will be correspondingly consistent** (please refer to Figure 3 in [the global rebuttal's attached PDF](https://openreview.net/attachment?id=1S5GhI6UFd&name=pdf) for empirical evidence). Even with consistent data quality, our method has **the advantage of adaptively finding the proper balance** over existing offline RL algorithms with human-chosen balances.\n\n---\n> Q2: The paper's story of balancing policy improvement and constraint can be slightly over-complicated and somewhat confusing.\n\nThank you for your question, and apologies for any confusion. Here's the logic behind our method:\n- Our method aims to balance policy improvement and constraint, equating to different upper bounds $\\epsilon_{\\mathbf{s}}$ (see Equation 3). \n- Unlike AWAC or IQL, which use a single constraint term (Equation 5), different $\\epsilon_{\\mathbf{s}}$ create $|\\mathcal{S}|$ constraints (Equation 4), requiring $|\\mathcal{S}|$ Lagrange multipliers. \n- Given $\\beta = d_{\\pi_\\beta}(\\mathbf{s})/\\mu$ (Line 537), we must have $|\\mathcal{S}|$ different $\\beta$ values, hence the necessity for state-adaptive balance (Proposition 3.5). \n- To generate these $\\beta$ values, we introduce $\\pi_b: \\mathcal{S}\\mapsto \\mathbb{R}$, outputting $\\beta$ based on $\\mathbf{s}$.\n\nWe will add this explanation to the next version of our paper.\n\n---\n> Q3: The proposed method is in spirit similar to decision-transformer-style methods.\n\nThank you for pointing out the similarity. **Please refer to [our response to Reviewer APCr's Q3](https://openreview.net/forum?id=vtoY8qJjTR&noteId=EQ1sNpef2S)**, where we compare our method with other works **including decision transformer [9]**.\n\n---\n> Q4: How does the updating rule Eqn. (10) train the balance model $\\pi_b$ to control policy improvement and constraint? Why does maximizing the $Q^k$ values achieve this balance?\n\nThank you for your question. In Equation (9), the universal model $\\pi_u$ is trained to adjust the conservative/radical balance of the policy based on the inputted balance coefficient $\\beta_s$. Building on this, Equation (10) trains the balance model to select the $\\beta_s$ that enables $\\pi_u$ to maximize the Q value. This approach is grounded in the understanding that **the Q value serves as an estimate of future return, which is our ultimate goal of striking a balance between policy improvement and constraint**. This is in alignment with Equation (2), where it's important to note that $V(s)$ has no gradients concerning actions, and consequently, no gradients with respect to $\\pi_b$. We will include this clarification in the next version of our paper.\n\n---\n> Q5: Is the phenomenon of data-quality variation ubiquitous or specific?\n\nAs depicted in **Figure 1 of [the global rebuttal's PDF](https://openreview.net/attachment?id=1S5GhI6UFd&name=pdf)**, the phenomenon of substantial variation in data quality is a **widespread occurrence** in offline datasets. It's worth reiterating, as previously noted in response to your Q1, that while our method can harness data across a range of qualities, **it does not fundamentally depend on data quality diversity**.\n\n---\n> Q6: Could you explain more on the benefit of \"state-adaptive improvement-constraint balances\"?\n\nIndeed, as you suggest, online feedback can help the agent determine which actions are good and which are bad even without our state-adaptive improvement-constraint balances. However, **the existing fixed balance methods present two primary problems, which can be addressed by the implementation of state-adaptive balances**:\n\n1. **Difficulty in Deciding Proper Balance:** In the absence of abundant information about an offline dataset and online environment, during offline pre-training, it's **difficult to pre-specify a proper improvement-constraint balance** that can optimally deal with every future state encountered during online interaction.\n\n2. **Inflexibility of Fixed Balances:** As indicated by [3], existing offline-to-online algorithms with fixed balances inhibit drastic changes in an agent's behavior (e.g., from conservative to radical, or vice versa) due to \"primacy bias.\" In contrast, our approach facilitates flexible adjustment of conservative or radical degrees for different states during online inference, as stated in our paper. **Without our proposed state-adaptive balances, the algorithms would struggle to adapt the conservative/radical degrees appropriately for varying states**.\n\n---\n**Because of the character limit and our desire to respond to your questions thoughtfully, we've placed the remaining rebuttal in the [global rebuttal](https://openreview.net/forum?id=vtoY8qJjTR&noteId=1S5GhI6UFd). Please refer to it for the follow-up.**\n\n---\n\nAgain, we thank you for your invaluable insights and support in enhancing our work. We are eager to engage in further discussions or address any additional concerns to continue improving our manuscript."}, {"review_id": "0jmPQrk7ms", "reviewer": "Reviewer_BkWu", "summary": "Authors propose a framework for offline-to-online tuning of offline RL algorithms tunning. The idea is to train an additional network which desides helps to keep improvement-constraint balance during finetuning.", "questions": "How does the modification affect the compute time required to  train algorithms?", "limitations": "Adding another network requires additional choice of  hyperparameters which might be hard to find", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Approach improves all of the checked algorithms performance and can be applied to different offline RL algorithms. Good range of benchmarking tasks and  algorithms.", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your keen observations and valuable suggestions are highly appreciated, and we thank you for helping us to strengthen our paper.\n\n> Q1: How does the modification affect the compute time required to train algorithms?\n\nThank you for raising this question. To examine the computational overhead introduced by FamO2O, we utilized IQL and FamO2O+IQL as examples, assessing their training time using their JAX implementations over 1M offline steps and 1M online steps. The result is provided in the table below. From this analysis, we can discern that FamO2O augments the training time of IQL by roughly 8 minutes. **Given the substantial performance improvement brought by FamO2O, this increase in training time can be deemed acceptable.**\n\n| (unit: minute)             | IQL    | FamO2O+IQL |\n| -------------------------- | ------ | ---------- |\n| offline pre-training phase | 9.278  | 11.48      |\n| online fine-tuning phase   | 12.532 | 18.25      |\n| total                      | 21.81  | 29.73      |\n\n> Q2: Adding another network requires additional choice of hyperparameters which might be hard to find.\n\nThank you for your insightful observation. We do recognize the complexity that adding another network introduces due to the need to choose additional hyperparameters, such as the selection of the space of balance coefficients $\\mathcal{B}$. Nevertheless, our empirical results, as presented in Figure 11 and Figure 12 of our paper, demonstrate that **FamO2O's performance is largely unaffected by the choice of hyperparameters, e.g., $\\mathcal{B}$, within a sensible range**.\n\n---\n\nWe extend our heartfelt gratitude once more for your thorough review and thoughtful comments. We anticipate further dialogue and collaboration, and are open to any more thoughts you may have to help refine our work."}]

Author rebuttal:
### References for All Reviewers:

**Dear reviewers, due to the rebuttals' character limit, we've placed the references for all rebuttals below. Thank you for your time and consideration.**

[1] Luo, Yicheng, et al. "Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions." *arXiv preprint arXiv:2303.17396* (2023).

[2] Piche, Alexandre, et al. "Implicit Offline Reinforcement Learning via Supervised Learning." *arXiv preprint arXiv:2210.12272* (2022).

[3] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." *International conference on machine learning*. PMLR, 2022.

[4] Lee, Seunghyun. "Offline-to-online reinforcement learning via balanced experience replay and pessimistic Q-ensemble." (2021).

[5] Hong, Joey, Aviral Kumar, and Sergey Levine. "Confidence-conditioned value functions for offline reinforcement learning." *arXiv preprint arXiv:2212.04607* (2022).

[6] Swazinna, Phillip, Steffen Udluft, and Thomas Runkler. "User-Interactive Offline Reinforcement Learning." *arXiv preprint arXiv:2205.10629* (2022).

[7] Ghosh, Dibya, et al. "Offline rl policies should be trained to be adaptive." *International Conference on Machine Learning*. PMLR, 2022.

[8] Emmons, Scott, et al. "Rvs: What is essential for offline rl via supervised learning?." *arXiv preprint arXiv:2112.10751* (2021).

[9] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." *Advances in neural information processing systems* 34 (2021): 15084-15097.

[10] Nair, Ashvin, et al. "Awac: Accelerating online reinforcement learning with offline datasets." arXiv preprint arXiv:2006.09359 (2020).

---

### Follow-up Rebuttal for Reviewer urL3:

> Q7: Explain why you consider the term $\log\pi(a|s)$ in Eqn. (1) as a policy constraint. What is the target of this constraint, especially when $(s, a)$ is an online interaction sample?

**Explanation on policy constraint.** If we remove the policy improvement term in Equation 1, the Equation 1 will become $L_\pi = \mathbb{E}_{(\mathbf{s}, \mathbf{a})\sim\mathcal{D}}[\exp(\beta)\cdot \log\pi(\mathbf{a}|\mathbf{s})]$. As $\exp(\beta)$ is a constant that can be absorbed in the learning rate, the equation can be further simplified into 

$L_\pi = \mathbb{E}_{(\mathbf{s}, \mathbf{a})\sim\mathcal{D}}[\log\pi(\mathbf{a}|\mathbf{s})]$, 

which is behavior cloning that maximizes the log-likelihood of the action $\mathbf{a}$ under the state $\mathbf{s}$. Therefore, we term $\log\pi(\mathbf{a}|\mathbf{s})$ as a policy constraint due to it forcing the behavior of policy $\pi$ to be close to that of the collected dataset $\mathcal{D}$.

**Target of policy constraint.** The constraint stops excessive updates and exploration of unknown areas, particularly during the switch from offline pre-training to online fine-tuning, where it stabilizes training and prevents a performance decline (analyzed by AWAC [10]).

**Effect of policy constraint for online samples.** As new online interaction samples are added to dataset $\mathcal{D}$, they may be part of the sampled data, ensuring that the behavior policy, to which policy $\pi$ is closely constrained, gradually aligns with the online state-action distribution (also analyzed by AWAC [10]).

> Q8: Explain more on how to get Eqn. (12) and (13) from Eqn. (5)? Expand Appendix C.1 to include more details.

Thanks for your advice, and we will include more details in Appendix C.1 in the next version. Due to the character limits, please refer to the attached PDF of this global rebuttal for an explanation of how to get Eq. (12), (13) from Eq. (5).

> Q9: Miss a "$∀\epsilon\ge0$" before $\exists\{\cdots\}$ in Eqn. (6)?

Thank you for pointing this out. We will add "$\forall \epsilon \ge 0$" to Eqn. (6) in the revised version.

> Q10: [L161] Explain on the cooperation between $\pi_u$ and $\pi_b$. Why not still randomly sample $\beta_{s}$ during online fine-tuning?

Thank you for the feedback. The intuitions behind the interplay between the universal model $\pi_u$ and the balance model $\pi_b$ are as follows:
1. **Offline Pre-training Phase**: At this stage, due to the absence of online feedback, it's uncertain which balance coefficient $\beta_{\mathbf{s}}$ results in an optimal policy for any given state $\mathbf{s}$. Consequently, **$\pi_u$ is exposed to random $\beta_{\mathbf{s}}$ values, enabling it to learn from a diverse range of policies**. On the other hand, the balance model $\pi_b$ is trained by maximizing the Q-value, but the Q-value might not always be accurate, particularly for unseen areas. As such, during offline pre-training, **it's not feasible for $\pi_b$ to pinpoint the ideal balance coefficients for $\pi_u$.**
2. **Online Fine-tuning Phase**: With the advantage of online feedback, the Q-value refines, enhancing the reliability of $\pi_b$. Allowing $\pi_b$ to ascertain $\beta_{\mathbf{s}}$ for $\pi_u$ not only **benefits from $\pi_b$'s improved performance** but also **compels $\pi_u$ to emphasize the candidate policies frequently opted by $\pi_b$**.

> Q11: [L159] How many $\beta_{\mathbf{s}}$ vectors are required to learn $\pi_u$? How does this number scale with the number of states in the dataset? And how to select/design the balance coefficient space $\mathcal{B}$?

Thank you for your question concerning the number of $\beta_{\mathbf{s}}$ vectors and the selection of the balance coefficient space $\mathcal{B}$. In our paper, $\beta_{\mathbf{s}}$ is actually a scalar, sampled from $\mathcal{B} = [\beta_{\text{low}},\beta_{\text{high}}]$, thus **the number of $\beta_{\mathbf{s}}$ is infinite**.

As for the selection of $\mathcal{B}$, we have discussed this in lines L171-172, Appendix E.2 and F.2. Briefly, our method's performance is insensitive to the choice of $\beta_{\text{low}}$ and $\beta_{\text{high}}$, provided we refrain from using extremely radical values in $\mathcal{B}$. More details can be found in the aforementioned parts of the paper.

Meta-review: This paper is concerned with addressing the distributional shift problem in offline-to-online reinforcement learning (RL) through introducing FamO2O, a framework that enables existing algorithms to determine state-adaptive improvement-constraint balances. The proposed FamO2O framework leverages a universal model to train policies with varying improvement and constraint intensities, and a balance model to select the most suitable policy for each state, emphasizing the necessity of state-adaptive balances for achieving higher performance bounds. Experiments demonstrate that FamO2O outperforms previous baselines. Generally, this paper is strong in its simple yet effective methods, with both theoretical and experimental support. The presentation is also satisfactory.
Decision: Accept (spotlight)