Title: Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

offline-to-online reinforcement learning

offline reinforcement learning

policy improvement

policy constraint
Abstract: Offline-to-online reinforcement learning (RL) is a training paradigm that combines pre-training on a pre-collected dataset with fine-tuning in an online environment. However, the incorporation of online fine-tuning can intensify the well-known distributional shift problem. Existing solutions tackle this problem by imposing a policy constraint on the policy improvement objective in both offline and online learning. They typically advocate a single balance between policy improvement and constraints across diverse data collections. This one-size-fits-all manner may not optimally leverage each collected sample due to the significant variation in data quality across different states. To this end, we introduce Family Offline-to-Online RL (FamO2O), a simple yet effective framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O utilizes a universal model to train a family of policies with different improvement/constraint intensities, and a balance model to select a suitable policy for each state. Theoretically, we prove that state-adaptive balances are necessary for achieving a higher policy performance upper bound. Empirically, extensive experiments show that FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark. Codes are available at https://github.com/LeapLabTHU/FamO2O.

Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning

Abstract

Offline-to-online reinforcement learning (RL) is a training paradigm that combines pre-training on a pre-collected dataset with fine-tuning in an online environment. However, the incorporation of online fine-tuning can intensify the well-known distributional shift problem. Existing solutions tackle this problem by imposing a policy constraint on the policy improvement objective in both offline and online learning. They typically advocate a single balance between policy improvement and constraints across diverse data collections. This one-size-fits-all manner may not optimally leverage each collected sample due to the significant variation in data quality across different states. To this end, we introduce Family Offline-to-Online RL (FamO2O), a simple yet effective framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O utilizes a universal model to train a family of policies with different improvement/constraint intensities, and a balance model to select a suitable policy for each state. Theoretically, we prove that state-adaptive balances are necessary for achieving a higher policy performance upper bound. Empirically, extensive experiments show that FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark. Codes are available at https://github.com/LeapLabTHU/FamO2O .

Train a policy family based

on the collected dataset

Learn policy selection from the

online environment feedbacks

vanilla

w/ FamO2O

(a) General intuitions of our proposed FamO2O

(b) FamO2O's performance enhancement

𝑫

varied-balance policy family

conservative

radical

train

⋯

feedback

action

online environment

state-wise select

⋯

collected dataset

policy

selector

state

3

1 Introduction

Offline reinforcement learning (RL) provides a pragmatic methodology for acquiring policies utilizing pre-existing datasets, circumventing the need for direct environment interaction [29]. Nonetheless, the attainable policy performance in offline RL is frequently constrained by the quality of the dataset [66]. The offline-to-online RL paradigm addresses this limitation by refining the offline RL policy through fine-tuning in an online setting [35].

While online fine-tuning can indeed elevate policy performance, it also potentially exacerbates the issue of distributional shift [29], where policy behavior diverges from the dataset distribution. Such shifts typically ensue from drastic policy improvements and are further amplified by state distribution changes when transitioning from offline learning to online fine-tuning [11, 25, 9, 26]. Prior works have attempted to counter this by imposing policy constraints on the policy improvement objective to deter excessive exploration of uncharted policy space [25, 54, 35]. However, this conservative approach can inadvertently stifle policy improvement [35]. In essence, offline-to-online RL necessitates an effective balance between policy improvement and policy constraint during policy optimization.

Regrettably, prior offline-to-online RL algorithms tend to adopt a monolithic approach towards this improvement-constraint trade-off, indiscriminately applying it to all data in a mini-batch [10, 66, 53] or the entire dataset [35, 24, 27, 30]. Given the inherent data quality variation across states (see Figure 2), we argue that this one-size-fits-all manner may fail to optimally exploit each sample. In fact, data yielding high trajectory returns should encourage more 'conservative" policies, while data leading to poor returns should incite more 'radical" policy improvement.

high-quality

low-quality

(a) HalfCheetah

(b) Hopper

In this paper, we introduce a novel framework, Family Offline-to-Online RL (FamO2O), which can discern a state-adaptive improvement-constraint balance for each state. FamO2O's design is founded on two key insights delineated in Figure 1(a): (i) The collected dataset, abundant in environmental information, could facilitate the training of a diverse policy family ranging from conservative to radical, and (ii) feedback from online learning might assist in selecting an appropriate policy from this family for each state. As depicted in Figure 3, FamO2O incorporates a universal model and a balance model . The universal model, conditioned on a balance coefficient , determines the degree of policy conservatism or radicalism, while the balance model learns a mapping from states to balance coefficients, aiding the universal model in tailoring its behavior to each specific state.

FamO2O represents, to the best of our knowledge, the first offline-to-online RL approach harnessing the power of state-adaptive improvement-constraint balances. Theoretically, we establish that in policy optimization, point-wise KL constraints afford a superior performance upper bound compared to the distributional KL constraints adopted in prior works [37, 35, 24]. Importantly, these state-adaptive balances become indispensable when addressing point-wise KL constraints, thereby underlining the necessity of incorporating such balances in offline-to-online RL. Experimental results, as summarized in Figure 1(b), reveal that FamO2O is a simple yet effective framework that statistically significantly improves various offline-to-online RL algorithms and achieves state-of-the-art performance.

2 Preliminaries

We introduce essential RL and offline-to-online RL concepts and notations here. For descriptive convenience and theoretical analysis, we use the advantaged-weight regression (AWR) algorithm framework [37, 35, 24], but FamO2O isn't limited to the AWR framework . We later demonstrate its integration with non-AWR algorithms in Section 5.3 and Appendix D.

2.1 Reinforcement learning formulation

RL is typically expressed as a Markov decision process (MDP) [43], denoted as ( S , A , P, d 0 , R, γ ) . Here, S and A are the state 1 and action space; P ( s t +1 | s t , a ) is the environmental state transition probability; d 0 ( s 0 ) represents initial state distribution; R ( s t , a t , s t +1 ) is the reward function; and γ ∈ (0 , 1] is the discount factor.

2.2 Offline-to-online reinforcement learning

Offline-to-online RL is a training paradigm including two phases: (i) offline pre-training : pre-training a policy based on an offline dataset; (ii) online fine-tuning : fine-tuning the pre-trained policy by interacting with the environment. Note that in the offline pre-training phase, the policy cannot interact with the environment, but in the online fine-tuning phase, the policy has access to the offline dataset.

Similar to offline RL algorithms, offline-to-online RL algorithms' training objectives usually consist of two terms, either explicitly [35, 24] or implicitly [27, 28]: (i) policy improvement , which aims to optimize the policy according to current value functions; (ii) policy constraint , which keeps the policy around the distribution of the offline dataset or current replay buffer. Using the AWR algorithm framework [37, 35, 24], we demonstrate our method (also applicable to non-AWR algorithms) and define notations in Equation (1) for later use.

L π is a maximization objective, while D represents the collected dataset. Initially, during offline pretraining, D starts with the pre-collected offline dataset D offline = { ( s k , a k , s ′ k , r k ) | k = 1 , 2 , · · · , N } . As we move to the online fine-tuning phase, online interaction samples are continuously incorporated into D [35, 24]. The balance coefficient β is a predefined hyperparameter moderating between the policy improvement and policy constraint terms, while the imitation weight sets the imitation intensity for the state-action pair ( s, a ) .

3 State-Adaptive Balance Coefficients

Our design of state-adaptive improvement-constraint balances is motivated by the observations that (i) the quality of the dataset's behavior, i.e. , the trajectory returns, fluctuates greatly with different states, as shown in Figure 2; (ii) state-dependent balances are conducive to a higher performance upper bound. In this section, we will theoretically validate the latter point.

We first present the policy optimization problem with point-wise KL constraints in Definition 3.1, which is the focus of FamO2O:

Definition 3.1 (Point-wise KL Constrained Optimization Problem) . Weconsider a policy optimization problem defined as follows:

Here, π k ( k ∈ N ) denotes the policy at iteration k , π β signifies a behavior policy representing the action selection way in the collected dataset D , d π β ( s ) refers to the state distribution of π β , and ϵ s is a state-related constant. The optimal policy derived from Equations (2) to (4) is designated as π k +1 .

The optimization problem common in previous work [37, 35, 24] is shown in Definition 3.2:

Definition 3.2 (Optimization problem with distributional KL constraints) . The definition of the policy optimization problem with distributional KL constraints is the same as Definition 3.1, except that

Equation (3) in Definition 3.1 is substituted by Equation (5), where ϵ is a constant:

Remark 3.3 . The update rule in Equation (1) is based on the optimization problem in Definition 3.2.

The point-wise constraints' superiority over distributional constraints is shown in Proposition 3.4:

Proposition 3.4 (Advantage of point-wise KL constraints) . Denote the optimal value in Definition 3.1 as J k ∗ [ { ϵ s , s ∈ S} ] , the optimal value in Definition 3.2 as J k ∗ [ ϵ ] . These optimal values satisfy:

Proposition 3.4 indicates that the optimal value under the point-wise KL constraints, given suitable point-wise constraints, is no less than that under distributional KL constraints. This finding justifies our approach under point-wise constraints.

Proposition 3.5 shows the necessity of state-dependent balance coefficient design in solving the point-wise KL constraint optimization problem:

Proposition 3.5 (State-dependent balance coefficient) . Consider the optimization problem in Definition 3.1. Assume that the state space S = [ s min , s max ] l ( l is the state dimension), and the feasible space constrained by Equations (3) to (4) is not empty for every s ∈ S . Then the optimal solution of π k +1 , denoted as π k +1 ∗ , satisfies that ∀ s ∈ S , a ∈ A ,

where β s is a state-dependent balance coefficient, and Z s is a normalization term. When utilizing a parameterized policy π ϕ to approximate the optimal policy π k +1 ∗ , the training objective can be formulated as:

Proof.

In contrast to AWR [37] and AWAC [35], Proposition 3.5 highlights state-dependent (marked in blue) balance coefficients in Equations (7) to (8), as opposed to a pre-defined hyperparameter in Equation (1). This state-adaptiveness is due to Proposition 3.5 considering the finer-grained constraints in Definition 3.1. Together, Proposition 3.4 and Proposition 3.5 indicate state-adaptive balance coefficients contribute to a higher performance upper bound.

4 Family Offline-to-Online RL

Section 3 theoretically proves that finer-grained policy constraints enhance performance upper bounds, necessitating state-adaptive balance coefficients. Accordingly, we introduce FamO2O, a framework adaptively assigning a balance coefficient to each state, easily implemented over various offline-to-online algorithms like [35, 24, 27], hereafter called the 'base algorithm'.

Essentially, FamO2O trains a policy family with varying balance coefficients during offline pretraining. During online fine-tuning, FamO2O

𝛽𝑠

balance model

𝜋𝑏

𝛽𝑠

𝑠

state-adaptive

balance coefficient

𝜋𝑢(𝑎|𝑠, 𝛽 𝑠

)

action

𝑎

state

𝑠

environment

𝑃(𝑠′|𝑠, 𝑎)

identifies the appropriate policy, corresponding to the suitable balance coefficient, for each state from this policy family. In this section, we present FamO2O using the AWR algorithm framework (Equation (1)). FamO2O's compatibility with non-AWR algorithms is discussed in Appendix D.

As shown in Figure 3, FamO2O's policy consists of two components: a universal model π u and a balance model π b . Denote the space of balance coefficients as B . For every state s , the balance model 2 π b : S ↦→ B figures out a suitable balance coefficient β s ; based on the state s and state-related balance coefficient β s , the universal model π u : S × B ↦→ A outputs an action. The balance coefficient β s is to control the conservative/radical degree of the universal model π u in dealing with the state s .

4.1 Learning universal model

We initially address training the universal model π u , aimed at learning a policy family with varying balances between policy improvement and constraint. The formal optimization target of π u is:

Q k and V k , detailed in Section 4.3, represent Q and V functions at iteration k . Equation (9) echoes Equation (8), but the policy also takes a balance coefficient β s as input (highlighted in blue). In the offline pre-training phase, β s is randomly sampled from balance coefficient space B . This encourages π u to learn varied strategies. During online fine-tuning, β s is set by balance model π b before input to universal model π u , which prompts cooperation between π u and π b .

4.2 Learning balance model

Next, we outline how the balance model π b chooses an appropriate policy for each state from the policy family trained by the universal model π u . As indicated in Section 4.2, every β s ∈ B corresponds to a unique policy. Consequently, to select the optimal policy, π b needs to determine the appropriate balance coefficient β s for each state s . Given this rationale, the update rule for π b is:

Here, π k +1 u represents the updated universal model in Equation (9). Intuitively, Equation (10) aims to find a π b that maximizes Q k value by translating balance coefficients into actions with π k +1 u . This design is grounded in the understanding that the Q value serves as an estimate of future return, which is our ultimate goal of striking a balance between policy improvement and constraint. Concerns may arise about Q k 's extrapolation error in Equation (10) potentially misguiding π b 's update. Empirical evidence suggests this is less of an issue if we avoid extremely radical values in the balance coefficient space B . Following the update rule in Equation (10), π b effectively assigns balance coefficients to states, demonstrated in Section 6.1.

4.3 Learning value functions

Furthermore, we explain the value functions update. As per Equations (9) to (10), a single set of Q and V functions evaluate both π u and π b . This is due to π b : S ↦→ B and π u : S × B ↦→ A collectively forming a standard RL policy π u ( · , π b ( · )) : S ↦→ A . Hence, the value functions update mirrors that in the base algorithm, simply replacing the original policy with π u ( · , π b ( · )) .

Finally, we offer a pseudo-code of FamO2O's training process in Appendix B.

5 Experimental Evaluation

In this section, we substantiate the efficacy of FamO2O through empirical validation. We commence by showcasing its state-of-the-art performance on the D4RL benchmark [8] with IQL [24] in Section 5.1. We then evidence its performance improvement's generalizability and statistical significance in Section 5.2. Moreover, Section 5.3 reveals FamO2O's compatibility with non-AWR-style algorithms like CQL [27], yielding significant performance enhancement. Lastly, we reserve detailed FamO2O analyses for Section 6 and Appendix F. For more information on implementation details, please refer to Appendix E.2.

0.45

0.60

0.75

0.90

BC

SAC

TD3+BC

AWAC

CQL

BR

IQL

Ours

IQM

0.45

0.60

0.75

0.90

Median

0.30

0.45

0.60

Optimality Gap

0.45

0.60

0.75

Mean

D4RL Normalized Score (Normalized to [0, 1])

1

Datasets Our method is validated on two D4RL [8] benchmarks: Locomotion and AntMaze. Locomotion includes diverse environment datasets collected by varying quality policies. We utilize IQL [24] settings, assessing algorithms on hopper, halfcheetah, and walker2d environment datasets, each with three quality levels. AntMaze tasks involve guiding an ant-like robot in mazes of three sizes (umaze, medium, large), each with two different goal location datasets. The evaluation environments are listed in Table 1's first column.

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Overall

Antmaze

Locomotion

P(FamO2O > Base)

CQL

CQL+FamO2O

CQL+BR

115

Scores

110

105

Normalized

100

95

90

85

medium-replaymediummedium-expertoverall

DatasetTypes

(a) larger imitation weights

Base

FamO2O

(b) closer to the trajectories

Base

FamO2O

normalizedtrajectoryreturn

Metrics Considering RL's inherent variability, we adopt robust evaluation methods per rliable [3]. Besides conventional Medium and Mean, we integrate IQM and Optimality Gap metrics for broader assessment. We also employ rliable's probability of improvement metric for gauging the likelihood of our method outperforming others. We confirm our performance enhancement's statistical significance using 95% Confidence Intervals (CIs).

5.1 Benchmark Comparison

FamO2O's state-of-the-art performance is demonstrated by implementing it over IQL [24] and comparing with baseline methods.

Baselines We benchmark FamO2O against: (i) offline-to-online RL , including IQL [24], Balanced Replay (BR) [28], CQL [27], AWAC [35], and TD3+BC [10]. For IQL, BR, and AWAC, official implementations are used. In the case of CQL and TD3+BC, we implement online fine-tuning based on the author-provided offline pre-training codes, following the procedures in IQL and AWAC; (ii) online RL method , SAC [13], to highlight offline pre-training's efficacy; (iii) behavior cloning (BC) , which is implemented by maximizing the log-likelihood of the samples in the offline dataset. For SAC and BC, we utilize the implementations of CQL. Further details are in Appendix E.1.

Comparison As shown in Figure 5, FamO2O outperforms competitors across all metrics (IQM, Medium, Optimality Gap, and Mean). Specifically, for IQM, Optimality Gap, and Mean, FamO2O's 95% CIs don't overlap with the competitors'. Even for Medium, all baseline expectations fall below the lower limit of FamO2O's 95% CIs. The results underscore the significant edge of our state-adaptive policy constraint mechanism over competing methods.

5.2 Analyzing FamO2O's Performance Enhancement

Though FamO2O demonstrated superior performance on the D4RL benchmark in Section 5.1, it's vital to discern the actual contributions of FamO2O from its base policy, IQL. Therefore, we address two key questions: (i) Does FamO2O consistently enhance other offline-to-online RL algorithms? (ii) Is the performance boost by FamO2O statistically significant given RL's inherent variability?

Setup We apply FamO2O to AWAC and IQL. AWAC [35] is one of the most famous offline-toonline algorithms, and IQL [24] is a recently proposed method that achieves great performance on D4RL [8]. We use the authors' codes, maintaining the same hyperparameters for a fair comparison. Further details are in Appendix E.2.

Comparison Table 1 shows AWAC's and IQL's performances w/ and w/o FamO2O. FamO2O generally enhances performance by a statistically significant margin across most datasets, regardless of the base algorithm, highlighting its versatility. Even on complex datasets where AWAC barely succeeds, e.g. , AntMaze medium-diverse and medium-play, FamO2O still achieves commendable performance.

Pursuing rliable's recommendation [3], we evaluated FamO2O's statistical significance by calculating average probabilities of improvement against base policies (Figure 4). In all three cases (Locomotion,

AntMaze, and Overall), the lower CI bounds exceed 50%, denoting the statistical significance of FamO2O's improvement. Specifically, the upper CI on Locomotion surpasses 75%, demonstrating statistical meaning as per the Neyman-Pearson criterion.

5.3 Versatility of FamO2O with Non-AWR Algorithms

To demonstrate FamO2O's versatility beyond AWR-based algorithms, we extended it to CQL [27] in addition to AWAC [35] and IQL [24]. The implementation specifics are in Appendix D. As Figure 6 reveals, FamO2O significantly outperforms CQL. Even when compared to Balance Replay (BR) [28], an offline-to-online method designed specifically for CQL, FamO2O still shows statistically significant superior performance. These results highlight FamO2O's adaptability to non-AWR algorithms.

6 Discussion

In this section, we further provide some in-depth studies on FamO2O, including visualization (Section 6.1) and quantitative analyses (Sections 6.2 to 6.6). More analyses are deferred to Appendix F.

6.1 Does FamO2O really have state-wise adaptivity?

Here, we design a simple maze environment to visualize the state-wise adaptivity of FamO2O. As shown in Figure 8(a), the agent starts at a random cell in the top row and is encouraged to reach the goal at the bottom right corner through two crossing points. During data collection, guidance is provided when the agent passes through the lower crossing point, but no guidance for the upper crossing point. To elaborate, the guidance refers to compelling the agent to adhere to the route and direction that yields the shortest path to the goal. Without it, the agent moves randomly. It can be observed in Figure 8(b) that the agent generally outputs lower balance coefficients for the states with high-quality samples ( i.e. , those derived from the agent's movement with guidance) while generating higher

possible starting points

w/o guidance

w/ guidance

goal

(a)

(b)

high

low

balance coefficients for the states with low-quality data ( i.e. , data gathered when the agent moves without guidance). This result shows FamO2O's state-wise adaptivity in choosing proper balance coefficients according to the data quality related to the current state.

6.2 What is the effect of state-adaptive balances?

In this section, we explore the impact of state-adaptive improvement-constraint balance. In order to encompass data of varying quality levels, we assess FamO2O using the medium-replay datasets of D4RL Locomotion [8]. Our analysis focuses on two metrics: imitation weights and action distances .

Imitation weights are defined in Equation (1), with larger (or smaller) values prompting the agent to align more closely (or less) with the replay buffer D 's behavior. Action distance , delineated in Equation (11), quantifies the discrepancy in action selection between a policy π and a trajectory τ :

Here, a lower (or higher) action distance d π,τ action signifies a greater (or lesser) behavioral similarity between the policy π and the trajectory τ .

We evaluate IQL with and without FamO2O ('Base' and 'FamO2O') regarding imitation weights and action distances, depicted in Figure 7. Figure 7(a) computes the average imitation weight difference (AIWD) per trajectory in the offline dataset. AIWD indicates the mean imitation weight difference between FamO2O and the base algorithm for each ( s, a ) pair within a trajectory. Figure 7(b) likewise determines an average action distance difference per offline dataset trajectory.

Balance Coefficient

(a) Ablation on AWAC

(b) Ablation on IQL

adaptive balance

95% CIs

fixed balance

FamO2O

Base

Figure 7(a) reveals that FamO2O typically shows higher imitation weights than the base algorithm for high-return trajectories. Figure 7(b) indicates that FamO2O aligns more with high-quality trajectories and less with low-quality ones than the base algorithm. These results highlight the state-adaptive balance's role in promoting emulation of high-quality behavior and avoidance of low-quality behavior.

6.3 State-adaptive balances vs. fixed balances?

To prove that our adaptive balance coefficients are better than traditional fixed balance coefficients, we compare FamO2O against the base algorithms with different balance coefficients as hyperparameters. As shown in Figure 9, on both AWAC [35] and IQL [24], our adaptive balance coefficients outperform all the fixed balance coefficients. Significantly, the 95% CIs of adaptive and fixed balance coefficients have no overlap. These comparison results indicate that our adaptive balance coefficient approach surpasses the fixed balance coefficient method by a statistically significant margin.

6.4 Does FamO2O's efficacy rely on varied data qualities?

It's worth noting that our method's efficacy doesn't rely on varied data qualities. Table 1 clearly demonstrates that FamO2O surpasses the base algorithms in performance across all the medium datasets, namely hopper-medium-v2, halfcheetah-medium-v2, and walker2dmedium-v2, which all maintain a relatively consistent data quality. We claim that FamO2O can determine the suitable conservative/radical balance for each state in online scenarios based on the data quality in the collected dataset. If the dataset is diverse in quality, the balances will be diverse; if the quality is consistent, the balances will be correspondingly consistent. The above claim is supported by Figure 11, which indi-

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

Step

1e6

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Balance Coefficient

medium

cates that (i) in datasets with more (or less) diverse data qualities, i.e., medium-expert (or medium), the balance coefficients are more (or less) diverse, with corresponding larger (or smaller) standard deviations; (ii) with higher (or lower) quality datasets, the balance coefficients are averagely lower (or higher), leading to a more conservative (or radical) policy.

6.5 Does FamO2O mitigate the performance drop stemming from the distributional shift?

Despite FamO2O's primary objective not being direct distributional shift handling, its state-adaptive improvement-constraint balances prove beneficial in mitigating performance degradation during the

offline pre-training to online fine-tuning transition, attributed to the distributional shift. Existing offline-to-online RL algorithms [24, 35] already incorporate mechanisms to counter distributional shift, hence significant performance drops are infrequent. Figure 10 illustrates the training curves of IQL and IQL+FamO2O across both offline pre-training (negative steps) and online fine-tuning (positive steps) on antmaze-umaze-diverse, on which IQL exhibits the most significant decline in performance when transitioning from offline pre-training to online fine-tuning. As evidenced, while IQL+FamO2O initially experiences a performance drop akin to IQL during online fine-tuning, it recovers rapidly and attains high performance, in stark contrast to IQL's sustained performance decline throughout the fine-tuning stage.

6.6 Balance model vs. random selector?

To validate the effect of the balance model π b in choosing balance coefficients, we present a FamO2O variant, denoted as random-FamO2O , with the balance model replaced with a random balance coefficient selector. Other training settings keep the same for FamO2O and random-FamO2O. Table 2 shows the improvement percentages and probability of improvement of FamO2O against random-FamO2O. As we can observe, FamO2O outperforms random-FamO2O on almost all the datasets of D4RL Locomotion [8]. Furthermore, the lower CI of the probability of improvement is much higher than 50%, and the upper CI exceeds 75%. This indicates that the effect of the balance model is not only statistically significant but also statistically meaningful as per the Neyman-Person statistical testing criterion.

7 Conclusion

This work underscores the significance of state-adaptive improvement-constraint balances in offline-toonline RL. We establish a theoretical framework demonstrating the advantages of these state-adaptive balance coefficients for enhancing policy performance. Leveraging this analysis, we put forth Family Offline-to-Online RL (FamO2O), a versatile framework that equips existing offline-to-online RL algorithms with the ability to discern appropriate balance coefficients for each state.

Our experimental results, garnered from a variety of offline-to-online RL algorithms, offer substantial evidence of FamO2O's ability to significantly improve performance, attaining leading scores on the D4RL benchmark. In addition, we shed light on FamO2O's adaptive computation of state-adaptive improvement-constraint balances and their consequential effects through comprehensive analyses. Ablation studies on the adaptive balances and balance model further corroborate FamO2O's efficacy.

The limitation of our work is that FamO2O has been evaluated on just a handful of representative offline-to-online RL algorithms, leaving a vast array of such algorithms unexamined. Additionally, FamO2O's utility is somewhat limited, as it is applicable exclusively to offline-to-online algorithms. In future work, we aim to expand the applicability of FamO2O by integrating it with a broader spectrum of offline-to-online algorithms. Additionally, we intend to investigate the feasibility of incorporating state-adaptive improvement-constraint balances in offline RL settings, where online feedback is intrinsically absent, or even applying our adaptive design to the LLM-as-agent domain [49, 50].

References

Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , 2020.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems , 2021.

Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. In Advances in Neural Information Processing Systems , 2020.

Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems , 1992.

Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Artificial Intelligence Research , 2000.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.

Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning , 2019.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems , 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , 2019.

Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning , 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , 2018.

Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation. In International Conference on Machine Learning , 2017.

Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song, Junfeng Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic early-exiting networks. In European Conference on Computer Vision , 2022.

Yizeng Han, Zhihang Yuan, Yifan Pu, Chenhao Xue, Shiji Song, Guangyu Sun, and Gao Huang. Latencyaware spatial-wise dynamic networks. In Advances in Neural Information Processing Systems , 2022.

Bernhard Hengst. Discovering hierarchy in reinforcement learning with hexq. In International Conference on Machine Learning , 2002.

Joey Hong, Aviral Kumar, and Sergey Levine. Confidence-conditioned value functions for offline reinforcement learning. In International Conference on Learning Representations , 2023.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , 2016.

William Karush. Minima of Functions of Several Variables with Inequalities as Side Conditions . Springer Basel, 2014.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems , 2009.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems , 2019.

Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. In Advances in Neural Information Processing Systems , 2020.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems , 2020.

Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning , 2022.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS , 2022.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019.

Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems , 2018.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems , 2020.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Yifan Pu, Yizeng Han, Yulin Wang, Junlan Feng, Chao Deng, and Gao Huang. Fine-grained recognition with learnable semantic data augmentation. arXiv preprint arXiv:2309.00399 , 2023.

Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao Huang. Rank-detr for high quality object detection. In Advances in Neural Information Processing Systems , 2023.

Yifan Pu, Yiru Wang, Zhuofan Xia, Yizeng Han, Yulin Wang, Weihao Gan, Zidong Wang, Shiji Song, and Gao Huang. Adaptive rotated convolution for rotated object detection. In International Conference on Computer Vision , 2023.

Rembert Reemtsen. Discretization methods for the solution of semi-infinite programming problems. Journal of Optimization Theory and Applications , 1991.

Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin A. Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations , 2020.

Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning . MIT press Cambridge, 1998.

Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 1999.

Phillip Swazinna, Steffen Udluft, and Thomas Runkler. User-interactive offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , 2017.

Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning , 2017.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432 , 2023.

Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320 , 2023.

Shenzhi Wang, Liwei Wu, Lei Cui, and Yujun Shen. Glancing at the patch: Anomaly localization with global and local feature comparison. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. In Advances in Neural Information Processing Systems , 2020.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Qisen Yang, Shenzhi Wang, Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. arXiv preprint arXiv:2306.03362 , 2023.

Qisen Yang, Shenzhi Wang, Qihang Zhang, Gao Huang, and Shiji Song. Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance. arXiv preprint arXiv:2309.01448 , 2023.

Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In International Conference on Learning Representations , 2019.

Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. In Robotics: Science and Systems , 2017.

Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In International Conference on Machine Learning , 2023.

Yang Yue, Bingyi Kang, Xiao Ma, Gao Huang, Shiji Song, and Shuicheng Yan. Offline prioritized experience replay. arXiv preprint arXiv:2306.05412 , 2023.

Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:2210.09241 , 2022.

Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Value-consistent representation learning for data-efficient reinforcement learning. In AAAI , 2023.

Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. In International Conference on Learning Representations (ICLR) , 2023.

Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations , 2021.

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072 , 2020.

Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210.13846 , 2022.

Appendices

Contents

A Related Work

Offline RL Reinforcement learning (RL) usually improves a policy through continuous online interactions with the environment [43, 32, 62]. To reduce the huge demand for online interactions, especially when they are costly or risky, researchers proposed offline RL that utilizes pre-collected datasets to improve the policy without interacting with the environment [29]. Directly applying off-policy algorithms in offline RL usually leads to poor performance. This phenomenon is due to distributional shift , where the agent may learn inaccurate value estimates of out-of-distribution (OOD) state-action pairs [29, 61, 60]. Existing algorithms address this issue via policy constraints [11, 25, 54, 23, 37, 35], importance sampling [19, 14, 65], regularization [33, 27], uncertainty estimation [2], and imitation learning [4, 42, 52].

Offline-to-online RL Offline-to-online RL adds online fine-tuning to offline RL to enhance policy performance. Similar to the distributional shift problem in offline RL, offline-to-online (O2O) RL also suffers from off-policy bootstrapping error accumulation caused by OOD data, which causes a large 'dip" in the initial performance of online fine-tuning [35]. Moreover, O2O RL algorithms tend to be excessively conservative and result in plateauing performance improvement [35]. This emphasizes the importance of finding a balance between being optimistic about improving the policy during the online phase and still being constrained to the conservative offline policy. However, as previously discussed, existing O2O RL algorithms generally adopt a 'one-size-fits-all' trade-off, either in the process of Q-learning [27, 30, 24, 31], along with the policy improvement [35], via a revised replay buffer [28], or through some alignment approaches [53, 59]. It's worth noting that, [66] also proposes an adaptive weight in online fine-tuning. However, the adaptive weight is a single parameter for all the samples. In addition, its update requires an extra human-defined target score, which is absent in the setting considered by our competitors and us. Furthermore, [63] presents a policy expansion strategy that adaptively selects a policy from a set, which includes both offline pre-trained policies and online learned policies. Nevertheless, this policy set comprises only two policies, in contrast to the infinite number of policies contained in FamO2O's policy family.

Conditioned RL policy Algorithms in this scheme can be divided into two stages: (i) a set of policies is trained; (ii) the policy with the highest performance is selected. Generally, these methods learn a family of policies that exhibit different behaviors and search for the best one that successfully executes the desired task [36, 57]. A similar method to our FamO2O is UP-OSI [58], where a Universal Policy (UP) within a parameterized space of dynamic models and an Online System Identification (OSI) function that predicts dynamic model parameters based on recent state and action history are proposed. Unlike these prior methods utilizing models to identify the environmental parameters, the balance model in our method aims to determine a proper balance between policy improvement and constraints for each state. It's worth noting that [18, 45, 12] also exhibit conditioned designs. However, [18] focuses on conditioned value functions and pure offline RL; [45] relies on human interactions; [12] concentrates on adjusting offline RL policies according to the estimated probabilities of various MDPs in uncovered areas, but uncoverage might not be a primary concern in offline-to-online RL due to the accessibility to the online environment. Moreover, [56, 55] both exhibit a comparable understanding of how to adjust policy improvement and policy constraint across samples of varying quality. Nevertheless, the focus of [56, 55] is on offline RL. Specifically, [56] requires a small number of expert demonstrations, while [55] necessitates annotated preferences to appropriately calibrate the balance for each individual sample. Additionally, in areas such as computer vision, there are existing works that condition their training and/or inference on various input samples [15, 16, 40, 38, 39, 51]. These studies align with the adaptive approach in our work.

Hierarchical RL Hierarchical Reinforcement Learning (HRL) decomposes complex tasks into manageable subtasks, enhancing sample efficiency, interpretability, and generalization in RL. HRL approaches bifurcate into predefined and learned options (or skills). Predefined options, guided by domain knowledge or human demonstrations, shape high-level policy or constrain action space, exemplified by MAXQ [6], options framework [44], skill chaining [22], and feudal RL [5]. Learned options, optimizing objectives like entropy or reward, are seen in option-critic, skill-discovering [22], feudal networks [48], HEXQ [17], DIAYN [7], HIRO [34], and HIDIO [64]. While FamO2O shares some design elements with HRL, such as hierarchical policy decision-making, the objectives diverge: FamO2O focuses on balancing policy improvement and constraints per state, while HRL seeks to simplify complex tasks via goals or options.

B Training Process of FamO2O

Here we provide the pseudo-codes of FamO2O (see Algorithm 1) to demonstrate its training process.

Algorithm 1 FamO2O Algorithm

Require: replay buffer D , offline dataset D offline, offline and online training steps N off , N on

2: Initialize D with D offline

1: Initialize π 0 b , π 0 u , Q 0 , V 0

3: for k = 0 → N off -1 do

4: Sample a mini-batch M k from D

5: Assign a random balance coefficient for each sample in M k , denoting the balance coefficient set as B M k

6: Update π k u to π k +1 u with M k , B M k by Equation (9)

8: Update Q k , V k to Q k +1 , V k +1 respectively with M k according to the base algorithm

7: Update π k b to π k +1 b with M k by Equation (10)

9: end for

10: for k = N off → N on -1 do

12: Sample a mini-batch M k from D

11: Collect samples with π k u , π k b and add samples to D

13: π k b computes a balance coefficient for each sample in M k . Denote the balance coefficient set as B M k

14: Update π k u to π k +1 u with M k , B M k by Equation (9)

16: Update Q k , V k to Q k +1 , V k +1 respectively with M k according to the base algorithm

15: Update π k b to π k +1 b with M k by Equation (10)

17: end for

▷ offline pre-training phase

▷ online fine-tuning phase

C Theoretical Proofs

C.1 Proof of Proposition 3.4

Lemma C.1. Equation (5) is equivalent to:

Proof. We aim to prove Equation (5) ⇒ Equations (12) to (13) in our manuscript through a contradiction approach. Let's assume that Equation (5) ⇏ Equations (12) to (13), signifying the following:

Condition 1 According to Equation (5), ∫ s ∈S d π β ( s ) D KL ( π ( ·| s ) ∥ π β ( ·| s ))d s ≤ ϵ ;

Condition 2 The converse proposition of Equations (12) to (13) is: ∀ ϵ s ≥ 0 ( s ∈ S ) which satisfy ∫ s ∈S d π β ( s ) ϵ s d s = ϵ , there exists s ∈ S , D KL ( π ( ·| s ) ∥ π β ( ·| s )) > ϵ s .

Next, we form a special set { ϵ s , s ∈ S} . Selecting an arbitrary s 0 ∈ S , the set fulfills:

̸

For all s = s 0 , ϵ s = D KL ( π ( ·| s ) ∥ π β ( ·| s )) ,

̸

It can be clearly seen that the set adheres to ∫ s ∈S d π β ( s ) ϵ s d s = ϵ . Based on Condition 2, we can deduce that D KL ( π ( ·| s 0 ) ∥ π β ( ·| s 0 )) > ϵ s 0 . Hence,

̸

This stands in contradiction to condition 1, where ∫ s ∈S d π β ( s ) D KL ( π ( ·| s ) ∥ π β ( ·| s ))d s ≤ ϵ . Thus, the assumption that Equation (5) ⇏ Equations (12) to (13) is proven false, which confirms that Equation (5) ⇒ Equations (12) to (13).

Denote the feasible region of the problem in Definition 3.2 as F k ( ϵ ) , and the feasible region of the problem in Definition 3.1 as F k ( { ϵ s , s ∈ S} ) . According to Lemma C.1, we can infer that:

Considering that the problems in Definition 3.1 and Definition 3.2 shares the same objective function, we have:

where π k +1 ∗ [ ϵ ] is the optimal solution corresponding to the optimal value J k ∗ [ ϵ ] for the problem in Definition 3.2.

Based on Equation (15), we can derive that

̸

Here J k π [ { ϵ s , s ∈ S} ] is the objective function value in 3.1 with the solution π and KL constraints { ϵ s , s ∈ S} .

Under the KL constraints { ϵ s , s ∈ S} , the optimal value J k ∗ [ { ϵ s , s ∈ S} ] of the problem in Definition 3.1 is no less than J k π [ { ϵ s , s ∈ S} ] . Therefore

Proposition 3.4 is proven. Q.E.D.

C.2 Proof of Proposition 3.5

Because the state space S possibly contains an infinite number of states, the optimization problem in Definition 3.1 is probably a problem with infinite constraints, which is not easy to deal with. Therefore, we first start with a simplified version of the optimization problem, where the state space contains only a finite number of states, and further extend the conclusion on the simplified optimization problem to the original one in Definition 3.1.

The simplified optimization problem is stated as follows:

Definition C.2 (Simplified optimization problem with point-wise constraints) . The simplified optimization problem with point-wise KL constraints on a state space with only a finite number of states S = { s 1 , s 2 , · · · , s m } is defined as

Here, π k ( k ∈ N ) denotes the policy at the k -th iteration, π β is a behavior policy representing the action selection way in the offline dataset or current replay buffer, and d π β ( s ) is the state distribution of π β . we utilize the optimal solution for Definition C.2 as π k +1 .

For this simplified optimization problem, we have a lemma below, whose derivation is related to AWR [37] and AWAC [35]:

Lemma C.3. Consider the optimization problem in Definition C.2. The optimal solution of π k +1 , denoted as π k +1 ∗ , satisfies that ∀ s ∈ { s 1 , s 2 , · · · , s m } , a ∈ A ,

Proof. The Lagrangian function L ( π, λ, µ ) is given by:

where λ = ( λ 1 , λ 2 , · · · , λ m ) , µ = ( µ 1 , µ 2 , · · · , µ m ) are Lagrange multipliers. According to the KKT necessary conditions [20], the optimal solution satisfies that ∀ a ∈ A , i ∈ { 1 , 2 , · · · , m } ,

Therefore, the optimal policy π k +1 ∗ is:

where Z s i = exp ( λ i µ i +1 ) , β s i = d π β ( s i ) µ i are state-dependent factors.

Furthermore, due to the constraint in Equation (20), Z s i also equals to ∫ a ∈A π β ( a | s i ) exp ( β s i ( Q π k ( s i , a ) -V π k ( s i )) ) d a .

Now we consider extending the conclusion in Lemma C.3 to the more complex optimization problem in Definition 3.1, where the state space S = [ s min , s max ] l is a compact set and contains an infinite number of states. Without loss of generality, suppose that S = [0 , 1] . The derivation below is easy to transfer to S = [ s min , s max ] l with small modifications.

Specifically, we construct a sequence sets { B i , i = 0 , 1 , 2 , · · · } with each element B i = { j/ 2 i , j = 0 , 1 , · · · , 2 i } . We can observe that { B i , i = 0 , 1 , 2 , · · · } satisfies:

∀ i ∈ N + , | B i | = 2 i +1 < ∞ , i.e. , all B i are finite sets, and therefore all B i are compact ; (27)

The qualities in Equations (25) to (28), together with the assumption in Proposition 3.5 that the feasible space constrained by Equations (2) to (4) is not empty for every s ∈ S , meets the prerequisites of the discretization method proposed by [41]. Set π k +1 ∗ [ A ] as the optimal solution of the following optimization problem:

where A is a subset of S . According to the Theorem 2.1 in [41],

For any B i , because | B i | < ∞ , Equation (21) holds for π k +1 ∗ [ B i ] . By combining Equation (21) with Equation (32), we succeed in proving Equation (7) in Proposition 3.5.

Furthermore, we utilize a parameterized policy π ϕ to approximate the optimal policy π k +1 ∗ in Equation (7), i.e. ,

In practice, Z s in Equation (34) is challenging to calculate. We follow the derivation of AWR [37] and AWAC [35], and omit the term 1 Z s . Therefore Equation (34) can be rewritten as

Thus Equation (8) in Proposition 3.5 is also proven. Q.E.D.

D FamO2O's Extension to Non-AWR Algorithms

In this section, we discuss extending FamO2O to non-AWR algorithms, specifically considering the Conservative Q-Learning (CQL) [27].

CQL's policy update rule is given in Equation (37):

where π k and Q k represent the policy and Q function at iteration k respectively, and α is a hyperparamter. CQL already incorporates a conservative estimation for out-of-distribution data in its Q function update rule, thus lacking a policy constraint for policy updates in Equation (37). However, Equation (37) presents a balance between exploitation (the blue part) and exploration (the red part), determined by α . We thus deploy FamO2O to adjust this balance adaptively per state.

Under FamO2O, the policy update rule morphs to Equation (38):

where π u is the universal model taking a state and balance coefficient as input to yield an action, and π b is the balance model mapping a state to a balance coefficient. The changes, denoted in red and blue, depict FamO2O's additional input and the substitution of the adaptive balance coefficient α s for the pre-defined one α , respectively.

Additionally, the balance model update rule (Equation (40)) aims to maximize the corresponding Q value by finding an optimal α s , akin to Equation (10).

As for Q function updates, by consolidating balance model π b : S ↦→ B and π u : S × B ↦→ A into a standard policy π u ( · , π b ( · )) : S ↦→ A , these updates remain identical to standard CQL.

E Implementation Details

The following section outlines the specifics of baseline (Appendix E.1) and FamO2O (Appendix E.2) implementations.

E.1 Baseline Implementation

We commence by detailing our baseline implementation. For IQL [24], AWAC [35], and Balanced Replay [28], which have been tested in an offline-to-online RL setting in their respective papers, we utilize the official codes 34 .

For CQL [27] and TD3+BC [10], we introduce an online fine-tuning process based on the recommended offline RL codes by the authors 56 . Specifically, our online fine-tuning for CQL and TD3+BC mirrors the implementation in rlkit, that is, appending samples acquired during online interactions to the offline dataset while maintaining the same training objective during offline pre-training and online fine-tuning.

For SAC [13] and BC, we leverage CQL's codebase, which has already incorporated these two algorithms.

Regarding training steps, we adhere to IQL's experiment setting for a fair comparison across all methods in Figure 5, which involves 10 6 gradient steps each for offline pre-training and online fine-tuning. It should be noted that due to AWAC's overfitting issue, as discussed by [24], we limit AWAC's offline pre-training to 2 . 5 × 10 4 gradient steps, as recommended. For non-AWR algorithms comparison, namely CQL [27], BR [28], and CQL+FamO2O, we employ 2 × 10 6 offline gradient steps, as suggested in CQL's implementation, and 1 × 10 6 online gradient steps, aligning with IQL.

E.2 Implementation of FamO2O

Next, we provide FamO2O's implementation details. We implement our FamO2O on the official codes of AWAC [35], IQL [24], and CQL [27] discussed above. Our major modification is (i) adding a balance model and (ii) making the original policy conditioned on the balance coefficient (called 'universal model" in our paper). Except for these two modifications, we do not change any training procedures and hyperparameters in the original codes.

For the first modification, we employ the stochastic model used by SAC in rlkit to construct our balance model and implement the training process of the balance model described in Section 4.2. For the second modification, we change the input dimension of the original policy from the state dimension l to l + l b , where l b is the dimension of balance coefficient encodings. We encode balance coefficients in the same way as the positional encoding proposed by [47]. This encoding design expands the dimension of the balance coefficients and avoids the balance coefficients being neglected by the universal model.

For IQL with FamO2O, adaptive balance coefficients are no less than 1 . 0 (Locomotion) / 8 . 0 (AntMaze) and no larger than 5 . 0 (Locomotion) / 14 . 0 (AntMaze). For AWAC with FamO2O, the adaptive balance coefficients are no less than 2 . 0 (Locomotion) / 9 . 0 (AntMaze) and no larger than 3 . 0 (Locomotion) / 11 . 0 (AntMaze). For CQL with FamO2O, the adaptive balance coefficients are no less than 0 . 5 and no larger than 1 . 5 . To ensure a fair comparison, in Figure 9, the fixed balance coefficient ranges cover the adaptive ranges discussed above.

As for training steps, for IQL with FamO2O, we use 10 6 gradient steps in the offline pre-training phase and 10 6 gradient steps in the online fine-tuning phase. For AWAC with FamO2O, we use 7 . 5 × 10 4 gradient steps in the offline pre-training phase and 2 × 10 5 (Locomotion) / 1 × 10 6 (AntMaze) gradient steps in the online fine-tuning phase. For CQL with FamO2O, we use 2 × 10 6 gradient steps in the offline pre-training phase and 1 × 10 6 in the online fine-tuning phase.

For more implementation details, please refer to Tables 4 to 6.

F More Experimental Results

F.1 Sensitivity Analysis on Different Ranges of Balance Coefficients

To find out whether FamO2O's performance is sensitive to the range of balance coefficients, we implement FamO2O on IQL [24] with 3 different balance coefficient ranges ( i.e. , [1 , 5] , [1 , 4] , and [2 , 4] ) on 3 datasets of different qualities ( i.e. , medium-replay, medium, and medium-expert). The results can be found in Figure 12. We further implement FamO2O on IQL with 4 different balance coefficient ranges ( i.e. , [6 , 12] , [8 , 12] , [8 , 14] , and [9 , 11] ) on 3 datasets of different maze sizes ( i.e. , umaze, medium, large). The results can be viewed in Figure 13. It can be observed that on various datasets of different qualities or maze sizes, the performance of FamO2O does not vary significantly with different ranges of the balance coefficients. This indicates that within a reasonable scope, FamO2O is insensitive to the range of coefficient balances.

medium-replay

medium

medium-expert

medium

large

umaze

F.2 Comparing FamO2O and Balance Coefficient Annealing

To delve deeper into the efficacy of FamO2O's balance model more thoroughly, we set up a comparison against the balance coefficient annealing. The balance coefficient annealing is a process that gradually augments the balance coefficient value as the fine-tuning stage proceeds. This method is intuitively reasonable, as the initial conservativeness due to the balance coefficient annealing helps to curb extrapolation errors at the commencement of finetuning, and as the process continues, this conservativeness is gradually eased to facilitate policy improvement.

We incorporate balance coefficient annealing into IQL [24], named IQL+annealing, providing a basis for comparison with IQL+FamO2O. The comparison results are presented in Table 3. The results show that FamO2O exceeds the performance of balance coefficient annealing across most of the D4RL Locomotion datasets [8].

In addition, we calculate the 95% CIs for both IQL+FamO2O and IQL+annealing in Table 3, following the suggestion by rliable [3]. The lower CI for IQL+FamO2O is mush higher than the upper CI for IQL+annealing, signifying a statistically significant improvement of FamO2O over balance coefficient annealing. These findings establish that in comparison to annealing, FamO2O showcases a more sophisticated and effective approach in adaptively modulating the balance between improvement and constraint.

G Potential Negative Societal Impacts

This work proposes a plug-in framework, named FamO2O, which can be implemented on top of the existing offline-to-online RL algorithms to further improve policy performance. Therefore, the potential negative societal impacts of our method are similar to those of the offline-to-online RL. Generally, FamO2O would greatly improve policy performance. However, as with most RL algorithms, FamO2O cannot guarantee to take safe and effective actions in all kinds of scenarios. We advocate that the users of FamO2O should be aware of the potential consequences and utilize FamO2O safely, especially in online environments such as self-driving, robotics, and healthcare.

Peer reviews:
comments: SUMMARY:
Authors propose a framework for offline-to-online tuning of offline RL algorithms tunning. The idea is to train an additional network which desides helps to keep improvement-constraint balance during finetuning.

STRENGTHS:
Approach improves all of the checked algorithms performance and can be applied to different offline RL algorithms. Good range of benchmarking tasks and  algorithms.

WEAKNESSES:
-

QUESTIONS:
How does the modification affect the compute time required to  train algorithms?

LIMITATIONS:
Adding another network requires additional choice of  hyperparameters which might be hard to find
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper approaches offline-to-online RL from the intuition that at a particular state, if the dataset already contains good actions, then the subsequent online tuning should be more conservative to retain the good actions in the dataset; but if the dataset's actions is poor, then more radical policy improvement is needed.\nTo this end, this paper introduces a framework, Family Offline-to-Online RL (FamO2O), which aims at a state-adaptive improvement-constraint balance for each state.\nSpecifically, from the collected dataset, the authors train a diverse policy family ranging from conservative to radical and use the environmental feedback to select an appropriate policy from this family at each state.\nPractically, this is achieved by a universal model, which determines the degree of policy conservatism; and a balance model, which learns the balance coefficients at each state.\nExperimental results show that FamO2O improves on various offline-to-online RL methods and achieves competitive performance.\n\nSTRENGTHS:\n1. The paper is well-written and generally easy-to-follow.\n2. The proposed method is well theoretically justified.\n3. The empirical discussion and ablation study are thorough.\n\nWEAKNESSES:\n1. The proposed method seems require abundant diverse data, which may not be feasible on harder settings, e.g., the Adroit domain in the D4RL benchmark where the data is limited and the data distribution is narrow and lack of diversity. \n2. Maybe I misunderstand something. I think the purpose of the $\\pi_b(s)$ model is to find the $\\beta_s$ that corresponds to the optimal sequence of constraints $\\\\{\\epsilon_s, s \\in \\mathcal{S}\\\\}$  in Eqn. (6), which relates to $\\beta_s$ by the Language multipliers $\\mu(s)$ (Line 537, Appendix C.2). In this regard, the paper's story of balancing policy improvement and constraint can be slightly over-complicated and somewhat confusing.\n3. The proposed method is in spirit similar to decision-transformer-style methods, and is therefore less a surprise.\n\nQUESTIONS:\n1. Could you explain more on how the updating rule Eqn. (10) train the balance model $\\pi_b$ to control policy improvement and constraint? Why does maximizing the $Q^k$ values achieve this balance? \n2. The significant variation in data quality across different states depicted by Figure 2 may come from the nature of the medium-expert datasets, which by construction contains medium and expert trajectories. Does this variation in data quality exist in other type of datasets, say expert, medium, or medium replay? More generally, is this phenomenon of significant data-quality variation ubiquitous or specific?\n3. Could you explain more on the benefit of \"state-adaptive improvement-constraint balances\"? Even if we do not make such a balance and simply just online finetune an offline-pretrained policy, the environmental feedback should be able to tell us which actions are good and should be retained and which actions are poor and should be improved. \n4. Could you explain more why you consider the term $\\log \\pi (a|s)$ in Eqn. (1) as a policy constraint? What is the target of this constraint, especially when $(s,a)$ is an online interaction samples? \n5. Could you explain more on how you get Eqn. (12) and (13) from Eqn. (5)? It would be better if you can expand Appendix C.1 to include more details and explanations. The current version is a bit hard to follow.\n6. nit: in Eqn. (6), are you missing a \"$\\forall \\epsilon > 0$\" before $\\exists \\\\{...\\\\}$?\n7. [L161] Could you explain more on the cooperation between $\\pi_u$ and $\\pi_b$? Why couldn't we still randomly sample $\\beta_s$ during online fine-tuning?\n8. [L159] How many $\\beta_s$ vectors are required to learn the universal model? How does this number scale with the number of states in the dataset? And how to select/design the balance coefficient space $\\mathcal{B}$?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm to perform offline-to-online reinforcement learning. The core idea is to consider a state-adaptive balance parameter, which aims to encourage imitation of dataset behavior only if the corresponding advantages / values are high, while prior works have mostly assumed fixed balances. The authors provide detailed experimental evaluation of their approach and show superiority over relevant baselines.\n\nSTRENGTHS:\nI think offline-to-online RL is a very relevant and promising direction for research since if reliable it would enable much more practical applications of RL in real-world tasks. The core idea of the paper, training a collection of policies that adapts on a per state level, appears logical and powerful (however please see [2,3,4], which I think put forward similar ideas) and will in the future probably significantly influence the way offline-to-online RL is performed and thought about. The paper is overall very well written and understandable & offers a strong statistical analysis of the proposed algorithms performance compared to relevant baselines.\n\nWEAKNESSES:\nThe term data quality is frequently used but not really introduced (could mean accuracy / truth of the information in the data, but I think it means something like return).\n\nNot all D4RL locomotion datasets were used - its unclear whether that's an issue since the selection is not justified.\n\nThe considered baselines appear relevant (TD3+BC & CQL less so since its already shown in the AWAC paper that using the same conservative formulations in the online setting does not work well), however I think the closest existing methods are missing: E.g. Confidence-conditioned value functions [1] automatically adjusts its policy confidence level (as far as I understand on a per state level since every state goes into the considered history). Also methods like RvS [4], which condition on return, could easily be extended to this setting (i.e. always try to maximise conditioned return during online data collection). Generally I have the feeling that some very relevant related work regarding offline adaptive policies was not considered, i.e. also:\n[2] User-Interactive Offline RL (which considers policies conditioned on a balance between conservative and very liberal)\n[3] offline policies should be trained to be adaptive (which adapts policies based on the online collected history)\nI'm not sure the state-adaptive balance between conservative and radical idea for offline-to-online is thus entirely novel.\n\nI see some issues regarding clarity that could easily be fixed (see questions)\n\nQUESTIONS:\nCould you please define how you use the term data quality (I'm assuming it has something to do with the data's return, but I'm not sure)?\n\nPlease elaborate why the random datasets in the D4RL datasets were not considered.\n\nThere exist offline-to-online methods that already consider something like state-adaptive policy changes, like confidence-conditioned value functions [1], as well as other adaptive offline-to-online policy concepts [2,3] that could easily be extended to your case - why were they not considered? (I realise that some are rather recent, but I think they deserve at least brief discussion)\n\nI do not understand section 6.1 / figure 8 at all - what is meant by guidance? what is meant by high / low quality data? Starting positions are denoted as triangles, suggesting a moving direction but I don't think that's intended. The text says high balance coefficients are found where data quality is also high (quality=reward / return?) & that higher quality data is found at the lower crossing point in the 5th row, however from the color encoding it seems that the balance coefficient there is actually the lowest...\n\nI think the ratio between offline and online training steps as well as collected interaction steps is a crucial parameter for reproducibility / future comparisons, however that information is only conveyed in the figure 10 axis & the appendix - I think it should be explicitly stated somewhere in the experimental section text.\n\nWhen I look at figure 10, it seems that after all online training steps, the attained performance is almost identical to that at the end of offline training - isn't that a very disappointing result and couldn't you just throw away the whole online part altogether then? Since FamO2O outperforms prior offline-to-online baselines, does that mean that these baselines only get worse with online training?\n\n[1] Hong, J., Kumar, A., & Levine, S. (2022). Confidence-Conditioned Value Functions for Offline Reinforcement Learning. ICLR 2023\n[2] Swazinna, P., Udluft, S., & Runkler, T. (2022). User-Interactive Offline Reinforcement Learning. ICLR 2023\n[3] Ghosh, D., Ajay, A., Agrawal, P., & Levine, S. (2022, June). Offline rl policies should be trained to be adaptive. ICML 2022\n[4] Emmons, S., Eysenbach, B., Kostrikov, I., & Levine, S. (2021). RvS: What is Essential for Offline RL via Supervised Learning?. arXiv preprint arXiv:2112.10751.\n\nLIMITATIONS:\nno, limitations are not discussed - perhaps looking at the not improved performance after online fine-tuning in figure 10, as well as the often only small improvements over standard IQL, a brief discussion would be good.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method to mitigate the distribution shift problem in the offline to online RL problem. The paper states the intuition that the policy should behave differently on states with different values, that is, the policy should be more conservative on high return states and exploratory on the low return states. With this intuition, the paper proposed to train a family of policies in the offline to online setting, specifically, train another \"policy\" to parameterize the rollout policy via the state-adaptive balance coefficient. The experiments show that FamO2O outperforms previous O2O baselines, and show via a toy experiment that FamO2O indeed leans the state-wise adaptivity, and various ablations show the importance of each design choices.\n\nSTRENGTHS:\n1. The experiment result is solid as it evaluates on extensive D4RL dataset with different data quality and both locomotion and maze tasks. \n\n2. The discussion sections validate the algorithm design choices, and section 6.1 verify that the algorithm indeed learns a state-wise adaptive policy, which support the intuition and motivation of the algorithm.\n\nWEAKNESSES:\n1. The exploitation vs. exploration intuition is not brand new in the offline to online setting, there is also some work with similar intuition [1]. I believe proper comparison is required given the similarity of the intuition, although I believe training a family of policies (or conditionally parametrized the policy) seems like a slight generalization.\n\n2. It is confusing that in the universal model training (eq. (9)), the exponential of the advantages are weighted by the balance parameter $\\beta$, but when training the balance model, the Q-function (as in the loss) is not weighted by the balance parameter. There seems to be some consistency. I can tell that the unweighted objective (the current form of eq (10)) would be more computationally friendly, but theoretically, it seems more natural to optimize over a weighted version where eq. 10 is also weighted by $\\beta, which sample from the balance model.\n\n3. The action distance in eq. (11) may not be the best metric to measure the discrepancy between a policy and a trajectory. For example, if the policy that induces the trajectory only takes $a_1$ in $s$, and the evaluated policy takes $a_1$ with $p=0.51$ and $a_2$ with $p=0.49$, which may induce a very negative reward, or cause great trajectory derailment (which is not recorded in the offline trajectory), the proposed metric will still be 0 but in reality the evaluated policy is not that close to the trajectory.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}