Title: Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

offline-to-online
Abstract: Offline-to-online (O2O) reinforcement learning (RL) provides an effective means of leveraging an offline pre-trained policy as initialization to improve performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. To deal with this problem, we disclose that there are evaluation and improvement mismatches between the offline dataset and the online environment, which hinders the direct application of pre-trained policies to online fine-tuning. In this paper, we propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method. Before online fine-tuning, we re-evaluate the pessimistic critic trained on the offline dataset in an optimistic way and then calibrate the misaligned critic with the reliable offline actor to avoid erroneous update. After obtaining an optimistic and and aligned critic, we perform constrained fine-tuning to combat distribution shift during online learning. We show empirically that the proposed method can achieve stable and efficient performance improvement on multiple simulated tasks when compared to the state-of-the-art methods.

Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL

Abstract

Offline-to-online (O2O) reinforcement learning (RL) provides an effective means of leveraging an offline pre-trained policy as initialization to improve performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. To deal with this problem, we disclose that there are evaluation and improvement mismatches between the offline dataset and the online environment, which hinders the direct application of pre-trained policies to online fine-tuning. In this paper, we propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method. Before online fine-tuning, we re-evaluate the pessimistic critic trained on the offline dataset in an optimistic way and then calibrate the misaligned critic with the reliable offline actor to avoid erroneous update. After obtaining an optimistic and and aligned critic, we perform constrained fine-tuning to combat distribution shift during online learning. We show empirically that the proposed method can achieve stable and efficient performance improvement on multiple simulated tasks when compared to the state-of-the-art methods. The implementation is available at https://github.com/QinwenLuo/OCR-CFT.

1 Introduction

Offline reinforcement learning (RL) aims to learn a policy from a fixed dataset without additional interactions with the environment. This characteristic makes it particularly promising for critical applications such as healthcare decision-making [39], human-AI coordination [14] and autonomous driving [8]. Generally, the performance of the learned policy relies on the quality of the dataset. Given that the offline data is limited, fine-tuning the policy through interactions with the environment is still necessary to achieve favorable performance. Consequently, offline-to-online (O2O) RL tends to achieve faster performance improvements based on better initializations.

To effectively fine-tune offline policies, O2O methods are typically designed based on specific offline RL algorithms. Existing methods can be roughly divided into two groups according to the base offline methods they use. The first group relies on policy constraint methods. These approaches aim to improve online performance by either adaptively adjusting constraints [5, 52, 43] or directly applying offline algorithms to online fine-tuning [32, 20]. Unfortunately, these methods often suffer from inefficient performance improvement due to restricted action exploration caused by policy constraints. The second group builds on value regularization methods. These methods aim to prevent excessively

low Q-values resulting from pessimistic evaluations, such as those in CQL [21]. The goal is to enhance the generalization of the value function and mitigate potential performance declines [27, 33]. Some other methods adopt ensemble Q-learning [51, 23, 30, 19] address these issues. However, these methods often face high computational costs due to the need to train multiple Q-networks.

Considering that the aforementioned methods develop online fine-tuning algorithms based on specific offline methods, they often struggle to be applied to other offline methods. To establish a general O2O framework, it is essential to address the core issues associated with transitioning from offline to online environments. Inspired of the recent work [48], which highlights the misalignment between the actor and the critic in an explicit policy constraint method, we identify two mismatches in general O2O RL: evaluation mismatches and improvement mismatches. Evaluation mismatches primarily occur in value regularization methods. These refer to the differences in policy evaluation methods between online and offline settings, which cause severe fluctuations in Q-value estimation during the initial stages of fine-tuning. Improvement mismatches, on the other hand, are prevalent in policy constraint methods. They are often caused by differences in objectives for updating the policy, leading to a misalignment between the probabilities of actions and their Q-values. Thanks to another recent work by Xu et al [47], which connects value regularization and policy constraint methods, we bridge these two types of mismatches within a unified framework for general RL-based offline algorithms.

In this paper, we propose a general O2O framework designed to address both evaluation and improvement mismatches simultaneously, aiming for stable and favorable fine-tuning performance from any offline method to representative online methods. To address the evaluation mismatch in value regularization methods, we propose re-evaluating the offline policy in an optimistic manner using an off-policy evaluation method. This approach allows us to obtain optimistic Q-value estimates, preventing the dramatic fluctuations in Q-values that could potentially cause the policy to collapse. Although the re-evaluated critic can estimate Q-values optimistically, it suffers from the misalignment with the offline policy, causing the improvement mismatch. To handle the improvement mismatch in the re-evaluated critics and policy constraint methods, we introduce value alignment to calibrate the critic so that it aligns with the probabilities predicted by the policy. Our approach involves using the Q-value of the most likely action as an anchor and then calibrating other Q-values by either exploiting the correlation between Q-values of different state-action pairs or modeling Q-values as a Gaussian distribution. Finally, we propose a constrained fine-tuning framework to guide the policy update by adding a regularization term, with the target of mitigating the negative impact of data shift. Extensive experimental results on multiple benchmark environments validate that the proposed methods can achieve better or comparable performance when compared to state-of-the-art methods.

Our contributions can be summarized as follows:

We systemically study that there exist evaluation and improvement mismatches for offline RL methods from the perspective of online RL. We show that resolving these two types of mismatches is essential for achieving general O2O RL.

We develop two techniques to address these mismatches. Policy re-evaluation aims to achieve optimistic Q-value estimates, preventing instability in Q-value estimation. Value alignment calibrates the critic to align with the policy, ensuring consistency between action probabilities and their corresponding Q-values.

We introduce a constrained fine-tuning framework that incorporates a regularization term into the policy objective, combating the inevitable distribution shift and ensuring stable and optimal performance when fine-tuning the policy in online environments.

2 Preliminaries

In this section, we introduce necessary preliminaries about RL, including Markov decision process, and three target online RL methods.

Markov Decision Process (MDP) A Markov Decision Process M is defined by the tuple ( S,A,R,P, µ , γ ) [38], where S is the state space, A is the action space, P : S × A → ∆( S ) is the transition function, R : S × A → R is the reward function, µ is the initial state distribution, and γ is a discount factor. The goal is to learn a policy that maximizes the expected return as

Online RL The three most commonly used online RL algorithms are SAC [17], TD3 [11], and PPO [37]. Below, we will introduce these three methods one by one.

SAC first introduces entropy maximization into the RL scenarios and updates the actor and the critic by minimizing the following objectives:

where α > 0 , ¯ µ i are the parameters of the target Q network, and ¯ H is the target entropy.

TD3 models a deterministic policy and uses tricks, including clipped double Q-learning and policy smoothing, to address the issue of function approximation error. The deterministic policy gradient used to update policy is defined as

The objective function of updating the critic is defined as

Here, y ( r, s ′ ) = r + γ min i =1 , 2 Q ¯ µ i ( s ′ , ˜ a ) , where ˜ a = π ¯ θ ( s ′ ) + ϵ and ϵ ∼ clip ( N (0 , σ ) , -c, c ) .

PPO provides a simple implementation for TRPO [35], which updates the actor and the critic by minimizing the following objective functions, respectively,

where r ( θ ) = π θ /π θ k and the advantage A π θ k is computed by GAE [36].

It is noteworthy that previous works mostly adopted SAC and TD3 for online fine-tuning as they are off-policy methods. In our work, given the widespread use of PPO in online RL, we also employ it for fine-tuning though it is an on-policy method.

3 Evaluation and Improvement Mismatches

In this section, we focus on the differences between online and offline RL and study how these differences impact the performance of online fine-tuning. In general, we summarize these differences as two types of mismatches, evaluation mismatch and improvement mismatch . The former arises from the changes in policy evaluation functions during the transition from offline to online environments; while the latter represents the inconsistency in the objectives for policy updates between offline and online RL. Most offline RL methods suffer from one or both of these issues, which underscores the importance of addressing these mismatches to achieve stable and effective online fine-tuning.

Evaluation mismatch often occurs in the value regularization methods. For example, in CQL [21], a representative offline method, the policy evaluation function transitions from a pessimistic estimation inherent to offline learning to a more optimistic estimation during online training. This shift frequently results in a sharp increase in Q-values at the beginning of online fine-tuning, which can hinder stable performance improvements. To address this problem, several attempts have been made to mitigate the excessive underestimation of out-of-the-distribution (OOD) actions during offline learning [33, 27]

or to initialize a pessimistic Q-ensemble to maintain pessimism during online fine-tuning [23]. Unfortunately, these approaches are predominantly tailored to the transition from CQL to SAC, limiting their applicability to other offline algorithms.

Improvement mismatch is commonly found in policy constraint methods. Typical examples include TD3+BC [10] and AWAC [32], where the objective of actor updates differs significantly from typical online methods. In these models, updates to the actor are not solely reliant on the critic's evaluation. Consequently, actions that have high Q-values may not automatically translate to high probabilities of being selected, and vice versa. This divergence often misguides the update of policy at the beginning stage of online fine-tuning, resulting in unfavourable performance.

Besides, One-step RL and non-iterative methods, e.g. , IQL, exhibit both evaluation and improvement mismatches. During the policy evaluation stage, these methods estimate the Q-values based on the behavior policy or an unknown policy [18] instead of the target policy as in online methods; during the improvement stage, because they impose additional constraints on policy updates, they also encounter the same issue as discussed above. This can lead to discrepancies in both the assessment of action values and the subsequent policy optimization, making it hard to achieve effective policy improvement.

Thanks to the recent work [47], which presents a unified framework for understanding offline RL, we bridge these two mismatches for general RL-based offline algorithms. Formally, the offline RL problem can be defined by the behavior-regularized MDP problem [47, 13] via maximizing the following objective:

where f ( · ) is a regularization function and µ is the behavior policy.

From Eq. (1) and Eq. (9), we observe a significant divergence in the relation between the actor and the critic in online versus offline scenarios. Unlike in online cases where the policy update is solely dependent on the Q-function, in offline scenarios, it also critically depends on the data distribution, as indicated by the regularization term in Eq. (9). This distinction highlights why offline methods often face evaluation and improvement mismatches when applied to online fine-tuning. Using offline actor and critic trained by Eq. (9) for initialization in online fine-tuning introduces both types of mismatches, resulting in unstable and inefficient updates.

4 Method

In this section, we introduce our proposed O2O method for handling the mismatches discussed in Section 3 and the distribution shift problem. To address the pessimistic or unreliable evaluation, such as in CQL and IQL, we develop a policy re-evaluation technique. This technique optimistically re-evaluates the well-trained offline policy using an off-policy evaluation method. Although this re-evaluation helps the critic achieve more optimistic Q-value estimates, unavoidable factors such as function approximation errors and partial data coverage can still lead to a misalignment between the critic and the offline policy. This misalignment means that the action with the highest probability predicted by the policy does not necessarily have the highest Q-value, leading to what we have termed improvement mismatch.

As discussed in in Section 3, both the value regularization (after re-evaluation) and policy constraint methods exhibit improvement mismatch, though the reasons for the mismatch differ between these two approaches. To address this issue, we propose value alignment, which aims to align the critic's estimates with the policy's action probabilities, effectively tackling the improvement mismatch in both types of methods. Finally, to deal with the inevitable distribution shift between offline and online environments, we develop a constrained fine-tuning framework. This framework ensures that the policy consistently updates in the optimal direction by incorporating a regularization term into the policy objective.

We propose methods for various representative online RL algorithms within a unified framework, including SAC [17], TD3 [11], and PPO [37]. These algorithms represent the mainstream approaches in online RL, and are targeted respectively for the off-policy approach with stochastic policies, the off-policy approach with deterministic policies, and the on-policy approach.

4.1 Policy Re-evaluation

The critic trained on an offline dataset typically maintains pessimistic estimates of Q-values. When using online evaluation methods to fine-tune this critic without any value regularization, the Qvalues can experience a dramatic jump, especially for OOD actions, leading to inaccurate Q-value estimations. To mitigate this problem, we propose re-evaluating the offline policy to acquire a new critic by employing an off-policy evaluation (OPE) method. The goal is to enable the critic to have optimistic estimates of Q-values that more closely approximate the true values.

However, directly applying OPE methods for policy evaluation on offline datasets often leads to large extrapolation errors, as discussed in the previous work [12]. These errors arise due to absent data and training mismatches. Thanks to the pessimism in offline RL, it is reasonable to assume that a well-trained policy is close enough to the behavior policy or even captures the support set of the behavior policy. A common assumption is single-policy concentrability [45, 34], which demonstrates how concentrated a learned policy is within the given dataset and can be defined as follows.

Assumption 4.1. ( π θ -concentrability [45]) The behavior policy µ and learned policy π θ satisfy

where d π θ ( s, a ) is the occupancy measure of π θ and C is a constant. Single-policy concentrability measures the degree to which the state-action distribution induced by the learned policy is covered by the dataset used for training. By ensuring that the learned policy does not deviate significantly from the behavior policy, the extrapolation error can be greatly reduced [22, 29]. When using a representative OPE method called fitted Q-evaluation (FQE), based on Assumption 4.1 and Theorem 4.2 in [22], the upper bound of extrapolation error can be obtained.

Corollary 4.2. Under Assumption 4.1, by denoting Q-value function class as F , for δ ∈ (0 , 1) , after K iterations of FQE on the dataset D , with probability 1 -δ , we have:

where d π F is inherent Bellman evaluation error (Definition 4.1 in [22]), and ¯ V is the maximum of V , which can be bounded by R max / (1 -γ ) .

With a powerful neural network and sufficient data, the inherent Bellman evaluation error could be tiny. Accordingly, with a large training step K , the error will be bounded by an acceptable value. This implies that, given sufficient data, one can achieve a critic with optimistic property and minor extrapolation error through policy re-evaluation. In practical implementation, for off-policy methods SAC and TD3, we can directly use Eq. (3) and Eq. (6) to re-evaluate the offline policy. For the on-policy method PPO, we train a critic by fitting the returns of offline trajectories. Considering that the critic can only approximate V µ ( s ) rather than the true value function, we propose a regularization term in Section 4.2, which ensures the critic's estimates are reliable and conducive to effective policy improvement.

4.2 Value Alignment

Although the critic after policy re-evaluation possesses the optimistic property needed in the online environment, it often does not align well with the offline policy due to factors such as function approximation errors, generalization errors of neural networks, and partial data coverage. Moreover, as discussed in Section 3, policy constraint offline methods also suffer from misalignment between the critic and policy. The misalignment means that the



































































action with the highest Q-value does not necessarily have the highest probability, often leading to misleading updates of the policy. To verify this observation, we trained the policy using SAC and TD3

with three different critics: the fixed re-evaluated critic, the iterative re-evaluated critic (which updates with the policy), and our aligned critic. From Figure 1, the performance of re-evaluated critics sharply declines at the initial stage and does not recover in the subsequent training; while our aligned critic achieves stable and favorable performance. These results indicates that the misalignment between the re-evaluated critic and the offline policy can make it difficult for the policy to optimize in a correct direction, leading to an irreversible decline in performance. Given that the well-trained offline policy is reliable, the desirable critic should not only have optimistic Q-value estimates but also maintain alignment with the offline policy. To achieve this, we propose performing value alignment to calibrate the critic. The main idea is to use the Q-values of the offline policy actions as anchors, keeping them unaltered, and to suppress any overestimated Q-value that exceed these anchors, thereby aligning the critic with the offline policy. Below, we will discuss how to implement value alignment for different online methods.

O2SAC Recall that in SAC [16, 17], the optimal policy is defined as

With a simple transformation of Eq. (10), we have

One intuition behind our method is that actions with higher probabilities typically have more accurate Q-value estimates because these actions are closer to the dataset, and there is sufficient data nearby to obtain a precise estimate. This motivates us to use the Q-value Q ¯ µ ( s, ˙ a ) of the optimal action ˙ a to calibrate any other overestimated action. Assuming that π off is the offline policy, we perform value alignment for any state-action pair ( s, a ) as follows:

where min( · , · ) operator is used to maintain the Q-values of actions that are not overestimated and consistent with OPE results.

Formally, we define the objective function of value alignment as follows:

Considering that the overall Q-values have decreased, to maintain an optimistic estimate, we use the regularization term L retain Q to prevent the underestimation of Q-values.

We derive Proposition 4.3 to show that the aligned state values V align ( s ) can be maintained within an appropriate range, meaning that the estimates remain optimistic while avoiding overestimation.

Proposition 4.3. After the value alignment process of Eq. (13) , the state value function V align ( s ) satisfies

where Q fqe ( s, a ) are the low Q-values after policy re-evaluation, which do not require calibration, V fqe ( s )= Q fqe ( s, a ) -α log π ( a | s ) and V ˙ a ( s )= Q ( s, ˙ a ) -α log π (˙ a | s ) .

During value alignment, we update the policy with Eq. (2) simultaneously for sampling the overestimated actions. The whole process iterates Eq. (2) and Eq. (13) to obtain a policy that performs equivalently to the offline policy and a critic aligned with it. We denote the policy as π on and the aligned critic as Q on for sequential training. Note that Q on, which is modified from the critic obtained in the policy re-evaluation, does not depend on specific offline critics. This flexibility allows us to implement the transition to SAC from different offline algorithms.

O2TD3 As done in the case of O2SAC, we can use the Q-values of the policy actions ˙ a to maintain the optimistic property of policy re-evaluation and calibrate the Q-values of other actions.

Unfortunately, in TD3, the actor is modeled as a deterministic policy, lacking an explicit expression for Q-values, which prevents us from directly aligning Q-values with the policy.

To solve this problem, our main idea is to model the distribution of Q-values around the policy action ˙ a as a Gaussian distribution. Specifically, in Eq. (5), when we use the offline policy as π θ , the gradient of the policy is only related to the the gradient of Q ( s, a ) with respect to a . It is easy to see that the gradient around ˙ a should tend to 0 as ˙ a is the optimal action selected by the offline policy. Due to the use of smoothing regularization in the critic's update in TD3, the Q-values of actions near ˙ a differ only slightly from the Q-value of ˙ a itself. This enables us to assume that normalized Q-values around ˙ a follow a Gaussian distribution Q ( s, a ) /Q ( s, ˙ a ) ∼ N (˙ a, Σ) . Formally, we can calibrate the Q-values of other actions as (see Appendix G.2 for detailed derivation)

where k is a constant, which is set as 1 across all tasks, and d ( a, ˙ a ) is a distance measure, which is defined as the euclidean distance divided by the square root of the action dimension in our implementation.

From Eq. (15), after calibration, the Q-values of the actions around ˙ a are only slightly lower than Q ( s, ˙ a ) , which ensures that ˙ a is the output action of the policy while maintaining a smoothing and optimistic property of Q-values. Moreover, for the actions that differ greatly from ˙ a , we limit the maximum of the distance measure d ( a, ˙ a ) in Eq. (15) to the policy noise used in Eq. (6) to avoid severe underestimation of their Q-values. Formally, we define the objective loss of value alignment for O2TD3 as

where Q ′ µ ( s, ˜ a ) = min( Q ¯ µ ( s, ˜ a ) , Q ′ µ ( s, ˜ a )) and ˜ a is a perturbed action defined in Eq (6).

O2PPO In PPO, only the critic V ( s ) is used to estimate the advantages for policy update. During the re-evaluation process, we train a critic by fitting the returns of offline trajectories as mentioned in Section 4.1. This indicates that the re-evaluated critic only approximate V µ ( s ) instead of the true one, misguiding the update of the policy. To mitigate this problem, we propose an auxiliary advantage function to correct erroneous updates.

Generally, a desirable auxiliary advantage function should satisfy the following two conditions: 1) it enables the policy to update in a reliable region; 2) its value must be zero at the beginning of online fine-tuning to enable the policy to transition smoothly from offline to online. Considering that the well-trained offline policy is reliable, we define the auxiliary advantage function as

where H is the entropy of action probabilities predicted by π off . It is easy to verify the second condition that A α ( s, a ) = 0 at the beginning of offline fine-tuning. To verify the first condition, we derive the following proposition. Its proof can be found in Appendix F.

Proposition 4.4. With A α ( s, a ) in Eq. (17) , the policy update is regularized by the cross-entropy loss about the offline policy, thereby constraining the policy update in a reliable region.

Accordingly, we define the advantage function for policy update as

where β anneals to 0 from 1. With the auxiliary advantage, Eq. (18) prevent the update direction from deviating too far from the offline policy.

4.3 Constrained Fine-Tuning

In the previous subsections, we discussed how to address the mismatch issues in the O2O problems. However, due to the distribution shift between the offline dataset and the online environment, along

AWAC

IQL

PEX

Off2On

Cal_QL

ACA

O2SAC

O2TD3

0

25

50

75

100

Evaluation Epoch

20

40

60

80

Normalized Return

0

25

50

75

100

Evaluation Epoch

0

20

40

60

80

Normalized Return

0

25

50

75

100

Evaluation Epoch

40

60

80

100

Normalized Return

0

25

50

75

100

Evaluation Epoch

80

100

Normalized Return

halfcheetah-expert-v2

0

25

50

75

100

Evaluation Epoch

60

80

100

Normalized Return

hopper-medium-v2

0

25

50

75

100

Evaluation Epoch

40

60

80

100

Normalized Return

hopper-medium-replay-v2

0

25

50

75

100

Evaluation Epoch

40

60

80

100

Normalized Return

hopper-medium-expert-v2

0

25

50

75

100

Evaluation Epoch

20

40

60

80

100

Normalized Return

hopper-expert-v2

0

25

50

75

100

Evaluation Epoch

80

100

Normalized Return

walker2d-medium-v2

0

25

50

75

100

Evaluation Epoch

80

100

120

Normalized Return

walker2d-medium-replay-v2

0

25

50

75

100

Evaluation Epoch

80

100

120

Normalized Return

walker2d-medium-expert-v2

0

25

50

75

100

Evaluation Epoch

40

60

80

100

120

Normalized Return

walker2d-expert-v2

with the optimism in online RL, encountering out-of-distribution (OOD) states and actions becomes inevitable, potentially leading to significant performance fluctuations. Especially for OOD states that are absent in the offline dataset, even though the policy was trained well in the offline phase, it may still fail to output favourable actions, thereby causing erroneous policy update.

Considering that the critic maintains an optimistic nature after undergoing policy re-evaluation and value alignment, with the optimistic update way during online fine-tuning, it typically overestimates the Q-values of OOD state-action pairs, which can mislead the update of the policy. Inspired of CMDP [1, 41, 7], we develop constrained fine-tuning (CFT) to introduce a regularization term to constrain the current actor and critic, which prevents the policy update from being severely misguided.

Specifically, we impose a constraint term f ( π, π ref ) on the policy objective to ensure that it updates within the credible region of π ref, where f ( · , · ) is a divergence measurement and π ref is the optimal historical policy during online evaluations. Formally, we define the policy objective of CFT as

By incorporating the constraint term into the reward function akin to RCPO [41], we can solve the problem by minimizing the following objective functions with initializing π ref as π on obtained in Section 4.2 at the beginning of online fine-tuning.

We provide a theoretical guarantee for the proposed CFT framework.

Corollary 4.5. With the penalty f ( π, π ref ) defined before and appropriate learning rates, algorithm of Eq. (20) almost surely to a fixed point ( θ ⋆ , µ ⋆ , λ ⋆ ), where λ ⋆ = 0 , θ ⋆ and µ ⋆ are corresponded to π ⋆ and Q ⋆ , which are optimal in the MDP without constraint.

In our implementation, we use KL divergence and MSE function as f ( · , · ) for the transitions of O2SAC and O2TD3, respectively. For O2PPO, the auxiliary advantage function already has the





































(a) From TD3+BC to SAC







































(b) From online DT to TD3

0

20

40

60

80

100

Environment Steps(×2.5e3)

30

35

40

45

50

55

60

Normalized Return

antmaze-umaze-diverse-v2

O2PPO from ODT

ODT

(c) From online DT to PPO

ability to constrain the policy update, we only need to replace π off with π ref during online finetuning. Note that in our methods, at the beginning of online fine-tuning, the regularization function f ( π θ ( a | s ) , π on ( a | s )) is zero for any state, which guarantees no destruction of the alignment for the actor and the critic obtained in Section 4.2.

5 Experiments

In this section, we perform experiments to validate the effectiveness of the proposed method on D4RL [9] MuJoCo and AntMaze tasks, including HalfCheetah, Hopper, Walker2d and AntMaze environments. Specifically, we compare our methods with AWAC [32], IQL [20], PEX [50], Off2On [23], Cal-QL [33] and ACA [48]. For all methods except for O2PPO, we run 100,000 interaction steps with the environment and evaluate the policy per 1000 steps, as they all use off-policy methods for online fine-tuning. For our O2PPO method, due to the low efficiency of the on-policy method, we run 250,000 interaction steps to validate its performance, with 2500 steps as the evaluation interval. We run all methods with five random seeds and report their averaging results. Due to the space limitation, more experimental results can be found in Appendix B and C.

5.1 Comparison with State-of-the-Art Methods

We respectively initialize the policies of O2SAC, O2TD3 and O2PPO from the results of CQL, TD3+BC and IQL. Figure 2 shows the performance curves of our methods and comparing methods on Mujoco tasks. Similar to the previous works [48, 5], for O2SAC and O2TD3, we use CQL and TD3+BC for offline policy pre-training. From the figure, we can see that our method can converge more stably and rapidly than other methods and achieve the optimal performance in most cases. Although the ensemble method Off2On can achieve better final performance than our methods in some cases, it often suffers from a dramatic drop in performance at the beginning of fine-tuning, which is often unacceptable in the O2O problems. We report the results of O2PPO in Appendix A because of its different number of interaction steps. Although PPO is an on-policy method with low efficiency, from the results in Table 1, it shows significant superiority on sparse reward tasks, even though with equal interactions.

5.2 Study on Transferability

In this section, we perform experiments to verify the powerful transferability of the proposed method. As mentioned earlier, one of the advantages of our method is that it imposes no requirements on offline

algorithms. This means we can achieve the transfer from any offline RL algorithm to three online RL algorithms. Figure 3 illustrates the fine-tuning performance of three online algorithms transferring from their heterogeneous offline algorithms. From Figure 3(a), pre-trained with TD3+BC, O2SAC outperforms SAC trained from scratch with a larger margin. Although Online Decision Transformer (ODT) achieves favourable performance in the offline environment, it converges slowly during online fine-tuning due to the architecture of the transformer. Our O2TD3 and O2PPO significantly enhances its performance through online fine-tuning. These results convincingly verify that our method show the strong transferability from various offline methods.

6 Conclusion

In this paper, we disclose there exist two types of mismatches when online fine-tuning offline RL methods. To address these two mismatches in O2O RL, we proposed optimistic critic reconstruction to re-evaluate an optimistic critic and align it with the offline actor before online fine-tuning, ensuring the stable performance improvement at the beginning stage and potentially better efficiency due to the reconstructed optimism consistent with online RL. Furthermore, to combat the inevitable distribution shift that can hinder the stable performance improvement, we introduce constrained fine-tuning to constrain the divergence of current policy and the best foregoing policy to maintain the stability of online fine-tuning. These two components form a versatile O2O framework, allowing the transition from any offline algorithms to three state-of-the-art online algorithms. Experiments show our framework can converge to optimal performance without affecting the aligned critic at the beginning of online fine-tuning and achieve strong empirical performance.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , 32:96, 2019.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in Neural Information Processing Systems , 35:28955-28971, 2022.

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948 , 2023.

Alex Beeson and Giovanni Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211.11802 , 2022.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757 , 2018.

Christopher Diehl, Timo Sebastian Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Uncertainty-aware model-based offline reinforcement learning for automated driving. IEEE Robotics and Automation Letters , 8(2):1167-1174, 2023.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328 , 2023.

Cong GUAN, Ke XUE, Chunpeng FAN, Feng CHEN, Lichao ZHANG, Lei YUAN, Chao QIAN, and Yang YU. Open and real-world human-ai coordination by heterogeneous training with communication. Frontiers of Computer Science , 19(4):194314, 2025.

Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang, Hechang Chen, Haiyin Piao, Lichao Sun, and Yi Chang. A simple unified uncertainty-guided framework for offline-to-online reinforcement learning. arXiv preprint arXiv:2306.07541 , 2023.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573 , 2023.

Ingook Jang and Seonghyun Kim. Uncertainty-driven pessimistic q-ensemble for offline-toonline reinforcement learning. In 3rd Offline RL Workshop: Offline RL as a'Launchpad' , 2022.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning , pages 1702-1712. PMLR, 2022.

Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:2311.03351 , 2023.

Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305.15669 , 2023.

Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. arXiv preprint arXiv:2303.17396 , 2023.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:1711-1724, 2022.

Yihuan Mao, Chao Wang, Bin Wang, and Chongjie Zhang. Moore: Model-based offline-toonline reinforcement learning. arXiv preprint arXiv:2201.10070 , 2022.

Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning , pages 23829-23851. PMLR, 2023.

Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022 , 2022.

Trevor McInroe, Stefano V Albrecht, and Amos Storkey. Planning to go out-of-distribution in offline-to-online reinforcement learning. arXiv preprint arXiv:2310.05723 , 2023.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479 , 2023.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

RichardS. Sutton and AndrewG. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks , page 285-286, Jan 2005.

Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems , 35:34272-34286, 2022.

Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a 'Launchpad' , 2022.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 , 2018.

Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning , pages 34556-34583. PMLR, 2023.

Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Gaetan Lin, HAO CHEN, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-toonline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:31278-31291, 2022.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:40854098, 2022.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810 , 2023.

Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In International Conference on Machine Learning , pages 40452-40474. PMLR, 2023.

Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. arXiv preprint arXiv:2310.04411 , 2023.

Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935 , 2023.

Kai Zhao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Ensemble-based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration. arXiv preprint arXiv:2306.06871 , 2023.

Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210.13846 , 2022.

Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning , pages 27042-27059. PMLR, 2022.

A Detailed Data

Compared with the baseline methods, our methods outperform much in D4RL Mujoco locomotion tasks, as showned in Table 2. We can compare the fine-tuning performance in groups based on online update way. Off2On, Cal-QL, ACA and our O2SAC are updated in the SAC way during online fine-tuning. Although there is a lack of the baseline methods that update the policy in the TD3 way, we compare our O2TD3 with a recent O2O method PROTO+TD3 [25] in Appendix C.4 to demonstrate the superiority of our methods. And we compare our O2PPO with the policy constraint methods AWAC, IQL and PEX, that are implemented in the IQL way, since the idea of PPO is similar to a kind of policy constraint.

With such groups for comparison, our methods get the best performance improvements respectively. Note that in our implementation, Off2On achieves much better performance than the original paper [23] and the implementations in other papers [48] and [50]. However, our O2SAC still outperforms it with less computational cost during online fine-tuning and less requirements for offline policy. In

addition, our methods bridge different offline algorithms and three SOTA online algorithms, and permit additional modifications for improved algorithms based on the online algorithms.



























































































































































































































































Due to the low sample efficiency of PPO, we run 250,000 interaction steps for the implementation of O2PPO, which makes it unreasonable to compare the results of different O2O methods in one graph. So we demonstrate the results of O2PPO without any baseline method, but compare our method with the direct way of fine-tuning the offline policy of IQL in the PPO way in the online phase. Note that our purpose is to achieve stable performance improvement, the efficiency depends mainly on the online algorithm, so in some environments, such as walker2d-medium-replay-v2 , the performance improvement may be less than the direct fine-tuning. This phenomenon could arise due to the critic's capability to approximate the genuine values effectively, thereby facilitating accurate evaluation of the policy, obviating the necessity for corrective adjustments in the policy update direction. And our O2PPO uses an auxiliary advantage function to constrain the policy update, resulting in slower performance improvement. However, since we cannot judge when the critic is reliable, it is necessary to use O2PPO to achieve stable performance improvement, and there is no remarkable discrepancy between O2PPO and the direct way, although in the above scenarios. In general, our O2PPO can achieve stable and efficient performance improvement with limited interactions.

B Ablation Study

Before the analysis of ablation studies, we need to clarify the roles and applicability of the different components of our methods for different offline methods.

The benefit and applicability of policy re-evaluation As talked before, the purpose of policy reevaluation is to get optimistic Q-value functions, avoiding the drop of Q-value due to underestimated Q-values in offline. Many previous works [27, 33] discussed such a problem and solve it by alleviating excessive pessimism. Our methods do not need initialize critic from offline results, but re-evaluate the policy in the online way to get optimistic Q-value functions, which is applicable to different algorithms and avoids the performance drop caused by the drastic drop of Q-value. Note that optimistic Q-value functions obtained by policy re-evaluation may be unreasonable, especially for OOD actions, hence value alignment is needed for redress overestimated values.

In some policy constraint methods [32, 10], only the policy is limited to update in a region close to the dataset, but the critic is obtained in the same way as the online update. Therefore, for such offline methods, the online critic can be initialized directly by the offline one, and just need to align the values with the offline policy. In addition, policy re-evaluation can be applied to the scenarios where only the offline policy is provided, such as where the offline policy is obtained in the style of sequence modeling [6, 53, 18] or non-standard RL [46].

The necessity of value alignment With the optimistic property, Q-value functions obtained in policy re-evaluation will overestimate OOD actions and induce the policy to take such actions, resulting in performance degradation. For the transition from those policy constraint methods that do not require re-evaluation of the policy, this case still holds. The purpose of value alignment is to approximate the optimistic property and suppress the Q-values of OOD actions to a reasonable estimation. For those offline algorithms where the actor and the critic are misaligned like explicit policy constraint methods, if the constraint is removed during online fine-tuning, the critic will induce actor to align with it. However, the actor is reliable but the critic is not, so such process leads to unknown performance changes that are most likely to be worse. In addition, for algorithms induced by behavior-regularized MDP, such problem still exists because the update way between offline and online changes, leading to drastic variation of the critic. Therefore, for O2O RL, it is necessary to align the critic with the actor instead of aligning the actor with the critic.

The necessity of constrained fine-tuning Although we keep the optimistic property for Q-value functions and align the critic with actor, it is still challenging to achieve stable online fine-tuning. In general, most of the current offline algorithms focus on how to avoid OOD actions and train a reliable policy on the states of the dataset. Due to the optimism in online RL, OOD states and actions are inevitable, may leading to drastic performance fluctuations, which is undesirable for important scenarios especially for high-risk scenarios. Especially for OOD states, even trained well in the offline phase, the policy still fails to output favourable actions, that may causes erroneous policy update. Therefore, for stable O2O RL, online fine-tuning with the constraint of ensuring safe exploration is necessary.





































(a) Ablation results of O2SAC







































(b) Ablation results of O2SAC







































(c) Ablation results of O2TD3





































(d) Ablation results of O2TD3







































(e) Ablation results of O2PPO





































(f) Ablation results of O2PPO

The results of Figure 5 show that even with a narrow dataset e.g. hopper-expert-v2 , our methods still achieve stable online fine-tuning. For O2SAC, the fine-tuning process without the optimistic critic reconstruction can lead a sudden performance drop at the beginning phase, e.g. in Figure 5(a), as the offline critic may be severely pessimistic. Due to the alignment of the offline critic and the actor in CQL, using constrained fine-tuning alone can result in overall stable performance improvement,

but O2SAC demonstrates a more efficient result. As for O2TD3, fine-tuning from offline directly suffers from severe performance degradation due to the mismatch of the actor and the critic in offline training, such as in Figure 5(c). Thanks to value alignment, we can address the mismatch, thereby achieving stable performance improvement at the beginning stage. However, due to OOD states during online fine-tuning, constrained fine-tuning is necessary for the long-term stability. At last, PPO is an on-policy method that has a high data quality requirement, so it cannot improve performance if the value is incorrectly evaluated, e.g. in Figure 5(e). Through the reliable auxiliary function, our O2PPO can improve performance stably even with imperfect value evaluation.

Although our methods are universal for any offline algorithm to SAC, TD3 and PPO, some steps can be omitted according to the type of the offline algorithm. For example, for value regularization method e.g. CQL, we can only use constrained fine-tuning for stable and efficient O2O RL if the constraint term with a low coefficient has little effect on the critic, as the offline actor and critic are aligned, constrained fine-tuning is enough to combat the problem caused by the drastic jump of Q-values. And for policy constraint methods with explicit constraint e.g. TD3+BC and AWAC, as talker above, the critic is evaluated in the same way as online way, so policy re-evaluation can be omitted. For other offline methods with the update way that does not match the online way, such as IQL [20], IVR [47] and χ -QL [13], and methods induced by behavior-regularized MDP, and with special policy form decision transformer , all steps of our methods are needed for stable O2O RL.

C Additional Experiments

C.1 Difference in policy performance caused by optimistic critic reconstruction

Note that we use π on as the initialization of the actor at the beginning of online fine-tuning. In accordance with our analysis, the performance of the initial policy π on is expected to align with the characteristics of the actual offline policy. As depicted in Table 3 from our empirical experiments, the results substantiate our analysis, revealing minimal disparities in the performance of π on compared to the actual offline policy. In most environments, π on even demonstrates superior performance than the offline policy. We attribute this to the effectiveness of our min operator, which sensibly tightens the policy distribution.

C.2 Experiments on D4RL AntMaze navigation tasks

For the difficult tasks of AntMaze navigation, such as medium and large environments, TD3+BC is almost completely incapable of training a favourable policy [40]. Fine-tuning with the poor

initialization from such a policy helps little and has minor differences with training a policy from scratch, that should be the concern of hybrid learning [4], so we do not consider it.

Here we demonstrate the results of O2SAC and O2PPO on the the difficult tasks of AntMaze navigation, including antmaze-medium-play-v2 , antmaze-medium-diverse-v2 , antmaze-large-play-v2 , antmaze-large-diverse-v2 environments.

















































































































































































































































Note that here we run O2PPO with 250,000 environments steps and run O2SAC with 200,000 environments steps, while in Section 5.1, we run O2PPO with 200,000 environments steps.

Note that as some previous work, in our O2SAC implementation on AntMaze tasks, we do not use the double Q networks trick to avoid overly underestimation, and the threshold τ reaches the maximum in step 100,000.

Since there is a lack of hyper-parameters for these tasks in many related work [48, 25, 50, 53], we do not compare the results of our methods with other methods. However, according to our simple reproduction, our methods achieve competitive results.



















































































































































































































































































































































C.3 Comparisons with the results of updating the reference policy update at a fixed interval

We set the reference policy as the optimal historical policy during online evaluations in our preceding experiments, and it indeed introduces additional testing information. However, for most of the scenarios that offline RL focus on, this would be acceptable as a well-learned policy can be deployed directly without serious consequence due to its decent performance, e.g. in autonomous driving.

In addition, thanks to the abundant data, another way that does not require practical evaluation is to train a transition model that can be used to evaluate the policy by generating synthetic rollouts as the initial states are provided. Since such a methods is similar to model-based methods, we will leave it for future work.

If evaluation is forbidden, we can replace π ref with a recent policy at a fixed interaction interval. The Corollary 4.3. still holds because the policy performance will be almost certainly improved with a long interaction interval, especially compared to the last reference policy. Considering that the policy performance is most likely to fluctuate in the beginning stage during online fine-tuning, we can set a large update interval for this stage and a small update interval for the subsequent stage, or reduce the threshold change range.

We experiment our methods with a given update interval on D4RL Mujoco locomotion tasks and compare the results with those of the methods with the optimal reference policy, shown as Figure 8 and Figure 9. Without special hyper-parameter optimization, we just update the reference policy per 1000 steps for most environments, but for hopper-medium-expert-v2 and hopper-expert-v2 , we update the reference policy per 10000 steps, and as well as for walker2d-expert-v2 in O2TD3, because these datasets are narrow, making it easy for the policy to suffer from OOD states and actions. With an appropriate interval, the policy performance can achieve competitive improvement when compared with the methods with the optimal reference policy. Policy re-evaluation and Value alignment guarantee the smoothing and stable performance improvement at the beginning stage, and constrained fine-tuning is necessary for the stability of the subsequent stage. Although in hopper-medium-v2 , O2SAC with a fixed update interval suffers from the performance degradation at the latter stage, because the threshold is large and the Lagrange multiplier λ is low, the lack of constraint on the policy update in the reliable region. This phenomenon can easily be amended by reducing the threshold change range.





















































































































































































































































C.4 Comparisons with PROTO

In the absence of O2O methods on the deterministic policy, we compare our methods with PROTO [25], a recent O2O method that supports fine-tuning from an offline policy for both the stochastic and the deterministic policy. We reproduce the code of PROTO in Pytorch for the initialization of the policies derived from CQL and TD3+BC, and set all hyper-parameters as in the official paper [25]. As shown in Figure 10, our methods demonstrate significant superiority in both stability and effectiveness, especially for the deterministic policy.

C.5 Comparisons on initialization of different offline methods

We conducted some experiments for O2SAC with the initialization of different offline methods, including CQL, IQL and ODT. The results are shown in Figure 11. The initial performance of O2SAC initialized from ODT is lower than others since the simple behavior cloning (we directly maximize the likelihood of the actions output by offline ODT while keeping an appropriate entropy) could harm the performance, as discussed in Appendix I. But in hopper-medium-v2, the performance improves quickly. We analyze that by the constraint, the policy can recover the offline performance (about 97 normalized score of ODT), as the output of the cloned policy is near the ODT policy. The results demonstrate that our methods are suitable for any offline algorithm, even the policies are heterogeneous.

C.6 Accelerate learning with sample-efficient RL

An additional benefit of our methods is their straightforward compatibility with sample-efficient online RL algorithms. Since we only add a constraint that can be considered as part of the reward, the policy iteration process remains consistent with the normal online approach, which makes it feasible to incorporate techniques from advanced efficient RL algorithms. Drawing from [4], we conducted some experiments using a high UTD ratio of 10 (but still update the lagrangian multiplier once per step) and achieved better performance improvements, as shown in Figure 12.













































































































































































































































































































































































C.7 Computational cost for each component

We conducted an experiment on hopper-medium-v2 environment using the O2SAC method and listed the time cost for different phases in Table 5, evaluated on an Nvidia 3070 GPU.













































































In policy re-evaluation, since the policy is fixed and the re-evaluation of the critic is straightforward, the computational cost of re-evaluation is significantly lower than that of offline learning. The time cost of value alignment is somewhat higher but still less than that of the offline phase. In fact, the time cost is approximately proportional to the offline phase according to the alignment steps, since both the actor and critic are updated in the value alignment phase.

However, it is worth noting that although we set the training steps for value alignment at 500k, in some environments, only a few alignment steps are needed to calibrate the critic with the offline actor, as shown in Fig. 1 and Fig. 11. Only in antmaze environments, where it is hard for the critic to capture the sparse reward signal, more alignment steps are necessary. Additionally, in constrained fine-tuning, since only the lagrangian multiplier is added to be updated and the interaction cost dominates, the time cost increases very little.

Moreover, in O2O RL, we are typically not concerned about the time cost in the offline process, as different offline methods take different amounts of time. Instead, we prioritize the cost of interactions during online fine-tuning. Our method re-evaluates and aligns the critic with the offline actor solely within the offline dataset, making the time cost less critical.

D Related Work

Unified algorithms for offline-to-online Many offline algorithms are unified across phases in O2O RL [32, 20, 13, 43, 44], who share the common philosophy of designing an RL algorithm that is suitable for both offline and online phases and then the network parameters trained in the offline phase can be reused for further learning in the online phase. Most of them are policy constraint methods which learn policy without querying OOD samples [20, 43] or penalize action probabilities of them by an explicit estimation of behavior policy [44], which is beneficial for offline performance but limits efficient performance improvements in online fine-tuning, as talked about in [33]. For explicit policy constraint, recent works focus on how to make the constraint adaptive [52, 26] or loosen it gradually [5]. Although such algorithms achieve great performance in offline phase, there is a lack of research on from such algorithms to more efficient online algorithms.

Efficient online fine-tuning Some recent works aim to online fine-tune efficiently with optimistic exploration. [15] propose a unified uncertainty-guided framework to explore optimistically in online fine-tuning phase and keep the offline constraint for OOD actions to avoid erroneous update. [30] and [51] utilize ensemble Q-learning to alleviate distribution shift, and implement optimistic exploration by some approaches about ensemble in online RL. For model-based O2O RL, [31] explores regions with high uncertainty and returns in learned model. [42] and [50] concatenate different offline and online algorithms by resetting a new online policy learning from the offline policy gradually, where optimistic exploration is implemented by the new policy. With a given policy and dataset, [3] focus how to recover the performance of the policy rapidly, whose setting is suitable for our method too.

Stable online fine-tuning As offline RL works for some important scenarios with high risk, the stable performance is considerable for O2O RL. After [23] put forward distribution shift problem in O2O RL which is alleviated by pessimistic Q-ensemble and balanced replay proposed by them, many methods are proposed to combat this problem. From offline to SAC, [33] mitigates the penalty for

OODactions in CQL and calibrates Q-values of them by Monte Carlo returns, while [48] reconstructs Q-functions aligned with offline policy. [25] also induces KL divergence to policy objective, which is about current policy and the policy at the last iteration to perform a trust-region-style update, but the performance will drop suddenly at the beginning of online fine-tuning. [19] and [28] utilize environment dynamics model ensemble to obtain uncertainty to penalty OOD actions.

E Detailed Discussions about Mismatch in SOTA Offline Algorithms

CQL For common implementation of CQL, [21] do not specify exactly how π k is updated but only provide properties on the policy exp( f k ( s, a ) /Z ( s )) like SAC. In this way, the actor of CQL is only related to the critic, and the action probability is directly proportional to Q ( s, a ) like online RL. However, as the conservative policy evaluation operator used to update the Q-function (according to [21] Appendix C, equation 13), the policy is not only related to rewards, but also related to the distribution of behavior policy. The conservative policy evaluation operator in CQL is:

where B π is the standard Bellman operator B π Q ( s, a ) = r ( s, a ) + γ E s ′ ∼ P ( ·| s,a ) [ V ( s ′ )] and

If we define Q o as the Q-function derived from Eq. (1), which satisfies:

Therefore, the relationship of conservative V ( s ) in CQL and Q o ( s, a ) is:

The objective of policy is to maximize V ( s ) , so compared with online RL, there is a difference in learning objective due to pessimism. In fact, drawing from SAC, we can derive the objective of CQL:

which is in according with behavior-regularized MDP proposed by [47] at f ( x ) = x -1 . The conservative policy objective causes pessimistic Q-values estimation, hence in online fine-tuning, the change of policy objective leads to drastic Q-values jump and performance degeneration.

IQL As expectile regression used to estimate expectiles of the state value function with respect to random actions, Q ( s, a ) and V ( s ) are not induced by the policy π , and we denote them as Q µ ( s, a ) and V µ ( s ) , where µ represents a special policy related to the expectile regression [18]. With initialization of such Q µ ( s, a ) and V µ ( s ) in online fine-tuning phase, if we update the actor and the critic in online algorithm, the actor update will tends to µ at the beginning, which causes unknown performance change because the performance of ν is unknown. And usually the performance will deteriorate significantly the probability of a satisfactory policy ν is low.

On the other hand, the work of [47] reveals that the optimal critic of IQL can be derived from behavior-regularized MDP at f ( x ) = log( x ) . Meanwhile, the update of actor is derived from return maximization with a KL divergence constraint, it is easily to discovery that the learned policy is relevant to dataset. Therefore, the O2O RL for IQL still suffers from the problem of policy objective change. And compared with value regularization methods, there is not only pessimistic estimation problem but also initial misalignment problem, which means offline actor cannot be derived from offline critic in online update way. In fact, in offline RL, the policy update function is to maximize Eq. (9), which is highly related to behavior policy as α is a fixed hyper-parameter. If 1 is directly used to fine-tune policy, the policy will tends to be only proportional to Q ( s, a ) of offline critic, hence the policy performance will drop sharply with a great probability.

TD3+BC With a explicit constraint of policy update, the improvement mismatch is obvious: the critic update follows the Bellman backup, but the actor update does not only follow Q-values maximization, that constrains the actor only update in a region close to the dataset. A little different from the

mismatch in IQL, TD3+BC does not suffer from pessimistic estimation since the critic is updated in an optimistic way as the online way, but such improvement mismatch still leads to performance drop. The probability of action with the largest Q-value may be not largest, as talked about in [48], which is not in according with online RL and leads to erroneous update to destroy performance at the beginning of online fine-tuning. Such improvement mismatch exists in almost all explicit constraint methods, but their critics are optimistic because the update ways is the same as online RL, so value alignment is necessary for O2O RL from such offline methods.

F Proof

Proof of Corollary 4.2 The theoretical analysis about FQE can be found in Theorem 4.2 of [22] or Theorem 4.9 of [29], here we use the latter result. Full proof is similar to the proof of [29], but a little different from [29], we fix the evaluated policy as the offline policy.

Here we simply show the proof process and emphasis the difference, see [29] for more details.

Since FQE deals with the following regression problem

we can have

With the inherent Bellman evaluation error d π F , we can apply least squares generalization bound here (Lemma A.11 in [2]). With probability at least 1 -δ , we have

Note that here we fix the policy π as the offline policy, that means for a given Q 0 , Q k -1 is welldetermined, so we do not need to apply a union bound over all possible Q k -1 .

Then we first bound ∥ Q k -Q π ∥ 2 ,d π t × π as the same as [29] did.

The fifth derivation uses Jensen's inequality. With

Therefore

Proof of Proposition 4.3 We define Q fqe ( s, a ) as the state-action values for some actions after the process of policy re-evaluation and lower than the calibrated value, i.e. Q fqe ( s, a ) ≤ Q ( s, ˙ a ) -α (log π off (˙ a | s ) -log π off ( a | s )) . With the min operator in Eq. (12), we can divide actions into two categories, calibrated actions or standard actions . The former refer to those actions with Q-values to be calibrated, and the latter refer to those actions with unchanged Q-values since they are lower than the calibrated values. We denote them as a cal and a fqe respectively.

Due to the min operator, for standard actions a fqe, the calibrated Q-values satisfy Q fqe ( s, a fqe ) ≤ Q cal ( s, a fqe ) , therefore

On the other hand, it is easy to see that Q fqe ( s, a fqe ) -α log π ( a fqe | s ) ≤ Q cal ( s, a cal ) -α log π ( a cal | s ) , so

So V fqe ( s ) ≤ V align ( s ) ≤ V ˙ a ( s ) , i.e. Proposition 4.3 is proved.

Proof of Proposition 4.4 With simplicity, we consider the term of the auxiliary advantage function without probability ratio clipping

Therefore, the term of the auxiliary advantage function regularizes the policy update near the reference policy, thereby ensuring reliable update even with inaccurate value estimation. And at the beginning of online fine-tuning, the reference policy is initialized from π off .

Proof of Corollary 4.5 Since our constrained fine-tuning method solves a constrained MDP problem by the method in [41], Theorem 2 in [41] still holds in our method, which can be described as With the constraint satisfied, RCPO algorithm converges almost surely to a fixed point ( θ ⋆ ( µ ⋆ , λ ⋆ ) , µ ⋆ ( λ ⋆ ) , λ ⋆ ) .

Such theorem guarantees the convergence of RCPO, and with the our proposed constraint, we can utilize contradiction to proof Corollary 4.5.

̸

Assumption According to the convergence of RCPO, we can assume the convergent point ( θ ⋆ ( µ ⋆ , λ ⋆ ) , µ ⋆ ( λ ⋆ ) , λ ⋆ ) with λ ⋆ = 0 .

Derivation As the constraint is E π [ f ( π ( a t | s t ) , π ref ( a t | s t ))] < α and π ref is the best one among old policies during online evaluations, π ref = π ⋆ when the algorithm converges, so the constraint term tends to -α . According to the update function Eq. (20) of λ , λ tends to be reduced, which is in contradiction to the Assumption. Therefore, λ ⋆ = 0 when the algorithm converges.

Meanwhile, when λ ⋆ = 0 , the constraint is removed, hence the π ⋆ and Q ⋆ are the same with the optimal results in the MDP without constraint, which means our method converges the optimal point like online RL but keeps stable improvement.

G Algorithm Details

G.1 O2SAC

the choice of α

Although α is generally smaller than 1 after offline training, as we use the energy policy to align critic with actor, a small α has no influence on the recovery of offline policy but leads to a wide distribution for policy, which means the policy modeled by Gaussian distribution has a large standard deviation, which is harmful for online exploration. If the standard deviation is large, some OOD actions may be taken and dangerous state may arise during online exploration, which dose not suit the setting of high risk scenarios and affects favourable performance. Therefore we choose a suitable α which is smaller than 1 but not excessively small. For Mujoco locomotion tasks, we determine that alpha is 0.2 for the dataset with medium quality, and is 0.5 for the dataset with expert







































quality, because small standard deviation induced by a large α is advantageous for online exploration of well trained policy. For AntMaze navigation tasks, we set alpha as 0.5 for medium and large datasets and 0.2 for umaze datasets.

Moreover, as we use min operator for value alignment, the limit for α is not so strict. If Q-values induced by policy re-evaluation is smaller than alignment objective, as a Gaussian distribution is used to fit energy policy, such Q-values will leads to a smaller standard deviation.

Note that the choice of α has no effect on the recovery of offline policy, as shown in Figure 13.

policy re-evaluation As talked about Section 4.1, with a given α , online policy evaluation of SAC Eq. (3) can be used to policy re-evaluation to obtain an optimistic critic.

value alignment In Section 4.2, the value alignment method to O2SAC has been described in detail. In brief, we use online policy improvement Eq. (2) to get the actions which have overestimated Q-values, and use Eq. (13) to inhibit their Q-values to a reliable value calculated by maximum entropy RL.

According to the min operator, there is no change if Q-values induced by policy re-evaluation is smaller than alignment objective, so our value alignment method not only aligns Q-values with actor but also is as consistent with the results of online evaluation as possible, which reduces Bellman error in online fine-tuning to keep Q-values more stable.

constrained fine-tuning For O2SAC, the constraint is KL divergence, and the policy objective of CMDP in O2O RL is:

And corresponding loss functions are:

A constraint related to offline policy is necessary for online fine-tuning, because OOD states will appear surely during online exploration, especially in narrow dataset, which may lead to erroneous overestimation and destroy old policy. Such constraint can avoid overestimation of actions far away from current policy, hence trust-region style update guarantees stable performance improvement.

Compared with the constraint of current policy and offline policy, our proposed constraint guarantees more optimal update because the policy updated in trust-region of offline policy generally has similar performance to offline policy, which leads to not much performance improvement.

And compared with the tight constraint of current policy and the policy at last iteration, our constraint guarantees rapid recovery when some erroneous update occurs which results in performance degradation, because the Lagrange multiplier λ will be larger and make the Q-values of OOD actions lower, hence the policy tends to be close to π ref. In such situation, our method can recover a similar performance to π ref for current policy rapidly, but the tight constraint needs to take much time to do that as it constrains the current policy close to the poor policy at last iteration.

G.2 O2TD3

policy re-evaluation Similar to O2SAC, online policy evaluation of TD3 Eq. (6) can be used directly to evaluate an optimistic critic, and in TD3, there is no need for α .

value alignment As TD3 models a deterministic policy, value alignment method can not be derived directly from the relationship of the actor and the critic like O2SAC. However, from Eq. (5), the gradient of policy is only related to the the gradient of Q(s, a) to a if we fix the policy as offline policy.

So, we have two insights for the actor in TD3: Q ( s, π ( s )) is the maximal in Q ( s, · ) and the gradient around π ( s ) should tend to 0, and the latter also corresponds to the idea of policy smoothing update in TD3.

Note that in most cases, the correlation among different dimensions of action is not be considered, hence we use one-dimensional Gaussian distribution for the following analysis.

Therefore, with the definition that ˙ a = π ( s ) , we consider that normalized Q-values around ˙ a follow a Gaussian distribution Q ( s, a ) /Q ( s, ˙ a ) ∼ N (˙ a, Σ) . And for Q ( s, ˙ a + δ ) , we utilize Taylor expansion of first order and omit higher order terms, then we can get:

From Eq. (40) to Eq. (41), we consider the continuous derivative. From Eq. (41) to Eq. (42), we utilize the derivative of a Gaussian distribution ∇ x f ( x ) = -( x -µ ) / Σ , where f ( x ) is a onedimensional Gaussian distribution f ( x ) ∼ N ( µ, Σ) .

Note that Q ( s, ˙ a ) = Q ( s, ˙ a ) / √ 2 π Σ , which means Σ = 1 / 2 π , so we can get:

In our implementation, we replace Eq. (43) to Eq. (15), where k is used to control penalty for overestimated values, and as action range is from -1 to 1, the alignment objective avoids severe underestimation because the minimum Q does not tends to 0 if we set a small k , which also reduces Bellman error during online update as the same thing as min operator does. And we use the euclidean distance divided by the square root of action dimension to calculate δ , which is δ ( a, ˙ a ) = √∑ ( a i -˙ a i ) 2 / | A | .

Note that in O2TD3, we use the same trick of reward normalization as TD3+BC, which means we subtract 1 from all rewards, so we need to consider the situation of negative rewards. In Eq. (43), we set Q ( s, a ) /Q ( s, ˙ a ) ∼ N (˙ a, Σ) when rewards are positive, so it is natural to set Q ( s, ˙ a ) /Q ( s, a ) ∼ N (˙ a, Σ) at the situation of negative rewards. Therefore, for the environment with negative Q-values, we use following way to redress critic:

In a word, we multiply Q ( s, a ) by (1 + kδ 2 ) if Q ( s, a ) > 0 , otherwise divide it by (1 + kδ 2 ) .

constrained fine-tuning For O2TD3, the constraint is MSE loss, so the loss functions are:

G.3 O2PPO

policy re-evaluation Different from O2SAC and O2TD3, as Eq. (8) is not related to offline policy, in practice we can only obtain V µ ( s ) through fitting the returns. Advantages A π off ( s, a ) computed by such V µ ( s ) may be awfully incorrect, which sequentially causes error update. However, in order to follow the update way of PPO (only V ( s ) is used to update), we stick to update state value function by fitting the returns, and we introduce a modification to advantages used to update.

value alignment As V π off ( s ) obtained in policy re-evaluation is actually V µ ( s ) , which is incorrect to compute advantages A π ( s, a ) for on-policy update, we propose an auxiliary advantage to redress

erroneous update. Let us think about the property of the auxiliary advantage. First, as an advantage function, its mathematical expectation of offline policy π off should be 0. Second, the function should be able to output positive values and negative values for different actions to distinguish the quality of actions. Last, better actions should correspond higher values. Drawing from the policy form of SAC, we propose the auxiliary advantage as:

When the policy is modeled as Gaussian distribution, it satisfies: (1) E a ∼ π off ( ·| s ) [ A α ( s, a )] = 0 . (2) A α ( s, a ) > 0 when ∥ a -µ ∥ 2 < σ . (3) A α ( s, a ) ∝ log π off ( a | s ) . And such properties are in accord with requirements of auxiliary advantage function.

It is notable that here is an implicit assumption that the actions with higher probability for π off are better, and for a well trained offline policy, such assumption should be valid, at least for the beginning of online fine-tuning. With such auxiliary advantage function, we can redress advantages computed by incorrect V π ( s ) to ensure the stable performance improvement during online fine-tuning, especially for the beginning stage.

constrained fine-tuning As the auxiliary advantage function has the ability to constrain policy to update in a reliable region near π off, that means π ref = π off, we just need to replace π ref as the optimal policy during online evaluations to constrain online fine-tuning. Meanwhile, with the increase of interactions, the critic gradually becomes accurate, approximating V π . And on-policy method is naturally stable to improve performance by updating in a reliable region, therefore the constraint factor β anneals to 0 from 1 during online fine-tuning.

H Implementation Details

H.1 baseline implementation

Offline results For obtaining offline policy, we choose CQL [21], IQL [20] and TD3+BC [10], and we reproduce offline results according to a deep offline RL library CORL [40] https://github.com/tinkoffai/CORL, all hyper-parameters are the same with the official implementations. The offline results are used in our methods, Off2On and PROTO.

1 CQL We reproduce the results of CQL by the code from CORL https://github.com/tinkoffai/CORL/blob/main/algorithms/offline/cql.py.

2 IQL Like the implementation of CQL, we reproduce the results of IQL by the code from CORL https://github.com/tinkoff-ai/CORL/blob/main/algorithms/offline/iql.py.

3 TD3+BC Similar to above, we reproduce the results of TD3+BC by the code from CORL https://github.com/tinkoff-ai/CORL/blob/main/algorithms/offline/td3_bc.py.

Online fine-tuning results For offline-to-online algorithms, for most methods, we reproduce the results according to their official implementations. For AWAC [32], IQL [20] and Cal-QL [33], we reproduce the results by the code from CORL [40]. For ACA [48] and PEX [50], we reproduce the results by the official open-source code https://github.com/ZishunYu/Actor-Critic-Alignment, https://github.com/Haichao-Zhang/PEX.

Although PROTO focus on online fine-tuning, but initialize the policy from other offline algorithms in the official implementation. For the sake of fairness, we rewrite the code according to the paper and official code (https://github.com/Facebear-ljx/PROTO) of PROTO in Pytorch, and we use the offline results of CQL and TD3+BC for initialization respectively.

In addition, in the official implementation of Off2On [23], the policy update 1,000 times per 1,000 environment steps, that is different from the common implementation. Therefore, we reproduce by using all parts that related to the prioritized replay in the official code from https://github.com/shlee94/Off2OnRL, and we use offline results of CQL for initialization with the ensemble size of 5, that is the same as the official paper and implementation.

All hyper-parameters are the same as the paper. And as O2O baseline methods are all off-policy methods, they are allowed to interact with the environment in 100,000 steps.

H.2 General implementation of our methods

Network Architecture As we need to re-evaluate policy, which means we only need offline policy and do not use the offline critic, we can modify the critic network architecture for stable online fine-tuning. In our implementation, we adopt the same network architecture as offline phase but apply Layer Normalization (LayerNorm) for the output of hidden layers after activation function.

[49] indicate that in offline RL, LayerNorm is a good solution to effectively avoid divergence without introducing detrimental bias, leading to superior performance. Moreover, [4] find that LayerNorm is favourable for efficient online RL with offline data. Therefore, for stable evaluation and future consideration, we decide to apply LayerNorm to online critic network.

Initialization of online replay buffer We test our method by initializing the online replay buffer with four different types: (1) Initialize the buffer with the entire offline dataset akin to [20]. (2) Conduct a separate online buffer and sample symmetrically from both offline dataset and online buffer akin to [4]. (3) Initialize the buffer with a small number of offline data with high quality akin to [48]. (4) Initialize the buffer without any offline data. For simplicity, we denote them as All , Half , Part , Null respectively.

As the results shown in 6, there is little difference among different ways of initialization, except for all , as the the quality of the offline dataset may be low. Although the initialization ways of part and null also show the competitive results, however, to be consistent with [4] and future efficient update, we decide to adopt Half as our implementation way. It is notable that online policy is still favourable even if we adopt Null initialization, thanks to our adaptive constrained fine-tuning.

loss weight for λ update Since λ in Eq. (20) is a variable applied in all states, it may decrease greatly in a batch. In order to considering more about the situation that needs to be constrained, we modify the term about λ in the loss Eq. (20) with a weight.

where ω = | 0 . 7 -I (( f ( π θ ( a | s ) , π ref ( a | s ))) > τ ) | gives a large weight to negative coefficients, thereby constraining the abrupt decrease of λ .

The choice of the initial value of λ As λ is the Lagrange multiplier which is adaptive to the constraint, there is no need to design the initial value of λ specially. For medium and large datasets of Antmaze tasks, we set the initial value as 2.0 since the initial performance is low, and we set the initial value as 2.0 for other tasks.

H.3 O2SAC implementation

The choice of constraint threshold τ As we consider the constraint in O2SAC is KL divergence, and we model policy as a squashed Gaussian distribution, we can directly the KL divergence of Gaussian distribution to approximate the constraint. Since π ref is the best one among old policies during online evaluations and it will be close to π θ with the performance improvement, we can assume that the standard deviation of π ref is the same as the one of π θ . Therefore, according to the KL divergence of Gaussian distribution, we can get:

Therefore, we can determine constraint threshold τ based on the magnitude of change in the mean. For medium dataset, we constrain that | µ θ -µ b | < σ , hence τ = 0 . 5 . However, due to different standard deviations, this threshold may not accurate, and it is an intuitive idea to loosen the constraint at a later stage. So we set τ as a linearly increasing variable from 0.125 to 2.0 for medium and medium-replay datasets, which means the allowable range of policy distribution mean is from σ/ 2 to

2 σ . And for medium-expert and expert datasets, we set it from 0.005 to 0.125 for safe update, which means the allowable range of policy distribution mean is from σ/ 10 to σ/ 2 .

Clipped log likelihood Due to the limit of precision of Pytorch and the squashed Gaussian policy used in O2SAC, when the output action a is near 1.0, like 0.99999999, the log likelihood log π ( a | s ) is will be severely low as the value of action will be computed as 1.0, which will occurs when the log likelihood is need to be computed by given a action in value alignment phase and constrained fine-tuning phase. Therefore, for simple calculation, we set the minimum value of log likelihood is -50, a enough small number, which has litter influence on results.

H.4 O2TD3 implementation

The choice of constraint threshold τ As TD3 models deterministic policy, it is unable to determine the constraint threshold according to the standard deviation. However, when we inspect the interaction process of TD3, we can find that exploration noise is akin to the standard deviation. So we determine τ from the exploration noise. First we set exploration noise as 0.1 for medium dataset, which is the same as TD3 learning from scratch. And for expert dataset, we set exploration noise as 0.05 because offline policy is well trained in such dataset, and a lower exploration noise is favourable to avoid poor action samples.

After determination of exploration noise, with the idea of the equivalence of the standard deviation and the exploration noise, we set tau similar to O2SAC. For medium and medium-replay datasets, tau grows linearly from 0.0025 to 0.01, which means the allowable range of policy distribution mean is from σ/ 2 to 2 σ , when we consider the exploration noise is equal to the standard deviation. Similarly, for medium-expert and expert datasets, we set it from 0.000025 to 0.000625.

H.5 O2PPO implementation

We implement our O2PPO basically according to the code of Uni-O4 [24].

The decay rate of β In our implementation for O2PPO, we set the interaction steps as 250,000 since PPO is an on-policy method which improves performance more slowly than off-policy methods. Since we keep the idea that in medium-expert and expert dataset, the offline policy is well learned, we decay β from 1 to 0 linearly in 500,000 steps, which means in the end of our fine-tuning, β = 0 . 5 . And for policies trained in other datasets, including AntMaze tasks and medium and medium-replay datasets of Mujoco tasks, we decay β from 1 to 0 linearly during 250,000 steps to reserve more potential of performance improvement.

Clipped standard deviation As shown in 14, the policy trained by IQL algorithm has a large standard deviation which makes poor exploration performance, especially in hopper-medium-expert-v2 and hopper-expert-v2 . Therefore, we set the maximum value of the standard deviations of policies trained in the two datasets as 0.05, which corresponds to the set of exploration noise in O2TD3, because for a deterministic policy, exploration noise and standard deviation of a stochastic policy are equivalent during taking actions in exploration.

Shaped and weighted auxiliary advantage For one-dimensional Gaussian distribution f = N ( µ, σ 2 ) , the entropy is log( √ 2 πeσ 2 ) , which is equal to log( f ( µ ± σ )) , hence the maximum value of A α ( s, a ) in (48) is 1 / 2 . However, the minimum value of it may be much low, which leads to unstable training. Therefore, we use SoftPlus activation function to clip the range of A α ( s, a ) . For each dimensional, we clip A α ( s, a ) as follows:

By such clipping operator, the range of A α ( s, a ) is ( -4 , 1 / 2) approximately. Note that when the current policy is close to the reference policy, the actions with overly low logarithmic values are rare, thereby having a little effect on the results.

Note that in the implementation of PPO, the normalization of advantages is needed. In order to make A α ( s, a ) and A ( s, a ) the same order of magnitude, we reweight A α ( s, a ) by multiplying a coefficient that is the double value of the standard deviation of A ( s, a ) . Therefore, the maximum value of A α ( s, a ) is the standard deviation of the batch of A ( s, a ) .









































I Connect Different Offline and Online RL Methods

Our methods permit connecting different offline and online RL methods as only offline policy is needed in our methods. Therefore, for those RL-based methods, which means the policy is modeled as a stochastic policy or deterministic policy in traditional RL way, it is easy to implement stable online fine-tuning by our methods. For those methods which model policy with different form, such as decision transformer (DT) [6], our methods can be applied easily if a stochastic policy or deterministic policy is used to clone the offline policy.

For behavior cloning of a stochastic policy, the easiest way is to Maximize log likelihood, however, we recommend the form akin to [53], as follows:

The purpose to maximize the entropy is to avoid an excessively narrow distribution, which may cause drastic jump of Q-values in O2SAC as α will decay to zero quickly. However, in our experiment, such a way of behavior cloning will leads to performance degeneration, which is a question for imitation learning, It is more easy for the behavior cloning of the deterministic policy, MSE loss can be used to update the policy for the states in offline dataset.

It is notable that degree of coverage has an important influence on behavior cloning, which is still an intractable challenge in imitation learning. Our methods provide the opportunity for the application of advanced imitation learning to O2O RL. In addition, even though the policy initialized with imperfect performance, our methods achieve better online performance than the online algorithm learning from scratch.

J Pesudo-Codes

Algorithm 1 O2SAC

Algorithm 3 O2PPO

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly claim our contributions on offline-to-online RL in both abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Since we re-evaluate the policy under Assumption 4.1, the policy should have favourable performance after offline training. And in Section C.2 in Appendix C, we further discuss why we do not focus on fine-tune a policy with poor performance.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the references of our assumptions and proofs for our proposed propositions in Appendix F.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose our algorithm details in Appendix G and our implementation details in Appendix H, including the implementation of the baselines and our methods.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We upload our code in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We disclose our experimental setting in Section 5 and Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We run our methods in 5 random seeds and report the results including the average and standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Since our methods do not focus on computational cost, and we experiment our methods in many devices with different GPUs, we do not provide the information on the computer resources specially.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics adequately and we make sure we followed the code.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use the open-source code and dataset, and they are explicitly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a general offline-to-online (O2O) reinforcement learning method that can work with any offline RL algorithm. It addresses evaluation and improvement mismatches between offline datasets and online environments by (1) Re-evaluating the offline critic optimistically; (2) Calibrating the critic with the offline actor; (3) Performing constrained online fine-tuning. This approach shows stable and efficient performance improvements across various simulated tasks compared to existing methods.

STRENGTHS:
1. This paper is well-written and easy to follow.
2/ The re-evaluation and calibration procedures are indeed very crucial for improving offline-to-online RL training, the authors make a great effort to show the importance of such procedure, with both empirical evidence and theoretical guarantees.

WEAKNESSES:
There are several minor presentation issues, not very critical, but they significantly affect the visual quality of the paper.
1. In line 86, equation 2, the subscription under the two expectations is getting too close, making the formula a bit messy.
2. In line 132, equation 9, the parenthesis of the first $f$ should be larger, so that it covers the inputs.
3. In figure 3, the colors of some curves are very similar, making it hard to tell the performance of each method.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the offline-to-online (O2O) reinforcement learning problem with the goal of improving online performance by leveraging offline data. The primary contributions of this paper are twofold. First, it identifies and elaborates on two key challenges in O2O RL: evaluation and improvement mismatches, which differentiate offline and online RL. Second, it introduces a general method for transferring knowledge from any offline approach to three representative online methods. Both theoretical analysis and empirical experiments thoroughly validate the effectiveness of the proposed method.\n\nSTRENGTHS:\nSignificance: In contrast to existing works, this paper is the first to summarize two mismatches between offline and online methods. These mismatches, which relate to two types of offline approaches, reveal their negative effects on subsequent online fine-tuning.\n\nContribution: The proposed method effectively balances specificity and flexibility. To address the two distinct mismatches, the paper introduces policy re-evaluation and value alignment techniques, which yield optimistic Q-value estimates and accurate Q-value calibration, respectively. Furthermore, the method is applicable to a broad range of representative online methods, demonstrating its wide applicability.\n\nSoundness: The experiments are thorough and strongly support the claims of the paper, including the motivation and effectiveness. Compared to multiple SOTA methods, the proposed method shows superiority.\n\nWEAKNESSES:\nThe method consists of three components. The first two components are developed to solve two mismatches. How about the third component? Is there another key issue in O2O learning tasks, such as another mismatch?\n\nIn value alignment, regarding different online methods, the paper develops different strategies. Does this imply that the method lacks generalizability?\n\nQUESTIONS:\nAlthough the method appears straightforward, it comprises three components. What is its time complexity? The paper is suggested to include an analysis on time complexity.\n\nLIMITATIONS:\nThe paper does not have a Limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper proposes to handle evaluation and improvement mismatches in offline-to-online RL. To this end, the authors suggest to re-evaluate the pessimistic critic and calibrate the misaligned critic with the reliable offline actor. Then, they perform constrained fine-tuning. They evaluate the performance in the standard offline-to-online RL benchmark.

STRENGTHS:
- The framework is theoretically analyzed.
- Experimental evaluation is extensive.

WEAKNESSES:
- Difficult to follow up on technical novelties. I suggest adding concept figures or algorithm tables to highlight core contributions.

QUESTIONS:
- Does re-evaluation require heavy computation? Is it more efficient compared to RLPD [1], which initialize replay buffer with offline dataset instead of offline pre-training?

[1] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." ICML 2023

LIMITATIONS:
See Weaknesses and Questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a general framework to bridge offline-to-online RL. It first studies two types of issues in O2O RL: evaluation mismatch and improvement mismatch. The proposed method addresses these issues by combining policy re-evaluation, value alignment, and constrained online fine-tuning. Unlike prior methods, the proposed framework can be applied to any offline RL algorithm. In the experiment, the method was built on top of CQL/SAC, TD3+BC, and IQL/PPO, demonstrating its performance in D4RL tasks.\n\nSTRENGTHS:\n1. The breakdown of the two mismatch problems is interesting.\n\n2. The proposed method is compelling in that it can generally bridge any offline RL and online RL algorithms.\n\n3. The results in D4RL locomotion tasks are solid.\n\n4. Overall, the paper is well-written and contains informative ablation studies.\n\nWEAKNESSES:\n1. [Major] It is a bit confusing to me which offline pre-training methods were used for each result. Is it correct that all the results from O2SAC are pre-trained with CQL, O2TD3 is pre-trained with TD3+BC, and O2PPO is pre-trained with IQL? I suggest mentioning these details more clearly in the paper.\n\n2. [Major] While the paper suggests that the proposed method is universal to the offline RL algorithm, there is not much comparison of using the same online RL algorithm with different offline pre-training methods. Can you provide ablations of running O2SAC on the same task with different offline methods, such as CQL, IQL, and ODT?\n\n3. [Major] While the results for the harder AntMaze tasks (antmaze-medium/large) are briefly mentioned in the appendix, could the authors provide the full comparisons to previous methods and add them to Table 1?\n\n4. [Major] Is it possible to include results on the Adroit binary task, as in [1, 2, 3], which is a common benchmark for studying the sample efficiency of online RL with offline data?\n\n5. [Minor] Figure 5 (a) is a bit confusing to me. Which plot corresponds to the unconstrained fine-tuning mentioned below?\n>(l.550) For O2SAC, unconstrained fine-tuning suffers from a performance fluctuation, and direct fine-tuning from offline may lead to faster performance improvement, but drops sharply in the subsequent phase, e.g. in Figure 5(a)\n\n6. [Minor] For the value alignment objective in Eq.13, can the authors explain more on why L_retain is needed? What will happen if we only use L_align as the objective?\n\n7. [Minor] It would be interesting to see if the proposed method can be combined into a recent sample-efficient online RL algorithm that can use a high UTD ratio, such as RLPD [3].\n\n8.  [Minor] The legend for Figure 3 (c) seems to be incorrect: O2TD3 -> O2PPO\n\n[1] Nair et al., AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, 2020\n\n[2] Nakamoto et al. Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning, 2023\n\n[3] Ball et al., Efficient Online Reinforcement Learning with Offline Data, 2023\n\nQUESTIONS:\nThe questions are included in the previous section.\n\nLIMITATIONS:\nThe limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}