Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
Authors: Qin-Wen Luo
Ming-Kun Xie
Ye-Wen Wang
Sheng-Jun Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
offline-to-online
Primary area: reinforcement_learning

Abstract: Offline-to-online (O2O) reinforcement learning (RL) provides an effective means of leveraging an offline pre-trained policy as initialization to improve performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. To deal with this problem, we disclose that there are evaluation and improvement mismatches between the offline dataset and the online environment, which hinders the direct application of pre-trained policies to online fine-tuning. In this paper, we propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method. Before online fine-tuning, we re-evaluate the pessimistic critic trained on the offline dataset in an optimistic way and then calibrate the misaligned critic with the reliable offline actor to avoid erroneous update. After obtaining an optimistic and and aligned critic, we perform constrained fine-tuning to combat distribution shift during online learning. We show empirically that the proposed method can achieve stable and efficient performance improvement on multiple simulated tasks when compared to the state-of-the-art methods.

Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offlnn-to-Onnlie RL
Qin Qin-Wen Luo"I, Ming-Kun Xie Ye-Wen Wang" Sheng-Jun Huang!
Nanjing University of Aeronautics and Astronautics, Nanjing, China 2 RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
{1uoqu8, waiisuaagg,,uaaa (raangs))@naaa denuaa.edu.co. ming-kun. kun.. xie@riken. jp
Abstract
Offline-to-onlie (020) reinforcement learning (RL) provides effective means of leveraging an offline pre-traine rre-tra policy as initialization to improve performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O20 learn- ing from any offline method. To deal with this problem, disclose that there are evaluation and improvement mismatches between the offline dataset and the online environment, which hinders the direct application of pre-trained policies to online fine-tuning. In this paper, we propose to handle these two mismatches simultane- ously, which aims achieve general 020 learning from any offline method any online method. Before online fine-tuning, we re-evaluate the pessimistic critic trained on the offline dataset in an optimistic way and then calibrate the misaligned critic with the reliable offline actor to avoid erroneous update. After obtaining an optimistic and and aligned critic, we perform constrained fine-tuning to combat distribution shift during online learning. We show empirically that the proposed method can achieve stable and efficient performance improvement on multiple sim- ulated tasks when compared to the state-of-the--eeeee art methods. The implementation is available https:/ggitubbcccmm/mmvveeeeuood
Introduction
Offline reinforcement learning (RL) aims to learn a policy from fixed dataset without additional interactions with the environment. This characteristic makes it particularly promising for critical applications such as healtheare decision-making (39], human-Al coordination [14] and autonomous driving []. Generally, the performance of the learned policy relies the quality of the dataset. Given that the offline data is limited, fine-tuning the policy through interactions with the environment is still necessary achieve favorable performance. Consequently, tolline-to-onliee (O20) RL tends to achieve faster performance improvements based on better initializations.
To effectively fine-tune offline policies, 020 methods are typically designed based on specific offline RL algorithms. Existing methods can be roughly divided into two groups according the base offline methods they use. The first group relies on policy constraint methods. These approaches aim to improve online performance by either adaptively adjusting constraints 5
52 43
or directly applying
offline algorithms to online fine-tuning inefficient performance improvement due to restricted action exploration caused by policy constraints. The second group builds on value regularization methods. These methods aim to prevent excessively
32 20). Unfortunately, these methods often suffer from
"The authors contribute equally.
Correspondence to: Sheng-Jur Jun Huang Mlangsj@nuaa.eda
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

low Q-values resulting from pessimistic evaluations, such as those in CQL 21. The goal is to enhance the generalization of the value function and mitigate potential performance declines [27 33
Some other methods adopt ensemble Q-learning [51 23 30 19 address these issues. However, these methods often face high computational costs due to the need to train multiple Q-networks.
Considering that the aforementioned methods develop online fine-tuning algorithms based on specific offline methods, they often struggle be applied other offline methods. To establish a general 020 framework, it is essential to address the core issues associated with transitioning from offline to online environments. Inspired of the recent work [48], which highlights the misalignment between the actor and the critic in an explicit policy constraint method, we identify two mismatches in general O20 RL: evaluation mismatches and improvement mismatches. Evaluation mismatches primarily occur in value regularization methods. These refer to the differences in policy evaluation methods between online and offline settings, which cause severe fluctuations in Q-value estimation during the initial stages of fine-tuning. Improvement mismatches, on the other hand, are prevalent in policy constraint methods. They are often caused by differences in objectives for updating the policy, leading to misalignment between the probabilities of actions and their Q-values. Thanks to another recent work by Xu [47, which connects value regularization and policy constraint methods, we bridge these two types of mismatches within unified framework for general RL-based offline algorithms.
In this paper. we propose a general O20 framework designed address both evaluation and im- provement mismatches simultaneously, aiming for stable and favorable fine-tuning performance from any offline method to representative online methods. To address the evaluation mismatch in value regularization methods, we propose re-evaluating the offline policy in an optimistic manner using an off-policy evaluation method. This approach allows us to obtain optimistic Q-value estimates, preventing the dramatic fluctuations in Q-values that could potentially cause the policy to collapse. Although the re-evaluated critic can estimate Q-values optimistically, it suffers from the misalignment with the offline policy, causing the improvement mismatch. To handle the improvement mismatch in the re-evaluated critics and policy constraint methods, we introduce value alignment to calibrate the critic so that it aligns with the probabilities predicted by the policy. Our approach involves using the Q-value of the most likely action as an anchor and then calibrating other Q-values by either exploiting the correlation between Q-values of different state-action pairs modeling Q-values a Gaussian distribution. Finally, we propose a constrained fine-tuning framework to guide the policy update by adding regularization term, with the target of mitigating the negative impact of data shift. Extensive experimental results on multiple benchmark environments validate that the proposed methods can achieve better comparable performance when compared to sate--of-the-art methods.
Our contributions can be summarized as follows:
* We systemically study that there exist evaluation and improvement mismatches for offline RL methods from the perspective of online RL. We show that resolving these two types of mismatches is essential for achieving general 020 RL.
We develop two techniques to address these mismatches. Policy re-evaluation aims to achieve optimistic Q-value estimates, preventing instability in Q-value estimation. Value alignment calibrates the critic to align with the policy, ensuring consistency between action probabilities and their corresponding Q-values.
We introduce a constrained fine-tuning framework that incorporates a regularization term into the policy objective, combating the inevitable distribution shift and ensuring stable and optimal performance when fine-tuning the policy in online environments.
2 Preliminaries
In this section, we introduce necessary preliminaries about RL, including Markov decision process, and three target online RL methods.
Markov Decision Process (MDP) A Markov Decision Process M is defined by the tuple (S.A, (38), where S is the state space, A is the action space, P : S A - ASS) is the transition function, R: S x A + IR is the reward function, μ is the initial state distribution, and y is discount factor. The goal is to learn a policy that maximizes the expected return as
(T) = E Σ t
>'r]] = Eo(eo..n) [Q (No: un)] 8(50, ao) = (s0 ~ 14, ag ~ n (-180)). (1)
2

---

Online RL The three most commonly used online RL algorithms are SAC [7]. TD3 [1, and PPO 37.. Below, we will introduce these three methods one by one.
SAC first introduces entropy maximization into the RL scenarios and updates the actor and the critic by minimizing the following objectives:
C (0)
E s-f amool-11))
E
[a log тo (als) Qu (s,a)],
(2)
LSAC (p:) (3,0,,,)")
E
[Qqm - (r, -s (r,
(3)
y 8') = r 7 CSAC (a) a E
minQp, (8, a) - a log no (a'ls))
1-122
E R am)Ta(11s)
[lo To (als) - -H].
(4)
where a: > 0, aa are the parameters of the target Q network, and 7c is the target entropy.
TD3 models deterministic policy and uses tricks, including clipped double Q-learnnn and policy
smoothing, to address the issue of function approximation error. The deterministic policy gradient used update policy is defined
V TD3 (0)
E S
Qu (a=m(s) VaTe (s)]
(5)
The objective function of updating the critic is defined
22
TD3 (μ4)
E (s,0,T,0)~ J
[Qum (s, (r, (r,
(6)
Here, y (r, min Qz. (s',a), a), where à Th (s') + and clip (N (0, o) cc. 1=1.2
PPO provides simple implementation for TRPO 35 which updates the actor and the critic by minimizing the following objective functions, respectively,
CPO (0) =
min
ccip -- A (0,
swap T e (-1s)
PPO (v)
E [AAT (8, Vo V. (s)) (s.r,8')~RR
(8)
where r0) To/ nT, and the advantage Th is computed by GAE [36..
is noteworthy that previous works mostly adopted SAC and TD3 for online fine-tuning as they off-policy methods. In our work, given the widespread use of PPO in online RL, we also employ
for fine-tuning though is an on- on-policy method.
Evaluation and Improvement Mismatches
In this section, we focus on the differences between online and offline RL and study how these differences impact the performance of online fine-tuning. In general, we summarize these differences as two types of mismatches, evaluation mismaich and improvement mismatch. The former arises from the changes in policy evaluation functions during the transition from offline to online environments; while the latter represents the inconsistency in the objectives for policy updates between offline and online RL. Most offline RL methods suffer from one or both these issues, which underscores the importance of addressing these mismatches to achieve stable and effective online fine-tuning.
Evaluation mismatch often occurs in the value regularization methods. For example, in CQL 211, representative offline method, the policy evaluation function transitions from a pessimistic estimation inherent to offline learning to more optimistic estimation during online training. This shift frequently results in sharp increase in Q-values at the beginning of online fine-tuning, which can hinder stable performance improvements. address this problem, several attempts have been made mitigate the excessive underestimation of
(OOD) actions during offline learning [33 27

---

or to initialize pessimistic Q-ensemble to maintain pessimism during online fine-tuning [23] Unfortunately, these approaches are predominantly tailored to the transition from CQL SAC, limiting their applicability to other offline algorithms.
Improvement mismatch is commonly found in policy constraint methods. Typical examples include TD3+BC [10] and AWAC (32), where the objective of actor updates differs significantly from typical online methods. In these models, updates the actor are not solely reliant on the critic's evaluation. Consequently, actions that have high Q-values may not automatically translate to high probabilities of being selected, and vice versa. This divergence often misguides the update of policy at the beginning stage of online fine-tuning, resulting in unfavourable performance.
Besides, One-step RL and non -en-iteratvve methods, e.g., IQL, exhibit both evaluation and improvement mismatches. During the policy evaluation stage, these methods estimate the Q-values based on the behavior policy
unknown policy [18] instead of the target policy as in online methods; during the improvement stage, because they impose additional constraints on policy updates, they also encounter the same issue as discussed above. This can lead to discrepancies in both the assessment of action values and the subsequent policy optimization, making it hard to achieve effective policy improvement.
Thanks to the recent work [47], which presents a unified framework for understanding offline RL, bridge these two mismatches for general RL-based offline algorithms. Formally, the offline RL problem can be defined by the behavior- reguurio-regularrzed MDP problem [47 13 via maximizing the following objective:
-a. Σ 1= Ennor' 7 (r(stna)- - (((((())))) [u(ap]s)) ))]
B8(s0.00) Q (so, (s), ao) - -a.f ( (an) (su) (ag (so) )
(ao]ss)
E
where f(.) is regularization function and ju is the behavior policy.
From Eq. 1 and Eq. ( we observe a significant divergence in the relation between the actor and the critic in online versus offline scenarios. Unlike in online cases where the policy update is solely dependent on the Q-function, in offline scenarios, also critically depends on the data distribution, as indicated by the regularization term in Eq. (9). This distinction highlights why offline methods often face evaluation and improvement mismatches when applied to online fine-tuning. Using offline actor and critic trained by Eq. (9) for initialization in online fine-tuning introduces both types of mismatches, resulting in unstable and inefficient updates.
4 Method
In this section, we introduce our proposed 020 method for handling the mismatches discussed in Section 3 and the distribution shift problem. To address the pessimistic or unreliable evaluation, such in CQL and IQL, we develop policy re-evaluation technique. This technique optimistically re-evaluates the well-trained offline policy using an off-policy evaluation method. Although this re-evaluation helps the critic achieve more optimistic Q-value estimates, unavoidable factors such as function approximation errors and partial data coverage can still lead to a misalignment between the critic and the offline policy. This misalignment means that the action with the highest probability predicted by the policy does not necessarily have the highest Q-value, leading to what we have termed improvement mismatch.
As discussed in in Section 3 both the value regularization (after re-evaluation) and policy constraint methods exhibit improvement mismatch, though the reasons for the mismatch differ between these two approaches. To address this issue, we propose value aligmment, which aims to align the critic's s estimates with the policy's action probabilities, effectively tackling the improvement mismatch in both types of methods. Finally, to deal with the inevitable distribution shift between offline and online environments, we develop constrained fine-tuning framework. This framework ensures that the policy consistently updates in the optimal direction by incorporating a regularization term into the policy objective.
We propose methods for various representative online RL algorithms within a unified framework, including SAC [77. TD3 [1I, and PPO 337 These algorithms represent the mainstream approaches in online RI,, and are targeted respectively for the off-policy approach with stochastic policies, the off- off-policy approach with deterministic policcie,, and the on-policy approach.

---

4.1 Policy Re-evaluation
The critic trained on an offline dataset typically maintains pessimistic estimates of Q-values. When using online evaluation methods to fine-tune this critic without any value regularization, the Q- values can experience dramatic jump, especially for OOD actions, leading inaccurate Q-value estimations. To mitigate this problem, we propose re-evaluating the offline policy to acquire new critic by employing an off-policy evaluation (OPE) method. The goal is enable the critic to have optimistic estimates of Q-values that more closely approximate the true values.
However, directly applying OPE methods for policy evaluation on offline datasets often leads to large extrapolation errors, as discussed in the previous work [2]. These errors arise due to absent data and training mismatches. Thanks to the pessimism in offline RL, reasonable assume that a well-trained policy is close enough to the behavior policy or even captures the support set of the behavior policy. A common assumption is single-policy concentrability 45 (4], which demonstrates how concentrated a learned policy is within the given dataset and can be defined as follows.
Assumption 4.1.
(me-concentrability
45
The behavior policy p and learned policy vo satisfy
d (8,a) SC. d' (s.
max (s.a)ExXA
where d (s, a) is the occupancy measure of mo and C' is constant. Single-policy concentrability measures the degree to which the state-action distribution induced by the learned policy is covered by the dataset used for training. By ensuring that the learned policy does not deviate significantly from the behavior policy, the extrapolation error can be greatly reduced [22 29]. When using a
representative OPE method called fitted Q-evaluation (FQE), based on Assumption 4. and Theorem 4.2 in (22), the upper bound of extrapolation error can be obtained.
Corollary 4.2. Under Assumption 4.1 by denoting Q-value function class as F. for 8 E (0. 1), after K iterations of FQE on the dutasel D, with probubility 1 8, we have:
1-7 K T-7 VC KV, where : 22V2 log( [ F1/8)
1Q- - Q-1
+ 20d
[
where di. is inherent Bellman evaluation error (Definition 4.1 in 22 and V is the maximum of V. which can be bounded by Rmaz/ (1 - 7).
With powerful neural network and sufficient data, the inherent Bellman evaluation error could be tiny. Accordingly, with a large training step K, the error will be bounded by an acceptable value. This implies that, given sufficient data, one can achieve a critic with optimistic property and minor extrapolation error through policy re-evaluation. In practical implementation, for off-policy methods SAC and TD3, can directly use Eq. (3) and Eq. (6) to re-evaluate the offline policy. For the on-policy method PPO, we train a critic by fitting the returns of offline trajectories. Considering that the critic can only approximate VP(s) rather than the true value function, we propose regularization term in Section 4.2] which ensures the critic's estimates are reliable and conducive effective policy improvement.
bopper mne diam- +2
fied critic
Nerated critic
allgned orite
Traidig (ops
(a) The performance of SAC
hopper aed medium
Esed crilic
4.2 Value Alignment
allgaed crtiic
offline
Although the critic after policy re-evaluation possesses the opti- mistic property needed in the online environment, often does not align well with the offline policy due to factors such as function approximation errors, generalization errors of neural networks, and partial data coverage. Moreover, as discussed in Section 3 policy constraint offline methods also suffer from misalignment between the critic and policy. The misalignment means that the
Training Stops (-Sed))
(h) The performance of TD3
The results of actors updated with different critics.
Figure 1:
action with the highest Q-value does not necessarily have the highest probability, often leading to misleading updates of the policy. To verify this observation, we trained the policy using SAC and TD3
5

---

with three different crities: the fixed re-evaluated critic, the iterative re-evaluated critic (which updates with the policy), and our aligned critic. From Figure the performance of re-evaluated critics sharply declines at the initial stage and does not recover in the subsequent training; while our aligned critic achieves stable and favorable performance. These results indicates that the misalignment between the re-evaluated critic and the offline policy can make it difficult for the policy to optimize in correct direction, leading to an irreversible decline in performance. Given that the well-trained offline policy is reliable, the desirable critic should not only have optimistic Q-value estimates but also maintain alignment with the offline policy. To achieve this, we propose performing value alignment to calibrate the critic. The main idea is use the Q-values of the offline policy actions anchors, keeping them unaltered, and to suppress any overestimated Q-value that exceed these anchors, thereby aligning the critic with the offline policy. Below, we will discuss how to implement value alignment for different online methods.
O2SAC Recall that in SAC []6 171, the optimal policy is defined
(als) = exp( 1 Q(s a))/ /xppp (epp--Ve Vs))
1
(10)
With a simple transformation of Eq.
10 we have
Q(s, V(s) a log (a|s)
(11)
One intuition behind our method that actions with higher probabilities typically have more accurate Q-value estimates because these actions closer to the dataset, and there is sufficient data nearby to obtain a precise estimate. This motivates us to use the Q-value Q (s, à) of the optimal action à to calibrate any other overestimated action. Assuming that noa is the offline policy, perform value alignment for any state-action pair (s, a) as follows:
Qg(s, a) = min (Qn(s, a) a (log Toir (āls) - log Ton (a|s)) Qp (s, a)),
(12)
where min(.:) -) operator is used to maintain the Q-values of actions that are not overestimated and consistent with OPE results.
Formally, we define the objective function of value alignment as follows:
Calign (Ju)
TE [QQm (8, a) - 2/ (s, ()) "]]
s--.a--((1ss)
2 (ni) = E s-12
Cretain
(Qpm (s, Qa (s, a))² (a=
(13)
Caniti (ui) -Cllsu (u.) + Lo Cretain (p.)
Considering that the overall Q-values have decreased, to maintain an optimistic estimate, we use the regularization term Cretain to prevent the underestimation of Q-values.
We derive Proposition 4.3 to show that the aligned state values Valign() can be maintained within an appropriate range, meaning that the estimates remain optimistic while avoiding overestimation.
Proposition 4.3. After the value alignment process of Eq. 13). the state value function Volige (8) satisfies
(sye(s < Vadign (s) Va(s)
(14)
where Qja (s, a) are the low Q-values after policy re-evaluation. which do not require calibration, (s) (8, a log क (als) and (a)= ((+ à) -a logog x (áls).
During value alignment, we update the policy with Eq. 2 simultancously for sampling the overes- timated actions. The whole process iterates Eq. (2) and Eq. (13) to obtain a policy that performs cquivalently the offline policy and critic aligned with We denote the policy as non and the aligned critic as Qun for sequential training. Note that Q which is modified from the critic obtained in the policy re-evaluation, does not depend on specific offline critics. This flexibility allows to implement the transition to SAC from different offline algorithms.
02TD3 As done in the case of O2SAC, we can use the Q-values of the policy actions to maintain the optimistic property of policy re-evaluation and calibrate the Q-vauues of other actions.

---

Unfortunately, in TD3, the actor is modeled as deterministic policy, lacking an explicit expression for Q-values, which prevents us from directly aligning Q-values with the policy.
To solve this problem, our main idea is to model the distribution of Q-values around the policy action à Gaussian distribution. Specifically, in Eq. (5), when we the offline policy as nh, the gradient of the policy is only related to the the gradient of Q(s, a) with respect to a. It is easy to see that the gradient around should tend 0 as is the optimal action selected by the offline policy, Due to the use of smoothing regularization in the critic's update in TD3, the Q-values of actions near à differ only slightly from the Q-value of itself. This enables us to assume that normalized Q- Q-values around à follow Gaussian distribution ((. a)/aa//// 1Q((ssss à) ~ N(ē, E). Formally, we can calibrate the Q-values of other actions as (see Appendix G.2 for detailed derivation)
Q(s, ())
a'ss.a) =min min (Qu(s,a), 1+ k. max (d(a,
(15)
where k is a constant, which set as across all tasks, and d(a, a) a distance measure, which is defined as the euclidean distance divided by the square root of the action dimension in our implementation.
From Eq. (15), after calibration, the Q-values of the actions around à are only slightly lower than Q(s, å), which ensures that à is the output action of the policy while maintaining smoothing and optimistic property of Q-values. Moreover, for the actions that differ greatly from á, we limit the maximum of the distance measure d(a, à) in Eq. (15) to the policy noise used in Eq. (6) to avoid severe underestimation of their Q-values. Formally, we define the objective loss of value alignment for O2TD3 as
calen (ni) = E
[Qmm (s, - Q2 (s, ]la=x(+)+8 (I,aa))la===())))
swi
L Cggtann (p+) = E
[0pm (s. à) - 2p (s. (s, lamma(ss)
(16)
Coritic (M:) = Lo (μi) + L (pi)
where Y2 (s, =
â), 2j (s, a)) and à is perturbed action defined in Eq (6)
O2PPO In PPO, only the critic Vss) is used to estimate the advantages for policy update. During the re-evaluation process, we train critic by fitting the returns of offline trajectories mentioned in Section[4.] This indicates that the re-evaluated critic only approximate (*(s) instead of the true one, misguiding the update of the policy. To mitigate this problem, we propose an auxiliary advantage function to correct erroneous updates.
Generally, a desirable auxiliary advantage function should satisfy the following two conditions: 1) it enables the policy to update in a reliable region; 2) its value must be zero at the beginning of online fine-tuning to enable the policy transition smoothly from offline online. Considering that the well-trained offline policy is reliable, we define the auxiliary advantage function as
Aa(s, a) = a log (a)(s) + a<H(TTo((1ss)
(17)
where H is the entropy of action probabilities predicted by Tof-- It easy to verify the second condition that (s, a) = 0 at the beginning of offline fine-tuning. To verify the first condition, we derive the following proposition. Its proof can be found in Appendix [
Proposition 4.4. With (s, a) in Eq. (17). the policy update is regularized by the cross-entropy loss about the offline policy, thereby constraining the policy update in a reliable region.
Accordingly, we define the advantage function for policy update as
A'(s, a) = + (s, A(A,,))
(18)
where 33 anneals to 0 from 1. With the auxiliary advantage, Eq. (88 prevent the update direction from deviating too far from the offline policy.
4.3 Constrained Fine-Turee Tuning
In the previous subsections, we discussed how to address the mismatch issues in the 020 problems. However, due to the distribution shift between the offline dataset and the online environment, along
7

---

AWAC
JQL
PEX
OBE20n
Cal. QL
ACA
02SAC
O2TDS
ballchactallmmdiineee
hal halficheetal-mmdiieee -iddbeeolllmmdddm---e-------
ball@heetal-eppeet-2
Nomallzd Regumm
Normalteed Reeumm
Normalnze Rotuns
Normalized Ioetun
1ea
Evaliation Epoch
Evaluation Epoch
Evaluction Epoch
Evaluation Epoci
happer medium-v2
Moppoe-medium-oee replly------22222
bogpe: medium expert-v2
hopper exppre--epet-2-222
Nommalzza Remm
Normalizod Retur
Namallzz Remu
Nommalzzd Renum
&
50
Epoci
300
Evvaluatonn Epoch
100
Evalustion Epoch
100
Epoch
300
walkker2d medium-r2
wallker2d 2edium-reepla-vv2
walker one dium expert-s2
walkerr2d valler2ldexxerrt22
Nomalizo Rolum
Nomalized Ratmm
Nomulizel Raum
100
Nomullzaa
Toa
82.
60-
Evalaation Epuci
Evalustion Epoch
Exaluation Epoch
100
Evaluation Epoch
100
Figure 2: Performance curves on D4RL 9
MuloCo locomotion tasks during online fine-tuning.
with the optimism in online RL, encountering out-of-distribution (OOD) states and actions becomes inevitable, potentially leading to signifieant performance fluctuations. Especially for OOD states that are absent in the offline dataset, even though the policy was trained well in the offline phase, it may still fail to output favourable actions, thereby causing erroncous policy update.
Considering that the critic maintains an optimistic nature after undergoing policy re-evaluation and value alignment, with the optimistic update way during online fine-tuning, it typically overestimates the Q-values of OOD state-action pairs, which can mislead the update of the policy. Inspired of CMDP i we develop constrained ane-ttnnig (CFT) to introduce regularization term to constrain the current actor and critic, which prevents the policy update from being severely misguided.
Specifically, we impose constraint term f(x, Thet) on the policy objective to ensure that it updates within the credible region of Tet where f(...) a divergence measurement and ne is the optimal historical policy during online evaluations. Formally, define the policy objective of CFT
E Σ (rre(st
s.t. I2,,, (/ (:), Trrraa [sc))] <T
(19)
By incorporating the constraint term into the reward function akin RCPO we can solve the problem by minimizing the following objective functions with initializing as obtained in Section 4.2 at the beginning of online fine-tuning.
L (0) max E,,, Q (s, a) (f(s)(als) (mr((s)))
To (1)) = min E (eann))) (QQ (s, y)
(20)
= + ( 1 (s', a') Af(To(a'\s') s'), (ar(a'|ss))
T(A)= = min - [E (f (a) (s), Tre(())s))) -
A
We provide a theoretical guarantee for the proposed CFT framework.
Corollary 4.5. With the penalty f Teez) defined before and appropriate learning rates, algorithu of Eq. (20) almost surely to a fixed point (6* _p",, 1,",X, X*), where X O, 0* and μı" are corresponded to nt and Q", which optimal in the MDP without constraint.
In our implementation, we use KI. divergence and MSE function as f(..)) for the transitions of O2SAC and O2TD3, respectively. For O2PPO, the auxiliary advantage function already has the

---

Table Average normalized D4RL scores of our methods shown in AntMaze navigation tasks after 200k interactions with the environment. (U=umaze, D=diverse)
Dataset
IQI.
PEX
Cal-QI.
O2TD3
O2SAC
O2PPO
U-v2
80.8 +80.8
85.0-496.2
80.8 +97.0
92.8 +95.8
92.8 -93.6
77.3-+98.0
U-D-v2
56.6 +35.8
12.6->16.0
23,8
38.4-+52.2
+3.88
56.4-18..3
total
137.4-11166
97.6-+112.2
104.6->168.2
128.6-144..0
131.2-114..0
133.7-1184.
peh-cxpert-rl
walker26-mediumm22
untmaze umitze diverse-22
50-
ODT
Nermalkzed Rerurn
Nor mormalted Return
Normalteed Return
ODT
02SAC
SAC from scrutch
O2TD3 from OUT
ODT
Environment Steps(x1e()
Eavironment Steps(-1e3)
Environment Steps(
(a) From TD3+BC SAC
(b) From online DT TD3
(c) From online DT to PPO
Figure 3: The fine-tuning performance achieved by transferring to three online algorithms from their heterogeneous offline algorithms.
ability to constrain the policy update, we only need to replace Toll with Tee during online fine- tnning. Note that in our methods, the beginning of online fine-tuning, the regularization function (mT(a)s) s), Ton (a]s) is zero for any state, which guarantees no destruction of the alignment for the actor and the critic obtained in Section|.2
Experiments
In this section, we perform experiments to validate the effectiveness of the proposed method on D4RL [9] MuJoCo and AntMaze tasks, including HalfCheetah, Hopper, Walker2d and AntMaze environments. Specifically,
compare methods with AWAC [32. IQL 220. PEX (50), Off2On [23], Cal-QL [33] and ACA [48). For all methods except for O2PPO, we run 100,000 interaction steps with the environment and evaluate the policy per 1000 steps, they all use off- off-policy methods for online fine-tuning. For our O2PPO method, due to the low efficiency of the on-policy method, we 250,000 interaction steps to validate its performance, with 2500 steps the evaluation interval. We run all methods with five random seeds and report their averaging results. Due to the space limitation, more experimental results can be found in Appendix B and
5.1 Comparison with State-of-the-Art Methods
We respectively initialize the policies of O2SAC, O2TD3 and O2PPO from the results of CQL, TD3+BC and IQL. Figure shows the performance curves our methods and comparing methods on Mujoco tasks. Similar to the previous works [48] [ for O2SAC and O2TD3, we use CQL and TD3+BC for offline policy pre-training. From the figure, we can see that our method more stably and rapidly than other methods and achieve the optimal performance in most cases. Although the ensemble method Off2On can achieve better final performance than our methods in some cases, it often suffers from dramatic drop in performance at the beginning of fine- the-tunning, which is often unacceptable in the 020 problems. We report the results of O2PPO in Appendix A because of its different number of interaction steps. Although PPO is on-policy method with low efficiency, from the results in Table T shows significant superiority on sparse reward tasks, even though with equal interactions.
converge
5.2 Study on Transferability
In this section, we perform experiments to verify the powerful transferability of the proposed method. As mentioned earlier, one of the advantages of our method is that it imposes no requirements offline

---

algorithms. This means we can achieve the transfer from any offline RL algorithm to three online RL algorithms. F Fillustrates the fine-tuning performance of three online algorithms transferring from their heterogeneous offline algorithms. From Figure pre-trained with TD3+BC, O2SAC outperforms SAC trained from scratch with larger margin. Although Online Decision Transformer (ODT) achieves favourable performance in the offline environment, it converges slowly during online fine-tuning due to the architecture of the transformer. Our O2TD3 and O2PPO significantly enhances its performance through online fine-tuning. These results convincingly verify that our method show the strong transferability from various offline methods.
6 Conclusion
In this paper, we disclose there exist two types of mismatches when online fine-tuning offline RL methods. To address these two mismatches in 020 RI., we proposed optimistic critic reconstruction to re-evaluate an optimistic critic and align it with the offline actor before online fine-tuning, ensuring the stable performance improvement the beginning stage and potentially better efficiency due to the reconstructed optimism consistent with online RL. Furthermore, to combat the inevitable distribution shift that can hinder the stable performance improvement, we introduce constrained fine-tuning to constrain the divergence of current policy and the best foregoing policy to maintain the stability of online fine-tuning. These two components form a versatile O20 framework, allowing the transition from any offline algorithms to three state-of-the-art online algorithms. Experiments show our framework can converge to optimal performance without affecting the aligned critic the beginning of online fine-tuning and achieve strong empirical performance.
Acknowledgements
This work supported by the NSFC (62222605), the Natural Science Foundation of Jiangsu Province of China (BK20222012, 1517), and the National Science and Technology Major Project (2020AAA0107000),
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
[3] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in Neural Information Processing Systems, 35:28955-28977 2022.
[4] Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, arXiv preprint arXiv:2302.02445 2023.
[5] Alex Beeson and Giovanni Montana. Improving td3-be: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211. 2022.
[6] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14:45099-55097, 2021.
[7] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint / 008557700000,...0111 2018.
[8] Christopher Diehl, Timo Sebastian Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. pnccertainty-wwar model-based offline reinforcement learning for automated driving.
IEEE Robotics and Automation Letters, 1177:1167-1174 2023.
10

---

[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep deta-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[10] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132-20145, 2021.
[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine leurning, pages 1587-1596. PMLR, 2018.
[12] Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[13] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent without entropy. arXiv preprint arXiv:2301.02328, 2023.
[14] Cong GUAN, Ke XUE, Chumpeng FAN, Feng CHEN, Lichao ZHANG, Lei YUAN, Chao QIAN, and Yang YU. Open and real-world human-ai coordination by heterogeneous training with communication. Frontiers of Computer Science, 19(4):194314 2025.
[15] Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang, Hechang Chen, Haiyin Piao, Lichao Sun, and Yi Chang. A simple unified uncertainty -ruiddd framework for offline-to-oniine reinforcement learning. arXiv preprint arXi::2306.07544 2023.
[16] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352 1361. PMLR, 2017.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[18] Philippe Mansen-Estrucl Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[19] Ingook Jang and Seonghyun Kim. Uncertainty-driven pessimistic q-ensemble for offline-to- online reinforcement learning. In 3rd Offline RL Workshop: Offline RL L"Launchpad", 2022.
[20] Iyya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. In International Conference on Learning Representations, 2021.
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[22] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning. pages 3702-3712. PMLR, 2019.
[23] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-tee tofiie--to-oninee reinforcement learning via balanced replay and pessimistic geensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
[24] Kun Lei, Zhengmao He. Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:231 1 2023.
[25] Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint 156., 2023.

---

[26] Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. arXiv preprint arXiv:2303. /7396, 2023.
[27] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711 1-1724, 2022.
[28] Yihuan Mao, Chao Wang, Bin Wang, and Chongjie Zhang. Moore: Model-based offline-to- online reinforcement learning, arXiv preprint arXiv:2201.10070, 2022.
[29] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829-23851. PMLR, 2023.
[30] Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
[31] Trevor Mclnroe, Stefano V Albrecht, and Amos Storkey. Planning go out-of-distribttion in offliee-to-nllnee reinforcement learning. arXiv preprint 1rXiv:2310.05723, 2023.
[32] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09339, 2020.
[33] Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint 233::23333300777 2023.
[34] Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 14:11702-11716, 2021.
[35] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[36] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1506.02300002333 2015.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.000. 0.34477777 2017.
[38] RichardS. Sutton and AndrewG. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, page 285-286, Jan 2005.
[39] Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Veles and Jenna Wiens. Lever- aging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35:34272-34286, 2022.
[40] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL "Launchpud". 2022.
[41] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprini arXiv: 1805 1074, 2018
[42] Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et Jump-start reinforcement learning. In International Conference on Machine Learning, pages 34583.. PMLR, 2023.
12

---

[43] Shenzhi Wang, Qisen Yang, Jiawei Gao. Matthieu Gaetan Lin, HAO CHEN, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get family: State-adaptive balances for offline-to- online reinforcement learning. In shirtt-seventh Conference Neural Information Processing Systems, 2023.
[44] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning, Advances in Neural Information Processing Systems, 35:31278-3333 -12291, 2022.
[45] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27407, 2021.
[46] Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:4085- 4098, 2022.
[47] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint 1xXx:22303.18100 2023.
[48] Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In International Conference on Machine Learning, pages 40452-40474. PMLR, 2023.
[49] Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving valuu divergence in offline-rl. arXiv preprint arXiv:2310.04411, 2023.
[50] Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-onling reinforcement learning, arXiv preprint arXiv:2302.0000 2023.
[51] Kai Zhao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Ensemble-based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration. arXiv preprint arX1v:2306.0687J, 2023.
[52] Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behav- ior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210.1338 138.66666, 2022.
[53] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. international conference on machine learning, pages 27042-27059, PMLR, 2022.
13

---

A
Detailed Data
Table 2: Average normalized D4RL scores of 020 methods shown in Figure 2
Outside parenthesis: scores the end of 100k online steps. Inside parenthesis: the increase of that score upon the end of offline training. (M=medium, R-replay, E=expert)
Score(6)
Dataset
Off2On
Cal-QL
ACA
O2TD3
O2SAC
Hc-M
94.22(47.37)
54.36(6.81)
66.59(20.09)
65.58:10.7(17.455)
74.18:10.5(27.33)
Ho-M
90.34(29.85)
92.83(33.92)
99.76(41.08)
100.25=1.1(43.55
97.29:18.(38800
Wa-M
98.2(17.55)
83.93(1.06)
82.55(5.18)
96.32=1.6611.71)
106511=2.1(5..)))
Hc-M-R
88.74 (43.18)
49.36(3.31)
57.93(15.71)
18.58=1.8(14311)
71.65=0.9(26.7))
Ho-M-R
103.61(4.93)
98.64(2.58)
102.19(50.3)
105.12 1.4.22)+144(3.22)
Wa-M-R
102.43(20.78)
91.61(9.36)
85.65(8.86)
(...8:++22((1..644
106.05++28((8.75))
Hc-M-E
98.33(2.67)
92.67(43.62)
93.54(0.11)
1..07=11.14.90)
100.41=0..866220
Ho-M-E
99.47(14.65)
108.57(4.57)
109.72(14.17)
112.49=0.9(10.8)
107.4=6.0(23255))
Wa-M-E
118.01(8.51)
110.3(1.21)
110.36(2.92)
112.9710.4(2.33)
120.9510.6(11555)
Hc-E
100.99(4.59)
96.9(1.01)
87.93(-6.65)
99.96.10.4((155))
107.74
Ho-E
92.13(-19:44)
110.84(5.1)
107.79(-1.98)
112.410.5(0.66)
112.76:1.440.955)
Wa-E
118.02(8.3)
109.71(0.76)
108.2(0.21)
114.67=11114552)
820.35=0.8(10.5))
Total
(204.5(182.93)
1099.73(113.3)
1112.22(150.0)
1171.66(170882)
(230.41(225.54)
Score
Dataset
AWAC
IQL
PEX
O2PPO
Hc-M
51.11(1433
(8.83().33)
57.05(8.53)
59.96=0.8(11.54)
Ho-M
82.42(16.53)
60.48(-1.62)
95.7(33.59)
100.42=:1.148.661
Wa-M
87.03(1.50)
83.87(1.34)
88.18(5.65)
86.65-+-2.2(000))
Hc-M-R
47.81(2.03)
43.42(0.32)
45.44(2.34)
46.76+ 1.14.22)
Ho-M-R
98.86(0.07)
95.56(5.16)
101.75(1 1.35)
96.87-14.1(18))
Wa-M-R
86.16(6.49)
84.75(5.42)
94.1(14.78)
90.4315.1((116))
Hc-M-E
95.37(0.57)
(2.54(0
84.99(-7.3)
1..61_)..(
Ho-M-E
109.76(2.53)
108.29(11.23)
97.17(0.11)
107.61::5.48898)
Wa-M-E
106.87(1.23)
112.57(0.67)
116.47(4.57)
121.440..(9.35)
He-E
97.16(0.42)
96.24(-0.56)
96.69(0.11
96.2:10.7(2.77)
Ho-E
109.88(1100)
97.15(-5.33)
71.37(-31.12)
113.08=004(9.
Wa-F
111.31(0.64)
113.43(1.31)
97.54(-14.57)
117.49-009(9922)
Total
1083.74(31.44
1037.15(18.53)
1046.45(27.83)
1126.48(138.55))
Compared with the baseline methods, our methods outperform much D4RL Mujoco locomotion tasks, as showned in Table 2 We can compare the fine-tuning performance in groups based on online update way. Off2On, Cal-QL, ACA and our O2SAC are updated in the SAC way during online fine-tuning, Although there is lack of the baseline methods that update the policy in the TD3 way, we compare our O2TD3 with a recent 020 method PROTO+TD3 [25) in Appendix C.4 to demonstrate the superiority of our methods. And we compare O2PPO with the policy constrant methods AWAC, IQL and PEX, that are implemented in the IQL way, since the idea of PPO is similar to kind of policy constraint.
With such groups for comparison, our methods get the best performance improvements respectively. Note that in our implementation, Off2On achieves much better performance than the original paper 23 and the implementations other papers [48] and [50). However, our O2SAC still outperforms with less computational cost during online fine-tuning and less requirements for offline policy. In
14

---

addition, our methods bridge different offline algorithms and three SOTA online algorithms, and permit additional modifications for improved algorithms based on the online algorithms.
O2PPO
Direct PPO from IQL
halfcheefahh-m edlam-222
halfcheetnb: medium -eddamm-emuuuuulpp---- replay-22
halfebretahh medlum- exclem-eettt
mallsheetabbexpertte
Normalize Beturs
Normaitzes Rotura
Normalized Roturn
Normalized Roturn
Evaluation Egoch
30
15
100
Evalustion Egoch
100
Evalustion Epuch
190
Evaluation Epoch
50
390
mmppor-mmdiia--22
bogpa-mmdiin-eeplyy-
hopper-med sapperr-edimm-xpprtee
bopper-expertt222
Normattrod Return
Normelized Return
Normelized Return
tee
10o-
Normaltral
Evaluation Epoch
Evaluation Epoch
Evaluatoo Epuch
Evalration Epoch
399
walker 24-medium-42
walker 20-expert-22
Normaltned Return
Normatined Return
Aormalied Return
Nermalized Retali Return
120
Evaluation Epoch
Evaluation Epoch
Evaluation Epoch
Evaluation Epoch
Figure 4: Performance of our O2PPO and direct PPO from IQL on D4RL 9 MuJoCo locomotion
tasks during online fine-tuning. The solid lines and shaded regions represent mean and standard deviation.
Due to the low sample efficiency of PPO, we run 250,000 interaction steps for the implementation of O2PPO, which makes it unreasonable to compare the results of different 020 methods in one graph. So we demonstrate the results O2PPO without any baseline method, but compare our method with the direct way of fine-tuning the offline policy of IQL in the PPO way in the online phase. Note that our purpose is to achieve stable performance improvement, the efficiency depends mainly on the in some environments, such as
online algorithm, improvement may be less than the direct fine-tuning. This phenomenon could arise due to the critic's capability approximate the genuine values effectively, thereby facilitating accurate evaluation of the policy, obviating the necessity for corrective adjustments in the policy update direction. And O2PPO uses auxiliary advantage function constrain the policy update, resulting in slower performance improvement. However, since we cannot judge when the critic is reliable, it is necessary use O2PPO to achieve stable performance improvement, and there is no remarkable discrepancy between O2PPO and the direct way, although in the above scenarios. In general, our O2PPO can achieve stable and efficient performance improvement with limited interactions.
the performance
B Ablation Study
Before the analysis of ablation studies, we need to clarify the roles and applicability of the different components of our methods for different offline methods.
The benefit and applicability of policy re-evaluation As talked before, the purpose of policy re- evaluation is to get optimistic valuue functions, avoiding the drop of Q-value due to underestimated Q-values in offline. Many previous works B3) discussed such a problem and solve it by alleviating excessive pessimism. Our methods do not need initialize critic from offline results, but re-evaluate policy in the online way to get optimistic Q-value functions, which is applicable to different algorithms and avoids the performance drop caused by the drastic drop of Q-value. Note that optimistic Q-value functions obtained by policy re-evaluation may unreasonable, especially for OOD actions, hence value alignment is needed for redress overestimated values.
15

---

In some policy constraint methods [32 I0,, only the policy is limited to update in region close to the dataset, but the critic is obtained the same way as the online update. Therefore. for such offline methods, the online critic can be initialized directly by the offline one, and just need to align the values with the offline policy.
addition, policy re-evaluation can be applied to the scenarios where only the offline policy is provided, such as where the offiin policy is obtained in the style of sequence modeling [6 (3 [8]
or non-standard RL [46].
The necessity of value alignment With the optimistic property, Q-value functions obtained in policy re-evaluation will overestimate OOD actions and induce the policy to take such actions, resulting in performance degradation. For the transition from those policy constraint methods that do not require ee-evaluation of the policy, this case still holds. The purpose of value alignment is to approximate the optimistic property and suppress the Q-values of OOD actions to reasonable estimation. For those offline algorithms where the actor and the critic are misaligned like explicit policy constraint methods, if the constraint is removed during online fine-tuning, the critic will induce actor to align with it. However, the actor is reliable but the critic not, so such process leads to unknown performance changes that are most likely to be worse. In addition, for algorithms induced by sehhavior-reulariinn MDP, such problem still exists because the update way between offline and online changes, leading to drastic variation of the critic. Therefore, for 020 RL, it is necessary to align the critic with the actor instead of aligning the actor with the critic.
The necessity of constrained fine-tuning Although we keep the optimistic property for Q-value functions and align the critic with actor, it is still challenging to achieve stable online fine-tuning. In general, most of the current offline algorithms focus on how to avoid OOD actions and train reliable policy on the states of the dataset. Due to the optimism in online RI., OOD states and actions are inevitable, may leading to drastic performance fluctuations, which is undesirable for important scenarios especially for high-risk scenarios. Especially for OOD states, even trained well in the offline phase, the policy still fails to output favourable actions, that may causes erroneous policy update. Therefore, for stable O20 RL, online fine-tuning with the constraint of ensuring safe exploration is necessary.
hal8 hoetsh lem exprrte))
heppor-espert-22
hupper-medium-e?
Nernatted Rouup
Neroolted Romm
Cammllee Return
03SAC VA
O3TDA YA
wath CD
dbruct
Evaluatio Epach
Ecaluxtion Epuch
Eexhuation Fpech
(a) Ablation results of O2SAC
(b) Ablation results of O2SAC
(c) Ablation results of O2TD3
Balfii heetab expertin
belf-thetth medium- x2
hepper expert-33
Normotee Retura
Normoliad Retura
Evaluntion Epsch
Evaluation Epoch
Evalustion Epock
(d) Ablation results of O2TD3
(c) Ablation results of O2PPO
(f) Ablation results of O2PPO
Figure 5: Ablation results of our methods. PR=Policy re-evaluation, VA=Value Alignment, CF-Constrained Fine-tuning. For O2PPO, VA means the use of the auxiliary advantage, and CF means the update of the reference policy.
The results of Figure that even with a narrow dataset e.g. hopper-expert-v2, our methods still achieve stable online fine-tuning. For O2SAC, the fine-tuning process without the optimistic critic reconstruction can lead sudden performance drop at the beginning phase, e.g. in Figure 5(a) as the offline critic may severely pessimistic. Due to the alignment the offline critic and the actor in CQL, using constrained fine-tuning alone can result in overall stable performance improvement,
16

---

but O2SAC demonstrates more efficient result. As for O2TD3, fine-tuning from offline directly suffers from severe performance degradation due the mismatch of the actor and the critic in offline training, such as in Figure 5(c) Thanks to value alignment, we can address the mismatch, thereby achieving stable performance improvement at the beginning stage. However, due to OOD states during online fine-tuning, constrained fine-tuning is necessary for the long-term stability. At last, PPO an on-pont polpolicc method that has a high data quality requirement, so it cannot improve performance the value is incorrectly evaluated, e.g. in Figure 5(e) Through the reliable auxiliary function, our O2PPO can improve performance stably even with imperfect value evaluation.
Although our methods are universal for any offline algorithm to SAC, TD3 and PPO. some steps can be omitted according to the type of the offline algorithm. For example, for value regularization method e.g. CQL, we can only use constrained fine-tuning for stable and efficient 020 RL if the constraint term with a low coefficient has little effect on the critic, as the offline actor and critic are aligned, constrained fine-tuning is enough to combat the problem caused by the drastic jump of Q-values. And for policy constraint methods with explicit constraint e.g. TD3+BC and AWAC, as talker above, the critic is evaluated in the same way online way, policy re-evaluation can be omitted. For other offline methods with the update way that does not match the online way, such as MDP, and with special policy form decision transformer, all steps of our methods are needed for stable O20 RL.
IQL (20), IVR [47] and x-QL [I3, and methods induced by behavior-regularie
C Additional Experiments
C.1 Difference in policy performance caused by optimistic critic reconstruction
Note that we use Tea as the initialization of the actor at the beginning of online fine-tuning. In accordance with analysis, the performance of the initial policy Too is expected align with the characteristics of the actual offline policy. As depicted in Table 3 from our empirical experiments, the results substantiate the actual offline policy. In most environments, Fom even demonstrates superior performance than the offline policy. We attribute this the effectiveness of our min operator, which sensibly tightens the policy distribution.
analysis, revealing minimal disparities the performance of Ton compared to
Table 3: The performance of the offline policy noi and the policy Ten obtained after policy re- evaluation and value alignment
O2SAC
O2TD3
Dataset
Taff
Man
Roff
Tan
halfchectah medium-v2
46.85
53.04
48.12
48.46
hopper-medium--2
$9.29
65.28
56.29
61.44
walker2d-meedummyy
81.08
78.63
83.13
84.16
halfchectah hatpsacctta-mmciidu--epaaaaaa-
45.27
48.
44.28
45.21
pooper-mediumm-rephee
88.97
99.64
59.26
49.95
80,31
75.77
77.16
75.90
halfchhetah-mmcduum-oporrrr
94.14 14
95.43
91.99
78.64
hopper-mediumm-experi
84.15
94.15
101.60
106.44
walker2a-mmddium-xxpprt-=22
109.44
109.30
110.51
110.53
halfcheetah-mediin--epp-r------e
96.65
99.80
96.89
97.17
hopper-expert-v2
111.80
111.68
111.74
106.31
walker2d-expert-v2
109.84
109.87
110.15
110.3
C.2
Experiments on D4RL AntMaze navigation tasks
For the difficult tasks of AntMaze navigation, such as medium and large environments, TD3+BC
is almost completely incapable of training favourable policy [40]. Fine-tuning with the poor
17

---

initialization from such policy helps little and has minor differences with training policy from scratch, that should be the concem of hybrid learning 4 we do consider it.
Here we demonstrate the results of O2SAC and O2PPO on the the difficult tasks of AntMaze navigation, including antmoze-medium-playy ontmacc--eddinn-divveeee antmaze-large-play-t
nntmazz-large-diversees environments.
medium diverse-12
diverse-v2
Narqaalted Rour
Normalized Rerteeedeuurr Return
Nermalieed
M M
Curmaliin
Evalustion Epoch
21
Evatuntion Epoch
'Evauuation Epech
Exaduntion Epoch
Figure 6: The results of O2SAC D4RL 9
AntMaze navigation tasks during online fine-tuning.
The solid lines and shaded regions represent mean and standard deviation.
animarry- -bt con- player
anfimaze mairazz-mmdimm-dfeeeee.
animarre Sarge parg--ppp-o2
anfmaze- emree2
Normatteed Retur
Normalizd Requ
W un
m/h
Normalred
Normatted
Nu M
Mmmm
Evaluntion Epoch
Evaluation Epoch
Eraluation Epoch
Evaluation Epoch
Figure 7: The results of O2PPO on D4RL 9 AntMaze navigation tasks during online fine-tuning. The solid lines and shaded regions represent mean and standard deviation.
Table 4: Average normalized D4RL scores of our methods and some baselines on AntMaze navigation tasks.
Dataset
AWAC
IQL
PEX
O2SAC
O2PPO
medium-play-v2
0.0
64.8~>78.0
73.6~>81.0
67.0++78.8
66.5 66.3
medium-diverse--2
-
68.8->73.2
70.8->83.4
61.6++70.6
63.5 +85.5
large-play-v2
0.0 - 0.0
0.0
39.4+55022
46.2-454.6
19.8-+44,
38.0-455.83
large-diverse-v2
0.0 + 0.0
31.0-+36.0
40.0->58.4
24.2->40.2
26.83 +5.33
total
0.0 100+000
204.0++337.4
230.6-+277.4
172.6 +233.8
194.8-+272.0
A
+0.0
+33.4
+46.8
+61.2
+77.16
Note that here we run O2PPO with 250,000 environments steps and run O2SAC with 200,000 environments steps, while in Section 5.1 we run O2PPO with 200,000 environments steps.
Note that as some previous work, in our O2SAC implementation on AntMaze tasks, we do not use the double Q networks trick to avoid overly underestimation, and the threshold reaches the maximum in step 100,000.
Since there is lack of hyper-pppret parameters for these tasks in many related work 48 25 do not compare the results of our methods with other methods. However, according to our simple reproduction, methods achieve competitive results.
50
53
we
18

---

DSSAC with splanal refomme
00SAC with fired
02E01 wikh oplial
02190 spdaie interval
sallibbeeta--mediimm-
ballchheelahe merprit--mmiiiu--rre----
bulfecrretal-mmee Iullichetta-mmdiinneseett
ballchretta-espeett--
Normallzed Returs
Nor mormalled Retura
Normaliad Retare
Aormalized Return
1
Evaluation Epoch
Exaluation Epuch
Evalustion Epuch
190
Evaluation Epuch
399
2
hogpur-nedian--pppg--7
hopper-m -dam-spur-2
bopper-espert-22
Normaltao Return
Normaltedd Return
Normwlizd Return
Normalized Return
Evaluxtion Egoch
Evaluation Eguch
Evalustion Epoch
Evoluatio Epuch
walker 2
vallor2ddmmdium-ree
Normaltzed Return
Normaltred Retuen
Normalzed Return
Normalked Return
8
Evaluation Epoch
Evalnation Epoch
24 Evaluation Epoch
Evaluation Epoch
Figure 8: Comparisons on different ways of updating the reference policy for O2SAC and O2TD3.
C.3 Comparisons with the results of updating the reference policy update at a fixed interval
We set the reference policy as the optimal historical policy during online evaluations in our preceding experiments, and it indeed introduces additional testing information. However, for most of the scenarios that offline RL focus on, this would be acceptable as a well-learned policy can be deployed directly without serious consequence due its decent performance, e.g. in autonomous driving.
In addition, thanks the abundant data, another way that does not require practical evaluation is to train a transition model that can be used to evaluate the policy by generating synthetic rollouts as the initial states are provided. Since such a methods is similar to model-based methods, we will leave for future work.
If evaluation is forbidden, we can replace Net with recent policy at a fixed interaction interval. The Corollary 4.3. still holds because the policy performance will be almost certainly improved with long interaction interval, especially compared the last reference policy. Considering that the policy performance is most likely to fluctuate in the beginning stage during online fine-tuning, we can set large update interval for this stage and small update interval for the subsequent stage, or reduce the threshold change range.
We experiment our methods with a given update interval D4RL Mujoco locomotion tasks and compare the results with those of the methods with the optimal reference policy, shown as Figure B and Figure [ Without special haper-parameter optimization, we just update the reference policy per 1000 steps for most environments, but for supper-medium-expee
and hopper-expert--2, we update O2TD3, because these datasets are narrow, making it easy for the policy to suffer from OOD states and actions. With an appropriate interval, the policy performance can achieve competitive improvement when compared with the methods with the optimal reference policy. Policy re-evaluation and Value alignment guarantee the smoothing and stable performance improvement at the beginning stage, and constrained fine-tuning is necessary for the stability of the subsequent stage. Although in hopper-medium-v2. O2SAC with a fixed update interval suffers from the performance degradation at the latter stage, because the threshold is large and the Lagrange multiplier À is low, the lack of constraint on the policy update the reliable region. This phenomenon can easily amended by reducing the threshold change range.
the reference policy per 10000 steps, and well as for malker2d-exxerrte
19

---

uptimul reference policy
update interval
eddicbeeeel-meeiinw edium-22
mallcheefall-
balfcheeinh- ballcheinf-mcdiiu---e medium--repph--
bnlfcchetthh hattbrettaammmiimssseeetee ballcheetabbeepeet-22
Normalized Return
Normalzzed Return
Normalized Return
Normolized Return
Epoch 399
Epoch
Epoch
Epoch
2
hopper-mediam-reeeee reppn-y--
hopper-m eetnmn-epeertaee
hopper-expert-22
Normallred Return
Normallred Return
Normollee Return
Normalizedd Return
Evaaluation Epoch
Exalmation Epoch
Evaluation Epoch
Evaluation Epoch
walker 241
palller2d-medumm repla---------222222
2d-expert-v2
Norm allxed Return
Norm altdd Return
tee
Norm alledd Return
Norm elzed Return
100
Evaluation Epoch
Evalnation Epoch
Evaluation Epoch
Evaluation Epoch
Figure 9: Comparisons on different ways of updating the reference policy for O2PPO.
C.4 Comparisons with PROTO
In the absence of 020 methods the deterministic policy, compare methods with PROTO [25, a recent 020 method that supports fine-tuning from an offline policy for both the stochastic and the deterministic policy. We reproduce the code of PROTO Pytorch for the initialization of the policies derived from CQL and TD3+BC, and set all hyper-parameters as in the official paper 25 As shown in Figure TO our methods demonstrate significant superiority in both stability and effectiveness, especially for the deterministic policy.
C.5 Comparisons on initialization of different offline methods
We conducted some experiments for O2SAC with the initialization of different offline methods, including CQL, IQI. and ODT. The results are shown in Figure TT] The initial performance of O2SAC initialized from ODT is lower than others since the simple behavior cloning (we directly maximize the likelihood of the actions output by offline ODT while keeping an appropriate entropy) could harm the performance, as discussed in Appendix ! But in hopper-medium-Ve. the performance improves quickly. We analyze that by the constraint, the policy can recover the offline performance (about 97 normalized score of ODT), the output of the cloned policy near the ODT policy. The results demonstrate that our methods are suitable for any offline algorithm, even the policies are heterogeneous.
C.6 Accelerate learning with sample-efficient RL
An additional benefit of our methods their straightforward compatibility with sample-efficient online RL algorithms. Since we only add constraint that can be considered as part of the reward, the
policy iteration process remains consistent with the normal online approach, which makes it feasible to incorporate techniques from advanced efficient RI algorithms. Drawing from 4. we conducted some experiments using high UTD ratio of 10 (but still update the lagrangian multiplier once per step) and achieved better performance improvements, shown in Figure 12
20

---

O2SAC
mallcheefall-m ediummv2
balfcheeinb- ballcheeenk-mrcdiun-eee mediimm-eelllyeaeeee
hulfchretah- 2ediumeespert-
hallcchettab-exeet--2
Normaltzed Return
Normalzzed Return
Normalized Return
Normalized Return
Epoch
Epoch
Epoch
Epoch
393
hopper meddium-22
hopper-mediam-reilgee r2
hopper-m ed expert-v2
hopper-expert-22
Normallred Return
Normallreed Return
Norma Return
Normalizee Redliee Return
Evaluation Epoch
Exalmation Epoch
Evaluation Epoch
Evaluation Epoch
26-medlum--22
walkerr2d medium-rrplaage replay--v2
26-expertv22
Norm Cormatted Retura
Norm albed Return
Norm dlledd Return
Norm alleed Return
why
Y
40
Evaluation Epoch
Evalnation Epoch
Evaluation Epoch
Evaluation Epoch
Figure 10: Comparisons with PROTO and PROTO+IDS (25] on D4RL [ MuJoCo locomotion tasks during online fine-tuning. The solid lines and shaded regions represent mean and standard deviation.
OZSAC from CQL
O2SAC from IQL
O2SAC from ODT
haffchorial-mmddim--22
hopper-medinm--72
wallier Bd-iierrd---eeee dium-ny2
Normaltaed Return
Normallized Return
Normalited Raturn
Evaluation Epech
Evaluation Epoch
Evalustion Epoch
Figure 11: The performance of O2SAC with the initialization from different offline algorithms. The solid lines and shaded regions represent mean standard deviation.
C.7 Computational cost for each component
We conducted an experiment on hopper-medium--2 environment using the O2SAC method and listed the time cost for different phases in Table 5] evaluated Nvidia 3070 GPU.
Table 5: Computational cost for each component of O2SAC
Training Phase
Oftine(CQLL)
Policy Re-evaluation
Value alignment
Training Steps
IM
0.5M
0.5M
Time Cost
5.4h
0.95h
2.0h
21

---

ballchectal-mddimm-
hopper-mediam--2
walleer2d
UID-1 U1D-10
UTD-I
116-
Normalized Retum
Narmatian Iomaltted Ruturn
UTD-10
Naymalted ltzd
40
UTD-1 UTD-10
Evalustion Epech
Evaluatioo Epoch
Evalustion Epech
20
Figure 12: The performance with a high Opdate-To-Date ratio of O2SAC, The solid lines and shaded
regions represent mean and standard deviation.
In policy re-evaluation, since the policy is fixed and the re-evaluation of the critic is straightforward, the computational cost of re-evaluation is signifieantly lower than that of offline learning. The time cost of value alignment is somewhat higher but still less than that of the offline phase. In fact, the time cost is approximately proportional to the offline phase according to the alignment steps, since both the actor and critic are updated in the value alignment phase.
However, it is worth noting that although set the training steps for value alignment 500k, in some environments, only a few alignment steps are needed to calibrate the critic with the offline actor, shown in Fig. 1 and Fig. 11. Only in antmaze environments, where is hard for the critic to capture the sparse reward signal, more alignment steps are necessary. Additionally, in constrained fine-tuning, since only the lagrangian multiplier is added to be updated and the interaction cost. dominates, the time cost increases very little.
Moreover, in O20 RL, we typically not concerned about the time cost in the offline process, as different offline methods take different amounts of time, Instead, we prioritize the cost of interactions during online fine-tuning, Our method re-evaluates and aligns the critic with the offline actor solely within the offline dataset, making the time cost less critical.
D Related Work
Unified algorithms for offline-to-online Many offline algorithms are unified across phases in 020 RL 32 20
who share the common philosophy of designing an RL algorithm that suitable for both offline and online phases and then the network parameters trained in the offline phase can be reused for further learning in the online phase. Most of them are policy constraint methods which learn policy without querying OOD samples 120] 43 or penalize action probabilities of them by an explicit estimation of behavior policy [44), which is beneficial for offline performance but limits efficient performance improvements in online fine-tuning, talked about in [33. For explicit policy constraint, recent works focus on how to make the constraint adaptive [52, 26 or loosen
gradually [5). Although such algorithms achieve great performance in offline phase, there is lack of research on from such algorithms to more efficient online algorithms.
Efficient online fine-tuning Some recent works aim to online fine-tune efficiently with optimistic exploration. [15] propose unified uncertainty-guit guidded framework to explore optimistically online fine-tuning phase and keep the offline constraint for OOD actions to avoid erroneous update. 3Q and [51] utilize ensemble Q-learning to alleviate distribution shift, and implement optimistic exploration by some approaches about ensemble in online RL. For model-based 020 RL, [31] explores regions with high uncertainty and returns in learned model. 42 and [50] concatenate different offline and online algorithms by resetting new online policy learning from the offline policy gradually, where optimistic exploration is implemented by the new policy. With given policy and dataset, [3] focus how to recover the performance of the policy rapidly, whose setting is suitable for method too.
Stable online fine-tuning As offline RL works for some important scenarios with high risk, the stable performance is considerable for 020 RI. After 233 put forward distribution shift problem in 020 RI. which is alleviated by pessimistic Q-ensemble and balanced replay proposed by them, many methods are proposed to combat this problem. From offline to SAC, [33] mitigates the penalty for
22

---

OOD actions in CQL and calibrates Q-values of them by Monte Carlo returns, while 448 reconstructs O-functions aligned with offline policy. [25] also induces KL divergence to policy objective, which is about current policy and the policy at the last iteration to perform trust-region-style update, but
the performance will drop suddenly at the beginning of online fine-tuning. [19] and [28] utilize environment dynamics model ensemble to obtain uncertainty to penalty OOD actions.
E
Detailed Discussions about Mismatch in SOTA Offline Algorithms
CQL For common implementation of CQL. [21 do not specify exactly how тy is updated but only
provide properties on the policy exp(fx (s. a) (z) like SAC. In this way, the actor of CQL is only related to the critic, and the action probability is directly proportional to Q(s. a) like online RL. However, as the conservative policy evaluation operator used to update the Q-function (according to (11] Appendix C, equation 13), the policy is not only related to rewards, but also related to the distribution of behavior policy. The conservative policy evaluation operator in CQL is:
((aas)) 1]
Q(s.a) = B Q(s a) af
(21)
Tz(aas)
where 8" is the standard Bellman operator 8 Q(s, = r(s, a) + (V and
V(s) = Eu (Q a)]
(22)
If we define Qo as the Q-function derived from Eq. which satisfies:
Q0(8,a) B= (s)
(23)
Therefore, the relationship of conservative Vss) in CQL and (s a) is:
(s) = Ee (s) a. [(a s)
(24)
(a
The objective of policy is to maximize (s),, compared with online RI., there is difference in learning objective due to pessimism. In fact, drawing from SAC, we can derive the objective CQL:
max E (r( 0x) al p((ar(s)) 1])]
(25)
T (T(((())))
which is in according with rehavior-regularize MDP proposed by 47 at f(x) 1. The conservative policy objective causes pessimistic Q-values estimation, hence in online fine-tuning, the change of policy objective leads to drastic Q-values jump and performance degeneration.
IQL As expectile regression used to estimate expectiles of the state value function with respect to random actions, Q(s, a) and (s) are not induced by the policy æ, and we denote them as (s, a) and (s)s,, where μ represents a special policy related to the expectile regression [88. With initialization of such Q"(s, a) and V in online fine-tuning phase, if we update the actor and the critic in online algorithm, the actor update will tends to μ at the beginning, which causes unknown performance change because the performance of is unknown. And usually the performance will deteriorate significantly the probability of a satisfactory policy v is low.
On the other hand, the work of 47 reveals that the optimal critic of IQL can be derived from behuaior-regullrizee MDP at ((x) log(s). Meanwhile, the update actor derived from return maximization with a KI. divergence constraint, it is easily to discovery that the learned policy is relevant to dataset. Therefore, the 020 RL for IQL still suffers from the problem of policy objective change. And compared with value regularization methods, there is not only pessimistic estimation problem but also initial misalignment problem, which means offline actor cannot be derived from offline critic in online update way. In fact, in offline RL, the policy update function is to maximize Eq. ( which is highly related to behavior policy as a is a fixed parrr--arreeter.. Is directly used to fine-tune policy, the policy will tends to be only proportional to Q(s, a) of offline critic, hence the policy performance will drop sharply with a great probability.
TD3+BC With explicit constraint of policy update, the improvement mismatch is obvious: the critic update follows the Bellman backup, but the actor update does not only follow Q-values maximization, that constrains the actor only update in region close to the dataset. A little different from the
23

---

mismatch in IQL, TD3+BC does not suffer from pessimistic estimation since the critic is updated an optimistic way as the online way, but such improvement mismatch still leads to performance drop. The probability of action with the largest Q-value may be not largest, as talked about in [48], which is not in according with online RI. and leads to erroneous update destroy performance the beginning of online fine-tuning. Such improvement mismatch exists in almost all explicit constraint methods, but their critics are optimistic because the update ways the same as online RL, so value alignment is necessary for O20 RL from such offline methods.
F Proof
Proof of Corollary 4.2 The theoretical analysis about FQE can be found in Theorem 4.2 22 or
Theorem 4.9 of (2).. here use the latter result. Full proof is similar to the proof of [29 but
little different from 229]. we fix the evaluated policy as the offline policy.
Here we simply show the proof process and emphasis the difference, see 29 for more details.
Since FQE deals with the following regression problem
D
Qk + arg min QEF
> [Q(ss; - - -x0)--((c,,
(26)
can have
Irs + 77 < + <V
(27)
(T + 7B
< 7V s 22
With the inherent Bellman evaluation error di, we can apply least squares generalization bound here (Lemma A.11
2 With probability least have
22 log F1/8)
11Qx 10k-T"Q2 1
20d7
(28)
D
Note that here we fix the policy as the offline policy, that means for a given Qo. Qk-1 is well- determined, so we do not need to apply union bound over all possible Qk-
Then we first bound || - Q" 11l.,0 as the same as 29 did.
||Qx Q" 112.77 = 11Qx Q" Q
17'12.007x x
S 11Qu - Q=Q 12- X + 1 Qx-1 Q" 1| x
VOl1Q& T"Qu + Qk-
1 T 0112,07 xn (Assumption 1.1
VCe+ + (E(ra))dd x
(sss''a'))
VCe + // E (tss)
(=',
-0 x2
(29)
The fifth derivation uses Jensen's inequality. With
1-
|Q Q" 1"112,0fx=
Q @"112,0fxx5 1-7
VOC+gKD
(30)
K=0
24

---

Therefore
10t - -0-- = 1Q=
Qxl(K + 00)
S 11Qu - "Jense
Buy (Baonr(ll)) (Qx((, Q" (8.a)
E1 - 7) Σ
dd (s)
Σ -(dss[k((s.))
t=0
(1
d7 (8)
=(ab)]Qx(ssaa - (s)))))
(31)
1=0
(1
7 1xxx Q Q"112,0000 x
1=
1 - 1
7)
1 VV)+
t=0
K
VCe V
We define (iq a) as the state-action values for some actions after the process of policy re-evaluation and lower than the calibrated value, i.e. Qtge a) S ((, a) (log Toff (ā log ooff (als)). With the min operator in Eq. (12). we can divide actions into two categories, calibrated actions or standard actions. The former refer to those actions with Q-values to
Proof of Proposition 4.3
be calibrated, and the latter refer to those actions with unchanged Q-values since they are lower than the calibrated values. We denote them as Geal and ale respectively.
Valign (s) = Ea-naa [Qalign (s, a) a log (at((a)s))
Σ (ar(als) ( (s, a) - a log Tall (a|s))
(32)
Σ Toff( dcal (ea (s. deal)] + Σ [ort(Urgels)(C [S) (8, dtgz)]++
Σ Tott(al (a) s) log [a][]
Due to the min operator, for standard actions Ofge, the calibrated Q-values satisfy Q (s, asye) S Qeai(s, arge). therefore
Va(s) Q( à) a log (als)
Σ Taa(aas)) (car a) a log [a]s]
Σ Toar(aaal ()a deal) Σ Toff Toff( [s) (sppe (f,ea
(33)
Σ (an (a)ss)
Vatign(>)
On the other hand, it is easy to see that Q (s, Aite) a log (argels) S (sa acat) a log a (Qcals).
so
(s)
Q (s, ofge) - a log (afge s)
Tooe (#rge 1)) ())QQe (s) (rige) ar log सiत (atgels)}+
(34)
Σ
Totel (Ccal (s) (Qac Quat) log Tail [sc
Valign())
So (s) < Valign (s) Vs(s,, i.c. Proposition 4.3 is proved.
25

---

Proof of Proposition 4.4 With simplicity, we consider the term of the auxiliary advantage function without probability ratio clipping
E
RB
Aa (8,
= sm RawNTo(-s))
(s,
arw Rar (-1a)
no.
Σ
To(als))
a
E |- SMilt
Re log (err(als) E To (a(als)H(TTr((H
= -E [CELoss( (-0 Tee((-1s)) - H(Tm((-)s)) Σ Ta(a)s)]
s
= C'EssossTT neet) +
(35)
Therefore, the term the auxiliary advantage function regularizes the policy update near the reference policy, thereby ensuring reliable update with inaccurate value estimation. And at the beginning of online fine-tuning, the reference policy is initialized from Toff-
Proof of Corollary 4.5 Since our constrained fine-tuning method solves constrained MDP problem by the method in [41], Theorem 2 in 41 still holds in our method, which can be de- scribed as With the constraint satisfied, RCPO algorithm converges almost surely to fixed point (0* (w",
Such theorem guarantees the convergence of RCPO, and with the our proposed constraint, we can utilize contradiction proof Corollary 4.5
Assumption According to the convergence of RCPO, we can assume the convergent point (0* (j²", >*), (X*), X*) with X* * 0.
Derivation As the constraint is E, (x The (a: {))] a and " the best one among old policies during online evaluations, Tie x² when the algorithm converges, the constraint term tends to aa. According to the update function Eq. 200) of X, À tends to be reduced, which is in contradiction to the Assumption. Therefore, X when the algorithm converges.
Meanwhile, when optimal results in the MDP without constraint, which means our method converges the optimal point like online RL but keeps stable improvement.
O, the constraint removed, hence the and Q* are the same with the
G Algorithm Details
G.1 O2SAC
the choice of a
Although is generally smaller than I after offline training, as we use the energy policy to align critic with actor, small has no influence the recovery of offline policy but leads to wide distribution for policy, which means the policy modeled by Gaussian distribution has a large standard deviation, which harmful for online exploration. If the standard devi- ation is large, some OOD actions may be taken and dangerous state may arise during online exploration, which dose not suit the setting of high risk scenarios and affects favourable performance. Therefore we choose a suitable which is smaller than 1 but not excessively small. For Mujoco locomotion tasks, we determine that alpha is 0.2 for the dataset with medium quality, and is 0.5 for the dataset with expert quality, because small standard deviation induced by large is advantageous for online exploration of well trained policy. For AntMaze navigation tasks, we set alpha as 0.5 for medium and large datasets and 0.2 for umaze datasets.
walker 2d ciiumeespert-
0-0.1
0.2
0.5
Nermalized Return
112
110
106
104
Evaluaaion
Figure 13: Policy performance during value alignment with different
26

---

Moreover, as we use min operator for value alignment, the limit for a is not so strict. If Q-values induced by policy re-evaluation is smaller than alignment objective, as Gaussian distribution is used to fit energy policy, such Q-values will leads to smaller standard deviation.
Note that the choice of has effect the recovery of offline policy, as shown in Figure
1
policy re-evaluation As talked about Section 4.1 with given a, online policy evaluation of SAC Eq. can be used to policy re-evaluation to obtain optimistic critic.
value alignment In Section 4.2 the value alignment method to O2SAC has been described in detail. In brief, we use online policy improvement Eq. (2) to get the actions which have overestimated Q-values, and use Eq. (33) to inhibit their Q-values reliable value calculated by maximum entropy RI.
According the min operator, there is change if Q-values induced by policy re-evaluation is smaller than alignment objective, so our value alignment method not only aligns Q-values with actor but also is as consistent with the results of online evaluation as possible, which reduces Bellman error in online fine-tuning to keep Q-values more stable.
constrained fine-tuning For O2SAC, the constraint is KL divergence, and the policy objective of CMDP in 020 RL is:
s.t. E2 [log( {(an|se) )] <T
max E Ce(re(se, an) all (*(+ss)))
(36)
Frar(ar (st)
And corresponding loss functions are:
Alog( (a(a)s)
(( max Esg [45 a) - a log To Ta(als)
(re((a ]]
(37)
I(p;)= = min
E
[( (0% (s,a) y
(38)
(K,a,T,))) N
O (s',a') aog To (a'ls') Alog' To (a(a'(s²) Trer(a ss)
= r + 7
E
a c'>5to-115)
<o(a|s) -T
L(A) = min
llog(
X20
(als)
(39)
min
X
[lo No (a(a))) log (ar((a)s) -
X
s~ (
A constraint related to offline policy is necessary for online fine-tuning, because OOD states will appear surely during online exploration, especially in narrow dataset, which may lead to erroneous overestimation and destroy old policy. Such constraint can avoid overestimation of actions far away from current policy, hence trust-region style update guarantees stable performance improvement.
Compared with the constraint of current policy and offline policy, our proposed constraint guarantees more optimal update because the policy updated trust-region offline policy generally has similar performance to offline policy, which leads to not much performance improvement.
And compared with the tight constraint of current policy and the policy last iteration, our con- straint guarantees rapid recovery when some erroneous update occurs which results in performance degradation, because the Lagrange multiplier À will be larger and make the Q-values of OOD actions lower, hence the policy tends to be close to Tee-- In such situation, our method can recover a similar performance to Nee for current policy rapidly, but the tight constraint needs to take much time to do that as it constrains the current policy close to the poor policy at last iteration.
G.2 O2TD3
policy re-evaluation Similar to O2SAC, online policy evaluation of TD3 Eq. (6) can be used directly to evaluate an optimistic critic, and TD3, there no need for a.
value alignment As TD3 models a deterministic policy, value alignment method can not be derived directly from the relationship of the actor and the critic like O2SAC. However, from Eq. (5), the gradient of policy is only related to the the gradient of Q(s, a) to if we fix the policy offline policy.
27

---

So, we have two insights for the actor in TD3: ((s, (()) is the maximal in Q(s,)) -) and the gradient around (s) should tend to O, and the latter also corresponds to the idea of policy smoothing update in TD3.
Note that in most cases, the correlation among different dimensions of action is not be considered, hence we use one-dimensional Gaussian distribution for the following analysis.
Therefore, with the definition that à iT(s,, we consider that normalized Q-values around à follow Gaussian distribution Q(s, a)/Q(s, a) ~ N (ā, Σ). And for Q(s, à + 6), we utilize Taylor expansion of first order and omit higher order terms, then we can get:
Q( à 8) ~ ((x,a) + (s.(s.)) ......... 5 (Taylor expansion)
(40)
2 Q(s,a) V= +6)
(41)
8 = as, Q(s.++
(42)
From Eq. (40) to Eq.
(11g
we consider the continuous derivative. From Eq. 41) to Eq. 42)
we utilize the derivative of a Gaussian distribution Va.((x) = -(r - p0/2, where f(z) is one- dimensional Gaussian distribution f(x) N (1,, 2).
Note that (s. = Q(+,
which means E = 1/2TT so we can get:
Q(s,a+ 1 8
((s, 1 2x52 ((s.a)
(43)
In our implementation, we replace Eq. overestimated values, and as action range is from -1 to 1, the alignment objective avoids severe underestimation because the minimum Q does not tends to O if we set small k, which also reduces Bellman error during online update as the same thing as min operator does. And we use the euclidean distance divided by the square root of action dimension to calculate 6, which is 5(a, â) VE (a: /:)A/
43 to Eq. (15) where k is used to control penalty for
Note that O2TD3, we use the same trick of reward normalization as TD3+BC, which means we subtract 1 from all rewards, so we need to consider the situation of negative rewards. In Eq 443) we set Q(, (s,)(s ~ (à, 2;) when rewards are positive, it natural set a(s,a)(Q a) ((, Σ) at the situation of negative rewards. Therefore, for the environment with negative Q-values we use following way to redress critic:
Q(s,a+0) = (1 (1 (6)++)))(((0)) a)
(44)
In a word, we multiply Q(s, a) by (1 + k52) if (xx a) > 0, otherwise divide it by (1 + kō?).
constrained fine-tuning For O2TD3, the constraint is MSE loss, so the loss functions are:
((0) max E To 'QTH
A(To(s) Tee (s))
(45)
L(p:) min E [Q7 (s, y
(46)
(c
y = r + 7
E
[Q% (s',a') aa) {(T((s) - TT)r(
a' (-1s")
L(A)
min A20
A E (TT(() - x))r - r]
(47)
R
G.3 O2PPO
policy re-evaluation Different from O2SAC and O2TD3, Eq. (8) is not related to offline policy, in practice we can only obtain (*(s) through fitting the returns. Advantages (T a) computed by update. However, in order to
such V"(s) may be awfully incorrect, which sequentially causes follow the update way of PPO (only (s)) is used to update), we stick to update state value function by fitting the returns, and introduce a modification to advantages used to update.
value alignment As V (s) obtained in policy re-evaluation is actually V*(,,, which is incorrect to compute advantages A ( a) for on-policy update, propose auxiliary advantage to redress
28

---

erroneous update. Let us think about the property of the auxiliary advantage. First, as an advantage function, its mathematical expectation of offline policy oof should be Second, the function should be able to output positive values and negative values for different actions to distinguish the quality of actions. Last, better actions should correspond higher values. Drawing from the policy form of SAC, we propose the auxiliary advantage as:
Aa a) = a log (als) + a H(Toa((-ss)
Ea-na(-1s)) [og (Tor(a))) [a]s)
(48)
When the policy is modeled as Gaussian distribution, it satisfies: (1) 1a [Aa a)] = (2) (s, a) > O when ||a /=122 < a. (3) (s, a) x log Ton (als) And such properties are in accord with requirements of auxiliary advantage function.
It is notable that here is implicit assumption that the actions with higher probability for Toff are better, and for well trained offline policy, such assumption should be valid, at least for the beginning of online fine-tuning, With such auxiliary advantage function, we can redress advantages computed by incorrect (s) to ensure the stable performance improvement during online fine-tuning, especially for the beginning stage.
constrained fine-tuning As the auxiliary advantage function has the ability to constrain policy to update in reliable region near Toff that means N Ton. we just need replace net as the optimal policy during online evaluations to constrain online fine-tuning, Meanwhile, with the increase of interactions, the critic gradually becomes accurate, approximating V And on-policy method is naturally stable to improve performance by updating in a reliable region, therefore the constraint factor 3 anneals to 0 from during online fine-tuning.
H Implementation Details
H.1 baseline implementation
Offline results For obtaining offline policy, we choose CQL [21], IQL 20 and TD3+BC [00, and we
reproduce offline results according deep offline RL library CORL 40 https://githbb.co htt.ggithw...omiooooii.tt
ai/CORT, all hyper-parameters are the same with the official implementations. The offline results are used in methods, Off2On and PROTO.
COL We reproduce the results of CQL by the code from CORL https://github..om/iikk///
IQL Like the implementation of CQL., reproduce the results of IQL by the code from CORL https://githubb.o htts://github.commmii
TD3+BC Similar to above, we reproduce the results of TD3+BC by the code from CORL __cp____________.
Online fine-tuning results For offline-to-online algorithms, for most methods, we reproduce the results according to their official implementations. For AWAC 32 IQL, 20 and Cal-QL 33 we
reproduce the results by the code from CORL [40]. For ACA 48 and PEX [50, we reproduce the results by the official open-source source code httss//github..oom/iiiee http:/ihwwb..ooo/aaaaaaaoooiaiie
Yu/Actor Ttw//Ator-Critic-llieeeee Critic-Alignment
Although PROTO focus online fine-tuning, but initialize the policy from other offline algorithms in the official implementation. For the sake of fairness, we rewrite the code according to the paper of PROTO in Pytorch, and we use the
and official code (https://github.comm http://hhuuucooo/aeeeeeeee http///aeeeaali//POOOO offline results of CQL and TD3+BC for initialization respectively.
In addition, in the official implementation of Off2On 23 the policy update 1,000 times per 1,000 environment steps, that is different from the common implementation. Therefore, we reproduce by using all parts that related to the prioritized replay in the official code from https://github..om wwweesooo//aaaeeeeeeeee woom/hhle94/Off20nR
and we use offline results of CQL for initialization with the ensemble size of 5, that is the same the official paper and implementation.
All hyper-parametent are the same as the paper. And as 020 baseline methods are all off-policy methods, they are allowed to interact with the environment in 100,000 steps.
29

---

H.2 General implementation of our methods
Network Architecture As we need to re-evaluate policy, which means we only need offline policy and do not use the offline critic, we can modify the critic network architecture for stable online fine-tuning. In our implementation. we adopt the same network architecture as offline phase but apply Layer Normalization (LayerNorm) for the output of hidden layers after activation function.
49 indicate that in offline RI., LayerNorm is good solution effectively avoid divergence without introducing detrimental bias, leading to superior performance. Moreover, [4] find that LayerNorm favouralle for efficient online RL with offline data. Therefore, for stable evaluation and future consideration, we decide to apply LayerNorm to online critic network.
Initialization of online replay buffer We test method by initializing the online replay buffer with four different types: (1) Initialize the buffer with the entire offline dataset akin to [20]. (2) Conduct a separate online buffer and sample symmetrically from both offline dataset and online buffer akin to
[4]. (3) Initialize the buffer with a small number of offline data with high quality akin to [48]. (4)
Initialize the buffer without any offlin data. For simplicity, we denote them as All, Half, Part, Null respectively.
Table 6: Results for different initialization of online replay buffer.
All
Half
Part
Null
Total scores
163.08
225.54
220.26
201.63
As the results shown in 6 there is little difference among differrnnt ways of initialization, except for all, as the the quality of the offline dataset may be low. Although the initialization ways of part and wull also show the competitive results, however, to be consistent with 4 and future efficient update, we decide to adopt Half as our implementation way. It is notable that online policy is still favourable even if we adopt Null initialization, thanks to adaptive constrained fine-tuning.
loss weight for update Since À in Eq. (2) is a variable applied in all states, may decrease greatly in batch. In order to considering more about the situation that needs to be constrained, we modify the term about À in the loss Eq. 20p with a weight.
L(X)
min - [E (f (TT((s)s) (ee((al)))) -] X
(49)
where 10.7 - (( (a)
(alss) s))) > r) gives large weight to negative coefficients,
thereby constraining the abrupt decrease of X.
The choice of the initial value of X As À is the Lagrange multiplier which is adaptive to the constraint, there is no need to design the initial value of À specially. For medium and large datasets of Antmaze tasks, we set the initial value as 2.0 since the initial performance low, and we set the initial value as 2.0 for other tasks.
H.3 O2SAC implementation
The choice of constraint threshold т As we consider the constraint in O2SAC is KL divergence, and we model policy squashed Gaussian distribution, can directly the KL divergence of Gaussian distribution to approximate the constraint. Since Tee is the best one among old policies during online evaluations and it will be close to Te with the performance improvement, we can assume that the standard deviation of Nee is the same as the one of ng. Therefore, according to the KL divergence of Gaussian distribution, can get:
D (N (1),,0)), N 0²)) (110 - 1b6)
(50)
202
Therefore, we can determine constraint threshold r based on the magnitude of change in the mean. For medium dataset, we constrain that μoر nul < a, hence - = 0.5. However, due to different standard deviations, this threshold may not accurate, and it is an intuitive idea to loosen the constraint at later stage. So we set as a linearly increasing variable from 0.5 to 2.0 for medium and mediun-replay datasets, which means the allowable range of policy distribution mean is from a e/2 to
30

---

20. And for medium- expert and expert datasets, we set it from 0.005 to 0.125 for safe update, which means the allowable range of policy distribution mean is from 0/10 to o//2.
Clipped log likelihood Due to the limit of precision of Pytorch and the squashed Gaussian policy used in O2SAC, when the output action a is near 1.0, like 0.99999999, the log likelihood log i (a)ss) (8) is will be severely low as the value of action will be computed as 1.0, which will occurs when the log likelihood is need to be computed by given action value alignment phase and constrained fine-tuning phase. Therefore, for simple calculation, we set the minimum value of log likelihood is -50, enough small number, which has litter influence on results.
H.4 O2TD3 implementation
The choice of constraint threshold r As TD3 models deterministic policy, is unable to determine the constraint threshold according to the standard deviation. However, when we inspect the interaction process of TD3, we can find that exploration noise is akin to the standard deviation. So we determine from the exploration noise. First we set exploration noise as 0.1 for medium dataset, which is the same as TD3 learning from scratch. And for expert dataset, we set exploration noise as 0.05 because offline policy is well trained in such dataset, and lower exploration noise is favourable to avoid poor action samples.
After determination of exploration noise, with the idea of the equivalence of the standard deviation and the exploration noise, we set tau similar to O2SAC. For medium and medium-replle datasets, tau grows linearly from 0.0025 to 0.01, which means the allowable range of policy distribution mean is from 0/ 0/2 to 20, when we consider the exploration noise is equal to the standard deviation. Similarly, for medium-expert and expert datasets, we set it from 0.000025 to 0.000625.
H.5 O2PPO implementation
We implement our O2PPO basically according to the code of Uni-O4 (24).
The decay rate of 3 In our implementation for O2PPO, we set the interaction steps as 250,000 since PPO is an on-policy method which improves performance more slowly than off-policy methods. Since we keep the idea that in medium- expert and expert dataset, the offline policy is well learned, we decay 33 from to 0 linearly in 500,000 steps, which means in the end of our fine-tuning, B = 0.5. And for policies trained in other datasets, including AntMaze tasks and medium and medium-replay datasets of Mujoco tasks, we decay ,3 from 1 to 0 linearly during 250,000 steps to reserve more potential of performance improvement.
Clipped standard deviation As shown in [4 the policy trained by IQL algorithm has large standard deviation which makes poor exploration performance, especially in ceppr--meiiiu-eee---eeee and hopper-expert-v2. Therefore, we set the maximum value of the standard deviations of policies trained in the two datasets as 0.05, which corresponds to the set of exploration noise in O2TD3, because for a deterministic policy, exploration noise and standard deviation of a stochastic policy are equivalent during taking actions in exploration.
Shaped and weighted auxiliary advantage For one-dimensional Gaussian distribution / N (u, o²), the entropy is log( Log(V\rrea²), which is equal to log(f (p + o)), hence the maximum value of (s, a) in 48) is 1/2. However, the minimum value of may be much low, which leads to unstable training. Therefore, we use SoftPlus activation function to clip the range of Aa a). For each dimensional, we clip (s, a) as follows:
Clip(Au(s, a)) = SoftPlus(AA a) + +4)
(51)
By such clipping operator, the range of As, a) is ( 1/2) approximately. Note that when the current policy is close to the reference policy, the actions with overly low logarithmic values are rare, thereby having a little effect on the results.
Note that in the implementation make (s a) and A(s, a) the same order of magnitude, we reweight As, a) by multiplying coefficient that is the double value of the standard deviation of A(s,a). Therefore, the maximum value of A (s, a) is the standard deviation of the batch of A(s,a).
PPO, the normalization of advantages is needed. order to
31

---

hopper-mediung expert-v2
evaluation score
100
exploration score
ypu
Normalized Return
80
60
40
25
50
75
100
125
150
175
200
Environment Ste)s(xSe3)
Figure 14: Normalized return of evaluation and exploration during IQL offline training, where evaluation means policy output the mean of action distribution and exploration means actions are sampled from the distribution.
I Connect Different Offline and Online RL Methods
Our methods permit connecting different offline and online RL methods only offline policy is needed in our methods. Therefore, for those RL-based methods, which means the policy is modeled as a stochastic policy or deterministic policy in traditional RI. way, easy implement stable online fine-tuning by our methods. For those methods which model policy with different form, such as decision transformer (DTT [6], our methods can be applied easily if stochastic policy or deterministic policy is used to clone the offline policy.
For behavior cloning of stochastic policy, the easiest way Maximize log likelihood, however, we recommend the form akin to 133, as follows:
min (es log[/ (+o(s))
(52)
The purpose to maximize the entropy is avoid excessively narrow distribution, which may cause drastic jump of Q-values in O2SAC as will decay to zero quickly. However, in our experiment, such a way of behavior cloning will leads to performance degeneration, which is a question for imitation learning, It is more easy for the behavior cloning of the deterministic policy, MSE loss can be used to update the policy for the states in offline dataset.
is notable that degree of coverage has an important influence on behavior cloning, which is still an intractable challenge in imitation learning. Our methods provide the opportunity for the application of advanced imitation learning to 020 RL. In addition, even though the policy initialized with imperfect performance, our methods achieve better online performance than the online algorithm learning from scratch.
J Pesudo-Codes
32

---

Algorithm O2SAC
Require: offline policy Rott offline dataset D, factor a, threshold interaction interval T Optimistic critic reconstruction before online fine-tuning
1: Initialize the actor on with Joff and the critic Qon with random parameters
2: for iteration i 1, ...... do
3: Update the critic in the optimistic way by
b Policy re-evaluation
4: end for
5: for iteration i 2, do
6: Update Ton to seek overestimated actions by
7: Update Qun to suppress overestimated Q- Q-values by 13
Value Alignment
8: end for
Construined online fine-tuning
9: Set the reference policy Tref as Ton and initialize the replay buffer R with D)
10: for iteration i 1, do
11:
Interact with the environment and store the transition in replay buffer R
Update the temperature coefficient a by
Constrained Fine-tuning
13:
Update the Lagrange multiplier À by 39
Update the critic Qan by 38
15:
Update the policy by37
if evaluation permitted then
17:
if % interaction interval == 0 then (T > (Tett then Replace as Tan
19:
end if
21:
end if
22: else
23:
Replace Tret at given interval
24:
end
25:
end for
Algorithm 2 O2TD3
Require: offline policy Tat, offline dataset D. factor k. threshold т, interaction interval # Optimistic critic reconstruction before online fine-tuning
1: Initialize the actor Ton with Toff and the critic Qon with random parameters
2: for iteration i 1, 1, 2....... do
3: Update the critic in the optimistic way by
b Policy re-evaluation
4: end for
5: for iteration i = 1, 2.... do
6:
Update Won to seek overestimated actions by
7: Update Q.n to suppress overestimated Q- -aluuu by 16
> Value Alignment
8: end for
# Constrained online fine-tuning
9: Set the reference policy Tyef as Ton and initialize the replay buffer R with D
10: for iteration i 2, do
11:
Interact with the environment and store the transition in replay buffer R Update the Lagrange multiplier À by447 47
12:
Constrained Fine-tuning
13:
Update the critic Qon by 46
14:
Update the policy Ton by 46
15:
if evaluation permitted then
16:
if % interaction interval = 0 then
17:
J(TTn) > J(TTet then Replace Teel Ton
18:
19:
end
20:
end if
21: else
22:
Replace niee Ton given interval
23:
end if
end for
33

---

Algorithm 3 O2PPO
Require: offline policy Toft offline dataset D, factor a, interaction interval, update interval Optimistic critic reconstruction before online fine-tuning
1: Initialize the actor Tun. with Ton and the critic Von with random parameters > Policy re-evaluation
2: for iteration i = 1, ... do
3: Update the critie Von in the optimistic way by fitting the returns of offline trajectories
4: end for
Value Alignment Constrained online fine-tuning
5: Set the reference policy Wref as Ton and initialize an empty replay buffer R
6: for iteration i 1,2, ..... do
7: Interact with the environment and store the transition in replay buffer
8: if' update interval 0 then
9:
for iteration = 1, 2, ... ... do
> Value Alignment & Constrained Fine-tuning
Compute advantage by [8
11:
Update the policy Ton by Update the critic Von by 8 end for
13:
14:
Reset R to empty end if
15:
if evaluation permitted then
17:
if % interaction interval == 0 then
18:
( > (TTet) then Replace Tee as Won
19:
20:
end end if
21:
22: else
23:
Replace Teet Tea given interval
24:
end
25:
end for
34

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We clearly claim our contributions on offline-to-online RL in both abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Since we re-evaluate the policy under Assumption 4.1 the policy should have favourable performance after offline training. And Section C.2 in Appendix C we further discuss why do not focus fine-tune policy with poor performance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
35

---

Answer: [Yes]
Justification: We provide the references of our assumptions and proofs for our proposed propositions in Appendix F.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose our algorithm details in Appendix G and our implementation details in Appendix H including the implementation of the baselines and our methods.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We upload our code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/ppll wttpcccpppppppppeeeeeeeeeeeeeeee httpa////giieee/eeeSSbiiiiisssss Coooeooomisssssseeeeeeee ssinsoliccyyyyyc.ccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We disclose our experimental setting in Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run methods in 5 random seeds and report the results including the average and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Since our methods do not focus on computational cost, and we experiment our methods in many devices with different GPUs, we do not provide the information on the computer resources specially.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't Itdn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics btttps://heurips.ccppbbbb itt.e//eeeeehicccccccieeeeeeeee
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics adequately and we make sure we followed the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
38

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use the open-source code and dataset, and they are explicitly cited in the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
39

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
40

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "yeAJOGQCGA", "reviewer": "Reviewer_QxHN", "summary": "The paper proposes a general framework to bridge offline-to-online RL. It first studies two types of issues in O2O RL: evaluation mismatch and improvement mismatch. The proposed method addresses these issues by combining policy re-evaluation, value alignment, and constrained online fine-tuning. Unlike prior methods, the proposed framework can be applied to any offline RL algorithm. In the experiment, the method was built on top of CQL/SAC, TD3+BC, and IQL/PPO, demonstrating its performance in D4RL tasks.", "questions": "The questions are included in the previous section.", "limitations": "The limitations are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The breakdown of the two mismatch problems is interesting.\n\n2. The proposed method is compelling in that it can generally bridge any offline RL and online RL algorithms.\n\n3. The results in D4RL locomotion tasks are solid.\n\n4. Overall, the paper is well-written and contains informative ablation studies.", "weaknesses": "1. [Major] It is a bit confusing to me which offline pre-training methods were used for each result. Is it correct that all the results from O2SAC are pre-trained with CQL, O2TD3 is pre-trained with TD3+BC, and O2PPO is pre-trained with IQL? I suggest mentioning these details more clearly in the paper.\n\n2. [Major] While the paper suggests that the proposed method is universal to the offline RL algorithm, there is not much comparison of using the same online RL algorithm with different offline pre-training methods. Can you provide ablations of running O2SAC on the same task with different offline methods, such as CQL, IQL, and ODT?\n\n3. [Major] While the results for the harder AntMaze tasks (antmaze-medium/large) are briefly mentioned in the appendix, could the authors provide the full comparisons to previous methods and add them to Table 1?\n\n4. [Major] Is it possible to include results on the Adroit binary task, as in [1, 2, 3], which is a common benchmark for studying the sample efficiency of online RL with offline data?\n\n5. [Minor] Figure 5 (a) is a bit confusing to me. Which plot corresponds to the unconstrained fine-tuning mentioned below?\n>(l.550) For O2SAC, unconstrained fine-tuning suffers from a performance fluctuation, and direct fine-tuning from offline may lead to faster performance improvement, but drops sharply in the subsequent phase, e.g. in Figure 5(a)\n\n6. [Minor] For the value alignment objective in Eq.13, can the authors explain more on why L_retain is needed? What will happen if we only use L_align as the objective?\n\n7. [Minor] It would be interesting to see if the proposed method can be combined into a recent sample-efficient online RL algorithm that can use a high UTD ratio, such as RLPD [3].\n\n8.  [Minor] The legend for Figure 3 (c) seems to be incorrect: O2TD3 -> O2PPO\n\n[1] Nair et al., AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, 2020\n\n[2] Nakamoto et al. Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning, 2023\n\n[3] Ball et al., Efficient Online Reinforcement Learning with Offline Data, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and positive recognition of our paper. We are glad that you consider our work “interesting, solid, well-written”. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.\n\n**Q1:** Suggest mentioning the offline pretrained algorithms used for O2SAC, O2TD3 and O2PPO more clearly in the paper.\n\n**A1:** Thanks for your valuable suggestion. We adopt suitable offline algorithm according to the consistency of the policy form just for simple experiments. For example, both CQL and O2SAC adopt the squashed Gaussian policy. In fact, any offline algorithm can be used for the initialization of our methods with a change of policy form through behavior cloning, as discussed in Appendix I of our paper. And we will mention the offline pretrained algorithms used for O2SAC, O2TD3 and O2PPO more clearly in the paper.\n\n**Q2:** While the paper suggests that the proposed method is universal to the offline RL algorithm, there is not much comparison of using the same online RL algorithm with different offline pre-training methods. Can you provide ablations of running O2SAC on the same task with different offline methods, such as CQL, IQL, and ODT?\n\n**A2:** Thanks for your suggestion. We provide the results in Fig. 1 of the uploaded PDF. The initial performance of O2SAC initialized from ODT is lower than others since the simple behavior cloning (we directly maximize the likelihood of the actions output by offline ODT while keeping an appropriate entropy) could harm the performance, as discussed in Appendix I. But in _hopper-medium-v2_, the performance improves quickly. We analyze that by the constraint, the policy can recover the offline performance (about 97 normalized score of ODT), as the output of the cloned policy is near the ODT policy. We will include these results and discussion in the revised version.\n\n**Q3:** While the results for the harder AntMaze tasks (antmaze-medium/large) are briefly mentioned in the appendix, could the authors provide the full comparisons to previous methods and add them to Table 1?\n\n**A3:** Since some work does not give the hyper-parameters of these tasks and TD3+BC performs extremely poor (almost 0) on these tasks resulting in the terrible initialization for O2TD3, it may be inappropriate to add the comparisons to Table 1.  However, we did some comparisons shown in Table 1 of the uploaded PDF. We are willing to take more time to compare with more algorithms like ODT and give a separate table in the revised paper. \n\n**Q4:**   Is it possible to include results on the Adroit binary task, as in [1, 2, 3], which is a common benchmark for studying the sample efficiency of online RL with offline data?\n\n**A4:** We tried the experiments on Adroit binary tasks but found some difficulties of getting a favourable offline policy by CQL and TD3+BC since the corresponding hyper-parameters are not given in the papers. We tried the hyper-parameters of antmaze tasks but achieve extremely poor performance (almost 0). And we tried cloning a policy by the offline policy learned by IQL, but the performance of the cloned policy is still poor. We cannot provide the corresponding results at the moment for O2SAC and O2TD3. In Fig. 2 of the uploaded PDF, we show the results of O2PPO initialized by IQL. Since the Adroit binary tasks return the sparse rewards like antmaze, given the effectiveness of antmaze tasks, it is reasonable to think that our methods can be applied to the Adroit binary tasks.\n\n**Q5:**   Figure 5 (a) is a bit confusing to me. Which plot corresponds to the unconstrained fine-tuning mentioned below?\n\n**A5:** We apologize for the confusion caused by the mismatched expression. The expression refers to plots in a previous version of our paper, but we did not update the expression after revising the paper. Thanks for pointing this out and we will correct it in the revised paper.\n\n**Q6:**   For the value alignment objective in Eq.13, can the authors explain more on why L_retain is needed? What will happen if we only use L_align as the objective?\n\n**A6:** The role of L_align is to suppress the Q-values of OOD actions. At the beginning of value alignment, the target Q-values of OOD actions can be extremely low as $log \\pi_{off}(a_{ood}|s)$ can be extremely low, resulting in an overall or even catastrophic underestimation of Q-values, thereby destroying the optimistic property after policy re-evaluation. The role of L_retain is to keep the optimistic property by keeping the Q-values of reliable actions $\\dot{a}$, that is necessary for value alignment since we take them as the anchors to calibrate the Q-values. \n\n**Q7:**   It would be interesting to see if the proposed method can be combined into a recent sample-efficient online RL algorithm that can use a high UTD ratio, such as RLPD [3].\n\n**A7:** Yes, this is indeed another advantage of our method. Since we only add a constraint that can be considered as part of the reward, the policy iteration process remains consistent with the normal online approach, which makes it feasible to incorporate techniques from advanced efficient RL algorithms. We conducted some experiments using a high UTD ratio of 10 (but still update the lagrangian multiplier once per step) and achieved better performance improvement, as shown in Fig. 3 of the uploaded PDF. We will include these results and discussions in the revised version.\n\n**Q8:**   The legend for Figure 3 (c) seems to be incorrect: O2TD3 -> O2PPO\n\n**A8:** Thanks for your careful review. We will correct this mistake in our revised paper.\n\nThank you for your insightful review again. We hope we have resolved your concerns. We are always willing to answer any of your further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YqqCtKxkyF", "reviewer": "Reviewer_zZK3", "summary": "This paper proposes to handle evaluation and improvement mismatches in offline-to-online RL. To this end, the authors suggest to re-evaluate the pessimistic critic and calibrate the misaligned critic with the reliable offline actor. Then, they perform constrained fine-tuning. They evaluate the performance in the standard offline-to-online RL benchmark.", "questions": "- Does re-evaluation require heavy computation? Is it more efficient compared to RLPD [1], which initialize replay buffer with offline dataset instead of offline pre-training?\n\n[1] Ball, Philip J., et al. \"Efficient online reinforcement learning with offline data.\" ICML 2023", "limitations": "See Weaknesses and Questions.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework is theoretically analyzed.\n- Experimental evaluation is extensive.", "weaknesses": "- Difficult to follow up on technical novelties. I suggest adding concept figures or algorithm tables to highlight core contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our paper. We are glad that you consider our work “theoretical analyzed”. We are glad to answer all your questions.\n\n**Q1:**   Difficult to follow up on technical novelties. I suggest adding concept figures or algorithm tables to highlight core contributions.\n\n**A1:** Thanks for your valuable suggestion. To facilitate understanding, I would like to elucidate the techniques simply again. As the analysis in our paper, in O2O scenarios, evaluation and improvement mismatches are common. Recognizing that the offline policy is well-trained and trustworthy, we first utilize FQE to re-evaluate the critic optimistically, as in the online process. Given factors like partial data coverage, we then calibrate the critic with the offline actor to achieve an optimistic and reliable critic. Finally, to address distribution shift, we incorporate CMDP into online fine-tuning for stable and efficient performance improvement. We have included the pseudocodes in Appendix J.\n\n**Q2:**  Does re-evaluation require heavy computation? Is it more efficient compared to RLPD [1], which initialize replay buffer with offline dataset instead of offline pre-training?\n\n**A2:** Since the policy is fixed and the re-evaluation of the critic is straightforward, the computational cost of re-evaluation is significantly lower than that of offline learning. In our experiments, the re-evaluation takes about 1 hour for O2SAC (whereas CQL takes more than 5 hours) and 40 minutes for O2TD3. For O2PPO, updating the critic by fitting the returns independently of the policy results in even lower time costs.\n\nGenerally, in O2O RL, we are typically not concerned about the time cost in the offline process, as different offline methods take different amounts of time. Instead, we prioritize the cost of interactions during online fine-tuning. Our method re-evaluates and aligns the critic with the offline actor solely within the offline dataset, making the time cost less critical.\n\nMoreover, it is challenging to directly compare our method with RLPD because they address different objectives. RLPD focuses on learning a policy from scratch using a given dataset, whereas our method aims to improve a well-trained policy in O2O scenarios with limited interactions in a stable and efficient manner. Nonetheless, we acknowledge that integrating RLPD techniques could potentially enhance our method's performance, as our approach imposes minimal additional constraints during the online process. This is an interesting future direction of our work.\n\nThank you for your review again. We hope we have resolved your concerns. We are always willing to answer any of your further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NH6zA9j6jD", "reviewer": "Reviewer_9DG2", "summary": "The paper addresses the offline-to-online (O2O) reinforcement learning problem with the goal of improving online performance by leveraging offline data. The primary contributions of this paper are twofold. First, it identifies and elaborates on two key challenges in O2O RL: evaluation and improvement mismatches, which differentiate offline and online RL. Second, it introduces a general method for transferring knowledge from any offline approach to three representative online methods. Both theoretical analysis and empirical experiments thoroughly validate the effectiveness of the proposed method.", "questions": "Although the method appears straightforward, it comprises three components. What is its time complexity? The paper is suggested to include an analysis on time complexity.", "limitations": "The paper does not have a Limitation section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance: In contrast to existing works, this paper is the first to summarize two mismatches between offline and online methods. These mismatches, which relate to two types of offline approaches, reveal their negative effects on subsequent online fine-tuning.\n\nContribution: The proposed method effectively balances specificity and flexibility. To address the two distinct mismatches, the paper introduces policy re-evaluation and value alignment techniques, which yield optimistic Q-value estimates and accurate Q-value calibration, respectively. Furthermore, the method is applicable to a broad range of representative online methods, demonstrating its wide applicability.\n\nSoundness: The experiments are thorough and strongly support the claims of the paper, including the motivation and effectiveness. Compared to multiple SOTA methods, the proposed method shows superiority.", "weaknesses": "The method consists of three components. The first two components are developed to solve two mismatches. How about the third component? Is there another key issue in O2O learning tasks, such as another mismatch?\n\nIn value alignment, regarding different online methods, the paper develops different strategies. Does this imply that the method lacks generalizability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your high praise and comprehensive review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.\n\n**Q1:** The method consists of three components. The first two components are developed to solve two mismatches. How about the third component? Is there another key issue in O2O learning tasks, such as another mismatch?\n\n**A1:** There is indeed another key issue in O2O RL, known as _distribution shift_ in previous work [1][2][3]. This issue arises from the discrepancy between the offline dataset and the interactive data collected by the offline policy, which can negatively impact performance improvement. This challenge occurs whether the policy is evaluated using suboptimal offline data or high-quality but limited data. Although we strive to maintain an optimistic property for the critic and align the critic with the actor, achieving stable online fine-tuning remains challenging due to the inevitable _distribution shift_.\n\nMost current offline algorithms focus on avoiding OOD actions and training a reliable policy on the states present in the dataset. However, due to the inherent optimism in online RL, encountering OOD states and actions is unavoidable, leading to performance fluctuations. This is particularly problematic in critical scenarios, especially high-risk ones. For OOD states, even if the policy is well-trained during the offline phase, it may still fail to produce favorable actions, potentially causing erroneous policy updates.\n\nTo conclude, while optimistic critic reconstruction can guarantee stable and efficient performance improvement initially, it is essential to implement constrained fine-tuning in later stages to maintain continued stability.\n\n**Q2:** In value alignment, regarding different online methods, the paper develops different strategies. Does this imply that the method lacks generalizability?\n\n**A2:** It is important to note that the different strategies correspond to different policy types and update mechanisms, as we study the problems of O2O settings from the perspective of online RL. In online RL, the alignment ways between the critic and the actor can vary significantly due to different foundational designs.\n\nO2SAC is designed for stochastic policies updated in an off-policy manner; O2TD3 is for deterministic policies updated in an off-policy manner; and O2PPO is for stochastic policies updated in an on-policy manner. These three methods are representative and cover the major categories of existing mainstream online algorithms, making it straightforward to incorporate other advanced techniques. Thus, rather than indicating a lack of generalizability, the developed strategies demonstrate our method's adaptability to a wide range of policy types and update mechanisms in online RL.\n\n\n**Q3:** Although the method appears straightforward, it comprises three components. What is its time complexity? The paper is suggested to include an analysis on time complexity.\n\n**A3:** In policy re-evaluation, since the policy is fixed and the re-evaluation of the critic is straightforward, the computational cost of re-evaluation is significantly lower than that of offline learning. The time cost of value alignment is somewhat higher but still less than that of the offline phase. In fact, the time cost is approximately proportional to the offline phase according to the alignment steps, since both the actor and critic are updated in the value alignment phase. \nWe conducted an experiment on the _hopper-medium-v2_ environment using the O2SAC method and listed the time cost in different phases as follows.\n\n| Trainning Phase | Offline(CQL)| Policy Re-evaluation | Value alignment|\n| -----------     | ----------- | -----------          | -----------    |\n| Trainning Steps | 1M          | 0.5M                 |  0.5M          |\n| Time Cost       | 5.4h        | 0.95h                |  2.0h          |\n\nHowever, it is worth noting that although we set the training steps for value alignment at 500k, in some environments, only a few alignment steps are needed to calibrate the critic with the offline actor, as shown in Fig. 1 and Fig. 11. Only in _antmaze_ environments, where it is hard for the critic to capture the sparse reward signal, more alignment steps are necessary. Additionally, in constrained fine-tuning, since only the lagrangian multiplier is added to be updated and the interaction cost dominates, the time cost increases very little. \n\nMoreover, in O2O RL, we are typically not concerned about the time cost in the offline process, as different offline methods take different amounts of time. Instead, we prioritize the cost of interactions during online fine-tuning. Our method re-evaluates and aligns the critic with the offline actor solely within the offline dataset, making the time cost less critical.\n\nWhile the time cost is not the main concern, we will include a detailed analysis of the time complexity in the revised paper to provide a clearer understanding of the computational requirements for each component.\n\n[1] Lee et al. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble, CoRL 2022.\n\n[2] Yu et al. Actor-critic alignment for offline-to-online reinforcement learning, ICML 2023.\n\n[3] Nakamoto et al. Cal-ql: Calibrated offline rl pre-training for efficient online\nfine-tuning, NeurIPS 2023.\n\nThank you for your constructive review again. We hope we have resolved your concerns. We are always willing to answer any of your further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rkLIxK5jZ2", "reviewer": "Reviewer_aiGQ", "summary": "The paper proposes a general offline-to-online (O2O) reinforcement learning method that can work with any offline RL algorithm. It addresses evaluation and improvement mismatches between offline datasets and online environments by (1) Re-evaluating the offline critic optimistically; (2) Calibrating the critic with the offline actor; (3) Performing constrained online fine-tuning. This approach shows stable and efficient performance improvements across various simulated tasks compared to existing methods.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2/ The re-evaluation and calibration procedures are indeed very crucial for improving offline-to-online RL training, the authors make a great effort to show the importance of such procedure, with both empirical evidence and theoretical guarantees.", "weaknesses": "There are several minor presentation issues, not very critical, but they significantly affect the visual quality of the paper.\n1. In line 86, equation 2, the subscription under the two expectations is getting too close, making the formula a bit messy.\n2. In line 132, equation 9, the parenthesis of the first $f$ should be larger, so that it covers the inputs.\n3. In figure 3, the colors of some curves are very similar, making it hard to tell the performance of each method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you for the high praise and the comprehensive review of our paper.  \n\n**Q1:** There are several minor presentation issues, not very critical, but they significantly affect the visual quality of the paper.\n\n**A1:** Thanks for your suggestions. We have revised these language mistakes. We will proofread the paper and revise writing issues carefully.\n\nThank you for your review again and we appreciate your suggestions for a better presentation. We are always willing to answer any of your further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for your thorough evaluation of our paper. Your constructive comments have been invaluable in helping us enhance our work.

Meta-review: This paper analyzes two issues in offline-to-online RL: evaluation mismatch and improvement mismatch. They propose a method to address these  mismatch issues via a combination of techniques such as value re-evaluation, alignment, and constrained fine-tuning. Overall, the results in the paper are convincing in that this approach can improve over offline policies produced from arbitrary methods. 

The reviewers generally liked the paper and found their concerns addressed, so we are accepting this paper for now. The AC personally found the presentation of the paper and the ideas a bit confusing and not entirely motivated: with so many variants of the approach presented, and with some of these being a bit less concretely defined: e.g., the alignment loss for TD3, SAC are a bit ad-hoc mostly based on approximate intuition, the constrained fine-tuning loss is largely disconnected from the mismatch intuitions, the method is complex and needs dataset-specific tuning (as compared to domain-specific tuning). We encourage the authors to improve the clarity of the paper, make it clear upfront what the components of the method would be and strive to provide more guidance to researchers building on this approach.
Decision: Accept (poster)