Actor-Critic Alignment for Offline-to-Online Reinforcement Learning
Authors: Zishun Yu
Xinhua Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning
Offline-to-Online

Abstract: Deep offline reinforcement learning has recently demonstrated considerable promise in leveraging offline datasets, providing high-quality models that significantly reduce the online interactions required for fine-tuning. However, such a benefit is often diminished due to the marked state-action distribution shift, which causes significant bootstrap error and wipes out the good initial policy. Existing solutions resort to constraining the policy shift or balancing the sample replay based on their online-ness. However, they require online estimation of distribution divergence or density ratio. To avoid such complications, we propose deviating from existing actor-critic approaches that directly transfer the state-action value functions. Instead, we post-process them by aligning with the offline learned policy, so that the Q-values for actions *outside* the offline policy are also tamed. As a result, the online fine-tuning can be simply performed as in the standard actor-critic algorithms. We show empirically that the proposed method improves the performance of the fine-tuned robotic agents on various simulated tasks.

Under review as a conference paper at ICLR 2023
ACTOR-CRITIC ALIGNMENT FOR OFFLINE-TO-ONLING REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep offline reinforcement learning has recently demonstrated considerable promise in leveraging offline datasets, providing high-quality models that signif- icantly reduce the online interactions required for fine-tuning. However, such a benefit is often diminished due to the marked state-action distribution shift, which causes significant bootstrap error and wipes out the good initial policy. Exist- ing solutions resort to constraining the policy shift or balancing the sample replay based on their online-ness. However, they require online estimation of distribution divergence or density ratio. To avoid such complications, we propose deviating from existing actor-critic approaches that directly transfer the state-action value functions. Instead, we post-process them by aligning with the offline learned pol- icy, so that the Q-values for actions outside the offline policy are also tamed. As result, the online fine-tuning can be simply performed as in the standard actor- critic algorithms. We show empirically that the proposed method improves the performance of the fine-tuned robotic agents on various simulated tasks.
INTRODUCTION
Offline reinforcement learning (RL) provides novel tool that allows offline batch data to be lever- aged by RL algorithms without having to interact with the environment (Levine et al., 2020). This opens up new opportunities for important scenarios such as health care decision making, and goal- directed dialog learning. Due to the limitation of offline data, it generally remains beneficial and necessary to fine-tune the learned model through online interactions, and ideally the latter will enjoy a faster learning curve thanks to the favorable initialization.
Unfortunately, it has been long observed that direct offline-to-nllne (020) transfer often leads to catastrophic degradation of performance in the online stage, which is unacceptable in critical applications including medical treatment and autonomous driving. A key cause lies in the significant shift of state distribution at online phase compared with the offline data (Fujimoto et al., 2019; Kumar et al., 2019; Fu et al., 2019; Kumar et al., 2020a). As result, the Bellman backup suffers compounded error (Farahmand et al., 2010; Munos, 2005), because the Q-value has not been well estimated for the state-actions lying outside the offline distribution.
number of solutions have been developed to address this issue. The most straightforward approach importance sampling (Laroche al., 2019; Gelada & Bellemare, 2019; Zhang et al., 2020; Huang & Jiang, 2020), which requires an additional effort of estimating the behavior policy, and suffers from high variance, especially when it differs markedly from the learned policy (a more realistic issue for the offline setting than the conventional off- off-policy setting). The model-based approach, on the other hand, also suffers from the distribution shift in state marginals and actions (Mao al., 2022; Kidambi et al., 2020; Yu et al., 2020; Janner et al., 2019). It may exploit the model to pursue out-o--distribution states and actions where the model mis-believes to yield a high return. So they also require detecting and quantifying the shift. In addition, they suffer from standard challenges plaguing model-based RL algorithms such as long horizon and high dimensionality.
Dynamic programming proffers lower variance and directly learns the value functions and policy. Several approaches have been proposed to combat distribution shift. A natural idea is to constrain the policy to the proximity of the behavior policy, and this has been implemented by using probability divergences in (Nair et al., 2020; Siegel et al., 2020; Peng et al., 2019; Wu et al., 2019: Kumar al., 2019), by behavior cloning regularization (Zhao et al., 2021; Fujimoto & Gu, 2021). A

---

Under review as a conference paper at ICLR 2023
second class of approaches resort to pessimistic under-estimate of the state-action values (Kumar et al., 2020b; Kostrikov et al., 2021), especially for -ustof-distribution actions that could have an unjustified high value. Conservative Q Q-learning (CQL, Kumar et al., 2020b) has been shown to produce a relatively safer 020 transfer in balanced replay (Lee et al., 2022), which further prioritizes the experience transitions that closer to the current policy.
Unfortunately, all these methods require online estimation of distribution divergence or density ratio (for priority score regularization weight). Excess conservatism can also slow down the online fine-tuning. A third category of methods avoid these complications by estimating the epistemic uncertainty of the Q-function, so that -itto---istribttion actions carry a larger uncertainty which in turn yields conservative target values for Bellman backup (Jaksch et al., 2010; O' O'Donoghue et al., 2018; Oshand et al., 2016; Kumar et al., 2019). However, uncertainty estimates, especially for deep neural nets (Fujimoto et al., 2019).
generally hard to find calibrated
To resolve the aforementioned issues, we propose a novel alignment step for actor-critic RL that can be flexibly inserted between offline and online training, dispensing with any estimation of Q- function uncertainty, distribution divergence, or density ratio. Our key insight is drawn from soft actor-critic (SAC, Haarnoja et al., 2018), where the optimal entropy-regularizee policy is simply the softmax of the Q-function. Now that the Q-function is generally problematic for -iittr-diitiibttioon actions while the policy learned offline is assumed trustworthy (though still needs fine-tuning), it is natural to align the critic to the actor upon the completion of offline learning, so that the Q function tamed to be consistent with the policy under the softmax function, especially for those actions that lie outside the behavior policy, As a result, the online fine-tuning will only need to take the simple form of the standard SAC, and empirically the proposed method outperforms state-of-the-an fine-tuned robotic agents on various simulated tasks.
Our contributions and novelty can be summarized as follows:
We propose novel 020 RL approach that outperforms or matches the current SOTAs.
Our approach does not rely on offline pessimism conservatism, allowing it to transfer broader range of offline models.
We propose, for the first time, discarding Q-values learned offline as a means to combat distribu- tion shift in 020 RL. We also design novel reconstruction of Q-functions for online fine-tuning.
When offline data not available at online fine-tuning very realistic scenario due data privacy concerns, our method remains applicable and stable, while strong competitors such as balanced replay cease being applicable.
2 RELATED WORK
Decision transformer (Chen et al., 2021) and trajectory transformer (Janner et al., 2021) have cently been shown effective for offline reinforcement learning, where the batch trajectories' likeli- hood is maximized rutt-seerrssively to model action sequences conditioned (2022) extended them to online decision transformers (ODTs) by populating the replay buffer with online ODT rollouts labeled with hindsight experience replay. As result, sequence modeling be- comes effective for online fine-tuning. Our method remains in the actor-critic framework, and we demonstrate similar or superior empirical performance to ODT.
task. Zheng et al.
Behavior cloning often plays an important role in effective 020 RL. It can take the form of con- straining the policy around the behavior policy under certain probability discrepancy measure, or simply imposing least square or cross-entropy regularizer to drive the policy to imitate transitions (Zhao al., 2021: Fujimoto & Gu. 2021). Such regularizer often requires delicate annealing, and to this end, Zhao et al. (2021) designed heuristic rules based on reward feedback. Recently, Kostrikov et (2022) employ behavior cloning to guide the extraction of policy from an expectile- based implicit Q-learning.
is noteworthy that behavior cloning is also commonly used in imitation learning, where the goal is to imitate instead of outperforming the demonstrator, differing from the 020 setting. A number of efforts have been made to fuse with RL for improvement (Lu et al., 2021). A similar line of research is to boost online learning from demonstration, (e.g., Hester et al., 2018; Reddy et al., 2019). However, they focus on accelerating online learning by utilizing offline data, and are not concerned about the safety or performance drop in porting the pre-trained policy to online.
2

---

Under review as a conference paper at ICLR 2023
3 PRELIMINARY
We follow the standard protocol that formulates a RL environment as a Markov decision process (MDP). An MDP M is often described 5-tuple (S, A, 7), where S is the state-space, A the action space, P function, and y E [0, 1) is a discount factor. A policy is distribution ((a)s) E (AA), and the agent aims to find a policy that maximizes the expected return Ex -
x A + A(S) is the transition function, S.S. x A - is the reward
Soft actor-critic To learn from offline data generated by behavior policy, we will focus on off- policy RL methods. In particular, the soft actor-critic method (SAC, Haarnoja et al., 2017: 2018) learns a Q-function Qu a) with parameter μ., and a Gaussian policy (aaas)) whose sufficient statistics are determined by neural network with parameter A. Let d be the empirical distribution corresponding the replay buffer, and we intentionally left it flexible on state, state-action, or transition. Then SAC alternates between updating the critic and actor by minimizing the following respective objectives:
C, d) == -B [a log (e(a)s) Qu(s,)),,
(1)
CSAC (j,, (1, d) = (s,,,, dd) dir-d
(Qu(s,a) v(r, {
2
(2)
where y(r,s', ((r)s',d)== : 7(1 d)B 1Q (sss's') a log To (a'lss))
(3)
Here, ar >> is the temperature parameter, and is the delayed Q-function parameter. If na is based on universal neural network, its optimal value that minimizes LSAC (0, d) admits closed form:
To (als) exp (xp (HQu(s,))) EnEA
/ exp ())
In practice, one simply performs gradient descent steps on LSAC because even if the network is universal, the value that corresponds to the optimal solution (4) hard to find.
It is important to note that adding baseline function (s)) to Qn (s, a) does not change the optimal Ty in (4), as long as (s) does not depend on a. Therefore, given Ta. Qu(s, a) can be inferred as Qu(s, a) = ((s) + a logTTe(ass) Te(als),
(5)
where Z(s) provides additional freedom to fit other aspects of the problem; see Section 4.2.
ALIGNING CRITICS WITH ACTORS FOR OFFLINE-TO-ONILEE RL
We now detail our method that consists of three phases: offline, actor-critic (AC) alignment, and online. The whole procedure summarized in Table 5 in Appendix A.
4.1 OFFLINE TRAINING
Motivating our offline training is TD3+BC (Fujimoto & Gu, 2021), which runs TD3 (Fujimoto al., 2018) on the offline dataset with a behaviour cloning (BC) regularization (Bain & Sammut, 1995). Similar approaches such as SAC+BC can be found in Nair et al. (2020). However, we replaced TD3 with SAC to enable stochastic policies and to be consistent with the subsequent AC alignment, where the Q-function is obtained in closed-form under maximum entropy RI. We also replaced the BC regularization with a maximum likelihood (ML) regularizer, in order to be consistent with the online phase that also uses an ML regularizer (see Section 4.3), As a result, we naturally name our offline method SAC+ML. We will compare our SAC+ML against TD3+BC in Appendix C.
Actor update. Let d be the empirical distribution a mini- batch sampled from the offline dataset D. The actor update of TD3+BC and SAC+ML aims to minimize the following respective objec- tives:
CD3-BC (8, d) (
x)) (b- -0)-)
(6)
CSAC+ML (0,d) B..gaaad BB
((Qu(ss)b) (Blossb)-ologmm(bbs))) olog)o(bls)) log log
(7)
3

---

Under review as a conference paper at ICLR 2023
where the hyperparameter À balances Q values with the BC/ML regularization. In practice, employed the clipped double Q-learning technique (Hasselt, 2010) to train two Q-networks Qp, and Quz- It is beneficial for both offline and online training (Fujimoto al., 2018). is then set to
x: w/ (u,
where Q₂ min Quz),
(8)
and w is a predetermined syper-parameter So À is recomputed after every critic update, requiring almost no additional computation.
Critic update. SAC+ML follows the same critic update as SAC in (2), except for the double part: CQ
CSACHMI (Me d) (s.a.,, (Q (s. u(r, ((r,,''d)))
(9)
where v(r, s ('):s'd))=== :0)=- y(1 - d) (ro (Qp(s'
log To To (d'ls');
(10)
and j is delayed version of μ. a.k.a. target network, with Qn a) min Q (s, to Q,.- The temperature update tries to reduce the following objective over a > 0:
akin
LSACtMI (a, d) == ReseeBaa [- (log no (a|s) H}] Ciemp
(11)
Here, H. > 0 is the target entropy value, hyper-parameter specified priori. The Lagrange mul- tiplier a is automatically tuned in (11), enforcing the upper bound of the entropy of my by H. The pseudo-code of SAC+ML is relegated to Appendix A.
4.2 ACTOR-CRITIC ALIGNMENT
At the end of offline learning, the learned policy Te, often performs reasonably well, and is ready for online fine-tuning. So denote the policy with index O. conventional actor-critic, the critic is supposed to be updated frequently enough to accurately pursue the state-action values for the current policy. However, even if such updates are conducted proactively, the distribution shift problem in 020 still plagues the critic under deep net approximation. because the Q-values are not trustworthy beyond what has been visited under the behavior policy. So the over-estimated Q-values can rapidly destroy the learned actor and critic through Bellman backup.
In order to avoid this issue, we propose taming the out-of-distribution Q-values by directly aligning the critics with the actors, as a post-pros post-processing step for offline learning, or an initialization step for online learning. In particular, inspired by (5), we choose to discard the Qp. learned from the offline phase, and reset them into!
(i a) = log ma, (a|s) + Zo (s).
(12)
The baseline Za, (s) can be naturally calibrated by minimizing the Bellman residual on offline data:
C>
CSCC+ML (W5s, d) = E. (s,,,,,,'' .dydd (log To, (als) + Zo. (s) v(r,s',d))" d))²]
(13)
where y(r s' d) = + 7(1 d) E (1 [log Te, (a') (a'ls') + Z.
(14)
Z == m 7.
(15)
Here, Ze in (14) employs standard semi-gradient. This optimization simply a regression prob- lem and be conducted by Adam (Kingma & Ba, 2015). The details are deferred to Appendix A.1, where pseudo-code is also given in Algorithm 1.
Generality. Thanks this alignment step that disregards the Q-function learned offline, the offline learning algorithm is not limited to SAC+ML, even though it is more favorable. In Section 6.3, we will show that our alignment approach can be well applied to the offline policy learned from CQL.
4.3 ONLINE TRAINING
During the online fine-tuning, we restore the full flexibility of Q-functions by using the following parameterization:
Qs. a) log ne, (als) + (s, where Rø, (s, a) is initialized with Ze, (s).
(16)
Compared with (5), it appears that we have set a there to 1, while its value at the end of offline learning is rarely close to 1. This creates no contradiction, however, because log no, will be used to parameterize the online Q-function as in (16), and the a for online phase SAC is initialized to 1. So their product, passed through the softmax, will recover T8g-
4

---

Under review as a conference paper at ICLR 2023
Such initialization can be simply implemented by loading the weights of Zy; and setting the weights corresponding to action to zeros. It is noteworthy that one should refrain from constrain- ing Q to closed-form manifold induced by the latest To throughout the online phase, i.., setting Qo, (s, a) to log To ro(a(s) ) + Zo, (8) for some trainable baseline Ze.-- This is because it would lead to no improvement of the policy. As such, we only leverage the closed-form for initialization.
The update on temperature is exactly the same as (11), and the update on critic resembles that of the offline phase in (9), except that the training variable is now only Rg (s, a). In particular, we adapt SAC critic update to our Qo. along with standard tricks of target network and double Q-clipping:
Co(os, d) == Ess,a,,, (d)=a (
(oog To, (als) + Ro. a) y(r, ((rr,'' -
(17)
where v(r, 8 d) = + 7(1 Bdd
[lo To. (a'ls') + (5', - ar log To (a' [d|s')] (18)
The actor's objective follows from the vanilla SAC, and can be written as follows with d being a mini-batch sampled from the replay buffer, R2 == minsg (1,2) Rø. and Qa log Te + RØ:
(0, d) == -Bgaaalornod [Qo(s, a) a log To [a]s)]
(19)
= El (s, a) a log ne (a)as)) - Basd (11as [log ne (als))
(20)
penalizing deviation of from Rep
Behavior cloning in (20). Naturally unfolding from SAC using the parameterization (16) is the expectation of log- legliike of Ta, under To in (20), a maximun-likelihood term that enforces the actions favored by the new policy T to also enjoy a high log-likelihood under the offline policy Too Different from AdaBC (Zhao et al., 2021), we sidestepped an ad-hoc introduction of behavior cloning regularization and tweaking of its weight. This regularization is not applied on the offline data, but on the policy Te, achieved by offline learning. To be consistent, we also used the maximum- likelihood regularization in the offline training.
One might argue that this interpretation is artificial because, after all, the log TT, term can be sub- sumed into the free variable Rs, in (16), obliterating this BC regularizer in (20). This in fact makes sense if the entire optimization is convex and the range of RR, as a function set is closed under addition with log Te, However, since Rág is a neural network, such conditions do not hold true. As result, the composite form in (16) does play crucial role in the good empirical performance, which is manifested in our ablation study in Section 6.5.
In practice, we also introduced two techniques to stabilize online learning. The first f-clipping trick addresses the excessively large magnitude of log Te, by capping its absolute values. The second critic interpolation gives the flexibility to balance between safety transfer and policy improvement. For the sake of space, they are deferred to Appendix A.2.
ILLUSTRATION OF ALIGNMENT UNDER DISTRIBUTION SHIFT
We first demonstrate how the critic alignment makes the Q-function more consistent with the real ac- tions in the offline dataset, compared with the Q-values learned from offline actor-critic. We trained SAC+ML on the halfchectah-meddie out-of-distributio states, we resorted to the malfcheetah-expenr dataset, and the details are available in Appendix D. Figure 5 there further illustrates the difference of these two state distributions.
dataset, and sampled in in-distribution states from it. To sample
Figure 1 (top row) compares the Q-values learned from SAC+ML and our alignedrreconstruuted Q-values, where the state-action pair is sampled in-distribution. The bottom row shows similar comparison, but on out-of-distribution samples. The actions have 6 dimensions, and for the i-th subplot, we perturbed the i-th dimension in [-1, 1]. with all the other dimensions fixed. Clearly, the offline learned Q-values are often inconsistent with the real action from the dataset, even for in-distribution samples. But our alignment much improves the consistency, which encourages the policy to stay close to the offline policies, safeguarding the process of transfer.
EXPERIMENTS
We next compared our actor-critic alignment method (ACA) with a number of state-of of t-th-artt art meth- ods as summarized in Table 1, Although CQL was not developed for 020 transfer, we still included
5

---

Under review as a conference paper at ICLR 2023
action
action
in-diettr
0
-1
0 -1
-1
0
out-or-distr
SAC+ML real action gned
Figure 1: SAC+ML Q-values v.s. aligned Q-values for in n--iisrribution sample (top row) and out-of- distribution sample (bottom row). Left y-axis: SAC+ML Q-values, right y-axis: aligned Q-values. Since only the trend of each curve matters, we omit the y-axis tick values.
Table Baseline algorithms for 020 RL. See acronyms below.
Flag
Name
Offline
Online
Description
SAC-ACA (Ours)
SAC+ML SAC+ML
Algorithm
Our method init from SAC+ML
SAC-SAC
SAC
SAC init from SAC+ML
CQL-BR
CQL
SAC w/ BR
Balanced replay init from CQL SAC init from CQL
CQL-SSCC
CQL
SAC
AWAC
AWAC
AWAC
AWAC init from AWAC
SAC
SAC
SAC from scratch
due to its strong performance. Implementation our ACA algorithm can be found anonymously Online Supplementary, along with the pre-trained models.
Our experiments aim to demonstrate:
SAC-ACA matches or outperforms SOTAs, e.g. balanced replay (BR, Lee et al., 2022), ad- vantage weighted actor critie (AWAC, Nair et al., 2020), and online decision transformer (ODT, Zheng et al., 2022);
Direct transfer such SAC- and CQL->SAC suffers significant performance drop;
Transfer from offline method significantly outperforms training SAC online from scratch. will additionally present ablation studies to examine various components of ACA.
6.1 COMPARISON WITH BASELINE METHODS
We used three environments from the datasets D4RL-v2 (Fu et al., 2020), including HalfCheetah, Hopper, and Walker2d. Each of them has five levels. All offline/online experiments 5 random seeds. We ran all offline algorithms for 500 episodes with 1000 mini- batches each, and all on- line experiments for 100 episodes with 1000 environment interactions each. This protocol is quite commonly used. More implementation details are deferred to Appendix B.
Figure shows the average return as a function of training episodes, achieved each offline model (left half of the subplots) and online model (right half). Since SAC-AACA and SAC-SAC share the same offline method, their curves coincide on the left of the subplots, with the green curve shown only (no blue) the left. A similar situation occurs to CQL--BR and CQL-+SC, and only the purple curve shown on the left half (no pink).
In Figure 2, CQL-SSC (purple->pink) drops significantly on the expert level (fifth row) and medium-expert level (fourth row). SAC-SAC (green-<blue) drops in almost all cases, except random (first row) and medium-replay (third row). It is clear that our SAC- (green) barely suffers performance drop. The only exception Hopper-medium expert, but all other methods (ex- cept AWAC which performs poorly offline) also suffer drop there. while ours recover most rapidly. Besides. ours offers comparable policy improvement to the strongest baseline, which CQL-BBR in most cases.

---

Under review as a conference paper at ICLR 2023
Offiine:
SAC+ML
OOL
AWACC
Online:
HellChreddhh-aaddmm-22
Hopper-ondommw22
1200
4000
XCCC
3030
400
200
HoirCheetsh medium
Hopper medium
Walker2d medium v2
700
5030
3000
2030
3000
HalfChestah- HallChettal-meiiu--pppp -am.aa-mmmiii---pla--2
Hopper medium replayyv2
Walker22 medium maadidm-rrppay-...
5000
700
3030
3000
2000
Haiff SallTTeeetabbmmefiimmeeeo 2
Heppar-meell wraaaappettpp
Wellbwe mol-malli mw-.pprt--p
4000
1000
4000
2030
2900
3000
200
50
200
40
HalfChentah expart v2
Hopper expert 02
Wallkor2d expert
4000
5000
3030
300
2000
200
102
104
200
66
Figure 2: Comparing SAC-+ACA (ours) with other baselines for offline-to-online RL. The shaded areas stand for the standard deviation. Refer to Table I for legend meanings.
Table Average normalized D4RL scores of various 020 methods. Outside parenthesis: scores the end of 100k online steps. Inside parenthesis: the increase of that score upon the end of offline training. HC HalfCheetah, H Hopper, Walker2d.
Score(E) AWAC
Dataset
Env
SAC-SAC
CQL-+SAC
CQI--BR
SAC-AACA (ours)
IIC
54.60(37.31)
54.00(28.70)
34.26(18 94)
84,36(59.00 59.06)
72.60(55.31)
Random
17.62(9.69)
1.37(0.72)
16.85(3.03)
29.80(29.15)
81.85(73.92)
W
3.86(0.50)
3.91(3.25)
4.15(-0.58)
10.05(9.39)
12.42(9.06)
HC
75.20(28 86)
69.52(21.10)
50.48(1.54)
82.95(34.52)
66.58(20.25)
Medium
H
73.39(19.08)
89.77(16.30)
97.53(24.48)
98.14(24.67)
96.54(42.24)
W
79.63(-1.53)
81.78(-0.70)
1.93(-0.54)
76.36(-611 11)
74.66(-6.50)
HC
68 68.90(26.37)
63.91(18.01)
46.84(2.44 22))
78.36(32.46)
59.03(16.50)
Medl-Replay
H
74.04(25.22)
92.01(-3.95)
95, 95.98(0.00)
97.25(1 (1.2)
(5.54(36.72)
W
40(23.2))
(9.28(0.89)
80.81(2.97)
100.06(21 68)
85.17(22.98)
HC
92-155(11.55)
87.85(46.77)
68.75(32.30)
91. 91.80K50.73 73)
93.74(0.21)
Med.-Experi
H
(5.44(-27.64) 64)
80.46( 13.26)
73.13(47.50)
78.51((15.22)
98.02(4.94)
W
87.18(-20.95)
107.03(-2.63)
45.21(4.45)
(04.43(-5.22)
110.54(2.42)
HC
(8.177-55.42)
(6.39(6.86)
21, 21.23(14.83)
79.69(31.16)
93.14(-0.46)
Expert
11
28. -8.2208882.8)
67 88(- 186--15.4 48)
(7.97(-12.85)
68.55(-14.81)
110.21(-0.67)
67.76(-40.45)
81.92(-27.244
110.68(0.80)
110.65(1.50)
109.59(1.38)
Total
901.54((69.880
1016.08(79.35)
805.80 (05.80(139.29)
1190.96(254.23)
1249.65(278.31)
Since different baselines in Table employ different offline methods, is not reasonable compare fine-tuning methods based only on their online performance. Therefore, we provided in Table 2 the increase of return achieved online compared with the final offline policy. As the numbers in the parenthesis there show, SAC-ACA attains the highest improvement (as well as the final score), and BR appears the best among all other baseline methods.

---

Under review as a conference paper at ICLR 2023
6.2 COMPARISON WITH ONLINE DECISION TRANSFORMER
Since ODT is not based on dynamic programming, we compared it with SAC-ACA in this separate section. As Zheng al. (2022) experimented using 200k online samples and averaged over 10 seeds, we ran SAC+MI, with 5 additional seeds and ran 200k online steps for all 10 SAC+ML runs, make the comparison fair.
Table 3:
Comparing SAC-ACA with online decision transformer (ODT), with a focus on the online improvement upon offline policy (dODT and dAA)).
Dataset
Environment
ODT((fftine)
OIDT(200K)
ΔDDT
SAC+MI.
ACA(2000)
ÒΑCA
HalfCheetah
42.7210.66
42. 12.16=1.46
-0.56
46.40=0.30
72.07=3.01
26.28
Medium
Hopper
60.9513.26 26
97.54
30.59
56.93=4.12
99.32:77.2
42.39
Walker2d
72.1946.49
76. 76.79=2.30
4.60
79.36=2.25
76.05+220.67
-3.30
HalfCheetah
39 19.9911..6
10.42.1.00 1.61
0.43
42. 0..5
64.29
22.11
Medl-Replay
Hopper
$6.6445.41
88, 38.89
2.25
49. 19.255=6.0
103,17
53,92
Walker2d
68.9214.70
76.86
7.94
63.20=10.12
82.09 68
18.89
Total (w/o hopper-mr)
290.77
333.77
43.00
288.06
394.41
106.35
Total (all)
377.41
422.66
45.25
337.31
497.58
160.22
As shown in Table 3, for almost all medium and medium-replay tasks, our SAC outperforms ODT in both final performance and performance increase (d). We also note that ODT(oflinne) out- performs SAC+ML the hopper-mediun-repla.e task by a large margin, which leaves our approach more room to improve. Therefore, we made the same comparison by excluding the hopper-medium- replay task. this case, ODT and ours were initialized from roughly the same performance, and ours still outperforms ODT in both total final performance and total performance increase.
6.3 FLEXIBILITY IN INITIALIZATION FROM DIFFERENT OFFLINE METHODS
A key advantage of our alignment method lies in the flexibility of leveraging any offline RL method, as long as outputs parameterized Gaussian policy, because the Q-function reset anyway. In contrast, SOTA methods sometimes require certain properties in the offline method such pes- simism. For example, BR's performance depends critically on the use of CQL.
To demonstrate our flexibility, we adopted CQL for offline learning and made simple change the alignment step which, in (13), clips log Te, to O when it is negative, In comparison, we also tested BR by using SAC+ML as the offline learner. Figure shows the results of ACA/BR initialized from SAC+ML/CQL. While the performance of our approach does not change much when initializing from different offline models, BR shows significant performance drops when it is initialized from SAC+MI, i.e. non-pessimistic offline training.
HbifCheatah-mediimm-y
modium-22
Wit modium
+200
XR3
2000
1334
1.0
200
30
300
100
Figure 3: ACA and BR initialized from different offline methods. ACA could achieve similar per- formance while initializing from both SAC+ML/CQL. BR requires CQL initialization.
6.4 ONLINE TRAINING WITHOUT OFFLINE DATA
When the application precludes the accessibility of offline data during online fine-tuning, we re-ran the benchmarks for medium-rent meplay,-eppaa, medium-expert, expert. There is obviously no reason to replay offline data at random level, and empirically, we observed that online fine-tuning already performed well on medium when no offline data was replayed.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Scores for SAC-AACA at 100k online steps, and its increase from offline result (in paren- thesis). Comparison is made between with or without offline data. =C HalfCheetah, H Hop- per, W Walker2d.
Figure 6 in Appendix F shows the online aver- age return of our method without using offline data, compared with other baselines which also do not access offline data during online fine- tuning. The balance replay algorithm requires offline data. So compared with Figure 2. we no longer have the purple line that corresponds to CQL-BR. It turns out that all the other baselines retain similar online performance as in Figure 2, which had been shown inferior to our SAC-ACAA Table further highlights that SAC-AACA does not exhibit significant change in online performance in the absence of offline data.
Scorc(8)
Dutaset
Env
offline data
w/o offline data
HC
59.03(16,50)
59.48( 59.48(16.95)
Med.-Replay
85,54(36.72)
77.19(28.37)
85.17(22.98)
84.27(22.08)
HC
93.74(0.21)
93.81(0.28)
Med.-Expert
H
98 94)
105,67 59)
W
110.54(2.22)
100.93(2.81)
HC
93.14(-0.46)
90.76(-2.83)
Expert
H
110.21(-0.67)
109 109.22(-1 (2((1.66) 1.66)
W
109.59(
10.52((2.31)
6.5 ABLATION STUDY
Total
844,98(84.02)
841.84(80.88)
As mentioned in Section 4.3, log Te, in (20) can be considered as a "behaviour cloning" regularization. One may wonder whether this, instead of actor-critic alignment, is the primary contributor to the empirical effectiveness. We therefore con- ducted the following ablation study, which answers this question in the negative.
In contrast to the parameterization of online Q-function in (16), we designed two alternatives. The first directly copies the Q-function from the conclusion of offline learning, and then fine-tunes online. The second adopts the same decomposed parameterization as in (16), but initializes Ra, with the offline learned Qu,, instead of zu.-- As result, log Te + Rs, whose expectation serves in the actor objective, becomes regular offline-trained Q function with a regularizer log Te,, instead of an actor-critic alignment. The update objectives of the two ablations are relegated to Appendix E.
Figure 4 shows that on tasks vulnerable to transfer risk such hopper-medium and walker-medium (second and third subplots), the two ablation alternatives suffer clear performance drop due to the attributed eтror in the offline trained Q-functions. However, some tasks can be less vulnerable. For example, on halfcheetah-meddimn, Figure 2 shows that SAC-SAC (green-+blue) only suffers small amount of drop, although it employs no mechanism combat distribution shift. In such task, the two ablation alternatives remain competitive to no surprise (first subplot).
11500
480
3000
300
2030
DC only
by
0.0
4. 0
30 300 0 200
.52
300
Figure 4: Ablation study: We keep the actor update same as Eq (20), we however change the critic update (details found in Appendix 6.5) to show that the re- re-parameterized Q-function is critical. BC only (ablation 1): R& is seen as the online critic and is initialized by Q,- Critic updates are made regular SAC critic update without our re-parameterization. It therefore with behavior cloning regularized actor update. R, init by Q2 (ablation 2): We keep the ACA framework but initialize R& by Q2 instead of Zy to show the importance of the baseline.
be seen as SAC
CONCLUSION AND FUTURE WORK
We proposed new actor-critic alignment method that allows safe offline-to-online reinforcement learning and achieves strong empirical performance. To combat distribution shift, we designed novel approach that disregards offline learned Q-functions, and reconstructs it based on the learned policy using closed-form that is motivated from the entropy- regulai-ee actor update. Since it does not need an offline critic, online actor-critic fine-tuning is made possible for offline learned decision transformer, as well as other supervised learning methods such as Rvš (Emmons et al., 2022).

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement. Our ACA implementation can be found anonymously Online Sup- plementary, along with the pre-trained offline models. More implementation details regarding offline approaches and baselines are available in Appendix B, and our choice of nyper-parameters can be found in Appendix
REFERENCES
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15. pp. 103-129, 1995.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances Neural Information Processing Systems (NeurIPS), 2021.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS: What is essential for offline RL via supervised learning? In International Conference on Learning Representations (ICLR), 2022.
Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems, 2010.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. In International Conference Machine Learning (ICML), pp. 2021-2030, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning (ICML), pp. 1587-1596.
PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML). 2019.
Carles Gelada and Mare G. Bellemare. Off-policy deep reinforcement learning by bootstrapping the covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.
Tuomas Haamooja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-basee policies. In International Conference Machine Learning (ICML), pp. 1352- 1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning (ICML), pp. 1861-1870. PMLR, 2018.
Hado Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems (NeurIPS), 2010.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Amold, John Agapiou, Joel Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In Proceedings of the AAAI Con- ference on Artificial Intelligence (AAAT), 2018.
Jiawei Huang and Nan Jiang. From importance sampling to doubly robust policy gradient. In International Conference Machine Learning (ICMI.. 2020.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(51):1563-1600 1600, 2010. URL http:// jm r. ww.g/r//eee http//aper/////ee paperss/ill / //aa/////hhhl/ Ca html.
10

---

Under review as a conference paper at ICLR 2023
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems (NeurIPSS, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic gradient descent. In ICLR: International Conference on Learning Representations, pp. 1-15, 2015.
lyy Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning (ICML), 2021.
Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q- learning. International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q- learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforce- ment learning via distribution correction. In Advances Neural Information Processing Systems (NeurIPS), 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (ICML), 2019.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari. Dmitry Kalashnikov, and Sergey Levine. AW-opt: Learning robotic skills with imitation and reinforcement at scale. In 5th Annual Conference on Robot Learning, 2021.
Yihuan Mao, Chao Wang, Bin Wang, and Chongjie Zhang. MOORe: Model-based offline-to-online reinforcement learning. arXiv preprint arXiv:2201.10070 2022.
Rémi Munos. Error bounds for approximate value iteration. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2005.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint 9rXhv:2006.09359, 2020.
Brendan O' D'Donoguue Ian Osband, Remi Munos, and Vlad Mnih. The uncertainty Bellman equation and exploration. In International Conference on Machine Learning (ICML), 2018.
Online Supplementary. Supplementary material including code. https: http://wwwwrr. http://wwwwoobbo.oooo otgp://cu.eeuuu//////.ooooooooor
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems (NeurIPS), 2016.

---

Under review as a conference paper at ICLR 2023
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable ooflpc reinforcement learning. arXiv preprint arxiv:1910.00117,, 2019.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 19Xi:: 1088, 2019.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. International Conference on Learning Representations (ICLR), 2020.
Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv:1911.11386 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference Learning Representations (ICLR), 2020.
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. In Offline Reinforcement Learn- ing Workshop Neural Information Processing Systems, 2021.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference on Machine Learning (ICML), 2022. https: /tte:://proooo http///pwoceee....e//////oss.ssp mlr. press/ v1 221l2/heen22// 2tt.:/ee////////ng22....om zheng22c.pdf.
12

---

Under review as a conference paper at ICLR 2023
A ALGORITHM DETAILS
The podudo-code of the offline, alignment, and online phases is provided in Algorithm 1, 2, and 3, respectively.
Table 5: Learning in three phases for oflline-to-onlin RL with actor-critic alignment
Learnable Component
Phase
Description
Offline training
(ulals):u(ss.0
Run offline RL obtain offline policy TT(aas))
and Q-function (x a)
Actor-critic
(s(s)
Fit the baseline Z by minimizing the Bell- man residue.
alignment
Online
Run actor-critic policy ( and critic where
training
Qø(s, a) = log Te, (a|s) + Rs(,,a)
( a) is initialized by (s).
Algorithm Offline SAC+ML
Algorithm 2 Actor-critic Alignment
Initialize parameters 0, a, vi, Ps. P. for i E {1, 2} for each iteration do
Require: 0, a, 1hs, P.. for i E {1,2]
sample mlini-batch from dataset D
Initialize parameters di, His for E (1,2) Set Ray (s, a) <- Zo, (8), for i E [1,2]
update a with Eq. (11)
update μ4 with Eq. (9) for i E {1,2)
Copy di + di
update 9 with Eq. (7)
Copy 0o
update 3 with Eq. (13) for i {1,2) [ 4- T/4s + (1 T) for E [1,2]
Reset a +- 1
Delete μr and jú, for E {1,2)
end for
A.1 OPTIMIZATION OF Zv:
Since the alignment objective (13) needs access offline data, blended into the offline training as shown in the second last step of Algorithm nience of implementation, and the dh; values do have any influence on SAC+ML training itself. Conversely, the optimized value of vi provides a good initialization for a standalone optimization of objective (13). practice, we observed that the vi found from offline training good enough, and we just directly used them to initialize the online critic R
noteworthy that this is only for the conve-
A.2 TECHNIQUES TO STABILIZE ONLINE LEARNING
We propose 3-clipping trick and critic interpolation to achieve better empirical performance. As log To, is unbounded below, the training can be numerically unstable, 8-clipping trick bounds the term to stabilize training. And critic interpolation gives the flexibility balance between safety transfer and policy improvement.
A.2.1 3-CLIPPING TRICK
As in the course of online learning, the magnitude of log na, can sometimes be an/several order larger than Rg, which leads very instable critic training- Given Te, (als) is parameterized by squashed Gaussian distribution N(u,,
clip the log Te, term, as follows
(log Te, (als)) SoftPlus log To (als) Cg(s)) + Co(s)
(21)
where Cs(s) min flog To, (U,, Bozls), log To, (Mo 3a,ls))
(22)
13

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Online training
Require:
0. a, di o; from Algo. 2 if D is accessible then Initialize replay buffer B with top trajectories else
Initialize replay buffer +- Ø end if
for each iteration do
sample mini-batch from buffer B update a with Eq. (11)
update de with Eq. (30) for i {1,2)
update 9 with Eq. (29)
Φ 4 TØ; + (1 T)
end for
Here, 3 is a hyper-parameter, and SoftPlus(z) = log(1 + exp(x)). Essentially it clips log Te (als) Cs(s). This CLIPg(-) operator bounds the log Te, term in a reasonable range, and also requires minimal tuning of byper-parameter see Section G and Section H for details. Using CLIP we define 0
(3. a) CUPp((ooT + Ro(s,a).
(23)
Now, the clipped online actor/critic updates can be summarized by
(0, d) Egnd ao (a(as)) Q2(
(24)
-/ d) -E (0 (s, v(r, 27
(25)
v(r,s', d)) = + + ((l-dd)Bs d)
[ - a log (o())))
(26)
A.2.2 CRITIC INTERPOLATION
At the initial phase of online training. C Te, Te, (a|s)) dominates the actor update, safeguarding the policy. As training proceeds, Rs grows to overcome the barrier and starts to improve the policy. Ideally, we wish to finely control such a junetion so that the safety of 020 transition does not excessively slow down the policy improvement. To this end, we introduce an interpolation between closed-form initialized critic and restriction-free critic. We call it critic interpolation, which can be written
3 a) := ( Te, (als)) + Rs) + (1 k)
(27)
closed-from initialized critic
restriction-free
= k x CLIPA log Tap (a|s)) + Rs
(28)
We set h: 1 at f. course of online training, allowing a transition from closed-form initialization to free SAC update. The detailed decaying rate can be found in Appendix
0 to assert closed-form initialization. Then linearly decay 1 during the
A.3 CONCLUDED ONLINE TRAINING
Our final online update rules are summarized as follows:
Conline
(0,d) = Eg En T
aa (elogmo(als) To(als) - 2 25.0 (s.
(29)
(0 (s))(())))) - u(r, d)) 27
c Contine (Di d) = E.s.a.n, ,ayd
(30)
8(,,8,d) = r 7(1 d) E)
(t. (s', a') - a log To (a'ls'))
(31)
14

---

Under review as a conference paper at ICLR 2023
B IMPLEMENTATIONS
Overall, our implementations are from or based on d3rlpy (Takuma Seno, 2021), a popular RI. library that specialized for offline RI.. Using the same lib helps us to minimize the impact of imple- mentation difference. Many of our baselines (see Table 9) are implemented upon SAC, with changes proposed in their original papers, respectively.
B. GENERAL IMPLEMENTATION DETAILS
Evaluation protocol: All offline/online experiments 5 random seeds. We all offline algo- rithms for 500 episodes with 1000 mini-batches each, and all online experiments for 100 episodes with 1000 environment interactions each. After each episode, we conducted 10 evaluations and computed the average return. Results reported are mean and std of average returns, over 5 random seeds.
Choice of offline checkpoints: Evaluating in the offline phase, in fact, requires online interactions. Therefore we do not pick the perf-rerf checkpoints. Instead, we use the last checkpoints as our initialization models, for online.
Squashed Gaussian:
For methods with stochastic policies, we parameterized their policies by uni- modal Gaussian, and applied the squashed Gaussian trick (Haarnoja et al., 2018) to bound the range of action to [].
Buffer initialization: We followed the instructions in AWAC and BR papers on initializing online replay buffers. For AWAC, we added all transitions in to the buffer B. And for BR, we refer to their original implementation this URL for details. For SAC-SAC and CQL-sSC,, all transitions in D to the buffer as well, as there is no explicit instructions or common protocols. All replay buffer sizes were be 1e6, unless specified in the Appendix I.
added
B.2 OFFLINE
AWAC and CQL: We used d3rlpy implementations for AWAC and CQL.
SAC+ML: Our SAC+MI. implementation was adapted from d3rlpy's TD3+BC implementation, with changing the actor update rule to Eq. (7), and adding the learning of baseline Zg.
B.3 ONLINE
Training details for online: For all methods, we made temperature (if applicable), a critic, and an actor update after every environmental interaction, if there were enough transitions (i.e. more than batch size) in the replay buffer. Target networks were all updated in a Polyak averaging fashion, where the step size 0.005 for all experiments. See Section for more typer-parameter details. And online results, reported in tables, were also using the last checkpoints instead of best- ones.
SAC: We used d3ripy implementation for SAC.
SAC-SAC and CQL-SAC: We simply loaded filline-trained SAC+ML and CQL, respectively, and then ran SAC online.
BR: We adapted all parts that related the prioritized replay from the official BR implementation, to d3rlpy SAC implementation base, as the original BR paper also run SAC online.
ACA (ours:) Implementation of our approach be found anonymously at Online Supplementary. In addition to Algorithm 3, we also did gradient norm clipping to actor updates, which is commonly used in RI. implementations.
15

---

Under review as a conference paper at ICLR 2023
C SAC+ML vs. TD3+BC
We would like to emphasize that our goal is not to propose a stronger offline RL. method. Table is presented show that our SAC+ML modification performs comparably to the original SOTA method, TD3+BC.
TD3+BC results in Table 6 were copied from appendix C.3 of their paper (Fujimoto & Gu, 2021). The evaluation protocol is identical to theirs: (1) all experiments were done in D4RL-v2 datasets; (2) and the results reported were from the last evaluation step, averaged over random seeds.
Table 6: SAC+ML TD3+BC
Dataset
Environment
TD3+BC
SAC+ML
HalfChectah
11.0
17.3+2.
Random
Hopper
8.510.6
7.920.3
Walker2d
1.641.7
3.422.1
HalfCheetah
48.310.3
46.310.2
Medium
Hopper
59.314.2
54.31334
Walker2d
83.7+2.1
81.2+1.6
HalfCheetah
44.6.1.55
42.5
Medium-Reppay
Hopper
60.9=18.8
48.8
Walker2d
81.825.5
62.2+4.9
HalfCheetah
90.7+4..3
93.5+-4.0
Medium-Expers
Hopper
98.0+9.4
93.1+
Walker2d
110.1
108.111.6
HalfCheetah
96.7+1.1
93.6+b.8
Expert
Hopper
107.8:7
110.9
Walker2d
110.210.3
108.210.3
Total
1013.2
971.3
D ILLUSTRATION DISTRIBUTION SHIFT
Figure shows the histogram of the & norm of state vectors from the halfcheetah- maciettaaammmiuu--222 and halfcheetah-exwe -eprtaa---xeeet-222 datasets. Clearly, there a distribution shift. So we can obtain in on-iistribution sample and out-of-distribttio samples (with respect to the medium dataset) by (I) sample tran- sition from medium dataset and (2) sample from 98% to 100% percentile of the expert dataset (in terms of the & norm of the state vector), so that it is out-of- distibbbbtttion for medium agent.
40000
medium
35000
expert
30000
25000
20000
15000
10000
5000
20
40
60
80
100
120
Figure 5: Histogram of l norm of state vectors in halfcheetah medium and expert datasets.
16

---

Under review as a conference paper at ICLR 2023
Offinn: Online:
SAC NL
CQL
ANAC
CQL-SAC
Hopper
12300
2000
4003
300
402
20
HalT wwl.hhhiimmmmeerreeee
Hoppas
12000
033
HottChhwtth medium motoy
Happ per modly
Wallowdd modium rndby
11300
5000
0001
2000
2000
1008
X00
HorCCheetah medium e2
Hopper medium expertv2
Vallce mediu expert
6000
7003
300
3000
Hog 2r
Valle Cwwherro-e
1004
11800
300
2000
Figure When offline data are not accessible, vs. other baselines
E ABLATION STUDY OF ACTOR-CRITLL ALIGNMENT
Here we write out the detailed formula of the critic objective the two ablation studies in Section 6.5.
Cablation
(di, d) (s.ar,ssddd)) (R)(s, (1ls,a)-v(r.. -((
(32)
u(r,s',)) = = - d)) Rooss' a') a log To Ta(''ss))
(33)
(di, d) = E.aaaaiss .0nd ardd (0 (s, -irr,', - 21
LQ
cablation2
(34)
2(r,6'd) = + -11 dd)E d) ( (s,
(s',a') a') log mo(a'ls')
(35)
F MORE RESULTS ON ONLINE TRAINING WITHOUT OFFLINE DATA
The distributional shift issue would clearly be severer when offline data not accessible during the online phase. To be more conservative, we therefore set Bw/o 1.53w./ for experiments without offline data, excluding random and medium levels both used no offline data for our main results already. (Bw./ denotes the syper-parameter we used for our main results, see Table 8 for details.) All other Hyper-parametens remained unchanged.
G SENSITIVITY ON 8
Figure shows that the performance of SAC->ACA is not very sensitive the choice of 3.
17

---

Under review as a conference paper at ICLR 2023
HalfCh moduume
Walkor
300
0002
8.0
2000
1000
2.555
290 40
130
200
123
220
100
Figure 7: Results for different /
Besides, one could choose 3 without any online iteration, rather than tuning by grid search, which is impractical in offline RL 020 RL. One could compare the clipping threshold [Cp(s) to the magnitude of (s)]. where (s(s) as defined in Eq. (22).
is reasonable if these values comparable, or Cs(s) is slightly larger than than 1 (s)l. So the clipped log To, term creates a barrier for the online critic to overcome, which in turn makes the transfer safe. This allows one avoid running online evaluations tune 3, as the evaluation of (s) and Cs(s)) can be done completely offline. We show the comparison between Zyp (s) and [s](s] in Figure 8.
18

---

Under review as a conference paper at ICLR 2023
H MAGNITUDE COMPARISON BETWEEN [ AND |Cal
He Mwnehaaaamoorro
HolfChootah modayy-
Hopper vzden
Tolleerd-mediee roplay 12
.00
142
30
HolChhotbh medum esport v2
Happer medium expar.22
Wellerze medium expert
.00
300
300
300
40
30
120
HetChhetah exparti22
Tbpper
We Weler2o-oparsdd
300
600
00
400
550
500
400
Figure 8: x-axis stands for different random seeds (corresponding to different pretrained SAC+ML models). For each random seed, randomly sampled episode {(s,, $1+1, r;) 1,2,..., T from corresponding offline dataset. We then computed (ICa(s:)) and (z to make box plots, where outliers omitted for better visualization. We used the As in Table 8 to make this plot.
Figure 8 shows that (Cs(s)) and [s)l have comparable values. The empirical performance al- ready outperforms all baselines even though we did not extensively match their magnitude for every task. It in turn implies that tuning 3 requires minimal effort, in additional the advantage that can be chosen completely via offline comparison.
19

---

Under review as a conference paper at ICLR 2023
I HYPER-PARAMETERS
Table 7: Specific parametrreeeters for different baselines. Please refer to the original paper for the meaning of hyper-parameter names.
Algo.
Syyper-paaane name
Value
SAC+ML
30
AWAC
1.0
conservative weight # of actions sampled
10
CQL
10
offline buffer size
2.5e6
online buffer size
2.5e5
BR
density ratio estimation network arch. density ratio estimation network temp
[]S] [A], 256, 256, 1]
0.75
ACA
π grad norm clip
0.25
Table Hyper-params used for our main results reported in section 6.1. 1/4 z represents that decays from x to using y episodes.
hyper-param
HalfCheetah
Hopper
Walker2d
N (# of init trajs)
0
Random
8 (3-clipping)
7
(interpolation)
10
N
0
Medium
3
7
20
k
0.5
N
50
Medium-replay
3
7
k
> 0.5
N
50
Medium-expert
3
15
1 N/A
k
N
50
Expert
3
15
N/A
k
20

---

Under review as a conference paper at ICLR 2023
Table 9: General syper-parametern ACA and BR stand for SAC->ACA and CQL-BR, respec- tively, for the sake of space.
CQL
SAC+ML
ACA
SAC-sSAC
BR
CQL-sSAC
SAC (scratchh))
AWAC (off)
AWAC (oa)
Phase
offline
online
offline
oaline
Based on SAC?
Yes
No
General hyper params
Arch.
[ Al, 256. 256,
Q Arch.
[
A 256, 256, I]
[[5]. 256, 1]
# Q nets
# 2 nets
N/
(Polyak avg.)
0.005
Activation
ReLU Adami for all
Optimizer
Adam params
hetas (0.9, 0.999), eps = 1e-8, weight decay = 0
Ie-4
Q
3c-4
30-4
3e4
le-4
3x-4
3e-4
N/A
Z
3e.4
3e-4
N/A for online
# episodes
500
100
it/ep
# batch/ic
Batch size
Hyper-params for base SAC impl.
Entropy target R
[4
N/A N/A N/A
Uni model Gaussian
Squashed Gaussian
Yes
21

---

Under review as a conference paper at ICLR 2023
J IMPLICIT Q-LEARNING
We run IQL with the same experimental setting as Figure 2
waler2d.andoo-.22
140
300
309
100
300
200
4000
1000
1000
300
sullchectal-mellue wz
welkerztimmelurr
.000
2001
28
100
-00
300
.00
12200
1000
5000
303
UM
250
200
120
200
300
200
12:00
2502
20
400
50
100
-00
30
300
300
50
Figure 9: IQL
We observed that IQL is struggling with online improvement. This is in fact also observed by ODT (Zheng et al., 2022). See Table 10.
Table 10: IQL
Reported ODT [QL(200K)
reproduction (QL(100K)
Dataset
Eac
IQL
hgg.
[QL((ffiine)
fig.
SAC+MI.
ACA(100K)
SACA
BC
Random
11
8.81
8.81
0.00
7.93
81.85
73.92
W
5.46
5.44
-002
3.36
12.42
9.06
HC
47.37
47.41
0.04
46.61
46.76
0.15
46.33
66.58
20.25
Medium
1I
63.81
66.79
2.98
55.62
52.48
.3.14
54.31
96.54
42.24
W
79.89
80.33
0.44
78.08
80.91
81.16
74.66
-6.50
HC
44.10
44.14
0.04
44.40
44.85
0.45
42.53
$9.03
16.50
Modl-Replay
H
92.13
73.07
73.08
48.82
85,
36.72
W
73.67
70.55
-3.12
74.64
73.03
-1.61
62.19
85.17
22.98
Med. Expert
11
44.61
44.42
-0.19
93.08
98.02
4,94
W
HC
94.07
94.10
0.03
93.59
93.14
-0.46
Expert
W
108.98
109.10
0.13
108.21
109.59
1.38
Total (mmed & med-roplay)
400.97
405.45
4.48
372.42
371.11
-1.30
335.35
467.53
132.19
Total (all)
928.06
923.35
.471
971.34
1249.65
278.31
22

---

Under review as a conference paper at ICLR 2023
K
DEMONSTRATION OF MOTIVATION
How we design the experiment?
We load pre-trained models from medium/expert level
Disable critic update, target update, data collecting, etc. (In other words, only keep actor and entropy update.)
The actor/entropy update are the same SAC actor/entropy update.
Run update on random dataset simulate OOD data
We run total 10k steps and evaluate its performance every 100 steps.
test how different offlinealligned Q-functions affects the actor update.
Aligned CQL COL
Aligned SAC+ML SAC+ ML
Hopper-meidum-Vg medium-v2
Hopper-
3000
2000
1500
mww V M
1000
Hopper-expert-vv
Hopper-expert-v2
offiine score
3000
3000
COL (aligned)
2300
W
2000
offline score
SAC+ML
SAC+ML (aligned)
1000
1000
10k
4k
6k
8k
10k
(a) CQL (aligned) vs. CQL
(b) SAC+ML (aligned) vs. SAC+ML
Figure 10: Our aligmment (applied on both CQL and SAC+ML) could attain the offline performance better than the original performance.
Takeaway 1: our reparameterization is able to attain its offline performance better than other base- lines.
Takeaway 2: our reparameterization applies to different offline critics, as we also mentioned in Section 6.3.
23

---

Under review as a conference paper at ICLR 2023
As SAC+ML collapse really quick, opt it out for further comparison. We provide compar- ison of CQL vs. aligned CQL.
Aligned COL COL
Heppet-medium-
4003
3000
mm My M
5200
WW
5003
4003
1000
4000
100
HallCheetah expert.v2
Hopper expert-v2
Walker20 expert-v2
4003
3000
1003
2000
1002
Figure 11: Aligned CQL vs. CQL.
Table 11: Expert level policies are more fragile.
CQL relative change (S)
CQL (aligned) relative change (%)
Dataset
Eny
@ 1k
@2k
@3k
@4k
@.5k
@ lk
62k
@3k
64k
6:5k
HaltCheetahh
-0.54
-3,53
4,55
-6.71
-7,58
0.30
0.71
0.75
0.98
0.17
Medium
Hopper
19.62
-2,999
-2.27
-3.227
64.43
-3.56
-2.68
-11.11
99.52
-66.77
Walker2d
-5.2
-11.74
-25.66
-30.89
-26.73
-0.12
-0.09
-0.40
-0.70
0,01
Avg. (medium)
8..6
-12.76
-17.53
-24.29
-32.91
-1.13
-0.69
-359
-3.08
-5.33
HalfCheetah
100250
-10.322
-105,22
-105,48
-104.96
-68.42
-73.67
-68.21
-69.02
-62.44
Expert
Hopper
-4.96
-51.07
-77.36
-80.78
-82.59
-12.83
-5.23
-9.44
-3.31
-19,33
Walker2d
-32.15
-16.76
-29.06
-36.70
-34.42
-0.67
-0.54
-0.22
-0.26
40.42
Avg. (expert)
-59.87
-56.72
-70.55
-74.32
-73.99
-27.31
-26.48
-25.96
-24.20
-27.39
Avg. (expert excluding HC)
-38.55
-33.92
-53.21
-58.74
-58.51
-6.75
-2.89
-4.83
-1.79
-9.87
Additional observation: near optimal policies more fragile (even if we exclude HalfCheetah- expert), which highlights our advantage in expert-level datasets.
24

---

Under review as a conference paper at ICLR 2023
L AWR OBJECTIVES
Akin to experiments we conducted in Section K, we run actor update only experiments with AWAC (also categorized into AWR) actor objective instead of SAC actor objective. Critics trained offline by AWAC.
Hopper-medium-22
Hopper-mediumw2
2500
2000
2000
1500
1500
1000
1000
500
500
Happer-experte
exaper-xxper-v22
offline score
offline score
3000
2500
1500
2000
1000
1000
2k
8k
10k
20
60
80
100
(a) AWAC 10k updates
(b) AWAC 100 updates
Figure 12: (a) shows that AWAC also collapse to nearly zero performance. (b) shows that around 20 actor updates are already enough to destroy the performance.
25

---

Under review as a conference paper at ICLR 2023
M AVERAGED VERSION OF FIGURE
Figures 14 and 15 provide averaged versions of Figure for in-distribution and out-of-disrribution samples, respectively. The approach used to generate these two figures is illustrated in Figure 13.
adtin d
ad3in din
action dim
-000
Off petzabee
-004
-000
(a) Perturb action
(b) Centralization
(c) Multiple samples
(d) Aggregation
Figure 13: Demonstration of how Figure 14 and Figure 15 are created. (a) Given a sample (s, a), we
perturb along dimension plot (ss, â) and compare ((s, â) to Q(s, (s)); (b) We plot Q(s, with its deviation from (s)), so that (ss, (s)) is centered at = and (s) is centered 0: (c) Such centralization allows us to place multiple samples (different s) the same plot, where points above the z-axis correspond to "over-estimated" perturbations; (d) We aggregate multiple samples by counting how many points are above O. This way, the height of the red part in the bar plot quantifies the fraction of points that are "over-estimated" For an "over-estimated" point (s, a), its z-coordinate stands for the distance between a and the policy favored action (s).
Note: By "over-estimation" we mean for some a * T()) such that Q(s, a) Q(s; T())), which in certain degree means that the critic Q is "disagreeing" with the policy
dim
action
action
300
allgned
-300
6.9%
5.71%
3.10%
4.55%
4.55%
6.24%
-1000
5acmm
31.35%
35.34%
38,84%
36,54%
38.72%
99.98%
330
8
-100
0.207%
8.95%
8.71%
8.98%%
-1.00
GWBC
21.57%
22.5
20.09%
28.1.%
30.29%
-1000
Figure 14: Quantifying fraction of over-estimated perturbations for in-distribution samples.
26

---

Under review as a conference paper at ICLR 2023
action
200
aiggeed
-103
4.7%%%
5.11%
4.00%
0.2
-500
90.900%
44.01%
30.68%
-1000
300
63)
-100
11.59%
24. 75%
6.91%
16.51%
13.86%
17.52%
DNNC
28.90%
37. 76%
45.67%
29.41%
33.62%
37.08%
-1000
Figure 15: Quantifying fraction of over-estimated perturbations for out-of-distribution samples.
All agents are trained on malfcheetah-mediune v2 dataset. By in-distribution samples, we refer states drawn from the malfcheetah-mediin-- dataset. By out-of-distribution samples, use samples from the halfcheetah-experttV dataset. We randomly drew 200 samples per seed, which results in total of 1000 samples to make each plot.
Details:
27

---

Under review as a conference paper at ICLR 2023
N CONVERGENCE OF SOME TASKS
We run another IM steps on top of the original 500k steps, for WalfCheetah-MMeding Meiiim---dd-errr,, HallCheetab-Expert,, Hopper-Expert, with experimental setting and hyperparams that identical to Figure 2.
However we found that MerChheetah-Meiiu--EEEe still have high variance. CQL suffer from relatively high variance is also observed in TD3 (Fujimoto & Gu, 2021).
taiichettah-meduumeexpttttt
12000
10000
8000
6000
2000
0
50
HalfCheetah eeppetttt---
Hxpper-expert--2
14000
4000
12000
10000
3000
6000
2000
4000
1000
2000
COL->SAC
0
COL->BR
Figure 16: Run CQL until convergence.
We still observe unstable performance in Hopper-expert. Due to time limit, not able to HalfCheetahh-mediun medium-expert expeet further longer, but should not affect the results Table 12 as the online performance of Ceehhhhhtaammdium-eeeeeee
converged (roughly) to max score.
Table 12: New results with additional 1M CQL offline steps
CQL--SAC (prev.)
CQL--SAC (new)
CQL-BBR (prev.)
CQL--BR (new)
HC-ME
87.85(46.77)
93.94/28.755
91.80(50.73)
96.35(31.17)
93.74(0.21)
HC-E
55.39(6.86)
94.88(-0.98)
79.69(31.16
97.67(1.82)
93. 93.14(-0.46)
H-E
67.88(-15.48)
94.56(-13.95)
68.55(-14.81)
79.32(-29.19)
110.21(-0.67)
Total (three above)
211.12(38.15)
283.37(13.83)
240.04167.08)
273.34((.79)
297.09(-0.92)
Total (all)
1016.08(79.35)
1088.33(55.03)
1190.96/254.33)
1224.26(190.94)
1249.65(278.1))
As originally claim match BR's performance, and highlighted both BR and ours the strongest ones in Table 2. We do not see a nessceray change of this claim. The new results is in fact favoring us in terms of performance improvement. Our ultimate advantage could attain similar performance while offline data is not accessible while BR is not even applicable.
BR is that
28

---

Under review as a conference paper at ICLR 2023
O INITIALIZATION FROM BEST CHKPTS
We only saved checkpoints every 100 episodes. Therefore, we pick the best checkpoints among them. Sub-figures that are visible means the last checkpoint is the best checkpoint saved. (We determine the "best" checkpoint by the averaged performance over all seeds.)
We do not observe significant differences between initialization from best checkpoints and last ones.
ACA init from best checkpoint
rappee-randem--2
1000
1500
1002
1003
Ampger-medummer2
3000
4000
2000
1002
replay
2000
1003
Hopper expent-22
expert-v2
4002
20000
3000
7500
2000
5000
2500
Figure 17: ACA initialized from best checkpoint, if last is not the best.
chackpoint
Iulltheelahhenddmm-p
[ksppe-randdm-122
Wellber 2ww.aaddm--2222
1503
1002
Hopper
4000
5000
3020
4300
2000
3000
2000
1000
1002
Wolleer mendedi www.dddd---mmmeeeeeeeee www.paaallll
6000
DR inctt from lact BR inst from best offtine score (astt offiine score (bes)
20
Figure 18: BR initialized from best checkpoint, if last is not the best.
29

---

Under review as a conference paper at ICLR 2023
P DOES OVER-FITTING AFFECT TRANSFER?
In addition to Section O, we initialize from checkpoints at 100k, 200k, 300k, 400k and 500k re- spectively for walker2d-medium--2. whether initialization from different checkpoints make difference.
Walkeer2d-meiium-22
5000
3001
2009
COLIDCRI-SSC
COX (200k)--SAC
1009
-OU-BOOK)--AA
COL/AOKKI-SAC
COUS2CKI-SAC
Figure 19: CQL-SAC initialized from different checkpoints. Dashed lines are corresponding of- fline performance.
We do observe conclusive evidence showing that over-fitting might lead unstable transfer.
Q ASYMPTOTIC PERFORMANCE
HaliCheetth-medium-2
12000
8000
6000
offline score
4000
Figure hyper-params
2000
less conservative hyper-params
D4RL max score
100
200
400
Figure 20: Asymptotic performance
The blue curve uses the same hyper-parameter setting to Figure 2, where k decays from 1.0 to 0.5 in 20k steps. Keeping k 0.5 seemed to slow down the achievement of expert performance. However, one could always safely to decay & to after the transfer is stable. The green curve linearly anneals k 0 using 200k steps and able reach D4RL expert score much faster.
30

---

Under review as a conference paper at ICLR 2023
R RANDOM INITIALIZATION
We initialize Riss a) randomly
telfcheerallmeeeemee
hspper-maiuu--2
We Rearad medium 12
302
2500
000
2200
1200
300
150
200
Figure 21: Randomly initialized R(s,0) a)
is obvious that random initialization has poor performance as it violates our motivation.
31


Reviews:
[{"review_id": "UIVgreHoKZ", "reviewer": "Reviewer_LK37", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel alignment step for actor-critic RL approaches from offline to online settings. The paper relies on the observations that the Q functions OOD are much more untrustworthy than the entropy regularized policies learned from the offline data. They propose to disregard Q-values learned offline to combat overestimation under distribution shift and rather propose a procedure to rebuild these Q functions from the policy and a state-only value estimate during the online fine-tuning phase. They demonstrate little to no drop in performance when offline to online transfer happens on a range of controlled simulated tasks. ", "summary_of_the_review": "While the results are not significantly better than a strong competitive baseline CQL + BR, the ideas of aligning Q values from offline to online phase and reconstructing Q values for online finetuning are fairly interesting. The experiments have been run on limited  environments and not in large domains where the performance shift arising out of the distribution shift can be very evident. The paper has done a fair job of running ablations and demonstrating which components of the alignment step are important for the transfer. While there are certain clarifications highlighted earlier, in the interest of the idea, I am recommending to marginally accept the paper and hope that the authors do address the comments in subsequent iterations.", "strength_and_weaknesses": "Strengths\n\n* The approach to disregard learned Q values during the alignment phase and aligning them into the warmstart policy and state-only function is an interesting idea for transfer from offline to offline settings.\n* The paper does some good ablations (eg Figure 1 and Figure 4) where they first justify the alignment procedure and later verify that BC regularization alone does not guarantee offline -> online stability, both of which are very valuable observations.\n* The approach does not require any of the offline data during online phase, which can be suitable for applications that are covered by data privacy considerations such as healthcare.\n* The results are strong and consistent over the environments taken into consideration while all other baselines fail to transfer well in some cases or the other.\n\n\nAreas to improve/ Clarifications and Questions\n\n* Should we also consider offline RL approaches such as the model-based ones which learn the dynamics structure before the fine-tuning phase for a fair comparison to how they respond to OOD actions? Running full-scale experiments might be a costly exercise, but I still feel additional experiments or some discussion around this aspect can advise which approaches to embrace for offline-online RL.\n* Transferring Q values to policies might sometimes swing the pendulum to the other side from over-optimistic estimates to pessimistic estimates in the policy, which may prevent exploration in the online fine-tuning case. Prior works (eg [2]) have alluded to this phenomenon. It might be worth investigating whether such a phenomenon affects the model from learning more optimal policies in certain environments where it underperforms other baselines.\n* (please correct me if i misunderstand) The alignment procedure works only when the Q function estimates the values of the behavior policy, but I am not sure whether this transfer is possible for methods that do perform multi-step DP on the value functions such as Implicit Q Learning [1]. It might be worth having a discussion about this specific aspect in the paper and see how this transfer might work for those value functions.  \n* The paper uses the last checkpoint during offline training as the initialization for online training. This might not always be the best offline trained model as it does not account for overfitting and thus, exacerbating the offline -> online OOD effect even more. Since this is a study in controlled environments, policy selection could still be done by evaluating in the environment and picking the best possible policy to warmstart. This can expose whether overfitting might be a strong reason for the models to suffer in terms of generalizing to the online environment.  \n* Minor : The paper could have benefitted from demonstrating transfer from offline to online in more complicated real-life environments plagued by distribution shift, such as robotics where sample collection is costly. This would have made the impact of results even stronger.\n* Minor:  For baselines in transfer such as SAC -> SAC, while distribution shift is a problem, an interesting ablation would be to constrain the policies from diverging too much during the online training at each training step such as adding KL penalties for policy divergence, for instance. This would further explain whether we need this alignment step as compared to controlling policy divergence in the online phase from the bootstrapped policy.\n\nMinor comments\n\nFigure 2 might be useful to point users to table 1 to understand the legends better.\n\nPage 2 - Our approach does not “reply” on should have rather been rely on\n\nExposition in figure 1 is not clear. Kindly amend the description to better demonstrate how the alignment is benefitting over the baseline.\n\nFigure 4 can be made more self-contained as ablation 1 and ablation 2 are not really clear anyone trying to parse the figures.\n\nReferences\n​​1. Implicit Q Learning – https://arxiv.org/pdf/2110.06169.pdf\n\n2. https://arxiv.org/abs/2002.12174\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rw9EPp3A7i6", "reviewer": "Reviewer_Xgqq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new offline-to-online RL method that proposes to use SAC as the backbone RL algorithm and directly aligns the policy with the critic function at the end of the offline training phase and use the newly aligned critic as an initialization of the critic function during online fine-tuning. The authors show that the proposed approach can achieve reasonable performance on D4RL-v2 tasks.", "summary_of_the_review": "Based on the comments above, I would vote for a weak reject.", "strength_and_weaknesses": "Strength:\n\n1. The paper tackles offline-to-online finetuning, which is an important problem to study in RL.\n\n2. The empirical results suggest that proposed method is able prevent dip at the start of the fine-tuning, which is promising.\n\nWeaknesses:\n\n1. I'm confused by the fact that the authors need to do explicit alignment at the end of offline training as the added BC regularization should be able to handle distributional shift of Q-values on out-of-distribution actions. It is unclearly why the authors need to do this additional step. It appears to me that this step is rather ad-hoc.\n\n2. The authors also didn't compare to more recent offline-to-online methods such as IQL. It is important to add such a comparison to show the benefits of the method. It also appears that the improvement over other approaches is rather marginal, which is not conclusive enough to show the benefits of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LLAD-DAG6b", "reviewer": "Reviewer_mP3E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new approach for improving the transition between offline and online training of policies via RL. In actor-critic algorithms, the source of errors in the online phase comes from extrapolation errors in the pretrained Q-function. In the online phase, when the pretrained policy gets into states that are not in the offline dataset, the Q-function can overestimate the values of poor actions and then the policy optimization in turn deteriorates the policy. The proposal in this paper is to fix this by throwing away the trained Q-function and effectively creating a new one that is \"aligned\" with the policy in the sense that the Q-values are fit to the log-policy. This is also somewhat theoretically justified by the common interpretation of policies being Boltzmann distributions with the Q-function as the energy.", "summary_of_the_review": "Overall I think this paper introduces a neat and simple idea for offline-to-online RL. It's an area of great significance for real world RL applications. Unfortunately, the exposition is slightly lacking, particularly empirically. It seems the benefits of the method are really only significant when the dataset has a lot of expert trajectories. Given the alignment nature of the idea, I worry that this is a built-in weakness [1] and I'm not yet convinced by the experimental results.\n\nI was an emergency reviewer so I will drop my confidence score to reflect that.\n\n[1] The authors even state: \"the Q-function is generally problematic for out-of-distribution actions while the policy learned offline is assumed trustworthy.\"", "strength_and_weaknesses": "Strengths:\n- A nice and simple idea.\n- Attacks an important and well motivated.\n- The first third of the paper is quite clearly written.\n\nWeaknesses:\n- Clarity could be improved in parts (e.g. beta-clipping, discounts).\n- Figure 1 could be made more rigorous.\n- Experiments are thorough but the plots are hard to read.\n- Benefit of the approach is not clear empirically.\n  * Improvements are only clear in the expert offline data regime.\n- The additional implementational details in A.2 and Eq. 8 seem like a confounder. Are these used for all baselines as well?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CIW8ubOW-Uw", "reviewer": "Reviewer_XAHs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel offline-to-online RL method.\nThe proposed method introduces an offline critic re-training phase to explicitly align the critic with the offline-trained policy.\nIn the critic re-training phase, a refined critic parameterization deducted from SAC policy objective prevents value overestimation.\nExperimental evaluation shows its comparative fine-tuning sample efficiency compared to SOTA offline-to-online RL methods.", "summary_of_the_review": "The proposed method and empirical support are quite convincing.\nACA can replace the SOTA offline-to-online RL method even without access to offline data.\nHowever, as described in the weaknesses, some important points related to the motivation of the idea are not clearly shown.\nAlso, the quantitative comparison possibly includes unfair comparisons for several tasks.\nThus, I vote for rejecting this paper until those concerns are resolved.", "strength_and_weaknesses": "### Strength\n* The proposed method can achieve comparable online convergence with SOTA baselines without its access to the offline data or importance sampling strategy.\n\n### Weakness\n1. Unclear parts in experiment results.\n    - Some of the CQL offline training curves in Figure 2 seem to be not converged yet. (HC medium-expert, HC expert, and H expert, following the notation in table 2). It could have affected the initial stability and also the final score after a fixed amount of online training, which makes the quantitative comparison in Table 2 unreliable.\n    - In Figure 1 result, SAC+ML is not even aligned with the in-distribution sample which is considered to be from the bias introduced by ML term. (or possibly from not converged Q which is also problematic.) Thus, this qualitative result is showing the misalignment not only by distributional shift but also by biased objective. Consequently, the given result is not enough to illustrate the existence of the misalignment problem of prior offline RL methods. \n\n2. Ambiguous term - misalignment\n    - Despite the \"actor-critic misalignment\" being the key problem that this paper aims to solve, it is neither formally defined nor kindly explained.\n    - Moreover, as described in weakness-1, the qualitative result is not clearly showing whether the described problem really appears in other methods.\n\n3. IQL reports better benchmark results than AWAC even for online fine-tuning, thus more proper baseline for AWR-based offline RL methods. I think this comparison is also important to show whether AWR policy objective is also affected by the actor-critic misalignment. But the current bad results of AWAC seem to be from offline RL performance rather than the actor-critic mismatch.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7thJ2JvCab", "reviewer": "Reviewer_Px1x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to transfer (or align ) a trained method from offline learning to online learning. This method has three steps: (modified) offline learning, actor-critic alignment, and online training. In  modified  offline learning, they use a modified version of TD3-BC where they use soft-actor-critic instead  of TD3 and add behavior cloning term. During the alignment step, they fit the \"baseline\", and finally during online-training, they changed the Q function with their \"baseline\". To show the effectiveness of their method, they use a subset of D4RL benchmark (i.e HalfCheetah-*, Hopper-*, Walker2d-*) and they get interesting results.", "summary_of_the_review": "Offline to online alignment/transfer in RL is an important and interesting area and this paper looks at the right problem. However, the current paper has many issues and I don't think it meets ICLR acceptance bar. In addition, the paper doesn't experiment in which distribution shift really happens, this is important as this paper motivates the issue of offline to online transfer. Finally, it requires significant revision as it is hard to follow. ", "strength_and_weaknesses": "Strength:\n\n- Utilizing a trained model in the offline-rl and fine-tuned during online learning is an interesting direction to consider.\n- Some of the results are promising. \n\n\nWeaknesses:\n\n- Although authors claim their method \"outperforms or matches the current SOTAs\", I disagree with this statement. Take table 2 and Figure 2 as examples. In table 2, CQL + BR has pretty much the same performance as SAC + ACA ( proposed method). Although standard  deviation is not reported in these experiments, it is hard to say if these results are significant. In addition, if we look at Figure 2, green lines are only better in 5 to 6 cases out of 15 cases. Considering the strong performance of [1], why should someone select this method over [1]? What is the advantage of this method vs [1]? discarding Q-function by itself is not an advantage until it shows up in the results.\n\n- This method has lots of moving parts and it would be potentially hard to make if this method applies to new environments.  Consider training settings for each step of this method.\n\n- It is hard to follow the writing and understand the motivation for some choices. For instance, $Z$ is a function of only states but during the final step of this algorithm, it will be used to initialize R(s, a). Since R(s,a) is a function of both s and a, why does this initialization make sense? Also, it is not clear the motivation behind  $Z$. Can you explain?  It seems to me $Z$ behaves like a value function normalized by policy. But it is hard to tell. Also, can you explain how you derive equation 5 and motivation beyond that? Finally what happens if we ignore the first two steps of your method and just initialize R(s,a) randomly? How does performance change? \n\n- In my view, the issue of distribution shift is overlooked in this paper and there are no good cases for that in this work. Examples of distribution shift can be change of state distribution during online training, changing reward, changing transition functions, etc. \n\n- Why were only these environments selected? Are there reasons for not running experiments using Adroit, Maze environment form D4RL benchmarks? \n\nOther comments:\n- Compounding error is usually the problem in model-based RL. Can you expand and discuss how it is related to offline RL?\n\n- Extrapolation error is mainly the source of the issues in offline RL. However, this paper implies that it is a distribution shift. Since not all extrapolations are caused by distribution shift and vice versa. Can you explain how these two are related?\n\n\n[1] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject