Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning
Authors: Yi Zhao
Rinu Boney
Alexander Ilin
Juho Kannala
Joni Pajarinen
Venue: ICLR 2022 Submitted (2022)

Abstract: Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. However, depending on the quality of the offline dataset, such pre-trained agents may have limited performance and would further need to be fine-tuned online by interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. While constraints enforced by offline RL methods such as a behaviour cloning loss prevent this to an extent, these constraints also significantly slow down online fine-tuning by forcing the agent to stay close to the behavior policy. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Moreover, we use a randomized ensemble of Q functions to further increase the sample efficiency of online fine-tuning by performing a large number of learning updates. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.

Under review as a conference paper at ICLR 2022
ADAPTIVE BEHAVIOR CLONING REGULARIZATION FOR STABLE O CEMNTT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline reinforcement learning, by learning from fixed dataset, makes it pos- sible to learn agent behaviors without interacting with the environment. How- ever, depending on the quality the offline dataset, such pre-trained agents may have limited performance and would further need to be fine-tuned online by in- teracting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. While constraints enforced by offline RL. methods such as a behaviour cloning loss prevent this an extent, these constraints also significantly slow down online fine-tuning by forcing the agent to stay close to the behavior pol- icy. We propose to adaptively weigh the behavior cloning loss during online fine- tuning based on the agent's performance and training stability. Moreover, we use a randomized ensemble Q functions further increase the sample efficiency of online fine-tuning by performing large number of learning updates. Experiments show that the proposed method yields state-of-the-an offline-oo-nling reinforce- ment learning performance the popular D4RL benchmark.
INTRODUCTION
Offline or batch reinforcement learning (RL) deals with the training RL agents from fixed datasets generated by possibly unknown behavior policies, without any interactions with the environment. This is important in problems like robotics, autonomous driving, and healthcare where data collec- tion can be expensive or dangerous. Offline RL. has been challenging for model-free RL methods due to extrapolation error where the Q networks predict unrealistic values upon evaluations on out- of-distribution state-action pairs (Fujimoto
al. 2019). Recent methods overcome this issue by constraining the policy to stay close to the behavior policy that generated the offline data distribu- tion (Fujimoto al. 2019 Kumar al.) 2020; Kostrikov et al., 2021: FFujimoto & Gul [202I). to demonstrate even better performance than the behavior policy on several simulated and real-world
tasks (Siegel et al. 2020; Singh et al. 2020 Nair al. 2020).
However, the performance of pre-trained policies will be limited by the quality of the offline dataset and it is often necessary or desirable fine-tune them by interacting with the environment. Also, offline-to-online learning reduces the risks in online interaction as the offline pre-training results in reasonable policies that could be tested before deployment. In practice, offline RL methods often fail during online fine-tuning by interacting with the environment. This offline-to-onlinn challenging due to: (i) the sudden distribution shift from offline data online data. This could lead to severe bootstrapping errors which completely distorts the pre-trained policy leading to sudden performance drop from the very beginning of online fine-tuning, and (ii) constraints enforced by offline RL methods on the policy to stay close to the behavior policy. While these constraints help in dealing with the sudden distribution shift they significantly slow down online fine-tuning from newly collected samples.
RL setting is
We propose to adaptively weigh the offline RL constraints such as behavior cloning loss during online fine-tuning, This could prevent sudden performance collapses due to the distribution shift while also enabling sample-efficient learning from the newly collected samples. We propose to perform this adaptive weighing according the agent's performance and the training stability. We

---

Under review as a conference paper at ICLR 2022
start with TD3+BC, simple offline RL algorithm recently proposed by Fujimoto & Gu (2021)
which combines TD3 (Fujmoto et al. 2018) with a simple behavior cloning loss, weighted by an hyperparameter. We adaptively weigh this hyperparameter using control mechanism similar to the proportional-derivaiiee (PD) controller. The a value is decided based on two components: the difference between the current return and the target return (proportional term) well as the change of return between current episode and the last episode (derivative term). We demonstrate that these simple modifications lead stable online fine-tuning after offline pre-training on datasets of different quality. We also use randomized ensemble of Q functions (Chen et al. (022)) to further improve the sample-efficienyy We attain state-of-the -teh-art online fine-tuning performance on locomotion tasks from the popular D4RL benchmark.
2 RELATED WORK
Offline RL. Offline RL aims to learn policy from pre-collected fixed datasets without interacting
with the environment (Lange et [012] Agarwal et al. [2020; Fujimoto et al. [2019] Kumar et al.,
2019 Nachum et al.) 2019: Siegel et al. [020 Levine al.] 2020 Peng et al. 2019). Off-policy RL
algorithms allow for reuse of off off-policy data (Konda & Tsitsiklis 2000; Degris et al. 2012 Haarnoja et al., 2018 [Silver et al. 2014 Lillicrap al. 2015 Fujimoto al. 2018 Maih et al.) 2015) but
they typically fail when trained offline on fixed dataset, even if it's collected by a policy trained using the same algorithm (Fujimoto al. 2019, Kumar et al. 2019). In actor-critic methods, this is due to extrapolation error of the critic network on out-of-distriuutin state-action pairs Levine et al. (2020). Offline RL methods deal with this by constraining the policy to stay close to the behavioral policy that collected the offline dataset. BRAC (Wu et al.] (201)) achieves this by minimizing the Kullback-Leibler divergence between the behavior policy and the learned policy. BEAR (Kumar et al. 2019) minimizes the MMD distance between the two policies. TD3+BC (Fujimoto & Gul, 2021 proposes a simple yet efficient offline RL algorithm by adding an additional behavior cloning loss to the actor update. Another class of offline RL methods learns conservative Q functions, which prevents the policy network from exploiting out-of-distribution actions and forces them to stay close to the behavior policy. CQL (Kumar et al. 2020) changes the critic objective to also minimize the Q function on unseen actions. Fisher-BRC Kostrikov et al. (021) achieves conservative Q learning by constraining the gradient of the function on unseen data. Model-based offline RL methods Yu et al., 2020: Kidambi et al.] 2020) train policies based on the data generated by ensembles of dynamics models learned from offline data, while constraining the policy to stay within samples where the dynamics model is certain. In this paper, we focus on offline-to-online RI. with the goal of stable and sample-efficien online fine-tuning from policies pre-trained on offline datasets of different quality.
Offline pre-training in RL. Pre-training has been vastly investigated in the machine learning com- munity from computer vision (Sharif Razavian et al. 2014 Donahue et al. 2014, Yosinski et al.) 2014) to natural language processing (Devlin et al.] 2018: TTurian et al.] 2010). Offline pre-training
in RL could enable deployment of RL methods in domains where data collection can be expensive or dangerous. (Silver et al.] 2016;; Gupta et al. 2019; Rajeswaran et al.] 2017) pre-train the policy network with imitation learning to speed up RL. QT-opt (Kalashnikov et al.] 2018)) studies vision- based object manipulation using a diverse and large dataset collected by seven robots over several months and fine-tune the policy with 27K samples of online data. However, these methods pre-train using diverse, large, or expert datasets and it is also important to investigate the possibility of pre- training from offline datasets of different quality. Yang & Nachum (202T); Ajay et al. (2020) use offline pre- vee-raiinig to accelerate downstream tasks. AWAC (Nair et al. 2020) and Balanced Replay Lee et al. (02T)) are recent works that also focus on affline-to-online RL from datasets of different quality. AWAC updates the policy network such that it is constrained during offline training while not too conservative during fine-tuning. Balanced Replay trains an additional neural network to pri- oritize samples in order to effectively use new data well as near-on- near-on-police on-policy samples in the offline dataset. We compare with AWAC and Balanced Replay to attain sfate-of-the-at offline-t-onlimn RL performance on the popular D4RL benchmark.
Ensembles in RL. Ensemble methods are widely used for better performance in RL (Faußer & Schwenker 2015 Osband et al. 2016; Chua al. 2018: Janner et al. 2019). In model-based RL, PETS (Chua et al. 2018) and MBPO (Janerr et al. 2019 use probabilistic ensembles to effectively model the dynamics of the environment. In model-free RI,, ensembles of Q functions have been
2

---

Under review as a conference paper at ICLR 2022
shown to improve performance (Anschel al. 2017 Lan et al. 2020). REDQ (Chen et al. 2021 learns a randomized ensemble of Q functions to achieve similar sample efficiency as model based methods without learning dynamic model. We utilize REDQ in this work for improved sample- efficiency during online fine-tuning. Specifie to offline RI., REM (Agarwal et al. 2020) uses random convex combinations of multiple Q-value estimates to calculate the Q targets for effective offline RI. on Atari games. MOPO Yu et al. (2200) uses probabilistic ensembles from PETS to learn policies from offline data using uncertainty estimates based on model disagreement. MBOP (Argenson & Dulac-Amold 2020) uses ensembles of dynamic models, Q functions, and policy networks to get better performance on locomotion tasks. Balanced Replay (Lee et al.) (021) uses ensembles of pessimistic Q functions to mitigate instability caused by distribution shift in offline-to-onling RL.
While ensembling Q functions has been studied by several prior works (Lan et al.) [0200] Chen et al., 202T), we combine it with behavioral cloning loss for the purpose of robust and sample- efficient offline-to-online RL.
Adaptive balancing of multiple objectives in RL. Ball et al. 2020 train policies using learned dy- namics models with the objective visiting states that most likely lead to subsequent improvement in the dynamics model, using active online learning. They adaptively weigh the maximization of cu- mulative rewards and minimization of model uncertainty using an online learning mechanism based on exponential weights algorithm. In this paper, we focus affline-to-onlnee RL using model-free methods and propose to adaptively weigh the maximization of cumulative rewards and a behavioral cloning loss. Exploration of other online learning algorithms such as exponential weights algorithm is a line of future work.
3 BACKGROUND
3.1 REINFORCEMENT LEARNING
Reinforcement learning (RL) deals with sequential decision making to maximize cumulative re- wards. RL problems are often formalized as Markov decision processes (MDPs). An MDP consists of a set of states S, set actions A, transition dynamics 8141 ~ ((-18t, ax) that represents the probability of transitioning reward function rt- R(s,, at), and a discount factor y E [0, 1].
state 81+1 by taking action a, in state st at timestep scalar
A policy function π of an RL agent is a mapping from states to actions and defines the behavior of the agent. The value function V=(*) of a policy æ is defined as the expected cumulative rewards from state (*)
E E 7a R st, as) transitions 8141 ~ p(-1s, az) and policy function a; ~ T(t)). Similarly, the state-action value the expected cumulative rewards after taking action state s: t ' R((,, (s) = Gg a). The goal of RL is to learn an optimal policy function Te with parameters 8, that maximizes the expected cumulative rewards:
s). where the expectation is taken state
function Q" a) defined
(8, a) = E
Kg arg max Ear [s)) arg max E.- Q (s, To (T))
We use the TD3 algorithm for reinforcement learning (Fujimoto et al. 2018). TD3 is an actor-critic method that alternatingly trains: (i) the critic network Q+ to estimate the QT (s, a) values of the policy network wo, and (ii) the policy network produce actions that maximize the function: VoQo (s))..
3.2 OFFLINE PRE-TRAINING
Offline reinforcement learning or batch reinforcement learning assumes that the agent is not able to interact with the environment but is given fixed dataset D of (s, a, s') tuples to learn from. The data is assumed to be collected by an unknown behavioural policy (or collection of policies).
The problem with using actor-critic methods for offline RL is extrapolation error due to the evalua- tion of the critic network on the next state and next action values Q(s', a') to compute the temporal difference error. Here the next action a' is sampled from the policy network a' (Tess)) and this evaluations of the critic network. This is problematic as erroncous
could lead to nut-of-distributie predictions of the critic on unfamiliar actions could be propagated to other critic predictions due to
3

---

Under review as a conference paper at ICLR 2022
bootstrapping in temporal difference learning. This will also lead to the policy network preferring actions with unrealistic value predictions. This problem can be overcome either by constraining the policy network to stay close the data distribution (Fujimoto & Gu 2021) or by enforcing
conservative estimates of the critic network on out-of-distribution samples Kumar et al. (020))
Fujimoto & Gu (2021) propose TD3+BC, simple offline RL algorithm that regularizes policy learning in TD3 with behavior cloning loss that constraints the policy actions to stay close to the actions in the offline dataset D. This is achieved by adding behavior cloning term to the policy
loss:
= arg max (ssa)> ((5,TT(()) - a(To(() - -3²²
(1)
where a is weighing hyperparameter and
Q(S,TT(())
Q( (T( = *.,.0. Q( a.)
normalizes the Q values which help in balancing both losses. The sum in the denominator is taken over mini-batch and the gradients do not flow through the critie term in the denominator.
4 ONLINE FINE-TUNING
RL agents trained from offline data tend to have limited performance and would further need to be fine-tuned online by interacting with the environment. During online fine-tuning. the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline data to online data. Keeping the constrain used in offline pre-training, such as in Equation D could mitigate the collapse. However, this will force the policy to stay close to the behavior policy (used to collect the dataset), thus leads slow improvement. In this section, we describe the two components of our online-tuning algorithm that enables stable and sample- efficient online fine-tuning.
4.1 ADAPTIVE WEIGHING OF BELAVIIR CLONING LOSS
The most straightforward way fine-tune the pre-trained policy is by just removing the constrains used in offline pre-training. For example, Balanced Replay (Lee et al. (021)) uses CQL (Kumar et al.) 2020) during offline pre-training and uses SAC (Haarnoja al., 2018) in fine-tuning. However, this strategy often leads to a performance collapse at the beginning of fine-tuning, as shown in Fig. [ (with a = 0) and the TD3-ft Fig. 2 In the TD3+BC algorithm we consider in this paper, an hyperparameter is used to balance the RL objective and the behaviour cloning term which constrains the policy to stay close to the behavior policy (see Equation IT. We use Coffline and Cleoline to distinguish the a hyperparameter value used during offline and online training respectively. By default, we use Oudiliie = 0.4 in all our experiments. We use Oenaiiee = 0.4 for TD3+BC and we observe that this prevents sudden performance drops the initial steps of online fine-tuning, the cost of very slow learning due to the strong behavior cloning constraint. On the other hand, setting Oonline leads sample-efficient learning on some tasks at the cost of complete instability in other tasks. This is due to the sudden distribution shift causing the policy network to change significantly.
In Fig. we present the influence of Coaalin on the TD3+BC during fine-tuning by trying different values of Ouunline from [0.0, 0.1, 0.3]. We can clearly see that using the behavior cloning loss with proper Gunline enables stable mee-tuning. However, the value of Ouaaliee depends on the quality of the offline dataset and has significant influence of the fine-tuning performance. For example, Coonline O fits well on the Hopper-Random task while causes immediate collapse on Hopper-Medium and Hopper-Medium-
tasks.
In our experiments, we found that when the offline dataset has narrow distribution or when the policy has already converged to desired performance (comparable to the expert), it is usually beneficial to maintain a higher Qoallin- When the data distribution is broader or when we still need to improve not find
the agent by large margin, a smaller Oanlime works better. During experiments, single Ovealine that is suitable for all tasks and its value needs to be tuned carefully per tasks, which makes this method hard to be used in practice.
To solve this problem, we propose to adapt the weight of the behavior cloning loss according to two factors: (i) the difference between current episodic return and the target return, and (ii) the episodic
4

---

Under review as a conference paper at ICLR 2022
Hopoor
Hoppor
Happer Rupllyy
Happer
4000
10.5
M MV
3000
3000
1086
2000
2000
2000
2900
1900
93
100
190
2010
200
00 102
150
200
130
100
190
230
250
Figure 1: Results of online fine-tuning on the D4RL benchmark using TD3+BC with different Clooline hyperparameters. We plot the mean and standard deviation across runs. Using the behavior cloning loss with proper Okenline enables the stable fine-tuning. But the optimal value of Conline differs between datasets.
return between current episode and the last episode. We adaptively change the Q'aline hyperparameter as:
A (kanlinee = Kp
(Reurrent
Rearett
+ K max (0, Riast Reurrent)
(2)
where we constrain Cronline between 0 and 0.4 (the value used during offline pre-training). Reurrent and Riast are normalized following the return normalization procedure used D4RL. Rarget the target episodic return, which set (corresponding to the expert policy) for all tasks. Kp controls how fast we decrease the Clonline according to current performance and KD determines how fast we increase the Conline when the performance drops. Intuitivelly when the agent's 's performance reaches the target episodic return, we try to maintain it during fine-tuning. But when the agent's performance low, decrease the Oleatine allow the agent improving further. The second term increases the Conline when performance drops during training to mitigate performance collapse. Equation alllows for adaptive weighing of the behavior cloning loss throughout online fine-tuning. This learning algorithm can automatically adjust the constraint enforced by the behavior cloning loss.
After offline pre-training the replay buffer is filled with offline samples and during online fine- tuning, they are slowly replaced by online samples. Uniformly sampling mini-batches from this replay buffer for online fine-tuning inefficient it dominated by offline samples. After offline learning we simply remove 95% of random offline samples from the replay buffer to deal with this problem. Our results show that the data hown-sampling allows efficient usage of novel data without destroying the training.
4.2 RANDOMIZED ENSEMBLES OF CRITIC NETWORKS
We propose to use an ensemble Q functions better deal with the distribution shift from offline pre-training and to improve the sample-efficinccy of online fine-tuning, We use the Randomized Ensembled Double Q-learning (REDQ) method proposed by Chen al. 2021 learn an ensemble of critic networks.
The critic network is trained satisfy the Bellman equation: Q" a) + (Q" (s)))) REDQ maintains an ensemble of N critic networks and randomly samples M networks for each critic update. Given mini-batch B of B transitions (s, a,,,,,ss . all critic networks in the ensemble are updated towards the same target:
Σ (8.0.r,8')EB
- min Qo. (s' (s') iEM
where M is random subset of M critic networks and a' clip( (e + E, alowy ahigh)) Here e is Gaussian exploration noise with standard deviation policy and [atow, ahigh) is the action range.
REDQ updates the policy network to maximize the average predictions of the critic networks:
V
Σ 1=1
(g))))))).
8EB
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Offline-to-online RL with adaptive behaviour cloning and ensembles of critic networks Initialize REDQ agent with critic parameters .1...., Initialize target parameters 6' t- 0 and of <- Φi, for i = ,.....N Initialize replay buffer R. with offline data D
.......... ΩN and policy parameters 0
for k = 0 to K do
Sample mini-batch
transitions (s, a, (7,8) from
Update critic parameters 01,.... ON using Equation 3
Update actor parameters & using Equation with a = Oloffline
Update target networks 6' r (1 00' and di TO: + (1 T) end for
Randomly remove 95% of offline samples from R
Initialize Coonline Oloffline Initialize Reurrent and Rast to store the return of current and previous episodes Initialize environment for online fine-tuning for every training episode do for 1. to 7 do
Act with exploration noise No (8:) + (O, Texpl) Observe next state s and reward Tt. Add (st, ae Tt, 8t-1) to R
for g = to G do
Sample mini-batch B of B transitions (s, a, s') from R
Update critic parameters o1...., ΦN using Equation
Update actor parameters using Equation 4 with a Oontine
Update target networks 0' + r0 + +1 70' and of t- TØS + (1 - T) end for
end for
Set RIss = Rearrent and Reurrent 2 Pt
Adapt Olonline based on Riast and Reurrent using Equation 2 end for
We combine this REDQ policy update with a behaviour cloning loss (like in Equation learning (Fujimoto & Gu 202I):
1 for robust
V
Σ
1 N
M
Qo (8, - a(To(s) =a2
(4)
B
(s,a)EB
We show that this simple modification of ensembling the critic networks (which can be run in paral- lel) improves offline-to-onling learning. We call this algorithm REDQ+AdaptiveBC. Our algorithm is outlined as Algorithm
5
EXPERIMENTS
5.1 ONLINE FINE-TUNING ON D4RL BENCHMARK
The goal of our experiments is evaluate the stability and sample-efficiency of the proposed al- gorithm on online fine-tuning after offline pre-training on datasets of different quality. We evaluate our algorithm on online fine-tuning after offline pre-training on the D4RL benchmark (Fu et al., 2020). D4RL includes three locomotion tasks (halfcheetah, hopper, and walker) implemented in the MuloCo simulator (Todorov al.) 2012), wrapped in OpenAl Gym API (Brockman et al., 2016). D4RL provides five different offline datasets for each task: Random, Medium, Medium-Keplay, Medium-Expert, and Expert. The Random datasets are collected by random policies, Medium datasets are collected by early-stopped soft actor-critic (SAC) (Haarnoja et al. 2018) agent with medium-level performance, Kediium-Replat datasets consist all samples in the replay buffer after training medium-leve agent, Medium-Expor datasets mixed with expert demonstrations and sub-optimal demonstrations from a medium- agent, and Expert datasets are expert demonstra- tions. The "expert" in these datasets is fully trained soft-actor critic agent. We ignore the Expert

---

Under review as a conference paper at ICLR 2022
HulChhoosh-Meduun
looahh Medi Ruply
Modiute Export
1403
0203
0000
#103
0000
5300
9983
MMAG
2353
Cearoed Tapley
2000
2033
parseing
1C5
100
100
205
Hipporr
Hopport by
Hopport Expert
+003
4000
4003
3000
2303
2000
2003
2000
100
106
205
on 2ab-Bando
Wia Wallaar2th--We uni-lagray
Expert
1000
0003
5000
5003
3000
4000
2000
980
1000
5000
Time Stepe (10)
Times Stppa (1621
Time Shpps (163)
Time Steps (1453
Figure 2: Results of online fine-tuning on the D4RL benchmark. We plot the mean and standard deviation across runs. Our REDQ+AdapivvBCC method attains performance competitive to the state-of-the-arr. It should be noticed that, unlike other methods, our results do not collapse immedi- ately at the beginning training.
datasets in this paper as offline RL algorithms already achieve expert-level performance on these tasks and there is little to no benefit in online fine-tuning.
In Figure 2 we compare our REDQ+AdaptiveBC algorithm with two state-of-the-ar art offline-to- online RL algorithms (AWAC and Balanced Replay) and two bascline methods (TD3-ft and REDQ):
Advantage Weighted Actor-Critic (AWAC) (Nair et al. 2020) is an actor-critic method for offline-to-onliee RL that implicitly constraints the policy network to stay close to the behavior policy. We produce the results for AWAC using code taken from https:/7 jaxrl
github. Mattuub.cmmiin
Balanced Replay (Lee et al. 2021 is an offline-to-online RL method that prioritizes near- on-policy samples from the replay buffer. This method also uses an ensemble of Q func- tions prevent overestimation of Q values in the initial stages of online fine-tuning. We reproduced the results for this method using our own implementation. For a fair compari- son, we base our implementation on TD3+BC (instead of CQL originally used by et al.) (202I)) while ensuring that we able to reproduce the original results.
TD3-ft is the standard TD3 algorithm (Fujimoto et al. 2018 that was pre-trained offline using TD3+BC (Fujimoto Gu 2021).
REDQ (scratch) (Chen al.) 2021) is RL method trained from scratch, without any access to the offline data. This baseline emphasizes the importance of offline pre-training and online fine-tuning. We base REDQ implementation TD3 (instead of SAC used by Cheh et al. 220T)) for compatibility with TD3+BC.
All methods (except AWAC) are implemented on top on TD3 and are run from the same codebase for fair comparison. For simplicity, we do not perform any state normalization like in the original TD3+BC implementation (Fujimoto & Gu 2021).
During offline pre-training, all algorithms dient steps. After pre-training, we fine-tune the agents for 250,000 time steps by interacting with the environment. We evaluate the agent every 5000 time steps and each evaluation consists of 10
pre- treitrained the offline dataset for one million gra-
7

---

Under review as a conference paper at ICLR 2022
episodes. We attain performance competitive to the state-of-ther art in this benchmark with our method stably improving the performance during online fine-tuning.
We significantly outperform all others methods in the HalfChectah domain for three datasets. We perform slightly worse than Balanced Replay on Walker2d- and but outperform it get similar results in all other cases. We need to mention that in both Walker2d- Medium and Walker2d- 2alker2d-Medium Replly tasks, our method already reaches the target performance (predefined following D4RL), and the a is increased automatically to maintain the performance and thus fail to improve further. We significantly outperform REDQ on all tasks, which demonstrates that we considerably benefit from offline pre-training. TD3-ft is able to improve from online fine- tuning but suffers from significant performance drops due to the sudden distribution shift and the learning progress is slow due to the replay buffer being dominated by offinn samples. It should be noticed that, unlike other methods, our algorithm does not collapse immediately on all three Medium-Expert tasks.
Both Balanced Replay and our method (REDQ+AAdaptiveC) use an ensemble of 10 Q networks, but in different ways. Balanced Replay maintains a pair of five ensemble networks, average the predictions across each of the five networks and then takes the minimum of the averages as the final prediction. In our method, we simply consider the average of all 10 networks as the prediction but randomly sample pair of Q networks to compute the critic targets (Equation B3. We show that this simple modification enables stable and sample-efficient online fine-tuning without the need for any complex sampling scheme from the replay buffer.
Similar to prior works (Fujimoto & Gu 2021), we use feed-forward networks with two hidden layers as actor and critic networks for all the methods. We use batch size of 256 to train the network for all methods, except for AWAC where we use a larger batch size of 1024 (Nair et al.) 2020 During offline learning. we use deffline = 0.4 for all tasks, except Walker-Randos where we use Ooottiie = 100 since the dataset has very narrow distribution. We list all the hyperparameters used in our experiments in Table in the Appendix.
5.2 ALGORITIMICC INVESTIGATIONS
Adaptive Weighing of Contine: To evaluate whether the proposed method can correctly select good Gonline for stable online fine-tuning, in Figure 3 we compare the results obtained with the automat- ically tuned Conllnn with manually tuned results. To mamually tune the Claalines for each domain and each dataset, we do grid search Conline over [0.0, 0.1, 0.2, 0.3) and pick the best O'valine separately for each task.
We that with manually tuned Gontine our method consistently outperformances other methods in Figure except the Balanced Replay on Walker2d- Mediuu.. Our results with automatically tuned Contine are slightly worse than manually tuned results on HalfCheetah tasks. However, our method successfully find the similar Coonline on Halfcheetah tasks as we manually selected after roughly 10-15 episodes. On the rest tasks, our automatically tuned results are competitive the carefully picked results but saving lots of labor and computational resources.
Offline Dataset Downsampling: Balanced Replay (Lee et al. 2021 trains a neural network to estimate the priority of samples from offline data and online data. In their methods, three replay buffer need to be maintained: offline dataset (0.1-2M samples), online dataset (0.25M samples) and a prioritized replay buffer (0.35M-2.25M samples) (Schaul et al.) 2015). making it memory consuming (0.7M-4.5M samples). Our method simply dowsamples the offline dataset by 95%, thus, method only maintains replay buffer to store online data but is prefilled with 0.05M offline data points, roughly saves 65% 95% memory.
To demonstrate the effectiveness of dataset downsampling, we compare TD3_ft with and without dataset downsampling on three random datasets, shown in Figure 4 Our results show that the downsampling procedure allows the agent effectively sampling the novel data encountered during fine-tuning. This is even important when the data quality of the offline data is not good enough, such as when the dataset is collected by random policy.
8

---

Under review as a conference paper at ICLR 2022
HalChootsh Random
HulChheosh-MMddun
looahh Medi Ruplay
Moduute Expart
1400
0003
12:03
19303
1000
8000
1203
7900
5300
8000
9300
2033
00.00
190
1C5
156
100
50
250
Randion
Hoppor
Hoppor by
Hopport Expert
3.20
3503
4000
3003
3000
2093
2000
2593
2000
3300
1900
1333
2730
1000
1003
2000
1CB
156
205
onr Dandra
Wi Wawlaar2t---eee uni-llggaa
War Expert
5000
9000
4500
1003
5000
4003
3900
5099
3000
3000
2000
1003
Time Stepe (103)
Times Stppa (1621
Thme Stppa (163)
Time Staps (1403
Figure 3: Comparison of results with automatically tuned Gonline and carefully picked results. It shows that our proposed method can effectively find the suitable C'ooline for all tasks.
Hoppar-Random
Walker2d-Rarn Random
Horcheetah- Random
4000
14000
4000
10000
Returia
5000
8000
2000
6000
1000
down
1000
4000
sempiing
2000
100
200
50
150
200
100
150
250
Time
Figure 4: Comparison of TD3-ft with and without dataset downsampling. We plot the mean and standard deviation across 3 runs. Downsampling enables effective usage of novel data encountered during fine-tuning vuentun
6 CONCLUSION
We consider the problem of offline-to-online RL where an agent is first pre-trained on offline data collected (by a possibly unknown behavior policy) and the agent is then fine-tuned online by inter- acting with the environment. This is desirable as pre-trained agents may have limited performance depending on the quality of the offline dataset. Offline-to-online RL is challenging due to the sud- den distribution shift from offline data to online data, and also the constraints enforced by offline RL algorithms (such as a behavior cloning loss) during pre-training. In this paper, we propose a simple mechanism to adaptively weight a behavior cloning loss during online fine-tuning, based on agent performance and training stability. We demonstrate that randomized ensemble further helps to deal with these challenges to enable sample-efficis online fine-tuning performance. We attain performance competitive to the the popular D4RL. benchmark.
online fine-tuning methods on locomotion tasks from

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.1300 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQNN Variance reduction and stabi- lization for deep reinforcement learning. In International conference on machine learning, pp- 176-185, PMLR, 2017.
Arthur Argenson and Gabriel Aulac-Amold. Model-based offline planning. arXiv preprint N0fv:2000.0055566 2020.
Philip Ball, Jack Parker-Holder. Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Ready policy one: World building through active learning. In International Conference on Ma- chine Learning, pp. 591-601. PMLR, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: 1rX6v:1606.01540 2016.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double Q- learning: Learning fast without a model. In International Conference on Learming Representa- fions, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12111 2018.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint arXiv:1205.4833 2012.
Jacob Devlin, Ming- Ming Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810004805. 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference machine learning, pp. 647-655. PMLR, 2014.
Stefan Faußer and Friedhelm Schwenker. Neural network ensembles in reinforcement learning. Neural Processing Letters, 41(1):55-69, 2015.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06800 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing funetion approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1956, 2019.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: 18/2.05905 2018.
10

---

Under review as a conference paper at ICLR 2022
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. arXiv preprint arXiv:1906.08255. 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprini arXiv: 10293, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05955 2020.
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1011 1014, 2000.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning. pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 2019.
2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 04779, 2020.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Controlling the estimation bias of Q-learning International Conference on Learning Representations, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Seunghyun Lec, Younggyo Seo, Kimin Lee, Picter Abbeel, and Jinwoo Shin. Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble. arXiv preprint arXiv:2107.00591, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint urXiv:2005.01643 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (rXxv::509.02977 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. arXiv preprint arXiv:1906.04777 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Ian Oshand, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4034, 2016.
Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. wdvintage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2012
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint (xXi:1709.1008/) 2017.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: 2015.
11

---

Under review as a conference paper at ICLR 2022
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off- the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806-813, 2014.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. International Conference on Learning Representations, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. PMLR, 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(75877:484-48,, 2016.
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 14500, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 384-394, 2010.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint 19Xiv:191..11320. 2019.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci-
sion making. arXiv preprint arXiv:2 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.1792. 2014.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13238 13239, 2020.
12

---

Under review as a conference paper at ICLR 2022
Hopou
Hoapor
Happer Raploy
Happer Expert
MM
3009
3300
2000
2003
2530
190
1008
7
93
100
190 270
250
00
100
100
200
150
130
100
190
250
250
Figure 5: Comparison of usages of ensembles on the hopper domain. We plot the mean and standard deviation across 3 runs. Randomized ensembled double Q-Learning stabilizes the training, but is not necessary to avoid performance collapse during fine-tuning.
Table 1: Hyperparameters used in our experiments
Hyperparameter
Value
Optimizer
Adam
Learning rate
3e-4
Batch size
256
TD3
Target update rate
5e-3
Policy noise std
0.1
Policy noise clip
0.5
Policy update frequency
2
Hidden layers
2
Architecture
Hidden units Activation function
256
ReL.U
Number of networks N
10
REDQ
Randomly sampled networks M
2
Number of updates G
20
Offline BC
Oloftine
0.4
Adaptive BC
Kp
3e-5
K
8c-5
A ABLATION ON ENSEMBLES
In our experiments, we use ensembles to represent the critic network. In 5 we compare the training results with and without ensembles as well as the way to use ensembles. Our results show that using ensembles is not necessary to avoid performance collapse, however, it stabilizes training in most cases. Also, the way to use ensembles matters. We compare calculating target values with randomly sampled Q predictions and with minimal predictions. Our results show that using minimal Q predictions to calculate target values hurts the performance in most cases.
B HYPERPARAMETERS
All the hyperparameters and network architectures used in experiments are listed in Table
13


Reviews:
[{"review_id": "xR8pUjsKONq", "reviewer": "Reviewer_q4CJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is clearly written and easy to understand. The proposed algorithm is also novel in the sense that it gives an adaptive scheme of selecting the amount of conservatism during online training depending on the current policy performance. The empirical results also clearly show that the method can prevent the policy from degrading in medium-expert datasets where we don't need to change the current policy much, aligning with the initial motivation of the method.\n\nI also have a few concerns. First, while the results of the method generally align with the intuition that the policy should use large alpha on narrow datasets with good data and use large alpha on diverse datasets with low-quality data, the performance of the method on hopper and walker2d is a bit underwhelming. The method seems to be the same as hopper and also worse than Balanced Replay on three of the walker datasets. While the authors mentioned that the less satisfying results on walker2d are caused by the predefined target score in D4RL, I wonder if that is potentially a limitation of the method since it relies on the human-defined target score. Also, how would the method do if we change the target score to be higher (e.g. R_max * T)?\n\nMoreover, I think the paper lacks some theoretical understanding of the method. The intuition of the method makes perfect sense, but I would like to see how this adaptive scheme can give use some theoretical insights on policy improvement guarantees. Prior works [1] have studied this and I wonder how the proposed method can be connected to that.\n\nFinally, beyond the mujoco tasks, I think it would be good to evaluate the method on more realistic tasks such as the dexterous manipulation tasks used in the AWAC paper. \n\nMinor comment: it would be great to have an ablation study on the REDQ ensemble since that seems to be a bit disjoint to the theme of the paper.\n\n[1] Xie, Tengyang, et al. \"Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning.\" arXiv preprint arXiv:2106.04895 (2021).", "summary_of_the_paper": "This paper proposes a new offline RL with online fine-tuning method. The authors first pretrain the policy using recent offline RL method TD3+BC with offline data and then collect on-policy data to further improve the pretrained policy. To prevent the policy from either degrading performance or failing to improve at the fine-tuning stage, the authors propose an automatic scheme to adjust the $\\alpha$ term that controls the BC loss in TD3+BC to ensure that the policy does not continue to contain itself to the behavior policy if there's room for improvement and also is able to stick to the previous policy if the performance is already near optimal. The authors conduct evalutions in D4RL mujoco environments and show that the approach is able to outperform prior methods in halfcheetah.", "summary_of_the_review": "Given my comments in the above section, I think the paper could be improved if the authors can show the results of the method with a more realistic target score (e.g. R_max * T), provide the theoretical justifications, and evaluate the method on more realistic domains. I would vote for a weak reject given the current status.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tn0P7MYMyXj", "reviewer": "Reviewer_6n3w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n** This paper considers an important problem setting for RL.\n\n** Paper is well written and is easy to follow.\n\nCons:\n\n** The proposed solution is heuristic and is similar to other methods that use annealing to tune the regularizer weight. And similar to other heuristics in this space, this method also has this limitation that it requires pre knowledge about the final performance of the task. I would like to see a more principled solution that at-least does not require such knowledge. \n\n** Evaluations are done on simple low dimensional tasks. Given the limited technical contribution, i would like to see a study on more complicated tasks with for example visual inputs where representation learning introduce additional challenges for finetuning.\n\n** The other claimed contribution on using ensemble of Q functions is orthogonal to the problem under study as it has shown before that ensemble of Q-functions helps to stabilize learning. ", "summary_of_the_paper": "This paper proposes an update rule to adaptively change the regularizer term (BC here) weight during online learning for more stable fine tuning to improve over an expert.", "summary_of_the_review": "This paper considers an important topic for RL. However the proposed solution is heuristic and evaluations are not convincing given the simplicity of the tasks. Overall i think this paper does not offer enough contribution in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p6j7ZJYxFh", "reviewer": "Reviewer_aXQe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- This paper is well-written and easy to follow.\n- The focus of this paper, fine-tuning in RL seems to be a significant topic towards real-world applications, overcoming the bottleneck of current offline RL research.\n- The empirical comparisons seem to show better or competitive performance to AWAC, Balanced Replay, TD3_ft, and REDQ.\n\n## Weaknesses\n- The proposed method REDQ+AdaptiveBC seems to require a lot of hyper-parameter tuning. First, $\\alpha_{offline}=0.4$ is a very small value compared to original TD3+BC, which uses $\\alpha_{offline}=2.5$ after a grid search from \\{1.0, 2.0, 2.5, 3.0, 4.0\\}. This implies that $\\alpha_{offline}=0.4$ (and 100 for walker2d-random) might be a result of extensive hyper-parameter tuning. In addition, I guess $K_p$, $K_d$ (and even target return) must be well-balanced. I'm not sure how they are determined.\n- Also, REDQ+AdaptiveBC seems incremental on top of TD3+BC, combining the established Q-ensemble method (Chen et al. 2021) and carefully-designed coefficient tuning, and resembles Balanced Replay (Lee et al. 2021). I summarized the problems in fine-tuning and the remedies for them on the table. Both Balanced Replay and REDQ+AdaptiveBC overcome the data distribution shift issues using the ensembles of Q-functions. For imbalance issues of the offline-online replay buffer, Balanced Replay utilizes  Prioritized Experience Replay, and REDQ +AdaptiveBC downsamples the offline data, both of which aim to reduce the ratio of offline data and encourage to leverage the online samples when online fine-turning. During the online training phase, behavior regularization for stable offline training can prevent the improvement of the policy. Balanced Replay, proposed on top of CQL, removes the CQL loss and train value function and policy as online SAC, and REDQ+AdaptiveBC gradually reduce the ratio of BC loss. I think the novelty and contribution of this paper are not enough.\n\n|Problem|Balanced Replay (Lee et al. 2021) | REDQ+AdaptiveBC  |\n| ---- | ---- | ---- |\n|  **Distribution Shift**  |  Pessimistic Q-Ensemble  |  REDQ (Chen et al. 2021)  |\n|  **Imbalanced Replay Buffer** |  PER  |  Downsampling  |\n|  **Behavior Regularization**  | CQL-SAC switching |  Adaptive $\\alpha_{online}$  |\n\n- I think the comparison might be insufficient, since Balanced Replay, proposed on top of CQL, is based on TD3+BC in this paper. I don't think the comparison to CQL-based Balanced Replay isn't fair. In addition, while adaptive BC heavily depends on TD3+BC algorithms, Balanced Replay seems to work on both CQL and TD3+BC. I also think the authors can add TD3+BC baseline (keeping $\\alpha_{offline}$ during online training) and +REDQ/-REDQ ablations.\n\n### Minor Comments\n- I'm not sure how can I interpret the results of Figure 4. In my understanding, downsampling of replay buffer is introduced to overcome the instability of the beginning of the online training due to the imbalanced replay buffer (as discussed in Lee et al. 2021), but the caption says \"Downsampling does not hurt the training\". Doesn't downsampling improve the performance?\n- You can consider adding Ghasemipour et al. (2021), that proposes the ensemble-based offline RL algorithms, to the related work.\n- Prior works of offline-to-online fine-tuning (AWAC, Balanced Replay) work on robotic environments. Comparison on these environments, not only current MuJoCo locomotion, might be helpful.\n\n\n### Reference\nFujimoto and Gu. A Minimalist Approach to Offline Reinforcement Learning.  ArXiv preprint arXiv:2106.06860 (2021).\n\nLee et al. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble. Conference on Robot Learning (2021).\n\nChen et al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. International Conference on Learning Representations (2021).\n\nGhasemipour et al. EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL. International Conference on Machine Learning (2021).\n", "summary_of_the_paper": "This paper studies the fine-tuning problem from offline to online RL. While the naive approach to fine-tune offline policy suffers from a sudden distributional shift by online samples and too much behavior constraint in offline algorithms. The proposed method leverage (1) the adaptive coefficient tuning in TD3+BC loss, (2) randomized ensembles of Q-functions (proposed by Chen et al. 2021), and (3) down-sampling of offline data. The experiments seem to show better or competitive results to other approaches.", "summary_of_the_review": "As I described in Main Review, this paper seems to require extensive hyper-parameter search (proper $\\alpha_{offline}$ and well-balanced $K_p$, $K_d$), and to propose the incremental algorithms on top of TD3+BC (and resemble Lee et al. 2021), combining Q-ensemble method proposed by Chen et al. (2021), downsampling for imbalanced replay buffer, and adaptive $\\alpha_{online}$ tuning. Due to the lack of novelty and significant contributions to the ICLR (or RL) community, I vote for rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PJEI2lBTwYX", "reviewer": "Reviewer_bnwi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized and the motivation is really clear. Offline-to-online learning considered in this paper is an important problem. I agree that adaptively updating the behavior policy constraint is the key point in online fine-tuning. In the experiments, the authors compare the two SOTA offline-to-online methods AWAC and Balanced Replay, and ablation studies show the effectiveness of adaptive $\\alpha$.\n\nMy main concern is the novelty. REDQ and TD3+BC are prior work, and it seems that the main contribution in this paper is the adaptive mechanism of $\\alpha$. Although this paper tries to solve an important problem, the contribution of the proposed method is limited.\n\nThe update rule of $\\alpha$ is heuristic. Although it is well-motivated, more theoretical analysis is expected to demonstrate how the adaptive  $\\alpha$ achieves the balance between performance improvement and performance dropping. And the selection of hyperparameters KP and KD is arbitrary. More empirical results are needed to show how the two hyperparameters affect the update of $\\alpha$​. In Figure 3, AdaptiveBC does not significantly outperform ManualBC. Although ManualBC uses grid search, the search space is not too large. \n\nIn Figure 2, if no expert data is involved, REDQ (scratch) could achieve a good performance in 250000 timesteps, which shows pure online learning is enough to deal with this setting. In that case, it is important to consider the sample efficiency and deployment efficiency[1] in offline-to-online learning.\n\n[1]https://arxiv.org/abs/2006.03647", "summary_of_the_paper": "The authors combine REDQ and TD3+BC and propose an adaptive mechanism to update $\\alpha$ during the online fine-tuning. The proposed method decreases $\\alpha$ for further improvement and increases  $\\alpha$  to avoid performance collapse.", "summary_of_the_review": "The paper aims to solve an important problem: adaptive behavior policy constraint in offline-to-online learning. However, the proposed method is heuristic, and more theoretical analysis is expected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach that allows online finetuning of an offline RL policy by adaptively changing a BC regularization term.

Even after discussions with the authors, the reviewers had several concerns. First, the paper seems to be limited in novelty as the "REDQ+AdaptiveBC seems incremental on top of TD3+BC". Second, there were concerns that the adaptive regularization term was insufficient as a contribution given its heuristic nature.

Given the consensus among reviewers of this paper, I recommend rejecting this paper.
Decision: Reject