Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
Authors: Seunghyun Lee
Younggyo Seo
Kimin Lee
Pieter Abbeel
Jinwoo Shin
Venue: CoRL2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Offline RL
Fine-tuning

Abstract: Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RL. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method improves sample-efficiency and final performance of the fine-tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: https://github.com/shlee94/Off2OnRL.

Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
Seunghyun Leest
Younggyo Seo
Kimin Lee:
Pieter Abbeel
Jinwoo Shin
Korea Advanced Institute of Science and Technology UUniversity of California, Berkeley
Recent advance in deep offline reinforcement learning (RL) has made it possible train strong robotic agents from offline datasets. However, depend- ing on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RI. To address this issue, we first propose a balanced replay scheme that priori- tizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfa- miliar actions at novel states during the initial training phase. We show that the proposed method improves tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: https://gitae tttbbb..t..u///g///g/ com wthheee9//0fff20//
Abstract:
and final performance of the fine-
Keywords: Deep Reinforcement Learning, Offline RL, Fine-tuning
Introduction
Deep offline reinforcement learning (RL) [1] has the potential to train strong robotic agents without any further environment interaction by leveraging deep neural networks and huge offline datasets. Accordingly, the research community has demonstrated that offline RL can train both simulated [2, 3, 4, 5, 6, 7, 8, 9] and real [5, 10] robots that are often more performant than the behavior policy that generated the offline dataset. However, thusly trained offline RL agents may be suboptimal, for (a) the dataset they were trained on may be suboptimal: and (b) environment in which they are deployed may be different from the euvironment which the dataset was generated. This necessitates an online fine-tuning procedure, where the robot improves by gathering additional samples.
Off-policy RL algorithms well-suited for offline-to-online RL, since they can leverage both of- fline and online samples. Fine-tuning an offline RI. agent using a conventional off-policy RL. algo- rithm, however, is difficult due to distribution shift, i.e., the robot may encounter unfamiliar state- action regime that not covered by the offline dataset. The Q-function cannot provide accurate value estimates for such duitrf-distributiin (OOD) online samples, and updates with such samples lead to severe bootstrap error. This leads to policy updates in an arbitrary direction, destroying the good initial policy obtained by offline RL.
To address state-action distribution shift, we first introduce a balanced replay scheme that enables us provide the robotic agent with ce-r-on-policy samples from the offline dataset, in addition to samples gathered online. Specifically, we train a network that measures the online-ness of available samples, then prioritize samples according this measure. This adjusts the sampling distribution for Q-learning to be closer to online samples, which enables timely value propagation and more accurate policy evaluation in the novel state-action regime.
However, we find that the above sampling scheme not enough, for the Q-function may be overop- timistic about unseen actions at novel online states. This misleads the robot to prefer potentially
Equal Contribution. Correspondence to (seunghyun.c lee, younggyo. seo) @kaaatttt............ ac.kr
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

bad actions, and in turn, more severe distribution shift and bootstrap error. We therefore propose a pessimistic Qeensemble scheme. particular, we first observe that a specific class of offline RL algorithms that train pessimistic Q-functions [8, 9] make an excellent starting point for offline-to- online RI. When trained as such, the Q-function implicitly constrains the policy to stay near the behavior policy during the initial fine-tuning phase. Building on this observation, we leverage mul- tiple pessimistic Q-functions, which guides the robotic agent with a more high-resolution pessimism and stabilizes fine-tuning.
In our experiments, we demonstrate the strength of our method based on (1) MuJoCo [11] loco- motion tasks from the D4RL [12] benchmark suite, and (2) vision-based robotic manipulation tasks from Singh et al. [10]. We show that our method achieves stable training during fine-tuning, while outperforming all baseline methods considered, both in terms of final performance and sample- efficiency. We provide a thorough analysis each component of our method.
Background
Reinforcement learning. We consider the standard RL framework, where agent interacts with the environment so as to maximize the expected total return. More formally, at each timestep t, the agent observes state 8t, and performs action a, according to its policy The environment wards the agent with rr, then transitions to the next state 8t41- The agent's 's objective to maximize the expected return E,, 1-> r<r,). where 7 E [0, 1) is the discount factor. The unnormalized sta- tionary state-action distribution under и is defined d= (s, = Σi= > d (s, a). where di (s, a) denotes the state-action distribution at timestep of the Markov chain defined by the fixed policy m.
Soft actor-critic. We mainly consider off-policy RL algorithms, class of algorithms that can, in principle, train an agent with samples generated by any behavior policy. In particular, soft actor-critic [SAC; 13] is an off- ooflccl actor-critt algorithm that learns soft Q-function Qs a) parameterized by 0 and a stochastic policy Fe modeled as Gaussian, parameterized by Φ. SAC alternates between critic and actor updates by minimizing the following objectives, respectively:
CSAC Critsc (0)
E (~B
(Qo(s,a) - r -7 - E [Qa(s', a') log (a(d(s))))
(1)
d'
actor (Ø)
E ar Baw
a log (als)) Q0(s,a)].
where B is the replay buffer, the delayed parameters, and a the temperature parameter.
Conservative Q-learning. Offine RL algorithms are off-policy RI algorithms that utilize static datasets for training an agent. particular, conservative Q-learning [CQL; 9] pessimistically evalu- ates the current policy, and learns lower bound (in expectation) of the ground-truth Q-function. To be specific, policy evaluation step of CQL minimizes the following:
CCOL Ceritic (0)
1
E (=)
[Qo -8* + 0o E [og Σ a() E [Q(s, a)], (3) anii
amB
where ( (sp) = Enees [(==s),a=ou) E.eg 1[s=s0] is the empirical behavior policy, ag the trade-off factor, and 8 the bellman operator. The first term is the usual Bellman backup, and the second term is the
regularization term that decreases the Q-values for unseen actions, while increasing the Q-values for seen actions. We argue that thusly trained pessimistic Q-function is beneficial for fine-tuning as well (see Figure lc), Policy improvement step is the same as SAC defined in (2).
3 Fine-tuning Offline RL Agent
In this section, we investigate the distribution shift problem in offline-to-online RL. We first ex- plain why an agent being fine-tuned can be susceptible to distribution shift, and why distribution shift is problematic. Then, we demonstrate two important design choices that decide the effect of distribution shift on fine-tuning sample selection and choice of offline Q-function.

---

haircheelah--anddee
had merddmm
2000
0.02
Offine
Orline
Nonatoos Fegueno
7000
6.44
5000
20
60
Steas (4117)
60
100
203
2000
602
4000
Lis (ssympp)
Laikerm
10000
001
Ortiie
5000
Online
6.02
-70
-00 Log Boad
-50
2000
20
40
Steau
(a) State-action distribution shift
(b) Sample selection
(c) Choice of offline Q-function
Figure 1: (a) Log-likelihood estimates of (i) offline samples and (ii) online samples gathered by the offline RL. agent, based on a VAE model trained on the offline dataset. (b) Fine-tuning per- formance on halfcheet mediimm task when using online samples exclusively (Online only), or when using both offline and online data drawn uniformly at random (Uniform). (c) Fine-tuning performance on pessimistic (denoted CQI.-init) and a non-pessimistic (denoted FQE-init) Q-function, respectively.
and half cheetah-medi -mme-mmeei---eeeett---e-- tasks, when using
3.1 Distribution Shift in Offline-to-Online RL
In offline-to-online RL, there exists a distribution shift between (s. and dott (s. a), where the former denotes the state-action distribution of online samples in the online buffer goD and the latter that of offline samples in the offline buffer Botr_ Figure la visualizes such distribution shift. Specifically, we trained variational autoencoder [14] reconstruct state-action pairs in the dataset that contains uniform random policy rollouts in the halfcheetah en- vironment. Then, we compared the log-likelihood of (a) offline samples and (b) online samples collected by a CQL agent trained on the same dataset. There is clear difference between offline and online state-action distributions.
Such distribution shift is problematic, for the agent will enter the unseen state-action regime, where Q-values (hence value estimates used for bootstrapping)
be very inaccurate. Updates in such unseen regime results in erroneous policy evaluation and arbitrary policy updates, which destroys the good initial policy obtained via offline RL. Distribution shift can be especially severe in offline- to-online RL, for the offline RL agent is often much more performant than the behavior policy (e.g.. CQL can train a medium-level agent capable of running, using transitions generated by random policy only). Also, when the offline dataset narrowly distributed, e.g., when is generated by single policy, the agent is more prone to distribution shift, for the agent easily deviates from the narrow, seen distribution.
3.2 Sample Selection
In light of the above discussion, we study how sample selection affects fine-tuning, We find that online samples, which are essential for fine-tuning, are also potentially dangerous OOD samples due to distribution shift. Meanwhile, offline samples are in-distribution and safe, but leads to slow fine-tuning. As a concept experiment, we trained an agent offline via CQL (3, 2) on the dataset containing medium-level transitions, then fine-tuned the agent via SAC (1, 2). We see that using online samples exclusively for updates (denoted Online only in Figure 1b) leads unstable finc-tuning, where the average return drops from about 4500 to be- low 3000. This demonstrates the harmful effect of distribution shift, where novel, OOD samples collected online cause severe bootstrap error.
On the other hand, when using a single replay buffer for both offline and online samples then sam- pling uniformly at random (denoted Uniform in Figure 1b), the agent does not use enough online samples for updates, especially when the offline dataset is large. As a result, value propagation is slow, and as seen in Figure lb, this scheme achieves initial stability at the cost of asymptotic perfor- mance. This motivates a balanced replay scheme that modulates the trade-off between using online samples (useful, but potentially dangerous), and offline samples (stable, but slow fine-tuning).
3

---

3.3 Choice of Offline Q-function
Another important design choice in offline-to tonoline RL is the offline training of Q-function. In par- ticular, we show that a pessimistically trained Q-function mitigates the effect of distribution shift, by staying conservative about OOD actions in the initial training phase. As a concept experiment, we compared the fine-tuning performance when using pessimistically trained Q-function and when using a Q-function trained without any pessimistic regularization. Specifically, for a given offline dataset, we first trained a policy T6 and its pessimistic Q-function Qoae via CQL (3). Then we trained a non-pessiminn pon-pessimistic Q-function Q+e of the pre-trained offline policy Fg via Fitted Q Eval- uation [FQE; 15], off-policy policy evaluation method that trains given policy's is Q-function. Finally, we fine-tuned { QPea. and { Qepe) via SAC (1, 2). See Section for more details.
As shown in Figure 1c, both pessimistic and non pessimisiii Q-functions show similar fine-tuning performance on the random dataset, which contains random policy rollouts with good action space coverage. However, when fine-tuning an offline RL agent trained on the medium experr dataset, which contains transitions obtained by a mixture of more selective and performant policies, non- pessimistic Q-function loses the good initial policy, reaching zero average return at one point.
The reason is that QFqE can be overly optimistic about OOD actions at novel states when bootstrap- ping from them. In turn, the policy may prefer potentially bad actions, straying further away from the safe, seen trajectory. On the other hand, Qun. remains pessimistic in the states encountered on- line initially, for (1) these states are incrementally different from seen states, and (2) Q-function will thus have similar pessimistic estimates due to generalization. This points to a strategy where we first train pessimistic Q- Q-function offline, then let it gradually lose the pessimism as the agent gains access to balanced mix of offline and online samples via balanced replay during fine-tuning. Fur- thermore, since a single agent's s Q-function may not be pessimistic enough, we may train multiple agents offline in parallel, then deploy online the ensemble agent equipped with a regher-resolution pessimism of the Q-ensemble (see Section 4.2 for a more detailed explanation).
4 Method
Pessimietic Offiine Training
Actio
Ensembre Agent
Environment Interaction
We propose simple yet effective framework that addresses the state- action distribution shift described in Section 3. Our method comprises of two parts: (a) balanced experience re-
2 Crttic 2
Offinee
Balanced Replay
Actor N
Prion
Chiline Buffer
play scheme, and (b) a pessimistic Q- ensemble scheme.
Offline RL
Online RL
Figure 2: Illustration of our framework. We first train an ensemble of N CQL agents on the offline dataset. Then we fine-tune the ensemble agent using both offline and online transitions via balanced replay. In particular, we train density ratio estimator that measures the online- ness of a given sample, then store all samples in the pri- oritized replay buffer with their respective density ratios as priority values. In turn, samples are drawn with proba- bility proportional their respective priority values.
4.1 Balanced Experience Replay
We introduce balanced replay scheme that enables us to safely utilize online samples by leveraging relevant, near- on-policy offline samples. By doing so, widen the sampling distribution for updates around the on-policy sam- ples and enable timely value propaga- tion. The challenge here is how to design a scheme that locates and retrieves such relevant, near-on- policy samples from the offline dataset, which can often be huge. To achieve this, we measure the online-ness of all available samples, and prioritize the samples according to this measure.
In particular, when updating the agent, we propose to sample transition (s, a, a,)") s') E goft U with a probability proportional to the density ratio w(s, a) := dd (s, a) /d (s, a) of the given sample. retrieve relevant, near-on-policy sample (s, a, s') E Bott by locating transition with high density ratio wiss, a). However, estimating the likelihoods doff a) and det a) is difficult, since they in principle be stationary distributions of complex policy mixtures To avoid this problem, we utilize a likelihood-free density ratio estimation method that estimates w/ss, a) by training a network Wp (s, a) parametrized by vh solely based on samples from Boff and gen.
This way,
We remark that d=ft a) is the stationary distribution of the (arbitrary) behavior policy that generated gott, and (s, a) the stationary distribution of the policy that generated "" which corresponds to the mixture of online policies observed over the course of tuning.

---

Training details. Here we describe the training procedure for the density ratio estimator (y a) in detail. Following the idea of Sinha et al. [16], we use the variational representation of f-divergences [17]. Let P and Q be probability measures defined on some measurable space X, with P absolutely continuous w.r.t. Q, and f(y)= == y log y+T
2- 2y + log T+T.. Then the Jensen-Shannol (IS) divergence is
defined as (Ps(P)l fx [ddP()) (=)))dd()))dQ(U))) (z). We then estimate the density ratio dP/dQ with parametric model (2), by maximizing the lower bound of Dss( [17]:
C (+) = Ef
- of (f(ww)))))
(4)
where (a) 2 0 parametrized by a neural network whose outputs are forced to non-negative via activation functions, and /* denotes convex conjugate. In particular, we obtain an estimate (g.(s, a) of don /doff by considering probability distributions P and Q with densities dos and doff, respectively. In practice, we sample from 8 for the first term in (4), and from Bott for the latter. For more stable density ratio estimates, we apply normaliaaiiinnnnnn [18] to the estimated density ratios over gott similar to Sinha et al. [16]. More details can be found in Section D.
4.2 Pessimistic Q-Ensemble
In order to mitigate distribution shift more effectively, we leverage multiple pessimistically trained Q-functions, We consider an ensemble of N CQL agents pre-trained via update rules (2, 3), i.c., [Qe, To. Ji=1. where 0; and o; denote the parameters of the i-th agent's 's Q-function and policy, respectively. Then we use the ensemble of actor-critic agents whose Q-function and policy are defined as follows:
N Σ Qo (-(11)] -N N
N Me Me,(s) (+),
N E (6) + (Z) - n2(+))
Qo = N
(()()))))((()))))))))))))))) (5)
N
where 0 = {0.) and { Note that the policy is simply modeled Gaussian with mean and variance of the Gaussian mixture policy N Ei= To.- In turn, O and @ are updated via update rules (1) and (2), respectively, during fine-tuning.
By using pessimistic Q-function, the agent remains pessimistic with regard to the unseen actions at states encountered online during initial fine-tuning. This is because during early fine-tuning, states resemble those present in the offline dataset, and Q-function generalizes to these states. As we show in our experiments, this protects the good initial policy from severe bootstrap error. And by leverag- ing multiple pessimistically trained Q-functions, we obtain a more high-resolution pessimism about the unseen data regime. That is, when individual Q-function may erroncously have high values for unseen samples, Q-ensemble is more robust to these individual errors, and more reliably assigns lower values to unseen samples. Computational overhead of ensemble is discussed in Section B.
5 Related work
Offiine RL. Offline RL algorithms aim to train RL agents exclusively with pre-collected datasets. To address the stte-cooniiional action distribution shift, prior methods (a) explicitly constrain the policy to be closed to the behavior policy [2, 3, 4, 5, 19], or (b) train pessimistic value functions [7, 8, 9]. In particular, CQL [9] was used to learn various robotic manipulation tasks [10]. We also build on CQL, so as to leverage pessimism regarding data encountered online during fine-tuning.
Online RL with offline datasets. Several works have explored employing offline datasets for online RL to improve sample efficiency. Some assume access demonstration data [20, 21, 22, 23, 24], which is limited in that they assume optimality of the dataset. To overcome this, Nair et al. [25] proposed Advantage Weighted Actor Critic (AWAC), which performs regularized policy updates so that the policy stays close to the observed data during both offline and online phases. We instead advocate adopting pessimistic initialization, such that may prevent overoptimism and bootstrap error in the initial online phase, and lift such pessimism once unnecessary, as more online samples gathered. Some recent works extract behavior primitives from offline data, then learn to compose them online [26, 27, 28]. It would be interesting to apply our method in these setups.
Experience replay. The idea of retrieving important samples for RL was introduced in Schaul et al. [29], where they prioritize samples with high temporal-differeee
error. The work closest to ours is Sinha et al. [16], which utilizes the density ratios between off-policy and near-on-policy state-action distributions as importance weights for policy evaluation. Our approach differs in that we utilize density ratios for retrieving relevant samples from the offline dataset.
5

---

Ours
AWAC
BCQ
SAC-R
SAC
hat Kiieeahh vancom
hat wwweeaaaeneedmmm
hu www.eeiianeee......m
1400
9000
13303
10030
KOCC
0101
2010
2003
120
3.00
2500
Tesp Inspee-maduu
3000
hepper-ered Sarpper-rdddam--gg
4000
2.00
250C
2200
1.03
100
200
150
Driforment
-000
2eleaaannn elamm
5000
wal mal-madiuum
walferril Lnm-naplay
5000
wi wrallae mel----eauu----- sogurt
200
3000
1000
100
Endermment Steps
Ennorment Crre
Eriorrent
Figure 3: Performance D4RL [12] MuJoCo locomotion tasks during online fine-tuning. The solid lines and shaded regions represent mean and standard deviation, respectively, across four runs.
Ensemble methods. In the context of model-free RL, ensemble methods have been studied for addressing Q-function's overestimation bias [30, 31, 32, 33, 34], for better exploration [35, 36, 37], or for reducing bootstrap error propagation [38]. The closest to our approach is Anschel et al. [32] that stabilizes Q-learning by using the average of previously learned Q-values the target Q- value. While prior works mostly focus on online RL and estimate the ground-trute Q-functions, we leverage ensemble of pessimistically pre- trained Q-functions for safe offline-to-onling RL.
6 Experiments
We designed our experiments answer the following questions:
How does our method compare to existing offline-to-online RL methods and an online RL method that learns from scratch (see Figure 3)?
Can our balanced replay scheme locate offline samples relevant to the current policy (see Fig-
ure 4a) and improve the fine-tuning performance by utilizing these samples (see Figure 4c)?
Can our pessimistic Q-ensemble scheme discriminate unseen actions (see Figure 4b) and suc- cessfully stabilize the fine-tuning procedure by mitigating distribution shift (see Figure 4d)?
Does our method scale to vision-based robotic manipulation tasks (see Figure 5)?
6.1 Locomotion Tasks
Setup. We consider MuJoCo [11] locomotion tasks, i.e., halfcheetah, hopper, and walker2d, from the D4RL benchmark suite [12]. To demonstrate the applicability of our method on var- ious suboptimal datasets, we use four dataset types: random, medium, medium-replay and medium-expens Specificallyy, random and medium datasets contain samples collected by a ran- dom policy and medium-levas policy, respectively. medium-replay datasets contain all samples encountered while training a medium-level agent from scratch, and medium medium-expert datasets con- tain samples collected by both medium-level and expert-level policies. For our method, we ensemble size
More experimental details are provided in Section D.
Comparative Evaluation. We consider the methods outlined below baselines for comparative evaluation. For fair comparison, we applied ensemble to all baselines except SAC-ft, since the results for SAC-ft with ensemble can be found in the ablation studies (see Figure 4c).
Advantage Weighted Actor Critie [AWAC; 25]: an offline-to-online RI. method that trains the policy imitate actions with high advantage estimates.

---

Nolle/2d
walkor25
04-
Ortine buloreet
5
00-
AUDOO
2005
4000
2--
02
10c
100
Covtreeen Dapa
Conforment
(a) Buffer analysis
(b) Ensemble analysis
(c) Buffer ablation
(d) Ensemble ablation
Figure 4: (a) Proportion of offline samples used for updates as the agent is fine-tuned online, for walker2d tasks. (b) AUROC (%) over the course of fine-tuning
um expert, where the Q-function is interpreted as a binary classifier that classifies a given action pair (s, a) as either seen pair (s, Ozeen) or an unseen pair (s, Ouniforn), for state encountered online. Pes- simistic Q-ensemble shows a stronger discriminative ability. (c) Performance on wwekeerrrrrraaee with and without balanced experience replay. We consider two setups where balanced experience replay is not used: (i) Uniform, where offline and online samples are sampled uniformly from the same buffer for updates, and (ii) Online only, where the offiin agent is fine-tuned using online sam- ples only. (d) Performance on walker2d-random with varying ensemble size N E {1, 5). The solid lines and shaded regions represent mean and standard deviation, respectively, across four runs.
BCQ-ft: Satch-Constrained deep Q-learning [BCQ: 2], is an offline RL method that updates policy by modeling the behavior policy using a conditional VAE [39]. We extend BCQ to the online fine-tuning setup by applying the same update rules as offline training.
SAC-ft: Starting from a CQL agent trained via (3, 2). we fine-tune the agent via SAC updates
(1, 2). Justification for excluding the CQL regularization term from (3) during fine-tuning can be found in Section E.
SAC: a SAC agent trained from scratch via (I, 2), i.e., the agent has no access to the offline dataset. This baseline highlights the benefit of offline-to-onlin RL, as opposed to fully online RL, in terms of sample efficiency.
Figure shows the performances of our method and baseline methods considered during the online RL phase. In most tasks, our method outperforms the baseline methods in terms of both sample- efficiency and final performance. In particular, our method significantly outperforms SAC-ft, which shows that balanced replay and pessimistic Qeenseblee are indeed essential.
We also emphasize that our method performs consistently well across all tasks, while the perfor- mances of AWAC and BCQ-ft are highly dependent on the quality of the offline dataset. For exam- ple, we observe that AWAC and BCQ-ft show competitive performances in tasks where the datasets are generated by high-quality policies, i.e., medium iedimeeeepert tasks, but perform worse than SAC on random tasks. This is because AWAC and BCQ-ft employ the same regularized, pessimistic update rule for offline and online setups alike, either explicitly (BCQ-ft) or implicitly (AWAC), which leads to slow fine-tuning. Our method instead relies on pessimistic initialization, and hence enjoys much faster fine-tuning, while not sacrificing the initial training stability.
Balanced replay analysis. To investigate the effectiveness of our balanced experience replay scheme for locating near-on-police samples in the offline dataset, we report the ratios of offline samples used for updates fine-tuning proceeds. Figure 4a shows that for the random task, offline samples quickly become obsolete, as they quickly become irrelevant to the policy being fine-tuned. However, for the are relevant to the current policy, hence are replayed throughout the online training. This shows that our balanced replay scheme is capable of utilizing offline samples only when appropriate.
task, offline samples include useful expert-level transitions that
Q-ensemble analysis. We quantitatively demonstrate that pessimistic Q-ensemble indeed provides more discriminative value estimates, i.e., having distinguishably lower Q-values for unseen actions than for seen actions. In particular, we consider a medium medium-expedr expert dataset, where the offline data distribution is narrow, and the near-optimal offline policy can be brittle. Let Dipai := {(s, a, ) be the samples collected online up until timestep T. We construct a "fake" dataset by replacing the actions in Drsal with random actions, i.e., Diske confidence value for classifying real and fake transitions, we measure the area under ROC (AUROC) curve values over the course of fine-tuning As seen in Figure 4b, Qeensemble demonstrates superior discriminative ability, which leads to stable fine-tuning.
{(st, auiis) (T T Interpreting ((s, a) as the
7

---

dosed
grass blocked
celterart ne nanden vardes
0.8
0.8
COL- Oriiee unb
00L On only
O Ontine only
Oure
Ouro
Ouro
00
200
630
900
830
1900
0.
200
480
630
690
1006
0.0
200
400
900
680
900
200
400
1200
400
1256
(a) Performance
(b) Buffer analysis
Figure 5: (a) Fine-tuning performance for robotic manipulation tasks considered. (b) Proportion of random data used for during fine-tuning decreases over time. The solid lines and shaded regions represent mean and standard deviation, respectively, across eight runs.
Ablation studies. Figure 4c shows that balanced replay improves fine-tuning performance by sam- pling near-on-policy transitions. On the other hand, two other naïve sampling schemes (a) Uni- form, where offline and online samples are sampled uniformly from the same buffer, and (b) Online only, where the offline agent is fined-tuned using online samples exclusively suffer from slow and unstable improvement, even with pessimistic Q-ensemble. This shows that balanced replay is crucial for reducing the harmful effects of distribution shift.
Also, Figure 4d shows that fine-tuning performance improves as the ensemble size N increases, which shows that larger ensemble size provides higher-resolution pessimism, leading to more stable policy updates. Ablation studies for all tasks can be found in Section E.
6.2 Robotic Manipulation Tasks
Setup. We consider three sparse-reward pixel-based manipulation tasks from Singh et al. [10]: (1) pick-place: pick an object and put it the tray; (2) grasp-closedddraas drawer:: grasp an object in the initially closed bottom drawer; (3) grasp-blocked -draarrrrdedd.rrr grasp an object in the initially closed bottom drawer, where the initially open top drawer blocks the handle for the bottom drawer. Episode lengths for the tasks are 40, 50, 80, respectively.
The original dataset [10] for each task consists of scripted exploratory policy rollouts. For example, for pi sick-place the dataset contains scripted pick attempts and place attempts. However, it is rarely the case that logged data 'in the wild' contains such structured, high-quality transitions only. We consider a more realistic setup where the dataset also includes undirected, exploratory samples we replace a subset of the original dataset with uniform random policy rollouts. Note that random policy rollouts are common in robotic tasks [40, 41]. We used ensemble size N = 4 for our method. More details about the tasks and dataset construction are provided in Section D.2.
Comparative Evaluation. We compare our method with the method considered in Singh et al. [10), namely, CQL fine-tuning with online samples only. CQL-ft fails to solve the task in some of the seeds, resulting in high variance as seen in Figure 5a. This is because CQL shows inconsistent offline performance across random seeds due to such factors difficulty of training on mixture data [12]. instability of offline agents over stopping point of training [42], and sparsity of rewards. With no access to peseudo-)expent offline data and due heavy regularization of CQL, such CQL agents hardly improve. Meanwhile, our method consistently learns to perform the task within a reasonable amount of additional environment interaction (40K to 80K steps).
Buffer analysis. We analyze whether balanced replay scales to image-based based robotic tasks. As seen in Figure 5b, without any privileged information, balanced replay automatically selects relevant offline samples for updates, while filtering out task-irrelevant, random data as fine-tuning proceeds.
7 Conclusion
In this paper, we identify state-action distribution shift as the major obstacle in offline-to-online RL. To address this, we present a simple framework that incorporates (I) balanced experience replay scheme, and (2) a pessimistic Q-ensemble scheme. Our experiments show that the proposed method performs well across many continuous control robotic tasks, including locomotion and manipulation tasks. We expect our method to enable more sample-efficient training of robotic agents by leveraging offline samples both for offline and online learning. We also believe our method could prove to be useful for other relevant topics such as scalable RL [43] and RL safety [44].
8

---

Acknowledgments
This work was supported by Microsoft and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gDV) (N..2019-0-00075 Artificial Intelligence Graduate School Program(KAIST)), We would like to thank anonymous viewers for providing helpful feedbacks and suggestions in improving our paper.
References
[1] Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[2] Fujimoto, D. Meger, and D. Precup. Off-policcy deep reinforcement learning without explo- ration. In International Conference on Machine Learning. 2019.
[3] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019. 2019.
[4] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arviv; 1911.11361, 2019.
[5] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for of- fline reinforcement learning. In International Conference on Learning Representations, 2020.
[6] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforce- ment learning. International Conference on Machine Learning, 2020.
[7] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model- based offline rein- forcement learning. In Advances Neural Information Processing Systems, 2020.
[8] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020.
[9] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. In Advances in Neural Information Processing Systems, 2020.
[10] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and Levine. Cog: Connecting new skills to
past experience with offline reinforcement learning. In Conference on Robot Learning, 2020.
[11] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
[12] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint Xii.:2004.002199,.. 2020.
[13] T. Haarojja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning, 2018
[14] D. P. Kingma and M. Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[15] т. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXi::2007.0005. 2020.
[16] S. Sinha, J. Song, A. Garg, and S. Ermon. Experience replay with likelihood-free importance weights, 2021. URL
Sf-a.
[17] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances Neural Information Processing Systems, 2008.

---

[18] W. G. Cochran. Sampling techniques. John Wiley & Sons, 2007.
[19] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. Emaq: Expected-mas max q- g-learning operator for simple yet effective offline and online rl. arXiv preprint WIXiv:2007.1109/ 2020.
[20] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Learning attractor landscapes for learning motor primitives. Advances in Neural Information Processing Systems, 2003.
[21] B. Kim, A.-m. Farahmand, J. Pineau, and D. Precup. Learning from limited demonstrations.
In Advances in Neural Information Processing Systems, 2013.
[22] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Robotics: Science and Systems, 2018.
[23] H. Zhu, A. Gupta, A. Rajeswaran, Levine, and V. Kumar. Dexterous manipulation with deep reinforeement learning: Efficient, general, and low-cost. In International Conference Robotics and Automation, 2019.
[24] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.088/1 2017.
[25] A. Nair, M. Dalal, A. Gupta, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2021.
[26] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors.
In Conference on Robot Learning, 2020.
[27] A. Ajay, A. Kumar, P. Agrawal, Levine, and O. Nachum. (OPAL): Offline primitive discov- ery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021. URL https http:/:///pp/eei/eeeeee
mett/ffrummidddVV9G....
[28] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinchart, and S. Levine. Parrot: Data-driven behavioral
priors for reinforcement learning, In International Conference on Learning Representations, 2021. URL https: / //penreviiee
[29] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay, In International Conference on Learning Representations, 2016.
[30] H. V. Hasselt. Double q-learning. In Advances in Neural Information Processing Systems, 2010.
[31] H. v. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In AAAI Conference Artificial Intelligence, 2016.
[32] O. Anschel, N. Baram, and N. Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International Conference on Machine Learning, 2017.
[33] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, 2018.
[34] Q Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
[35] I. Osband, C. Blundell, Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn, In Advances in Neural Information Processing Systems, 2016.
[36] R. Y. Chen, S. Sidor, P. Abbeel, and Schulman. Ucb exploration via g-ensembles. arXiv preprint arXiv: 1706:0000...222 2017.
[37] K. Lee, M. Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning deep reinforcement learning. International Conference on Machine Learning, pages 6131-6141. PMLR, 2021.
10

---

[38] K. Lee, M. Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.04938. 2020.
[39] K. Sohn, H. Lee, and Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, 2015.
[40] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793, IEEE, 2017.
[41] Ehert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for dision-based robotic control. arXiv preprint arXiv:1812.00568, 2018.
[42] S. Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.006 06860, 2021.
[43] D. Kalashnikov, A. Irpan, Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. In Conference on Robot Learning. 2018.
[44] Garcia and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1488) 2015.
[45] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[46] Y. Burda, R. Grosse, and R. Salakhuudinov. Importance weighted autoencoders. In Interna- tional Conference on Learning Representations, 2016.
[47] Matsushima, Furuta, Matsuo, O. Nachum, and Gu. Deployment-efficient reinforce- ment learning via model-based offline optimization. arXiv preprint
2020.


Reviews:
[{"review_id": "q0MhiyM-Lrl", "reviewer": "Reviewer_Jt9W", "summary": "This paper identifies state-action distribution shift as the major problem in offline-to-online RL, and proposes two mechanisms, balanced replay buffer and pessimistic Q-ensemble to solve it. Through well-organized experiments, the paper shows that the balance experience replay can effectively balance the usage of online data and offline data and the pessimistic Q-ensemble is able to mitigate distributions shift.  The proposed method obtains good empirical results on D4RL Mujoco locomotion tasks and vision-based robotic manipulation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The motivation of this paper is strong and the proposed method effectively solves the identified state-action distribution shift problem. \n2. The empirical results are good. It outperforms previous methods in most cases. \n3. The paper is well written and easy to follow. \n4. The experiments are well-designed and show the benefits of the proposed methods.\n\nWeaknesses: \n1. In section 3.3, the author compares the choice of CQL and FQE and shows that the CQL is more suitable for offline-to-online RL. But according to the CQL paper, for many offline datasets, directly using non-pessimistic Q-function suffers from distribution shift and can’t learn well during offline training. Thus, I think it will be good to compare CQL with another offline RL method, e.g. BCQ etc. \n2. In section 4.1, the training details are not very clear for me. For example, the author uses P, Q to represent d^{off} and d^{on}, thus the density ratio w(s,a):=d^{on}(s,a)/d^{off}(s,a) = Q/P, but in the last sentence of page 4, the author uses ”estimate the density ratio dP/dQ with a parametric model w_{phi}(x)”, which is inconsistent.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation is \"Weak Accept\".\n\nThis paper identifies state-action distribution shift as the major problem in offline-to-online RL, and proposes two mechanisms, balanced replay buffer and pessimistic Q-ensemble to solve it. The motivation of this paper is strong and the proposed method effectively solves the identified state-action distribution shift problem. The proposed method obtains good empirical results on D4RL Mujoco locomotion tasks and vision-based robotic manipulation tasks. But the description of the training details can be more clear. Thus, my recommendation is \"Weak Accept\".", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. In section 3.3, the author compares the choice of CQL and FQE and shows that the CQL is more suitable for offline-to-online RL. But according to the CQL paper, for many offline datasets, directly using non-pessimistic Q-function suffers from distribution shift and can’t learn well during offline training. Thus if possible, I think it will be good to compare CQL with another offline RL method, e.g. BCQ etc. \n\n2. In section 4.1, the training details are not very clear for me. For example, the author uses P, Q to represent d^{off} and d^{on}, thus the density ratio w(s,a):=d^{on}(s,a)/d^{off}(s,a) = Q/P, but in the last sentence of page 4, the author uses ”estimate the density ratio dP/dQ with a parametric model w_{phi}(x)”, which is inconsistent. "}}, {"review_id": "JpuaAIo7-iA", "reviewer": "Reviewer_yjtU", "summary": "The paper presents a method for off-line to on-line reinforcement learning. First, given a dataset of behavior in a task, an ensemble of \"pessimistic\" Q-functions and policies is trained off-line where the Q-function is learned conservatively to avoid over-estimation of Q-values in out of distribution state-action samples. Subsequently, on-line fine-tuning is done using a Balanced Experience Replay. This replay buffer consists of old samples from the off-line learning phase and new samples collected during on-line learning. Transitions are sampled based on their \"online-ness\", a value proportional to the density ratio between on-line samples and off-line samples. The further traning progresses and moves away from the initial data, the fewer off-line samples are chosen to update the policy.\nThe method is evaluated on simulated locomotion tasks from open-ai gym, as well as vision based manipulation tasks. It performs especially well in terms of learning speed if the intial policy is random.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n- The overall method seems very intuitive and well motivated\n- The efficacy of each component is shown via ablation studies\n\nWeaknesses:\n- Especially on halfcheetah, the asymptotic performance seems to stay well below what SAC achieves alone (12k vs 15k). An intuition why this is the case would be interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "First of all, I am not an expert in off-line RL, so I'm not familiar with all the work that is out there on the topic. Most of the parts like CQL, ensembles, and prioritzed replay based on likelihoods is not entirely new but the parts are modified nicely to fit the off-line to on-line RL scheme and put together well and results look promising. Therefore, I would recommend accept but put my confidence score rather low.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- I don't fully get the definition of the behavior policy in line 69. It looks like it is the number of times an action a_0 is taken in state s_0 divided by how often state s_0 occurs in the replay buffer. But, given states and actions are continuous, how often will this be different from 1?\n- Why is the performance of half cheetah worse, if the policy is initially trained with the medium dataset?\n- Why does the performance using the medium-expert dataset on hopper and walker2d not reach the performance of AWAC and partially BCQ-ft?\n- Is there a difference in the entropy of the policy after off-line training with a random, medium or expert data set? For example does the policy not explore enough if trained on an expert data set?\n- Why are the manipulation tasks only compared to [10]?\n\n"}}, {"review_id": "57Ty1CbRLR", "reviewer": "Reviewer_rPXZ", "summary": "This main contribution of this paper is a balanced replay scheme to balance the offline and online data. It uses an offline policy evaluation method to compute the importance sampling ratio between offline samples and online samples, which is further used to prioritize experience replay of offline samples. The motivation is clear and experimental results are promising. Overall, it's a good paper. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n  (1) The idea of Leveraging offline samples which relate to the current policy to boost online policy evaluation and improvement is reasonable since offline dataset may consist of samples with different styles and quality, some of which may harm the currently learned policy, let alone the policy improvement.\n  (2) Extensive experimental results demonstrate the effectiveness of the proposed balanced replay scheme.\n\n\nWeaknesses:\n(1) Though the authors claim that their method can prevent the policy from being destroyed by online OOD samples, we can still observe a severe performance drop at the early period of online interaction in most cases, why?\n(2) While pessimism protects the policy from severe bootstrapping error,  it may also prevent the online policy from efficiently exploring the unknown environment (in fact, there have been many works using pessimism for online exploration). In other words, the online policy may be too conservative for online policy improvement. I think it's necessary to provide such ablation studies telling if pessimism is helpful for the ultimate online policy performance", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this is a good paper. But If the authors could provide with more experiments to clarify the two weaknesses, I will raise my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "As mentioned in \"Strengths and Weaknesses\".", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "4Cd29BPLjs8", "reviewer": "Reviewer_SrrQ", "summary": "The authors attempted to address the challenge posed by distribution shift to fine tuning policies learned through offline RL. The proposed solution includes two parts: (1) a method called “balanced replay” that uses a density approximation of how similar the old offline samples are to the new online samples and accordingly determines how offline and online samples in the replay buffer should be mixed during fine tuning; and (2) an ensemble of “pessimistic” Q functions (trained with conservative deep Q learning) to further strengthen the stability of the fine tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\ni.\tThe case for the need to address distribution shift in fine tuning policies learned through offline RL is well motivated: Section 3 and Figure 1 are well structured and well presented. Kudos to the authors!\n\nii.\tBalanced replay is a relatively novel approach.\n\niii.\tThe ensemble of pessimistic Q functions is empirically effective.\n\nWeaknesses\n\ni.\tThe relationship between the two parts of the solution is unclear. The pessimistic ensemble seems to be an extra add-on. It seems to be an afterthought for better performance  rather than an integrated part of an architecturally cohesive solution.\n\nii.\tDiscussion of the ensemble, about why and how it works, stopped at vague allusions to “resolution” without detail. (Technical Issue v below).\n\niii.\tThe treatment of BCQ as a baseline seems quite unfair. See Technical Issues (vi) below for more detail.\n\niv.\tThe effectiveness of the method in the real world is doubtful. See Technical Issues (ix and x) below for more detail.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "a.\tThere is an interesting idea that is worth discussing by the RL community: “balanced replay”. Also, the analysis of distribution shift through a careful study of one concrete case is quite helpful and illuminating.\n\nb.\tThe relationship between the pessimistic ensemble and distribution shift is not clearly established. From the experiments, it seems that the ensemble is mainly contributing towards stabilizing training. Overall, the ensemble part seems to be a separate proposal.\n\nc.\tThe effectiveness of the approach for real-world robotics is doubtful. See below in \"Issues\".\n\nd.\tThere are multiple technical issues that need to be fixed, addressed or discussed. See below in “Issues”.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Technical issues\n\ni.\tSection 3 and Figure 1: Can the discussion and presentation here be used to motivate the necessity of the use of ensemble in the solution resolution? How should the discussion and Figure 1 (or experiments therein) be changed?\n\nii.\tShould we also consider offline RL approaches (e.g. model-based) that learn the structure more explicitly before fine-tuning? How would that affect the distribution shift situation? Should we also consider model-based baselines? (No need to actually include such baselines. Just theoretically speaking.)\n\niii.\tHow does our analysis (e.g. in Section 3) inform about data collection & selection for the offline phase itself? Should we also compare with approaches that do data collection differently for the offline phase to improve fine-tuning? Note that the injection of uniform random policy rollouts (Line 291) seems to be an example. Should we also consider this in our theoretical treatment at least?\n\niv.\tLine 155: In the design space here, other than d^on / d^off used in the paper, d^on / d^(on union off) also seems a sensible choice that might have the added benefit of trending towards 1 asymptotically.\n\nv.\t“Ensemble methods” (paragraph starting on Line 207): it is not clear that “obtain a high-resolution pessimism” is mutually exclusive. I.e. it is not clear “instead” is warranted.\n\nvi.\tLine 236-237: Is it justified to apply the same update rules as offline training when using BCQ as a baseline? Why not transfer/fine-tune using e.g. SAC? That seems to be a much more fair comparison. This is so especially in light of Figure 3 in which BCQ is barely learning during online fine-tuning. But BCQ was not designed for fine tuning. To do justice to BCQ, the comparison should presumably be about the quality of policy initialization.\n\nvii.\tLine 268: “Interpreting Q(s, a) as the confidence value for classifying real and fake transitions” -- How is this justifiable at all? Are we making a category mistake here? Action value and confidence about real vs. fake are radically different kinds of things.\n\nviii.\tFigure 5: “eight runs” and everywhere else is 4 runs. Why do we need to go up to eight runs here? What’s the reason behind this shift from the practice elsewhere in the paper.\n\nix.\tFigure 13 and 14: the main expected effect or phenomenon -- stable and efficient fine-tuning -- seems highly unstable. All methods see big drops during the first 100 episodes. This raises questions about whether the method is going to be truly effective when used with physical robots. Of course, an alternative conclusion here could be that the method is good but that these environments are inadequate for testing the new architecture. It seems that adequate discussion of these curves is missing.\n\nx.\tLine 291: “we replace a subset of the original dataset with uniform random policy rollouts.” Is this justified? In real world applications, data are more likely from medium-level policies, less likely from uniform random policies. In fact, it could be the case that in the real world it’s much harder to obtain data with uniform random policy rollouts. That this sort of uniform random rollouts are needed at all seems worrying, especially if offline RL is supposed to use real world data.\n\nWriting issues\n\ni.\tLine 83-84 “dataset of random halfcheetah transitions”. The notion of a “random dataset”  should be more appropriately introduced. More detail was provided later around Line 123. But it is more desirable if the notion is explained before it is used.\n\nii.\tLine 131-133: It may be more helpful to say right there and then positively and explicitly that that’s what the balanced replay approach is intended to do. That wrap up Section 3 more adequately.\n\niii.\tFigure 2: Diagram is good. But it is misleading to have labels of the parts of the diagram in shaded boxes. Consider remove the boxes around “Pessimistic Online Training”, “Environment Interaction” and “Balanced Replay”.\n\niv.\tLine 196: the acronym AWAC should be introduced here rather than later on Line 233.\n\nv.\tFigure 1 and Figure 5: It will be helpful and make the figures more self-contained if the environments used in the experiments are explicitly identified in the caption.\n\nvi.\tFigure 13: should “CQL objective” in the caption be “SAC objective”?\n"}}]

Author rebuttal:

Meta-review: The reviewers generally appreciate the strengths of the paper, with a well-motivated and mostly novel approach to offline RL. I am glad the authors were active during the rebuttal period and were able to address many reviewer concerns. I note that Reviewer Jt9W still has one concern about the method; namely, I advise the authors to add a paragraph or two conceding and elaborating on potential shortcomings of their proposed approach, including the potential instability issues.
Decision: Accept (Poster)