Addressing Distribution Shift in Online Reinforcement Learning with Offline Datasets
Authors: Seunghyun Lee
Younggyo Seo
Kimin Lee
Pieter Abbeel
Jinwoo Shin
Venue:  (2021)
Keywords: reinforcement learning
offline reinforcement learning
control
distribution shift

Abstract: Recent progress in offline reinforcement learning (RL) has made it possible to train strong RL agents from previously-collected, static datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to improve such offline RL agents with further online interaction. As it turns out, fine-tuning offline RL agents is a non-trivial challenge, due to distribution shift – the agent encounters out-of-distribution samples during online interaction, which may cause bootstrapping error in Q-learning and instability during fine-tuning. In order to address the issue, we present a simple yet effective framework, which incorporates a balanced replay scheme and an ensemble distillation scheme. First, we propose to keep separate offline and online replay buffers, and carefully balance the number of samples from each buffer during updates. By utilizing samples from a wider distribution, i.e., both online and offline samples, we stabilize the Q-learning. Next, we present an ensemble distillation scheme, where we train an ensemble of independent actor-critic agents, then distill the policies into a single policy. In turn, we improve the policy using the Q-ensemble during fine-tuning, which allows the policy updates to be more robust to error in each individual Q-function. We demonstrate the superiority of our method on MuJoCo datasets from the recently proposed D4RL benchmark suite.


Under review as a conference paper at ICLR 2021
ADDRESSING DISTRIBUTION SHIFT IN REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent progress in offline reinforcement learning (RL) has made it possible train strong RL agents from pending on the quality of the trained agents and the application being considered, it is often desirable to improve such offline RL agents with further online interac- tion. As turns out, fine-tuning offline RL agents is a non-trivial challenge, due to distribution shift the agent encounters out-of cut-of-diisribution samples during on- line interaction, which may cause bootstrapping error in Q-learning and instability during fine-tuning. In order to address the issue, present simple yet effective framework, which incorporates a balanced replay scheme and an ensemble distil- lation scheme. First, we propose keep separate offline and online replay buffers, and carefully balance the number of samples from each buffer during updates. By utilizing samples from a wider distribution, i.e., both online and offline samples, we stabilize the Q-learning. Next, we present an ensemble distillation scheme, where we train an ensemble of independent actor-critic agents, then distill the policies into single policy. In turn, we improve the policy using the Q-ensemble during fine-tuning, which allows the policy updates to be more robust to error in each individual Q-function. We demonstrate the superiority of our method on MuJoCo datasets from the recently proposed D4RL benchmark suite.
static datasets. However, de-
INTRODUCTION
Offline reinforcement learning (RL), the task of training a sequential decision-makee making agent with static offline dataset, holds the promise of a data-driven approach to reinforcement learning, thereby bypassing the laborious and often dangerous process of sample collection (Levine al., 2020). Accordingly, various offline RL. methods have been developed, some of which are often capable of training agents that are more performant than the behavior policy (Fujimoto al., 2019; Kumar et al., 2019; Wu et al., 2019; Siegel et al., 2020; Agarwal et al., 2020; Kidambi et al., 2020; Yu et al., 2020; Kumar et al., 2020). However, agents trained via offline RL methods may be suboptimal, for (a) the dataset they were trained may only contain suboptimal data; and (b) environment in which they are deployed may be different from the environment in which dataset was generated. This necessitates online fine-tuning procedure, where the agent improves by interacting with the environment and gathering additional information.
Fine-tuning offline RL agent, however, poses certain challenges. For example, Nair et al. (2020) pointed out that offline RL algorithms based on modeling the behavior policy (Fujimoto et al., 2019;
Kumar al., 2019; Wu et al., 2019; Siegel et al., 2020) not amenable fine-tuning. This is because such methods require sampling actions from the modeled behavior policy for updating the agent, and fine-tuning such generative model online for reliable sample generation a challenging task. On the other hand, more recent state-of satt--of-the-art offline RL algorithm, conservative Q-lcarning (CQL; Kumar al., 2020), does not require explicit behavior modeling, and one might expect CQL to be amenable to fine-tuning. However, we make an observation that fine-tuning a CQL agent a non-trivial task, due to the so-called distribution shift problem - the agent encounters out-of- distribution samples, and in turn loses its good initial policy from offline RL training. This can be attributed to the bootstrapping error, i.e., error introduced when Q-function is updated with an inaccurate target value evaluated at unfamiliar states and actions. Such initial training instability is severe limitation, given the appeal of offline RL lies in safe deployment at test time, and losing such safety guarantees directly conflicts with the goal of offline RI.

---

Under review as a conference paper at ICLR 2021
Contribution. In this paper, first demonstrate that fine-tuning CQL agent may lead to unstable training due to distribution shift (see Section for more details). To handle this issue, we introduce simple yet effective framework, which incorporates balanced replay scheme and ensemble distillation scheme. Specifically, we propose to maintain two separate replay buffers for offline and online samples, respectively. Then modulate the sampling ratio between the two, in order bal- ance the effects of (a) widening the data distribution the agent sees (offline data), and (b) exploiting the environment feedback (online data). Furthermore, we propose an ensemble distillation scheme: first, we learn an ensemble of independent CQL agents, then distill the multiple policies into a sin- gle policy. During fine-tuning, we improve the policy using the mean of Q-functions, so that policy updates are more robust to error in each individual Q-function.
In our experiments, we demonstrate the strength of our method based on MuJoCo (Todorov et al., 2012) datasets from the D4RL (Fu et al., 2020) benchmark suite. Our goal is to achieve both (a) strong initial performance as well as maintaining it during the initial training phase, and (b) better sample-efficiency. For evaluation, we measure the final performance and sample-efficinnc of RL agents throughout the fine-tuning procedure. We demonstrate that our method achieves stable train- ing, while significantly outperforming all baseline methods considered, including BCQ (Fujimoto et al., 2019) and AWAC (Nair et al., 2020), both in terms of final performance and sample-efficieng
2 BACKGROUND
Reinforcement learning. We consider the standard RL framework, where an agent interacts with the environment as to maximize the expected total return. More formally, at each timestep t, the agent observes a state 8t, and performs an action O according to its policy л. The environment rewards the agent with rı, then transitions to the next state 8141- The agent's objective is to maximize the expected return Ex, [>> >*r,], where 7 E [0, 1) is the discount factor. In this work, we mainly consider off-policy RL algorithms, class of algorithms that can, in principle, train an agent with samples generated by any behavior policy. These algorithms are well-suited for fine-tuning pre- trained RL agent, for they can leverage both offline and online samples. Offline RL algorithms are off-policy RL algorithms that only utilize static datasets for training. Here, we introduce an off-policy RI. algorithm and an offline RL algorithm we build on in this work.
Soft Actor-Critic, SAC (Haarnoja et al., 2018) is an off-policy actor-critic algorithm that learns soft Q-function Q0(s,a) a) parameterized by 0 and stochastic policy Te modeled as a Gaussian with its parameters の. To update parameters 0 and o. SAC alternates between a soft policy evaluation and a soft policy improvement. During soft policy evaluation, soft Q-function parameter is updated to minimize the soft Bellman residual:
Cetitic (0) = ET. (T,,
(1)
< (Te. 0) = (Qo(se, at) (re + 7 BB ( (0 ax+1) - a log To (a ((t+11))))))))))))))))
()))
(2)
where (St, az, Te, St+1) is transition, B is the replay buffer, Ø is the moving average of soft Q-function parameter 0, and a is the temperature parameter. During soft policy improvement, policy parameter @ is updated to minimize the following objective:
actor (Ø) = EE,, (( (st, +)]; where (8e, Φ) = Eaiwy fa log n (e) - Qo(s,, 4t)] (3)
CSAC
Conservative Q-Learning, CQL (Kumar et al., 2020) is an offline RL algorithm that learns a lower bound of the Q-function Qo a). in order to prevent extrapolation error value overestimation caused by bootstrapping from duitt--distribution actions. To this end, CQL(H), a variant of CQL, imposes regularization that minimizes the expected Q-value at unseen actions, and maximizes the expected Q-value at seen actions by minimizing the following objective:
1
corl (f)
2 -L
COL
(O)
(4)
[ (O))(0) ET,ND (Qo(s(, - (rr + /Ba 1Q (0((542,44+111))
(5)
CraL (8) = E,,,D
log Σ exp(QQ((5,0)) Earni [Qo a:)l
(6)
2

---

Under review as a conference paper at ICLR 2021
Offine Online
808
Offine
5.500
COLi Uniormm
Online
4.000
CQL- (SAC, Online only)
005-
2.000
0:04
2.000
0:11
0.w1ct C.211" a.sao" 10H10"
fodiloood
(a) L-SNE visualization
(b) Negative log-likelihood
(c) Walker2d-medium
Figure (a) I-SNE visualization of offline and online samples from the wa malkeedddmeee um dataset from the D4RL benchmark (Fu et al., 2020), and online samples collected by CQL agent trained on the same dataset. (b) We visualize the log- leglliklihhod estimates of offline and online samples observed in wa balker walker2dmmmed um task, based
VAE pre-trained the offline dataset. Offline and online samples follow different distributions. (c) Performance on wa 1 kerkk
um task during fine-tuning. We first train CQL agent via (4), and fine-tune it via SAC updates (1), (3), either using both offline and online data drawn uniformly at random (Uniform replay), or using online samples only (Online only).
Eingee (s=s),ama))
where a CLL is the tradeoff' parameter, D the offline dataset, and 7 (st) the empirical behavior policy.
L.eD [(smst)
CHALLENGE: DISTRIBUTION SHIFT
In the context of fine-tuning an offline RL agent, there may exist a distribution shift between offline and online data distribution: an offline RL agent encounters data distributed away from the offline data, as the agent starts interacting with the environment (see Figure Ia for a visualization). More formally, let (s a) dm (s) (a|s) denote the discounted stationary state-action distribution of the policy, where dd denotes the discounted marginal state distribution, and π denotes the policy. Then distribution shift refers to the difference between ps(8, a) and Pr, (8, a), where 3 refers to the behavior policy, and ny refers to the current policy. To demonstrate the existence of distribution shift empirically, report the negative log- eigllikelihiodd estimates of offline and online samples in Figure lb. Specifically, we first split the walker2d-me validation datasets, then train variational autoencoder (VAE; Kingma & Welling, 2014) the training dataset. Then, we estimate the log-likelihoor of online samples and offline samples from the validation dataset, using the importance weight estimator (Burda et al., 2016). We can observe difference between offline and online data distributions, i.c., they follow different distributions.
offline dataset into training and
This shift in turn affects fine-tuning in very complicated manner, for it involves an interplay be- tween actor and critic updates with newly collected -ont---disttibttion samples. To elaborate in more detail, we train an agent offline via state-of-the-ar CQL algorithm in (4), then fine-tune via SAC algorithm in (1) and (3)', either using samples drawn uniformly at random (Uniform replay) as in Nair al. (2020), or using online samples exclusively (Online only). Figure shows how fine-tuning may suffer from instability in both cases. Essentially, this instability occurs due to the shift between offline and online data distribution. In the case of using both offline and online data, the chance of agent seeing online samples for update becomes too low, especially when the offline dataset contains massive amount of data. This prevents timely updates at unfamiliar states encoun- tered online, and may lead to performance degradation seen in Figure Ic. On the other hand, when using online samples exclusively, the agent is exposed to unseen samples only, for which Q function does not provide reliable value estimate. This may lead to bootstrapping error, and hence a dip in performance as seen Figure Cc. This points to leveraging a mixture offline and online samples for fine-tuning, so that the agent can balance the trade-off between them.
tine-tuning with CQL updates (4) leads to slow (if at all) improvement in most setups we consider (see Ap- pendix B for more results). We also provide additional experimental results when using different RI. algorithms than SAC for fine-tuning pre-trained offline CQI. agent in Appendix C.
3

---

Under review as a conference paper at ICLR 2021
Online Early Stage
Online Later Stage
Actor
Actor
Actor
Critic
Critic
Critic
Actor 2
Critic 2
Distilled Actor
Offline
Actor
Offline
Online
Offline
Online
Replay Buffer
Critic N
(a) Balanced replay
(b) Ensemble distillation
Figure 2: Illustrations of our framework. (a) For updating the agent during fine-tuning, we draw balanced number of samples from both offline and online replay buffers initially, but focus exploiting online samples at later training stage. (b) We first train an ensemble of independent offline RL agents with the given offline dataset. Then, we distill the ensemble of policies into single policy, such that the policy may receive more accurate learning signal from the Q-ensemble.
4 BRED: BALANCED REPLAY WITH ENSEMBLE DISTILLATION
In this section, we present BRED: Balanced Replay with Ensemble Distillation, order address the distribution shift between offline and online data distributions. First, introduce separate offline and online replay buffers order to select a balanced mix of samples from them for better Q-function updates. This has the dual advantage of updating Q-function with wide distribution of samples, while making sure that Q-values are updated at novel, unseen states from online interaction. Furthermore, train multiple actor-critic offline RL agents, and distill the policies into single policy. Then, during fine-tuning, we improve the distilled policy via Q-ensemble.
4.1 BALANCING EXPERIENCES FROM ONLINE AND OFFLINE REPLAY BUFFERS
In order to address the distribution shift between offline and online samples, we introduce separate replay buffers for offline and online samples, respectively. Then, update the agent using mix of both offline and online samples. Specifically, at timestep i, we draw B (1- pt samples from the offline replay buffer, and B pe samples from the online replay buffer, where B is the batch size, and the fraction of online samples at timestep t, Pt is defined as follows:
min t4imal)
Pt = 00 + (1 poo)
trinal
where trimal denotes the final timestep of the annealing schedule, and p is the initial fraction of online samples. other words, we initially let the agent see a mix of both offline samples and online samples, and linearly increase the proportion of online samples (see Figure 2a). This has the effect of (a) better Q-fuction updates with wide distribution of both offline and online samples, and (b) eventually exploiting the online samples later when there enough online samples gathered. We empirically show that this simple strategy improves the fine-tuning performance by balancing the experiences from online and offline replay buffers (see Figure 4 for supporting experimental results).
4.2 ENSEMBLE OF OFFLINE RL AGENTS FOR ONLINE FUNE-TUNING
During fine-tuning, each individual Q-function may be inaccurate due to bootstrapping error from unfamiliar online samples. In order to prevent the policy from learning with such erroncous Q- function, we propose to train multiple actor-critic offline RL agents, then distill the policies into single policy. turn, we improve the policy using more accurate value estimate the Q-ensemble.
Formally, consider ensemble of N CQL agents pre pre-trained via (4), i.e., {Qo,, Te, I.N. where 8. and Φ; denote the parameters of the i-th Q-function and i-th policy, respectively. We then distill the ensemble of independent policies (To, J.N into single distilled policy Tays (see Figure 2h) by

---

Under review as a conference paper at ICLR 2021
minimizing the following objective before online interaction:
Cad Ciittill (Opp)) = EaNDD [ll (st) -m(s))) )| + | (st) - (s)))1
(8)
N N (8)) (~()))
1 N
(+x+)
(*)) + n (8:)) - (sss)
(9)
N
1=1
with Mpa. pa denoting the mean and variance of the distilled policy Tay and Jr, 2² the mean and variance of the policy mixture Ei=1 TTee. Then, we update the distilled policy by minimizing the following objective instead of (3):
Lacter (0pa) Bs ( (sr, Spa)],
(10)
N
Σ N Qo. (ot, an)
CP (st: Øpa) = Eng a log o (arls))
(11)
where a is the temperature parameter. As for Q-function updates, we keep a separate target Q- function Qø for each Qø then minimize the loss (I) independently, in order ensure diversity among the Q-functions. Trained this way, the distilled policy receives a more accurate learning signal from the Q-ensemble, although each individual Q-function may be relatively inaccurate. Al- ternatively, one may also consider an ensemble of independent policies, where actions are samples from N ( (ss)) timestep t, and each policy no. is updated with its corresponding Qo, via (3). However, we show that our ensemble distillation scheme allows for a much more stable improvement over using an ensemble of independent policies in more complex environments (see Figure for supporting experimental results).
5 RELATED WORK
Offline RL. Offline RL algorithms aim to learn RL agents exclusively with pre-collected datasets. While Agarwal et al. (2020) showed that conventional off-policy RL algorithms work well for large and diverse datasets, it has been observed that Q-function evaluation can be erroneous due to out-of- distribution actions (Fujimoto al., 2018) otherwise. To address this problem, numerous approaches have been proposed, including policy constraint methods that constrain the policy to stay close to the modeled dataset behavior (Fujimoto et al., 2019; Kumar et al., 2019; Wu al., 2019; Siegel et al., 2020), and conservative Q-learning methods that train the Q-function to be pessimistic in the unseen regime (Kidambi et al., 2020; Yu et al., 2020; Kumar et al., 2020). Our work builds on conservative Q-learning method (Kumar et al., 2020), updates once fine-tuning starts.
to allow fast, unconstrained policy
Online RL with offline datasets. Several prior works have explored employing offline datasets for online RL, which offers the advantage of improving both sample-efficiency and exploration. Some assume access to expert demonstrations, then either improve the behavior-cloned policy via on-policy RL (ljspeert et al., 2003; Kim et al., 2013; Rajeswaran et al., 2018; Zhu et al., 2019), or train a (pre-trained) policy via off-policy RL (Vecerik et al., 2017). These methods are limited, for their success highly depends on the optimality of the dataset. To overcome this. Nair et al. (2020) proposed train the policy so that it imitates actions with high advantage estimates in offline and online setups alike. However, this method relies on regression, hence the learned policy seldom outperforms the behavior policy that was used to generate the offline dataset. We instead advocate an alternative approach of improving the policy based on the generalization ability of Q-function.
Replay buffer. Using replay buffers for managing samples of different categories (e.g., expert and learner, different tasks, offline and online, etc.) has been explored for solving hard exploration problems (Vecerik et al., 2017; Gulcehre et al., 2020), or for continual learning (Rolnick et al., 2019). Our work differs from these, in that we do not assume access optimal data, nor do consider multi-task setups. A similar balanced replay scheme as ours has been used in a robot learning setup (Kalashnikov al., 2018), but their focus is preventing overfitting. We instead study diverse scenarios with datasets of varying suboptimality, where the agent's 's initial fine-tuning procedure may suffer due to distribution shift.
Ensemble methods. In the context of model-free RL, ensemble methods have been studied for addressing Q-function's overestimation bias (Hasselt, 2010; Hasselt et al., 2016; Anschel et al.,
5

---

Under review as a conference paper at ICLR 2021
2017: Fujimoto et al., 2018; Lan al., 2020), for better exploration (Osband al., 2016; Chen et al., 2017: Lee et al., 2020), or for reducing bootstrap error propagation (Lee et al., 2020). The closest to our approach is Anschel al. (2017) that proposed to stabilize Q-learning by using the average of previously learned Q-values as a target Q-value. Our approach instead utilizes an ensemble of independently trained Q-functions, in order maintain diversity among Q-functions.
EXPERIMENTS
We designed our experiments answer the following questions:
How does BRED compare to other fine-tuning methods (see Figure 3)?
How crucial is balanced replay for fine-tuning performance (see Figure 4)?
Can ensemble distillation stabilize the fine-tuning procedure (see Figure 5)?
How does ensemble size affect performance (see Figure 6)?
6.1
SETUPS
Tasks and implementation details. We evaluate BRED on MuJoCo tasks (Todorov et al., 2012), ie., hal Fcheetab ah, hopper, and walker2d, from the D4RL benchmark (Fu et al., 2020). The task to first train an offline RL agent then fine-tuning it We evaluate the strength of each method based two criteria: (a) strong initial performance
well maintaining it, and (b) achiev- ing good sample-efficiency. In order to demonstrate the applicability of our method on various suboptimal datasets, we use four dataset types, i..., random, medi um, medi um- remlayy and medi um expert. Specifically, random and medium um datasets contain samples collected by random policy and medium-level policy, respectively. medi um remlry datasets contain all samples encountered while training medium-level agent from scratch, and medium-expert datasets contain samples collected by both medium-leved and expert-level policies. Following the setup in Kumar et al. (2020), we trained offline RL agents for 1000 epochs without early stopping. For our method, we use = 5 for the number of CQL agents, and (rsnal = 125K for the final step of anncaling schedule in (7). For the initial fraction of online samples p , use the best performing hyperparameter for each dataset by conducting grid search over (0.25, 0.5, 0.75, 0.9). More details provided in Appendix A.
Baselines. We consider the following methods as baselines:
Advantage-Wai Weighted Actor Critic (AWAC; Nair al., 2020): actor-critic scheme for fine- tuning, where the policy is trained to imitate actions with high advantage estimates. Comparison of BRED to AWAC shows the benefit of exploiting the generalization ability of Q-function for policy learning.
BCQ-ft: Batch- Boach-Constrained deep Q-learning (BCQ; Fujimoto et al., 2019), is an offline RL method that updates policy by modeling the behavior policy using conditional VAE (Sohn et al., 2015). We extend BCQ to the online fine-tuning setup by applying the same update rules offline training.
Starting from CQL agent trained via (4), we fine-tune the agent via SAC updates (1), (3). Justification for excluding the CQL regularization term (6) during fine-tuning can found in Appendix B.
CQL-ft:
SAC: a SAC agent trained from scratch. In particular, we assume the agent does not have access to the offline dataset. We include this baseline show the benefit fine-tuning pre-trained agent in terms of sample-efficieng
For fairness comparison, we consider ensembles of baseline methods for comparative evaluation.
6.2 COMPARATIVE EVALUATION
Figure shows the performances of BRED and baseline methods during fine-tuning, based on vari- ous type of datasets from the D4RL benchmark (Fu et al., 2020). In most tasks, BRED outperforms all baselines in terms of both sample-effciinccy and final performance. In particular, method

---

Under review as a conference paper at ICLR 2021
BRED (Ours)
AWAC
BCQ-ft
CQL-ft
SAC
14,022
5,000
Average Return
12,022
10,022
Average Return
3.000
Averagge Retum
4,002
2,000
1,000
15-10"
20H10"
25410"
Timeedipp
Timesteps
(a) Halfchectah-random
(b) Hopper-random
(c) 2alker2d-random
10,000
5,002
Avorape Avorape Rotumn
Nedgg Rotum
Meagg
6,000
5,000
2,000
4,003
2,000
2,000
1,000
10-10"
10w12"
10x10"
20100
20"10
0-12"
25.10"
Timestops
Timestops
Timestops
(d) Halfcheetah-medium
(c) Hopper-medium
(f) Walker2d-medium
8,000
Average Retum
Average 9otm
8,000
0,000
Average
4,002
2,000
1.000
1,000
0x10"
S412"
30x10"
18410"
20-10"
25-m0
0.10
5x12"
10.10"
15x10"
20410"
25410"
0-12"
5-10"
10k10"
19410"
20010"
25416"
Timestops
Timestops
Timesteps
(g) Halfcheetah medium-replatt replay
(h) Hhopper-eedium-rpph
Walker2d feeeeor2d--miiu--rre replayy
14,000
12.000
6,000
10.000
Avcrogs Retum
Neoage Retum
8,000
Mengg
6,000
2,000
3.000
4,000
1000
2,000
1,000
D-10"
10-10"
15+10"
20
20nd
Dx10"
5+12"
10+10"
15-10"
20"30"
25.10
0-12"
2-10""
10x
15"10"
20v1o"
25
Timsstops
Timeslapa
Timestape
(i Halfchectah- Baafchoctah-medium-oxpoee
(k) Soopper-medium-expie expert
(l) Walker2d wiweeer/ddmmiiuu-ppppoee
Figure 3: Performance on MuJoCo tasks from the D4RL benchmark (Fu et al., 2020) during online fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respec- tively, across four runs. Dotted lines indicate the performance of offline RL agents before online fine-tuning.
significantly outperforms CQL-ft, which shows that balanced replay and ensemble distillation play an essential role in stabilizing the fine-tuning process overall. Note that BRED removes the initial performance dip observed in CQL-ft by addressing the harmful effect coming from distribution shift.
We also emphasize that BRED performs consistently well across all tasks, while the performances of AWAC and BCQ-ft are highly dependent on the quality of the dataset used. For example, we observe that AWAC and BCQ-ft show competitive performances in tasks where the datasets are generated by high-quality policies, i.e., me medium-es deueeerp tasks, but perform worse than SAC on r random tasks. This is because AWAC and BCQ-ft employ the same regularized update rule for offline and online setups alike, and fail to balance the offline and online experiences for replay.

---

Under review as a conference paper at ICLR 2021
3,900
5.000
5,000
BRIED
Online only
4,000
Urifom ruplay
4.000
6,000
Awtpge Smmm
Jastgge Rutun
2,00
3.000
65000
2,000
2000
2,000
1,00
1.000
1,000
Online only
Online only
Unilorm ruplay
Unitomm raplay
15
20010"
0x12"
(a) Walker2d-ramdom
(b) Walker2d-medium
(c) Walker2d-meddiun-repla
Figure 4: Performance on wa walker2d tasks from the D4RL benchmark (Fu et al., 2020) with and without balanced replay. Specifically, consider two setups: Uniform replay, where offline and online samples are sampled uniformly from the same buffer for updates, and Online only, where the offline agent is fined-tuned using online samples exclusively.
5,000
Ensembile diailition
3,000
Aurttgg Retm
0,000
2,000
2,00
1,800
Ensembre dis lation
Ennomble dasitation
Indopen dont ensomble
Indepondent ensombte
10w10"
15-10"
18410" 20010"
0-16"
25410'
(a) Walker2d-random
(b) Walker2d-medium
(c) Walker2d ceeerr2dd-miiiu--rpe
Figure 5: We compare BRED (using ensemble distillation) to its variant, an ensemble of independent policies (i.e., no distillation), on wa welker2d tasks. One can observe that ensemble distillation improves both stability and performance during online fine-tuning.
6.3 ABLATION AND ANALYSIS
Effects of balanced replay. In order to investigate the effectiveness of balanced replay, we compare the performance of BRED with the performance of BRED without balanced replay. In particular, instead of balanced replay scheme, online samples are sampled uniformly from the same buffer, and (ii) Online only, where the offline agent is fined-tuned using online samples exclusively. For all methods, we applied the proposed ensemble distillation. As seen in Figure 4, BRED is the only method that learns in a stable manner, while the other two methods suffer from slow and unstable improvement. This shows that balanced replay is crucial for addressing distribution shift and stabilizing fine-tuning. Results for all other setups can be found in Appendix D.
consider two setups: (i) Uniform replay, where offline and
We also analyze the effect of the proposed ensemble distillation method on fine-tuning performance. To this end, we compare BRED to ensemble of indepen- dent policies trained via (3). As shown in Figure 5, ensemble distillation significantly improves performance in complex tasks such as walker2d, where policy updates must be more robust to distribution shift. In particular, for the wa weelkdrrarr policies fails to learn highly performant policy, while BRED achieves a near-expert score at the end. Overall trends are similar for other setups considered (see Appendix E).
Effeets of ensemble distillation.
task, the ensemble of independent
Effeets of ensemble size. We investigate the performance of our method while varying the ensemble size N E {1, (1,2,5] case, i.e., N = 1, we did not use policy distillation. Figure 6 shows that that the fine-tuning per- formance of BRED improves N increases, which shows that larger ensemble size leads more stable policy updates. More experimental results for all tasks can be found in Appendix F, where the trends are similar.
walker2d med um rerlleeply task. We remark that for the single-model
8

---

Under review as a conference paper at ICLR 2021
3,900
5.000
5,000
BRED
(N-
4,000
BRED (N
4.000
6,000
Awspge Samm
bastgg Reau
2
0.000
65000
2,000
2.00
2,000
1,00
1.000
(N-5)
1,000
BRELD
(N=2)
BRED (N- 11
BRED (N-1)
0-12"
(a) Walker2d-random
(b) Walker2d-medium
(c) Walker22dmmediun--relay
Figure 6: Performance on wa ker2d tasks from the D4RL benchmark (Fu et al., 2020) with vary- ing ensemble size N E {1, We observe that performance of BRED improves as N increases.
CONCLUSION
In this paper, we present BRED, which mitigates the harmful effects of distribution shift between offline and online data distributions, thereby facilitating stable fine-tuning. BRED incorporates two components, namely, a balanced experience replay scheme that mixes offline and online samples for training. and an ensemble distillation scheme for stabilizing policy learning. Our experiments show that BRED performs well across many different setups, and particular, outperforms prior works that tackle the problem of online reinforcement learning with offline datasets. We believe BRED could prove to be useful for other relevant topics such as sim-to-real transfer (Rusu et al., 2017), scalable RL (Kalashnikov et al., 2018), and RL safety (Garcia & Fernández, 2015).
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In International Conference on Machine Learning. 2017.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. ICLR, 2016.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q- ensembles. arXiv preprint arXiv: 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning. 2019.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Caglar Gulcehre, Tom Le Paine, Bobak Shahriari. Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, Gabriel Barth-Maron, Ziyu Wang, Nando de Freitas, and Worlds Team. Making efficient use of demonstrations to solve hard exploration problems. In International Conference on Learning Representattinns 2020.
Tuomas Haamojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, 2018.

---

Under review as a conference paper at ICLR 2021
Hado V Hasselt. Double q-learning. In Advances in Neural Information Processing Systems, 2010.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI Conference on Artificial Intelligence, 2016.
Auke J Ijspeert, Jun Nakanishi, and Stefan Schaal. Learning attractor landscapes for learning motor primitives. In Advances neural information processing systems, 2003.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz. Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-basee robotic manipulation. In Conference on Robot Learning, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Beomjoon Kim, Amir- massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems, 2013.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 2020.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: estimation bias of g-learning. In International Conference on Learning Representattinss 2020.
Controlling the
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. arXiv preprint M.Niv:000.00493: 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.01663 2020.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, 2016.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2018
David Rolnick, Arun Ahuja, Jonathan Schwar7, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. In Conference on Robot Learning, 2017.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Be- havioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.0839 2020.
10

---

Under review as a conference paper at ICLR 2021
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:170.008817, 2017.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint ar 1911111611 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Infor- mation Processing Systems, 2020.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In International Conference on Robotics and Automation, 2019.

---

Under review as a conference paper at ICLR 2021
Appendix
A EXPERIMENTAL DETAILS FOR BRED
Offiine datasets. For training offline RL agents with offline datasets, we use the publicly available datasets from the D4RL benchmark (https: / //g//ttttis thub. com/ ccoo/raal-biee rwil-bbrkeeeeeeeeeee / with- out any modification to the datasets. Note that we use datasets available from the commit c4bd3de, i.e. renewed version of datasets.
Training details for offiine RL agents. For training CQL agents, following Kumar et al. (2020), we built CQL on top of the publicly available implementation of SAC (https: (attps://ithub: htn//itchyy//lll /rkit) without any modification to hyperparameters. As for network archi- tecture, we use 2-layer multi-layer perceptrons (MLPs) for value and policy networks (except for hal halfcheetah-medi um-expert task where we found that 3-layer MLPs is more effective for training CQL agents). For training BCQ (Fujimoto et al., 2019), we use the publicly available imple- mentation from https: /ttg//gtttti wtt.thb..../...oo com rail i1 berkelek master /boq. For all experiments, following the setup in Kumar et al. (2020), we trained offline RL agents for 1000 epochs without early stopping.
aelllevva eva auatiiiinn /ons/treee
Training details for fine-tuning. To fine-tune a CQL agent with our method, we initialized pa- rameters of policy and value networks using the parameters from the pre-trained offline CQL agent. For all experiments, we report the performance during the 250K timesteps for random seeds. For method, we used Adam optimizer (Kingma & Ba, 2015) with policy learning rate of (3e-4, 3,, 5--6] and value learning rate of 3e-4. We found that taking 10x more training steps (10000 opposed the usual 1000 steps) after collecting the first 1000 online samples slightly im- proves the fine-tuning performance. After that, we trained for 1000 training steps every time 1000 additional samples were collected. For AWAC (Nair et al., 2020), we use the publicly available im- plementation from the authors (https: : //gi thub. com/ wwwtthtrrlliiiiiii /tree//////eeeeeeee paree/moc tateee/mmseee//// exampl 1 es / awwa)) without any modification to hyperparameters and architectures. For BCQ-ft, we use the same implementation as the original BCQ, with the only difference being that we used additional online samples for training.
Training details for balaneed replay. For fine-tuning offline RL agents with the proposed balanced replay, we use p8a E {0.5, 0.75] for initial fraction of online samples, 256 for batch size B, and trinal = 125K for the final step of annealing schedule in (7).
Training details for ensemble distillation. For the proposed ensemble distillation method, we use N CQL agents. To distill the ensemble of policies into a single distilled policy, we optimize dis- tillation objective (8) for 1000 epochs with early stopping using the pre-defined validation samples. For action selection during evaluation,
the mean of distilled policy Tap-
12

---

Under review as a conference paper at ICLR 2021
B EFFECTS OF CQL REGULARIZATION ON FINE-TUNING
Here, we show that removing the regularization term (6) from the CQL update is crucial to the fine- tuning performance. As shown Figure 7, fine-tuning with CQL regularization prevents the agent from improving via online interaction, for the Q-function is updated to be pessimistic for unseen actions. This essentially results in failure explore. For BRED, we do away with regularization, and instead use the modified SAC update (10).
0000-
DOL-ft (Nol regularized)
CQL-Il 41l (Not regularized) CQL-ft (Regularized)
CQL-ft (Ragularized)
7000
4000
2000
Aerage Retur
NN00- www.opoooo....... Roum
8000
-0:00
2000
5000
1000
4020
CQL-ft (Not regularized)
1050
CQL-At (Regularized)
5030
S-10"
10-10" "5-10"" 20-00" 20H1
6+10
10w-1g 18-10" 20-10"
25-10f
5-100
10-10" 15 20
26
Timwstape.
Timestaps
Timesiegs.
(a) halfchhetah-mmdiuu-rrpaaya
(b) Hopper-medium-repie
(c) Walker2d-medium-repla
Figure 7: Performance of CQL-ft methods on (a) halfcheetah senthh--eiium mediun-replay. (b) hopper medii um- ceeedduu-reppall. aeplaay and (c) wa 2el replay ay tasks from the D4RL benchmark (Fu et al., 2020) during online mne-tuning. The solid lines and shaded regions repre- sent mean and 95% confidence interval, respectively, across four runs.
C EFFECTS OF ONLINE RL ALGORITHM ON FINE-TUNING
Since SAC optimizes different critic objective (1) which is different from the objective (5) of offline CQL algorithm, this difference in objectives might cause the instability in fine-tuning performance. To investigate how the difference in objective affects the fine-tuning performance, we provide addi- tional experimental results by considering following two online RI. algorithms that do not change the critic objective: (i) SAC with deterministic backup (SAC-D) that learns Q-function by opti- mizing the same critic objective (5) instead of (1), and (ii) Twin Delayed Deep Deterministic Policy Gradient (TD3; Fujimoto et al., 2018), an off-policy RL algorithm that learns deterministic policy. As shown in Figure 8b and 8c, agents fine-tuned with SAC-D and TD3 also suffer from instability in the initial phase of fine-tuning, which implies that performance drop is due to the distribution shift problem, not the difference in critic objectives. As shown in Figure 8b and 8c, agents fine-tuned with SAC-D and TD3 also suffer from instability in the initial phase of fine-tuning, which implies that performance drop is due to the distribution shift problem, not the difference in critic objectives.
CGL-ft (SAC, Unirmm reptay)
CQL-R (SAC-D, Uniform replay)
DOLLtt (1D3, Unifom malayy
(SAC. Online
(SAC Online
COL Online
4.000
4,000
tesggs Rdbit
3,000
Aveeog Retar
Frongu-Ruaar
3,000
2,008
2,000
1,00
1.000
1,000
0-
0.54107
04H1p" 03413"
0,2-10
S,041p 0.6110 0.841g Tinostags
0.5H10
6.2k10
0.64100 09e1g" 0 1/De100
Tasssopo
Timestppo
(a) Walker2d-meding (SAC)
(b) Walker2d-medimm (SAC-D)
(c) Walker2d-medium (TD3)
Figure 8: Fine-tuning performance on walker2d-medi um task when using (a) SAC, (b) SAC- D. and (c) TD3 for fine-tuning pre-trained offline CQL agents. The solid lines and shaded regions represent mean and 95% confidence interval. respectively, across four runs.
13

---

Under review as a conference paper at ICLR 2021
D EFFECTS OF BALANCED EXPERIENCE REPLAY SCHEME
We provide learning curves for all tasks with and without balanced replay. The proposed balanced replay scheme is indeed crucial for stable fine-tuning procedure.
14.00
4,000
5.000
BRED
only
3,000
4.000
Unifomm roplay
Aotraa Roam
0,002
0.000
2,000
2.000
4.000
1,000
1.000
2,002
Oriine ony
Online onl
Uniform rop ay
Unfbrmm reslay
22x10"
25412'
20010"
(a) Halfcheetah-ramee random
(b) Hopper-random
(c) Walker2d-random
10,000
4,000
$,000
8,002
1,000
4,000
Roorgg Rotmm
Acceape Ittam
6.000
8,000
2,000
4.002
2,000
2,002
BRED
1,000
ony Uniform rep
Online only
Online only
Uniform replay
Uniform reptay
15x00"
20010"
25-10'
(d) Kalfcheetah-mediune
(c) Hopper- medium
(f) Walker2d
Walker2d-medium medium
4,000
5,000
8.002
3,000
Aoorge Rabin
Acaraee Rakur
0,002
3,000
2,000
feoggg
4.000
MWN.
2,000
2,002
1,000
Online only
Online only
Online only
Unilam replay
Uniform ruslayy
Uniuum ruplay
20-10"
25-10"
(g) Saaffhectah-mediim-rqppppp
(h) Hedper-mediumrrellay
(i) Walker2d walkerrdd-edduu-rrlllaa
4,002
5,000
Ancugg Neam
10:000
Pectogg Ream
3,000
Reaan
4,000
8.002
2,000
4.000
1,000
2,002
Online onl
Online only
1,000
Online only
Unifom rop
Undfrmm reslay
Unioorm replay
15x10"
20x10"
20410"
(i)
(k) Hopper-medium-exee expert
(1) Walker2d-medium-eo expert
Figure 9: Performance on MuloCo tasks from the D4RL benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
14

---

Under review as a conference paper at ICLR 2021
E EFFECTS OF ENSEMBLE DISTILLATION
We provide learning curves for all tasks with and without ensemble distillation scheme. One can see that ensemble distillation highly effective for more complex tasks such as walker. However, observe that BRED without ensemble distillation method, i.e., Independent ensemble, sometimes performs better in simple tasks such as hal Echeetah, as exploration with diverse policies is relatively more important for these tasks.
Ensemole distilation Independent ensemble
12,000
10,000
Arsoga Rebun
3,000
8,000
9.000
4.003
1,000
2,002
Insependent onsemble
Independent ensemble
0-10"
Seid
10+10'
15-10"
26-10'
25-10
D-10
10x10"
15x10'
22+10"
20rl2"
0-10"
5.10
10K10" Tinastop
1ovid"
20
22n10"
Tanzzape
Tinoliaa
(a) Halfcheetah-random
(b) Hopper-random
(c) Walker2d-random
1,000
Raters
Awarape fakers
8.000
2,000
4.002
2,002
Independent ensemble
Independent ensemble
Independent ensemble
0-10
2010
10-10"
1beld" 20-00"
2003d
C-10
5-12
10+10"
15x10"
2001d"
25-10"
O-10
5-10"
10
15-10"
20-10"
25n0d"
Trrssspo
Timestope
Timetops
(d) Halfehectah-mmddim
(c) Hopper-
(f) Walker2d-medhie
Arcagg Ratr
Average Foteer
6.002
2,000
4.000
1,00
2,00
disdilation
Independent ennemble
Indepsereett ensambbie
Independent enseemnin
D-50
S000"
10x10'
1pxld" "Inerppe
20eth
Ev10
5-10"
30.10"
15e10"
20010"
25
Oe10"
10
15-th
20.10"
Timadagee
Timodage
g Halfchectthh-medium-ee
(h) Hopper-medium-reee
(i) Walker2d
12,002
10.000
Anstapa Batum
6,000
2000
2,00
8.002
2,00
4.002
1,000
2,002
Indepancentt ereemole
Independentt arowmole
Independeent anseenbblee
1-
5410"
10w10" ipvid"
20-0d
25.1s
0-10
Swid
10xi5"
15x10'
20
20rld
0+12"
5m'0"
10-10"
ttxid
20w10"
25x10"
"Irarape
Trandaze
Timatice
( Halichhetah--eeduu---pppet
(k) Hopper-medium-exppet
Walker24dmmddium-epper
Figure 10: Performance MuJoCo tasks from the D4RL benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
15

---

Under review as a conference paper at ICLR 2021
F EFFECTS OF ENSEMBLE SIZE
We provide learning curves for all tasks with varying ensemble size E (1,2,5) 5). One can see that performance of BRED improves N grows.
14,00
4.00
5.000
3,000
6.000
BRED (N-
1.000
Netrag Roam
8,002
0.000
2,000
2.000
4.000
BRED (N-5)
1,000
BRED (N-5)
1.000
2,000
(N-2)
BRED (N-2)
BRED (N=1)
BREE (3
20-10"
(a) Halfcheetah-ranne random
(b) Hopper-randeee random
(c) Walker2d-random
10,000
4,000
5,000
8,002
1,000
4,000
Rocrgg Rotmm
Aceespe Roamm
6.002
3,000
2.000
4.002
2,000
2,002
BRED (N-5)
BRED (N-5)
1,000
BRED (N-5)
(N=2)
BRED (N-1)
BRED (N-11
BRED 11
15x00"
25x10'
(d) Halfcheetahhmediin
(c) Hopper medium
(f) Walker2d
Walker2d-medium medium
4,000
8.002
3,000
Aoorgg Rabin
Acarrpe Rakum
0,002
3,000
2,000
4.000
2,000
2,002
BRED (N-5)
BRED (N-5)
1,000
BRED (N
BRED (N
BRED (N=2)
BRED ()
BRED (N-11
BRED (N
(g) Haafchectah-medium-rpplg
(h) Hopper-medium-replag
(i) Walker2d maakkrrddmedduum-rppaa
4,002
5,000
Bocrog Neam
10:000
Iettog Ream
3,000
Reaan
4,000
8.002
2,000
4.000
BRED (N= 5
BRED IN 5)
BRED 6)
2,002
(M
(N
1,000
BRED (N- 1
BREED (N
BREED (N 1)
15x10"
20x10"
(i) Halfcheetah sewicheetth-mmeduu-exxxpee
(k) Hopper- expert
(1) Walker2d-medium-ee expert
Figure 11: Performance on MuJoCo tasks from the D4RI. benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
16

---

Under review as a conference paper at ICLR 2021
G EFFECTS OF INITIAL FRACTION IN BALANCED REPLAY
In this section, we provide sensitivity analysis of the balanced replay scheme to the initial frac- tion po in (7). As shown in Figure 12, fine-tuning performance of BRED stays relatively robust to different Po" in random, medi um tasks. is sometimes sensitive for medi um- replay and me medi um- expert tasks, where the offline datasets had been generated by a mixture of heteroge- neous policies.
14,002
4,000
6.000
11:0 (pr
BRED (xy-0.500
3,000
4.000
10,000
0,002
Aucaye Rabar
4,000
Avengaa
3.000
BRE0
200
0.002
2.000
1,002
1.000
2,002
BRED 1 751
(7.)
BRED Lop 50)
BRED (00))
10x10"
15"10"
10x12*
15x10"
25-10
25-22
0x10"
5-10f
2001e
25.10
(a) Halfcheetah random
(b) Hopper- random
(c) Salker2d-randon
10,002
4,000
5,000
0,002
3,000
4,000
Arogg Rahan
Avarge 7akar
8.002
3,000
2.000
4.000
2,000
1,000
(2-4-223)
(c-00
1,020
(pa00.50)
(2.0075)
BRED (au-6751
(0)
SIED (2-0..00)
BRED (op1
16x10"
2510
15x10"
20-13" 25xggr
0x10*
5"10"
10x10"
10eeg"
20/10"
25410"
(d) Seaffcheetah-medinn
(c) Hopper- medium
(f) Walker2d meddiu
10,002
4.000
5,000
0,002
3,000
4,000
Jecrgg Ream
Aurrag Neemm
6.002
3,000
4.000
2,000
(o
3,000
(o--0
2:000
1,000
(2-10
(2.00.00)
BRED (ou-8:301
Timestopo
(g) Hallcheetah-mmeiiu--eee
(h) Hopper-mediun-repla
ii)
14,000
4,000
6,000
5,000
Ancrgg Reamn
10,002
Averan Retar
5,000
4,000
0.002
4,000
8,000
6.002
2:000
1,002
(00-225)
2,002
1,000
(pent
(pent
(o. 000)
BRED (p-08000)
0-32"
15x12"
25-10
10-10*
15--1"
25-10"
0x12"
15"13"
20-10" 25-10*
(i) Halfcheetah wewpeettt--miiuum-eee exppert
(k) Hopper-medium wwweer-dddmiiiieeeeeeeee expert
()) Walker2d-medium expert
Figure/ 12: Performance on MuJoCo tasks from the D4RL. benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
17

---

Under review as a conference paper at ICLR 2021
H EFFECTS OF FINAL TIMESTEP FOR ANNEALING IN BALANCED REPLAY
In this section, we provide sensitivity analysis of the balanced replay scheme to the final timestep ginel in (7). As shown in Figure 13, fine-tuning performance of BRED stays relatively robust to different fginal in zandom, medium um tasks, is sometimes sensitive for medium replay and me dii um- expert tasks, where the offline datasets had been generated by a mixture of heteroge- neous policies.
14,002
4,000
6.000
BFCD
E2K3
BR2D
3,000
10
4.000
Avenga beam
12.00
Kvnaayo Roam
-25000
0,002
8.000
2,000
2.000
4.00
520
Now
125K)
BRED Prem
1228)
1.000
2,000
187K)
DREDD Powr
187K)
SRED
200K)
BRED
250K)
20410'
25x12"
5.10"
10x10"
20"10"
Tinccepe
(a) Halfcheetah- randoo
(b) Sooperrrandone
(c) Walker2d-random
10,002
4,000
$,000
8,002
4,000
1,000
Ream Relan
Acaerge Rakum
6,002
3,000
Roorge
2,000
4.002
2,000
Oow 9BK
2,002
SRED Oew
129K)
BRED Pua 12540
1,000
BRED four
12080
SRED 0em
122K)
BRED f... - 187K)
SRED 18990
SPED
2508)
BRED 2500
BRGD
2500)
15-10"
20-10"
25-10"
15-10"
(d) Halfeheetah-mediun
(c) Hopper- medium
(f) Walkke22d-medium
10,002
4,000
5,000
8.002
4,000
3,000
besrog Ream
Narrae Ieem
6.002
3,000
2,000
4.000
2,000
fl.
2,002
0um
129K]
BRED P... 12540)
1,000
1200
3FE.
Oem
187K)
BRED Nu...
107%)
DREDD 187K)
3PED P....
250K)
BRED 2600
BRED 250R)
15x10"
6-10
20-10" 25-10"
(g) Sallfchetta-mediuu--eee
(h) Hopper-medium-reply
ii)
14,000
4,000
0,000
5,000
Roouge Ream
10.002
Averapo Ream
3,000
Reaans
4,000
0.002
2,000
4.000
Oo..
1,000
Powr
2,000
Fowr
2.000
SPE. Os...
0om.
127K)
125K]
DRED Dow. 185K)
Prwr 1288)
1,000
BREED Pow 1979
BRED Pow 1358)
SPED 25081
BRED Pow 2809)
BREED 280K3
15x10"
20-10"
25-10" 25H12"
0x10"
(i) hwwpheetta--mmiiuu-xpeeeeee expert
(k) Hopper weaeer-dddiiduieeeee expert
()) Walker2d-medium expert
Figure 13: Performance on MuJoCo tasks from the D4RL. benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
18

---

Under review as a conference paper at ICLR 2021
I EFFECTS OF AN EXPONENTIAL ANNEALING SCHEDULE FOR BALANCED REPLAY
In this section, we provide additional experimental results with different annealing schedule instead of linear annealing schedule in (7). Specifically, we consider exponential annealing schedule:
P min (1, (1 + 7)1), where 7
Finel
Pü
Figure 14 shows that exponential annealing schedule sometimes degrades the stability of BRED in tasks such as walker2d medi um replay. We conjecture this is because the fraction of online samples increase abruptly in the middle of online fine-tuning, which leads to unstable Q-learning.
14,000
4,000
5,000
12/00
(Esponenta))
10,000
3,000
4,000
2,00
0.000
2,000
4,00
1,000
(Urear)
(Uirer)
BRED (Exporantia))
BRED (Esponentia)
3-10"
5-13"
12+10"
15-10" 20-12'
2010
0-12
5-10"
12+12"
13-10"
20-10"
2010
D-12
5-10"
10h12"
150m0
20-10
22-16"
(a) Halfcheetab-random
(b) Hopper-random
(c) Walker2d-random
3.100
3.000
4,000
Acotag Sakan
4.00
2,00
2,00
4.00
2000
1,000
BRED A. rwa))
BRED (lewwr)
BRED (Linear
ERI (Toporarkaa)
BRED (Esperentt))
BRED (fispeeerii)
0-10
2x10
10-15
'5-10" 23
25r00
D-12
5-10
10-15
12-10
20x10"
20100
0-10"
5-10"
12-10"
"SH10" 20110
20-15"
Tieadops
Treadios
Truncage
(d) Halfcheetah-medium
Hopper-medium
(f) Palker2d-medium
10.300
5,000
4.000
2,00
2,00
4.000
1,000
1,000
BRED J. nea))
BRED (Urww) aRED (Esperenta)
BRED (Lineer)
DREED (Fsponerkal)
BRRE (Eipaneetio)
10H10"
'5-10" 23-10" z3-0g
10-12" 1541d"
20-10" 2010
"5-10"
20-10
20-13
Timedogs
Treemso
Trawsage
(g) Halfcheetah- maaghbettth-mediiu--rellaaa
(h)
(i) falkke2ddmedium--rpph
14,000
4.90
5.000
Kcogge talan
11.000
3,00
0.300
4300
1,300
BRED (ipporeraa))
BRED (EspSTweEiO)
BRED (.rrer)
(Eqponerin))
10612
20-1d"
12-12"
10H10" "5H10"
(i) Halfcheetah-medium-ee
(k) Hopper-medium-experr
(1) Saller2d-mediim-eo
Figure 14: Performance on MuJoCo tasks from the D4RL benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
19

---

Under review as a conference paper at ICLR 2021
J COMPARATIVE EVALUATION OF UNIFORM REPLAY AGENTS
We further provide ablation studies for BRED, namely, compare the ensemble distillation agent, with Uniform replay. This is to measure the performance of distilled ensemble agent without the balanced replay scheme, i.. BRED, but without BR. We observe that agents trained such do not show a particularly strong performance compared to baselines, especially for a difficult control task like wa eerdd. This demonstrates the importance of balanced replay scheme for fine-tuning offline RL agent.
14,00
4.300
5.00
AWAN
10/06
COL-R
S00-t
3,000
4.000
SAC
2,000
6,00
2,00
1,000
2000
0-107
1-127
11x"y" 18-10'
20-33" 25410
0x127
10010"
25xtp*
0-37"
15-10"
25
2543gt
(a) Halfchectah-random
(b) Hopper-random
(c) Walker2d-random
13.00
5,000
4,100
4.000
3,000
2,300
4.000
Linfom repley
2,00
AMMC
1,000
2.300
BCQ-I
COL
0-10"
Sel0
10vidh
15e104 20w-ge
2011
0rS
Crig
10-10"
1CH15
20+1
25
0-
10H10t
20w10
20r1
Treosiss
(d) Halfcheetah-mediim
(e) Hopper-medium
Walker2d-medium
73.300
4.90
5,00
4.100
3,700
4,000
Accege Rolm
0.000
2,000
2000
2.300
B00%%
1,000
COL4
SAC
0-10"
10h10"
23+10"
10-10"
5-100
torig
23-10
(g) salichhetta-mmdiiu--eee
(h) Hopper-
(i) Miiiker2--mediuu-rptteee
12,000
1,90
Roam
4.500
2,30
3.000
0.000
1,000
2.300
1.000
0-10"
Sxid"
"Oxi0" 15-10"
20-10"
0-17
biigh
"0-10"
04"0"
S-10"
10
10-10"
20-10"
Triedage
(i) Halfchetaah-mediin-eenpp
(k) Hopper-medium-exper
(1) Walker2d-medium-eee
Figure 15: Performance on MuloCo tasks from the D4RL. benchmark (Fu et al., 2020) during on- line fine-tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs.
20

---

Under review as a conference paper at ICLR 2021
K EFFECTS OF ENSEMBLE DISTILLATION
Ensemble distillation provides robustness to Q-function approximation error, by reducing policy variance. To show this, for each update during training, given an observation in the sampled mini- batch, we sampled 10 actions from the policy being trained, then measured the variance of Q values across these 10 actions. Then we averaged the variance in Q values across the sampled minibatch. In other words, we calculated the following statistic with mini-batch size 256 and 10 sampled actions: B,-8,a-((1s)) (s a)], where we use Tzya as for the ensemble distillation agent, and Gaussian policy with mean Jr and variance 2² (9) as ι for the independent ensemble agent. Figure 16 shows that the variance of Q values for independent ensemble is much higher than for ensemble distillation. This suggests that ensemble distillation indeed provides learning stability.
Ensertiee bedie
Ereaetth daliation
Ensereei detilator
indepeecedd i wariile
indepeedett orcentt
1000
Inteprederseeebbee
19645
1080
50
2502
2+10"
60'0""
201e""
8-10"
10-1pf
Wootot Tresdegs
Ztriet
Sherpt
0-11"
1-127
10-10"
192"0"
(a) Halfcheetab-rrmdoom
(b) Hopper-random
(c) Walker2d-random
Ereemsile
20000
Essorelo disiator
300
Independeet sertle
independent
Independerrosssrtee
2000
1000
5008
900
5000
Terstags
Tirattpp
Toradges
(d) Halfchhetah-medium
(e) Hopper-medium
(f) meldem
250
Enserttle bador
56-
Creanthe dallttinn
1200
Enserrie dellaann
230
indeperdend a swarbii
Independent arcanch
Independerraeeerbie
120
30
28-
400
24107
Therof
1<12"
1x127
saar"" 2011""
9412"
10-17"
Texte"
20418" 2040
(g) Hallchhetta--mmdiuu--rrppaa
(h) Heppee--
(i) Walkerr2dmmediun--eplay
Enserrle dail Indoosrdent e msmmbl
202
Ersembee decilatin
Erserbre disilatin
250
175
130
Indeseedivt aniembe
230
142
(01
-25
.033
100
130
71-
Tirattps
Tressige
Tressigs
(i) Halfchettah-mcdiu--epprrt
(k) Hopper-medium-expeet
Wallerradmmediumeexpet
Figure 16: Variance in Q-values for 10 sampled actions averaged across mini-batch during fine- tuning. The solid lines and shaded regions represent mean and 95% confidence interval, respectively, across four runs
21

---

Under review as a conference paper at ICLR 2021
L COMPARISON Q-FUNCTION ESTIMATES AND TRUE Q FUNCTIONS
In order to demonstrate the harmful effects of distribution shift, and the effectiveness of BRED to deal with such harmful effects, we measured how well the Q-functions of BRED and On- line only agent represent the ground-true Q fuctions, respectively, in similar manner to Fu- jimoto et al. (2018). Here, the Online only agent is exactly the same as the BRED agent, except it only (10000, 20000, 30000, 40000, 50000), we randomly sampled 1000 state-action pairs gathered on- line from the replay buffer, then calculated the average of Q values across these samples. Next, we rolled out 25 episodes starting from each of the state-action pair, then averaged the discounted returns thusly obtained to obtain an estimate of the true value at the state-action pair. Finally, we calculated the average of Q value estimates over the 1000 samples, then compared it with the average of Q-function estimates obtained from (1).
online samples gathered during fine-tuning. In more detail, timesteps
As we see in Figure 17, Online only agent suffers from high variance and overestimation bias in three of four tasks considered, namely, rdddraaddm wa 1kker2d-meduu--rep and walker2d -ed2dddmeiin wam-mmemmm expert. On the other hand, BRED shows low-variance Q- function estimates across different runs, and are good estimates of the true Q-function in three out of four tasks, namely, walker2d-random, walker2d-medi um- and walker28 -:22dddmeiinm un-expert. This shows the benefit of balanced replay for stabilizing Q- learning.
1,200
1,000
BRED (GT)
400
Oninc only
only (GT)
500
Target
200
BRED
BRED (GT)
Onine only
Onlic only (GT)
100
2-10
S-10f Timuskapa
4x10
Sx10
1x10*
2x10f
3-10" Timestup
4-10"
5x10
(a) Walker2d-random
(b) Walker2d-medium
BRED
500
0,000
BRED (GT)
BRED (GT)
400
Onine only
6,000
Online only
only (GT)
Target Vilue
Online only (GT)
300
4/000
Taagel
200
2,000
-2,00
(c) seeiddd-mmiim--ppie replay
(d) Walker2d Walker2d-medium-oon expprrrteeepeetttte
Figure 17: Comparison of Q-function estimates and true Q functions for BRED and Online-only in
Walker2d tasks. We see that BRED shows low variance, and suffers less from overestimation bias.
22


Reviews:
[{"review_id": "yabqxVDP67L", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of policy learning in Markov Decision Process (MDP) from the combination of online and offline samples. The offline samples are generated by a behavior policy in the same MDP model, i.e., the behavior agent and the learning agent share the same state-action space. The learning procedure goes as follows. One first trains a MDP policy from the offline data; the online samples are then used to fine-tune the learned policy.\n\nThe authors propose a simple yet effective approach. First, the authors keep separate offline and online replay buffers, and carefully balance the number of samples from each buffer during updates. Then,  multiple actor-critic offline RL policies are trained, and a single policy is distilled from these policies using ensemble methods. Experiment results show that the proposed method consistently outperforms state-of-art algorithms.\n\nThis paper is clearly written and well organized. I am not sure about the novelty of the proposed method, since it seems to follow the line of carefully reweight online and offline samples. However, the experimental results show a significant improvement over existing methods.\n\nQuestion for the authors:\n1. In practice, what is a good heuristic for selecting the initial fraction p0 of online samples? How sensitive is the learned policy w.r.t. the initial fraction p0?", "extra_scores": {}, "extra_text": {}}, {"review_id": "N0pAGRn7lcp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\n\nOffline RL allows agents to be trained on static, offline datasets. If\nthe agent has already undergone offline training however, distribution\nshift makes further online training difficult. The authors propose to\naddress this problem by keeping separate offline and online replay\nbuffers, and sampling different proportions of each. In addition, they\npropose to ensemble a distillation policy. The proposed method is\nevaluated on logged MuJoCo datasets answering whether BRED improves over\nother fine-tuning methods and whether balancing or replay contribute\nmore to BRED's success.\n\nDecision\n--------\n\nBRED is an interesting approach, but the experiments do not demonstrate\nthat it is better than the baseline. As such, my preliminary rating of\nthis paper is rejection. I appreciate that the sampling scheme proposed\nis simple and well motivated to address the problem of distribution\nshift. It is not clear whether the sampling scheme actually addresses\nthe problem of distribution shift however, as it gives the agent access\nto additional experience. The distillation scheme also does not seem to\naddress distribution shift directly and the experiments do not\ndemonstrate that it helps performance outside of the Walker2d-random\ntask. Moreover, it is not clear whether the baselines are also\nensembles, which would make the experiment results less compelling.\n\nOriginality\n-----------\n\nThe contributions are: a sampling scheme combining offline and online\nsamples, as well as a distillation scheme. The sampling scheme is\nincremental and simple, but seems like a novel approach. Distillation on\nthe other hand, is well-explored in RL (Czarnecki et al. 2019) and does\nnot seem to contribute much to the problem addressed in the paper. Taken\ntogether, the paper has limited novelty.\n\nQuality and Clarity\n-------------------\n\nThe paper is easy-to-follow and well-written. However, I have some\nissues with the title. The title suggests that your aim is to address\ndistribution shift in online RL. However, distribution shift is a\nproblem due to offline training. Of course, the problem you are\naddressing is further online training after offline training. Perhaps\nthe title should be reworded to make this more clear.\n\nStrengths\n---------\n\n-   There is a clear exposition of the problem addressed, namely\n    distribution shift. This is well motivated by the t-SNE\n    visualization. Section 3 in particular highlights the correlation\n    between distribution shift and the performance of offline RL\n    algorithms.\n-   BRED, the proposed method, is a straightforward solution via\n    balancing offline and online replay buffers. In addition, the\n    schedule for updating the proportion sampled of each replay buffer\n    is well argued and also simple.\n-   The results are clearly motivated by questions in the beginning of\n    the section. Sensible baselines are used to compare the proposed\n    method (BRED), and these are discussed and reasoned in detail.\n    Furthermore, there is a good coverage of tasks (3 MuJoCo) and\n    difficulties (random, medium, medium-replay, medium-expert).\n\nWeaknesses\n----------\n\n-   While fine-tuning an offline RL agent can pose challenges, similar\n    challenges can arise in supervised learning. For example, Ash and\n    Adams (2019) note that in different regimes, warm-starting neural\n    network training can hamper generalization. Overall, I feel that the\n    correlation between distribution shift and online performance after\n    offline training is not shown to be causal, but correlative.\n\n-   It is unclear why distillation is included, as it does not directly\n    address the problem of distribution shift in offline RL, outside of\n    the intuition that it mitigates bad $Q$ estimates. This is not\n    enough motivation however, and the experiment results (Figure 5\n    and 6) does not show any significant benefit of ensemble\n    distillation over independent ensembles (besides Walker2d-random).\n    Furthermore, this may not be a fair comparison to other baseline\n    methods which do not seem to be ensembles.\n\n-   Despite a very well-motivated experiment section, the experimental\n    conclusions are far too strong. The results do not back this up,\n    when there is high overlap in the confidence intervals for many\n    Figures (in each section). Furthermore, there are unexplained\n    peculiarities in the results, such as very big drops in performance\n    or the fact that the agent trained with the offline dataset is\n    substantially better than the medium one.\n\nDetailed Comments\n-----------------\n\n-   \"pointed out that offline RL algorithms… are not amenable to\n    fine-tuning, due to the difficulty of modeling the\n    dataset-generating policy in the online setup.\"\n\n    This is not clear to me, how does fine-tuning connect to modelling\n    the dataset-generating policy? Furthermore, how does the\n    dataset-generating policy (I assume this is the behavior policy)\n    connect to online RL?\n\n-   Figure 1b: the agent is trained on the offline dataset and then that\n    dataset is thrown away for the online agent, whereas the agent in\n    red gets to keep using it. This does show the difficulty of\n    fine-tuning but does not provide conclusive evidence that\n    distributional shift is the cause. Fine-tuning can be difficult even\n    in supervised learning (Ash and Adams 2019).\n\n-   \"On the other hand, when using online samples exclusively, the agent\n    is exposed to unseen samples only, for which Q function does not\n    provide a reliable value estimate. This may lead to bootstrapping\n    error, and hence a dip in performance as seen in Figure 1b.\"\n\n    Can't this be shown experimentally in an environment to provide\n    conclusive evidence for distribution shift being the cause of the\n    dip in performance?\n\n-   Figure 3: It seems that halfcheetah-random results in a better\n    policy than halfcheetah-medium. This does not make sense to me. In\n    addition, there is a strange dip in performance for CQL-ft in Figure\n    3g. Many of the figures have overlapping confidence intervals\n    (although the shaded region is only the standard deviation, the\n    corresponding confidence interval would be approximately the\n    standard deviation. Anyway these should be confidence intervals\n    rather than standard deviation). While BRED does perform\n    statistically significantly better in some tasks, I don't think the\n    comparison is fair due to ensembling used only for BRED.\n\n-   Figure 4: Going from figure 4a to 4b, balanced replay is shown to be\n    more beneficial for a dataset generated by a random behavior policy.\n    This seems to suggest that the main benefit of BRED is artificial\n    exploration. Counter-intuitvely, this trend is reversed in Figure\n    4c. Moreover, the online-only curve is very different in Figure 4c\n    than 4a and 4b, and to my understanding there should not be any\n    difference. Due to the noisiness of the curves, its hard to make any\n    straightforward conclusion from these results.\n\n-   Figure 5 and 6: Again, there is no statistical difference in medium\n    and medium-replay between independent ensembles and a distillation\n    ensemble.\n\nMinor Comments\n--------------\n\n-   Equation 3: missing brackets on the left side.\n\n\nAsh, Jordan T., and Ryan P. Adams. 2019. “On Warm-Starting Neural Network Training.” *arXiv:1910.08475*.\n<http://arxiv.org/abs/1910.08475v2>.\n\nCzarnecki, Wojciech Marian, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. 2019. “Distilling Policy Distillation.” *arXiv:1902.02186*.\n<http://arxiv.org/abs/1902.02186v1>.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5gyKcia7mc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\nThe paper proposes a fine-tuning method for an offline RL algorithm, CQL. The method incorporates a balanced replay scheme for both online and offline samples, and an ensemble distillation to stabilize policy learning. The proposed method is evaluated on benchmark environments. \n\nThe paper is clearly written and easy to read. However, the problem setting in the paper is not well-motivated and the goal is unclear. The paper makes some claims without supporting them (details come later). Moreover, some important experiments details are missing which makes it hard to assess the empirical results. I think all of these issues needs to be addressed before publication.  Therefore, I recommend to reject the paper.\n\n##### Supporting arguments and clarification questions\nFirst of all, I think it is not motivated why online fine-tuning procedure is necessary (see first paragraph in the introduction). If online simulation/interaction with the real environment is practical, why do we need to train an offline RL agent first and perform online fine-tuning (given that the paper claims that fine-tuning is challenging)? If online simulation is not practical, how could we perform online fine-tuning? \n\nIt is also unclear to me what is the goal we want the proposed method to achieve  (e.g., what are the evaluation metrics)? Do we want the fine-tunning method achieve the best sample efficiency compared to purely online algorithm, or achieve the best asymptotic performance compared to other fine-tuning methods? Without a clear goal, it is hard to assess the significance or soundness of the proposed method.  \n\nIn section 3, what is the reason for fine-tuning a CQL agent by SAC updates? Have you tried fine-tuning by DQN updates or its variant? I think SAC is learning a value function with an extra entropy term and CQL is learning the true value function (with some regularization during the optimization procedure), so these algorithm are essentially learning two different targets. It makes sense that we would see instability during fine-tuning since they are optimizing towards different targets. Therefore, I don’t think the clam “this instability occurs due to the shift between offline and online data distribution” is supported in the experiment. It might be due to the fact that we are optimizing different objectives. \n\nIn the experiment, how were the hyper-parameters selected? The hyper-parameters should be selected based on the offline dataset, not the fine-tuning performance. I think the paper should also include a baseline, which is the performance achieved the offline RL agents (i.e., horizontal lines in the learning curve). In Figure 3, it seems like BRED is much more stable compared to other methods in Hopper medium, so I wonder if the reason is that BRED already learns a good policy with offline dataset and it does not change much during the fine-tuning procedures. \n\nIn section 5, the paper provides a nice discussion on the related works. However, the paper claims that “this method (Nair et al., 2020) relies on regression, hence the learned policy seldom surpasses the best data-generating policy”. Can you elaborate more on this?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0JXrCzbWa33", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes to deal with distribution shift problem between online and offline samples when the agent trained by offline data is fine-tuned with online interactions. Two mechanisms are introduced: (1) using two replay buffers for offline and online data respectively, and training the agent with data sampled from these two buffers with a certain ratio (the ratio changes in a way that more online data is used in later epochs); (2) learning an ensemble of independent agents in the offline phase, and distilling them into a mean policy to overcome bootstrapping error. Empirical results demonstrate that the proposed method perform well during fine-tuning when there is distribution shift.\n \n\nStrengths:\n1. The paper is overall well-written and easy to follow.\n2. The problem studied by this paper is well-motivated, and the proposed methods are simple yet effective, making intuitive sense.\n3. The experiment section shows that the proposed method (BRED) outperforms baselines on 3 mujoco tasks with multiple types of offline data. Further ablation study verifies the effectiveness of the two proposed mechanisms.\n\n \nWeakness: \n1. Although empirical results are good, my main concern is that the novelty of this paper is limited. The main contribution of this paper is combining CQL with two tricks: balanced replay and ensemble distillation. As already discussed in the related work section, these two tricks are commonly used in the literature, even though the concrete setting varies.\n2. The specific way of combining the offline buffer and the online buffer (online fraction grows linearly in timestep) could be effective in some cases, but might not always be the best choice. I am curious whether the authors have tried other ways. And especially, can the agent chooses the ratio adaptively based on the degree of distribution shift and the learning performance?\n3. The definition of distribution shift in this paper is descriptive and informal. It will be nice to provide deeper analysis or insights w.r.t. the distribution shift problem. For example, is there a way to rigorously characterize the distribution shift? Is there a measurement for how large the gap is? This will be relevant to addressing point 2 above. \n4. Although policy ensemble can make the learned Q-function more accurate and stable, it is not very clear to me why it tackles distribution shift. Even the distilled policy has a more accurate estimation of the Q values, the estimation is still w.r.t. the offline data distribution. It still suffers from the bootstrapping error when using out-of-distribution samples. \n\n \nMinor comments: \n1. Figure 1 is a good visualization. However, the figure is not very informative since details are not provided (e.g., what is contained in a sample, in what sense / which dimension the distribution is different). \n2. In the experiment, I am wondering how an “ensemble-only” method would work and compare to baselines, i.e., use multiple pertained CQL agents, and fine-tune the distilled policy. The “Effects of ensemble distillation” paragraph shows a comparison between the proposed ensemble distillation and the ensemble of independent policies. However, the result does not reflect how ensemble compares with non-ensemble.\n3. The paper focuses on the distribution shift problem in offline RL. However, there could be multiple interpretations of distribution shift, e.g., state distribution changes due to the change of behavior policy, or dynamics distribution changes due to the slight change of the environment, etc. I think this paper is mainly dealing with the state distribution change case. But it will be better if the authors can make it explicit.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses an distribution shift and biased Q-values that happens when offline agents are finetuned in an online manner. The final revision of the paper is very well written and easy to understand. The proposed method in the paper is interesting, and aiming to address an important issue in RL. The proposed method involves a combination of two well-known methods in RL to tackle the distribution shift issue, the paper first suggests to use a balanced replay mechanism a replay for online experiences and another one for the offline. The second improvement is coming from the ensemble distillation.

It seems like in the light of the reviews, the authors have improved manuscript. However, I would like to recommend the paper for rejection. I would like the authors to do further experiments on the individual components of the algorithms, for example what if we run all the experiments only with BR or only using ED how would the performance change. How much improvement is coming from each one of those individual components? As it stands, it is not clear to me right now, and the proposed solution looks a bit complicated and hacky. 

The balanced replay mechanism is very similar to the replay approaches that are used for learning from demos methods like R2D3 [1] and DQfD. Also the ensemble distillation approach is very akin to RAND [2] and distillation approaches that are used in lifelong learning algorithms. It is not clear, why it is that important for offline RL. It should potentially improve online RL as well, perhaps some experiments on online RL would be interesting.

Nevertheless, I think the paper is very interesting and attempting to address a very important problem in RL. I would recommend the authors to resubmit the paper to a different venue after doing some small changes on it.

[1] Gulcehre, C., Le Paine, T., Shahriari, B., Denil, M., Hoffman, M., Soyer, H., ... & Barth-Maron, G. (2019, September). Making Efficient Use of Demonstrations to Solve Hard Exploration Problems. In International Conference on Learning Representations.

[2] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894.




Decision: Reject