Adaptive Q-learning for Interaction-Limited Reinforcement Learning
Authors: Han Zheng
Xufang Luo
pengfei wei
Xuan Song
Dongsheng Li
Jing Jiang
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
offline reinforcement learning
online-to-offline
limited interactions

Abstract: Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when an online interaction is costly.
Offline RL provides an alternative solution by directly learning from the logged dataset. However, it usually yields unsatisfactory performance due to a pessimistic update scheme or/and the low quality of logged datasets.
Moreover, how to evaluate the policy under the offline setting is also a challenging problem.
In this paper, we propose a unified framework called Adaptive Q-learning for effectively taking advantage of offline and online learning.
Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, i.e., a pessimistic update strategy for the offline dataset and a greedy or no pessimistic update scheme for the online dataset.
When combining both, we can apply very limited online exploration steps to achieve expert performance even when the offline dataset is poor, e.g., random dataset.
Such a framework provides a unified way to mix the offline and online RL and gain the best of both worlds. 
To understand our framework better, we then provide an initialization following our framework's setting.
Extensive experiments are done to verify the effectiveness of our proposed method.

Under review as a conference paper at ICLR 2022
ADAPTIVE Q-LEARNING FOR INTERACTION-LMIIED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when an online interaction is costly. Offline RL pro- vides an alternative solution by directly learning from the logged dataset. How- ever, usually yields unsatisfactory performance due to pessimistic update scheme or/and the low quality of logged datasets. Moreover, how to evaluate the policy under the offline setting is also a challenging problem. In this paper, we propose unified framework called Adaptive Q-learning for effectively tak- ing advantage offline and online learning. Specifically, we explicitly consider the difference between the online and offline data and apply adaptive update scheme accordingly, i.e., a pessimistic update strategy for the offfine dataset and greedy pessimistic update scheme for the online dataset. When combin- ing both, we can apply very limited online exploration steps to achieve expert performance even when the offline dataset poor, c.g., random dataset. Such a framework provides unified way to mix the offline and online RI. and gain the best of both worlds. To understand our framework better, we then provide an ini- tialization following our framework. Extensive experiments are done to verify the effectiveness of our proposed method.
INTRODUCTION
Conventional online reinforcement learning (RL) methods (Haamoja et al., 2018: Fujimoto al. 2018) usually learn from experiences generated by interactions with the online environment. They are impractical in some real-world applications, e.g., dialog (Jaques et al. 2019) and educa- tion (Mandel et al.) 2014), where interactions are costly. Recently, offline RI. Leevne et al.] (020) has aroused much attention. It targets the above challenge by making the agent learn from an offline dataset collected by other policies in purely data-driven manner. The difference between online RL and offline RL is shown in Figure I
actions issue by employing a pessimistic update scheme (Kumar et al.) 2019 2020 or in combination with imi- tation learning (Fujimoto et al. 2019). However, when the dataset is fixed, offline RI. cannot learn the optimal policy (Kidambi et a 2020),, and even worse, when the dataset's quality is poor, offline RL usually gains relatively bad performance (Kumar al.) 2020 Fu al. 2020 Levine et al.) 2020). On the other hand, it is challenging to evaluate the learned policy when learning totally from the offline dataset. Even though some research topies, e.g., off-policy evaluation (Dann al. 2014), study how to evaluate the learned policy without the interaction with the environment, it is still not ideal for the practical purpose.
Existing offline RL studies try to target the distribution mismatch
Some recent works try to address the above issues by employing an offline-online setting. Such methods (Lee et al., 2021 Nair et al.) (020)) focus on pre-training a policy using the offline dataset and fine-tuning the policy through further online interactions. Even though their methods allevi- ate the above issues to some extent, their main bottleneck is that they do not consider the different characteristics of offline and online data. For instance, pre-existing offline data can prevent agents from converging prematurely due to the potential diverse offline dataset, while online data can im- prove stability and accelerate convergence. Generally, these different data mixed and used by pessimistic strategy to update the policy in their methods, which may be problematic since using a pessimistic strategy for online data may harm the policy performance (Nair et al. 2020). Moreover,

---

Under review as a conference paper at ICLR 2022
Agent
Logged dataset
Online RL
Offlin
Figure 1: Online RL collect the data by interacting with the environment and they don't utilize the existing logged dataset while offline RL only exploit logged dataset without any future performance improvement. By contrast, focus obtaining the best of both worlds.
with sufficiently large and diverse offline data, high-performing policy can be learned just using pure online RL algorithm (Agarwal et al. 2020). And the online near-on-policy data also play a key role in improving the RL algorithm's stability (Fujimoto al., 2019). Hence, we should take full advantage of both offline and online dataset.
To tackle the above problems, in this paper, we emphasize that online and offline RL should be cou- pled organically. First, separate updating strategy should be employed for online and offline data, respectively, considering their different characteristics. To do so, we present a framework called adaptive Q-learning that integrates the advantage of offline learning and online learning effectively. When learning from the offline dataset, we conduct a pessimistic update strategy. In contrast, we use greedy or non-pessimistic update strategy when learning from the online dataset. Second, we design a novel replay buffer to distinguish the offline from online datasets a simple way. By utilizing such novel framework and buffer design, the agent can achieve expert policy using lim- ited online interaction steps regardless the quality of the offinn dataset. the experiments, our proposed framework can achieve better performance by using only fifth number of interactions compared with the previous method (Nair et al. 2020
Our contributions can be summarized as below:
We propose a unified framework called Adaptive Q-learning that can effectively benefit from both the offline dataset and limited number of online interaction data.
Based on the general framework, we initialize practical algorithm, called Greedy- conservative Q-ensemble learning (GCQL) that builds on top of State-of-the-Art offline RL and online RL method.
We empirically verify the effectiveness our method by comprehensive experiments on the popular continuous control tasks MuloCo (Todorov al. 2012) with offline dataset coming from D4RL (Fu et al. 2020).
RELATED WORK
Online RL In general, online RL algorithms can be divided into two categories, i.e., on-policy and off-policy algorithms. On-policy methods (Schulman et al. 2015 2017) update the policy using data collected by its current behavior policy. As ignoring the logged data collected by its history behaviour policies, they usually have a lower sample efficiency than the off-polic RL. On the other hand, off- off-policy methods (Fujimoto et al. 2018 Chen al.) 202T) enable the policy to learn from experience collected by history behavior polices, however, they cannot learn well from history trajectories collected by other agents' behavior policies (Fujimoto et al. 2019 Kumar al.) 2020). Consequently, the need for huge online interaction makes online RL impractical for some real-world applications,such as dialog agents (Jaques et al. 2019) or education system (Mandel al. 2014).
Offline RL Offline RL algorithms assume the online environment is unavailable and learn policies only from the pre-collected dataset. As the value estimation error cannot be corrected using online interactions here, these methods tend to utilize a pessimistic updating strategy to relieve the distri-
bution mismatch problem (Fujimoto et al. 2019 Kumar al. 2019). To implement such strategy,
2

---

Under review as a conference paper at ICLR 2022
model-free offline RL methods generally employ value or policy penalties constrain the updated policy close to the data collecting policy (Wu et al. 2019 Kumar et al. 2020; Fujimoto et al. 2019 He & Hou 2020). And model-baset methods use predictive models to estimate uncertainties of states and then update the policy in a pessimistic way based on them (Kidambi et al. 2020 Yu et al., 2020). Those offline RL methods cannot guarantee good performance, especially when the data quality is poor (Kumar et al., (220)). Besides, policy evaluation when the online environment is un- available is also challenging. Even though off-policy evaluation (OPE) methods (Dann et al.) 2014) present alternative solutions, they are still far from perfect.
Above issues of online and offline RL motivate us to investigate the offline-online setting.
Offline-online RL Lee et al. (021) and Nair et al. (220)) focus on the mixed setting where the agent first learned from the offline dataset, and then trained online. ]air et al.) (2020) propose an advantage-weiihhted actor-critic (AWAC) method that restricts the policy to select actions close to those in the offline data by an implicit constraint. When online interactions are available, such conservative constraint may have adverse effects on the performance. [ee et al.] (2021) employ balanced replay scheme to address the distribution shift issue. It uses the offline data by only se- lecting near-on-policy samples. Unlike these two works, our method utilizes all online and offline data, and explicitly considers the difference between them by adaptively applying non-conservative or conservative updating schemes, respectively. Matsushima et al. (2021) focuses on optimizing deployment efficiency, i.e., the number of distinct data-collection policies used during learning, by employing a behavior-regulaarzze policy updating strategy. Although in terms of deployment effi ciency, their work is between online and offline RL, it ignores existing offline dataset, and dose not focusing on improving sample efficiency, while both are addressed in our paper. Some works (Zhu iet al. 2019: Vecerik et al., 2017 RRaesswaran et al.) 2018 Kim et al.) 2013) can also learn from on- line interactions and offline data. However, they need expert demonstrations instead of any dataset, and this limits their applicability.
3 PRELIMINARIES
In RL, the interaction between the agent and environment is usually modelled using Markov de- cision process (MDP)
PM. r, 7). with state space S (state E S), action space A (action E A). At each discrete time step, the agent takes an action based on the current state and the state changes into s' according to the transition dynamics Pas (s' s, s,a)) and the agent receives a reward r (s, a, 8') E R. The agent's objective is to maximize the return, which is defined as R. = Ei 75r (st, ass B1+1), where is the time step, and 7 E [0, 1) is the discounted factor. The mapping from s to is denoted by the stochastic policy π : a ~ T(-1s). Policy can stochastic or deterministic, and we use the stochastic from in this paper for generality. Each policy æ have a corresponding action-value function Q (s, a) = E2 [Rg s, a]. which is the expected return follow- ing the policy after taking action in state s. The policy IT's action-value function can be updated by the Bellman operator T5:
Q') - (++++++000 (s', (s'))]
Q-learning (Sutton & Barto 2011 directly learns the optimal cetion-valuue function (" a) =
max, Q* (s, a), and such Q-function can be modelled using neural networks (Mnih et al. 2015).
In principle, off-policy methods, such as Q-learning, can utilize experiences collected by any poli- cies, and thus they usually maintain replay buffer B store and repeatedly learn from experiences collected by behavior policies (Agarwal et al. 2020). Such capability also enables off off-policy meth- ods to be used in the offline setting. by storing offline data into the buffer B, and not updating the buffer during learning since no further interactions are available here (Levine et al. (020). But this simple adjusting cannot guarantee the agent to have a reasonable performance, especially when the dataset is not diverse (Kumar et al., 2020; Agarwal et al. 2020: Fujimoto et al. 2019), and this is also the problem tackled in most offline RL works.
In this paper, we focus on the offline-online setting, where the agent is first learned from the offline dataset, and then trained via online interactions. And without additional remarks, online RL meth- ods refer to off- off-policy algorithms in the rest of this paper. We only use off-policy methods because they can make more use of offline data than on-policy ones for gaining high sample efficiency, and on-policy methods are not compatible with our proposed framework introduced next.
3

---

Under review as a conference paper at ICLR 2022
GGL
40
REDQ
30
Normalized
-00
10
Environment Steps
104
Figure 2: Learning curves on the D4RL Fu et al. 2020 task Hopper-nediimm-replay-w The re- ported results are the averaged performance across five random seeds and the shaded areas represent the standard deviation across different seeds. The normalized score of 100 is the average returns of a Noomann-speciffe expert while normalized score of 0 corresponds to the average returns an agent taking actions uniformly at random across the action space.
4 METHODOLOGY
In this section, we first give an illustrative example to explain our motivation. After that, we in- troduce the proposed general framework, namely Adaptive Q-learning, trying to couple online and offline RL in an organic way. To implement the proposed framework, then we present the Online- Offline Replay Buffer, targeting retaining and distinguishing online and offline data simultaneously. And finally we incorporate State-of-the-Ar (SotA) online and offline RL algorithms into the frame- work, and introduce the proposed eoneev-Conseevttivee Q-ensemble Learning algorithm in detail.
4.1 AN ILLUSTRATIVE EXAMPLE
We test the SotA offline RL method conservative Q-learning (CQL) (Kumar et al. 2020) and on-
line RL method Randomized Ensembled Double Q-learning (REDQ) (Chen et al. [021] under the
offline-online setting. Here, we choose widely used locomotion task Hopper (Todorov et al., 2012), and use the dataset hopper-mediun-replayy dataset, which contains diverse experiences from different policies. Specifically, we first pre-train the agent with the offline dataset for 100K steps. Then, the agent is fine-tuned online by alternately conducting the interaction and updating process, where the agent interacts with the environment for 1K steps, and is updated for 10K steps. Such interleaving process ends until the total online interaction steps reach 90K.
in the D4RI. benchmark (Fu et al. 2020) as the offline
The offline-online setting in this task should be favourable one for policy learning, because both di- verse offline data and online interactions available here. Therefore, we expect following results: the initial offline training using the dataset will provide a relatively good but not perfect starting point for the agent, and then the agent can be further improved by the online alternating process since online interaction data can be obtained, and finally we may acquire a well-performer policy, better than 100 (i.e., the performance of an expert).
even with normalized score close
Nonetheless, experiment results are shown in Figure 2 and they do not totally meet our expectation. Specifically, starting points of two curves show that the offline algorithm CQL and online algorithm REDQ can both obtain normalized scores greater than 0 (i.c., the performance of a random policy) but not very high after the initial offline training process. This result what we expected, because CQL is designed for the offline setting, and online off- afflppolic methods, such as REDQ, can also learn from the diverse and large offline dataset even though although the starting scores (below 30) are far from the expert score and leave large room for further promoting. both algorithms have troubles in the following online process. REDQ suffers from severe instability issue and its performance drops significantly during online learning, and in the end, the policy almost degenerates into random one. One the other hand, even though the CQL agent can keep stable during learning, its improvement is very limited.
fixed (Agarwal et al. 2020). However,
4

---

Under review as a conference paper at ICLR 2022
These results indicate that pure online RL algorithm may be problematic for effectively handling offline data and online interaction data in a single training process, and pure offline RL algorithm cannot make good use of valuable online interaction data due to its conservative updating strategy. Such observation motivates us to couple them in an organic way.
4.2 ADAPTIVE Q-LEARNING FRAMEWORK
In this subsection, we introduce proposed framework. The underlying idea is simple and can be described as follows. When the agent learns from online and near-on-policy interaction data, we choose a more greedy or no-pessimistic updating strategy, since these data reflect the truth situation of the current policy. By contrast, when data are sampled from the offline dataset, we tend to use more pessimistic updating strategy. Through such an adaptive way, both online and offline data, and explicitly consider their differences by separately applying suitable updating schemes.
make full use of
Then, we give formalization for above intuition based on Q-learning, and call this framework Adaptive Q-learning. The updating function of this framework can be defined by the following equation:
Qh+l arg min [A(Q*) + W(s, a))(")) Q"
This function consists of two terms. The first term A()) stands for the greedy updating strategy. which a regular updating function for Q-value online RL algorithms, e.g., the bellman error. The second term (BQQ) stands for the pessimistic updating strategy, which is the value penalty, e-g., the Q-value regularizer (Wu et al. 2019). Besides, a weight function W(s, a) is applied to the penalty term (Q)). This weight function is based on the sampled data type. Specifically, when use online and near-on- policy interaction data, W(s, a) will be a smaller value, and the updating relies more the objective of online RL., leading to relatively greedy strategy. On the contrary, when we use offline data, W(s, a) will be bigger value, and the updating strategy is relatively pessimistic.
Remark: Our behind intuition can also be easily formalized via the policy learning objective. Such variant of our framework can be denoted by x < arg max,* [A (**) + W(s, a) (r*)] where Axx*) a objective for the policy in online RL and B(x*) is policy penalty term. We also provide implementation for this variant based on TD3+BC algorithm (Fujimoto & Gu 2021), and our framework can largely boost its performance within limited environment steps in most tasks (see Appendix D for details).
4.3 OORB: ONLINE-OPFLINE REPLAY BUFFER
Next, we introduce a simple but effective online-offline replay buffer (OORB) to distinguish between near-on-policy online interaction data, and the offline data. OORB consists of two replay buffers. One is the online buffer that collects the online interaction data. Besides, to ensure the data in the online buffer is near-on-policy, we set to be very small, and fresh online interaction data are stored into it by following a fourstuuf rule. The other is the offline buffer consisting of the newly generated online interaction data and the offline dataset which may come from any policies.
Data are sampled from OORB following a Bernoulli distribution, which means that with a proba- bility p. they sampled from the online buffer, and with probability - p. they are sampled from the offline buffer. To benefit from both online and offline data in a balanced way, we empirically set p to 0.5, and its effect the final performance is further tested via ablation studies in Section (54] Results show that p = 0.5 works best overall, which confirms our claim that offline data and online interaction data are both crucial for policy learning.
4.4 GCQL: GREEDY-CONSERVATIV Q-ENSEMBLE LEARNING
We then present a detailed implementation of our proposed framework, by incorporating SotA of- fline RL algorithm CQL and online RL algorithm REDQ, and call our implemented algorithm Greedy Conservative Q- Q-ensemble Learning (GCQL). Specifically, we use the updating function in REDQ the first term (Q) in Equation 2 and use the conservative regularizer in COL as the
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Greedy ciesevy--onservattive Q-ensemble learning
Initialization:
Initialize policy Tæ, ensemble Q functions Q0 E Noffliin training steps
Online exploration steps To., offline update steps Toss Initialize Tinitiat, sample possibility p, start sampling steps:''s
Initialize online buffer Bo to empty, offline buffer Bar + offline dataset Initialize online buffer size Sen <-
Initial offline learning:
Train the agent for Tinitial steps using the logged dataset while True do
10
f
Explore Ton steps online
12
Store the Ton steps experiences to both online buffer Ben and offline buffer Bal Son f SSn Ton
14
for t << Toss do
Sample random value ps U(0, 1)
16
if < and S.n > T, then
17
Sample batch (s, a) from online buffer Bon
Set the W(s, a) to
19
end else
20
21
Sample batch (s, a) from offline buffer Boft Set the W(s, a)
23
end
24
Update the Q functions by Equation Update the policy by Equation 5
25
26
++=]]
27
end
28
end
second term B(Q), and the updating function can be presented by the following equation:
( (s. -B
+ + arg min
E. Bs Dow..
Q
(3)
+ W(s, a) Doom
log exp(((s, - EarDowc 1Q [()
where the action à sampled from current policy, i.c. ~ TT(-)s), ~ DOORE and DOORB is the OORB replay buffer. - 3s. a) is defined by
r y min Q (s', a'), a' ~ To (+)s)). SeM
(4)
We randomly select two Q functions from the ensemble Q functions and the M represents the selected Q functions' index just following the REDQ's setting. The Q stands for target Q function for stabilizing the learning process (Mnih et al. 2015). The update function of policy is defined by:
Tov f- arg max EiEN (s, a log mox (a s)] a ~ Tox (. s)
(5)
When sampled data is from the online replay buffer, we set W(s, a) to O, otherwise 1, and this is why the term conervvvnnsrvv to describe our algorithm. In another word, greedy exploit the near-on-policy online data by the regular online RL scheme without any conservative regularizer. On the contrary, conservatively exploit the offline data by employing the offline RI.

---

Under review as a conference paper at ICLR 2022
Environment
GCQL (Ours)
CQL
REDQ
AWAC
walker2d-randon
53:227
1611
1913
12
hopper-random
84--40
11+1
12+17
63
halfcheetah-random
100+2
46
34
53
walker2d-meding
94
8341
5+3
80
hopper-medine
105+1
100
3+1
halfcheettah-medium
66:13
42:50
46
41
walker2d-expert
117=2
11311
6+1
103
hopper-expert
114:1
113=1
12#6
112
halfeheetah-exxpett
110::0
108+:
1+:0
106
11714
11311
1213
78
hopper-mediim-eepee
115+11
114+0
40+15
112
halfcheeuah-mediuu--eye
107+1
95++2
9+:3
41
gulker2ddmedium-reee
11416
64155
53111
Sroper--edium-reee halfcheetah- hatthttttta-mmmddu--eeelpaaa
96:1-9
3944
38:9
59-12
50:0
50+1
Table 1: Performance of policies trained using 90K online interaction steps for GCQL,, COL and REDQ, and 500K online interaction for AWAC whose results are taken from Nair et al. (2020). captures the standard deviation over seeds. The reported results are the average test performance across five random seeds in our experiments. The learning curves are showed in Appendix
regularizer. Formally, this strategy can be explained by Definition
(ss,)) + {
if (s, a) is sampled from the online replay buffer
(6)
otherwise
Algorithm Bannhhmmmmmmmaa summarizes our proposed method. And we also explain the main steps in the Algorithm as follows. Firstly, We first learn from the existing offline data for Tisstiat steps to leverage them. To make good use of the offline data, we usually a big value for Timitials e.g., 100K. Secondly, We begin the following interleaving learning steps. We conduct the online exploration process for Ton. steps and store the new experiences OORB. We set the online exploration steps Ton small value, e.g., 1K. In contrast, the offline update step Toff is set to be larger than Ton e.g., 10K. We sample batch from our OORB and update the policy and Q-functions. the sampled batch comes from the online buffer, then we set the weight value W(s, a) to 0, otherwise 1. The above interleaving learning process is repeated till the end.
EXPERIMENTS
In this section, we design experiments verify the effectiveness of method from three perspec- tives: (1) the performance superiority compared with other baselines; (2) ablation studies to test the effect of each component used in our method. (3) the influence of different param--aramesers.
5.1 SETTINGS
All experiments were done on the continuous control benchmark MuloCo (Todorov et al. 2012) and the offline dataset comes from the popular offline RL benchmark D4RL (Fu et al. 22200) To make interaction limited, we set the number of online interaction steps for each iteration to a small value, i.e., KK. To better exploit the offline dataset, we set the number of offline training steps to large value, i.e., 100K for T Tintial, and 10K for Toff- The training process ends until the number of all online interaction steps reach 90K, and the number of all offline updating steps reach 1M. For OORB, we set p 0.5 as described in Section 4.3 The size of the online buffer is set to 20K, and the size the offline buffer is set to 3M. For other setting in baselines, except for the number of the ensemble Q, which is 5 in experiments. The above configurations keep same across all tasks. As our main purpose is to present a new framework instead of gaining SotA performance, do not fine-tune these hyper-parametees, for each task.
we follow the default

---

Under review as a conference paper at ICLR 2022
120
Namalbod Spare
Namalizd Spore
Norolized Store
140
75
240
40
8
Environment Steps
Ted
Ervirone and Steeps
Twe
Enviromment Stace
(a) walker2d-expert
(b) wallkerdd-medium-exper
(c) walker2dd-medium-rrpp
120
WO
OCOLLWE WE
Normatazd Scort
202
Nornalzzed Score
OOOL_WO
WC
80
800
40
3
6
Envtronment Steps
Emvironment Steps
(d) walker2d-medium
(c) walker2d-random
Figure& 3: Ablation study on Walker2d task.
5.2 OVERALL PERFORMANCE
4 methods, i.e., GCQL (Ours), CQL, REDQ and AWAC, are tested on 3 tasks, i.e., Walker, Hop- per and HalfCheetah, and each task has 5 different kinds of offline dataset, which are random-vO, medium-vO, expert-vO, medium-expert-vv and medium-replay-vo, As online interactions are avail- able, we take the maximization over testing scores during the whole training process, and report the average of these max scores across five different seeds in Table [ Please note that the results of AWAC are directly taken from their paper (Nair et al. 2020). As shown in Table [] our method gains a better performance than the basclines in all tasks. Particularly. for the medium-replay and random dataset, our GCQL outperforms the baselines by a large margin. By contrast, CQL also achieves an expert performance on the high-quality dataset, e.g., expert and medium-expert tasks. For these high-quality datasets, our method achieves a comparable or slightly better performance over CQL. In terms of the online RL method REDQ, it fails on almost all tasks. Even though we employ an online limited exploration process, conventional online RL cannot benefit from such limited experiences. On the contrary, such limited online experiences could bring catastrophic consequences, e.g., on the task, REDQ suffers significant instability issue. On the other hand, when dataset quality is high, e.g., expert or medium-expert dataset, our method can leverage the benefit of the offline RL that gains an expert performance with very limited interaction steps, such as around 20K steps. By contrast, when dataset quality is poor, e.g., the random or medium-replay dataset, only our method can learn effectively with less than 100K online steps, which demonstrate that our method can take the advantage the limited online experiences as much as possible. In sum, our method can benefit the most from the limited online experience while still maintain the offtine learning's 's ability. Besides, We also include the corresponding learning curves in Appendix A for comprehensive understanding. These learning curves demonstrate that our method also achieves a better final average score at the last iteration in most cases.
halfcheetah halfchectah-meeium--tee
5.3 ABLATION STUDIES
To investigate each component's 's effect in our method, we conduct the following ablation studies. To do this, design four variants of our method. GCQL.WE: GCQL without the ensemble where we only use two Q functions same as the setting of CQL. GCQL WO: GCQL without the online replay buffer where we only sample from the offline buffer. GCQL WG: GCQL without the greedy
8

---

Under review as a conference paper at ICLR 2022
strategy where we fix the weighed W to 1. GCOL.WC: GCQL without the conservative term where we fix the W to O.
We test all 5 kinds of offline dataset for the task Walker2d, and results averaged over three differ- ent random seeds are shown in Figure 3 First, from Figure 3 it is easy to deduce that when the quality of data is high, i.e,, including the expert dataset, almost all variants perform well except the GCQL_WC The underlying reason is obvious as we firstly pre-train the policy by offline data. If no such conservative restriction, then the pre-trained policy would suffer serious distribution mis- match issue (Kumar et al. (2020)) and gain poor pre-trained policy at last. By contrast, with the conservative scheme, relatively good pre-trained policy can be obtained. That why the starting point of GCQL WC is much lower than other variants.
Secondly, when the dataset is diverse, which means that the dataset collected by different behavior policies and includes data from different distributions, such as the medium-replay dataset, the greedy scheme (including the offline-online replay buffer and the greedy update strategy) plays an important role while the ensemble feature seems have limited influence. For instance, the performance declined significantly for GCQL. and GCQL WG on walker2d--ediiu--rplayy
Thirdly, when the dataset is not diverse, such as the medium dataset which is collected by one medium policy, almost all curves grow slowly except GCQI._WC. This may be caused by the charac- teristic of the dataset, because dataset diversity plays important role for policy learning (Agarwal ct al. 2020). However, according to the learning curves, our method still has a clear performance improvement at the latter learning stage. For the random dataset, all variants fail due to the poor quality of the dataset. Instead, our method achieves a clear performance improvement under it, which indicates that every component is important in this case.
5.4 ANALYSIS ON HYPER-PARAMETER
As we do not fine-tune the hyper- parameeerssss in our experiments, one may wonder how the hyper- parameters affect the performance. To this end, we conduct experiments to investigate their influ- ence. The detailed learning curves are shown Appendix B As one main characteristic of our method is limited interaction, we fix the online exploration step as 1 KK for each iteration in the Mu- JoCo benchmark. We then try settings with different initial update steps Timstial, offline update steps Tors. and sample possibility from online buffer p. Specifically, Timitial is tested with 2e5 and 5e4; Tofs is tested with 2e4 and 5e3; p is tested with 0.3, 0.4, 0.5, 0.6 and 0.7.
According to the evaluation results, we may conclude that the performance of our method is insen- sitive to the Timttaal and Tors. especially for datasets with not poor quality, e.g., datasets except for the random one. By contrast, p has a bigger impact on the performance. Particularly. methods with higher performs better on the medium and medium-replay tasks, while the ones with lower perform better the other tasks. That indicates that when dataset quality is very good very bad, methods with more offline updating perform better. On the other hand, only the variant of p = 0.5 can achieve a clear performance improvement in both the medium and random datasets. Overall, the default setting p = 0.5 is the most suitable setting, which indicates that taking the online and offline data equally important may be the best option in most cases.
6 CONCLUSIONS
This paper first discusses the disadvantages of online and offline RL, and the shortcomings of current offline-online RL methods. Then, considering that offline and near-on-policy online datasets are both crucial for policy learning, we propose a unified framework that can adaptively and effectively take advantage of both offline and online data. Furthermore, a practical algorithm based on the framework that greedily exploits the online experiences and conservatively exploits the offline experiences is presented. We conduct comprehensive experiments to verify the effectiveness of our method. In terms of the shortcomings, although our framework can take advantage of the offline and limited online dataset. the quality of the offline dataset still has big impact the performance. For instance, our method is relatively slower when learning from the random dataset. At last, we hope this work could contribute to bridging the gap between the practice and DRL. research.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. ICML, 2020.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double qr learning: Learning fast without model. ICLR, abss2101 1oss//201.0988,,. 2021.
Christoph Dann, Gerhard Neumann, Jan Peters, et al. Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Research, 2014.
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning, 2021.
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, 2018.
Scott Fujimoto, D. Meger, and Doina Precup. Off-policy deep reinforcement learning without ex- ploration. In ICML, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01299, 2018.
Qiang He and X. Hou. Popo: Pessimistic offline policy optimization. ArXiv, abs/2012.1365) 2020.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-polic batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 0907. 00456, 2019.
R. Kidambi, A. Rajeswaran, Praneeth Netrapalli, and T. Joachims. Morel Model-based offline reinforcement learning. NeurlPS, abs/2005.0595 2020.
Beomjoon Kim, Amir massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. NeurlPS, 2013.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. NeurIPS, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble. arXiv preprint arXiv:2107.00591, 2021.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. ArXiv, abs/2005.01643, 2020.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, volume 1077, 2014.
Tatsuya Matsushima. Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. ICLR, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (7754)):55995533 2015.
10

---

Under review as a conference paper at ICLR 2022
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09359, 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1899-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 17Xiv:/777.0637,,, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2011.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.088/1 2017.
Y. Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911.11323 2019.
Tianhe Yu, G. Thomas, Lantao Yu, S. Ermon, Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. NeurlPS, abs/2005. 1ttp/22055.1339,, 2020.
Henry Zha, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. 2019 Inter- national Conference Robotics and Automation (ICRA), pp. 3651-3657. IEEE, 2019.
A LEARNING CURVES FOR ALL TASKS
Figure indicate the whole learning curves for GCQL, and CQL.
B EXTRA EXPERIMENTS ON HYPER-PARAMETER,
Figure 5 and 6 shows the different prper-parameter' setting the performance.
C EXTRA ABLATION STUDY ON HALFCHEETAH
Figure present extra ablation study on halfcheetah. GCQL_WG the GCQL without the greedy update scheme but still with the onlinc-offline two-level buffer, while GCQL_WGO is the GCQL without the greedy scheme and online-offline replay buffer. From Figure cd,,e it clear that the greedy scheme plays important role in boosting the performance when the dataset is not optimal or near-optimal. On the other hand, from Figure d, we can see the online-offline replay buffer is crucial for stabilizing the learning process where the GGQL_WGO and CQL suffer serious stability issues which may be caused by extrapolation error (Fujimoto et al. 2019)

---

Under review as a conference paper at ICLR 2022
CQL
GCOL
100
Nomelize Store
80
Narmalzadd Siote
40
80
40
60
20
40
Envirormeent Steaze
Twe
Ervironmeett Steps
Ted
Envi ammment Stups
Teel
(a) walker2d-random
(b) hopper-random
(c) halfcheetah-random
100
83
60
Normalzed Score
Nomalized Score
80
80
Nomaloor
80
80
40
80
30
20
20
Environment Steps
Environment Steps
Environment Stepe
(d) walker2d-medium
(c) hopper-medium
(f) malfchectah-medium
125
Nomeltzed Soore
100
100
Nomaized Soore
80
Nomaiieed
60
10
20
Enwironment Steps
104
Enwironment
1o4
Environment Steps
104
(g) walker2d-expert
(h) hopper-expert
halfcheetah-expert
120
120
Nomaltzdd Soore
100
Normalized Soore
80
80
Normelized
%
8
10
18
Enwiromment Steepe
Twel
Envirommment Steepe
Envi crmment Steps
walker2d-mediumeee
(k) hooper- medium-
nalfcheetahmmedium expert
120
Nommlized Score
100
uhpll
Normalizedd Score
8
row Whl Mart
Normllzed Scora
50
80
60
80
30
80
Lnwrronment Steps
104
(m) walker2dd-medum--eplla
(n) hopper-medium--rplay
(o) halfcheetuh-mediuu--eplay
Figure 4: Training curves D4RL continuous control benchmark across five random seeds. The
shaded areas represent the standard deviation across different seeds.
12

---

Under review as a conference paper at ICLR 2022
Nomated State
100
Nommalan Statn
771
80
Nornakaad
50
80
25
40
20
-20
two
Environment Stepps
Environment
(a) walker2d-expert
(b) walker2d-meddum-expprr
(c) walker2d-mediun--rplh
120
30
GOCL fott
Normatzed Stote
80
Nomelliadd Beore
80
GOGL- -de3
GCCL-c004
GOOL
40
20
1wd
(d) walker2d-medium
(e) walker2d-random
Figure 5: Different offline update steps walker2d task across three random seeds.GCQL
i2e5:initial steps is 2e5, GCQL-i5e4: initial stesp is 5e4, GCQL-02e4: offline update steps is 2e4, GCQL-05e3: offline update steps is 5e3.
125
Normllzed Spore
Nomdlace Soore
100
Normslized Score
103
80
25
Environment Steps
504
Environment
1o4
Environment
104
(a) walker2d-expert
(b)
(c) walker2d medium-replayy
GCOL-033
100
CCQL-04
Nomalized Scote
Normalzad Store
GCOL-00
6CQL.07
60
CCOL-0.5
2
20
Enviroement Steps
Sef
Environment Steps
let
(d) walker2d-medium
(e) walker2d-random
Figure 6: Different possibility setting on walker2d task across three random aeeds.GCQL-X means the sample possibility from online buffer is X.
D EXTRA EXPERIMENTS ON TD3+BC
Extra experiments on TD3+BC (Fujimoto & Gu 2021 (without states normalization) for the sub- optimal dataset, i.e., random, medium, and medium-replay, the offline RI method per-
13

---

Under review as a conference paper at ICLR 2022
100
Nornsaadd Store
100
Nommllae Score
80
Normakaed Spate
30
80
16
20
Environment
565
Environment
105
Environment Steps
105
halfchectah-cxpent
(c) halffhhcttal-meciu--ryyy
70
CGL
Noma Scare
00
Nommllzze Score
100
GGDI WiG
CCOL
GCOLWW
-00
80
60
60
20
0.0
0.2
0.4
0.5
3.5
1.0
0.0
0.2
0.4
0.6
0.8
Environment Snepa
Environment Steps
1e5
(d) halfchectab-mmddim
(c) halfchectab-raadonn
Figure 7: Extra ablation study on Halfcheetah.
form very well on expert or near-on-expert datasets. We simply apply our greedy-oon learning framework and online-offinee buffer to TD3+B without any fine-tuning and call it: TD3BCGC. The learning curves indicate that prove volicy-penalty--asee methods by a large margin most tasks. Moreover, is very framework to TD3+BC where less than 20 lines codes needed. We modify the policy objective from = argmax,, (ssa)) [AQ(s, ((s)) - (T(s) - a) to i = argmax,
creddy-conservative framework can still im-
simple to apply
duced in (s.))) [XQ(s, 7 (8)) W(s, (T((s) - a)"]. where W(s,a) a) follows our setting intro- section 4.4.
E EXTRA EXPERIMENTS ON OFF2ON
Here, we compared our method with OFF2ON (eee al. 2021 which also employs an online update scheme to fine-tune a pre-trained agent. As the author did not provide the pre-training code for CQL,, we use the pre-trained CQL agents provided by OFF2ON's author for online fine-tuning. All the hyper-parameter follow the default setting except the online interaction steps. Figure indicated the whole learning curves. From this figure, we can see GCQL has a comparable or better performance than OFF2ON. It is worth noting that OFF2ON fine-tuning their method, for instance, the critic's s neural network architecture is different from the original CQL paper. In contrast, we did not fine-tune these parameters, and all settings are the same for all tasks.
F EXTRA EXPERIMENTS ON OFFICIAL ONLINE REDQ
In this section, we conduct extra experiments on the online REDQ (Chen et al. 2021 To guarantee the reproduction performance, we use the official code from the author, and all hyper-parameters are following its default setting, e.g., the number Q is 10, and the utd-ratio is 20. In contrast, the number of Q in GCQL is 5, and utd-ratio is 10. REDQ-ONLINE updates its policy following a conventional online RL setting, i.e., learning from scratch without the offline pre-training. From Figure I0 it is clear that GCQL achieves better comparable performance than REDQ-ONLINE except for the walker2d-random dataset. Specifically, when the dataset is good, e.g., including the expert dataset, GCQL outperform the REDQ-ONLINE by large margin. On the other hand,
14

---

Under review as a conference paper at ICLR 2022
20
TOXII TD960OC
50
70
Noraatzed Scote
15
Nommatrd Seat
Nomalizad
80
40
Stepp
fws
(a) walker2d-random
(b) hopper-randos
(c) halfcheetah--random
120
120
80
Noraatedd Stote
Noymalodd Stoues
101
Nomellzed
-0
40
20
36
-20
0.0
Steps
Steps
(d) walker2d-medium
(e) hopper-medium
(I) malfchhetah-mediiue
100
Normatzed Stote
Narmalnn Stom
100
40
101
Nomellzad
20
Steps
(g) walker2d-meddiun--epla
(h) hopper--ediium-rpplee
(i) baalchhetaa--mediin---pppp
Figure 8: Training curves for TD3+BC D4RL continuous control benchmark across three dom seeds on tasks: random-vO, medium-vO and medium-replay-vo. TD3BC means the algorithm introduced by (Fujimoto & Gu (202I) while TD3BCGC the variant with our greedy- framework and maline-buffee replay buffer.
when the offline dataset is poor quality, i.e., random-datased GCQL can still learn a comparable or better policy than REDO-ONLINE ONLINE in two of three tasks, i.e., the hopper-random and halfcheetah- random.
15

---

Under review as a conference paper at ICLR 2022
OFF2ON GOOL
80
100
Nornelized Score
Norndlaad
20
000
40
Environmant Stupa
Text
Envioonmnnt Stepa
Environment
Ted
(a) walker2d-random
(b) hopper-random
(c) halfchectah-
100
100
Nornalaeed Score
Namnllzad Seore
Namallad Btam
180
80
16
80
45
20
Ensironment
Text
Envioonmntt Stepa
Environment
(d) walker2d-medium
(c) hopper-medium
(f) hawfchoctah-
120
100
Nornslezd Scaro
Nomelloa Sozea
(00
Nomallod
Awj
Hul M
50
40
20
Ensironment
Enviconment Stepas
Environment
(g) weledd-mdiiuu--ppe
(h) hopper-medium medium-replay
(i) saffcheetah-mediinee medium-replay
Nomalood Soare
Nomelloa Sczee
Nomellood
60
50
25
Enviconment Stepps
Envirorment
Twt
( walker2d medium- expert
(k) hhophh--medium -expert
(I) hawfchettah- expert
Figure 9: Training curves for OFF2ON Lee al. 2021) D4RL continuous control benchmark
across four random seeds on tasks: random- v0, medium v0, mediim-expprr--vO and cediummreppay- vO.
16

---

Under review as a conference paper at ICLR 2022
OCL
00
GOOL REDOOOM OWLOOONLLWE INE
100
100
Noraatzed Spore Spore
Nomel/ad Stoke
(00
-80
-00
Nomalized
00
40
20
20
Steppe
two
Environment Stepps
Environment Steps
(a) walker2d-random
(b) hopper-random
(c) halfcheetah-randomm
1143
60
Nornatzed Stote
Nommalred Stamn
091
Nomelized
40
40
-3
20
twe
Environment Stepps
Environment
Twe
(d) walker2d-medium
(c) hopper-mediune
(I)
125
120
Normatzed Stote
100
Nomellzed Bocre
Normalno Stoom
100
75
800
00
25
20
ted
Stepss
(g) walker2d-expert
(h) hopper-expert
(i) hallcheetab--expet
120
120
Nomallzzd Score
Normallo Stomm
Nomelized
00
40
1ed
Environment Steps
(j) walker2d-mcdium- wwl--mmddddierrtteeeeme
(k) hopper-medium-expor
(l) sittchhctta-mmdiumee expert
120
60
Noraatzed Stote
120
1000
Stowe Stevie
300
199
80
Nomalizad
-0
40
20
20
1ed
Steps
(m) walker2d- mralu
(n) hopper-medium-rreple
(0) salfichettah-mddium-rco
Figure 10: REDQ-ONLINE learn from seratch without the offline pre-training while GCQL and
CQL learn from both offline dataset and online interaction.
17


Reviews:
[{"review_id": "Y8px7UNV0Ma", "reviewer": "Reviewer_irZQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Poor writing. Inadequate experiments design. This paper, at its current status, is not ready to be submitted to conferences like ICLR. \nActionable feedback for the authors might be rethinking the storyline of this work.", "summary_of_the_paper": "This paper presents a framework called adaptive Q-learning that integrates the advantage of offline learning and online learning. ", "summary_of_the_review": "see above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O1PzZV75LFC", "reviewer": "Reviewer_qMWG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper studies an important problem of offline-online RL where the algorithm has access to both offline data and a limited amount of online interactions. The authors empirically show the disadvantage of using a traditional approach of simply fine-tuning on the online data, as most offline RL algorithms like CQL will behave too conservatively. The approach, while simple, makes sense intuitively as it has the desired property of behaving pessimistically according to the offline data but greedily on the online data. \n\nThe proposed algorithm GCQL combines two specific offline RL algorithms CQL and REDQ. The central premise of the paper, to only act conservatively on the offline data, is a general concept that is agnostic to the specific training objective. I feel that the paper could be made more general by considering other forms of pessimism. For example, I would be interested in seeing if similar performance gains can be obtained by using a policy constraint penalty i.e. the penalty in TD3 + BC, rather than the value penalty in CQL. That way, the paper could be made more general by giving the practitioner the flexibility to use the best base learning algorithm for their specific problem domain. \n\nOne concern is that the performance gain of GCQL over GCQL-wg (which I interpret as effectively being CQL with ensemble Q-learning) appears to be very marginal. The only exception is when the offline dataset is random or highly suboptimal. In practice, it is common to assume that the dataset was generated by a decently-performing policy, so it is unclear if the novel training objective will yield much improvement. I think the paper would benefit from a more nuanced characterization by measuring the performance gap as a function of the suboptimality of the behavior policy.", "summary_of_the_paper": "The paper proposes GCQL, a new RL algorithm that is trained on a mixture of offline data and online interactions. The novel component of the algorithm is a reweighting that balances acting pessimistically on offline data and greedily on online interactions. The authors propose a mixture replay buffer that consists of both offline and online samples. For online samples, the policy is trained as in the REDQ algorithm; meanwhile, for offline samples, the policy is additionally trained on a CQL-like value penalty. Finally, the authors show that GCQL outperforms existing SOTA offline RL algorithms that simply fine-tune on online data. \n", "summary_of_the_review": "Overall, though the proposed algorithm seems like an incremental change over existing ones, it clearly is preferable to use over existing methods that simply fine-tune on the online data. I believe that the realization that using the same training objective for both offline and online data yields overly pessimistic policies is an important contribution, and opens the door for future work that considers different objectives for offline and online data as this paper does. Because of this, I recommend that the paper be accepted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JCEmdwY_Wjr", "reviewer": "Reviewer_tsK8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are two distinct contributions in this paper. The first is in the proposed training with experience replay approach, and the second is the propose algorithm which combines elements of CQL and REDQ. Unfortunately, I'm not sure the experimental results adequately explore either of these contributions. What the results show is their method doing better in some cases when using their training approach, but this approach itself is not validated by these results. Why not compare to other methods mentioned in the intro, such as Lee et al. 2021, using the training procedure they were designed for? The authors highlight differences between what they propose and the related work, but not all of those differences preclude comparing between the mentioned offline-online methods and the propose method. The results here don't allow us to determine whether this approach improves on existing approaches to offline-online RL.\n\nAs for the other main contribution, it's equally difficult to assess the value of this new CQL/REDQ hybrid because it is only compared using the proposed training method which might disadvantage the methods used as comparison. This wouldn't be an issue if the training method was shown as an important or superior approach for doing offline-online RL, but, as I argue above, I don't believe that the authors have provided any evidence for this. For instance, looking at REDQ's results in table 1, it seems like this training regime causes it to underperform based on the results the REDQ paper report given the same number of interactions (I could be mistaken given the results in this paper use a normalized score who definition is unknown to me). \n\nIn terms of writing, the paper contains many typos and incorrect word choice. I don't think this prevented me from understanding what was meant but this paper would benefit from some careful proofreading. Otherwise, the paper is well structured and mostly easy to follow.\n\nMy concerns about the experimental results make me doubt that this paper is ready for publication. The authors should make sure that their claim, that this training approach is a superior way of doing offline-online RL, is properly validated by the experimental results. It isn't sufficient to just propose a method that works better when using this training approach.\n\n# Questions:\n\n- Is my understanding correct that newly collected samples are added to both the online and offline buffers?\n\n- Is the initial offline dataset not used to initialize the offline buffer?\n\n- p. 7, “we report the max average score as performance”, this is ambiguous. As I see it, we have different points along the x-axis and different seeds. What are taking the maximum over? What are we averaging over?\n\n\n# Comments:\n\n- p. 4, it’s difficult to appreciate this illustrative example without some more context. How was the offline dataset generated? How are the offline and online datasets used, are they just concatenated? How many seeds were used and what are do the shaded areas represent? Why would it be fair to say that CQL failed if it was able to improve, albeit slowly? Why is this not due to the starting performance being already quite good? What kind of performance should we be expecting in this domain?\n\n- p. 5, the term C should be defined here. I could not confidently find it in the cited work, and, regardless, it is central enough that is should be restated here to avoid any possible ambiguity.\n\n- p. 5, “we randomly select two Q functions just following the REDQ’s setting”, this is unclear. Eq. (4) shows only a single action-value function or index. What is the meaning of the hat over this action-value function? One of the major contributions is proposed combination of parts of CQL and REDQ. I expect this terms to be explicitly defined here.\n\n- p. 6, eq. (6), this isn’t a proper definition for a function. As I understand it, nothing prevents identical state-action tuples from being present in both buffers. The output of this function given that (s, a) is indeterminate. This function definition requires another argument.\n\n\n\n# Minor comments and nitpicks:\n\n- p. 1, \"pre-existing offline data can prevent agents from converging prematurely due to the potential diverse”, the potential diverse? typo?\n\n- p. 2, \"the agent can achieve an expert policy using very few online interaction steps regardless of the quality of the offline dataset”, how can this possibly be true? Wouldn’t a degenerate dataset mean that all the learning needs to be done online, which would like require more than “very few” interactions?\n\n- p. 3, later in the paper, policies are treated as probabilistic, e.g., eq. (3). The policy definition provided here is a deterministic mapping of states to actions.\n\n- p. 3, \"a corresponding value function”, this probably should be called the action-value function, not the value function as the term is usually used to refer the state-value function.\n\n- p. 3, “off-policy RL algorithms maintain a growing replay buffer”, this is true only at the start of learning. Buffers are typically kept at a fixed size max size.\n\n- p. 6, algorithm 1, the variable t is not defined and its value is never changed.\n\n- p. 7, \"The entire offline-gradient-update step is about 1 million, and the entire online exploration step is around 90K”, 1 million what? What are we counting, what are the units?\n\n- p. 7, table 1, what do the +/- ranges represent?\n\nPost rebuttal\n==============\n\nThe authors have improved the empirical evaluation and have mostly addressed my biggest concerns. I have increased my scores to reflect this.\n", "summary_of_the_paper": "The authors propose a mixed offline-online RL approach for which they design an algorithm. They propose to maintain 2 separate replay buffers, one for online data and one for offline data, to allow them to sample either an online or offline batch of data when doing an update step, and tailor the loss function based on the batch's provenance. In addition, the authors propose using a CQL variant which uses an ensemble of action-value function as done by REDQ to learn in this setting. They conclude by showing some empirical results on the D4RL Mujoco benchmark domains.", "summary_of_the_review": "The authors shows that the proposed method performs better under their proposed training regime but don't provide any supporting evidence for this training regime. This raises some concerns about the significance of the empirical results, and the usefulness of the proposed regime. These concerns prevent me from recommending acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "56sR7wcCboW", "reviewer": "Reviewer_VmYw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nMain points:\n- The topic is interesting and I find that combining both offline and online training is a fruitful direction of research, which is potentially relevant in many practical situations. \n- The main intuition for adaptive Q-learning make sense to me---treating the offline and online data differently, with modified updates for each. \n- In terms of clarity, some parts need to be explained further, such as how the algorithm uses the replay buffers.  See the section below for more detailed questions.\n- The experiments seemed to be done well generally and the performance of the proposed algorithm is substantially better than the baselines in certain settings e.g. medium-replay. It's also interesting to have experiments where the online interactions with the environment are limited. Other ablation experiments and sensitivity tests to hyperparameters are included, which are welcome additions. The ablation studies seem to indicate that the proposed algorithm could be simplified though and I've included a few questions below.\n\nQuestions and suggestions:\n- The description of the online-offline replay buffers and section 4.3 are unclear. What exactly does each replay buffer contain? The online replay buffer says it contains \"online exploration experience\" but then the same thins is said about the offline buffer, \"offline replay buffer consisting of the newly generated online exploration data\". I'm confused about this distinction. Does the offline replay buffer contain more than just the initial dataset?\n\n- Why is it that REDQ fails completely on these tasks? \nOther variants of Q-learning seem to work in batch RL such as the distributional RL algorithms or Random Ensemble Mixture (REM) from \"An Optimistic Perspective on Offline Reinforcement Learning\" (Agarwal et al.). REDQ seems to be fairly similar to REM so one could expect it to work decently well in the offline setting too.\nAlso, looking at the learning curves in the appendix, there doesn't seem to be any improvement even in the online part for REDQ. \nIs the offline training a hindrance to the algorithm? Would it perform better if it was simply initialized from scratch without the offline training? I think this would be a useful baseline to include too.\n\n- CGQL-wo \"without the online replay buffer where we only sample from the offline buffer\". Does this mean that all the experience gets pooled into a single replay buffer? Could clarify the mechanism?\nAlso, does that also include removing the conservative term? The full GCQL would set W=0 when sampling from the offline buffer.\n\n- Related to the previous point, how important is it that the online buffer is much smaller than the offline buffer? \n\n- To clarify, is the difference between REDQ and CGQL-wc (without the conservative term) only that the replay buffers are separated into two? Are all the other hyperparameters the same? Would this explain why REDQ fails but CGQL-wc can achieve decent performance?\n\n- Do you have any idea what happened to CQL in halfcheetah-medium? In this environment, it seems like training more online actually leads to decreased performance over time, strangely enough. \n\n- Looking at the ablations, it seems like the setting CGQL-wg does quite well overall. This would suggest that being more conservative the entire time is not a bad strategy. I'm a bit curious how these results compare to CQL. \n\n- In the ablations, CGQL-we does about as well as the full CGQL in every setting except walker2d-random. In that case, even full GCQL doesn't improve that much over training, only reaching an average performance of around 30 by the end. This would suggest that the algorithm could be simplified while retaining almost identical performance. I'm not very convinced that adding ensembles\nis crucial here. \n\n- Section 5.4. The results should be included in the main text, even if it is summarized. Currently, there are no results to look at. Also, to present the different hyperparameter settings, I would just write that T_off was tested with 2e4 and 5e3 instead of describing the legend for the learning curves. Same for T_initial and p. In the current draft, the legend is out-of-place since the plot isn't in the main text. Also, I would add the label descriptions with the learning curves in the appendix for easier reading. \n\nMinor points:\n- \"Moreover, how to evaluate the policy under the offline setting is also a challenging problem.\" While this is mentioned in the abstract, the paper does not tackle this problem so I think it would be best to remove it.\n\n- I'm not sure if \"framework\" is the proper word to use to describe adaptive Q-learning. I usually expect \"frameworks\" to be more general and adaptive Q-learning seems like a fairly specific template for an algorithm. Personally, I found this a bit confusing and had different expectations when reading the abstract.\n\nVarious typos and awkward sentences are scattered throughout and should be proofread. I'm assuming these can be easily fixed so I haven't counted them as a negative point in my review. Some examples:\n- p.1 \"or by combination with the imitation learning\" -> \"or in combination with imitation learning\"\n- p.1 \"pre-existing offline data can prevent agents\nfrom converging prematurely due to the potential diverse, ...\". Missing words?\n- p.2 \"a well-performed policy \" -> \"high-performing\"/\"strong\"/\"good\"\n- p.2 \"In general, Online RL include the on-policy and off-policy RL\" -> In general, online RL includes on-policy and off-policy RL\"\n- p.5 \"CQL (Fujimoto et al 2019)\". Wrong citation.\n- p.5 \"As explained in section 1, We take the online-offline buffer equally important\". Awkward phrasing.\n", "summary_of_the_paper": "This paper tackles a variation to the offline RL setting, where the agent is allowed some limited number of online interaction steps after learning offline. An algorithm, CGQL, is proposed for this setting and uses the idea that online and offline data should be used in different updates. Experiments on a common benchmark show that this approach can be more effective than standard batch RL methods and ablation studies are also included. \n", "summary_of_the_review": "The problem the paper tackles is interesting, offline RL with limited online interactions. I think this is a nice direction of research that merits further investigation. The proposed approach makes intuitive sense to me and the algorithm, CGQL, has good performance on the benchmark environments. I do have a number of clarification questions and the paper would require some editing to make these clear. Overall, I am currently leaning towards acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors studied reinforcement learning applications that have access to both online and offline data (with limited online interaction though).  In order to handle the mixture of online and offline data efficiently, the authors proposed a new paradigm called adaptive Q-learning, which treats offline and online data differently (as reflected by whether pessimism is implemented or not). The effectiveness of the proposed paradigm has been tested empirically. The reviewers have raised concerns about the sufficiency and significance of the experiments conducted in the paper, and pointed out that the proposed algorithmic idea is a somewhat incremental change over existing ones. The changes the authors promised to make will make the paper stronger.
Decision: Reject