Offline-Online Reinforcement Learning: Extending Batch and Online RL
Authors: Maryam Hashemzadeh
Wesley Chung
Martha White
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Batch RL
Online RL
Offline RL

Abstract: Batch RL has seen a surge in popularity and is applicable in many practical scenarios where past data is available. Unfortunately, the performance of batch RL agents is limited in both theory and practice without strong assumptions on the data-collection process e.g. sufficient coverage or a good policy. To enable better performance, we investigate the offline-online setting: The agent has access to a batch of data to train on but is also allowed to learn during the evaluation phase in an online manner. This is an extension to batch RL, allowing the agent to adapt to new situations without having to precommit to a policy. In our experiments, we find that agents trained in an offline-online manner can outperform agents trained only offline or online, sometimes by a large margin, for different dataset sizes and data-collection policies. Furthermore, we investigate the use of optimism vs. pessimism for value functions in the offline-online setting due to their use in batch and online RL.

Under review as a conference paper at ICLR 2022
OFFLINE-ONLINE REINFORCEMENT LEARNING: EXTENDING BATCH AND ONLINE RL
Anonymous authors Paper under double-blind review
ABSTRACT
Batch RL has seen a surge in popularity and is applicable in many practical scenarios where past data is available. Unfortunately, the performance of batch RL agents is limited in both theory and practice without strong assumptions on the data-collection process e.g. sufficient coverage or good policy. To enable better performance, we investigate the offline-online setting: the agent has access to a batch of data train on but is also allowed to learn during the evaluation phase online manner. This is an extension to batch RL, allowing the agent to adapt new situations without having to precommit to a policy, In our experiments, we find that standard RL agents trained in an offine--online manner can outperform agents trained only offline or online, sometimes by a large margin, highlighting the potential of this new setting.
INTRODUCTION
The batch RL setting has attracted much attention in recent times Fujimoto al. (2019b); Wu et al. (2021): Zhang et al. (2021 2020). In this problem, the agent has to learn a good policy from fixed batch of data, without acting in the environment (Ernst et al. 2005), making it a good fit for many practical applications where only past logged data is available.
This problem setting has proven to be difficult. Empirically, in the case where a behaviour policy has collected the data, batch RI. algorithms are only able to provide policy matching slightly exceeding the original behaviour policy's performance (Fujimoto al. 2019a Kumar et al.) 2019). From the theoretical side, lower bounds exist showing that without strong assumptions on the data distribution and features, it may take exponentially many samples to learn a good policy (Zanette) Chen & Jiang 2019). Otherwise it seems algorithm can only produce the best policy which 1s wel -overrd by state- iiate-actions contained within the batch (Liu et al. 2020).
2020
These findings contrast results in the online setting, where there exist many effective algorithms both in practice (Haarooj et al. 2018 Mnih et 2015)) and theory (Jin et al.) (020a)). As such, natural consider variant of the batch setting, closer online learning. In the standard batch RL problem, the learner observes a pre-collected dataset and outputs a policy, which evaluated by running it on the environment. The main difficulty is the lack of feedback from the environment. As such, the learner is unable to assess the quality of policy it proposes.
We may instead consider a setting where the agent is allowed to do further learning during the evaluation phase, which we call the offline-online RL setting. While a natural extension to batch RL, this setting remains largely unexplored. policy as quickly as possible. While this may involve starting with good policy learned offline as in batch RL, the agent now has the chance to correct its policy by observing new transitions. This can help avoid a major roadblock to offline el.-extrapolation error (Levine et al.) 2020 Fujimoto ct al., (2019b). This issue arises due to the agent encountering states not found the dataset when it is evaluated. Oftentimes, the agent may choose bad actions in these new states, without chance to learn from its mistakes. This implicitly limits the agent's performance based on which states are present in the dataset. In many applications, this problem is unavoidable since cannot expect to collect a batch of data covering all possible situations. In the offline-online setting, the agent sidesteps this issue it can improve its policy at new states.
the offline-online setting, the goal is to learn an effective

---

Under review as a conference paper at ICLR 2022
Compared to online RL, the offinee-olinee agent now has extra information in the form of this dataset. As such, the agent can be expected to perform better since we can jumpstart the policy to higher-level of performance through batch RL. This may be especially desirable in practical applications where one would require the initial deployed policy to perform well. In these settings, starting tabula rasa, with online RL. may be unacceptable solution. From a theoretical standpoint, it has been shown that online learning performance can be improved (Xie et al. (021) when the agent has access to a good reference policy, similar setting.
In this paper, we study the difference between the online, batch and offline-olliee RL settings from an empirical perspective. We look the simplest instance of offline-online RL where we transfer a policy (value function) and a learned representation from the offline phase to the online phase. While we can expect algorithms to perform better in the offline-online setting compared to batch RL as we allow additional learning, as far as we know, no work has delved deeper into the differences. In offline-online RI., the goal is to use the batch of data to learn faster online, similar to the goal of representation learning: learn features that facilitate effective linear updates. However, the representation learned offline will also be limited, and so should continue to be adapted online too. We show that an approach, called Timescceee Networks (TTNs) (Chung al.] 2018), that separate representation learning and value learning, are well-suited to this offline-on onlline-online setting. To the best of our knowledge, there are existing approaches that exploit this offline-online setting learn representation and policy offline, for further updating online. The batch RI. setting focuses only on learning a policy that is fixed and deployed online. Existing works often focuses on unsupervised RL (Schwarzer et al. 202 || Ghosh & Bellemare, (020), where a representation is learned without explicit rewards but the agent can control the data gathering policy.
As a part of this work, we show that there is (a) a significant benefit in comparison with offline methods to allow the policy to continue to adapt online and (b) a significant benefit to use the offline batch in comparison to purely online methods. The offline setting is often characterized by the need for safety, and concerns with allowing the policy to adjust. This work, however, highlights the large opportunity cost with doing so, and suggests that a more suitable path is to consider even small adaptation, such as has been considered for high-confidence policy improvement. On the flip side, also highlights that it is restrictive to only learn online from scratch; if data is available (as it often is), then this can be leveraged to significantly speed learning both the representation and policy.
Further experiments in batch RL measuring the performance to the size of the dataset, the data- collection policy and the amount of training done. Here, we find that larger datasets and a good policy can be critical good results. There no clear trends for the impact of the amount of training. In each environment, different amounts of training are best. Taken together, these experiments highlight the difficulty of batch RI,, where many different factors have to be correct to obtain reasonable results.
1.1 RELATED WORK
There has been quite bit of work considering transferring learning from one setting to another; here, that transfer occurs from learning in an offline data to then learn further online. However, most transfer settings involve transfering only the policy or transferring between related environments, where the agent can control data collection in both environments. There is however some work that has considered transferring objects other than a policy between the offline and online (evaluation) phases. One work has considered transferring options, where options (Sutton et al. 1999) are learned in the offline phase and then leveraged speed up learning during the online phase (Ajay al. 202T). Other directions of research are of a similar flavor, such as unsupervised representation learning, in which the agent seeks learn useful transformation of the observations for control without access to rewards (Schwarzer et al. 2021).
From batch RL, other examples include learning world models Yu al. 2020) for planning or learning uncertainty estimates (Osband et al.] 2018 Jaques et al., 2019) to induce either optimism Lai] & Robbins (1985) or pessimism (Jin et al. 2020b] Buckman et al. 2020 the online phase, either of which may be useful in certain situations. Different problems can also be viewed as part of the offline- online setting such as inverse RL (Ng et al. 2000). where we seek to learn the appropriate reward function a batch of data to optimize during the online phase. Offline-onlin
RL is also related to safe RL (Thomas et al. 2013] Kakade & Langford (202)) where the agent uses a fixed-policy for some time to collect enough data to make policy improvement step with high-confidence.
2

---

Under review as a conference paper at ICLR 2022
From the perspective of offline-online RL, imagine other extensions of algorithms designed for online RL, such as using an offline dataset to jumpstart the meta-learning process of update rules (Javed & White 2019) hyperparameters (Xu al. 2018). Through these connections, we that offline-online RI. is natural setting to study, suggesting new algorithms and relevant in practical situations.
BACKGROUND
We model the environment as an MDP M = (S.A, R. P.7) where S are the states, A are the actions, [0. denotes the transition probabilities, and
R x + the reward function, [0, is the discount factor.
Online RL (Suttoon & Barto 2018 Lai Robbins 1985). this setting, the agent directly interacts with the environment, At each timestep, the agent chooses an action a, and the environment returns the next state $1+1 and reward Tt+1- The agent can adapt to new data and change its policy after every single transition. For episodic task, the goal is to accumulate the highest amount of return in each episode and eventually obtain the optimal policy.
Batch RL (Erst al. 2005 Fujimoto al. 2019b Levine al. 2020). The agent receives dataset D {$;, aj T'. st) f-) consisting of N transitions from an enviromment M. These transitions be collected arbitrarily although they often obtained by running a policy and recording the transitions observed by the agent. The goal of the agent is to use this dataset to learn a policy that performs well in the underlying environment. A key aspect is that once the algorithm has output policy, it can no longer change it.
3 OFFLINE-ONLINE RL
In this section, we outline the proposed offline-online setting. There are two phases: The offline phase and the evaluation (online) phase. In standard batch RL, given dataset D, the agent would learn policy т in offline phase. Then, during evaluation, this policy is fixed and deployed in the environment assess its performance. Likewise, in the basic offline-online setting, policy is learned in the offline phase. The difference is that in the evaluation phase, the policy is allowed to adapt online it interacts with the environment. For generic RI. algorithm that has parameters and associated policy my and an episodic task, we have the following pseudocode in Algorithm
Algorithm 1
RL
procedure OFFLINE- ONLINE RL(0, D))
2: Given dataset D of transitions (s, r,
3: Initialize agent parameters
4: for number of updates do
5: Update using data from
6: end for
7: for N number of online episodes do
D. Evaluate the agent
8: while episode not done do
10: a f- action chosen by Te given s
9:
Get reward and next state > Add sample to the batch data Update using a, (a,,,'') s') (and possibly past samples) Allow additional learning If episode ends, record the total return.
s +- Environment(s,
11:
D + 2+++s (s, "s,arr,8'
13:
end while
15:
end for end procedure
Compared to online RL, the agent now has extra information in the form of this dataset. Thus, it could be expected that the agent perform better in this setting, with the amount of improvement depending properties of the offline dataset given the agent.
3

---

Under review as a conference paper at ICLR 2022
In this paper, consider simple approaches offine--nline RL where learned value function and a representation are directly transferred from the offline phase and the online phase. As we will sec, despite the simplicity, there already major benefits to the approach.
To give concrete example of how to set up an algorithm in this setting, we present the pseudocode for offline-online variant of Two-Timescale Networks (TTNs), in Algorithm 2.
The TTN algorithm splits representation learning and value learning. For the representation, we use a neural network and treat the final hidden layer learned representation. This neural network is trained by using SGD on a surrogate loss, which would be minimizing the squared TD-error in this case (with stop gradient the target value). For value learning, treat the representation as fixed and use fitted Q-iteration (FQI) (Emsst et al., (2055) tailored for linear function approximation, which solves for the parameters using the replay buffer. This may be advantage since the updates can be larger and takes into account all available data at once, unlike SGD approaches, which must make small incremental updates.
Algorithm 2 Offline- Online TIN
procedure TRAINÇ w, 0, L0 .T) Initialize 0, x with Xavier initialization, wo to 0, T to number of updates, and starting state according to the environment.
2:
3: Initialize data set D from a batch of data generated by an arbitrary policy.
4: while 10++1 Ogl 2 e or L S T
Offfine-step 0, ú < Do 10000 gradient descent feature learning using sample (s, a, r's) from
5:
mini-bater data D
6:
Do FQI update on value learning using sample (s, a, from batch data D
7:
t
8:
end while
9:
for N number of episodes do
f- action chosen by given s
11:
b Get reward and next state Add sample the batch data 0, à + Do SGD on feature learning using sample (s, a, r, s') from
S + Environment(s, a)
D < (s, r')
13:
data
d D
14:
if number of steps == 1000 then
W Do FQI update on value learning using sample (s, a, a, s') from batch data
16:
end if end for
17:
18:
end procedure
The difference with (original) online TTN is that, here, the agent first gets to learn the given batch of data. This is done by doing some number of SGD updates from the dataset to learn the features, followed by multiple iterations of Fitted Q-iteration to learn the value estimates. In this way, offline-online TTN has access to both a better representation and value function (policy) when starting the online phase. Compared offline TTN, offline-online TTN is allowed additional updates after the offline phase. In doing so, the agent can adapt to any new data it observes in the online phase and can continue to improve the policy and representation. The complete pseudocode for offline and online TTN and DQN can be found in appendix A.2.2
EXPERIMENTS
In the following sections, we investigate various aspects of the offline-online setting and compare to the more traditional batch and online RL settings. As first experiment, we focus on the batch RL setting and look at offline-online DQN (Mnih et al. 2015), an agent which gets to train first on a batch of data and then perform online updates. We compare its performance algorithm, discrete BCQ (Fujimoto et al.) 2019a). The batch of data collected from a policy which performs well the task and consists of 10 thousand transitions. Offline-online Two- Too-Timescale Networks (TTN) is included as an agent that separates the representation learning and value learning processes (Chung et al. 2018) alternative approach.
representative batch RL
4

---

Under review as a conference paper at ICLR 2022
For all experiments, use 30 runs of each algorithm in the Mountain Car, Acrobot, Cart- pole (Brockman et al., 2016), and Catcher (Tasfi 2016) environments. To tune the learn- the algorithms using grid
ing rate, a, and the regularization coefficients, À for TTN, search with the sweeps a E (10-1., 10-20, 10-2.5 10-3.0 10-35, 10-3.15 10-10) and E (0, 0.0003, 0.001, 0.002, 0.05, 0.01, 2, 5,). To find suitable number of updates for the of- fline phase, the offline-TTY TTN algorithm was assessed for {70, 100, 150, 200} number of updates. Afterwards, the number with the best performance is selected for the offline-online setting well.
MountainCar
Acrobot
1000
1000
DQN ottline- online
TTN offilne-online
800
800
sqúes lenuth
episoces length
Offline DON Offlliee TTN
800
200
200
25
150
175
125
175
episodes
eDisodes
CartPole
Catcher
160
1000
140
120
lenuth
100
800
80
40-
200
01
25
50
125 episodes
100
150
175
200
0
80
100
120
140
episodes
We compare the offline-online setting to batch RL. We that while training only offline does not perform well in any environment alone, allowing learning afterwards can significantly improve the performance of the policy.
Figure 1:
In Fig. I we see that the offfine-online algorithms outperforms the batch RL agents, discrete BCQ (Fujimoto et al. (2099) and offline DQN/TTN, by significant margin. In Mountain Car and Acrobot, lower episode lengths are better while in Cartpole and Catcher, longer episode lengths are better. The learning curves are smoothed by averaging over a window of five episodes. Other experimental details can be found in the appendix.
In CartPole and Catcher, we see that the batch RL agents are unable to learn a policy that achieves non-neglible rewards. In these cases, we see that adding online training steps enables the agent to achieve a decent level of performance after some time. This highlights the importance of letting the agent learn during the evaluation phase, after it is trained on the offline batch. Looking at other settings, even though offline TTN can achieve good performance mountain car, additional training in the online phase still improves performance. In all cases, the performance of the offline-online agent is never worse than only offline training.
Next, we turn to the online RL setting and compare offline-online DQN and TTN with their purely online versions. As an additional comparison, we consider using the offline batch of data to initialize the replay buffer of the algorithms. We can view this is as simple attempt to augment the online algorithms with an available batch of data and expect this to improve the performance the online agent.
5

---

Under review as a conference paper at ICLR 2022
MountainCer
Acrobot
1000
Init
800
DON online
evisodes lenuth
DON offline online
episociss lenezh
TTN online
IIN online
600
400
400
200
290
25
30
15
100
125
150
1/5
200
0
50
75
100
125
150
175
200
episodes
episodes
CartPole
Catcher
160
1000
140
120
100
esisodes
400
60
40
25
50
75
100
125
150
175
200
0 20
40
60
80
100
120
140
episodes
Figure 2: We compare the offline-online setting to purely online learning. We plot learning curves for standard online DQN/TTN, online DQN/TTN initialized with the offline dataset and offline-online DQN/TTN. We find that initializing with the offline dataset is better than the standard procedure but additionally training on is still best.
Fig. [ shows the learning curves for this setting. We see the offline-online agents improve on the purely online agent in every setting. Training offline before the online phase gives a jump-start in terms of the performance at the beginning and, in some cases, this advantage is retained throughout the entirety of training. The buffer initialization strategy also gives better performance across the board compared to the standard online agent and, in Mountain car and Acrobot, leads to a final performance similar to that of the offline-online agents' even if the performance is inferior in the early stages.
These experiments confirms that the offline-online setting can produce better performing agents than either setting alone. Since this setting can naturally occur many practical situations where some data is available beforehand, offline-online RL deserves further study. Specifically, compared to batch RL, there may be no need to restrict the agent to precommit to a policy after training on the dataset. We see that allowing adaptation incoming data during deployment may yield significant benefits.
4.1 LEARNING REPRESENTATION SEPARATELY
We investigate separating representation learning as in Two-Timescale Networks (TTN) or whether end-to-end learning better choice traditionally done.
The TTN algorithm splits representation learning and value learning. For the representation, we use neural network and treat the final hidden layer as a learned representation. This neural network trained by using SGD on surrogate loss, which would be minimizing the squared TD-error in this case (with a stop gradient on the target value). For value learning, we treat the representation as fixed and use fitted Q-iteration (FQI) (Emst et al., (2005), which solves for the parameters using the replay buffer as a batch of data. Since FQI is fairly computationally expensive, we only do this every 1000 steps. See the appendix for more details.

---

Under review as a conference paper at ICLR 2022
2 can focus comparing the performance of TTN and DQN. We find that TTN achieves better performance than DQN in the majority of cases. Notably, TTN seems to produce better solutions in the batch RL setting such as MountainCar, where offline DQN is not able to complete the task (it hits the timeout limit) but offinn TTN able to do so. In the online setting, from Fig. again have that TTN seems to have better performance overall, with larger difference near the beginning of the online phase. As a tradcoff,, since TTN uses FQI (Ernst et al., 2005) solve for the next set of linear weights to approximate the value function in addition to SGD steps to learn the representation, the computational cost of TTN is higher in practice than than of DQN although the asymptotic cost is of the same order. These results suggest that end-to aod-to-end training may not be necessary to achieve good performance on these tasks and, in fact, separating the representation can be superior.
Revisiting figures I aadd
4.2 CHOICES OFFLINE DATASET
In this section, we perform certain ablation studies on the offline dataset. We can expect the properties of the dataset to influence the performance of the agent. For the offline-online setting, we test the impact of the size of the offline batch of data with DQN and TTN. The results are shown in Fig.
MountainCar
Acrobot
500
offline-online 10k
800
TTN offline anline 50k
affiine-online
episodes lenqth
100
DQN offline online 50k
600
evisodes
300
200
200
100
100
0
125
150
175
200
eplsodes
episodes
CartPale
Catcher
1000
200
evisodes length
150
800
100
200
50
100 eDisodes
150
200
0
80
100
120
episodes
Figure 3: We present learning curves for two dataset sizes in the offline-online setting, thousand and 50 thousand transitions for DQN and TTN. The larger dataset provides noticeable boost.
From these plots, In Mountain Car, the difference is more remarkable as the larger dataset enables DQN to start the online phase with good policy. This may be due to better coverage of the state space which makes offline training more successful.
that the increasing the amount of data improves performance expected.
We repeat the experiments comparing the offline-online setting to batch RL (Fig. D with the larger dataset of 50 thousand transitions. The results are qualitatively similar with the larger dataset although the performance of all the algorithms is improved, particularly the offline algorithms.
7

---

that the size of the dataset Under review as a conference paper at ICLR 2022
MountainCer
Acrobot
350
dttlin-conline
TTN offline-online SOK
300
50:s0lss lenoth
250
BCQ sok
episociss lenezh
225
affiine sok
Uffline ITN sük
200
290
150
100
100
25
30
/5
100
125
150
1/5
200
0
50
75
100
125
150
175
200
episodes
episodes
CartPole
Catcher
1000
M
250
200
episodes lenguh
150
600
100
400
50
25
50
75
100
125
150
175
200
0
20
40
60
80
100
120
140
ebisodes
eDisodes
Figure 4: We compare the offline-online setting to batch RL. when a dataset of 50 thousand transitions used. We observe qualitatively similar behaviour as with 10 thousand transitions.
Furthermore, we provide more experiments for the batch RL setting. First, we test two sizes of datasets and two different data-collection strategies: using a near-optimal policy or random policy with dataset sizes of 10 thousand or 50 thousand transitions. Tanse are summarized in table
see that both the quality of the policy and the size of the dataset are important factors to the final performance of the agent. The data-collection policy impacts the results greatly. Often, the random policy does not seem to provide a dataset sufficient to achieve nontrivial performance (e.g. Mountain Car or Acrobot). Looking at TTN Catcher or DQN CartPole, can also be the difference between trivial and decent results. In general, we see that TTN performs better in the batch RI. setting with performance always equal better than DQN's. We hypothesize this may be due to fitted Q-iteration operating on the entire dataset at once for each update instead of stochastic updates as in DQN.
Finally, we investigate offline TTN further and look at how the amount of offline training impacts the performance. Since this setting resembles supervised learning, we may expect a similar trend where performance increases at first and then decreases due to "overfitting".
The results in Table suggest that the previous hypothesis is too simplistic since we do not show any clear trends across environments. In Catcher, performance tends to increase with additional training but, in CartPole, it is the opposite. In Acrobot, the best performance is achieved by either a low number or large number iterations, with the intermediate numbers producing worse results. The ideal amount of training seems to be highly specific to the environment and may difficult to tune for batch RL.
5 DISCUSSION
While batch RL is an appealing setting due to its relevance in many applications, we argue that allowing learning during the evaluation phase-the the offline-online setting-may be more natural
8

---

Under review as a conference paper at ICLR 2022
Mountain Car
DQN
TTN
Size/Policy
Good
Random
Good
Random
10k
1000 (0)
1000 (0)
401 (8.6)
1000 (0)
50k
289 (7.1)
1000 (0)
126 (8.1)
1000 (0)
CartPole
DQN
TTN
Size/Policy
Good
Random
Good
Random
10k
9.4 (1.5)
9.9 (1.7)
10.2 (0.9)
9.5 (0.8)
50k
121.6 (13.2)
10 (1.0)
313.5 (24.8)
106.3 (7.1)
Acrobot
DQN
TTN
Size/Policy
Good
Random
Good
Random
10k
782 (10.8)
1000 (0)
553 (2.0)
999.5 (2.1)
50k
232 (17.4)
905 (7.8)
146 (11.1)
999.3 (2.1)
Catcher
DQN
TTN
Size/Policy
Good
Random
Good
Random
10k
-0.78 (0.021)
-0.73 (0.031)
1.89 (0.19)
0.87 (0.024)
50k
-0.73 (0.032)
-0.73 (0.029)
167 (3.2)
16.6 (1.9)
Table 1: This table contains the results for training on an offline batch of data examining two factors: the size of the dataset (10 thousand or 50 thousand) and the quality of the policy good policy or random policy). The experiments are done for both DQN and TTN. The number in the table is the average of 30 runs with one standard error indicated in parentheses. Entries highlighted in green show settings where nontrivial performance was achieved.
Iterations
MountainCar
CartPole
Acrobot
Catcher
70
201 (7.4)
313.5 (24.8)
249 (30.7)
23.9 (1.3)
100
126 (8.1)
236.2 (16.7)
146 (11.1)
25.6 (1.2)
150
642 (12.0)
135.8 (13.2)
168.8 (32.4)
37.3 (1.2)
200
647 (10.1)
91.6 (10.4)
262.2 (28.1)
167 (3.2)
Table 2: This table presents the results for training dataset of 50 thousand transitions but varying the amount training measured by number of FQI iterations with TTN. The mean of 30 runs is included with one standard error in parentheses.
problem to study. We find that it can be difficult to get good performance empirically with batch RL algorithms, with a common algorithm, DQN, often failing to achieve nontrivial results. On the other hand, in the offline-online setting, the same algorithm can achieve decent performance after modest amount of training. The batch RL problem may be asking too much of RL algorithms since the agent has to commit single fixed policy. Allowing learning afterwards enables quick adaptation to unexpected situations. In practical applications, we are unlikely to be able to collect data for every possible situation, thus intuitive to enable the agent to adapt in this manner.
Compared to purely online training, offline- online RL may be more desirable since we can leverage an available dataset to jumpstart the learning process. real-world scenario, it may undesirable to learn purely online since we would like the initial deployed policy to be high-performing. unlike random policy. Training on an offline dataset one way to start with reasonable performance and keep improving thereafter.
We hope that this paper generates more interest into offline-online RL. We see this setting as being potentially relevant in many real-world problems where a batch of data is available and could be more practical solution than either batch or online RL on their own. We see many opportunities to further investigate this setting. From theoretical perspective, certain work has already started to extend batch RL (Xie et al. 202I), given the unsatisfactory lower bounds (Zanette, (222). We believe the offline-online RL. setting opens the door many interesting avenues of research.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. (OPAL): Offline primitive discovery for accelerating offline reinforcement learning. In International Confer-
ence on Learning Representations, 2021. httpthtt/peeereeeeeeeeee net.eeeeeweeeeeeeeeeeeem.....eee homerrrrrww.nee/rrruuummmm forum?ide
V69LGNJ01IN
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
Jinglin Chen and Nan Jiang. Information-theoretie considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042-1051, PMLR, 2019.
Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International conference on learning representations, 2018.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking hatch deep reinforcement learning algorithms. arXiv preprint arXiv: 191.0779,, 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2063 PMLR, 2019b.
Dibya Ghosh and Marc G Bellemare. Representations for stable off- policy reinforcement learning. In International Conference on Machine Learning, pp. 3556-3565, PMLR, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rxiv:1907.004566 2019.
Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv preprint arXiv: 2019.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement
learning with linear function approximation. In Conference on Learning Theory, pp. 2137-2143. PMLR, 2020a.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? arXiv preprint arXiv:2012. 15085, 2020b.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning, In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Aviral Kumar, Justin Fu. George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 00949, 2019.
Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, (1):4222, 1985.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643. 2020.
Yao Liu, Adith Swaminathan, Alckh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. arXiv preprint vrXiv:2007.08202. 2020.
10

---

Under review as a conference paper at ICLR 2022
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 5 518(7540):529-533 2015.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. arXiv preprint 1003v::880..0330 2018.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, Devon Hjelm, Philip Bachman, and Aaron Courville. Pretraining representations for data-efficient reinforcement learning. arXiv preprint arXiv:2106.04799 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-111 1999.
Norman Tasfi.
Pygame learning environment.
PPGane-LLaarrannng-EELLeoonnee 2016.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Conference on Machine Learning, pp. 2380-2388. PMLR, 2015.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted offline reinforcement learning, 2021. URL https
//openreview .ww. wwt.:eet//ooooe (ote://ooommmmmmMMelell
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. arXiv preprint arXiv:2106.0488 2021.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradien reinforcement learning. arXiv preprint arXiv:1805.09801 2018
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online arXiv preprint arXiv:2012.08005. 2020.
Michael Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, ziyu wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2021. URL https /oopnneview net forum?i 1
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. International Conference on Machine Learning, pp. 11194 11203. PMLR, 2020.
A APPENDIX
The appendix is split into two main sections. Additional experiments are found in the first part and experimental details can be found the second. Within the experimental details, we present the hyperparameters used for all the experiments along with pseudocode for the different settings and algorithms.
A. ADDITIONAL EXPERIMENTS
We repeat the experiments comparing the offline-online setting to batch RL (Fig. I and online RL (Fig. ( with the larger dataset of 50 thousand transitions. The results are qualitatively similar with the larger dataset although the performance of all the algorithms is improved.
11

---

Under review as a conference paper at ICLR 2022
MountainCer
Acrobot
350
dttlin-conline
TTN offline-online SOK
300
e0:s0lss lenoth
250
BCQ sok
episocies lenezh
225
affiie sok
Uffline ITN sük
200
290
150
100
100
25
30
/5
100
125
150
1/5
200
0
75
100
125
150
175
200
episodes
episodes
CartPole
Catcher
1000
250
200
evisedes length
150
100
400
200
100
150
175
200
80
100
120
140
ebisodes
episodes
Figure 5: We compare the offline-online setting to batch RL. dataset 50 thousand transitions used.
A.2 EXPERIMENTAL DETAILS
We present the hyperparameters for all the algorithms and the experiments.
Table 3: This table presents the common hyper-parameters for the TTN and DQN algorithms.
Parameter
Value
Learning rate for TTN
10-3
Learning rate for DQN
10 10-3.75 3.75
Minibatch size for DQN
32
Minibatch size for TTN
32
Minibatch size for offline training
64
Target estimation in FQI
Expected SARSA
for Expected SARSA
0.01
Table This table presents the regularization coefficients and number of offline training for offline- online setting for each environments separately.
Environments
MountainCar 10-3
Acrobot
CartPole
Catcher
Regularization coefficient in FQI
10-3
1
Number of offline training for offline-online setting
100
100
70
200
12

---

Under review as a conference paper at ICLR 2022
MountainCer
Acrobot
Init
DON online S0k
800
episdds length
DON offline online 50k
episociss lenezh
online 50k
..AAA IIN online 50k
600
50k
400
200
25
30
100
125
150
1/5
200
0
50
75
100
125
150
175
200
episodes
episodes
CartPole
Catcher
1000
250
200
length
150
600
100
200
0-
50
100
150 175 200
0
80
100
120
140
ebisodes
episodes
Figure We compare the offline-online setting to purely online learning. We plot learning curves for standard online DQN/TTN. online DQN/TTN initialized with the offline dataset and afffine-onling DQN/TTN. We use a dataset of 50 thousand transitions here and find qualitatively similar results to using 10 thousand transitions, although performance is usually increased.
To generate the datasets for good policy, we ran TTN online algorithm for sufficient number of steps to achieve a good policy and then we stored only the last K Ktsteps where K is the size of dataset. To collect the dataset of 10 thousand transactions, the algortihm was run for 50 thousand steps and for the dataset 50 thousand transitions, 100 thousands learning steps were used.
We limit each episode of MountainCar, Acrobot, and CartPole to 1000 steps. For Catcher, we cap the (undiscounted) sum of rewards to 300 and terminate the episode if the limit is reached.
A.2.1 TRAINING PROCEDURES
We include pseudocode for the training procedures of the three settings considered: batch RL, online RL and offline-online RL.
For generic RL algorithm that has parameters 0 and associated policy :T. we have the following pseudocode
A.2.2 ALGORITIMS
We provide pseudocode for TTN and DQN in the different settings here, along with their hyperpa- rameters.
13

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Online RL
1: procedure ONLINE RL(0)
2: Initialize agent parameters
3: Initialize state by the environment for number of episodes do while episode not done do
4:
5:
6:
f- action chosen by Th given s t Environment(s, a)
7:
D> Get reward and next state
8:
Update using (s, a, r, s') (and possibly past samples If episode ends, record the total return.
9:
10:
end while
11: end for
12: end procedure
Algorithm 4 Batch RL
1:
procedure BATCH RL(O, D)
2: Given dataset D of transitions (s, a, r'
3: Initialize agent parameters
4: for number of updates do
5: Update using data from D
6: end for
7:
for number of evaluation episodes do
b Evaluate the agent
8: Initialize state by the environment
9: Run the policy until episode terminates
10: Record the return for the episode
11: end for
12: end procedure
Algorithm 5 Offline-Online RL
1: procedure OFFLINE-ONIA RL(A, D)
2:
Given dataset D of transitions (s, a, r,
3: Initialize agent parameters
4: for number of updates do
5: Update 0 using data from D
6: end for
7:
for number of evaluation episodes do
> Evaluate the agent
8:
while episode not done do
9:
action chosen by mo given
10:
r +- Environment(s, a)
D Get reward and next state
Update 0 using (s, a, r, s') (and possibly past samples) episode ends, record the total return.
Allow additional learning
12:
end while
14:
end for
end procedure
14

---

end for Under review as a conference paper at ICLR 2022
Algorithm 6 Online TTN
1: procedure TRAIN( w. W. x)
2: Initialize 0, i with Xavier initialization, w to 0 and starting state according to the environ-
ment.
3: Initialize data set D = +
4: for N number of episodes do
5:
action chosen by given
6:
T. 8 + Environment(s, a) D <- (s, r,5))
P. Get reward and next state Add sample to the batch data 0, i + Do SGD on feature learning using sample (s, a, r, s') from mini-batch data
7:
8:
d
D
9:
number of steps == 1000 then
10:
Do FQI update value learning using sample (s, a, r,) from batch data D)
11:
end if end for
13:
end procedure
Algorithm 7 Offline TTN
1: procedure TRAIN( w, ú. x)
2: Initialize 0, w with Xavier initialization, to to 0, T to number of updates, and starting state
according the environment.
3: Initialize data set D from a batch of data generated by an arbitrary policy.
4: while 10c+1 0g| 2 e or S do
Offline-step 0, ú <- Do 10000 gradient descent feature learning using sample (s, a, r,,') from
5:
mini-batch data D
6:
Do FQI update on value learning using sample (s, ",,,,,,5), s') from batch data D
7: end while
t + 9: for N number of episodes do
8:
<- action chosen by given
11:
r,&' +- Environment(s, a)
D Get reward and next state
end for 13: end procedure
Algorithm 8 Online DQN
procedure TRAIN( w, 20 T)
2: Initialize with Xavier initialization, w to and starting state s according to the environment.
3: Initialize data set D + for N number of episodes do
4:
5:
< action chosen by given s
6:
D. Get reward and next state Add sample the batch data f. à Do SGD feature learning using sample (s, s') from mini-batch data d D
8' <- Environment(s,
7:
D <- (s, a, s')
8:
9:
number steps
= 1000 then
10:
Do FQI update value learning using sample (s, r, s') from batch data D
11: end
13: end procedure
15


Reviews:
[{"review_id": "aId2qt_c6ed", "reviewer": "Reviewer_FHnd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**The setting considered in this paper is not new.** As the authors also acknowledge, prior works such as Yu et al. 2020 have a similar setting. For example, Yu et al. 2020 run a random policy to collect a dataset. It then learns representations of the world and later uses them during online exploration and policy training. In prior works on online RL, replay buffers are used during training which is another related setting. The exact setting of this paper (policy fine-tuning) is presented by Xie et al. 2021. \n\n**The message of the paper is not new.** The messages presented in this paper are:\n- If we train a policy using an offline dataset and fine-tune it with additional online data, we get a better policy than only using the offline dataset.\n- If we train a policy using an offline dataset and fine-tune it with additional online data, we get a better policy compared to if we start from a random initial policy and fine-tune it with online data of the same budget.\n\nSuch improvements are expected to be observed in empirical results and the level of improvement depends on the quality of the dataset. Investigating this setting from a theoretical perspective (as done by Xie et al. 2021) provides insights as in theory, we are concerned with questions such as exact convergence rates, how many additional samples are needed during fine-tuning, what guarantees does the overall offline-online setting provide, etc. I don't see any new insights provided by the empirical studies in this paper.\n\n**The empirical comparisons are not fair.** \n- In Figure 1, offline-only RL and offline-online RL are compared. However, the offline-only RL is provided with strictly fewer samples and therefore this comparison is not fair.\n- In Figure 2, online-only and offline-online RL are compared. Again, offline-online RL is provided with a dataset in addition to the same online exploration budget and therefore this comparison is not fair.\n- Even if the sample sizes were the same, the comparison between these settings highly depends on the quality and type of dataset (e.g. does it come from policy rollouts? is it random? is it adversarial?) and I'm not sure how one can fairly compare these settings.\n\nQuestion: How are the batch datasets collected in the experiments?", "summary_of_the_paper": "This paper focuses on reinforcement learning, where the agent is given an offline dataset and is also allowed to collect additional data during online exploration. This allows the agent to finetune the policy and adapt to new situations. The authors run experiments showing that the offline-online RL outperforms online only or offline only RL.", "summary_of_the_review": "Overall the paper has limited novelty and new insights. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "THgQ0MGfe0", "reviewer": "Reviewer_aP1R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The setting introduced in this paper is interesting. It combines the online setting and the offline setting which allows the agent to achieve better performance than both settings. \n\nHowever, I have a few concerns that prevent  me from recommending an acceptance.\n\n(i) The proposed algorithm is a simple modification to an existing algorithm (Two-Timescale Networks), and therefore the algorithmic contribution is limited.\n\n(ii) The empirical observations are not supported by theoretical analysis. Also, the experiments are performed only on a small number of relative simple environments (acrobot, cartpole, etc). It is unclear if the same trend holds for other environments (Atari games, Mujoco, etc).\n\n(iii) In the experiments, the algorithm is effective only when the batch dataset is collected from a policy which performs well, and if we use random policy to collect the batch dataset, the algorithm cannot achieve non-trivial performance. However, with samples from a good policy, it is not surprising to me that the algorithm performs better than those in the online setting since those samples could help warm start the training process. The authors could consider performing experiments on other batch dataset. E.g., what will happen if the dataset is collected by a policy with middle-level performance? Such a policy can be obtained by early stopping online RL algorithms.\n\nDue to these concerns, I cannot recommend an acceptance at this point, and I am open to raise my score if the authors could address my concerns. ", "summary_of_the_paper": "In this paper, the authors explore the offline-online setting for RL, where the agent has access to a batch of data to train on but is also allowed to learn during the evaluation phase in an online manner. Compared to the offline setting, the agent is now allowed to take online samples during the evaluation phase. Compared to the online setting, the agent has access to a batch dataset which could potentially improve the performance.\n\nThe authors demonstrate that empirically, an offline-online variant of Two-Timescale Networks outperforms online RL / offline RL algorithms. The authors also perform ablation study to investigate factors that affect the performance. ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "INdHtkIpXHJ", "reviewer": "Reviewer_NS43", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Comments:\n\nI think the authors consider an important and practical setting of offline-online RL where the algorithm has access to both offline data and a limited amount of online interactions. However, I am concerned with the novelty of the paper. Offline-online RL has been considered before (for example, AWAC [1]), and to my knowledge, the authors do not discuss a comparison of their work to existing literature. The proposed algorithm can be interpreted as training on the offline dataset and fine-tuning online, which [1] also does just using a different base RL algorithm. Furthermore, the authors only compare their method to RL algorithms that learn solely offline or online, so it is unsurprising that learning on both improves performance. It would be useful if the authors also compared their algorithm to existing offline-online RL algorithms. \n\n\nMinor Comments:\n\nIt is not completely clear to me what the difference between offline-online TTN and offline-online DQN are. My hypothesis is that DQN does not fix the representation when learning the value, but that is not clear from the paper. \n\nIn Algorithm 1, it would be more clear if the operation showed (s, a, r, s’) as being appended to dataset D. The comment makes this clear, but the actual pseudocode makes it look like a straightforward assignment. \n\n[1] https://arxiv.org/pdf/2006.09359.pdf\n", "summary_of_the_paper": "This paper proposes an offline-online RL algorithm that learns jointly on offline data and online interactions. The proposed algorithm, offline-online TTN, learns by separately learning a representation, and values given a fixed representation, on two different timescales. The authors evaluate their algorithm against offline-online DQN, and TNN and DQN that only learns using offline data or online.", "summary_of_the_review": "Overall, I feel that the novel components of this paper compared to existing work in offline-online RL is not clear. Hence, I am recommending that the paper be rejected. However, if the authors provide a clear discussion and empirical comparison to prior work (AWAC, for example), I would be willing to raise my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7NwXGIAi63F", "reviewer": "Reviewer_Vqyy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper studies an important setting that combines online and offline RL, which seems more closely related to some real-world applications, while typically a large amount of historical data exists and could be used properly to warm start the algorithm.\n- The experiments using DQN and TTN to successfully illustrate the effectiveness of this combined setting, compared with purely online/offline setting, though there is some minor problem for a fair comparison. \n- It is interesting to see learning a representation layer separately is better than the end-to-end learning.\n\nWeakness:\n- This online/offline setting is well-studied in the literature, such as [1],[2] pointed in the above section. It would be great if the authors could compare with them explicitly.\n- This is an empirical paper, however the take-away from the experiments is trivial and well-known to the community. As accessing more online data should be better than purely offline learning, and I do not see any novelty or meaningful takeaway here.\n- The experiments are based on DQN and TTN in both offline and online cases. DQN is suffered from extrapolation error, and there are new algorithms to handle this, such as MOPO, CQL, etc. It is interesting to see the offline learning part based on some of these STOA algorithms, though a similar conclusion might hold there.\n- There is also some problem in the experiment design, as it is unfair for the offline learning algorithm, it should access the same batch data, as well as the similar quality online exploration data.", "summary_of_the_paper": "This paper studies the offline-online setting in RL, in which the agent is given a batch of offline data to train and could further refine it with exploration with the environment.\n\nBy testing on two specific algorithms, DQN and TTN, this paper shows empirically that (1). compared with a purely offline setting, it is beneficial to allow the policy to adapt online and (2). compared with the purely online setting, it is beneficial to use the offline/batch data to warm start. Beyond this, they also empirically verify how the amount and quality of offline data that is used to pre-train the policy help later adaptation stage.\n\nThe offline-online setting is interesting, but it is not novel, as there are some work studies this, such as [1], [2]. The empirical experiments are valuable but too specific to the particular algorithms, DQN and TTN. The conclusion from the experiments is trivial and well-known to the community as well.\n\n\n[1]. Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning\n[2]. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation\n", "summary_of_the_review": "Overall, though the setting this paper studied is interesting. The experiments do not provide any new insight into this setting and I would recommend a reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: It seems that the reviewers reached out a consensus that the paper is not ready for publication at ICLR. The reviewers raised concerns including “The empirical observations are not supported by theoretical analysis” , “The proposed algorithm is a simple modification to an existing algorithm”, concerns with “with the novelty of the paper”, “The message of the paper is not new. “ Please see the reviews for more detailed discussions about the paper.
Decision: Reject