Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
Authors: Haichao Zhang
Wei Xu
Haonan Yu
Venue: ICLR 2023 poster (2023)

Abstract: Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance. One natural approach is to initialize the policy for online learning with the one trained offline.
In this work, we introduce a policy expansion scheme for this task. After learning the offline policy, we use it as one candidate policy in a policy set, and further learn another policy that will be responsible for further learning as an expansion to the policy set. The two policies will be composed in an adaptive manner for interacting with the environment. With this approach, the policy previously learned offline is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the offline policy in the initial stage of online learning while allowing the offline policy participate in the  exploration  naturally in an adaptive manner. Moreover, new useful behaviors can potentially be captured by the newly added policy through learning. 
Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach.

Published as a conference paper at ICLR 2023
POLICY EXPANSION FOR BRIDGING OFFLINE-TO- ONLINE REINFORCEMENT LEARNING
Haichao Zhang Xu Haonan Yu
Horizon Robotics, Cupertino CA 95014 (haichas zhang, wei xu, haonan.yul @onnnannyy@horiieeeeee @oorizorzzz.c. ai
ABSTRACT
Pre-training with offline data and online fine-tuning using reinforcement learning is promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance. One natural approach is to initialize the policy for online learning with the one trained offline. In this work, we introduce policy expansion scheme for this task. After learning the offline policy, we use it as one candidate policy in a policy set. We then expand the policy set with another policy which will be responsible for further learning. The two policies will be composed in an adaptive manner for interacting with the environment. With this approach, the policy previously learned offline is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the offline policy in the initial stage of online learning while allowing the offline policy participate in the exploration naturally in an adaptive manner. Moreover, new useful behaviors can potentially be captured by the newly added policy through learning. Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach. Code is available: https: / /tte:://gttttut github. com Haiahhaaaahaooaaaa Zhang PPXX
INTRODUCTION
Reinforcement learning (RL) has shown great potential various fields, reaching or even surpassing human-level performances many tasks (e.g. Mnih et al. 2015 Silver et al., 2017, Schrittwieser jet al.) 2019; Tsividis et al., 202T). However, since the policy learned from scratch for given task in the standard setting. the number of samples required by RL for successfully solving task is usually large, which limits its applicability in many practical scenarios such as robotics, where physical interaction and data collection has non-trivial cost.
In many cases, there is a good amount of offline data that has already been available (Kober et al. 2013) Rastgoftar et al. 2018 Cabi et al., 2019). e.g., collected during previous iterations of experiments from human (e.g. for the task of driving). Instead of ab initio learning as in the common RL setting, how to effectively leverage the already available offline data for helping with online policy learning
an interesting and open problem (Vecerik et al. 2017 Hester et al. 2018 Nair et al. 2018).
Offline RL an active recent direction that aims learn policy by purely using the offline data, without any further online interactions (Fujimoto et al.] 2019 Kumar et al. [2220] Fujimoto & Gu
202T Levine et al.) 2020 Ghosh al. 2022 Chen al.) [222]] 2022I,Janner Janner et al.] [202]] T02l] Yang al.
2021
Lu et al. 2022; Zheng et al. 2022). D. holds the promise of learning from suboptimal data and
improving over the behavior policy that generates the dataset (Kumar et al. 20022) but its performance
could still be limited because of its full reliance on the provided offline data.
To benefit from further online learning, one possible way to pre-train with offline RI,, and warm start the policy of an online RL algorithm to help with learning and exploration when learning online. While this pre-training + fine-tuning paradigm is natural and intuitive, and has received great success in many fields like computer vision (Ge & Yu 2017 Kornblith al. 2019) and natural language processing (Devlin et al. 2018 Radford & Narasimhan 2018 Brown et al. 2020). it is less widely used in RL. Many early attempts in RL community report number of negative results along this direction. For example, it has been observed that initializing the policy with offline pre- training and then fine-tuning the policy with standard online RL algorithms (e.g. SAC (Haarnoja et al. 2018)

---

Published as a conference paper at ICLR 2023
sometimes suffers from non-recoverable performance drop under certain settings (Nair al. 2020 Uchendu et al. (2022). potentially due to the distribution shift between offline and online stages and the change of learning dynamics because of the algorithmic switch.
Another possible way is to use the same offline RL algorithm for online learning. However, it has been observed that standard offline RL methods generally not effective in fine-tuning with online data, due to reasons such as conservativeness of the method (Nair et al. 2020). Some recent works in offline RL also start focus on the setlinn-pre-raaningg- online fine-tuning paradigm (Nair et al.)
2020 Kostrikoy et al. 2022). For this purpose, they share the common philosophy of designing an RL algorithm that is suitable for both offline and online phases. Because of the unified algorithm
across phases, the network parameters (including those for both critics and actor) trained in the offline phase can be reused for further learning the online phase.
Our work shares the same objective of designing effective offline-to-oniing training schemes. However, we take a different perspective by focusing on how to bridge offline-online learning, and not on developing yet another offline or online RL method, which is orthogonal to the focus of this work. We will illustrate the idea concretely by instantiating our proposed scheme by applying it on existing RL algorithms (Kostrikov et al. 2022. Haarnoja et al. 2018 The contributions of this work are:
we highlight the value of properly connecting existing offline and online RL methods in order to enjoy the best of both worlds, a perspective that is alternative and orthogonal to developing completely new RL algorithms;
we propose a simple scheme termed as policy expansion for bridging offline and online reinforcement learning. The proposed approach is not only able to preserve the behavior learned in the offline stage, but can also leverage it adaptively during online exploration and along the process of learning:
we verify the effectiveness of the proposed approach by conducting extensive experiments on various tasks and settings, with comparison to a number of baseline methods.
PRELIMINARIES
We briefly review some related basics in this section, first model-free RL for online policy learning, and then on policy learning from offline dataset.
2.1 ONLINE REINFORCEMENT LEARNING
Standard frrel-free RL methods learn a policy that maps the current state 8 to a distribution of action a as (s). The policy is typically modeled with a neural network (s) with denoting the learnable parameters. To train this policy, there are different approaches including on-policy (Sutton) ct al. 2000; Schulman et al. 2017) and pollyy RL methods (Lillicrap et al. 2016 Haarnoja et al., 2018: Fujimoto et al. 2018: Zhang al., 2022). In this work, mainly focus on off off-policy RL for online learning because of its higher sample efficiency. Standard off policy RL methods rely on the state-action value function ((s, a) using TD-learnngg:
(ss, = r(s,a) + +yE "Tsa)=((s (x) (Q(s',a') .a)),
where T(s, a) denotes the dynamics function and r(s, a) the reward. 7 E (0, 1) is discount factor. By definition, Q(s, a) represents the accumulated discounted future reward starting from 8, taking action a, and then following policy ne thereafter.
The optimization of 0 is achieved by maximizing the following function:
max Eax Q(s,
8
where D denotes replay buffer for storing online trajectories. In the typical RL setting, learning is conducted from scratch by initializing all parameters randomly and interacting with the world with the randomly policy. Q(s,a) a) can be implemented neural network (& a) with parameter ő
2.2 POLICY LEARNING FROM OFFLINE DATASET
Policy learning from offline datasets has been investigated from different perspectives. Given expert- level demonstration data, behavior cloning (BC) (Pomerleau 1988 Bain & Sammut 1996) is
2

---

Published as a conference paper at ICLR 2023
effective approach for offline policy learning because of its simplicity and effectiveness. In fact, in some recent work, BC has shown to perform competitively with some offline RL methods (Fujimoto & Gu 2021 Chen et al. 202I). Given dataset Dottine {ss a,)} consisting of expert's 'ss state action pairs (8t, 0;), BC trains the policy with maximum likelihood over the data:
max (s.a)) log To (a)s).
Although BC has the benefits of reducing the policy learning task to an ordinary supervised learning task, it suffers from the well-known distributional shift issue (Codevilla al.. [2019] Muller et al.) 2005 de Haan et al. 2019: Wen et al.) (020). Another limitation is that BC has a relatively strong requirements the data quality, and is not good at learning from suboptimal data.
Offline RL is a category of methods that are more suitable for policy learning from noisy and suboptimal offline data (Kumar et al., (022). When focusing on offline learning only, the core challenge is how to address the extrapolation error due to querying the critic function with out-
of-distribution actions (Fujimoto constraining the actions to be close to dataset actions (Fujimoto et al., 2019; Fujimoto & Gu 2021). ). and constraining the critic to be conservative for out of data distribution actions (Kumar et al. 2020).
al. 2019 Kumar et al.] 2020). Common strategies include
The recent implicit Q-learning (IQL.) method (Kostrikov et al. 2022) addresses this issue by learning a value network to match the expectile of the critic network, thus avoiding querying the critic with the actions not in the offline dataset. For policy update, IQL use a weighted BC formulation
max ((s ()saa) w(s a) log (T(aa)))
0
where w(ssa) a) denotes a data dependent weight, typically calculated based on the estimated advan- tages (Nair et al. 2020 Xu et al. 2022 Kostrikov et al. 2022).
3 OFFLINE AND ONLINE RL REVISITED: CONNECTIONS AND GAPS
Connections between Offline and Online RL. Offline RL and online RL are closely connected in many aspects. Historically, many offline RL approaches are branched off from off-policy RL algorithms to the full offline setting (Fujimoto et al. 2019 Fujimoto & Gu 2021). Algorithms more specific to offline setting are then further developed, motivated by the challenges residing in the offline setting (Levine et al. 2020 Kumar et al. 2020). Besides algorithmic connections, offline and each other in terms of strengths and weaknesses. Offline RL is sample efficient since no online interactions are required. but the performance is bounded by the fixed data set. Online RI. enjoys more opportunities for performance improvement, but is comparatively much less sample efficient. Because of the connections and complementary strengths, instead of treating them as two isolated topics, it is more natural to connect both in pursuit of a performant policy in practice.
online RL are also complementary
The Direct Offline-Online Approach. Because of the above mentioned connections, it is tempting to directly use the same algorithm (thus the same network architectures as well) for both phases. Unfortunately, this is ineffective in either directions: either directly using offline RI algorithms for online learning (forward direction), or directly using online RL algorithms for offline learning (reverse direction). The reverse direction has been explored extensively in the offline RL community. The current wisdom is that instead of directly using an existing online RL algorithm (e.g. TD3 (Fujimoto) al. 2018)), special treatments need be incorporated into the algorithm (e.g. incorporating BC into TD3 (Fujimoto& & Gu, 2)21) for handling the challenges arising in offline learning, due to issues such as querying the critie function with out-of outoo-diisrriutton actions.
In the forward direction, as noted in previous work (Nar et al. 2020), it is exceptionally difficult to first train a policy using offline data and then further improve using online RI. There are some efforts in the literature on directly transferring the parameters learned offline for online fine-tuning, i.e., by initializing the policy in Eqn. (T) with parameters learned offline. This scheme illustrated in Figure [ as the Direct approach for Offline-to-nllnee RL. While simple, this approach has several potential issues as noted in the literature. For example, one common issue is that the behavior of the offline policy can be compromised or even destroyed at the initial phase of online training, e.g. because of the noisy gradient for policy update due to cold start learning of critic network (Uchendu et al. 2022 (as in the case of reward-free pre-training) or distribution shift between offline and
3

---

Published as a conference paper at ICLR 2023
Offine
Oflline to Online RL
Unrol
Oft Buffer Pottie
Learn
Offine Zuffer Dattes
Learn Nd
Q
Learn
Puttis
Online
Policy Exparsion
Unroll
(Ix A-1
Offine Buffer
Dullix
Learn
Ta
Learn
Duttir
Figure 1: Illustration of Different Training Schemes. Offline training and online RL have been developed within their own training stages. Direct Cefline-Online learning approach continues the online training stage after the offline stage is finished, updating the same policy network. The proposed Policy Expansion approach bridges offline and online training by retaining the policy after offline learning (TT)), and expand the policy set with another learnable policy (TT) for capturing further performance improvements. The two policies both participate in interactions with environment and learning in adaptive fashion.
online dataset (Lee et al. 2021). Another issue is the conservativeness of the offline RL algorithms. While this a desirable feature when considering only offline training, is not preferred for online
learning, where exploration is valuable for further improvement (Rezaeifar et al.) 2022). This is
a phenomenon that is commonly observed and reported in the literature (Lee et al.
2021 Campos
et al. 2021 Uchendu et al., 2022).
4 BRIDGING OFFLINE AND ONLINE RL VIA POLICY EXPANSION
In this section, we will introduce simple scheme called Policy Expansion for bridging offline and online training. It is worthwhile to note that the proposed scheme is orthogonal to specific off/online RL algorithms and is compatible to be used with different value-based offline and online algorithms. The final performance of such combination may vary depend on the selection of methods.
4.1 POLICY EXPANSION AND ADAPTIVE COMPOSITION
Policy Expansion. To mitigate the above mentioned issues, we propose an alternative scheme that can be readily combined with existing algorithms. The proposed approach is illustrated in Figure [ Given policy Ty obtained from offline training phase, instead of directly fine-tuning the parameters. we freeze Th and add it into a policy set 11 = [T. To enable further learning, instead of directly
modifying Tg as in the Direct method, which has the potential of destroying useful behaviors learned
offline, we freeze Ty and expand the policy set II with another learnable policy as
(4) which is responsible for further performance improvement during online training. We refer this type of policy construction as Policy Expansion (PEX). It intuitive to understand that the behavior of offline policy T3 is free from being negatively impacted, while the newly added policy can be
[TT.
updated. The policies in the policy set II will all get involved into exploration and learning in an collaborative and adaptive manner as detailed in the following.
Adaptive Policy Composition. Both policies in the policy set 1I will form a single composite policy i, which will be used both exploration and learning. More specifically, given the current state we first sample actions for each member of the policy set II and form a proposal actions set = {a, ~ (s)mm, E II]. Then all the action proposals will be taken into consideration and they will be selected with the probability related to their potential utilities (e.g. values). For example, we can compute their values at the current state Q Qo=(Qo(s, a))a, E A] E k with K denotes the cardinality of 11 (here K = 2), and construct categorical distribution for selecting the final action:
cxp(Qo(s.a://a) Pwld Σ, exp(Qo(s,aa)/aa)
vi [ie -1.-..KK
(5)
where is temperature. Then sample from it for selecting the actions Pw to decide which action will be used during unroll for interacting with environment. Using value for policy composition has been used in different contexts in the literature (Yu et al. 2021 Shah et al. 2022)
4

---

Published as a conference paper at ICLR 2023
Conceptually, the composite policy it can be represented follows:
# [a]ss) (s)) (s)ma(s)) ww w W~ Pw
(6)
where "RKK one-hot vector, indicating the policy that is selected for the current state dawn denotes the Dirac delta distribution centered at a which is sampled from л.
By allowing only the newly added policy (Ty in this case) to be fine-tuned while freezing all others (TT)), we can avoid the problem of compromising (e.g. destroying or forgetting) the behavior of offline policy. At the same time, we have the advantage of adaptiveness in the sense of allowing learning of new abilities. From this perspective, the policy expansion plays the role of bridging the offline and online learning phases, while mitigating commonly encountered issues. It is interesting to note that similar compositional form of policy has appeared in DAgger (Ross et al. (011)) ID, although with uniform weight across states and under a different context of imitation learning. Here our compositional weight is state adaptive and the compositional policy is used for bridging offline-to-online reinforcement learning.
PEX has several advantages compared to direct offline-online learning (illustrated in Figure
1. offline policy preservation: can retain the useful behaviors learned during offline training phase by retaining the policy and avoid it being destroyed in the initial online training phase;
2. flexibility in policy form: the offline policy does not need to be of the same form with the online policy (e.g. same network structure) as in the direct offline-online approach, offering more flexibilities in design;
3. adaptive behavior selection: both the behavior of the offline policy and the online learning policy are used in interacting with the environment and they are involved in an adaptive manner, e.g., according to their respective expertise in handling different states.
Algorithm 1 PEX: Policy Expansion for Offline-to-Online RL
Input: offline RL algorithm { ofiige L L Joffinee online RL algorithm {Loninee Th
online
Initialize: network parameters B, 0, offline replay buffer Dottine
while in offline training phase do % offline policy training using batches from the offline replay buffer Dattine + Xo V V offtine (Ø), B Ax g (B)
end while
Policy Expansion: म [xg, uo): transfer Qo while in online training phase do for each environment step do
~ 5 (at]s:) according to (6) 81+1 1(8441181,01) D+ DU((S,, ((s,,)))) end for for each gradient step do % online training using batches from both Dottine and D
o + Φ XQ V Lontine()))
-- 0 - Ag V on (0)
end for end while
4.2 BRIDGED OFFLINE- ONLINEE TRAINING WITH POLICY EXPANSION
We focus on value-based RL algorithms for both stages in this work. For offline-training, we conduct offline RL with an offline RL algorithm (e.g. IQL) on the offline dataset obtain the offline policy TH. Then we can construct policy expansion following Eqn.(4) before entering the online phase. And we also transfer the Q function (critic) learned in the offline stage online stage for further learning. We also transfer the offline buffer to online stage as an additional buffer, as shown in Figure
For online training, we use the policy adaptively composed from the policy set as in Eqn. 6 and then conduct online training by interleaving environmental interaction and gradient update. The newly collected transitions are stored into the online replay buffer D. For training, batches randomly sampled from both D and Doftine are used. The value loss and policy loss are calculated based on the losses corresponding to the chosen algorithm. The proposed scheme can be used together with different existing RL algorithms. The complete procedure is summarized in Algorithm taking an offline RL and online RL algorithm as inputs.
We represent value-based RI. algorithm succinetly with pair of value and policy losses (1.9 L.).
5

---

Published as a conference paper at ICLR 2023
5 RELATED WORK
Pre- Tre-Trainin in RL. A number of different directions have been explored in RL pre-training,
including representation pre-training and policy pre-training. Note that for RL, pre-training can be either offline or online. Representative works include pre-training of the feature representation
using standard representation learning methods (e-g. contrastive learning (Yan & Nachum 202Th) D),
dynamics learning-based representation learning (e.g. |Schwarzer et al. 2021] Seo et al. 2002).
unsupervised RL driven by intrinsic rewards in reward-free environments (c.g. Liu & Abbeel 2021 J.
or directly using ImageNet pre- tre-training for visual RL tasks (e.g. (Shah (c.g.[Shah& & Kumar 202 Yuan et al.
2022). Apart from representation pre-training, another category of work is on policy pre-training, with the goal of acquiring behaviors during the pre-training phase that are useful for the online phase. When the downstream task is unknown, there are approaches for unsupervised pre- pre-training. e.g., maximizing behavior diversity (Eysenbach et al. 2019) or converting the action space Singh et al. (20TT) with the hope of discovering some behaviors that are useful for the downstream task. When the offline-online task is more aligned. there are some early attempts on directly transferring policy parameters (Rajeswaran et al. 2018), based on the intuition that policy initialized this way can produce more meaningful behaviors than randomly initialized networks. Some recent work focuses on behavior transferring, iee. leveraging the offline trained policy for exploration during online training (Campos et al., |02T] et al., 2022). Our work falls into this latter category of methods. One notable difference compared to to|Campos et al. (2021); Uchendu et al. (2022) is that for the proposed approach, the offline policy is one part of the final policy, and with its role been determined adaptively.
Data-Driven RL and Offline RL. Training a policy by leveraging a large amount of existing data (i.e. data-driven RL) is a promising approach that is valuable to many real-world scenarios, where offline data is abundant. Offline RL is one active topic towards this direction. The main motivation of offline RL. is train a policy by leveraging a pre-collected dataset, without requiring additional environmental interactions (Levine et al. (020). Many research works in this direction focus on addressing the special challenges brought by offline learning, including outtof-distribution value issue (Kumar et al., 2020). Common strategies include constraining the value (Kumar et al. (020)) to be small for out of distribution actions or the policy to be close to the action distribution of dataset (Fujimoto et al.) 2019 Fujimoto & Gu 2021). Recently, Kostrikov et al. (2022) proposes an implicit Q-learning (IQL.) method as an alternative way to handle this issue. It learns a value function
that predicts certain expectile of the values for state-action pairs from the dataset, which can be used for computing the value target without querying the critic with out-of -uiti--iittiibution actions.
Offline Training with Online Fine-tuning. Combining offline data with online learning is an effec- tive approach that have been demonstrated by several early attempts with demonstration data (Vecerík) et al. [017]] Hester et al. [0188] |air et al. 2018 Rajeswaran et al.) 2018). Traditionally, the offline RL methods are purely focused on the offline training setting. However. the offline learned policy could be limited in performance given a fixed dataset. In the case when further online interaction is allowed, it would be natural to fine-tune the policy further with data collected online. This paradigm of two-stage policy training is related to the iterative interleaved policy and data collection schemes used in imitation learning (Ross et al. 2011 Ross & Bagnell| 2012). Several different approaches have been explored towards online fine- tunint of offline pre-trained policy, including balancing offline-online replay data (Lee et al., al., 202T), > parameter transferring (Rajeswaran et al.) 2018 Xie] et al. 2021). policy regularization (Rudner al. 2021 Tirumala et al.) (020)) and guided explo- ration (Campos et al., 2021 Uchendu et al.) 2022). has been observed that directly applying some offline RL methods does not benefit from online interactions (Nair et al.) 2020: Uchendu et al. 2022). potentially due to the conservative nature of the offline policy (Nair et al., 2020). Based on this observation, there are some recent efforts on developing algorithms that are not only suitable for offline training, but also can leverage online interactions for further learning. Nair et al. (2020) shows that advantage- weighted form of actor-critic method is suitable for this purpose. TQL. (Kostrikov ct al. 2022) also leverages a similar form for policy learning and shows that it can benefit from online fine-tuning, Our work falls into this category and we provide an alternative approach for leveraging offline pre-training for helping with online training.
EXPERIMENTS
In this section, we first evaluate the effectiveness of the proposed approach on various types of benchmark tasks with comparison to number of baseline methods. Then we further show number of extensions where the proposed approach can also be applied.

---

Published as a conference paper at ICLR 2023
antmara-ameze
untmoe--eddum play
antrann-madiure
antranee- warge-plyy
B.Arr
Disci
KWAZ
15RL
PDX
antmaze largo diverse
haltcheesh
nooper-rendom
We ker20 mnoo
hoPcheetch medium
wed vodher2d-mmddmm
walken 2o-rraci
1.0
0.5
-22
2.
0.1
3.5
500
02a
0.4
0.4
3.
0.2
Figure 2: Normalized Return Curves of different methods on benchmark tasks from D4RI. (Fu
et al. 2020). IQL is used as the backbone apart from the AWAC baseline (Nair et al. 2020).
6.1
OFFLINE-TO-ONLIN RL EXPERIMENTS
Tasks and Settings. We use the standard D4RL benchmark which has been widely used in offline RL community (Fu et al. 2020). For offline learning, we use the provided dataset for training. For online learning, we use the accompanied simulator for interaction and training. For offline phase, 1M training steps are used for training. Then we run online fine-tuning for another 1M environmental steps. Here we use IQL (Kostrikov et al. 2022) as the backbone algorithm for all methods listed below. The training is repeated with 5 different random seeds.
Baselines. We compare the proposed approach with the following baselines: (i) Offline: offline training using IQL, without online fine-tuning; (ii) Scratch: train IQL online from scratch, without praitieppr--rriinng.. (iii) Buffer (Vecerik et al.] 2017): train IQL online without offline pre- training, but has access to the offline buffer during online training, i.e. using a buffer as D DDfttiieee (iv) Direct (Kostrikov et al. 2022): direct offline-to-onlin approach by directly transferring parameters trained offline to online stage using IQL (Kostrikov et al. (022). which is a recent and representative RL algorithm that shows state-of-the-art performance on offline RL while allows online fine-tuning; (v) AWAC (Nair et al. 2020): an approach that uses an advantage- actor-critic method for offline-to-online RL; (vi) Off2On (Lee al. 2021): a recent offline-to-online RL method that uses an ensemble of offline trained value and policies together with a balanced offline-online replay scheme; (vii) BT (Campos al. 202T): Behavior Transfer which is approach that leverages an offline learned policy in exploration, where the offline policy is used for exploration for a consecutive number of steps sampled from distribution once activated; (viii) JSRL (Uchendu et al., 2022): ; Jump Start RL, which divides the rollout of a trajectory into two parts, using the offline learned policy for the first part and then unrolling with the online learning policy for the rest of trajectory. PEX denotes the proposed approach, which has the same offline and online RL algorithms
form of
as Direct, and is only different in using Policy Expansion for connecting the two stages. More resources are available on the project page.
Offine
The return curves for all the tasks are shown in Figure The aggregated return across all tasks is shown in Figure The returns are first averaged across task and then across runs. It can be observed that all methods show some im- provements after online training in general, compared to the initial performance before online training. Seratch has
Scratch
Butler
Direct
AWAC
Of200n
JSRL
88
Envirement
httpa: //s//////aaa tew. goog) ie wwo.///.......mmmm...o.oooo...oo twwhaaanggaaaaaaaaaaaaaaaaaaaaaa htwp://wwwwaggggppooooooooco///e /ptpo/ccs//ppt
Figure 3:
Aggregated Return Curves across tasks (IQL-based).
7

---

Published as a conference paper at ICLR 2023
the lowest overall performance across all tasks. On the challenging sparse reward antmaze tasks, Scratch cannot learn a meaningful policy at all. Buffer has better performance than Scratch when incorporating the offline buffer, indicating that the offline buffer has some benefits in help- ing with learning. Notably, on the previously zero-performance antmaze tasks, Buffer achieves reasonable performance with the help of the offline buffer, leading large overall improvement over Scratch, (c.f. Figure 3). Direct (IQL-based) shows large improvements over Offline on average as shown in Figure implying the benefits brought by the the additional online training over pure offline training. Off2On also shows large improvement during fine-tuning and achieves strong overall performance (Figure 3). BT shows some improvements over IQL on some tasks such as antmaze-mer meree-mediue umllpp and
with an overall performance comparable to that of Direct. JSRL outperforms Direct and BT on some tasks (e.g. hopper-men um, hopper-med um- un-replay), potentially due to its different way of leveraging the offiin policy, and its overall performance is similar to BT. The proposed PEX approach performs comparably to other baselines on some tasks while outperforming all baseline methods most of the other tasks (c.f. Figure 2) and outperforms baselines methods overall (c.f. Figure 3). demonstrating its effectiveness.
6.2 HETEROGENEOUS OFFL RL BRIDGING VIA POLICY-EXPANSION
We have shown the application of PEX to the case where both the offline and online algorithms are the same (referred to as PEX-IQL here since both are IQL) in Section 6. T In this section, we further show the applicability of the proposed scheme in bridging heterogeneous RI. algorithms, L.e. different RL methods
09
Nometzed Retu
0.8
used for the offline and
rf
online stages. As an example. here we use IQL (Kostrikov et al.
0.4
2022) and SAC (Haarnoja et al. 2018) for offline and online RL stages respectively. We compare with Scratch (vanilla SAC (Haarnoja jet al., 2018)), Buffer (SAC with additional offline replay buffer) as well as Direct (directly transferring policy and critic parameters learned with offline IQL to SAC for further online learning). Again PEX uses the same offline and online algorithms as in Direct but uses policy expansion instead of the direct transferring approach. The normalized return curves aggregated across tasks are shown in Figure Individual return curves are shown in Appendix AA8 It can be observed that there is an overall improvement by simply applying PEX to the heterogeneous offline-online RL setting as well.
Diesi
RIMI
04
00
Environment
146
Figure 4: Aggregated Re- turn Curves across bench- mark tasks (SAC-based).
6.3
ABLATION STUDIES
We will inspect the impact of several factors on the performance of the proposed method in the sequel.
Offline Buffer
Critic Transfer
Policy Transfer
Offline Policy Freeze
3.0
3.6
Osabled
S
ed
Dbubled
09
(a)
Unvrenvent Skags
(b)
(c)
380
(d)
Enelomno thpps
Figure 5: Ablation Results on a number of factors. Orange curves correspond to ablation variants.
Offine-Buffer.
This experiment investigates the impacts of including the offline replay buffer in online training stage. The results are shown in Figure As can be observed, the inclusion of the offline replay buffer helps with the performance, but the performance drop caused by disabling offline replay buffer is smaller than the change of other algorithmic components (c.f. Figure (acuureebbyaaa (d))
This experiment studies the impacts of transferring the critic parameters trained from offline to online stage. As can be observed from Figure 5bb). disabling critic transferring will also greatly decrease the performance both in terms of sample efficiency as well as final performance.
Critic Transfer.
Policy Transfer. This experiment examines the impacts of transferring the offline pre- trained policy in the online training stage. When policy transferring is disabled, there is need to use policy
8

---

Published as a conference paper at ICLR 2023
expansion in the online training stage. The results shown in Figure 5(c). It can be observed that there is a clear performance drop when policy transferring is disabled.
Offline Policy Freeze. The offline learned nœ is freezed during the online learning stage in PEX. This experiment investigates the impact of this factor. If disabled, Ty will be trained the same way as during online learning. It is observed from Figure ( that freezing the offline policy is important. Training by disabling policy freezing has clear performance drop. This consistent with the intuition on the usefulness of offline policy preservation, which is one advantage of our approach. Another set of ablation results are deferred Appendix A.4
6.4
VISUALIZATION AND ANALYSIS
Policy Composition Probability. Since a composite policy is involved in the proposed approach, is interesting inspect the participation of each member policy from the policy set II ( ne when interacting with the environment. We visualize the policy compositional probability Pw during the rollout within trajectory after training. shown in Figure It can be observed that composition probability for each member policy is state-adaptive and is changing along the progress of a trajectory, implying that both policies contribute to the final policy in an adaptive manner.
antmaze umaze diverse
antmaze medium medium-diverse
1.0
1.
0.8
02
0.6
7
Fi
0.0
(a)
20
Environment Step
80
120
140
(b)
Environment Step
100
150
200
Figure 6: Visualization of Policy Composition Probability during the rollout of one trajectory. The probability curves smoothed for visualization purpose.
State Space Associations of Member Policies. To get better under- stand of the role of the member policies in the policy set, we visualize the association between the states and its selected policy. For this purpose, we embed set of states into a 2D space using t-SNE (van der Maaten & Hinton 2008), and then visualize the association of offline policy T and the newly expanded policy He to states in the projected space. States that select the offline policy TT colored with blue and states that select Ty are colored with red. It can be observed that To and To cover differ- parts of the state space, indicating that they have some complementary functionalities and preferred differently at different states.
CONCLUSIONS, LIMITATIONS AND FUTURE WORK
We highlight the usefulness of properly connecting offline and online stages of reinforcement learning gain the benefits of both worlds and present policy expansion scheme as an attempt toward this direction. This scheme an instance in the direction orthogonal to developing completely new offline-to-online RL algorithms. The proposed approach simple and can be combined with existing RL algorithms, and is illustrated with two different combinations in this work. Experiments demonstrate the effectiveness of the proposed approach on number of benchmark tasks.
While achieving promising performance, the proposed approach also has some limitations. One limitation that the number of parameters grows with the number of policies in the policy set. While this might not be an issue when the policy set is small as the case in this work, it will be less parameter efficient in the presence of large policy set. One possible way to address this issue is by introducing a distillation stage (Rusu et al. (2016). by consolidating the multiple policies into network with smaller number of parameters. Generalizing the proposed scheme to the case with a set pre-trained skill policies is interesting direction (Eysenbach et al.) 2019 Shu et al. 2018).. For offline learning, we have built upon the strong IQL method. It would be interesting to see how much we can gain by upgrading it with more recently developed offline RL methods together with different online methods. The idea of using a policy set itself can potentially be applied to other cases beyond offline to online RL. We leave the exploration of its generalization and application as interesting future work.

---

Published as a conference paper at ICLR 2023
REFERENCES
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, 1996.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rac Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerík, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. A framework for data-driven robotics. CoRR, abs/1909. 122000, 2019.
Víctor Campos, Pablo Sprechmann, Steven Hansen, André Barreto, Steven Kapturowski, Alex Vitvitskyi, Adrià Puigdomènech Badia, and Charles Blundell. Beyond fine-tuning: Transferring behavior in reinforcement learning. CoRR, 11512100211555,, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021.
Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In International Conference on Computer Vision, 2019.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, 2019.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Gu. minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, 2018
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline RL policiis should be trained to be adaptive. In International Conference on Machine Learning, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dunoc-Arood, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep Q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, 2018.
10

---

Published as a conference paper at ICLR 2023
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Jens Kober, J. Andrew Bagnell, and Jan Peters, Reinforcement learning robotics: A survey, The International Journal of Robotics Research,
2013.
Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better ImageNet models transfer better? In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-leariig. In International Conference on Learning Representattions 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should run offline reinforcement learning behavioral cloning? In International Conference on Learning Representations, 2022.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. Annual Conference on Robot Learning, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01 1000000.16433 643, 2020.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. CoRR, abs/2206.04777 044799, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann LeCun. Off-road obstacle avoidance through end-to-end learning. In Advances in Neural Information Processing Systems, 2005.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Picter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations, In IEEE International Conference on Robotics and Automation, 2018
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. CoRR, ahs/2006.09935) 2020.
Dean A. Pomerleau. ALVINN: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, 1988.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training.
In Tech. Report, 2018
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems, 2018.
Hossein Rastgoftar, Bingxin Zhang, and Ella M. Atkins. A data-driven approach for autonomous motion planning and control in oof-road driving scenarios. Annual American Control Conference, 2018.
11

---

Published as a conference paper at ICLR 2023
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In AAAI Confer- ence on Artificial Intelligence, 2022.
Stephane Ross and Drew Bagnell. Agnostic system identification for model reinforcement learning. International Conference on Machine Learning, 2012.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2011.
Tim G. J. Rudner, Cong Lu, Michael Osborne, Yarin Gal, and Yee Whye Tch. On pathologies in KL-regularized reinforcement learning from expert demonstrations. In Advances in Neural Information Processing Systems, 2021.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In International Conference on Learning Representations, 2016.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmiut, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillierap, and David Silver. Mastering Atari, Go, Chess and Shogi by planning with a learned model. CoRR. abs/1911.08226 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 005/1707.063441 2017.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference Machine Learning. 2022.
Dhruv Shah, Alexander Toshev, Sergey Levine, and Brian Ichter. Value function spaces: Skill-
centric state abstractions for long-horizon reasoning. In International Conference on Learning Representations, 2022.
Rutav Shah and Vikash Kumar. RRL: Resnet as representation for reinforcement learning. In
International Conference Machine Learning, 2021.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition
in multi-task reinforcement learning. In International Conference Learning Representations, 2018.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354 2017.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference Learning Representations, 2021.
R. Sutton, D. Mcallester, Singh, and Y. Mansour, Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, 2000.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, and Nicolas Heess. Behavior priors for efficient reinforcement learning. CoRR, 2020.
12

---

Published as a conference paper at ICLR 2023
Pedro A. Tsividis. João Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J. Gershman, and Joshua B. Tenenbaum. Human-level reinforcement learning through theory-basee modeling, exploration, and planning. CoRR, abs/2107. 12544, 2021.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-start reinforcement learning. In arXiv, 2022.
Laurens van der Maaten and Geoffrey Hinton. Visualizing high-dimensional data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. CoRR, abs/1707.08817, 2017.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. CoRR, abs/2010. 14876, 2020.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weisn weighted offline imitation learning from suboptimal demonstrations. International Conference on Machine Learning. 2022.
Chao-Han Huck Yang, Zhengling Qi, Yifan Cui, and Pin- Yu Chen. Pessimistic model selection for offline deep reinforcement learning. CoRR, abs/21 11. 1.14646 2021.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In International Conference Machine Learning, 2021.
Haonan Yu, Wei Xu, and Haichao Zhang. TAAC: Temporally abstract actor-critic for continuous control. In Advances Neural Information Processing Systems, 2021.
Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. First Workshop on Pre-training Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
Haichao Zhang, Wei Xu, and Haonan Yu. Generative planning for temporally coordinated exploration in reinforcement learning. In International Conference on Learning Representations, 2022.
Qinqing Zheng, Mikael Henaff, Brandon Amos, and Aditya Grover. Semi- semi-supervised offline rein- forcement learning with action-free trajectories. CoRR, abs/2210.06518,, 2022.
A APPENDIX
A.1 DETAILS ON BASELINES
In this section, we provide more details on the baseline methods.
Behavior Transfer (BT). This approach originally proposed for the case pre-training using unsupervised reinforcement learning and then transfers the learned behaviors for later learning. It also originally used in discrete action cases. Here we adapt this idea to build a baseline in our setting based on the approach of transferring behaviors in (Campos al. 2021 Following (Campos) et al. 2021). a Zeta distribution with parameter a = 2 is used for determining the number of steps of persistent unroll using the offline policy. The number of persistent unroll steps is re-sampled when currently not in the middle persistent unroll and with a random number sampled from [0, 1] is 0.1 is used in experiments.
smaller than threshold
13

---

Published as a conference paper at ICLR 2023
Jump Start RL (JSRL). JSRL divides the complete unroll of one trajectory into two parts, with the first part obtained with the offline policy T., while the second part is obtained with the current learning policy Tg- The core idea of JSRL is initially make the first part longer, and then progressively reducing the length of the first part, thus forming type of curriculum to help with exploration and learning. We follow this core idea by using linear scheduler that anneals from the max episode length to 0 for progressively decreasing the first stage of unroll using the prior policy πg.
Ofline-to-Online RL (Off2On). Off2On is an approach that aims to improve offline to online RL performance (Lee et al. 2021). It incorporates two components to cope with the challenges of offline-to-oninee RL: I) an ensemble of critic networks and policy networks are trained during offline training stage, in contrast to typical offline RI, which only learns one policy network and critic network. Following the setting in Lee et al. (2021). ensemble size of 5 used. 2) balanced replay: a balanced offline-oniine replay scheme properly trade off between the usage of offline and online samples. We used the code released by the authors of (Lee et al. 2021)
A.2 IMPLEMENTATION DETAILS ON POLICY EXPANSION
Policy Expansion (PEX with IQL-backbone). The proposed scheme can be combined with many different existing RL methods. Here we illustrate with an concrete instantiation using IQL. (Kostrikoy al., 2022). For pre-training, we conduct offline RL with IQL on the offline dataset to obtain the offline policy Tg. Then we conduct policy expansion as Eqn. before switching to online phase.
For offline training, we follow IQL (Kostrikov et al. 2022 train the parameters of the value function using the TD-learnig approach as follows:
Lo(d) ((s.anna) Qo(s,a) (r
(bb) = ((s.a))Doftin.. 1 12)0)(8))) (x)]
(8)
where Lī(u) 7
0) Kostrikov al. 2022). with t denotes the expectile value Kostrikov et al. 2022). denotes set of target parameters that are periodically copied from @. The policy network Te can be further updated by leveraging the updated critic network. Here
1(u
we again follow IQL. (Kostrikov et al. 2022) and use the weighted form for actor training:
L (8) E(s,a) [wos a)] log TT(aas))
with ass,a) exp((((xss))))
(s))/aa ()((s))) and [.] denotes the gradient stopping operator.
For online training, we first expand the policy set as in Eqn.(4). and then conduct online training by interleaving environmental interaction and gradient update. The newly collected transitions are saved into replay buffer D. For training, batches randomly sampled from both and Doffine are used. The values losses are the same as the offline phase, and the policy loss is computed against Ty:
L (0) = EE JD (w))()) - [wr(,, a)] log my (als),
(10)
For on ~ (9t),,, in practice, we simply take the greedy action from the policy. Following Kostrikov et al. 2022), we model the policy using a non-squashed Gaussian distribution with mean squashed to action range, and use state independent standard deviations.
Policy Expansion (PEX with with SAC as online backbone). The main procedure of online training the same as SAC (Haarnoja al. 2018 with an adaptation for the actor training by changing the
part of the loss based on Q-function from E.- BaNtY
BQ(s.a)
+ aa aa
(which is an equivalent
2
form of loss to Ea-DE - Q(s, a)) to E Ba
(Q(a.)) Ba
+ a]
40
3 Here refer the commonly used double critic replica to reduce value overestimation as in SAC Haaroja ct (018)) and TD3 (Fujimoto et al. 2018) as single critic network. The ensemble used in Tee et al. 2021 applied on top of this.
httppa http://ggttt.oooo/o..tooo shi /
14

---

Published as a conference paper at ICLR 2023
A.3 THEORETICAL JUSTIFICATION OF PEX
Here we provide some theoretical justification of the proposed form of adaptive policy compostional distribution Pw in Eqn. 5):
Pw exp(Qo(s,a))/o)
Σ, exp(Qa(s.a;))/)) ei (..... K],
(1I)
where K denotes the number of actions and in our case K 2.
Pw can be viewed as a discrete policy with action dimension of two, which selects between two candidate actions d1, 02, one from offline policy स and the other from Na i.e., d1 πs and 0g TT- Ideally, assigns a higher probability to actions with higher values at the current state s. The definition of Pw essentially a reflection of this intuition, Actually, below.
also be derived shown
By definition, have the following form for the compositional policy म:
*(als) = -f =s(orls)To(azls) Σ Pw (ils, a1 02)6(a a;)daıda2
(12)
which is essentially another form of Eqn.
For V-value, we can have the following derivations
((s) = J. (a(als)@ola a)da
(13)
(plug in the definition of policy in Eqn.
(14)
T3(a215)To(a21s)) [ en
E (ilss a1, 02)5(a = a;)da daz Qols, a)da(15)
/ an
[. 8(a a.)Q a)da da daz(1)))
[0((1)))To(a22s))
Σ Pw (ils,
[0(01ss)To(a2(s)
Pw (ils, a1 a2)Q(ss (lsssaa)des da) Σ
(17)
Jayna
Ea, O2
Pw(lls,an, a2)
(18)
Applying entropy regularization term for policy learning, we have
Eay Ea
Σ Pw a1, az) (Qo(s, a:) log Pw (ils, 01,)
(19)
Solving for Pw given Th and ne. have
Pw (ils, a1 a2) a exp(Qo(s,a))aa,, Vi [1.2].
(20)
Therefore, for the case of two actions, we have
(1))s,ar,00)) exp(Qo(s,a:/al
Vi [1.]].
(21)
E, exp((Q((ss,))))
A.4 ADDITIONAL ABLATION RESULTS
We provide more ablations experiments and results in this section.
Double Parameters. Since there are two member policies in the pro- posed approach. A natural question to ask is whether properly doubling the number of parameters policy could achieve similar performance.
15

---

Published as a conference paper at ICLR 2023
0.6
Douple Parameters
To verify this, we construct type of policy network with (roughly) doubled number of parameters as (T(s)= - 9e, ofe, (s), where fo, (s) encodes the observation into feature vector and g maps the feature vector to the action distribution. To (approximately) double the number of parameters, we implement fo, as fe, - for fo, fa, is initialized by transferring parameters learned in the pre- train stage. The aggregated return curves are shown on the right, for doubling the number of policy parameters as described above (Double-Pon Param), and the proposed approach. It can be observed that by comparison that simply increasing the number of parameters cannot help much with the performance, implying the importance of the proposed structure in PEX.
0.6
0.5
Double Param
%
Envronment
146
A.5 MORE DETAILS ON TASKS
We summarized the tasks used for experiments and their corresponding D4RL environment names in
Table
Task
D4RL Environment Name
antmare-umaze
antmaze-umsze-vl
antmaze-umsze-diveere
antmaze-umaze-diiereeyy
antmarre-moduumpplayy
antmare-modium-pllyyyy
antmaze-meddum-diverss
antmaze-meddum-diieeeee
antmaze-large-play
antmaxe-large-playy-0
antmaze-larre-diieezss
antmazze-aargp-diieeee--0
hal.chheetah-raandomvzz
hopper-random
hopper-randommv2
walker-random
walker-raandom-v2
halfcheetah-mmediu
halfcheettal-mmddiumwv
hopper-medium
hopper-medium-v2
walker-medium
walker-medium--2
horper-medium-repllayy-
walker-mediun-repllay
walkar-madium-replay--rr
Table 1: Tasks and the corresponding environment names in D4RL.
A.6 SPPER-PARAMETERS
The hyper-parameters and values are summarized in Table There are two algorithmic related hyper- parameters inherited from IQL. since we instantiate PEX based on temperature a (corresponding to 3- in IQL paper (Kostrikov et al. 2022)) and expectile value т. The rest of the are common ones in RL algorithms, including for example batch size, network structure and size, target update etc. We use the hyper-parameter values from the IQL paper (Kostrikov et al. 2022) in our experiments.
A.7 SPPER-PARAMETER ABLATIONS
Impact of a. a scales the Q values before constructing Pw:
Pw[1] exp(a Qo(s,-)Q((s,u)))
Wi (....K
(22)
We show the impacts of its value performance in Figure We have use fixed value (a-1 10 for all antmaze tasks and a -1 3 for all locomotion tasks according to the IQL paper (Kostrikov et al. 2022) (the parameter of IQL). Here we show the impacts of different values for a by setting the inverse temperature as 0.5a 1a -1 2a We note that other values are possible for different environments. While it is impractical to tune this parameter for each task, it is possible to make it more adaptive by auto-tume auto-tuning in a way similar to the temperature turning scheme used in SAC (Haarnoja et al. 2018).
16

---

Published as a conference paper at ICLR 2023
Hyper-parameters
Values
number of parallel discount
0.99
replay buffer batch size
le6
256
MLP hidden layer size learning rate
[256. 256]
3e-4
initial collection steps target update speed expectile value
5000
5c-3
0.9 (0.7)
inverse temperature a -1 number offfine iterations number of online iterations number of iteration rollout step target entropy (SAC)
(3)
IM
1M
-d
Table 2: values. Values in brackets are used for locomotion tasks. The rest of the hyper-parameter values are shared across all tasks, d denotes the action dimension.
entmaaze-umaze-ivett perinaze-mediumppeee
hellcheetta-mmedium
hopper-meddum
08
05
%
150
Figure 7: Impact of inverse temperature a Results with inverse temperature as 0.5a la 2a respectively.
Impact of Policy Entropy. The entropy of Ty is auto-tuned to match a target entropy (Haarnoja ct al. 2018), which is empirically set as Etarget A -d where denotes the dimensionality of action. This is inherited from SAC and the same setting (-d) is applied to all tasks. To show the impact of different target entropy values, we experiment with three different settings: 0.5etarget= Ctarget- target several tasks. The results summarized in Figure 8 It can be observed that overly small or large target entropy values could decrease the performance on some tasks.
animaze-umsze dinezeeeiiveee
antmaae antmmaze-medimme medium-play
hallchhestah-medium
hopper-medium
06
68
Unsumsent
Envrrneent
Enovceenn
166
Figure Impact policy entropy (SAC-based online stage). Results with target entropy 0.5etarget- 1etarget 2etarget respectively.
A.8
MORE RESULTS ON HETEROGENEOUS OFFLINE-ONLING RL
Here we show detailed return curves for each task when using SAC (Haarnoja et al., 2018) as the backbone online RL algorithm. The results are shown in Figure 9
A.9
OFFLINE- TRAINING WITH REWARD-FREE OFFLINE SETTING
We have shown using policy expansion for bridging offline IQL with online IQL/SAC in the main paper. Here further demonstrate the applicability of the proposed approach by using BC the
17

---

Published as a conference paper at ICLR 2023
antmaow mediam-plyy
antmooe-meddnm undiiasa
aromaza large- play
Scritsh
Bullie
Diivet
Cryionme Geegs
Divii rooe dere
arrmaze large-deeee
popper-andam
Walker mandom
haticheeth--meccm
369
hoppor modium
walkerdd modium
haltchectah med um roploy
hopper medi roplay
walkorde modium reploy
0.5
Dovernent
04
140
50
6.0
0.5
34
Figure 9: Return Curves of different methods using SAC (Haarnoja et al. 2018 the online RL backbone algorithm, including the Olf2On method (Lee et al. 2021). The returns are normalized according to D4RL benchmark (Fu et al. 2020).
offline learning module. We use IQL the online module for illustration. In this section, we further demonstrate the applicability of the proposed approach to the setting with offline dataset without reward annotations. This sometimes an encountered setting in practice, where plenty of offline data exist (e.g. from previous iteration of experiments or other agents), but without reward annotations. Since the offline dataset is only broadly related to the task and could potentially contain irrelevant or noisy data (w.Ff. the current task in consideration), direct behavior cloning could be sub-optimal.
Standard offline RL methods are inapplicable for offline pre-training in this case. We therefore use behavior cloning (BC) (Pomerleau 1988), which is simple and widely used approach for imitation learning. After offline pre-training using BC, the offline policy Ty is obtained and transferred to the second stage of online training. In terms of Algorithm I only the policy loss Lz is present in the offline training phase. The offline datasets not used in online training since has no rewards.
hxppaa-tttellm
wea mobeer2d-mecc..m
0.8
08
35%
02
61
J5RL
640
Prafronmen Sanps
Enviniranc Saaye
Bosieemeet Aags
165
Figure 10: Return Curves for reward-free offline pre-training setting.
We use subset of tasks from D4RL (uu et al. 2020) where BC has non-trivial performances. The results are shown in Figure 10 It can be observed that JSRL, which performs well under the case of having offline rewards, performs less well in the reward-free case. One potential reason is that it hinges on the offline policy to induce the state distribution (Uchendu et al.) 2022). In the case where the offline policy is not highly performant, the case here obtained from BC, its effectiveness could be compromised. BT performs comparable to or better than Direct online fine-tuning with IQL., potentially due to the fact that BT leverage the म8 in way that has shorter commitment
18

---

Published as a conference paper at ICLR 2023
length for each time the offline policy is selected. The proposed approach also shows similar online improvements comparable or sometimes better than the best performing methods.
A.10 POLICY USAGE
We show the policy usage of 3g and me during online unroll along the process of training session for BT and JSRL as well as BC-PEX (pre-trained with BC) and IQL-PEX (pre-trate pree-trained with IQL) in Figure IT] The return curves of BC-PEX are shown in Figure TO The corresponding return curves of IQL-PEX are shown in Figure 2 As can be observed, the usage of offline policy is changing along the course of training, and the dynamics of the usage evolution is different for different tasks.
BT
Haloy
0.3
0.3
0.0
04
0.0
0.2
0.4
0.6
0.6
60
02
0.6
Taaning Derationa
Thainigg Barations
0.5
1oko thage
Palyy Usage
11000
JSRL
6.66
02-
0.3
70%
0.4
0.5
0.5
0.5
BC-PEX
velcy 100g
0.4
0.
0.2
2.6
0.5
Taaning Recations
100
1.6
146
166
1.5
0.2
IOL PEX
0.c
Potcy Usagg
0.4
DA
04
0.0
0.2
0.5
0.
0.0
0.0
044
0.5
3.0
Bolniing Tlavalans
Thaniig dilio
Tukkri Bavelott
Bal Ranaloos
1sc
Figure 11: Policy Usage of F. and me along the process of online training for different methods.
19


Reviews:
[{"review_id": "MPGyyDubEGu", "reviewer": "Reviewer_NB3z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses how online RL can benefit from an existing dataset and/or a policy created by offline RL. The suggested method proposes that both the offline policy (fixed) and the online policy (to be trained) provide action candidates, which are then evaluated using a Q-function (to be learned online). The evaluations are converted into selection probabilities and the action to be executed is drawn accordingly. The method is being tested on several D4RL benchmark tasks.", "summary_of_the_review": "A surprisingly simple approach to solve an important problem. The description of the approach should be more explicit in parts and thus more reproducible. The language should be improved in parts, errors should be fixed. ", "strength_and_weaknesses": "**Strengths**\n- The paper deals with a problem that is relevant for practical applications\n- The method is very interesting. I find it surprising that it works.\n\n**Weaknesses**\n- The, in my opinion, central aspect of how actions are selected is presented somewhat scarcely.  I am not sure that the method can be reliably re-implemented using the explanations provided.\n\n**Further suggestions for improvement**\n- I think the calculus of how the selection probabilities are calculated from the Q-values of the candidate actions should be explicitly presented.\n- The temperature parameter $\\alpha$ probably has central influence on the functioning of the method. It would be interesting to see this influence through experiments.\n- Also the entropy of the two policies may have a significant influence. It would be interesting to see this influence through experiments.\n- Some sentences could be worded more concisely. For example, sentence „After learning the offline policy, we use it as one candidate policy in a policy set, and further learn another policy that will be responsible for further learning as an expansion to the policy set\" seems a bit awkward and thus difficult to understand.\\\nI stumbled over „limiting its applicability to many practical scenarios“, wouldn't „limiting its applicability in many practical scenarios“ or „which limits its applicability in many practical scenarios“ be better?\n- Why do you write \"e.g.\" in italics while \"et al.\" is not italic? This seems inconsistent to me.\n\n„to a a number“\\\n„onehot“ -> „one-hot“\\\n„the policy set set as“\\\n„q-learning“ -> „Q-learning“\\\n„Should i run“ -> „Should I run“\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AdIbsqPAt_", "reviewer": "Reviewer_EsUG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the offline-to-online RL setting and proposes policy expansion (PEX) by constructing the stochastic online policy: a mixture of the offline policy (trained using the offline data) and the online policy (trained with the online rollouts). In the experiments, the paper shows that PEX has some performance gain over some very recent methods that consider a similar setting. ", "summary_of_the_review": "Overall, there are several flaws in the current version of the paper: 1. insufficient literature review; 2. baselines not strong enough (at least no argument to show that the baselines are strong enough); 3. some technical part includes heuristic that may seem rather random. Thus I think the current version of the paper does not deliver enough contribution to the community and thus would recommend a reject at the current stage. \n\n===========================\nraised my score after the rebuttal", "strength_and_weaknesses": "# Strength \n* The idea of policy expansion, i.e., instead of using the offline policy as a warm start, one can use the offline policy as a checkpoint and combine it with the online policy (which could warm start from the offline training) and still query the checkpoint online, seems a general framework. Also as the paper already points out, the PEX framework could work agnostically to the particular online and offline RL methods. \n* The experiments study is on the general D4RL benchmark, and the paper shows that PEX is competitive in most environments and outperforms some recent baselines such as JSRL and BT.\n* The ablation study gives us more insights into some specific design choices such as the benefit of transferring the offline Q-function and why we should freeze the offline policy. \n\n# Weakness\n* The work seems to overlook a few more previous works in this online RL with offline dataset setup, such as [1,2,3,4]. \n* [5] is also very relevant to this paper's setting, and seems even more relevant than the current baselines. I wonder why [5] is not considered as in the experiment section. Also, I believe the paper should contain more explanation on why the current baselines are significant enough (I might be wrong but the baselines seem both from unpublished papers?)\n* From the technical side, there are a few points that confuse me:\n 1. when does $P_W$ make sense? First there is minor ambiguity in the algorithm: when it says that sampling $a \\sim \\tilde \\pi$ according to (6), in the definition of $P_w$, $\\textrm{Q}$ seems arbitrary although we can assume in the context of the algorithm it may refer to $Q_{\\phi}$. However, $P_W$ is only a valid weight if $Q_{\\phi}$ is accurate, otherwise, it seems just some random heuristic to select which policy to execute. But if $Q_{\\phi}$ is accurate, then we should be done because an accurate $Q_{\\phi}$ should implies a near optimal policy and thus we don't need to rollout and collect data anymore. Adding on this, there is no explanation of Fig.9, which records the usage of the two policies: it seems like the coefficient stays almost the same during the entire training, does this mean the $Q_{\\phi}$ is already accurate at the beginning so minor updates of $Q_\\phi \\implies$ no changes in the amount of policy switching?\n\n2. Another issue is why we should ever rollout the offline policy $\\pi_\\beta$. The ablation shows that it is beneficial to include the offline dataset into the online replay buffer, which makes perfect sense. However, the action take by $\\pi_\\beta$ is just a reflection of the offline data distribution, because the pessimism/penalty applied in the offline RL algorithms. Thus why should we rollout with $\\pi_\\beta$ because it does not seem to help with exploration/increasing data diversity?\n\n* Finally, the stochastic policy constructed with both the offline policy/expert policy and the online policy seems relevant to the DAgger algorithm [6], with the different being that DAgger is using imitation learning loss while PEX is using RL loss. Nevertheless, it seems fair to include a discussion about such stochastic policy approaches. \n\n[1] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep Q-learning from demonstrations.\n\n[2] Stephane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning.\n\n[3] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations.\n\n[4] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rotho ̈rl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.\n\n[5] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets.\n\n[6] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "47YrRAFDAPg", "reviewer": "Reviewer_U35N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a scheme of policy expansion (PEX) to bridge offline and online RL. Instead of directly fine-tuning the policy learned by offline dataset during online interaction, PEX freezes the learned policy and expands the policy set with a newly added policy. This new policy is optimized during the online learning phase. Then, a single policy is derived from the policy set, where action candidates are sampled by each policy and then the final action is selected by a resampling distribution. In the D4RL benchmark, PEX overall outperforms all of the baseline algorithms.\n", "summary_of_the_review": "The idea of policy expansion is interesting, but it lacks theoretical justification. For the experiments, it would be great to make a comparison with more baselines (see Weaknesses part) and to see the generality of the proposed PEX framework by conducting experiments with different offline RL backbones.\n\n== post-rebutal\nThank you very much for your detailed responses. The added experiments are also greatly appreciated. I raised my score accordingly.", "strength_and_weaknesses": "[Strengths]\n1. This work provides a simple yet effective scheme for offline-to-online RL. The proposed method is easy to implement.\n2. In the experiments, PEX significantly outperforms baseline algorithms.\n\n[Weaknesses]\n1. In the experiments, comparisons with existing methods for offline-to-online RL are missing, e.g. [1,2].\n2. To claim that PEX is a general framework, rather than an IQL-specific algorithm, it would be great to demonstrate the effectiveness of diverse combinations of PEX and existing offline RL algorithm backbones. For example, it would be great to see the performance of CQL+PEX or FisherBRC+PEX and whether PEX can consistently outperforms the baselines even for those different backbones.\n3. The proposed method is designed heuristically, without theoretical justification.\n\n[Questions]\n1. In the PEX framework, the offline training phase is to learn a Q function and a policy, where the policy should be frozen while the value should be fine-tuned (i.e. transfer Q). I am wondering why value-finetuning is beneficial but policy-finetuning is detrimental to offline-to-online performance. It would be great to provide a more detailed explanation about this.\n2. In Figure 2, why is the performance of PEX in halfcheetah-random better than in halfcheetah-medium-replay even though medium-replay dataset would contain more high-reward experiences?\n3. It would be great to see the result using Q-transfer only without the policy expansion, i.e. initialize the Q-function with the pretrained one, and run an online RL algorithm using the Q initialization.\n4. How sensitive is the algorithm with respect to the hyperparameter alpha?\n\n\n[1] Nair et al., AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, 2020\n\n[2] Lee et al., Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble, 2021\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3d191jkVZTz", "reviewer": "Reviewer_2rLA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new offline-to-online finetuning approach that proposes to maintain both the policy learned offline and the new online policy during online-finetuning. The authors first train the offline policy and Q-function, select the policy from a categorical distribution with the learned Q-values during online data collection, and optimize the online policy using data sampled from both the offline dataset and the newly collected online samples. The offline policy is kept frozen during the finetuning step. The authors show that doing so can alleviate the distributional shift and over-conservatism issues when switching from offline training to online training with a single policy. Empirical results suggest that the method can outperform prior method IQL and its variants on the D4RL benchmark.", "summary_of_the_review": "Based on my comments above, I vote for a weak accept of the paper.", "strength_and_weaknesses": "Strength:\n\n1. The idea of using a policy set to reduce the distributional shift issue and conservatism when switching from offline to online training is neat.\n2. The empirical results are pretty impressive. the ablation studies are also quite thorough, clearly suggesting the benefits of different components of the algorithm.\n\nWeaknesses:\n\n1. I think the authors should compare the method to more offline-to-online works such as Lee et al., 2021, Yang et al,. 2021, Lu et al., 2022, Zheng et al., 2022 and etc. I think it is important to see if the method can do better than approaches that are not like IQL.\n2. The story of using a policy set seems a bit disconnected to the main point of the paper. The main idea of the paper is essentially to train a different online policy and maintain the offline policy for exploration whereas the policy set idea is more than this. I think the authors should make the writing more clear and to the point rather than making too many less relevant statements.\n3. The authors don't provide much theoretical insight of the method.\n4. The error bars of the experiments are quite high. I wonder if the authors should run more random seeds / do hypothesis testing to make sure the results are statistically significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster