Pre-Training for Robots: Leveraging Diverse Multitask Data via Offline Reinforcement Learning
Authors: Aviral Kumar
Anikait Singh
Frederik Ebert
Yanlai Yang
Chelsea Finn
Sergey Levine
Venue: Submitted to ICLR 2023 (2023)
Keywords: pre-training
robotics
finetuning
offline RL

Abstract: Recent progress in deep learning highlights the tremendous potential of utilizing diverse datasets for achieving effective generalization and makes it enticing to consider leveraging broad datasets for attaining more robust generalization in robotic learning as well. However, in practice we likely will want to learn a new skill in a new environment that is unlikely to be contained in the prior data. Therefore we ask: how can we leverage existing diverse offline datasets in combination with small amounts of task-specific data to solve new tasks, while still enjoying the generalization benefits of training on large amounts of data? In this paper, we demonstrate that end-to-end offline RL can be an effective approach for doing this, without the need for any representation learning or vision-based pre-training. We present pre-training for robots (PTR), a framework based on offline RL that attempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as a few as 10 demonstrations. At its core, PTR applies an existing offline RL method such as conservative Q-learning (CQL), but extends it to include several crucial design decisions that enable PTR to actually work and outperform a variety of prior methods. To the best of our knowledge, PTR is the first offline RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens. We present an accompanying overview video at https://www.youtube.com/watch?v=yAWgyLJD5lY&ab_channel=PTRICLR

Under review as a conference paper at ICLR 2023
PRE- TRAINING FOR ROBOTS: LEVERAGING DIVERSE MULTITASK DATA VIA OFFLINE RL
Anonymous authors Paper under double-blind review
ABSTRACT
Recent progress in deep learning highlights the tremendous potential of utiliz- ing diverse datasets for achieving effective generalization and makes it enticing to consider leveraging broad datasets for attaining more robust generalization in robotic learning as well. However, in practice we likely will want to learn a new skill in a new environment that is unlikely to be contained in the prior data. There- fore we ask: how can we leverage existing diverse offline datasets in combination with small amounts of task-specific data to solve new tasks, while still enjoying the generalization benefits of training on large amounts of data? In this paper, we demonstrate that end-to-end offline RL can be an effective approach for doing this, without the need for any representation learning or vision-based pre-training. We present pre-training for robots (PTR), a framework based on offline RL that at- tempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning At its core, PTR applies an existing offline RI. method such as conservative Q- learning (CQL), but extends it to include several crucial design decisions that able PTR to actually work and outperform a variety of prior methods. To the best of our knowledge, PTR the first offline RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demon- strations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in variety of toy kitchens. We present an accompanying overview video at this Anonymous URI: https: /www. youtube com wa wattch ch v yAWgyI.JDS Y&ab_ wwwpeechhnneeeeeeeeeeee
new task, with as few as 10 demonstrations.
INTRODUCTION
Robotic learning methods based reinforcement learning (RL) imitation learning (II.) have led to a number of impressive results (Levine et al., 2016; Kalashnikov et al., 2018; Young et al., 2020; Kalashnikov al., 2021; Ahn et al., 2022), but the generalization abilities of policies learned in this way are typically limited by the quantity and breadth of the data available to train them. In prac- tice, the cost of real-wond weal-world data collection for each task means that such methods often use smaller datasets, which leads to more limited generalization. A natural way to circumvent this limitation to incorporate existing diverse robotic datasets into the training pipeline of a robot learning al- gorithm, analogously to how pretraining on diverse prior datasets has enabled rapid finetuning in supervised learning fields, such as computer vision and NLP. But how can we devise algorithms that enable effective pretraining for robotic RL?
In most cases, answering this question requires a method that can pre-train on existing data from a wide range of tasks and domains, and then provide good starting point for efficiently learning new task in a new domain. Prior approaches utilize such existing data by running imitation learn- ing (IL) (Young al., 2020; Ebert et al., 2021; Shafiullah et al., 2022) by using representation learning (Nair et al., 2022) methods for pre-training and then fine-tuning with imitation learning. However, this may not necessarily lead to representations that can reason about the consequences of their actions. In contrast, end-to-end RL can offer more general paradigm. that can be effective for both pre-training and fine-tuning, and is applicable even when assumptions prior work are vi- olated. Therefore we ask, can we devise simple and unified framework where both the pretraining and finetuning process uses RL? This presents significant challenges pertaining leveraging large amounts of offline multi-task datasets, which would require high capacity models and this can be very challenging (Bjorck al., 2021).

---

Under review as a conference paper at ICLR 2023
In this paper, we show that multi-task offline di-
Offina Rt Pretmiring on Bridge Dition
Offine RL Fine-luning Targei Davse Bridge Data
RL pretraining verse multi-task demon- stration data followed by offline RL finetuning on very small num- ber of trajectories (as few as 10 trials, max-
dommene 150 lak dencee
pet san not Taxt
NJt Pur
Geathrikiin bridge andeee
imum of 15) can in- deed be made into an ef- fective robotic learning strategy that in practice can significantly outper- form methods based on imitation learning pre-training well as RL-based methods that do not employ pre- me--trining. This is surprising and significant, since prior work (Mandlekar et al., 2021) has claimed that IL methods are superior to offline RL when provided with human demonstrations. Our framework, which we call PTR (pre- (pre-training training for robots). is based on the CQL algorithm (Kumar et al., 2020), but introduces a number of design decisions that we show are critical for good performance and enable large-scale pre-training. These choices include a specific choice of architecture for providing high capacity while preserving spatial information, the use of group normalization, and an approach for feeding actions into the model that ensures that actions are used properly for value prediction. We exper- imentally validate these design decisions and show that PTR benefits from increasing the network capacity, even with large ResNet50 architectures, which have never been previously shown to work with offline RL. Our experiments utilize the bridge dataset (Ebert et al., 2021), which is an exten- sive previously collected dataset consisting of thousands of trials for a very large number of robotic manipulation tasks in multiple environments. A schematic of our framework is shown in Figure 1.
Papetrate bridge date
Brinige Dnte and Target Date
Figure I: Overview of PTR: We first perform general offline pre- wve-trainnng on diverse multi-task rohot data and subsequently finetune on one or several target tasks while mixing batches between the prior data and the target dataset using hatch mixing ratio of r.
The main contribution of this work is demonstration that PTR enable offline RL pre-training on diverse real-world robotic data, and that these pre-trained policies can be fine-tuned to learn new tasks with just 10-15 demonstrations. This is significant improvement over prior RL-based pre- training and finetuning methods, which typically require hundreds or even thousands of trials (Singh al., 2020; Kalashnikov et al., 2021; Julian et al., 2020; Chebotar al., 2021; Lee al., 2022a). We present detailed analysis of the design decisions that enable offline RL to provide an effective pretraining framework, and show empirically that these design decisions are crucial for good per- formance. Although the individual components that constitute PTR are based closely on prior work, we show that the novel combination of these components in PTR is important to make offline RI. into viable pre-training tool that can outperform other pre- pre-training approaches and other RL-based policy learning strategies.
2 RELATED WORK
A number of prior works have proposed algorithms for offline RL (Fujimoto et al., 2018; Kumar et 2019; 2020; Kostrikov et al., 2021a;b; Wu et al., 2019; Jaques et al., 2019; Fujimoto & Gu, 2021; Siegel et al., 2020). Especially, many prior works study offline RL. with multi-task data and devise techniques that perform parameter sharing(Wilson et al., 2007: Parisotto et al., 2015; Teh et al., 2017; Espeholt et al., 2018; Hessel et al., 2019), or perform data sharing or relabeling (Yu et al., 2021; Andrychowicz et al., 2017: Yu et al., 2022: Kalashnikov al., 2021; Xie & Finn, 2021). In this paper, our goal is not to develop new offline RL algorithms, but to show that these offline RL algorithms can be an effective tool to pre-train from prior data and then fine-tune to new tasks, and we illustrate the design decisions required to get such methods to work well.
Unlike methods for fine-tuning from a learned initialization (Nair et al., 2020; Kostrikov et al., 2021b; Lee et al., 2022c), which typically perform online interaction, we consider the setting where we do not use any online interaction and do not require access to a reward function. This resembles the problem setting considered by offline meta-RL methods (Li et al., 2019; Dorfman & Tamar, 2020: Mitchell et al., 2021; Pong et al., 2021; Lin et al., 2022). However, our approach is simpler as we simply fine-tune the very same offline RL algorithm and method, PTR, outperforms the meta-RL method from Mitchell et al. (2021). Xu et al. (2022) present a prompting- based few-shot adaptation approach based on transforrss that is also related to our work.
2

---

Under review as a conference paper at ICLR 2023
In our experiments, we compare our approach to other approaches that attempt to leverage large, diverse datasets via representation learning (Mandlekar et al., 2020; Yang & Nachum, 2021; Yang et al., 2021: Nair et al., 2022; He et al., 2021), as well as other methods for learning from human demonstrations, such as behavioral cloning methods with expressive policy architectures (Shafiullah al., 2022). We find that PTR generally outperforms these methods. We also perform an empirical analysis to identify the design decisions behind the improved performance of RL-based PTR on demonstration data compared to BC, and find that the gains largely come from the efficacy of the value function in identifying the most "critical" decisions in a trajectory. This insight is in contrast to prior work (Mandlekar al., 2021), which argues that offline RL methods often do not work on real robots with human demonstration data, and supports the analysis in prior work (Kumar et al., 2022) that aims to understand when offline RL outperforms BC with demonstration data.
Perhaps the most closely related to our work are prior methods that run model-free offline RL. on diverse real-world data and then finetune to new tasks (Singh et al., 2020; Kalashnikov et al., 2021; Julian et al., 2020; Chebotar et al., 2021; Lee et al., 2022a). These prior methods typically per- form online fine-tuning, which requires complex reset mechanisms and uses significantly more data. (Chebotar et al., 2021; Kalashnikov et al., 2021) require thousand trials for et al., 2020; Lee et al., 2022a) require multiple hours or even days of online collection to learn new tasks. While these prior methods are autonomous, our framework PTR performs offline fine-tuning to learn a new task with as few as 10 demonstrations and thus provides a complementary advantage.
task, and (Julian
3 PRELIMINARIES AND PROBLEM STATEMENT
RL methods are derived under the formal model of a Markov decision process (MDP), which is a tuple M = (S, A, r, no. 7). where S, A denote the state and action spaces, and 7 (s' (s, a). r(s, a) represent the dynamics and reward function respectively. (s(() denotes the initial state distribution, and y E 1) denotes the discount factor. The policy π (als) learned by RL agents must optimize the long-term cumulative reward, max, (TT) == as |E, (rr(s,,
Problem statement. Our goal is to learn ceneral-purposse initializations from broad, multi-task offline dataset and then finetune these initializations to specific downstream tasks. We denote the general- purpose offline dataset by D, which is partitioned into & chunks. Each chunk contains data for a given robotic task (e.g. picking and placing a given object) collected in a given domain (e.g. a particular kitchen). See Figure for an illustration. Denoting the task/domain abstractly using an identifier i, the dataset can be formally represented as D = U (i, Di), where we de- note the set of training tasks concisely as Tuain = [*+]. Chunk D, consists of data for given task identifier i, and consists of a collection of transition tuples, DD; = {(s), a.. ry, r,,, )== col- lected by demonstrator on task Our goal is to utilize this multi-task dataset D, to find the best possible policy for one or multiple target tasks (denoted without loss of generality as task Trarget = [k + I,... n)), for which no experience is observed in D. While the diverse dataset D does not contain any experience for the target tasks, we are provided with very small dataset of demonstrations D' := {Di+1, Di B+11"" the size of D* is extremely small: in our experiments for a given target task, such that a policy that simply ignores the diverse offline dataset is unlikely to succeed. Our goal is to attain the best possible policy for tasks Taiget the end.
D;, corresponding to each of the target tasks. Note that
consider between 10 to 15 demonstrations
Background and preliminaries. The Q-value of a given state-action tuple Q" (s. a) for a policy n the long-term discounted reward attained by executing action a state s and following policy n thereafter. The Q-function satisfies the Bellman equation Q7 (s. a) r(s, a) + >/ a [Q" a a')] Typical model-free offline RL methods (Fujimoto et al., 2018; Kumar et al., 2019; 2020) alternate between estimating the f-function of fixed policy ' using the offline dataset D and then improving the policy π to maximize the learned Q-function. Our system, PTR, utilizes one such model-free offline-Ri RI. method, conservative Q-learning (CQL) (Kumar et al., 2020). We discuss how we adapt CQL. for pre-training on diverse data followed by single-task finetuning Section 4.
Tasks and domains. As discussed, our problem involves pre-training on data from many tasks and domains, which we source from the bridge dataset Ebert et al. (2021), and finetuning to a new task in new domain. Our terminology for "task" and "domain" follows Ebert et al. (2021): task corresponds to skill-object pair, such as "put potato in pot" and a domain corresponds to a particular environment, which in the case of the bridge dataset consists of different toy kitchens, potentially with different viewpoints and robot placements. We assume the new tasks and environments come from the same training distribution, but are not seen in the prior data.
3

---

Under review as a conference paper at ICLR 2023
4 LEARNING POLICIES FOR NEW TASKS FROM OFFLINE RL PRE-TRAINING
To effectively solve new tasks from diverse offline datasets, a robotic learning framework must: (1) extract useful skills out of the diverse robotic dataset, and (2) rapidly specialize the learned skills towards an unseen target task, given only a minimal amount of experience from this target task. In this section, we present our framework, PTR, that provides these benefits by training a single, seghly-expressiing deep neural via offline RL, and then specializes it on the target task with a small amount of data. We will first present the key components of our robotic framework in Section 4.1 and then discuss our novel technical contributions, the practical design choices that are crucial for attaining good performance in Section 4.2.
4.1 THE COMPONENTS OF PTR
To satisfy both requirements (1) and (2) from above, our framework uses a multi-tast tallii-askkaa offline RL approach, where the policy and Q-function are conditioned on a task identifier. This allows us to share a single set of weights for all possible tasks in the diverse offline dataset, providing general- purpose pre-training procedure that can use diverse data. Once a policy is obtained via this multi- task pre-training process, adapt this policy for solving new target task by utilizing very small amount of target task data. We describe the two phases, pre- pre-training and fine-tuning, below:
Phase 1: Multi-task offfine RL pre-training. In the first phase, PTR learns single Q-function and policy for all tasks E Tuain conditioned on the task identifier i, i... Qa(s, a; i) and Ty (a|s, i), via multi-task offline RI. We use one-hot task identifier that imposes minimal assumptions on the task structure. For multi-task offline RI., we use the conservative Q-learning (CQL) (Kumar et al., 2020) algorithm, extending to the multi-task setting. This amounts to training the multi- task Q-function against a temporal difference error objective along with a regularizer that explicitly minimizes the expected Q-value under the learned policy To (a(a{s;)), to prevent overestimation of Q-values for unseen actions, which can lead to poor offline RL performance (Kumar et al., 2019). Formally, the training objective for our multi-task Q-function, as prescribed by CQL, is given by:
min a
E (Q;(s,a;
E 1Q0(s, a; T- 8,
+ 2
1
E
(Q) - (Q(('aa)))) [(0():))))---(((aa))}
i-
s.a,5' a/OuT
Q denotes a target Q-network, which delayed copy of the current Q-network. We train o by running gradient descent on the above objective, and then optimize the learned policy to maximize the learned Q Q-values, along with an additional entropy regularizer as shown below:
max ET [Ea-to(Iss] Qo a: i)]l + (M((T))
At the end of this multi-task offline training phase, we obtain policy xy and Q-function Qg! that ready be finetuned a new downstream task.
Phase 2: Offline fine-tuning of off and Q to target tasks Turrett In the second phase, PTR attempts to learn a policy to solve one more downstream tasks by adapting =g using limited set of user-provided demonstrations that we denote D Our method for adaptation is simple yet effective: we incorporate the target task data into the replay buffer of the very same offline multi-task CQL algorithm from the previous phase and resume training from Phase 1. However, naively incorporating the target task data into the replay buffer might still not be effective since this scheme would hardly ever train on the target task data during adaptation due to the large imbalance between the sizes of the few target demonstrations and the large pre-training dataset. To address this imbalance, each minibatch passed to multi-task CQL during offline fine-tuning consists of a fraction of transitions from bridge demonstration data and target dataset. By setting to be small, we are able to prioritize multi-task CQL to look at target task data frequently, enabling it to make progress on the downstream task without overfitting-
fraction of transitions from the
Handling task identifiers for new tasks. The description of our system so far has assumed that the downstream test tasks are identified via a task-identifier. In practice, we utilize a one-hot vector to indicate the index of a task. While such a scheme is simple to implement, it is not quite obvious how we should incorporate new tasks with one-hot task identifiers. In our experiments, we use two approaches for solving this problem: first, can utilize a larger one-hot hot encoding that incorporates tasks in both Train and Turgets but never train the network corresponding to Traget- The f-function and the policy trained these placeholder task identifiers only during fine-tuning in Phase 2.
4

---

Under review as a conference paper at ICLR 2023
Another approach for handling new tasks is to not use unique task identifiers for every new task, but rather "re-target" or re- re-purpose existing task identifiers for new target tasks in the fine-tuning phase. PTR provides the option: can simply assign an already existing task identifier to the target demonstration data before fine-tuning the learned Q-function and the policy. For example, in our experiments in Section 5 we re-target the put sushi in pot task which uses orange transparent pots to instead put the sushi into a metal pot, which was never seen during training.
A complete overview of our approach is shown in Figure 1. task CQL and t = 0.8 for mixing the pre-training dataset and the target task dataset in most of our experiments in the real-world, without requiring any domain -spaiii tuning.
We use value of 10.0 in multi-
4.2 IMPORTANT DESIGN CHOICES AND PRACTICAL CONSIDERATIONS
Even though the components discussed in Section 4.1 are sufficient to give rise to an offline pre- training and fine-tuning algorithm, as we show in Section 4, this approach does not lead very good results just on its own. Instead, we must make some crucial design decisions, including designing neural network architectures that can learn from diverse data, cross- valsidaiiiitionnn metrics to identify policies expect be effective after fine-tuning, and the design of the reward functions that can be used to label the pre-training dataset. We will show that making the right choices for these com- ponents leads significant improvement (more than 3.5x in final real-world performance). Thus, describing, analyzing, and evaluating these design choices is an important part of this work that we hope will facilitate real-world applications of offline RL pretraining.
Policy and Q-function architectures. Perhaps the most crucial design decision for our ap- proach is the neural network architecture for representing nf and Q Since we wish to fine-tune the policy for different tasks, we must use high-capacity neural network models for representing the policy and the Q-function. We experimented with a variety of standard (high- capacity) architectures for vision- robotic RL. This includes standard convolutional archi- tectures (Singh al., 2020) and IMPALA chitectures (Espeholt et al., 2018). However, we observed that these standard models were unable to effectively handle the diversity of the pre-training data, and often collapsed. Then, we attempted to utilize standard ResNets (He et al., 2016) (ResNet-18 18, Resnet-34, and their adap- tations to imitation problems from Ebert et al. (2021)) to represent Qs but faced divergence chal- lenges similar to prior efforts with batch normalization (Bjorck et al., 2021; Bhatt et al., 2019). We found that by simply replacing batch normalization layers - known to be hard to train with TD- learning (Bhatt et al.. 2019) with group normalization layers (Wu & He, 2018), we were able to stably train ResNet Q-functions. See Appendix E for quantitative studies comparing these choices. Unlike prior work (Lee et al., 2022b), observed that with group normalization, we attain favor- able scaling properties of PTR with parameters: the more the parameters, the hetter the performance as shown in Figure 6. We also observed that choosing an appropriate method for converting the 3-dimensional feature-map tensor produced by the ResNet into a one-dimensional embedding plays crucial role for learning accurate Q-functions and obtaining functioning policies. Unlike standard ResNet architectures for supervised learning, simply computing global average pooling (as used in many classification architectures) performs poorly. Instead we point-wise multiply the learned feature-map with a 3-dimensional parameter tensor before computing sums over the spatial dimen- sions which allows the network to explicitly encode spatial information. We refer to this technique as "learned spatial embeddings". An illustration of this architecture is provided in Figure 2. As detailed in Appendix E, Table 11, we find that utilizing this leads to improved performance.
Lyyes
ResNer34
Outu Reaher
Tex E
eYbecdr go
speld
depilcaadd
The Q-function architecture for PTR. The encoder is a ResNet34 with group normalization along with learned spatial embeddings (left). The decoder (right) is MLP with the action vector duplicated and passed in at each layer. A one-hot task identifier is also passed into the input of the decoder.
Figure 2:
Next, we found that a Q-function (s a) obtained by running naive multi-task CQL tends to not use the action input a effectively, due to strong correlations between s and a in the offline data, which is almost always the case for lose-too-optin trajectories. As a result, policy improvement against such a Q-fucctoon overfits to these correlations, producing poor policies. To resolve this issue, we modified the architecture of Q-network, to pass the action input at each fully connected
5

---

Under review as a conference paper at ICLR 2023
layer, which (as shown in Figure 2 and Appendix E, Table 12), greatly alleviates the issue, and significantly improves the performance (significantly compared to naive CQL).
Cross-validatio after finetuning. Since we wish to learn task-specific policies that do not overfit to small amounts of data, thereby los- ing their generalization ability, we must ap- ply the right number of gradient steps during finetuning: too few gradient steps will pro- duce policies that do not succeed at the tar- get tasks, while too many gradient steps will give policies that have likely lose the gener- alization ability of the pre-trained policy. To handle this trade-off, we adopt the following heuristic as loose guideline: we run fine- tuning for many iterations while also plotting the learned Q-values over held-out dataset of trajectories from the target task. Then, we pick the checkpoint for which the learned Q-values most closely resemble a valid Q- function, which must increase over the time steps in the trajectory (see Figure 3; Appendix F). Empirically we find that this heuristic guides us to identify good checkpoints (more details in Appendix F).
X
Zeoo QrArees Ove Fiietand
Chosen Fixtuned Chockuo
Figure 3: Top: PTR policy rollout of task "put sushi in pot" re-targeted to metal-pot. Bottom: left: Q-value over time for a target task trajectory before fine-tuning begins (zero-shot), middle: Q-values for a checkpoint that has started to overfit after being trained for long and exhibits drastic changes in Q-values over the course of trajectory, and right: Q-values for a checkpoint that attains high per- formance.
Reward specification. In this paper, we aim to pre-train on existing robotic datasets, such as the bridge dataset (Ebert et al., 2021), which consists of cema-tteleoperaeed demonstration data. Al- though the demonstrations are all successful, they are not annotated with any reward function. Per- haps an obvious choice is to label the last transition of each trajectory as a success, and give it binary reward. However, in several of the datasets we use, there can be 0.5-1.0 second lag be- tween task completion and when the episode is terminated by the data collection. To ensure that a successful transition is not incorrectly labeled as O, we utilized the practical heuristic of annotating the last n = 3 transitions of every trajectory with a reward of +1 and and annotated other states with a 0 reward. We show in Appendix C that this provided the best results. In principle, more complicated methods of reward labeling (Eysenbach et al., 2021) could be used. However, we found the presented rule be simple and yet effective to learn good policies.
5 EXPERIMENTAL EVALUATION OF PTR AND TAKEAWAY'S FOR ROBOTIC RL
The goal of our experiments is to validate if PTR can learn effective policies from only a handful of user-provided demonstrations for a given target task, by effectively utilizing previously-collected robotic datasets for pre-training. We also aim to understand whether the design decisions introduced in Section 4.2 are crucial for attaining good robotic manipulation performance. To this end, we evaluate PTR in variety of robotic manipulation settings, and compare it to state of the art methods that either do not use any form of offline RL or do not learn end-to-end by employing some form of visual representation learning. We are considering three scenarios: (a) when the target task requires retargeting the behavior of an existing skill, in this case changing the type of object types it interacts with, (b) when the target task requires performing previously observed task but this time in previously unseen domain, and (c) when the target task requires learning a new skill in a new domain, by using the target demonstrations. For more details and visuals, please visit our anonymous website at https: /bitt.itt//.. 1y/pTR_I/PCTCA
Real-world experimental setup. We directly utilize the publicly available bridge dataset Ebert et al. (2021) for pre-training, as it provides a large number of robot demonstrations for diverse set of tasks in multiple domains, i.e. multiple different toykitchens. We use the same WidowX250 robot platform for our evaluations. The bridge dataset contains distinct tasks, each differing in terms of the objects that the robot interacts with and the domain the task is situated in. We assign a different task identifier to each task in the dataset for pre-training. We also evaluate an additional door-on task not present in the bridge dataset, where we collected demonstrations for opening and closing a variety of doors, and test our system new, unseen doors. More details are in Appendix A.
Comparisons. Since the datasets we use (both the pre-training bridge dataset from (Ebert et al., 2021) and the newly collected door opening data) consist of human demonstrations, indicated by

---

Under review as a conference paper at ICLR 2023
Sahtin Poi
medil
Leamplex
out Target Domain
Bomples ridge c)
b)
Figure 4: Illustrations of the three real-world experimental setups we evaluate PTR on: (a) the "put sushi in metallic pot" task which requires retargeting, (b) the task of opening unseen door, and (c) fine-tuning on several novel target tasks in held out toykitchen environment.
prior work (Mandlekar et al., 2021), the strongest prior method in this setting is behavioral cloning (BC), which attempts to simply imitate the action of the demonstrator based on the current state. We incorporate BC in a pipeline similar to PTR, denoted as BC (finetune), where pre-training dataset, and then finetune it using the demonstrations on the target task using the same batch mixing in PTR. Additionally, the validation error on a held-out offline dataset. Next, to assess the importance of performing pre- compare PTR to (i) the COG approach of Singh et al. (2020),
first run BC on the
tune the hyperparameters of all BC baselines against
training followed by fine-tuning, which is equivalent to jointly training with CQL on the pre-training data and the target task data from scratch, and (ii) multi-task offline CQL (CQL (0-shof)) that does not use the target demonstrations at all. We also make the analogous comparison for BC, jointly training BC on the pre-training and target task data from scratch (BC (joint)) which is equivalent (Ebert al., 2021). For fairness of comparison, BC, CQL, and PTR (both for O-shot, joint-training and fine-tuning) use the same exact architecture including our learned-spatial embedding described in subsection 4.2. In Scenario 3, compare PTR to other state-of-the the-art representation learning methods, and policy architectures. More implementation details can be found on our anonymous website.
Scenario I: Re-targeting skills for existing tasks to act new objects during finetuning. We utilized the subset of the bridge data with all pick-and- place tasks in one toy kitchen for pre-training, and selected the "put sushi in pot" task as our target task. This task is demonstrated in the bridge dataset, but only using an orange trans- parent pot (see Figure 4 (a)). In order to pose a scenario where the offline policy at the end of pre-training must be re-targeted to act on a different object, we collected only ten demonstrations that place the sushi in metallic pot. This scenario is challenging since the metallic pot drastically differs from the orange transparent pot visu- ally. By pre-training on all pick-and-place tasks in this domain (32 tasks) and jointly fine-tuning on this data and 10 demonstrations, PTR able to obtain a policy that is re-targeted towards the metal pot. BC appears to be mistaking arbitrary patches on the tabletop with the pot. Quantitatively, observe in Table 1, that PTR is able to complete the task with reasonable accuracy, whereas 0-shot and fine-tuned BC are completely unable to solve the task. The fact that O-shot CQL has great difficulty solving the task indicates that target demonstrations are necessary for solving this task, and PTR is able to make efficient use of these demonstrations (prior work Ebert et al. (2021) also found BC performs poorly with even 50 demonstrations)
Method BC (0-shot)
Success rate
0/30
BC (finetune)
0/30
CQL (0-shot)
2/30
PTR (Ours)
14/30
Table 1:
Performance of PTR for "put sushi in metallic pot" in Scenario 1. PTR substantially outperforms BC (finetune), even though it is provided access to only demonstration data. We also show some examples comparing some trajectories of BC and PTR in Appendix C.
Scenario 2: Generaliz- ing to previously unseen domains. Next, we study whether PTR can adapt behaviors seen in the pre- training data to new do- mains. We study door opening task, which re- quires significantly more complex maneuvers and precise control compared to the pick-and-ples -lacce tasks from above (video on the anonymous website). The target door (shown in Figure 4(b)) we wish open and the corresponding toy kitchen domain is never seen previously in the pre-training
O-shot
Joint Training
Target data only
Task
PTR (Ours)
BC (fine.)
CQL
BC
COG
BC
CQL
BC
Open Door
12/20
10/20
0/20
0/20
5/20
7/20
4/20
7/20
Table 2: Successes vs. total trials for opening a new target door in See- nario 2. PTR outperforms both BC (finetune) and BC (joint) given access to the same data. Note that joint training is worse than finetuning from the pre-trained initialization.
7

---

Under review as a conference paper at ICLR 2023
data, and doors in the pre-training data exhibit different sizes, shapes, handle types and visual ap- pearances. Due to the limited number of demonstrations and the associated task complexity, in order to succeed, an algorithm must effectively leverage the pre-training data. Concretely, for pre- training, we used a dataset of 800 door-opening demonstrations on 12 different doors in 4 different toy kitchen domains, and utilize 15 demonstrations on a held-out door for finetuning. Table 2 shows that PTR improves over both BC baselines and COG.
Interestingly, Table 2 shows that while COG by itself does not outperform BC (joint), the pre- training and fine-tuning approach in PTR leads to significantly better performance, improving over the best BC baseline despite using the same CQL algorithm as COG. Since CQL (joint) is equivalent to PTR, but with no Phase 1, this large performance gap indicates the efficacy of offline RL methods trained on large diverse datasets at providing good initializations for learning new downstream tasks. We believe that this finding may be of independent interest to robotic offline RI practitioners: when utilizing multi-task offline RI, it might be better first to run multi-task pre-training followed by fine-tuning, opposed to jointly training from scratch.
BC finetoning
training Target data only
Pre-train. rep.
Meta-leaminn
Tink
(Oun)
(Banc)
Autureg: BC
|QLL
BC
MACAW
Take emrieant from metal hool
2/10
5110
5100
1/10
4710
4/10
0/10
1/10
1/10
3/10
0/10
Put sowol us plnz
7/20
1220
1/20
0/20
0020
0.20
0/30
0/30
0.20
1/20
0/20
Plase kniffe pul
4/10
2210
2110
0710
1710
3/10
371
0/10
0/10
0/10
0/10
Put Puscuncneer put
5/00
0/10
110
3/10
2/0
1/10
0730
0/0
0/0
0/10
0/10
Table 3: Performance of PTR and other baseline methods for new tasks in Scenario 3. Note that PTR out-
performs all other baselines including BC (finetune). BC with more expressive policy classes (BeT (Shafiullah al., 2022), Augo-regressivv),, representation learning methods (Nair et al., 2022; He et al., 2021) and offiine RL with no pre-training ("Target data only") and joint training (Singh et al., 2020; Ebert al., 2021).
Scenario 3: Learning to solve new tasks in new domains. Unlike the two scenarios studied above, in this scenario, we attempt to solve a new task in a new kitchen scene. This task is represented via unique task identifier, and we are not provided with any data for this task identifier, even any data with from the kitchen scene where this task is situated during pre-training. We pre-train on all 80 pick-and-place style tasks from the bridge dataset, while holding out any data from the new task kitchen scene, and then fine-tune on 10 demonstrations for 4 target tasks independently in this new kitchen, as shown in Table 3. Recent representation learning methods (R3M, MAE+BC) more expressive policy architectures (Auto-regressive and Be'T) do not lead to improved performance compared the standard BC (finetune) approach. and we find that PTR outperforms all of these approaches. Please find more details on the implementation of R3M, MAE+BC Auto-regressive and BeT in Appendix D. This might appear surprising, and perhaps just a hyperparameter tuning artifact at first, but we present additional qualitative and quantitative analysis aiming at understanding the reasons behind why our offline RL-based PTR approach works better in the following paragraph. Figure 5 shows comparison of rollouts of the final policy found by PTR with rollouts from BC finetuning policy on the "take croissant from metal bowl" and "put cucumber in bowl" tasks. We also present additional rollouts in our anonymous website.
Understanding why PTR outperforms BC baselines. One natural question to ask given the results in this paper is: why does utilizing offline RL method for pre-training and finetuning as in PTR out- perform BC-based methods even though the dataset is quite "BC-friendly", consist- ing of only demonstrations? The answer to this question is not obvious, especially since joint training with BC (BC (joint)) still outperforms COG in our results in Ta- ble 3. Intuitively, we might expect that the RL-based PTR method might be better able to identify important decision points in the data, thus learning more control- centric representations, as suggested in prior work (Kumar et al., 2022). But can we analyze this question more precisely?
Qualitative Comparison BC (finetune) and PTR
Fabro graps bovel sataue paaAA
Succesc grani OISAAE pata
and
GuOCILL:
Figure 5: Qualitative successes of PTR visualized along- side failures of BC (finetune). As an example, observe that while PTR is accurately able to reach to the croissant and grasp it solve the task, BC (finetune) imprecise and grasps the bowl instead of the croissant resulting in failure.
8

---

Under review as a conference paper at ICLR 2023
To understand the reason behind improvements from RL, we perform a qualitative evaluation of the policies learned by PTR and BC (finetune) on two tasks: take croissant from metal bowl and put cucumber in bowl in Figure 5. We find that the failure mode of BC policies can be primarily explained as a lack of precision in locating the object, or a prematurely-executee grasping action. This is especially prevalent in settings where the object of interest is farther away from the robot gripper at the initial state, and hints at the inability of BC to prioritize learning the critical decisions (e.g., precisely moving over the object before the grasping action) over non-critical ones (e.g., the action to take to reach nearby the object from farther away). On the other hand, RL can learn to make such critical decisions correctly as shown in Figure 5.
More coneretely, to verify if the performance benefits can be explained entirely by the abil- ity of Q-learning to prioritize critical deci- sions, we run version of weighted behavioral cloning, where the weights are derived from the advantage estimates computed using a frozen Q-function learned by PTR after fine-tuning: (o a)= ei=pexp(4e (s, a) maxx (s a').. As shown in Table 6, we find that this advantage- weighted BC (AW-BC) approach performs sig- nificantly better than BC (finetune) method and comparably to PTR, for two tasks (croissant and cucumber from Table 3. Since AW-BC is essen- tially the same as BC, just with a modified weight to indicate the importance of any transition, this performance improvement clearly indicates the benefits of learning value functions via RL in a pre-training then fine-tuning setting, even when we only have demonstration data. Note that since AW-BC uses the Pee-htsghts after fine-tuning, it cannot serve as an independent method.
Task
BC (finctune)
PTR AW-BC (finctune)
Cucumber Croissunt
0/10
5/10
5/10
3/10
7/10
6/10
Table 4: Performance of ddvantage-weightee BC on two tasks from Table 3. Observe that weighting the BC objective using advantage estimates from the Q-function learned by PTR leads to much better per- formance than standard BC (finetune), almost recover- ing PTR performance. This test indicates that the Q- function in PTR allows us be accurate on the more critical decisions, thereby preventing the failures of BC.
Our design decisions enable us to effectively leverage high- capacity neural networks. To understand the importance of designing techniques that enable us to use high-capacity mod- els for offline RL, we examine the efficacy of PTR with dif- ferent Q-networks, on the open door task from Scenario 2, and the put cucumber in pot and take croissant out of metallic bowl tasks from Scenario 3. We compare to standard three-layer convolutional network architectures used by prior work for DM-control tasks (see for example, Kostrikov et al. (2020)), an IMPALA (Espeholt et al., 2018) ResNet that consists of 15 convolutional layers spread across a stack of 3 residual blocks, and the ResNet 18, 34, and 50 architectures with our proposed design decisions. Observe in Figure 6, that the performance of smaller networks (Small, IMPALA) is significantly worse than the ResNet in the door opening task. For the pick-and-place tasks that contain a much larger dataset, Small, IMPALA and ResNet18 all perform much worse than ResNet 34 and ResNet 50. In Appendix E we show that ResNet 34 models perform much worse our prescribed design decisions are not used. We also perform diagnostic study in simulation, whose details can be found in Appendix C (Table 7), and these support our real-world results.
Scaling Trand for PTR
Open
Place
Average Baroee Date
0.525
02175
Small Cany
IVPNLA
Theete Resber Raellad5)
Figure 6: Scaling trends for PTR on the open door task from Scenario 2, and average over two pick and place tasks from Scenario 3. Note that with our design decisions, PTR is able to effectively benefit from high capacity function approximators.
CONCLUSION AND DISCUSSION
We presented a system that uses diverse prior data for general-purpose offline RL pretraining, fol- lowed by fine-tuning to downstream tasks. The prior data, sourced from a publicly available dataset, consists of over hundred tasks across ten scenes and our policies can be fine-tuned with as few as 10 demonstrations. We show that this approach outperforms prior pre-training and fine-tuning methods based on imitation learning. One of the most exciting directions for future work is to further scale up this pre-training to provide a single policy initialization, that can be utilized as a starting point, similar to GPT3 (Brown et al., 2020). A limitation of our method is that it requires the prior data and new tasks to be structurally similar and an exciting future direction is to scale it up to more complex settings, including novel robots. As observed that joint training with offline RL (COG) was worse than pre-training and then fine- tuning with PTR, another exciting direction for future work is to understand the pros and cons of joint training and fine-tuning in the context of robot learning.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances neural information processing systems, 30, 2017.
Aditya Bhatt, Max Argus, Artemij Amiranashvili, and Thomas Brox. CrossNorm: Normalization for Off-Policy TD Reinforcement Learning. arXiv e-prints, art. arXiv:1902.05605, February 2019.
Johan Bjorek, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learn- ing. arXiv preprint arXiv:2/06.0/14 151, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749 2021.
Ron Dorfman and Aviv Tamar. Offline meta reinforcement learning. arXiv e-prints, pp. arXiv-2008, 2020.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 13396, 2021.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. International Conference on Machine Learning. pp. 1407-1416. PMLR, 2018.
Ben Eysenbach, Sergey Levine, and Russ Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Pro- cessing Systems, 34:1 15411114-11152 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.0688 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.029000 2018.
He, Chen, S Xie, Li, Dollár, and RB Girshick. Masked autoencoders are scalable vision learners. arxiv. 2021 doi: 10.48550. arXiv preprint urXiv.2111.0000 1.06377, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu. and Rosalind Picard. Way off- off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint mxXiv:1907 004.000456 2019.
10

---

Under review as a conference paper at ICLR 2023
Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning.
arXiv preprint ar a0X1v:2004.10190, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Scalable deep reinforce- ment learning for rision-based rohotic manipulation. In Conference on Robot Learning, pp. 651- 673, 2018.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic re- inforcement learning scale. arXiv preprint arXhv:2104.08212, 2021.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 1,00004.33649 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 2XXv:20006047779 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should i run offline reinforcement
learning or behavioral cloning? In International Conference on Learning Representations, 2022.
URL https::t//..m /etetp::/oppeee /ttp///pp//////eeeeeeeeee http://pppnneeeeeeeeeeee net forem/oormmmmm wwt.:iddd-PP1311111.
Alex X Lee, Coline Devin, Jost Tobias Springenberg. Yuxiang Zhou, Thomas Lampe, Abbas Ab- dolmaleki, and Konstantinos Bousmalis. How spend your robot time: Bridging kickstart- ing and offline reinforcement learning for vision-based robotic manipulation. arXiv preprint 20Xi::2005.03333, 2022a.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint arXiv:2205.1 15241, 2022b.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference Robot Learning, pp. 1702-1712. PMLR, 2022c.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1375 2016.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Keith Ross, Henrik Iskov Chris- tensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. arXiv preprint arXiv: 1337, 2019.
Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang, and Junshan Zhang. Model-based offline meta- reinforcement learning with regularization. arXiv preprint arXiv:2202.02929 2022.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4414-4420 IEEE, 2020.

---

Under review as a conference paper at ICLR 2023
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkamn,, Fei-Fei Li, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In 5th Anmual Conference Robot Learning, 2021. URL
ew net/forune forum? ddddrsffJtDddddd oe
Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline Meta- Reinforcement Learning with Advantage Weighting. arXiv e-prints, art. arXiv:2008.06045 Au- gust 2020.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint 26X60 2022.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint 1.063432, 2015.
Vitchyr H Pong, Ashvin Nair, Laura Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. arXiv preprint arXiv:2107.03974 2021.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning & modes with one stone. arXiv preprint arXiv:2206.1122: 2022.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Be- havioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills past experience with offline reinforcement learning. arXiv preprint 1.Xiv:0010.145000 2020.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: 1110100077.0111 2017.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference Machine learning, pp. 1015-1022, 2007.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911:v1111.11300. 2019.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences. arXiv preprint 109.99900 2021.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference on Machine Learning, pp. 24631-24645. PMLR, 2022.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci- sion making. arXiv preprint 1rXiv:2102.05815. 2021.
Mengjiao Yang, Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data. arXiv preprint arXiv:21 2021.
12

---

Under review as a conference paper at ICLR 2023
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto.
Visual imitation made easy, 2020.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural In- formation Processing Systems, 34, 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022.
13

---

Under review as a conference paper at ICLR 2023
Appendices
A DETAILS OF OUR EXPERIMENTAL SETUP
A.1 REAL- WEALD EXPERIMENTAL SETUP
A picture of our real-world experimental setup shown in Figure 7. The scenarios considered in our experiments (Section 5) are designed to evaluate the performance of our method under a variety of situations and therefore we set up these tasks different toykitchen domains (see Figure 7) on three different WidowX 250 robot arms. We data from the bridge dataset (Ebert et al., 2021) consisting of data collected with many robots in many domains for training but exclude the task / domain that use for evaluation from the training dataset.
(3)
(4)
Figure 7: Setup Overview:
Following Ebert et al. (2021), we use a toykitchen setup described that prior work for our experiments. This utilizes 6-DoF WidowX 250 robot. (1): Held-out toykitchen used for exper- iments in Scenario 3 (denoted "toykitchen 6"), (2): Re-targeting toykitchen used for experiments in Scenario 2 (denoted "toykitchen 2"), (3); target objects used in the experiments of scenario 3., (4): the held-out kitchen setup used for door opening ("toykitchen 1").
A.2 DIAGNOSTIC EXPERIMENTAL SETUP IN SIMULATION
In simulation, we evaluate our approach in a simulated bin-sorting task on the simulated WidowX 250 platform, aimed to mimic the setup we use for our real-world eval- uations. This setup is designed in the PyBullet simulation framework provided by Singh et al. (2020). A picture is shown in Figure 8. In this task, two different bins and two different objects are placed in front of the Widow X robot. The goal of the robot is correctly sort each of the two objects to their designated bin (e.g the cylinder supposed be placed in the left bin and teapot should be placed in the right bin. We refer to this task as a com- pound task since it requires successfully combining be- haviors of two different pick-and-place skills one after the other in a single trajectory while also adequately identify- ing the correct bin associated with each object. A success counted only when the robot can accurately sort both of the objects into their corresponding bins.
Figure 8: Bin-Sorting task used for our simulated evaluations. The task requires sorting the cylinder into the left bin and the teapot into the right bin.
14

---

Under review as a conference paper at ICLR 2023
Figure 9: Some trajectories from the pre- pre-training data used in the simulated bin-sort task.
Figure 10: The five demonstration trajectories used for Phase of PTR.
Offline pre-training dataset. The dataset provided for offline pre-training only consists of demon- strations that show how the robot should pick one of the two objects and place it into one of the two bins. Each episode in the pre-training dataset is about 30-40 timesteps long. A picture showing some trajectories from the pre-training dataset are shown in Figure 9. While the downstream task only requires solving this sorting task with two specific objects (shown in Figure 10), the pre-training data consists of 10 unique objects (some shown Figure 9). The two target objects that appear together in the downstream target scene are pre-training data only demonstrates how the robot must pick up one of the objects and place it in of the two bins (not necessarily in the target bin that the target task requires), it neither consists of any behavior that places objects into bins sequentially, nor does it consist of any behavior where one of objects is placed one of the bins while the other one is not. This what makes this task particularly challenging.
together in the pre-training data. Since the
Target demonstration data. The target task data provided to the algorithm consists of only five demonstrations that show how the robot must complete both the stages of placing both objects (see Figure 10). Each episode in the target demonstration data is 80 timesteps long, which is substantially longer than any trajectory in the pre- pre-training data, though one would hope that good representations learned from the pick and place tasks are still useful for this target task. While methods are able to generally solve the first segment of placing the first object into the correct bin, the primary challenge in this task is to effectively sort the second object, and we find that PTR attains substantially better success rate than other baselines in this exact step.
B DESCRIPTION OF THE WORLD EVALUATION SCENARIOS
In this section, describe the real-world evaluation scenarios considered in Section 5. We addi- tionally include a much more challenging version of Scenario 3, for which we present results in Appendix C. These harder test evaluate the finetuning performance four different tasks,
15

---

Under review as a conference paper at ICLR 2023
Ratarget metal-pol, using demos in target domain
Pm--Trn Brdge Data toysinhen
taskoy
Illustration of pre-training data and finetuning data used for Scenario 1: re-targeting the put sushi in metal-po pot behavior to put the object the metal pot instead of the orange transparent pot.
Figure 11:
2. Fine-Ture on Targer Domain Datu: door, 15
demonstrations
Tran on Bridge Uata, 12 doons 800 demonstrations. dfferent toy- ctchens
Figure 12:
Illustration of pre-training data and fine-tuning data used for Scenario 2 (door opening): transferring behavior to a held-out domain.
starting from the same initialization trained bridge data except the toykitchen 6 domain in which these four tasks were set up. In the following sections, nomenclature for the toy kitchens is drawn from Ebert al. (2021) and described in the caption of Figure 7.
B. SCENARIO 1: RE-TARGETING SKILLS FOR EXISTING TO SOLVE NEW TASKS
Pre-training data. The pre-training data comprises of all of the pick and place data from the bridge dataset (Ebert et al., 2021) from toykitchen 2. This includes data corresponding the task of putting the sushi in the transparent orange pot (Figure 11).
Target task and data. Since our goal in this scenario is re-target the skill for putting the sushi in the transparent orange pot to the task of putting the sushi in the metallic pot, we utilize dataset of 20 demonstrations that place the sushi in a metallic pot as our target task data that we fine-tune with (shown in Figure 11).
Quantitative evaluation protocol. For our quantitative evaluations in Table 1, we run 10 controlled evaluation rollouts that place the sushi and the metallic pot in different locations of the workspace. In runs the arm starts at up to 10 em distance above the target object. The initial object and arm poses and positions are matched as closely possible for different methods.
B.2 SCENARIO 2: GENERALIZING TO PREVIOUSLY UNSEEN DOMAINS
Pre-training data. The pre-training data in Scenario consists 800 door opening demonstrations on 12 different doors across 3 different toykitchen domains.
16

---

Under review as a conference paper at ICLR 2023
2. Fine-Tune
10 demoe each
Put knife in pot
Put cucumber in pot
Pre-Train on Bridge lasks, trajectories
Take croissant out of pol.
on Plate
Figure 13: Mustraaioo of pre-training data and fine-tuning data used for the tasks have added Scenario 3. The goal learn to solve new tasks in new domains starting from the same pre-trained initialization and when fine-tuning is only performed using 10-20 demonstrations of the target task.
Target task and data. The target task requires opening the door of an unseen microwave in toyk- itchen I using a target dataset of only 15 demonstrations.
Quantitative evaluation protocol. We run 20 rollouts with each method, counting successes when the robot opened the door by at least 45 degrees. To perform this successfully, there is degree of complexity as the robot has to initial open the door till it's open to about 30 degrees. Then due to physical constraints, the robot needs to wrap around the door and push it open from the inside. To begin an evaluation rollout, we reset the robot to randomly sampled poses obtained from held-out demonstrations the target door. This is compound task requiring the robot first grab the door by the handle, next move around the door, and finally push the door open. As before, we match the initial pose of the robot as closely as possible for all the methods.
B.3 SCENARIO 3: LEARNING TO SOLVE NEW TASKS IN NEW DOMAINS
Pre-training data. All pick-and-place data in the bridge dataset (Ebert et al., 2021) except any demonstration data collected in toykitchen 6, where our evaluations are performed.
Target task and data. The target task requires placing corn in a pot in the sink in the new target domain and the target dataset provides 10 demonstrations for this task. These target demonstrations are sampled from the bridge dataset itself.
Quantitative evaluation protocol. During evaluation we unable exactly match the camera orientation used to collect the target demonstration trajectories, and therefore ran evaluations with slightly modified camera view. This presents additional challenge for any method as it must now generalize to a modified camera view of the target toykitchen domain, without having ever observed this domain or this camera view during training. We sampled initial poses for our method by choosing transitions from a held out dataset of demonstrations of the target task and resetting the robot those initial pose for each method. We attempted to match the positions of objects across methods closely as possible.
B.4 MORE TASKS IN SCENARIO 3: LEARNING TO SOLVE MULTIPLE NEW TASKS IN NEW DOMAINS FROM THE SAME INITIALIZATION
In Appendix C, we have now added results for more tasks in Scenario 3. The details of these tasks
follows:
Pre-training data. All pick-and-place data from bridge dataset (Ebert et al., 2021) except data from toykitchen 6.
Target task and data. We consider four downstream tasks: take croissant from a metallic bowl, put sweet potato on plate, place knife in pot, and put eucumber in bowl, and collected 10 target demon-
17

---

Under review as a conference paper at ICLR 2023
strations for croissant, sweet potato and put cucumber in bowl tasks, and 20 target demonstrations for the knife in pot task. A picture of these target tasks shown in Figure 13.
Qualitative evaluation protocol. For our evaluations, utilize either 10 or 20 evaluation rollouts. As with all of our other quantitative results, we evaluate the baseline approaches and PTR starting from an identical the poses that appear in the first 10 timesteps of the held-out demonstration trajectories for this target task. For the configure inffiuuatiinn of objects, we test our policies in variety of task-specific configurations that we discuss below:
of initial poses for the robot. These initial poses randomly sampled from
Take croissant from metallic bowl: positions for the metallic bowl. In the "easy" positions, the metallic bowl placed roughly vertically beneath the robot's initial starting pose, whereas in the "hard" positions, the robot must first move itself to the right location the bowl and then execute the policy.
For this task, we alternate between two kinds of
Put cucumber in bowl: We 10 evaluation rollouts starting from 10 randomly sampled initial poses of the robot for our evaluations. Here we moved the bowl between the two stovetops in each trial.
Put sweet potato plate:
For this task, we performed 20 evaluation rollouts. We only sampled 10 initial poses for the robot, but for each position, we evaluated every policy on orientations of the sweet-potato (i.e., the sweet potato is placed on the table its flat face or on its curved face). Each of these orientations present some unique challenges, and evaluating both of them allows to gauge how robust the learned policy is changes in orientation. The demonstration data had variety of orientations for the sweet potato object that differed for each collected trajectory.
Place knife in pot: We evaluate this task over 10 evaluation rollouts, where the first five rollouts use a smaller knife, while the other five rollouts use a larger knife (shown in Fig- ure 7). Each knife seen in the demanstration dataset with equal probability.
We will discuss the results obtained on these new tasks in Appendix C.
C ADDITIONAL EXPERIMENTAL RESULTS
Croissant Task Multiple Viewpoint Experiment
Original Mi
Elessted Viowpoint
Rotated Viowoont
Sample observations from different camera viewpoints, only used during fine-tuning. Left: the original camera viewpoint found Figure 13. Middle: an elevated camera viewpoint where the robot and camera has been raised em. Right: rotated camera viewpoint where the kitchen has been slightly translated and rotated 15 degrees counterclockwise relative to the camera and robot.
Figure 14:
Even though Scenario already presents novel toy- kitchen domain and previously unseen objects during finetuning, we also evaluate PTR on a more challenging scenario where we additionally alter the camera viewpoint during finetuning. We apply two kinds of alterations to the camera: (a) we elevate the mount platform of the camera by 7 cm, which necessitates adapting the way the physical coordinates of the robot end-effector are interpreted by the policy, and (b) we rotate the camera by about 15 degrees to induce more oblique image observation than what was ever seen during pre-training. Note that both of these scenarios, the robot has never encountered such camera viewpoints during pre-training, which makes this scenario even more challenging. The original dataset (Ebert et al., 2021) had the camera elevated to the same position for eadch domain and always ensured the kitchen was parallel to the camera platform, with translations being the primary changes in scene for each domain. In Table 5, we present our results comparing PTR and BC (finetune). Observe that PTR still clearly outperforms BC (finetune),
Finetuning to novel camera viewpoints:
18

---

Under review as a conference paper at ICLR 2023
and attains performance close to that of PTR in Table 3, indicating that such shifts in the camera do not drastically hurt PTR.
Method
Elevated Viewpoint
Rotated Viewpoint
BC (finetune)
2/10
3/10
PTR (Ours)
6/10
7/10
Table 5: Comparison of PTR and BC (finetune), when evaluated on novel camera viewpoints with ele- vated and rotated cameras shown in Figure 14 for the croissant task. Observe that PTR still outperforms BC (finetune) this setting and attains more than 2x success rate of BC (finetune).
C.I EXPANDED DISCUSSION: WHY DOES PTR OUTPERFORM BC-BASED METHODS, EVEN WITH DEMONSTRATION DATA?
One natural question to ask given the results in this paper is: why does utilizing an offline RL method for pre-training and finetuning as in PTR outperform BC-based methods even though the dataset is quite "BC-friendly", consisting of only demonstrations? One might speculate that answer to this question is that our BC baseline can be tuned to be much better. However, note that our BC baseline is not suboptimally tuned. We utilize the procedure prescribed by prior work (Ebert al., 2021) for tuning BC as we discuss in Appendix D. In addition, the fact that BC (joint) does actually outperform CQL (joint) in many of our experiments, indicates that our BC baselines are well tuned. To explain the contrast to Ebert et al. (2021), note that the setup in this prior work utilized many more target task demonstrations (2 50 demonstrations from the target task) compared to our evaluations, which might explain why our BC- numbers are lower in an absolute sense. Therefore, the technical question still remains: why would we expect PTR to perform better than BC? We will attempt to answer this question using some empirical evidence and visualizations. Also, we will aim to provide intuition for why our approach PTR outperforms the baseline.
Qualitative Comparison of BC (finetune) and PTR
Taskc Croissant Metal Bowl
BC (Tinetune) Failure: graspe bowl instead of croissant when crosslant not underneath
PIR
Successi graaps croissant puts sink
Taok: Cucumber Bowl
BC (Tinetuns)
Fallure: exccutes impreaise grasp, and fails to locate accurately
PTR Cucumber
Figure 15: Qualitative successes of PTR visualized alongside failures of BC (finetune). As an example, observe that while PTR accurately able reach to the croissant and grasp it to solve the task, BC (finetune) imprecisc and grasps the bowl instead of the croissant resulting in failure.
To begin answering this question, it is instructive to visualize some failures for a BC-based method and qualitatively attempt understand why BC is worse than utilizing PTR. We visualize some evaluation rollouts for BC (finetune) and PTR as film strips in Figure 15. Specifically, we visualize evaluation rollouts that present challenging initial state. For example, for the rollout from the take crossiant out of metallic pot task, the robot must first accurately position itself over the crossiant before executing the grasping action. Similarly, for the rollout from the cucumber task, the robot must accurately locate the bowl and precisely try to grasp the cucumber. Observe in Figure 5 that BC (finetune) typically fails to accurately reach the objects of interest (croissant and the bowl) and executes the grasping action prematurely. On the other hand, PTR is more robust in these situations, and is able to accurately reach the object of interest before executes the grasping action or the releasing action. Why does this happen?
19

---

Under review as a conference paper at ICLR 2023
To understand why this happens, one mental model is appeal to the critical states argument from Kumar et al. (2022). Intuitively, this argument suggests that in tasks where the robot must precisely accomplish actions only few specific states (called "critical states") to succeed, but the actions at other states (called "non-critical states") do not matter as much. Thus, offline -tyle methods can outperform BC-based methods even with demonstration data. This is because learning a value function can enable the robot to reason about which states are more important than others, and the resulting policy optimization can "focus" on taking correct actions such critical states. Our real-world evaluation scenarios exhibit such structure. The majority of the actions that the robot must take to reach the object do not need to be precise as long as they generally move the robot in the right direction. However, in order to succeed, the robot must critically ensure to position the arm right above the object in correct orientation and position itself right above the container in which the object must be placed. These are the critical states and special care must be taken to execute the right action at these states. In such scenarios, the argument of Kumar et al. (2022) would suggest that offline RL should be better. We believe that we observe similar effect in our experiments: the learned BC policies are often not precise-enough at those critical states where taking the right action is critical to succeed.
Q Values Learnt After Finetuning
Before Offline Finetuning
Finetuned Chosen Checkpoint
Duger Datk Put Dasaster Road
Thalcory Tiraspo
Tojecsoy Treedpp
Figure 16: Evolution of Q-values on the target task over the process of fine-tuning with PTR. Observe that while the learned Q-values on held-out trajectories from the dataset just at the beginning of Phase 2 (finetuning) do not exhibit roughly increasing trend, the checkpoint of PTR we choose evaluate exhibits a generally increasing trend in the Q- Q-values despite having access to only 10 demonstrations for these target tasks.
As supporting evidence to the discussion above, we further visualize the Q-values over held-out trajectories from the target demonstration data that were never seen by PTR during fine-tuning in Figure 16. To demonstrate the contrast, we present the trend in Q-values before fine-tuning and for the checkpoint selected for evaluation after fine-tuning on the target task. Observe that the Q-values for the chosen checkpoint generally increase over the course of the trajectory indicating that the learned Q-function is able to fit well to the target data. Also, the learned Q-function generalizes to held-out trajectories despite the fact that only 10 demonstrations were provided during the fine- tuning phase. This evidence supports the claim that it is reasonable to expect the learned Q-function to be able to focus on the more critical decisions in the trajectory.
To further support our hypothesis that PTR outperforms BC-based methods because the learned value function enables us to learn about "critical" decisions, we run an experiment that essentially runs weighted version of BC during finetuning, where the weights are provided by exponentiated advantage values, where the advantages are defined as: As max.' Qo a') under Q-function learned by PTR. This approaches essentially matches BC fine- tuning in all aspects: the policy parameterization, the loss function (mean imean-squared error), and the details of training are kept identical to our BC baselines, with the exception of an additional weight given by exp(As (s, a)) on a given transition (s, a, r, s') observed in the set of limited -pecifcc demonstrations. We refer to this approach
Qe a)
Sadvantage-weighte. BC finetuning".
In contrast to our BC (finetune) results from Table 3, where PTR significantly outperformed BC (finetune), observe in Table 6, that divantage-weightee BC (finetune) performs comparably to PTR on the two tasks we studied for our analysis. This result is significant since it implics that all other factors kept identical, utilizing the weights given by the Q-function is the crucial factor in improving
20

---

Under review as a conference paper at ICLR 2023
the performance of BC and avoids the qualitative failure modes associated with BC methods shown in Figure 15.
Task
BC (finetune)
PTR (Ours)
Advantage-weighted BC (finetune)
Put cucumber in pot Take croissant from metal bowl
0/10
5/10
5/10
3/10
7/10
6/10
Table 6: Performance of advintage--eiighted BC on twa tasks from Table 3. Observe that weighting the BC objective using advantage-weights computed using the Q-function learned by PTR leads to much hetter performance than standard BC (finetune), and close PTR. This test indicates that the Q-function in PTR allows us to focus on more critical points, thereby preventing the failures discussed Figure
C.2 RESULTS IN SIMULATION
In this section, we present some additional results comparing BC-based methods and CQL to PTR in our simulated bin-sorting task. Recall that goal of this task was solve a two-stage. compound task with only five target demonstrations as discussed in Figure 8, Appendix A.2. Most importantly, the pre-training data does not show any instance of the robot attempting solve this two-stage task.
The performance numbers (along with confidenee intervals) are shown in Table 7. Observe that PTR improves upon prior methods in statistically significant manner, outperforming the BC baselines by a significant margin. This validates the efficacy of PTR in simulation, and corroborates our results.
Method
Success rate
BC (joint training)
7,00 + 0.00 %
COG (joint training)
8.00 + 1.00 %
BC (finetune)
4.88 + 4.07 %
PTR (Ours)
17.41 = 1.77 %
Table 7: Performance of PTR in comparison with other methods on the simulated bin sorting task, trained for many more gradient steps for methods until each one of them converges. Observe that PTR substantially outperforms other prior methods, including joint training on the same data with BC or CQL. Training on target data only is unable to recover a non-zere zero performance, and so we do not report it in this table. Since the 95%-confidence intervals do not overlap between PTR and other methods, it indicates that PTR improves upon baselines in a statistically significant manner.
D HYPERPARAMETERE FOR PTR AND BASELINE METHODS
In this section, we will present the hyperparameters we use in our experiments and explain how we tune the other hyperparameters for both our method PTR and the baselines we consider.
Since PTR utilizes CQL as the base offline RL method, it trains two Q-functions and sep- arate policy, and maintains a delayed copy of the two Q-functions, commonly referred as target Q-functions. We utilize completely independent networks to represent each of these five models (2 Q-functions, target Q-functions and the policy). We also do not share the convolutional encoders among them. As discussed in the main text, we rescaled the action space [-1, 1]1-1 match the one used by actor-critic algorithms, and utilized a tanh squashing function at the end of the policy. We used a CQL value of 10.0 for our pick and place experiments. The rest of the hyperparameters for training the Q-function, the target network updates and the policy are taken from the standard training for image-bases CQL from Singh et al. (2020), and are presented in Table below for completeness. The hyperparameters choose are essentially the network design decisions of (1) utilizing group normalization instead batch normalization, (2) utilizing learned spatial embed- dings instead of standard mean pooling, (3) passing in actions each of the fully connected layers of the nentwwork and the hyperparameter a in CQL that must be adjusted since our data consists of demonstrations. We will ablate the new design decisions explicitly in Appendix E.
PTR.
The only other hyperparameter used by PTR the mixing ratio T that determines the proportion of samples drawn from the pre-training dataset and the target dataset during the offline finetuning
21

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
O-function learning rate
3e-4
Policy learning rate
le-4
Target update rate
0.005 (soft update with Polyak averaging)
Optimizer type
Adam
Discount factor y
0.96 (since trajectories have a length of only about 30-40)
Use terminals
True
Reward shift and scale
shift = -1, scale 10.0
CQL
10.0
Use Color Jitter
True
Use Random Cropping
True
Table 8: Main hyperparameters for CQL training in our real-world experiments. In simulation, we utilize a smaller for CQL, 1.0 and larger discount 0.98 since trajectories in simulation are about 60-70 timesteps in length.
phase in PTR. We utilize 0.7 for our experiments with PTR in the main paper, and for the additional experiments we added in the Appendix. This is because т = 0.9 (more bridge data, and smaller amount of target data) was helpful in scenarios with very limited target data.
0.9
In order to perform checkpoint selection for PTR, we utilized the trends in the learned Q-values over of held-out trajectories the target data as diseussed in Section 4.2. We did not tune any other algorithmic hyperparameters for CQL., as these were taken directly from (Singh et al., 2020).
BC (finetune). We trained BC in similar manner as Ebert et al. (2021), utilizing the design decisions that this prior work found optimal for their experiments. The policy for BC utilizes the very same ResNet 34 backbone our RL policy since backbone based on ResNet 34 found to be quite effective in Ebert et al. (2021). Following the recommendations of Ebert et al. (2021) and based on result trends from our own preliminary experiments, we chose to not utilize the tanh squashing function at the end of the policy for any BC-based method, but trained a deterministic BC policy that was trained to regress to the action in the demonstration with mean-squared error (MSE) objective.
Hyperparameter
Value
Policy learning rate
le-4
Optimizer type
Adam
Use Color Jitter
True
Use Random Cropping
True
Dropout
0.4
Table 9: Main hyperparameters for Behavior Cloning Baseline Training in our real-world and simula- tion experiments. Note: architecture design choices follow closely to PTR design choices.
In order to perform vross-validation, checkpoint and model selection for our BC policies, we follow guidelines from prior work (Ebert al., 2021; Emmons et al., 2021) and track the MSE on held- out validation dataset similar to standard supervised learning. We found that ResNet 34 BC policy attained the smallest validation MSEs in general, and for our evaluations, we utilized checkpoint of ResNet 34 BC policy that attained the smallest MSE.
Analogous to the case of PTR discussed above, we also ablated the performance of BC for set of varying values of the mixing ratio but found that large value of - = 0.9 was the most effective for BC, and hence utilized 5 = 0.9 for BC (finetune) and BC (joint).
BC (joint) and CQL (joint). The primary distinction between training BC (joint) and BC (fine- tune) and correspondingly. CQL (joint) and PTR was that in the case of joint training, the target dataset was introduced right at the beginning of Phase I ppreniraining phase), and mixed the target data with the pre-training data using the same value of the mixing ratio 7 used in for our fine-tuning experiments to ensure a fair comparison.
Few-shot offline meta-RL (MACAW) (Mitchell et al., 2020): We compare to two variants this algorithm and perform extensive sweep over several hyperparameters, shown in Table 10.
22

---

Under review as a conference paper at ICLR 2023
We trained two different variants of MACAW in our evaluation: (1) Pre-training the bridge data in Scenario 3 and then fine-tuning on target data of interest, and (2) adapting a set of existing task identifiers to the target task of interest utiliizing the same pre-training and fine-tuning domains. We performed early stopping on the meta-training based on validation losses. From there, we started the meta-testing phase, adapting to the target domain of interest. Following Mitchell et al. (2020), use a task mini-batch of 8 tasks at each step of optimization rather than using all of the training tasks. We clipped the advantage weight logits the scale of 20 and attempted utilize policy network with fixed and learned standard deviation. Additionally, we varied the number of Adaptation steps following prior work. Our evaluation protocol for MACAW entails utilizing the validation losses to choose an initial checkpoint for evaluation. Then, we consider checkpoints in the neighborhood (d: 50K gradient steps) to for evaluations as well and chose the max over all of these checkpoints as the final evaluation success rate.
Quantitatively, as seen in Table 3, MACAW was unable to get a non-zero success rates on any of the tasks we study. However, we did qualitatively observe nontrivial behavior seen in our evaluation rollouts. For instance, we found that the policies trained via MACAW could consistently grasp the object of interest but were unable to localize where to place the object correctly. Several trials involved hovering around with the object of interest and not placing the object in the container. Other trials involved the agent failing grasp the object.
Hy Hyperparameter
Value
Optimizer
Adam
Outer Policy learning rate
le-4
Outer Value learning rate
le-5, le-6
Inner Policy learning rate
le-2, le-3
Inner Value learning rate
le-3, le-4
Auxilary Advantage Coefficient
le-2, 1e-3, le-4 Fixed std, Learned std
Policy Parameterization
AWR Policy Temperature
1, 10, 20
Number of Adaptation Steps
1, 2, 3
Task Batch Size
Train Adaptation Batch Size
64
Eval Adaptation Batch Size
64
Max Advantage Clip
20
Use Color Jitter
True
Use Random Cropping
True
Table 10: Main hyperparameters for Training MACAW (Mitchell et al., 2020) in our real-world exper- iments. Note: architecture design choices follow closely to PTR design choices but hyperparameter design choices follow closely the suggestions in Mitchell et al. (2020).
Pre-trained R3M initialization (Nair et al., 2022); Next we compare PTR to utilizing an off-the- shelf pre-trained representation given by R3M (Nair et al., 2022). We compare two baselines that attempt to train a MIP policy on top of the R3M state representation by using BC (finetuning) and CQL (linetuning) respectively. To ensure that this baseline is well-tuned, we tried variety of network sizes with 2. 3 or 4 MLP layers and also tuned the hidden dimension sizes in [256, 512, 1024]. We also utilized dropout as regularization to prevent overfitting and tuned variety of values of dropout probability in [0. 0.1, 0.2, 0.4, 0.6, 0.8]. We observe in Table 3, that on the four tasks we evaluate on, PTR outperforms R3M, which indicates that training on the bridge dataset can indeed give rise to effective visual representations that are more suited to finetuning in our setting. The numbers we report in the table are the best over each parametric policy corresponding to each hyperparameter in abalation. Checkpoint selection done utilizing early stopping which is the last iteration where the validation error stops decreasing. Learning curves for this bascline can be found in our Anonymous Website.
Pre-trained MAE initialization (He et al., 2021): We took a similar training procedure to R3M for our MAE representation. We used an MAE trained on the every image from the bridge dataset Ebert et al. (2021). We then finetuned on a specific target task with a similar ablation on network size, hidden dimension size and regularization techniques such dropout. We observe in Table ??
23

---

Under review as a conference paper at ICLR 2023
that on the four tasks evaluate on, PTR outperforms R3M, which indicates that training on the bridge dataset can indeed give rise to effective visual representations that are more suited to finetuning in setting. The numbers we report in the table are the best over each parametric policy corresponding to each hyperparameter in our abalation. Checkpoint selection was done utilizing early stopping which is the last iteration where the validation error stops decreasing. Learning curves for this baseline can be found in our Anonymous Website.
Policy expressiveness study. We considered two policy expressiveness choices for BC to compare with our reference BC implementation that is implementated with a set of MLP layers. These first of the two choices was an autoregressive policy where the 7 dimensional action space was discretized into 100 bins. Each action was then predicted autoregressively conditioned on the observation, task id and the action component from the previous dimension(s). The sencond approach was with the BeT Architecture from Shafiullah et al. (2022). We utilized the reference implementation from the paper with the default suggested hyperparameters for this set of abalations. The window size for the mingpt transformer was abalated over between 1, 2, and 10.
E VALIDATING THE DESIGN CHOICES FROM SECTION 4.2 VIA ABLATIONS
In this section, we will present ablation studies aimed to validate the design choices utilized by PTR. We found these design choices quite crucial for attaining good performance. The concrete research questions we wish to answer are: (1) How important is utilizing a large network for attaining good performance with PTR, and how does the performance of PTR scale with the size of the Q-function?, (2) How effective is learned spatial embedding compared to other approaches for aggregating spatial information?
(3) Is concatenating actions at each fully-connected layer of the Q-function crucial for good performance?, (4) Is group normalization good alternative to batch normalization? and (5) How does our choice of creating binary rewards for training affect the performance of PTR?. We will answer these questions next.
Scaling Trend for PTR
Highly expressive Q-networks are essential for good performance. To assess the importance of highly expres- sive Q-functions, we evaluate the performance of PTR with varying sizes and architectures on three tasks: the open door task from Scenario 2, and the put cucumber in pot and take croissant out of metallic bowl tasks from
Open Doer
Pidu-iaaca (Aneraal
heerage Suncesn Ratt
0.525
3.35
Scenario 3. Our choice of architectures is as follows: (a) a standard three-layer convolutional network typically used by prior work for DM-control tasks (see for exam- ple, Kostrikov et al. (2021a)). (b) an IMPALA (Espeholt et al., 2018) ResNet that consists of 15 convolutional lay- ers spread across a stack of 3 residual blocks, (c) ResNet 18 with group normalization and learned spatial embed- dings, (d) ResNet 34 that we use in our experiments, and (e) even bigger ResNet 50 with group normalization and learned spatial embeddings.
0.175
Reshie RosNNs50
Networks
Figure 17: Scaling trends for PTR the open door task from Scenario 2, and average over two pick and place tasks (lake crois- sant out of metallic pot and put cucumber in bowl) from Scenario 3. Note that more high capacity and expressive function approxima- tors lead to the best results.
We present our results in Figure 6. To obtain more accu- rate scaling trends, we plot the trend in the average success rates for the pick and place tasks from Scenario 3 along with the trend in the success rate for the open door task separately since these tasks use different pre-training datasets. Observe that the performance of smaller networks (Small, IM- PALA) sigmificantly worse than the ResNet in the door opening task. For the pick and place tasks that contain a much larger dataset, Small, IMPALA and ResNet18 all perform much worse than ResNet 34 and ResNet 50. We believe this result is quite exciting since highlights the possiblity of actually benefitting from using hxphsy-expressive neural network models with TD-lcarning based RL methods trained on lots of diverse multi-task data (contrary to prior work (Lee et al., 2022b)). We believe that this result is valuable starting point for further scaling and innovation.
Learned spatial embeddings are crucial for performance. Next we study the impact of utilizing the learned spatial embeddings for encoding spatial information when converting the feature maps from the convolutional stack into a vector that is fed into the fully-connected part of the Q-function. We compare our choice utilizing spatial softmax as in Ebert et al. (2021), and also global average
24

---

Under review as a conference paper at ICLR 2023
pooling (GAP) that simply averages over the spatial information, typically utilized in supervised learning with ResNets.
Method
Success rate
PTR with spatial softmax
4/10
PTR with global average pooling
4/10
PTR with learned spatial embeddings (Ours)
7/10
Table 11: Ablation PTR with spatial softmax and GAP on the croissant task. Observe that PTR with learned spatial embeddings performs significantly better than using a spatial softmax or global average pooling.
As shown in Table 11 learned spatial embeddings outperform both these prior approaches on the put croissant in pot task. We suspect that spatial softmax does not perform much better than the GAP approach since the softmax operation can easily get saturated when running gradient descent to fit value targets that are not centered in some range, which would effectively hinder its expressivity. This indicates that the approach of retaining spatial information like in PTR is required for attaining good performance.
Concatenating actions at each layer is crucial for performance. Next, we run PTR without passing in actions at each fully connected layer of the Q-function on the take croissant out of metallic bowl task and only directly concatenate the actions with the output of the convolutional layers before passing it into the fully-connected component of the network. On the croissant task, we find that not passing in actions at each layer only succeeds in 2/10 evaluation rollouts, which is significantly worse than the default PTR which passes in actions at each layer and succeeds in 7/10 evaluation rollouts (Table 12).
Method
Success rate
PTR without actions passed in at each FC layer PTR with actions passed in at each FC layer (Ours)
2/10
7/10
Table 12: Ablation of PTR with actions passed in at each layer. Observe that passing in actions at cach fully-connected layer does lead to quite good performance.
Group normalization is more consistent than batch normalization. Next, ablate the usage of group normalization over batch normalization in the ResNet 34 Q-functions that PTR uses. We found that batch normalization was generally harder train attain Q-function plots that exhibit roughly increasing trend over the course of a trajectory. That said, on some tasks such as the croissant in pot task, we did get reasonable Q-function, and found that batch normalization can perform well. On the other hand, on the put cucumber in pot task, we found that batch normalization was really ineffective. These results are shown in Table 13, and they demonstrate that batch normalization may not be consistent and reliable with PTR as group normalization.
Method
Croissant out of metallic bowl Cucumber in pot
PTR with batch norm. (relative)
+ 28.0% (7/10- - 9/10) 60.0% (5/10 - 2/10)
Table 13: Relative performance of PTR with batch normalization with respect to PTR with group malization. Observe that while utilizing batch normalization in PTR can be sometimes more effective than using group normalization (c.g., take croissant out of metallic bowl task), it may also be highly ineffective and can reduce success rates significantly in other tasks. The performance numbers to the left of the + corresponds to the performance of PTR with group normalization and the performance to the right of + is the performance with batch normalization.
Choice of reward function. Finally, we present some results that ablate the choice of the reward function utilized for training PTR from data that entirely consists of demonstrations. In our main set of experiments, we labeled the last three timesteps of every trajectory with a reward of +1 and annotated all other timesteps with 0 reward. We tried perhaps the most natural choice of labeling only the last timestep with a 0 reward on the croissant task, and found that this choice succeeds 0/10 times, compared to annotating the last three timesteps with +1 reward which succeeds 7/10 times.
25

---

Under review as a conference paper at ICLR 2023
We suspect that this is because only annotating the last timestep with +1 reward is not ideal for two reasons: first, the task is often completed in the dataset much earlier than the observation shows the task complete, and hence the last-step annotation procedure induces non non-Markovian reward function, and second, only labeling the last step with a +1 leads to overly conservative Q-functions when used with PTR, which may not lead good policies.
F MORE DETAILS AND EXAMPLES OF OUR EARLY STOPPING CRITERION
Our guideline for selecting checkpoints for evaluation was to pick the checkpoints that presented Q-function with the Q-values appearing closest to having a monotonically increasing trend in trajectory. This is a relative guideline and must be performed within the checkpoints observed in run, just like how the absolute value of validation error may not be as informative in supervised learning, as a comparison of its value across a few models. The reason for this heuristic choice is that a valid Q-function, in the end, must be valid estimator for long-term discounted return and hence, must increase over time-steps of demonstration trajectories for given task. Of course, this heuristic does not hold for arbitrary sub-optimal offline data, but all of data comes from human- collected demonstrations. In principle, this heuristic can be wrapped into a metric quantifying degree of monotonicity, but in our experiments we felt this was not necessary as we will show below, we were able to narrow down the checkpoints to essentially one or at most, two checkpoints by just visual inspection. Of course, designing an accurate metric would be helpful for future work.
We present two worked-out examples of our checkpoint selection strategy for two tasks from Sce- nario and Scenario below. Observe that checkpoints early in training exhibit Q-values that fluctuate arbitrarily in the beginning of training, that are clearly non-monotonic. This is because the the lack of sufficient gradient steps for fine-tuning on the target task. Once sufficient gradient steps are performed, the Q-values visibly improve the monotonicity property. Training further leads to much flatter Q-values, which start becoming more noisy and are visibly less monotonic.
Gradient Steps
D Gradient Stops
Undertrained Chreoppriins
Undertrained Checkpoints
50K Gradiont Stops
50K Gradient Stens
Gradient Steps
100K Gradiint Staps
Ressonaable Checkpoinnss
Reasonatle Checkpoint
150K Gradient Stans
150K Gradient Stops
Overtralned Checkpaints
200K Gradiett Steps
Overtrained Checkpaint
200K Gradient Staps
Figure 18: Evolution of Q-values on the target task over the process of fine- with PTR. Observe that while the learned Q-values held-out trajectories from the dataset just the heginning of Phase (finetuning) do not exhibit a roughly increasing trend, the checkpoint of PTR we choose to evaluate exhibits visible more increasing trend in the Q valuus despite having access to only 10 demonstrations for these target tasks.
26


Reviews:
[{"review_id": "sJkTk6i37-_", "reviewer": "Reviewer_D87g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper experimented with using offline RL to pretrain and fine-tune policy and Q function. The results show that using offline RL to first pretrain the policy and Q function on a diverse offline dataset can improve the policy adaptation speed on new tasks. The paper discussed about several important design choices, such as network architectures to achieve better performance. Overall, the paper is more like an engineered work that combines some ideas from prior works and achieves fast policy learning on new tasks.", "summary_of_the_review": "Overall I think the paper presents a decent amount of experiments and compare its proposed pipeline with some reasonable baselines. However, the paper writing needs to be improved to make the descriptions accurate.", "strength_and_weaknesses": "**Strength**:\n\n* The paper presents a few techniques, such as network architecture design and normalization trick, that are empirically shown to improve downstream task learning.\n\n* The paper shows real-world robot experiments. This is a big plus as such experiments do require a significant amount of effort.\n\n * The paper compares the proposed pipeline with a decent number of baselines, including behavior cloning and especially prior works on first doing representation learning and then doing policy learning such as R3M.\n\n**Weaknesses**:\n\n* As the paper also says itself, the techniques presented in the paper are not new.\n\n* Since the number of real-world tests is small (like 20?), I am not sure how one can report success rates with so many significant figures. Similarly, the paper tends to overclaim the method's effectiveness. For example, in page 5, the paper says that using \"learned spatial embeddings\" leads to `2x` improvement. However, if we actually look at table 10, we see that one case has 4/10 success rate and the other one has 7/10 success rate. With such a small number of tests, it is very confusing and not rigorous at all to make such claims (`2x` improvements). There are other similar claims in the paper. The authors should remove such claims.\n\n* In table 2, I cannot find the 95% confidence interval.\n\n* It's unclear how convincing the claim about generalizing to previously unseen domain is. From the video, I can only tell that the testing door is not that different from the training doors. Can authors add a figure showing all the training doors and the testing door? Since the claim is about generalization, it is important for readers to understand how much difference the training and testing distribution has. As a research paper, it is important to make each claim precise and accurate. While I enjoy reading the papers, especially seeing the real-world testings, reading such exaggerated claims only negatively affects my judgment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j0zFGvO1FwS", "reviewer": "Reviewer_tq7f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets to solve real-world robotic manipulation tasks by leveraging a large demonstration dataset and a small amount of target task demonstrations.\nThe proposed method first pre-train a policy on the large dataset with multi-task offline RL, then fine-tune on the small demonstration data again with same objective.\nDuring the fine-tuning phase, a small amount of data is sampled from the dataset used for pre-training to prevent overfitting.\nThe fine-tuning process is early stopped by heuristic validation rule.\nIn experimental results, several challenging robotic manipulation tasks are solved by the proposed method using proper design choices of network architecture and network size.", "summary_of_the_review": "Although this paper provides interesting results on challenging manipulation tasks leveraging offline data,\nthe relationship between the proposed method and existing methods is not enough addressed.\nMoreover, the proposed design choices are incremental but evaluated on a single domain.\nBecause the generality of the proposed method is not proven, it is difficult to be introduced to the community as a generic learning method for such a problem setup.\nThus, I vote to reject this paper.", "strength_and_weaknesses": "### Strength\n* Provided results on robotic manipulation tasks can provide empirical knowledge on handling robot control tasks using offline dataset.\n\n### Weakness\n* Novelty\n    - The problem setup of this paper is very relevant to fully offline meta-RL and the paper is stating \"our approach is much more data-efficient and simple\". However, any comparison or justification is not given in the paper, thereby misleading readers to be impressed as the problem setup is novel.\n    - Given architectural choices are interesting and reasonable, but neither surprising nor significant. Moreover, in order to introduce the network design as a technical novel of this paper, the architecture should be applied and evaluated on more diverse domains.\n    - The early stopping criteria is bizarre. The Q-value plot of passed example in Figure 3 is not monotonically increasing over the time. If there is an exact rule deciding whether it is \"almost\" monotonically increasing it is not described in the main text. If a human decide this, any of given results is reliable as there is no human-related study for this procedure.\n\n* Comparison\n    - As mentioned before, the comparison with offline meta-RL is missing. Also, any naive application of imitation learning method other than behavior cloning also deserves to be a baseline.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "A_-wQqDQzKZ", "reviewer": "Reviewer_WMAS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework pre-training for robots (PTR) based on CQL that attempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as few as 10 demonstrations.\nIn particular, their method modifies CQL to include several crucial design choices that enable PTR to have strong generalization. These choices include using task indicators for policy and Q-function networks, using group normalization, and using mixed data of prior data and the target dataset.\nIn their experiments, they train the model on the Bridge dataset and finetune it with real-world demonstrations. Their method shows signiﬁcant improvement over prior RL-based pretraining and ﬁnetuning methods.\n", "summary_of_the_review": "I acknowledge the empirical contribution of this paper to the generalization problem of offline RL. However, without a fair comparison with state-of-the-art meta-RL algorithms, I suggest rejecting this paper for now.", "strength_and_weaknesses": "### **Strength**\n\n* Generalization of offline reinforcement learning algorithms is an interesting topic. Although there is no new algorithm or framework in this paper, the exploration of useful combinations of existing technics still contributes to this area.\n\n* The presentation of this paper is good. All details are clearly explained. \n\n\n### **Weaknesses**\n\n* Missing baselines of offline meta-reinforcement learning [1][2]. Without comparing with existing methods, it is hard to position the empirical value of this paper.\n\n* Sort of limited novelty. As the authors mentioned, “Although the individual components that constitute PTR are not especially innovative and are based closely on prior work, the combination of these components is novel.”Since there are already some existing novel meta-RL methods, properly comparing them could empirically improve the novelty of this paper.\n\n---\n\n[1] Mitchell, Eric, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. \"Offline meta-reinforcement learning with advantage weighting.\" In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.\n\n[2] Xu, Mengdi, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. \"Prompting decision transformer for few-shot policy generalization.\" In International Conference on Machine Learning, pp. 24631-24645. PMLR, 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3es6DGFqpRc", "reviewer": "Reviewer_mM3r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of letting robots learn to solve novel target tasks by leveraging diverse offline datasets in combination with only small amounts of task-specific data. To this end, the paper proposes a framework that uses multi-task offline reinforcement learning approaches for pre-training and then fine-tuning its policy on target tasks. The experimental results show that the proposed framework outperforms behavior cloning and offline reinforcement learning baselines including CQL and COG. I am leaning toward accepting this paper since it studies a promising research direction and presents a reasonable framework to address the problem with supporting experimental results.", "summary_of_the_review": "I am leaning toward accepting this paper since it studies a promising research direction and presents a reasonable framework to address the problem with supporting experimental results.", "strength_and_weaknesses": "## Paper strengths and contributions\n\n**Novelty**\nIn my opinion, the idea of bridging offline reinforcement learning and behavior cloning to solve novel target tasks in a new domain with only a few demonstrations is intuitive and convincing. This paper presents an effective way to implement this idea.\n\n**Clarity**\nThe overall writing is clear. The authors utilize figures well to illustrate the ideas.\n\n**Related work**\nThe authors give a clear description of the related prior works from both the perspectives of offline reinforcement learning and behavior cloning.\n\n**Experimental Results**\nThe experimental results show that the proposed framework outperforms behavior cloning and offline reinforcement learning baselines including CQL and COG.\n\n## Paper weaknesses and questions\n\n**Experimental results**\n- It is mentioned in scenario two that COG does not outperform BC (joint), which is pretty interesting. Unfortunately, COG was only compared to BC but not PTR with this setup. It would be more informative to give a brief explanation or an intuition of why COG was at a disadvantage in that particular scenario.\n- It is not entirely intuitive to me why the proposed framework outperforms BC baselines. It would be helpful if the authors give more intuitions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject