TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
Authors: Zuxin Liu
Jesse Zhang
Kavosh Asadi
Yao Liu
Ding Zhao
Shoham Sabach
Rasool Fakoor
Venue: ICLR 2024 poster (2024)
Keywords: Efficient Adaptation
Continual Learning
Robot Learning
Language-Conditioned Visuomotor Control
Few-shot Adaptation
Large Pretrained Models
Imitation Learning
Transfer Learning
Primary area: reinforcement learning
TL;DR: We propose a method of efficient adaptation for large pretrained decision-making models. Low-Rank Adaptation with our framework outperforms traditional fine-tuning with only 1% new parameters while avoiding issues like catastrophic forgetting.

Abstract: The full potential of large pretrained models remains largely untapped in control domains like robotics. This is mainly because of the scarcity of data and the computational challenges associated with training or fine-tuning these large models for such applications. Prior work mainly emphasizes either effective \emph{pretraining} of large models for decision-making or single-task adaptation. But real-world problems will require data-efficient, \emph{continual adaptation} for new control tasks. Recognizing these constraints, we introduce TAIL (Task-specific Adapters for Imitation Learning), a framework for efficient adaptation to new control tasks. Inspired by recent advancements in parameter-efficient fine-tuning in language domains, we explore efficient fine-tuning techniques---e.g., Bottleneck Adapters, P-Tuning, and Low-Rank Adaptation (LoRA)---in TAIL to adapt large pretrained models for new tasks with limited demonstration data. Our extensive experiments comparing prevalent parameter-efficient fine-tuning techniques and adaptation baselines suggest that TAIL with LoRA can achieve the best post-adaptation performance with only 1\% of the trainable parameters of full fine-tuning, while avoiding catastrophic forgetting and preserving adaptation plasticity in continual learning settings.

Published as a conference paper at ICLR 2024
TAIL: TASK- ADAPTERS FOR IMITATION LEARNING WITH LARGE PRETRAINED MODELS
Zuxin Liu', Jesse Zhang', Kavosh Asadi', Yao Liu Ding Zhao', Shoham Sabach', Rasool Fakoor? Carnegie Mellon University, 2 University of Southern California, 3 Amazon Web Services
ABSTRACT
The full potential of large pretrained models remains largely untapped in control domains like robotics. This is mainly due to data scarcity and computational chal- lenges associated with training or fine-tuning large models for such applications. Prior work mainly emphasizes either effective pretraining of large models for decision-making or single-task adaptation. But real-world problems will require data-efficient, continual adaptation for new control tasks. Recognizing these con- straints, introduce TAIL (Task-specific Adapters for Imitation Learning), framework for efficient adaptation to a stream of new control tasks. Inspired by recent advancements in parameter-efficient fine-tuning language domains, we explore efficient fine-tuning techniques e.g., Bottleneck Adapters, P-Tuning, and Low-Rank Adaptation (LoRA)-in) in TAIL to adapt large pretrained models for new tasks with limited demonstration data. Our extensive experiments comparing prevalent parameter-efficien
mun-tuning techniques and adaptation baselines sug- gest that TAIL with LoRA can achieve the best post-adaptation performance with only 1% of the trainable parameters of full fine-tuning while avoiding catastrophic forgetting and preserving adaptation plasticity in continual learning settings.
INTRODUCTION
A desired property of an autonomous agent is the ability to adapt efficiently to novel tasks. In vision and language domains, large pretrained models have demonstrated adaptation new tasks with just few examples through prior knowledge obtained from internet-scale datasets (Brown al., 2020; Radford et al., 2021; Touvron et al., 2023). Similar methods have also been applied in decisiinnmaking and control applications (Brohan et al., 2022; Driess et al., 2023; Brohan et al., 2023). However, new control tasks are more difficult to adapt to than the aforementioned vision and language domains due to (1) the lack of internet-scale control data and (2) how optimal actions can vary significantly from tosk-to-task, even under shared observation spaces. As such, these large-scale decision-making models still rely a close alignment between training and testing tasks.
In contrast, agents deployed challenging environments need to adapt to major task variations takee for example, general household robot. Equipped with factory-pretraince policy, the robot will be employed in unique ways by every household. Thus, the robot will need to continually adapt in order best serve each one, e.g., by fine-tuning its capabilities et al., 2021; Lu et al., 2021; Kalashnkov et al., 2021; : Chen et al., 202 Yao et al., 2024). Because most prior decision-making papers adapt new tasks by fine-tuning the entire model (Gupta et al., 2022; Bousmalis et al.. 2023; Zhang et al., 2023a;b; Collaboration et al., 2023; Liu et al., 2023c), mastering each new skill requires great computational cost and often leads catastrophic forgetting of old ones. An alternative approach would be to store a separate policy per new task, which leads to unreasonable storage requirements. Some prior work investigates efficient adaptation of large models to single task suite (Liang et al., 2022; Schmied et al., 2023; Sharma et al., 2023), but this realistic continual learning setting brings out additional problems to consider, warranting further investigation. What would be the best way for agents efficiently adapt to stream of novel tasks without having to trade off computation, storage, and performance on older tasks?
few demonstrations (Chebotar
To answer this question, we propose Task-specific Adapters for Imitation Learning, shown in Fig. 1. framework for efficient adaptation to new control tasks. Through TAIL we (1) effectively incorporate lightweight adapter modules into pretrained decision-makinn models and (2) comprehensively

---

Published as a conference paper at ICLR 2024
pare efficient adaptation techniques implemented in TAIL in a continual imitation learning setting- Notably, we examine parameter-efficient adaptation techniques (PEFT) used for large language models; we explore the potential of adapters (Houlsby et al., 2019), prefix tuning (Li & Liang, 2021), and low-rank adaptation (LoRA) (Hu et al., 2021) in fostering efficient and continual adaptation in large pretrained decision-making models. These works stand out they introduce small number of new parameters which help: avoid catastrophic forgetting, maintain training plasticity for continual learning, avoid overfitting with limited adaptation data, and reduce computational and memory burden. Investigating these works in control tasks for a realistic continual learning setup specifically important because, unlike in language domains, test task losses are often not proportional to test task performance (Ross et al., 2011; Ke et al., 2020) effiieefficient adaptation insights from language models may not transfer to decision-making ones. Thus, independent investigation of these adaptation techniques for decision-making is crucial for deploying continually adapting agents in the real world.
We compare PEFT techniques implemented in TAIL against commonly used adaptation methods in the imitation learning literature. In our experiments, we discover that TAIL with LoRA leads to the best post-adaptation performance as it preserves the original pretrained representations while being resilient against overfitting in the limited-data regime. These capabilities especially important for agents operating in new, challenging environments, such as the aforementioned household robots. Our analysis also reveals important insights into the strengths and limitations of each adaptation strategy. Instead of performing full fine-tuning of the entire model, only introduces a small number of additional parameters without making changes the original model. These additional parameters make up a mere 1.17% of the size of the original model. Importantly, this results in approximately 23% less GPU memory consumption achieve 22% higher forward adaptation success rate than full the-ttning while avoiding catastrophic forgetting. Notably, these results are contrary to many results from the vision and language model literature which show that full fine-tuning works better (He et al., 2022; Mao et al., 2022; Chen et al., 2022; Schmied et al., 2023).
In summary, this work bridges a crucial gap in research into efficient and continual adaptation for pretrained decision models by introducing a framework for continual imitation learning, TAIL, and thoroughly analyzing the effects of different efficient adaptation methods. Comprehensive experiments demonstrate that TAIL outperforms standard continual learning and prior single-task adaptation baselines.
RELATED WORK
Pretrained Models for Control. Researchers have long studied the use of pretrained models for better downstream transfer to related tasks (Bozinovski & Fulgosi, 1976; Schmidhuber, 1992; Dietterich et al., 1997). Recent works have examined using the representations learned by pretrained visual models for control (Shridhar et al., 2022; Nair et al., 2022; Ma et al., 2022; 2023; Majumdar et al., 2023a). These methods leverage representations acquired from large task-agnostic datasets, such as Ego4D (Grauman et al., 2022), or through self-supervised objectives. However, there's evidence that simply utilizing these pretrained features may not be as useful for downstream task performance (Hansen et al., 2022). Meanwhile, another recent line of work directly trains large pretrained models for control (Brohan et al., 2022; Reed et al., 2022; Driess et al., 2023; Jiang et al., 2023; Brohan al., 2023; Bousmalis al., 2023). These methods either do not attempt adaptation to new tasks, or perform expensive full-finetuning for adaptation. In contrast, our method, TAIL., is a framework for efficient adaptation of fecision-making models, like the aforementioned large pretrained control models, and investigates ways to adapt such models efficiently to multiple new tasks.
Parameter- Efficient Fine-Tuning (PEFT). PEFT has gained traction as way adapt pretrained models without significantly increasing parameters. Rebuffi et al. (2018) 8) demonstrated that residual adapters for smaller, CNN-based vision models are effective in non-control supervised learning set- tings. More recently, counforrer-focused techniques such as transformer adapter modules (Houlsby et al., 2019), LoRA (Hu et al., 2021), and prompt tuning (Li & Liang, 2021) incorporate lightweight modules or prompts optimized for downstream tasks, all while preserving the original model weights. PEFT offers several advantages over full fine-tuning: it's faster, less susceptible to overfitting, retains prior capabilities, and facilitates efficient task-switching While PEFT has been successful in both language and vision domains (Chen et al., 2022; Schmied et al., 2023), its continuous adaptation for large decision-male making models is not yet thoroughly examined. Liang et al. (2022); Sharma et al. (2023), Xu et al. (2022), and Xu et al. (2023) propose the use of adapters, prompt-tuning and hyper-network in robotics settings, but they do not examine other PEFT methods and focus on
2

---

Published as a conference paper at ICLR 2024
anteer
lask head
Task head
Input Fusion
Temporal Decoder
Weights
Pretrained Weights
Hamb Frozen Pretrained Features
Task Embedeing
Token Sequence
Full Fine-Tuning
Instruction Encoder
Spatial Encoder
Spatial Encoder
Spatial
inters
teskceedd
Frocon
Pretrained Weights
Adapter
Quarile
Wriggns
patitohuu Is
description
Observations
Task-specific Adapters (TAIL)
(a)
Figure I: (a): The multi-modal, transformer policy architecture we utilize for pretraining. We encode language task descriptions with pretrained CLIP instruction encoder and image observations with pretrained CLIP spatial encoder. We additionally encode state observations (not pictured) which, along with the observation embeddings, embedded into sequence of tokens used by the temporal decoder transformer to predict single-step action distributions. We include an input fusion module to explicitly combine task embedding with the observation token sequence for better instraction-follwwinii ability. (b): The three types of fine-tuning paradigms we test, with TAIL at the bottom right. For further architecture details, see Appendix Sec. A.
adaptation to a single task suite. We instead examine the performance of various state-of-the-e PEFT techniques implemented with TAIL in the continual learning scenario.
Continual Learning. Continual learning in control (Thrun & Mitchell, 1995; McCloskey Cohen, 1989; Fu et al., 2022) is long-studied' problem with applications to many real-world situations. In general, agents should be able to transfer knowledge (e.g., by continually fine-tuning) or experience training data) from previously learned tasks new tasks (Lopez-Paz & Ranzato, 2017; Traoré et al, 2019; Fakoor et al., 2020; Caccia et al., 2023). However, with large pretrained models trained on large datasets, fine-tuning the entire model is computationally costly yet risks catastrophic forgetting. and transferring training data from other tasks is too memory inefficient in the face of a large stream of new tasks. Therefore. we present study into efficient fine-tuning techniques which, when integrated with TAIL, can help inform future research of continual learning.
PRELIMINARIES
In this section, introduce problem setting (Sec. 3.1), review large, pretrained models for decision-making (Sec. 3.2), and discuss traditional adaptation methods in this area (Sec. 3.3).
3.1 CONTINUAL. IMITATION LEARNING
The agent encounters a sequence K tasks, denoted as (T1...., Tx}. Each task To = (u4 9s) characterized by an initial state distribution μ and goal predicate gk- Goals for tasks can be specified using language instructions, providing clear context (Jang et al., 2021; Zhang et al., 2023a). For every task Tr. the agent receives N demonstration trajectories Dj = { Th NN). In this paper, we use the standard behavioral cloning loss to optimize the agent's 's policy over these demonstrations, however we note that TAIL can be used with other training objectives as well:
0 min
Σ c (F(alsse, Te; Ts; 0)
Here, C is supervised action prediction (e.g., mean squared error negative log likelihood) loss, l the length of demonstrations for task Tx. and 0 refers to the learnable parameters of the network. Notably, after learning task Tk. the agent cannot access additional data from preceding tasks. This presents a continual learning challenge, emphasizing the importance transferring knowledge across tasks without the risk of catastrophic forgetting (McCloskey & Cohen, 1989).
3.2 PRETRAINED DECISION- -MKIING MODELS
Here, briefly describe common features of large pretrained decision-maie model architectures used for embodied agents. We incorporate key components shared amongst these models into the architecture of the model that we pretrain to evaluate efficient adaptation, pictured in Fig. 1(a).
3

---

Published as a conference paper at ICLR 2024
Transformer Backbone. Most recent work training large-scale decision decision-making models (Brohan et al., 2022: Shafiullah et al., 2022; Brohan et al., 2023) utilize a transformer backbone (Vaswani et al., 2017) that attends to tokenized observations from prior timesteps. We adopt standard GPT- 2 (Radford et al., 2019a) transformer decoder (Fig. (()), temporal decoder) with separate encoders for each input modality and continuous action distribution outputs.
Pretrained Input Encoders. Encoders pretrained on large, diverse datasets can produce rich, well- structured embeddings which make it easier learn the downstream tasks (Jang al., 2021; Brohan et al., 2022). Therefore, we utilize pretrained CLIP image and textual encoders (Radford et al., 2021).
Input Modality Fusion. The idea of explicitly "fusing" different input modalities has seen great suc- cess not only in domains like vision and language (Perez et al., 2017 7), but also in agent learning (Jang al., 2021; Brohan al., 2022). Similarly, we utilize FiLM layers (Perez et al., 2017) (Fig. (a)), input fusion module) to fuse language task specifications with observations.
3.3 ADAPTING PRETRAINED MODELS FOR NEW TASKS
One standard adaptation method in prior research full fine-tuning (FFT) of all model parameters (Fig 1(b), top left). Though straightforward, it is resource-intensive and prone to overfitting with limited data (Bousmalis et al., 2023). There is also risk of distorting pretrained features, resulting in the loss of prior tasks a phenomenon known catastrophic forgetting (McCloskey & Cohen, 1989). Evidence also suggests that extensive fine-tuning might undermine a model's 's rapid adaptability to new tasks, effect referred to the loss of model plasticity and capacity (Kumar al., 2022; Lyle et al., 2022; Kumar et al., 2023). Such issues become more prominent in continual learning contexts (Lopez-Paz. & Ranzato, 2017). Moreover, duplicating sizable model for each subsequent task is neither efficient nor practical due to storage limitations.
Another standard adaptation method is the use of frozen pretrained features (FPF, Fig 1(b) top right). FPF ensures the retention of knowledge acquired from previous tasks by tuning a task-specific head. However, as noted in Sharma et al. (2023), it is not expressive enough for outto--distribution or especially complex tasks. Given these challenges, there's s clear need for more advanced fine-tuning paradigm that addresses catastrophic forgetting while maintaining model plasticity for adapting to new tasks, all in data and computationally resource-efficient manner.
4 TASK-SPECIFIC ADAPTERS FOR IMITATION LEARNING
In this section, we outline how we perform efficient adaptation on pretrained models through our Task-specific Adapters for Imitation Learning framework, depicted in Fig 1(b). Different from the FPF approach which simply substitutes the policy head for every new task, TAIL introduces small set of new weights, serving as a lightweight plugin to address specific tasks. This concept draws inspiration from parameter-efficien adaptation techniques prevalent in the language model area. These methods offer several advantages as they: (1) add a few parameters (typically between 0.1% 2%) to preserve the original features, thereby enhancing model plasticity for continual learning and avoiding catastrophic forgetting (Kumar et al., 2023), (2) are resilient to overfitting when adaptation data is scarce, (3) are more computationally and storage- than FFT.
Next, we delve into three prominent weight integration techniques for Transformer-base pretrained models Sec. 1. followed by case study illustrating the application of this framework in continual imitation learning scenarios in Sec. 4.2.
4.1 ADAPTER WEIGHTS INTEGRATION
The concept of an adapter can be best conceptualized as a modular plugin to the base model, cus- tomized for specific downstream tasks, that does not affect the model's pretrained representations. We mainly explore three prevalent styles of integration for TAII.: Parallel (Hu et al., 2021), Sequen- tial (Houlsby et al., 2019; Sharma et al., 2023), and Prefix Token (Li & Liang, 2021; Lester et al., 2021; Liu et al., 2023b), all of which are showcased with a Transformer block in Fig. 2. Parallel and sequential integration techniques are generally applicable to any model with feedforward layers, while the prefix token style method is especially tailored for Transformers.
Given a pretrained model, let's consider one layer weight matrix in it, denoted W E Raxk Its input and output hidden states are hig E Rd and hout E R² respectively. We have hout = WTh Next, we detail how apply parallel and sequential insertions to the pretrained weight matrix.
4

---

Published as a conference paper at ICLR 2024
laper
Trowen
bettleeett
Sequential
Prefix loken
Fedd Forward Layer
Add & laper Korm
up
Rettleneck
W
LotA
W
LotA
W
Lott
Mult-hoo Attertion
dean
Transformer
Figure 2: Demonstration three weight integration styles of TAIL for a Transformer block: sequential (bottleneck adapter), parallel (LoRA), and prefix token (prefix/prompt-tnnnng
Parallel Integration (LoRA). This integration method, often associated with Low-Rank Adaptation (LoRA) (Hu et al., 2021), introduces trainable low-rank matrices Wdoun E Rdxr and Wap E R Here, < min(d, k) represents the rank and is usually much smaller than the dimensions of the original matrix. These matrices are typically integrated in parallel with the original weight matrix W through addition, as shown as LoRA Fig. 2:
hoat w hin + a W W Woown hins up
(2)
with a being a hyperparameter to modulate task-specific adjustments. The above equation can also be formulated as: hout (W a W w.p) (T,)Thag han (W + TaW) Tha. where A W denotes the weight modifications for new tasks, and thus the columns of W down and W can be interpreted as new basis that contains task-specific knowledge. As observed by Aghajanyan et al. (2020), despite projecting to a condensed subspace with small "intrinsic dimensions," pretrained models can still learn effectively. By introducing the two low-rank matrices, the original weight matrices W can be adeptly tailored with minimal increase in parameters. Though LoRA was originally crafted for large language models- poddels-specffiallle
for the query and value projections matrices We and Wv in multi-head attention (Hu et al., 2021)-it it is easily applied to other linear layers as well, such as the Transformer's 's feedforward layers (Chen et al., 2022).
Sequential Integration (Bottleneck Adapter). Renowned in the language model domain, the Bottleneck Adapter introduces bottleneck layers within the model (Houlsby et al., 2019; Sharma et al., 2023) by appending trainable bottleneck layer after the feedforward network in each Transformer layer. Similar to LoRA, this bottleneck consists of down and up projections, Wdown and Wup- which first shrink then restore the dimensions of token hidden states. Formally, for the feedforward network's input hin and a bottleneck size the output Baut is:
Nout W db (WI
down (W (WThan)) hin)),
where & denotes a nonlinear activation function. The Bottleneck Adapter (Fig. 2) acts as filter, isolating relevant information for specific tasks. Yet, filtering often requires larger bottleneck size compared to that of LoRA, leading to more parameters. Additionally, the sequential insertion can increase latency compared to the parallel nature of LoRA (Hu et al., 2021).
Prefix Token Integration (Prefix & Prompt-Tuning). In this style, set of learnable prefix tokens are appended or prepended to the input sequence (Li Liang, 2021; Lester et al., 2021; Liu et al., 2023b). Let's consider an input sequence embedding dimension. The prefix tokens can be represented as p E mmmxd where m denotes the number of prefix tokens. These vectors act like virtual tokens which the original tokens attend to. They are initialized and learned during the task-specific adaptation phase. The modified input sequence, after appending the prefix tokens, can be expressed as S [p: E (m))x) The model then processes this extended sequence. These prefix tokens can be viewed as task descriptors that designed to guide the model towards the desired task specifcc behavior (see in Fig. 2).
Ra where is the sequence length and d is the
With adapters,
treat the optimization from Eq. the model is parametrized by 0 {0, w) and is the set of adapter weights we optimizing for.
over adapter weights instead, where
5

---

Published as a conference paper at ICLR 2024
Kitchen
Spatial
Goal
Object
Living Room
Study Room
Figure 3: Our task suites for continual imitation learning (excluding LIBERO-10) 10). The robot, placed in tabletop environment, equipped with 6-DOF arm and parallel gripper. It receives RGB images from two views, joint states, and language instructions, and tasked with producing continuous actions to control its arm.
4.2 TAIL FOR CONTINUAL IMITATION LEARNING
We consider the continual imitation learning problem as a typical application of the proposed TAIL adaptation paradigm. The goal of continual imitation learning is to ensure that the model performs effectively on the current task and without significant degradation of performance in past tasks. Given pretrained model weights, denoted as 0, and new task Ta with demonstrations Dj {Tk..... Tk. J. we initialize the task-specific adapter weight WP with far less parameters than the base model: |w < |0). The adapter weights are inserted into the model through the integration methods introduced in Sec. 4.1. By optimizing the behavior cloning loss in Eq. 1 w.r.t We while keeping the pretrained weights frozen, the policy adapts to Ta without interfering with previous tasks.
To execute a task, the corresponding lightweight adapters are loaded as plugin of the pretrained network weights. For example, when revisiting prior task Tj. where j I k. the model is configured to solely activate the j-th adapter wy. This entire procedure can be streamlined as follows:
1. For incoming task Tke acquire the training set Da, initialize task-specific adapter Wi.
2. Combine adapter wi with the base model 0 using either parallel, sequential, or prefix token.
3. Train the adapter on Di: optimize Eq. for wk. keeping pretrained parameters 0 frozen.
In essence, TAIL ensures task-specific knowledge is contained within the adapters, thereby enabling efficient adaptation without catastrophic forgetting. It's also worth noting that the TAIL framework is flexible. The choice of integration method or the specific architecture of the adapter can be tailored based on the complexity of the task or the available computational resources.
EXPERIMENTS
In this section, evaluate TAIL on wide range of tasks and benchmark its performance against other fine-tuning approaches. We mainly aim to answer the following questions: (1) Which efficient adaptation methods in TAIL work best? (2) Can TAIL prevent catastrophic forgetting of previously learned tasks, while allowing more efficient forward adaptation to new tasks over standard adaptation methods? (3) What are the computational efficiencies gained by using TAIL? Addressing them requires set of diverse tasks in realistic environments, we describe in the following section.
5.1 DATASETS AND BENCHMARK SUITES
We utilize the LIBERO robotic manipulation continual learning benchmark (Liu et al., 2023a), which features a diverse range of tasks that mirror human daily activities, such turning on stove, moving books, and opening drawers. Each task is specified via natural language instructions, for instance, "Open the top drawer of the cabinet, and put the bowl in it."
We craft a pretraining task suite, named Kitchen, involving 40 diverse tasks sourced from the LIBERO-90 dataset's kitchen scenes. We then evaluate adaptation separate task suites. LIBERO contains 3 task suites tailored for continual learning, focusing on evaluating different aspects of knowledge adaptation: the Spatial task contains the same objects in each scene but with different spatial layouts; each task in the Goal suite has distinct goals (such as open the drawer, turn on the stove), while keeping the objects and layout fixed; the Object suite contains pidk-and-place tasks for different objects in the scene but with the same layout. To create more comprehensive experimental setting, we also create 2 additional task suites (from LIBERO-90X Living Room, and Study Room. We adopt 8 tasks from each of the 5 adaptation task suites, respectively. Finally, separately evaluate each task sequentially in LIBERO-10, a benchmark with 10 challenging long-horizon tasks. See Fig. for task suite examples and Appendix Sec. D for more details.

---

Published as a conference paper at ICLR 2024
TAIL LORA
TAIL RoboAdapter
Bottleneck Adapter
TAIL- Tuning
Pretraining: Kitchen
S1i Spatial Tasks
52: Goal Tasks
S3: Object Taskss
Living Room
Study Room
0.8
0.6
0.4
0.2
0.0
100 200
500
400
500
600
Continual Training Epoch
Figure 4: Success rates for different types of adapters under our TAIL. framework. None of these methods suffer from catastrophic forgetting, so backward evaluation results are not presented here. LoRA performs hest. across all tasks, underscoring the benefits of the parallel integration approach.
5.2 EXPERIMENT SETUP
Evaluation metrics. The primary metric we report is average per-task suite success rate, measured by checking if current state aligns with pre-defined goal states. For continual learning, we also assess Forward Transfer (FWT) and Backward Transfer (BWT) across the curriculum of suites. Following the metric proposed in LIBERO (Liu al., 2023a), FWT is computed by the maximum success rate one algorithm can achieve when adapting to new task. We denote FWT at task & as FA. Meanwhile, BWT measures the success rate increase on previous tasks. Namely, when adapting to the k-th task, we record the best FWT model on this task and then evaluate this model on all previous Then we compute the success rate difference
I tasks, obtaining success rate S; 1 S S between the new model and the best FWT of the previous k tasks and then average among them to obtain the BWT metric: B
(S; F,). For both metrics, higher is better.
Model architecture. We adopt the CLIP-base model (Radford et al., 2021) both the spatial encoder and the language instruction encoder, each with 12 transformer layers. A 6-layer GPT2 structure (Radford et al., 2019b) serves our temporal encoder, with the FiL.M module (Perez et al., 201 77) handling input fusion. These components are well-regarded in the literature (Chen et al., 2021b; Brohan et al., 2022; Jiang al., 2023). Further architectural details can be found in Appendix A.
Continual Learning Baselines. We adopt four baselines: Full Fine-Tuning (FFT), Frozen Pretrained Features (FPF) which mirrors the linear probing method (Kumar et al., 2022) but also tunes both the policy head and fusion module per task, Experience Replay (ER) (Chaudhry et al., 2019) which uses 50-50 data split between new and previous task data while adapting to a new task (Rolnick et al., 2019), Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) which regularizes updates of crucial parameters from earlier tasks based on their Fisher information, and PackNet (Mallya Lazebnik, 2018) which prunes parameters to then be le-mnedd for every new task. These all use the same model and task conditioning, i.c., language, as TAIL. Further baseline details in Appendix B.
TAIL Adapters. We use LoRA (Hu et al., 2021), Bottleneck Adapter (Houlsby et al., 2019), and Prefix Tuning (Li & Liang, 2021) to represent parallel, sequential, and prefix integration styles. RoboAdapter (Sharma et al., 2023), specific implementation for decision- sequential integration style. Unlike the Bottleneck Adapter that applies weights at every transformer layer, it introduces weights only at specific transformer layers and exclusively after the feedforward layer. Configuration specifics and more details for these adapters are available in Appendix B.2.
stands as another
Training, Adaptation, and Evaluation. Each task provides 50 successful human demonstrations. These are divided into 40 training trajectories and 10 for validation. We report success rates over 10 scenes with initial states that are unseen in training. This limited demonstration setup offers an opportunity to determine which technique is less prone to overfitting in data-restricted conditions. Given our focus on evaluating the adaptation of large pretrained models, we further increase adaptation difficulty by training on and evaluating adaptation performance on all tasks within task suite simultaneously. ! We pretrain on Kitchen until performance convergence (100 epochs). Subsequent adaptations follow two setups: (1) sequential adaptation across the Spatial, Goal, Object, Living Room, and Study Room task suites for 100 epochs each, and (2) adaptation to each long-horizon task within the LIBERO-10 benchmark over 50 epochs. Each experiment is conducted with different random seeds. Except for the Experience Replay (ER) method, data from earlier tasks remains unavailable in later stages. Our diverse adaptation setup provides a thorough
We use one adapter per task suite. LIBERO (Liu al., 2023a) originally evaluated per-task hasis.
7

---

Published as a conference paper at ICLR 2024
TAIL
Roplay
EWC
Packlet
Stagee
Pretraining Stage:
Adapting Stage Spetial
Adapting Stage
Sucoess Rate
307
309
420
300
602
150
359
400
Adapting Stage 3: Object Tasks
Adapting Stage 4: Living Room
Adapting Stage S: Study Room
Epoch
Epoch
Epach
Figure 5: Success rates on the pretraining stage on 40 tasks in LIBERO Kitchen scene and adaptation stages, each with 8 tasks over 100 epochs, which are continuously evaluated in subsequent stages (shaded area).
and in-depth examination of knowledge transfer across a spectrum of domains, including spatial, procedural, visual, and compositional.
In the pretraining phase for TAIL,, we add trainable adapters to the CLIP spatial and instruction encoders while freezing the encoder weights. All other model weights are fully learnable. During adaptation, the CLIP encoders and the GPT2 decoder are frozen, while adapters for them, the fusion module, and the policy head are tuned. Adapter weights are initialized from previous adapters with minor random noise. A fusion module and policy head copy are maintained during the adaptation for both TAIL and FPF. The detailed hyperparameters are presented in Appendix B.
5.3 RESULTS AND ANALYSIS
Comparison of TAIL Integration Styles. Fig. 4 showcases the continual adaptation success rates for different TAIL methods. The efficacy of LoRA suggests that a well well-pretrainet model has a surprisingly low intrinsic dimension for imitation learning tasks (Aghajanyan et al., 2020). This implies the existence of low-rank reparameterization that is just adept for fine-tuning as the full parameter space. Further, the prefix tuning method outperforms the bottleneck based approach (Houlsby al., 2019), indicating that the sequential integration style may not be the optimal choice for continual learning, potentially due to its inherent "filtering" mechanism. Surprisingly, RoboAdapter (Sharma et al., 2023) generally performs the worst, potentially due to only introducing weights after the feedforward layer opposed to after (Houlsby et al., 2019) or within (Li & Liang, 2021; Hu et al., 2021) the attention layer. Due LoRA's pronounced effectiveness, it predominantly employed as our TAIL. integration method in subsequent experiments.
TAIL vs. Conventional Fine-tuning. Across all evaluations, TAIL vastly outperforms all baselines in both forward and backward transfer. demonstrating that conventional fine-tuning methods are weak data-scarce continual learning. In Fig. we plot continual learning success rates over task suites, where TAIL outperforms the best baselines by over 3x in some comparisons and generally achieves the best success rates. We display additional results on LIBERO- 10, long-horizar tasks, in Table 1. Here, TAIL again performs best, with perfect backward transfer and forward transfer capabilities significantly better than the baselines: FFT not only exhibits marked catastrophic forgetting of earlier evidenevdde by poor But-bbut also compromises the model's adaptability new tasks. This decline in forward transfer is characterized by a steady descent in success rates as training progresses, displayed in Appendix Table 7. Such deterioration in flexibility has been recognized in other studies as well (Lyle et al., 2022: Kumar et al., 2023). PackNet is able to adapt well on some task suites as it learns new parameters within different parts of the model, but overall is still outperformed by TAIL.
Table Adaptation results on 10 long horizon tasks, higher better. The BWT for TAIL methods are all 0 (no forgetting). FPF results were omitted due to its near-zero performance. See per-task results in Appendix Table 7.
Conventionll Fine- Tun-nuning Mathods
TAIL-bused Methods (Ours)
Tuning B
Experience Replay
EWC
Prefix PWT
Bottlenek FWT
RoboAdapter
FWT
FWT
BWT
PWT
BWT
FWT
PWT
Average
0.48
2.16
4055 4021
0.45 = 008
-0.49 403
0.30
836
41.43
0.200
0.70 2300
0.51
2013
0.46 #011
0.42 013
Adaptation Plasticity. Exhaustive fine-tume the-tunnng on specialized domains has been found to distort pretrained features (Kumar et al., 2022), undermining model adaptability. Our circle-back experiments
8

---

Published as a conference paper at ICLR 2024
Table Comparison of trainable parameters and memory usage for TAIL and FFT. We use (-%) and | (:9) to denote the percentage of trainable parameter and the decrease of GPU memory w.n. FFT.
Conventional
TAlL-bused Methods (Ours)
Method
Full Tine-Tuning
LoRA
RoboAd
Bottleneck Adapler
Prefix Tuning
CLIP CLIP(Spulial & Task Encoder)
149.62M
0.49M
1.29M
1.3M
0.58M
GPT2 (Temporal Encoder)
21.78M
0.69M
0.40M
0.40M
0.24M
Fusion module and policy head
0,84M
0,84M
0.84M
0,84M
0.84M
Total Parameters
172.24M
2.02M (1.17%)
2.53M
(1.47%)
2.55M (1.48%)
1.66M (0.93%)
GPU Memory (Batch 14)
20.1G
15,5G () 2356)
14.0G (00))
14.9G 26%)
15.8G (1 215))
in Table 2, where a full fine-tuned model is re-trained on prior task suites, demonstrate a steep performance drop upon re-visiting previously learned tasks. Additional experiments in Appendix C.3 further highlight this issue.
The training and validation losses, detailed in Appendix C.I and Fig. 7, highlight FFT's propensity to overfit. This translates to a notable decline in success rates, rein- forcing the challenges FFT faces in balancing retention of prior tasks with the assimilation of new ones.
Table 2:
The success rate of initial training and revisiting previous tasks with FFT. FFT suffers from catastrophic forgetting and performs worsc on re-visits despite re- re-training on the same data.
LIBERO Task Suite
While ER and the regularization baared method EWC exhibit some potential in mitigating catastrophic forget- ting, they were detrimental to forward transfer perfor- mance. Their downsides are also reflected in storage and computing costs: ER requires more storage for previous data than TAIL LoRA adapter weights (e.g., Kitchen dataset at 28GB vs 7.8MB for TAIL's LoRA adapter). Furthermore, EWC presents significant challenges for larger models because of the increased GPU memory consumption from maintaining a copy of the entire weights of the old model in memory. We also found it to exhibit unstable training due to the regularization loss. More discussions are presented in Appendix B.
Type
Spatial
Goal
Object
Initial
0.79
0.42
0.42
Re-visit 0.53 (1026)
0.20 (10.22)
0.27 (_0.15)
When does TAIL work best? The efficacy of TAIL hinges significantly on the base model's features. We compare TAIL under different pretraining strategies and models in Appendix Sec. C.2 and C.3. In short, TAIL works best with our pretraining architecture and frozen CLIP visual/language encoders, and performance drops when we fine-tune the pretrained encoders, likely as FFT contaminates the rich CLIP features when fine-tuned in niche domain with sparse data.
Analysis Summary. We argue in favor of a large pretrained base model augmented with numerous lightweight plugins tailored for diffferent downstream tasks. This framework. TAIL, holds considerable promise for advancing embodied intelligence in real-world applications; the storage footprint of our entire model is about 660MB, and duplicating this model for each task in stream of oncoming tasks is impractical. Meanwhile, the space occupied by one such model can accommodate as many as 84 task-specific adapters, which, as our experiments show, can outperform full fine-tuning regardless. Moreover, the features of the pretrained weights remain intact, ensuring their applicability across a broad array of domains. In summary. TAIL offers a promising avenue for the efficient adaptation of large decision- decision-making models. Despite the fact that our method requires significantly less computation and memory (and storage), our experiments show that it consistently outperforms all prior approaches in the continual learning setting. We would also like to highlight that the TAIL framework is not restricted imitation learning, but also other learning methods such as reinforcement learning.
6 CONCLUSION
In this study, we examined the challenges of efficiently adapting large pretrained models for decision- making and robotics applications. We proposed TAIL, an efficient adaptation framework for pretrained decision-making models. Through comprehensive exploration of parameter-efficiee fine-tuning (PEFT) techniques in TAIL., especially Low-Rank Adaptation (LoRA), we demonstrated their po- tential in enhancing adaptation efficiency, mitigating catastrophic forgetting, and ensuring robust performance across diverse tasks. Our empirical evaluations on the LIBERO benchmark further underscored the advantages of these techniques in continual learning scenarios. As the demand for adaptive, intelligent agents grows across various domains, the insights from this research offer a promising direction for the future of efficient model adaptation in decision-making contexts.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Reincarnating reinforcement learning: Reusing prior computation accelerate progress. Advances in Neural Information Processing Systems, 35:28955-28971. 2022.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint arXiv:2012.13255, 2020.
Christopher M Bishop. Mixture density networks. 1994.
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306. 1tt1:2306.11006, 2023.
Stevo Bozinovski and Ante Fulgosi. The influence of pattern similarity and transfer learning upon training of a base perceptron b2. In Proceedings of Symposium Informatica, volume 3, pp. 121-126. 1976.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabís, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Ri-1: Robotics transformer for real-workds control scale. arXiv preprint arXiv:2212.068/ 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gon- zalez Arenas, et al. Rt-2: Vision- aasioo-language-cctee models transfer web knowledge to robotic control. In arXiv preprint 2307: 158818 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, et al. Language models are few-shot learners, 2020.
Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Task-agnostic continual reinforcement learning: Gaining insights and overcoming challenges. Conference on Lifelong Learning Agents, 2023.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749 2021.
Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao. Context-aware safe reinforcement learning for non-stationary environments. 2021 IEEE Inter- national Conference Robotics and Automation (ICRA), pp. 10689-10695. IEEE, 2021a.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 50884150977 2021b.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer; Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16677 166788 2022.
Open X- E-Eboddiment Collaboration, Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, et al. Open X-Embodiment: Robotic learning datasets and RT-X thub. io, 2023.
models. https: htop:::/robbtte t
10

---

Published as a conference paper at ICLR 2024
Thomas G Dietterich, Lorien Pratt, and Sebastian Thrun. Special issue inductive transfer. Machine Learning, 28(1), 1997.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.033778 2023.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning. In International Conference Learning Representations, 2020.
Haotian Fu, Shangqun Yu, Michael Littman, and George Konidaris. Model-based lifelong reinforce- ment learning with bayesian exploration. Advances in Neural Information Processing Systems, 35: 92369-32382, 2022.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Abhishek Gupta, Corey Lynch, Brandon Kinman, Garrett Peake, Sergey Levine, and Karol Hausman. ainponstration-boostrrppa autonomous practicing via multi-task reinforcement learning. arXiv, 2022.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting baseline. arXiv preprint 12X.v:2212.05779, 2022.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficion transfer learning. International Conference on Learning Representations, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799 PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprins arXiv:2106.099 09680.0096,,.,.. 2021.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In Sth Annual Conference on Robot Learning, 2021.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference Machine Learning, 2023.
Dmitry Kalashnkov, Jake Varley, Yevgen Chebotar, Ben Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f f-divergence minimization. arXiv preprint 1905.12888. 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinska. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Ananya Kumar. Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of nut-of-distribution International Conference Learning Representations, 2022.
11

---

Published as a conference paper at ICLR 2024
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308. 11958,, 2023.
Yoonho Lee, Annie Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. International Conference on Learning Representations, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint 2 104. 104.08691, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 X11:22001.00190. 2021.
Anthony Liang, Ishika Singh, Karl Pertsch, and Jesse Thomason. Transformer adapters for robot learning. In CoRL. 2022 Workshop Pre-training Robot Learning, 2022.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.033100
2023a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023b.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351, 2023c.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NIPS), 2017.
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov. and Sergey Levine. Aw-opt: Learning robotic skills with imitation andreinforcement at scale. In 5th Annual Conference on Robot Learning, 2021.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint ww.iv:2210.00033 2022.
Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. arXiv preprint arXiv:2306.0 00000008,000088 2023.
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Olek- sandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? 2023a.
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piere Berges, Pieter Abbeel, Jitendra Malik, et al. Where are we the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:
2023b.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín Martin-Martín What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint 108.::201000322: 2021.
12

---

Published as a conference paper at ICLR 2024
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficien language model tuning. 2022.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivalion, volume 24, pp. 109-165. Elsevier, 1989.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12600 2022.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer, 2017.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint 40X2472005.00247, 2020a.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779 2020b.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019a.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9 2019b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Rebuffi, A. Vedaldi, and H. Bilen. Efficient parametrization of multi-domain deep neural networks. In 2018 IEEE/CVF Conference 8127, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society. doi: 10.1 10..109//....0010 2011 00847. URL httph:tt////mmm / / /oii. http//oiicc..ooo/// ieeecomppteracciitt....oo. wwwpeeiiiiiittttooooooeooeeeoeee 1109 C 00847.
Computer Vision and Pattern Recognition (CVPR), pp. 81 8119-
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, et al. A generalist agent. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. Featured Certification, Outstanding Certification.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635. PMLR, 11 11-13 Apr 2011.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compres- sion. Neural Computation, 4(2):234-242, 1992. doi: 10. 10.1162/ncco.11 1992.4....99222
Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models rl. arXiv preprint wwXiv:2306.14884, 2023.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning $kS modes with one stone. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Mohit Sharma, Claudio Fantacci, Yuxiang Zhou, Skanda Koppula, Nicolas Heess, Jon Scholz, and Yusuf Aytar Lossless adaptation of pretrained vision models for robotic manipulation. arXiv preprint ar arXiv:2304.066ii 2023.
13

---

Published as a conference paper at ICLR 2024
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 984-906. PMLR, 2022.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. In The biology and technology of intelligent autonomous agents, pp. 165-196. Springer, 1995.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
René Traoré, Hugo Caselles- Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz Ro- dríguez, and David Filliat. Discorl: Continual reinforcement learning via policy distillation. CoRR, abs/1907. 0058555 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Mengdi Xu, Yikang Shen, Shun Zhang. Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631-244 24645. PMLR, 2022.
Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper-decision transformer for efficient online policy adaptation. arXiv preprint arXiv:2304.08887 2023.
Yihang Yao, Zuxin Liu, Zhepeng Cen, Jiacheng Zhu, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constraint-conditionel policy optimization versatile safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Jesse Zhang, Karl Pertsch, Jiahui Zhang, and Joseph J. Lim. Sprint: Scalable policy pre- pre-training via language instruction relabeling. arXiv preprint arXiv:2306. 11886, 2023a.
Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. Bootstrap your own skills: Learning solve new tasks with large language model guidance. In 7th Annual Conference on Robot Learning, 2023b.
14

---

Published as a conference paper at ICLR 2024
Appendix: TAIL: Task-specific adapters for imitation learning with large pretrained models
A MODEL ARCHITECTURE DETAILS
GMM
GMM distribution
GMM distribution
distribution
action
t:2
Temporal Decoder (GPT2)
Input Fusion layer)
Inpurt
layer)
Input (FILM layer)
Task token
Image token
Image toller
state tobon
image token
image tolic
state toboe
image sokon
Image tollon
state tobes
Instruction Encoder (CLIP)
Spatial Encoder (CUP ViT)
State encoder
Spatial Encoder [CUP VIT)
Spatial Encoder (CUP ViT)
State encoder
encoder
Ojen
Low states
Low-dim states (t-]]
Low din states (1
the
description
Agant-stew Image (1-3
Eye-tn-hand
Image (1-2)
Agentsskww
Image (-:1
Bye-inhand Image (t-1]
Agent-vlew Image
Eye-in-hunn hat---aad Image (1)
Figure 6: A detailed view the multi-modal, transformer policy architecture we utilize for pretrain- ing. We encode language task descriptions with a pretrained CLIP instruction encoder and image observations with a pretrained CLIP spatial encoder. We additionally encode robot state observations which, along with the observation embeddings, are embedded into a sequence of tokens used by the temporal decoder transformer to predict single-step action distributions. We include an input fusion module (FiLM (Perez et al., 2017)) explicitly combine the task embedding with the observation and state embeddings for better instruction-followinn ability.
A.1 PRETRAINED INPUT ENCODERS
We utilize pretrained CLIP image and textual encoders (Radford et al., 2021) to encode image observations and language goal descriptions, respectively. Note that we do not use a pre-trained encoder for the low-dimensional state; the state encoder is learned from scratch.
A.2 INPUT MODALITY FUSTON
We utilize Feature-wise Linear Modulation (FiLM) layers (Perez et al., 2017) (Fig. (aa), input fusion module) to fuse language task specifications with image observations. FiLM is technique in multi-modal deep learning which modulates the intermediate activations of a neural network based on external information. Rather than explicitly designing architectures for conditional computation, FiLM layers simply use the features from one network to modulate the features of another.
Let's consider neural network f with intermediate activations x and an external network g which outputs modulation parameters 7 and B. The modulated features x' are given by:
7. 7 g(z)
(4)
x'=70x+3
(5)
where 2 is the input to the external network g: @ represents element-wise multiplication; and 3 are vectors having the same size 2r, with each element modulating corresponding feature in
15

---

Published as a conference paper at ICLR 2024
Thus, FiLM layers allow for dynamic and feature-wist conditional computation without needing explicit architectural changes. As such, task token (language) embeddings are given as input to a fully connected feedforward network, which outputs scale and translation parameters for the image and state embeddings. These parameters modulate the image and state embeddings before they are passed to the transformer backbone.
A.3 TEMPORAL TRANSFORMER BACKBONE
We utilize a standard GPT-2 (Radford et al., 2019a) transformer backbone for our policy. Its input sequence of image and low-dim state encodings (robot joint states in LIBERO) and it outputs an action distribution. Following the literature (Mandlekar et al., 2021; Liu et al., 2023a), we adopt (GMM) (Bishop, 1994).
a stochastic policy parametrization based on a Mauusian-Mixture-MMee Therefore, for every decision-making step, the transformer produces latent vector of Gaussian means and variances, one for each of the GMM modes. We optimize the parameters of the model with the negative log-likelihood loss the ground truth actions based on the parameters of the GMM distribution. At evaluation time, we deterministically select the next action parameterized by the mean of the Gaussian model with the highest density.
The cavironment configuration and the temporal decoder (GPT-2) hyperparameters are presented in Table 4.
Table 4: Environment configuration and GPT-2 model hyperparameters
Environment Configuration
GPT2 Temporal Encoder Configuration
Action Dim.
Max Seq Length
Activation
Gelu New
Raw State Dim.
Number of Heads
Number of Layers
6
Max Episode Length
500
GMM Min Std
0.0001
GMM Modes
5
Image Resolution
128 128
FiLM Layers 2
Dropout
0.15
Image Views
Agent Front, Fye-in-Hand
B IMPLEMENTATION AND TRAINING DETAILS
B.1 BASELINE DETAILS
Experience Replay (ER). ER (Chaudhry et al., 2019; Rolnick et al., 2019) is rehearsal-based approach that retains a buffer of samples from previous tasks to facilitate the learning of new tasks. After completing the learning process for task, subset of the data is saved into this buffer. During the training of subsequent tasks, ER draws samples from this buffer and mixes them with current task data. This process ensures that the training data closely resembles the distribution of data across all tasks. In our setup, we store all the previous trajectories in a replay buffer. For each training iteration on new task, we uniformly sample 50% trajectories from this buffer and 50% from the new task's training data, respectively.
Elastic Weight Consolidation (EWC). EWC (Kirkpatrick al., 2017) a regularization method that adds term to the standard single-tast task learning objective to constrain the updates of the neural network. This constraint uses the Fisher information matrix gauge the significance of each network parameter. The loss function for task A is represented as:
Lewc, (0) LaC (0) + Σ
Ri 1,,1)
Here, À is hyperparameter penalty, and F; the diagonal of the Fisher information matrix given by:
F₂ Ex-De. (Va, log ps, (als))²
For our experiments, we adopt the online version of EWC. It updates the Fisher information matrix using an exponential moving average throughout the lifelong learning process. The actual Fisher Information Matrix estimate used is:
F = 11= (1
with F3- E,,,,)DD, (Ve, log Po, (als))² and l representing the task number. Following the bench- mark implementation (Liu al., 2023a), the hyperparameters set as 7 0.9 and X 5 10a.
16

---

Published as a conference paper at ICLR 2024
Discussions. Both Experience Replay (ER) and Elastic Weight Consolidation (EWC) demonstrate potential in mitigating catastrophic forgetting. However, they each come with notable limitations, particularly with respect to forward transfer performance, storage, and computational efficiency.
Storage Overhead: ER demands significant storage space to maintain samples from prior tasks. This becomes particularly evident when comparing the storage needs of ER for larger datasets, such the Kitchen dataset which requires 28GB, with the lightweight LoRA adapter occupies only 7.8MB. The vast difference in storage demands underscores the inefficiency of the ER approach.
Computational Challenges: EWC, by design, necessitates the maintenance of a copy of the weights of the previous model in GPU memory. This leads to escalated GPU memory consumption, making EWC tends to reduce the training batch size, subsequently slowing down the training process.
Training Instability: The regularization approach of EWC can introduce instability during training, owing to the regularization loss. This is also reflected by the poor forward transfer capability, as shown in Table 1.
Scalability Concern:: While EWC might be manageable for smaller networks, it is ill-suited for the fine-tuning of larger decision models due its computational and storage challenges.
Given these outlined limitations, we advocate TAIL for alternative approaches that are both storage- efficient and computationally scalable, especially for large pretrained model adaptation.
B.2 TAIL ADAPTER CONFIGURATIONS
To establish our TAIL adapter configurations, primarily draw from the AdapterHub implementa- tion, setup and hyperparameters (Pfeiffer et al., 2020b).
We utilize the default hyperparameters for LoRA, with the rank 8 and scaling factor = 8. These low-rank matrices are applied in parallel to the Transformer's 's query and value matrices (Hu al., 2021). We also adopt the default for prefix token length of 30 for the prefix tuning (Li & Liang, 2021) method across all tasks. To improve the training stability, Low-rank matrices (r = 16) are employed during training represent the prefix tokens. The Bottleneck Adapter (Houlsby 2019) employs the bottleneck size of 32, and is applied to both the output layer of the attention and the intermediate feedforward layers. The RoboAdapter method (Sharma et al., 2023), as the closest work to us, also applies the sequential adapters to the decision-making domain. It differs from the Bottleneck Adapter in that they adopt special insertion of weights to specific layers of the Transformer, namely, layers 0, 1,5, 6, 10, 11. They selectively skip certain layers, aiming to increase the bottleneck size on the remaining layers. Therefore, the bottleneck size doubled to 64 for this approach, such that all methods share similar amount of parameters.
al.,
In order to maintain balanced adapter parameters number between the two CLIP-based (spatial and instruction) encoders, and the temporal transformer GPT2 decoder, the rank size for the GPT2 decoder doubled across all methodologies. This adjustment compensates for the GPT2 decoder's fewer layers relative the encoders.
For the continual learning setup, we use the previous stage's s adapter weight (if any) plus a small random Gaussian noise with standard deviation 0.001 as initialization of the current stage. The goal for adding a minor random noise aims to improve the adapter weight capacity (Kumar et al., 2022; Agarwal et al., 2022; Lyle al., 2022), preventing the weights from being trapped into local optimum. There is potential to better utilize the trained adapter weights on preceding tasks. We outline several promising exploration directions in Appendix Section B.4.
B.3 TRAINING HYPERPARAMETERS AND EXPERIMENT CONFIGURATIONS
Following similar setup in the LIBERO benchmark (Liu et al., 2023a), we perform data augmenta- tion for the image observation data for all methods. We adopt the color, affine, and random crase augmentations improve the robustness. The hyperparameters are presented in Table 5.
For our training process, we employed the AdamW optimizer combined with a linear learning rate scheduler. The majority of our task tiites-Kitchenn, Spatial, Goal, Object, Living Room, and Study Room-underwen underwent training for 100 epochs. Notably, each suite encompasses multiple tasks, with Kitchen having 40 and the others containing 8 each. In contrast, the 10 long-horizon adaptation tasks,
17

---

Published as a conference paper at ICLR 2024
Table 5: Image data augmentation and training hyperparameters
Image Augmentation
Training and Optimizer Configuration
Brightness
0.3
Contrast
0.3
Traiinn Epoclas
T0020
Batch Size (per device)
10/14/18
Saturation
0.3
Hue
0.3
Truining Epochs per Eval
Eval Episodes/Tusk
Color Aug Pmnh.
0.9
Affine Degrees
Warm up Steps
500
Weight Decay
0.1
Affine Translate
0.1
Affine Prob.
0.6
Learning Rate (LR)
lc-4
LR Scheduler
Lineur
Random Erase Prob
0.1
Training Demo Num
40
Validation Demo Num
40
termed LIBERO-10, 10, were trained for 50 epochs, with each task trained sequentially. We performed evaluations after every 5 training epochs over episodes (unseen in training) for each task.
Computing machine. Our experimental platform was powered by an AMD EPYC 7R32 CPU running Ubuntu 20.04.06. All trainings utilized 8 NVIDIA A10G GPUs, each with memory of 22731 MiB, equipped with driver version 470.199.02 199902 and CUDA version 11.4. We employ Distributed Data Parallel (DDP) for parallel training across GPUs, and utilize the 16-bit floating point precision (FP16) training mode to accelerate the training process. To ensure reproducibility, we adopted 3 distinct random seeds: 0, 21, and 42.
Training time. For holistic perspective on training duration: FFT and ER methods demanded between 120 140 hours per experiment (1.5 1.75 hours per task) for the task suites shown in Fig. 5, including the evaluation time. In stark contrast, TAIL-based techniques slashed this to 60 ~ 66 hours (0.75 ~ 0.825 hours per task). Hence, TAIL would also be much cheaper to train, considering its significantly shorter training time under identical computing machines.
Batch sizes varied by training method. EWC employed a batch size of 10, given its added memory demands to store distinct full parameter set. FFT and ER utilized batch sizes of 14. Owing to TAIL's more efficient memory muliizaioondeeta which can maximize GPU resource usage our machine, reducing training duration and cost.
in Table 3-a larger batch size of 18 was feasible,
B.4 MORE DISCUSSION AND FUTURE DIRECTIONS
The TAIL framework paves the way for a myriad of research opportunities:
1. Better Weight Allocation Method Across Layers: An interesting question within this framework discerning which layers, early or later, derive the most benefit from weight modifications. This can offer insights into the adaptability of neural architectures (Lee et al., 2023).
2. Enhanced Reusability of Trained Adapters: Exploring methods to efficiently reuse adapters from prior tasks, especially in scenarios with limited data, is a promising direction. AdapterFusion techniques (Pfeiffer et al., 2020a) can be potentially useful, enabling the composition of knowledge from multiple pre-existing adapters.
Building on Knowledge with Parallel Integration: The parallel integration method, particularly with LoRA weights, offers the capability to merge trained weights back into the main model. This iterative buildup of knowledge makes the approach valuable for continual learning, allowing new adapters to capitalize on the expertise of their predecessors.
4. Combining with Established Continual Learning Strategies: The potential to merge the TAIL framework with existing continual learning methods, like Experience Replay and EWC, can be a beneficial avenue. Such integrations can accommodate the strengths of each method, crafting models that are both efficient in memory and robust against forgetting.
5. Extension beyond the Imitation Learning Domain:
Taking the TAILframework into other decision-making domains, such as reinforcement learning (RL), is also promising. TAIL has the potential to address the model capacity loss issue in RL (Agarwal et al., 2022: Lyle et al., 2022). Leveraging the TAIL framework can also aid in multitask learning, meta-learning, and efficiently adapting trained RL models to new tasks without the necessity of vast amounts of data or extensive fine-tuning, thereby potentially accelerating convergence to optimal policies.
The avenues above elucidate the adaptability and potential of the TAIL. framework, setting the stage for future research in this domain.
18

---

Published as a conference paper at ICLR 2024
C MORE EXPERIMENT RESULTS
C.I OVERFITTING
For each task, used 40 demonstrations for training and 10 for validation. We interested in the following question: In scenarios where data limited, how resilient is TAIL against overfitting compared to traditional fine-tuning methods? ? To answer this, present the training and validation loss cross the Kitchen, Spatial, Goal, Object, Living Room and Study Room task suites, each with 100 epochs, in Fig. 7.
TAIL (LoRA)
TAIL (Bottleneck Adapter)
TAIL (Prefix Tuning)
FFT
Training Loss
Validation Loss
-200
100
200
300 Epoch
400
500
600
100
200
300
400
500
600
Epoch
Figure 7: Adaptation loss trends: Training versus validation. The graph shows that the TAIL model consistently has more stable validation losses, which means that it is more robust to contexts with limited data. On the other hand, the full fine-tuning model (FFT) has larger validation losses, which means that it is more likely to overfit to the training data.
A noteworthy observation from Fig. 7 is the behavior of FFT. Despite achieving the lowest training loss across all stages, its validation loss spikes significantly after just few epochs. This pattern suggests severe overfitting when FFT is applied to the entire parameter space using limited data. Intriguingly, this overfitting intensifies the later adaptation phases, potentially signifying a distortion of pretrained features alluded by Kumar al. (2022). Such distortion could be contributor to the suboptimal success rate observed in Fig. 5, and the loss of learning capacity when revisiting a previous task, presented in Table 2.
In constrast, Tale-based methods shows strong resilience against overfitting. Drawing from the Occam's razor prin- ciple, TAIL leverages fewer trainable parameters, inher- ently reducing its potential overfit with scarce data. Ad- ditional, different integration styles provide the flexibility effectively utilize the features from pretrained models while preserving them across all the adaptation stages.
Pretrain on Kitchen vla FFT
Success Rate
0.2
0.0
VC-1
CUPP
Random
This observation underscores the disparities between our docision-making problem, characterized by its limited data, and the traditional language or vision domains, which have data in abundance. Prior studies utilizing parameter- efficient fine-tuning techniques for language or vision tasks often reported superior performance with full fine- tuning due to its low training loss (He et al., 2022; Mao
Spatial Encoder Weights
Figure 8: Training on the Kitchen task with different pretrained CLIP-VIT coder weight. Random means using ran- dom initialization weight.
results demonstrate, lower training loss does not invariably translate to superior performance, especially in the context a data-scarce sequential decision-makinn tasks.
et al., 2022; Chen et al., 2022; Sharma et al., 2023). However,
C.2 ANALYSIS OF PRETRAINED WEIGHTS' INFLUENCE
aim to answer the following question: how does the underlying pretrained base model influence the performance of TAIL, and are certain pretrained weights more conducive to this kind of adaptation? We initiated our investigation by analyzing the success rates of 40 Kitchen tasks using different pretrained weights for the spatial encoder. Apart from the CLIP-ViT pretrained encodings adopted in our main results, two other initialization weights were considered: one sourced from the Visual Cortex (VC-1) (Majumdar et al., 2023b), recognized for being leading pretrained
19

---

Published as a conference paper at ICLR 2024
model for embodied agent tasks, and another using randomly initialized weights. The language instruction encoder consistently utilized the CLIP text model. From the results in Fig. 8, the VC-1 pretrained weights delivered performance on par with the CLIP-ViT encodings. Both considerably outperformed the randomly initialized weights, suggesting that large-scale pretraining can indeed enhance downstream fine-tuning. We then study how does the pretrained base model influence the performance of TAIL.
C.3 FURTHER EVALUATIONS ON TAIL WITH DIFFERENT BASE MODELS
To understand the influence of the base model's features on the performance of TAIL., we conducted additional evaluations. In Table 6, the methods column showcases different configurations:
LoRA (CLIP): The main setup we adopted in the experiment section 5, which keeps the pretrained CLIP encodings frozen across all the adaptation stages.
LoRA (CLIP with FFT):: Starting with the CLIP model, we applied FFT pretraining on the Kitchen task before using LoRA for subsequent adaptations. This helps us test out whether adaptation plasticity suffers after full fine-tuning as the only difference between this and the above method is the addition of full fine-tuning before using LoRA.
LoRA (VC-1 with FFT); The VC-1 model, after FFT pretraining on the Kitchen task, was adapted using LoRA.
LoRA (Random with FFT): A model with randomly initialized weights underwent FFT pretraining on the Kitchen task, followed by adaptation with LoRA.
All the pretrained encodings implemented in the same model architecture as described in Appendix Section A.
Observations from Table 6 highlight several findings:
The pure CLIP base model, when combined with LoRA, yielded the highest success rates across all task suites, suggesting the inherent quality and robustness of the original CLIP features for these tasks.
Dominance of Original CLIP:
FFT's Mixed Impact: While FFT pretraining aids in task-specific fine-tuning, when combined with CLIP, it leads to a degradation in performance. This could be attributed FFT potentially diluting the comprehensive and rich features within CLIP while also reducing adaptation plasticity (Kumar et al., 2022), especially when exposed to more constrained domain with limited data.
The VC-1 model, though renowndd in the domain of embodied agent tasks, delivered results that were only marginally better than the randomly initialized weights when both were subjected FFT pretraining and then adapted with LoRA. This emphasizes the unique advantages of the original CL.IP features.
VC-1's Comparable Performance:
Interestingly, it observed that CLIP is pretrained on the most comprehensive dataset, followed by VC-1. In contrast, the model with random weights only underwent pretraining on the Kitchen tasks. The success rates mirror this order, underscoring the idea that the efficacy of TAIL is closely tied to a base model pretrained with rich features on extensive datasets. So summary, the choice of base model significantly affects the performance of TAIL, with CLIP's original features showing remarkable compatibility and resilience across various task suites
Table Evaluation results FWT for LoRA with different pretrained model weights. The higher, the better.
We highlight the best method with highest FWT as bold.
Method
Spatial
Goal
Object
Living Room
Study Room
Average
LoRA (CLIP)
0.76 2002
0.79 2002
0.73
0.73 2007
0.55
0.71
LoRA (CI.IP with FFT)
0.62 004
0.67 n113
0.38 no0
0.32 no0
0.32 n00
0.46 1007
LoRA (Random with FFT)
0.38 n10
0.60
0.37 n00
0.23 =001
0.47
0.41 1007
LoRA (VC-1 with FFT)
0.56 000
0.66 = nos
0.25 0.00
0.20 0.05
0.48 2007
0.43 1005
20

---

Published as a conference paper at ICLR 2024
C.4 RANK SIZE ABLATION STUDY
In order to understand the impact of rank-size on adaptation performance, we conducted experiments using varying rank sizes for the LoRA and Bottleneck Adapter methods. The results, illustrated in Fig. 9, present the average success rates across the Spatial, Goal, and Object task suites. It is evident that increasing the rank size generally enhances performance up certain point. Beyond this optimal threshold, further increasing the rank size does not necessarily lead to higher success rates, potentially because of overfitting. Notably, in our continual learning context, the parallel insertion approach of LoRA consistently surpasses the sequential style of the Bottleneck Adapter method.
Success Rate vs. Rank Size
0.8
TNI-LORA
FFT-CUP
0.6
0.7
0.6
0.4
Success
0.5
0.2
0.3
0.2
LoRA Bottleneck Adapter
0.8 1Ztdhen
Spattal
Continual Learning Task Suites
Good
Object
Dving
Study
Figure 10: Comparison for TAIL-LoRA, sequential FFT with pre-trained CLIP weights, and FFT-In-Domain. FFT-In-Domain is trained from scratch with task-suite demonstration data only, which saves a copy of the entire model for each task. FFT with CLIP excels in initial Kitchen and Spatial suites, highlighting the value of pretrained models; however, its performance declines subsequent tasks, suggesting reduced adaptability. In contrast, TAIL-LoRA demonstrates consistent superior performance across all suites.
Figure 9: Ablation study of the rank-size of LoRA and Bottleneck adapters. Increasing the rank size generally enhances performance up certain point. Beyond this optimal threshold, further in- creasing the rank size does not necessarily lead to higher success rates. The parallel insertion ap- proach of LoRA consistently surpasses the sequen- tial style the Bottleneck Adapter method
Additionally, we would like to note that our TAIL framework exhibits data adaptivity, suggesting that the rank size could be adjusted based on the quantity of adaptation data. In scenarios with smaller datasets, a smaller rank size could be more effective, and vice versa.
C.5 COMPARISON BETWERN TRAINING FROM SCRATCH AND USING PRETRAINED MODELS
Fig. 10 compares the success rates across task suites for TAIL-LoRA, sequential FFT with pre-trained CLIP weights, and FFT-In- In--omain. Unlike FFT-CLIP, IFT-In-Domai is trained from scratch with task-suite demonstration data only, i.e., we need to maintain copy of the entire model for each task suite. There are three observations:
1. Pretrained Weights Advantage: In the initial Kitchen and Spatial task suites, FFT with CLIP pretrained weights demonstrates a higher success rate compared to FFT trained from scratch. This indicates the effectiveness of leveraging pretrained models, particularly in the context of the Kitchen suite where the benefit is more pronounced.
Despite the initial advantage, sequential FFT with CLIP shows marked decline in performance in the remaining four task suites Goal, Object, Living, and Study. This trend may be indicative of loss in model plasticity, where the pre-trained model performs well in the carly stages but struggles to adapt to new tasks after the pre-trained weights are contaminated.
2. Decline in Model Adaptability:
3. TAIL-LoRA's Consistent Performance: Throughout all the task suites, TAIL-LoRA with pretrained CLIP consistently outperforms the other methods. This suggests that the LoRA approach, combined with the advantages of pretrained CLIP weights, provides robust and adaptable framework capable of handling a variety of tasks with greater efficiency.
21

---

Published as a conference paper at ICLR 2024
C.6 ABLATION STUDY FOR DIFFERENT INTEGRATION STYLE COMBINATIONS
It's noteworthy that our method allows for the simultaneous use of multiple integration techniques (Mao al., 2022). This flexibility lets explore the performance impact of combining LoRA (parallel integration), bottleneck adapter (sequential integration), and prefix token (concatenation). To this end, we conduct ablation study for each of the combinations over the Spatial, Goal, and Object task suites. The experiment result is shown in Fig. 11, where the y-axis is the averaged success rate.
A key finding the critical role of LoRA (parallel integration) in enhanc- ing adaptation performance. Combi- nations involving LoRA consistently outperform those without For in- stance, the standalone use of LoRA yields a comparable success rate w.r.t the combination with others. This pattern underscores LoRA's effective- ness, either used alone or in conjunc- tion with other methods. In contrast, the combination of Prefix and Adapter without LoRA results in notably lower success rate (0.6641), highlight- ing LoR A's indispensability.
Prefix
Predix
Prefix LoRA
Adapter LoRA
Ablation study of different combinations
0.7
Adaapter
+LoRA
0.4
0.3
LoRA
Preffx
Adapter
PrafixtAdapter +ooRA
Interation Method
The integration of all three methods-Prefix, Adapter, and achieves success rate that comparable to LoRA's standalone per- formance. This outcome suggests that while the combination of different integration methods does not detract from performance, LoRA remains the primary driver of successful adaptation. These findings emphasize the importance of LoRA in adapter weight integration strategies and provide valuable guidance for future approaches in this domain.
Figure 11: Ablation study integration style combinations. LoRA (parallel integration) plays crucial role in enhancing adap- tation performance, consistently outperforming methods without it. Whether used alone in combination with other methods like Prefix and Adapter, LoRA shows superior effectiveness.
C.7 DETAILED PER-TASK RESULTS IN THE LIBERO-LONG TASK SUITE
Table 7:
Adaptation results 10 long horizon tasks. The f symbol means the higher, the better. The BWT for TAIL. methods are (no catastrophic forgetting). We highlight hest method (highest FWT f) in bold. FPF results were omitted due to its near-zero performance.
Conventional Pine- Mun-Tuning Mathods
TAll-bused Methods (Ours)
Task
Fine- Tunnng
Experience Replay FWT
EWC
RoboAdapter
FWT1
1WT1
BWT
P
BWT
FWT
PWT
FWT
PWT
Task
0.42 om
0.25 0.12
0.38 0:111
0.62 : 0.80
0.38 2:11
0.21 = 0.14
0.12 0m00
Task
0.58
-0/2
0.58 007
-0.25 0:10
0.54 0:00
40.38 0.10
0.75
0.58
0.75 0.12
=
Task
0.71 om
0.50 4:0:::00 0.10
0.67 009
-0.42 0:11
0.38 011
-0.46 0.12
0.96 : 0.7
0.88 0.21
0.71 0.19
0.50 025
Task
0.96
-0.57
0.92 007
-0.50 0:20
0.75 (7552:00
2.112
0.88
0.71 600
0.71 0.19
0.58:0014
Task
0.21 om
0.67 021
0.33 = 0.14
-0.60 4:25
0.17 0:1
-0.50 1.500::0..
0.62 0.12
0.17 c00
0.25 0.00
0.29 am
Task
0.83
-0.57
0.71
-0.55
0.50 2:00=:00
40.12
0.75
0.75 0000
0.75
Task
0.17 om
-0.62 027
0.12 = 0.00
-0.58 4:225
0.04 0,5
00.44
0.54 : 0.26
0.38 6.12
0.31 = 029
0.33 0:
Task
0:42
-0.55 n:29
0.29 007
-0.51
0.12
40.16
0.75
0.67 0.19
0.25 2.18
=.50
Tak
0.17 = om0
-0.54 028
0.12 005
-0.50 4:028
0.00 0,00
0:20
0.38 : 0.12
0.08 600
0.19 : 0.9
0,21 0:00
Task
0.33
-0.50
0.50
-0,16
0.12
40.38 0:31
0.79
0.50 =5000033
0.44 008
Averge
0.48 0.10
-0.55 0.1
0.45 =: 0.99
-0,49 a
0:36
-0.43 0.000
0.70 8.10
0.51 2:15
0.46 = 0.11
0.42 0.13
D EVALUATION TASK DETAILS
We list all the language instructions describing the tasks we adopted in our experiments below. Note that while certain tasks may share similar descriptions, they the same due to variations in the environment configurations (c.g., different spatial layouts, objects, or goal positions).
22

---

Published as a conference paper at ICLR 2024
Task Suite
Instructions
close the top drawer of the cabinet
close the top drawer of the cabinet and put the black bowl top of it put the black bowl in the top drawer of the cabinet put the butter at the back in the top drawer of the cabinet and close it put the butter at the front in the top drawer of the cabinet and close it put the chocolate pudding in the top drawer of the cabinet and close open the bottom drawer of the cabinet open the top drawer of the cabinet open the top drawer of the cabinet and put the bowl in it put the black bowl on plate put the black bowl on top of the cabinet open the top drawer the cabinet put the black bowl the back on the plate put the black bowl at the front on the plate put the middle black bowl on the plate put the middle black bowl on top of the cabinet stack the black bowl at the front the black bowl in the middle stack the middle black bowl on the back black bowl put the frying pan on the stove put the moka pot on the stove turn on the stove
Kitchen
turn on the stove and put the frying pan on it close the bottom drawer of the cabinet
close the bottom drawer of the cabinet and open the top drawer put the black bowl in the bottom drawer of the cabinet put the black bowl on top of the cabinet
put the wine bottle in the bottom drawer of the cabinet put the wine bottle the wine rack
close the top drawer of the cabinet put the black bowl in the top drawer of the cabinet
put the black bowl on the plate put the black bowl on top of the cabinet
put the ketchup in the top drawer of the cabinet close the microwave put the yellow and white mug to the front of the white mug open the microwave put the white bowl on the plate
put the white bowl to the right of the plate put the right moka pot the stove
turn off the stove
Table 8: 40 Kitchen scene pretraining tasks
23

---

Published as a conference paper at ICLR 2024
Task Suite
Instructions
put both the alphabet soup and the tomato sauce in the basket put both the cream cheese box and the butter in the basket turn on the stove and put the moka pot on put the black bowl in the bottom drawer of the cabinet and close it put the white mug on the left plate and put the yellow and white mug on the right
Long-horizon (LIBERO 10)
plate
pick up the book and place it in the back compartment of the caddy put the white mug on the plate and put the chocolate pudding to the right of the plate
put both the alphabet soup and the cream cheese box in the basket put both moka pots on the stove put the yellow and white mug in the microwave and close it pick up the black bowl between the plate and the ramekin and place it the plate
Spatial
pick up the black bowl next to the ramekin and place it on the plate
pick up the black bowl from table center and place it on the plate pick up the black bowl on the cookie box and place it on the plate
pick up the black bowl in the top drawer of the wooden cabinet and place it on
the plate
pick up the black bowl on the ramekin and place it on the plate pick up the black bowl next to the cookie box and place on the plate pick up the black bowl on the stove and place it on the plate open the middle drawer of the cabinet
Goal
put the bowl on the stove put the wine bottle on top of the cabinet open the top drawer and put the bowl inside put the bowl on top of the cabinet push the plate to the front of the stove put the cream cheese in the bowl turn on the stove
pick up the alphabet soup and place it in the basket pick up the cream cheese and place it in the basket pick up the salad dressing and place it in the basket pick up the bbq sauce and place it in the basket pick up the ketchup and place it in the basket pick up the tomato sauce and place it in the basket pick up the butter and place in the basket pick up the milk and place it in the basket pick up the alphabet soup and put it in the basket pick up the butter and put it in the basket pick up the milk and put it in the basket pick up the orange juice and put it in the basket pick up the tomato sauce and put it in the basket pick up the alphabet soup and put it in the tray pick up the butter and put it in the tray pick up the cream cheese and put it the tray pick up the book and place in the right compartment of the caddy pick up the book and place in the front compartment the caddy pick up the book and place it in the left compartment of the caddy pick up the book and place in the right compartment of the caddy pick up the red mug and place it the right of the caddy
Object
Living Room
Study Room
pick up the white mug and place it to the right of the caddy pick up the book in the middle and place it on the cabinet shelf pick up the book on the left and place it on top of the shelf
Table 9: Adaptation task suites
24


Reviews:
[{"review_id": "r5LGGuWxKW", "reviewer": "Reviewer_q7Ls", "summary": "This paper introduces a new architecture named TAIL for imitation learning, which effectively adapts large pretrained models to new tasks with limited demonstration data. The main contribution of this method is to effectively incorporate lightweight adapter modules into pretrained models using various integration techniques, including parallel integration (with LoRA weights), sequential integration (Bottleneck Adapter), and prefix token integration. \n\nThe paper provides a comprehensive comparison of these adaptation techniques on the LIBERO robotic manipulation continual learning benchmark. The results indicate that TAIL, particularly when utilizing LoRA integration, outperforms the compared methods in terms of both forward and backward transfer.\n\nOverall, while this paper may not introduce strong technical novelty, it offers a good empirical study of lightweight continual learning techniques for imitation learning of robot control tasks.", "questions": "1. Is a separate adapter $\\omega_k$ trained for each task suite? My understanding is that when re-evaluating previous task suites $j$, the previously trained adapter $\\omega_j$ is reloaded instead of testing the previous task on the adapter $\\omega_k$ trained for the current task $k$. \n2. In Figure 4, it would be insightful to determine whether combining all integration techniques, including LoRA, Bottleneck Adapter, and Prefix Token, would yield further performance improvements.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The paper includes sufficient experiments on the LIBERO benchmark, which can effectively demonstrate the advantages of employing TAIL in conjunction with the LoRA integration technique.\n3. The proposed TAIL-based methods greatly outperform the conventional fine-tuning method in both forward transfer results and adaptation efficiency.", "weaknesses": "1. My primary concern is the technical novelty of this paper, although I acknowledge the significance of benchmarking and conducting an extensive empirical investigation of existing parameter-efficient fine-tuning techniques. The proposed TAIL framework, which incorporates lightweight adapters and task-specific heads, shares similarities with existing methods in continual learning and multi-task learning, as demonstrated in the paper from Rebuffi et al. (2017) titled 'Efficient Parametrization of Multi-Domain Deep Neural Networks.' Although TAIL introduces a new multi-modal, Transformer-based architecture, the core idea bears a strong resemblance to prior work.\n2. In Figure 4, the authors primarily compare various design choices within the TAIL framework when presenting forward adaptation results. It would be valuable if the authors could extend this analysis to include a comparison of TAIL's best performance with the state of the art in lifelong imitation learning. Furthermore, in Table 1, the comparison of TAIL's forward transfer performance with previous continual learning methods may not be fair enough. This is because EWC and ER primarily address catastrophic forgetting and may not contain task-specific model parameters. \n3. In Figure 5, a comparison between TAIL and EWC/ER may not be entirely equitable, considering that these prior methods do not assume known task IDs or retain task-specific model parameters. I would like to suggest the authors include a comparison with multi-task learning methods for a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8ABrL7GFB", "reviewer": "Reviewer_86yf", "summary": "The paper considers the continual imitation learning setting where a stream of tasks arrive one at a time. Existing methods are susceptible to catastrophic forgetting or loss of model plasticity as the number of tasks increase. Consequently, the paper proposes task-specific adapters for imitation learning (TAIL), a method that uses additional per-task tuneable parameters and synthesizes them to the pretrained base model's parameters. The paper demonstrates through experiments that TAIL introduces little computational overhead and mitigates catastrophic forgetting and model plasticity problems.", "questions": "- Under subsection **Training, Adaptation, and Evaluation**, what exactly is a validation scene?\n- Does ER retrain parameters from scratch, or does it continue training from the current set of parameters? Clearly if it is the latter the model will experience plasticity loss similar to the reinforcement learning setting [1].\n- It appears that some tasks are repeated based on appendix. I am wondering if there is any result regarding the per-task success rate. Do we understand whether the parameters are well-adapted to specific tasks, or are they similar in performance?\n\n**Possible typos**\n- On page 2, subsection PEFT, line 5: \"It is\" instead of \"it's\".\n\n**References**\n[1]: Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P. L., & Courville, A. (2022, June). The primacy bias in deep reinforcement learning. In International conference on machine learning (pp. 16828-16847). PMLR.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to follow generally.\n- The method is very intuitive and simple---the fact that the overhead is low for a new task while mitigating catastrophic forgetting (e.g. better than experience replay) is appealing.\n- The empirical analysis demonstrates that using TAIL with low-rank adaptation (LoRA) perform better than existing baselines convincingly in LIBERO, interestingly that it performs that well with rank $r = 8$.", "weaknesses": "**Comments**\n- It appears that the number of tasks can easily explode over time since each task instruction (and initial-state distribution) corresponds to a new task. A natural extension will be to mitigate the amount of adapters based on the similarity of task instructions, as hinted in appendix.\n- Since the problem setting indicates that a task definition is not w.r.t. the state-action space, it will be more convincing if there are experiments conducted on cross-embodiment (e.g. different arms or same arm with different inertial properties.)\n- The evaluation metric is unclear---in particular under BWT, the equation seems to be different from what is described---should the *best FWT model* be the best $F_i$ for task $i$ after seeing $k$ tasks?\n- For the 10 validation episodes, I believe it has been shown multiple times that validation error does not necessarily correspond to success rates [1, 2, 3]---a model may achieve high validation error while still achieving high success rate.\n- Regarding success rates, what are the baseline performances? That is, what is the success rate of the expert demonstrations? Do we expect the policy that trains purely using the task-specific data to perform better than the models in Figures 5 and 6? How would a random policy perform?\n\n**References**\n[1]: Hussenot, L., Andrychowicz, M., Vincent, D., Dadashi, R., Raichuk, A., Ramos, S., ... & Pietquin, O. (2021, July). Hyperparameter selection for imitation learning. In International Conference on Machine Learning (pp. 4511-4522). PMLR.  \n[2]: Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., ... & Martín-Martín, R. (2021). What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298.  \n[3]: Ablett, T., Chan, B., & Kelly, J. (2023). Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks. IEEE Robotics and Automation Letters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1r2sloHcQ", "reviewer": "Reviewer_FCxQ", "summary": "The paper considers the adaptation problem in continual imitation learning and proposes task-specific adapters when fine-tuning the imitation policy to new tasks. Specifically, the task-specific adapter is an add-on to the imitation policy and can be updated for each task without changing the pretrained weights in the backbone policy. As shown by the paper, this adapter can be implemented by three ways: parallel, i.e., Low-Rank Adaptation (LoRA) in this paper, sequential, i.e., Bottleneck Adapter, & prefix token, i.e., prompt-tuning. The paper evaluates the proposed adapter in LIBERO robotic manipulation continual learning benchmark and shows that LoRA adapter performs the best across all tasks.", "questions": "1. In appendix, “all methods share similar amount of parameters”. I’m not quite sure how to interpret this. Why all methods need to have a similar number of parameters? Even though the adapter integration mechanisms are different? Could this constraint on the number of parameters be biased towards to LoRA adapter? (since “filtering often requires a larger bottleneck size compared to that of LoRA, leading to more parameters”)\n\n2. In the experiment Table 1, “The BWT for TAIL methods are all 0 (no catastrophic forgetting)”. Is it self-evident? since the TAIL methods produce an ensemble of models (fixed pretrained weights + task-specific adapter weights), each of which aims to solve a dedicated task and is updated independently to each other. Then why do we need to use BWT as a performance indicator. \n\n3. The colours for Prefix Token & Frozen cannot be easily distinguished in Figure 2.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "### Clarity \nThe structure and presentation of the paper is of very good quality. The flow of the paper is excellent with a good and appropriate structure. The underlying motivation of studying the task-specific adapters is well justified.", "weaknesses": "### Quality & Contribution\nOne of the main weaknesses of this paper is the lack of comprehensive empirical analysis over multiple different benchmarks, especially those used by previous baseline methods. Particularly, **the conclusions and empirical insights were drawn only from a single dataset, with no results from any other public datasets**. As the main contribution, this paper shows that the LoRA-type integration of the adapter in continual imitation learning performs the best. The paper thus concludes that they “are contrary to many results from the vision and language model literature which show that full fine-tuning works better”. However, these results are drawn based on LIBERO robotic manipulation continual learning benchmark ONLY. They have not been verified by any other datasets. It is thus unclear if and how the results and conclusions can be generalized to other robot control tasks, e.g., Franka-Kitchen. \n\nFurthermore, it is unclear if the baseline methods have been well tuned for the LIBERO benchmark. For example, the RoboAdapter method may need to tune the adapter locations and different pre-trained representations for its optimal performance. The paper, however, seems to take the default configs, even though those configs were originally proposed for other robot tasks. To have a fair comparison with RoboAdapter, the paper should consider some of datasets used by RoboAdapter paper, i.e., Metaworld, Franka-Kitchen, and RGB-Stacking task suites, replicate the reported performance of RoboAdapter and then demonstrates on at least one of those datasets that the TAIL performs better than RoboAdapter. \n\n### Originality & Significance\nThe originality of this paper can be limited and incremental. \n\n**In terms of technique**, the adapter idea has been well studied by the RoboAdapter paper, which introduces the adapter layer (Sequential Integration in this paper) to imitation learning and considers its application in robot controls. Though the paper proposes two other adapter mechanisms, LoRA & Prefix prompt tuning, both of them are taken directly from other papers, without any substantial modifications or creative combinations to the continual imitation learning or robot control tasks. \n\n**In terms of the empirical results**, they may not offer any in-depth understanding of using adapters in continual imitation learning and the significance can be limited. The paper mentions that TAIL with LoRA “avoiding catastrophic forgetting and preserving adaptation plasticity”. Avoiding catastrophic forgetting can be evident since the adapter weights are task-specific and not shared across different tasks. But the empirical results shed little light on the adaptation plasticity. It is unclear if and how the adaptation plasticity is preserved given that the empirical results focus only on the success rate, as in Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkyoXQVOea", "reviewer": "Reviewer_JmE1", "summary": "The authors applied parameter-efficient fine-tuning methods, including bottleneck adapters, P-tuning, and low-rank adaptation, to the continual imitation learning task. TAIL is designed to adjust large pre-trained models for new tasks even with limited demonstration data, with GPT2 as the backbone and CLIP-based modules as encoders. It avoids issues like catastrophic forgetting by activating the corresponding adapter for that task.", "questions": "* Could you show the results when setting the rank for LORA to 32, 256, 512, and 768 (the dimensionality of the embeddings and hidden states of GPT2)? The ablation studies regarding the number of parameters should be added.\n\n* Could you provide some insights on whether combining the Lora and P-tuning can improve the performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper rigorously tests TAIL with prevalent PEFT techniques on the imitating continual imitation learning task.\n\n* TAIL’s ability to achieve superior post-adaptation performance using only 1% of the trainable parameters of full fine-tuning highlights the framework’s efficiency, making it a potentially valuable tool for resource-constrained settings.", "weaknesses": "* The framework's capability to circumvent catastrophic forgetting is primarily attributed to its unique configuration that allows for the selection of adapters tailored to individual tasks. Its efficiency largely draws from existing PEFT methodologies, which limits the novelty and distinctiveness of the presented framework.\n\n* Drawing a comparison between TAIL and the fine-tuning baseline methods cited in the paper is unfair. This is because TAIL is designed to learn dedicated adapters for each distinct task. Thus, the claims regarding its efficacy become less compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfwqtu8g30", "reviewer": "Reviewer_Lua3", "summary": "The work presents Task-specific Adapters for Imitation Learning (TAIL), a framework for fine-tuning control policies in continual learning setting by borrowing ideas from large language model space. The agent is trained on multiple tasks, one task at the time, and the objective is to learn the new tasks while retaining the performance in old tasks. TAIL algorithms update only a small set of parameters, inspired by parameter efficient training from language model space. The results show that TAIL combined with LoRA outperforms all tested alternatives, especially full finetuning (tuning the whole network) or other methods that fine-tune the whole network.", "questions": "1) Page 7, \"Training, Adaptation and Evaluation\" paragraph: Manuscript says \"This limited data setup...\". How do you consider this a \"limited data\" setup? Can you give context/examples what would not be \"data limited\" setup?\n\n2) Page 7, last paragraph. The description of what parts were finetuned/update/adapted and when is somewhat unclear and involved. How did you come up with this setup? Could you clarify this paragraph or provide a figure to help with understand what was updated in which stage?\n\n## Comments\n- Define acornyms only once and consistently use the acornym ever since (e.g., TAIL is defined multiple times).\n\n\n## Rebuttal update 20th Nov\n\nI have read authors' answers to my questions, and I have kept my original review score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Overall nicely wrapped work. While not the most novel solution (see weaknesses), I feel the amount of results and experiments conducted here merit acceptance. I believe these results will be useful for many readers, and the instructions on how to fine-tune these models will support future work.\n\n### Originality\n\nAuthors bring insights from language space to control space in the form of efficient finetuning algorithms, and test those algorithms in a new setting (continual learning).\n\n### Quality\n\nExperiments are throughout with enough ablations and baselines. The methods are evaluated from different angles (performance increase in new task, retainibility in previous tasks, number of parameters / computational requirements.)\n\nAccurate description of the experiment setup provided in the appendix (e.g., hardware, driver versions).\n\n### Clarity\n\nThe manuscript is written well and is clear to read.\n\n### Significance\n\nThe insights are also helpful for simple fine-tuning of models for different tasks: results show that LoRA is also very fast to adapt in general (better than full finetuning).\n\nThe manuscript also provides a list of ablations on what settings are recommended for finetuning, which are useful for people to follow.\n\nGiven the popularity of transformer models and their adaptation in different domains, I see this work being useful for many readers. Focus on experiments that can be ran on lesser hardware (e.g., single GPU, less space) also allows more people to experiment with these features.", "weaknesses": "- Using a separate set of weights (\"adapters\") per task feels bit of cheating when comparing the model against the baselines. This alone ensures that you keep high performance in the previous tasks. While the argument for this is valid (with TAIL, you only need a handful of parameters vs. full network), I'd prefer a more apples-to-apples comparison.\n- Limited to only one environment. While I understand data for right setup is hard to come by, it is hard to tell if the results generalize outside this tested environment and data. Showing results in an another type of environment would solidify these insights.\n- (Minor) Proposed framework, despite having its own name, is not especially novel or specific: it is an umbrella term for existing solutions applied on the continual learning setting. This is evident from figures, where the same method (TAIL) appears multiple times, and fundamentally, all these variants are significantly different from each other. I'd perhaps call different instiations of this setup \"TAIL-LoRA\", instead of \"TAIL (LoRA)\", to signify it is LoRA applied in TAIL's fashion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)