Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptation

Authors: daehee  lee
minjong  yoo
♠  woo
kyung  kim
wonje  choi
honguk  woo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:27:37

Abstract: Continual Imitation Learning (CiL) involves extracting and accumulating task knowledge from demonstrations across multiple stages and tasks to achieve a multi-task policy. With recent advancements in foundation models, there has been a growing interest in adapter-based CiL approaches, where adapters are established parameter-efficiently for tasks newly demonstrated. While these approaches isolate parameters for specific tasks and tend to mitigate catastrophic forgetting, they limit knowledge sharing among different demonstrations. We introduce IsCiL, an adapter-based CiL framework that addresses this limitation of knowledge sharing by incrementally learning shareable skills from different demonstrations, thus enabling sample-efficient task adaptation using the skills particularly in non-stationary CiL environments. In IsCiL, demonstrations are mapped into the state embedding space, where proper skills can be retrieved upon input states through prototype-based memory. These retrievable skills are incrementally learned on their corresponding adapters. Our CiL experiments with complex tasks in the Franka-Kitchen and Meta-World demonstrate the robust performance of IsCiL in both task adaptation and sample-efficiency. We also show a simple extension of IsCiL for task unlearning scenarios.

Main paper: 
 introduction: 
 Lifelong agents such as home robots are required to continually adapt to new tasks in sequential decision-making situations by leveraging knowledge from past experiences. However, many realworld domains pose substantial challenges for these lifelong agents; the complexity and ever-changing nature of these tasks make it difficult for agents to constantly adapt, leading to difficulties in retaining knowledge and maintaining operational efficiency [1] . For instance, a home robot agent, operating within a single household, needs to continuously adapt, learning specific tasks in various areas such as cooking assistance in the kitchen or cleaning in the bathroom. At the same time, it is crucial that the agent not only retains but also improves its proficiency in the tasks it has previously learned, ensuring that it maintains consistent efficiency throughout the home.For these lifelong agents, Continual Imitation Learning (CiL) has been explored, in which an agent progressively learns a series of tasks by leveraging expert demonstrations over time to achieve a multi-task policy. Yet, CiL often encounters practical challenges: (1) the high costs and inefficiencies associated with comprehensive expert demonstrations [2] that are required for imitation, (2) frequently shifting tasks in dynamic, non-stationary environments, and (3) privacy concerns [3] related to learning from expert demonstrations. In this context, CiL faces significant issues in terms of cost, adaptability, and privacy, complicating its implementation in real-world scenarios. 
related work: 
 Continual imitation learning. To tackle the problem of catastrophic forgetting in continual learning, numerous studies have employed rehearsal techniques [8, 9, 10, 11] , which involve replaying past experiences to maintain performance on previously learned tasks. Another approach involves utilizing additional model parameters to progressively extend the model architecture [12, 13, 14, 15, 16] . These methods adapt the model's structure over time to accommodate new tasks. However, rehearsal techniques exhibit high variability in forgetting depending on the replay ratio and often demand substantial training to incorporate new knowledge [17] . Progressive models, on the other hand, require stage identification during evaluation and often overlook unseen tasks [13] . In this work, we propose a CiL framework that enables effective learning and expansion without requiring rehearsal and stage identification, leveraging pre-trained goal-based model knowledge.Continual task adaptation with pre-trained models. Several recent works use pre-trained models, accumulating knowledge continually through additional Parameter Efficient Tuning (PET) modules such as adapters [18, 17, 19, 20, 21, 6, 22] . These methods enhance the flexibility and scalability of continual learning systems. However, they suffer from inaccurate matching between adapter selection and trained knowledge, leading to a misalignment between the knowledge learned during training and the knowledge used during evaluation [17, 20] , which hinders overall performance. In the realm of sequential decision making, some studies have explored adapting pre-trained models. In [6] , the state space of tasks is fully partitioned, restricting its applicability in more integrated environments. Meanwhile, [7] relies on comprehensive demonstrations for learning, which may be impractical in real-world scenarios. Our study aims to enhance task adaptation efficiency by using incrementally generalized skills with accurate matching on state space.Skill adaptation. Reinforcement learning research has enhanced fast adaptation through skill exploration [23] and skill priors [24] , focusing on improving sample efficiency with offline datasets. Despite these advancements, adapting fixed skill decoders to new environments remains challenging. To overcome these limitations, skill-based few-shot imitation learning methods have been developedFigure 1 : The scenario demonstrating how IsCiL enhances continual imitation learning efficiency through retrievable skills: (i) Prototype-based skill incremental learning: despite the failure of τ 1 , skills are incrementally learned from the available demonstrations. In later stages, missing skills for τ 1 are retrieved from other tasks, achieving the resolution of τ 1 and illustrating the reversibility and efficiency of retrievable skills. (ii) Task-wise selective adaptation: IsCiL effectively retrieves relevant learned skills, facilitating rapid task adaptation. [25, 26] . However, these methods require extensive past data and struggle with scalability and generalization. Even skill-based approaches used in continual imitation learning [8] still require rehearsal data to mitigate knowledge loss and face difficulties addressing privacy issues through unlearning. Our IsCiL employs parameter-efficient skill adapters to prevent catastrophic forgetting and maintain efficiency, providing a scalable solution for unlearning. 
approaches: 
 Our work addresses three key challenges of CiL: (1) data inefficiency, (2) non-stationarity, and (3) privacy concerns, by adopting retrievable skills in the CiL context. Specifically, our IsCiL framework not only enhances data-efficient continual task evaluation in a non-stationary environment but also supports unlearning as a task adaptation strategy, thereby mitigating privacy concerns.In CiL scenarios, we consider a data stream of task datasets {D i } p i=1 , where D i contains an expert demonstration D i = {d 1 i , ..., d N i } for its associated task τ i . To effectively represent complex longhorizon tasks, each task τ i is comprised of sub-goal list, τ = {g 1 i , ..., g M i }. Each task dataset is sampled in a finite-horizon markov decision process (S, A, P, R, µ 0 , H), where S is a state space, A is a action space, P is a transition probability, R is a reward function, µ 0 is an initial state distribution, and H is an environment horizon.For demonstration d = {(s t , a t )} H t=1 , a state s t ∈ S represents a tuple (o t , g t ) consisting of an observation o t and a sub-goal g t . In our work, we represent sub-goals through language and use language-based goal embeddings for g t to achieve language-conditioned policies. Then, the objective of IsCiL is to obtain a multi-task policy π * , by which the performance on the tasks in the data stream can be comparable to that of respective expert policies. This is formulated asEQUATIONwhere πτ represents an expert policy for τ and T i denotes a set of evaluation tasks at stage i. In this context, the evaluation tasks continuously vary across different stages.To effectively handle complicated CiL scenarios, we present the IsCiL framework which involves (i) prototype-based skill incremental learning and (ii) task-wise selective adaptation.As illustrated in Figure 1 , in (i) the prototype-based skill incremental learning, we use a two-level hierarchy structure with a skill retriever π R composing the skills for each sub-goal, and a skill decoder π D producing short-horizon actions based on state-skill pairs. For this two-level policy hierarchy, we employ a skill prototype-based approach, in which skill prototypes capture the sequential patterns of actions and associated environmental states, as observed from expert demonstrations. These prototypes serve as a reference for skills learned from a multi-stage data stream. Using these skill prototypes, we can effectively translate task-specific instructions or demonstrations into a series of appropriate skills.Through this prototype-based skill retrieval method, the policy flexibly uses skills that are shareable among tasks, potentially learned in the past or future, for policy evaluation. This enables the CiL agent to effectively learn diverse tasks and rapidly adapt to variations, while incrementally accumulating skill knowledge from a multi-stage data stream. Furthermore, to facilitate sample-efficient learning and enhance stability in CiL, we employ parameter-efficient adapters that are continually fine-tuned on a base model. Each skill knowledge is encapsulated within a dedicated adapter and incorporated into the skill decoder π D to infer expert actions.In (ii) the task-wise selective adaptation, we devise efficient task adaptation procedures in the policy hierarchy to adapt to specific tasks using incrementally learned skills. This enables the CiL agent to not only facilitate adaptation to shifts in task distribution (e.g., due to non-stationary environment conditions) but also support task unlearning upon explicit user request (e.g., due to privacy concerns).Suppose that the smart home environment undergoes an upgrade with the installation of new smart lighting systems throughout the house. In this case, task-wise selective adaptation can be used for rapid adaptation by removing outdated control routines associated with the previous systems.State encoder and prototype-based skill retriever. To facilitate skill retrieval from demonstrations, we encode observation and goal pairs (o t , g t ) into state embeddings s t using a function f : (o t , g t ) → s t . We implement f as a fixed function to ensure consistent retrieval results for learning efficiency, mitigating the negative effects of input distribution shifts.To effectively handle the multi-modality of the state distribution in non-stationary environments, we employ a skill retriever π R . For this, we use multifaceted skill prototypes χ z ∈ X , where X is the set of learned skill prototypes. These prototypes capture the sequential patterns of expert demonstrations associated with specific goal-reaching tasks.θ z = π R (s t ; X ) = h argmax χz∈X S(χ z , s t ) , where S(χ z , s t ) = max b∈χz sim(b, s t ) (2)Here, h : χ z → θ z denotes a one-to-one function that maps each skill prototype χ z to its dedicated adapter parameters θ z , while the similarity function S is defined as the maximum similarity between state s and bases b ∈ χ z . Each χ z consists of multiple bases (e.g., 20 bases), and each basis b is a representative vector containing its corresponding centroid, shaped identically to the state s t .Adapter conditioned skill decoder. To effectively use the knowledge of the pre-trained base model without forgetting, even in a non-stationary changing environment, the skill decoder is conditioned based on parameters. The skill decoder policy π D (â t |o t , g t ; θ pre , θ z ) operates with the skill adapter parameters θ z and the pre-trained base model θ pre , using the Low-Rank Adaptation [27] .Skill incremental learning. To incrementally learn new retrievable skills, we update the skill prototype and adapter pair (χ z * , θ z * ) for a novel skill z * . The skill prototype χ z * is created by dividing a dataset of a single skill into several clusters based on similarity. From each cluster, a representative value is extracted to serve as the basis b, representing z * . We use the KMeans algorithm [28] to determine these bases, ensuring that the number of bases |χ z | adequately captures the diversity within the dataset of the novel skill. This multifaceted set of bases allows the skill prototype to capture an accurate multi-modal distribution of the skill represented in the state space, enabling effective retrieval as described in Eq. 2. In our experiment, z * is created for each sub-goal g in the given dataset D i for each stage i. The learning of the skill adapter is divided into two phases: initialization and update. During the initialization phase, θ z * is initialized using existing skill adapters. Predictions with the existing skill dataset and skill prototypes χ z ∈ X are used to identify the most frequently selected skill. Average scores are computed for each skill prototype using the dataset involved in training z * , as defined in Eq. 2. The skill with the highest average score, denoted as z, is selected. Consequently, θ z * is initialized by θ z . Then, the initialized adapter is updated through the following imitation loss.EQUATIONThe novel skill z * is incorporated into the learned prototypes X ← X ∪ χ z * , and the novel prototype and adapter pair (χ z * , θ z * ) updates the function h for pair mapping. Figure 2 presents an overview of this methodology, along with the algorithm for incremental learning is detailed in Appendix B.1.Task evaluation. Given the pre-trained model θ pre and learned skill prototypes X , for given inputs (o t , g t ) from the environment, IsCiL performs the following evaluation process.EQUATIONThe evaluation process adapts to novel tasks and sub-goal sequences from the environment by modifying the goal g t . This adjustment enables the inference of appropriate current actions, in a manner of similar to handling learned tasks. For example, a kitchen robot tailored to a specific user's kitchen setup can continuously and instantly adapt to changes in recipes without additional training.Task unlearning. To ensure privacy protection for incrementally learned skills, our architecture allows for task unlearning by removing task-specific skill prototypes and adapters. In IsCiL, the separation of skill adapters for each task facilitates easy tagging of task information on each skill. When an unlearning request is given with a task identifier τ , the corresponding skill prototypes and adapters are removed. This approach ensures exceptionally efficient and effective unlearning, aligning with the strong unlearning strategies in continual learning discussed in [3] . 
environments and data streams: 
 To investigate the sample efficiency and adaptation performance, we construct complex CiL scenarios using diverse long-horizon tasks [29, 30, 31] . We then analyze the sample efficiency across different stages and tasks with three types of scenarios: Complete, Semi-complete, and Incomplete, depending on how the samples are utilized and shared. Each scenario consists of a pre-training stage followed by 20 CiL stages. Figure 3 illustrates these scenarios.Evolving Kitchen. Evolving Kitchen is a data stream based on long-horizon tasks in the Franka-Kitchen environment [29, 30] . Each task requires sequentially achieving four out of seven sub-goals.The scenario consists of a pre-training stage in the environment with only four objects: kettle, bottom burner, top burner, and light switch, followed by continual adaptation to tasks involving seven objects. In Semi-Complete, the first 10 stages are repeated twice, with tasks presented alongside incomplete demonstrations, where specific sub-goals are missing from the trajectories. In Incomplete, the same sequence of tasks from the Complete scenario is used, but all stages feature incomplete demonstrations, requiring the system to handle tasks with missing sub-goal trajectories.Evolving World. Evolving World is a data stream based on the Meta-World environment [31] with long-horizon tasks, similar to [32, 33, 34] . Each task requires sequentially achieving four out of eight sub-goals. The scenario consists of a pre-training stage in the environment with only four objects, followed by continual adaptation to an entire environment with all eight Meta-World objects. More detailed configurations are provided in Appendix A.Baselines. We implement continual imitation learning and continual adaptation methods for sequential decision-making problems, which do not use rehearsal. First, we consider continual learning algorithms which involve full-model updates (Seq, EWC [35] ). We also implement several continual adaptation approaches that utilize pre-trained models with adapters (L2M [6] , TAIL [7] ). L2M learns a key and adapter pair to modulate the pre-trained model, where the key is a retrievable state embedding similar to our prototypes. TAIL, unlike L2M, incrementally constructs task identifiers and corresponding adapters to modulate the pre-trained model with new task data without forgetting previous tasks. Each method is categorized based on the values used for adapter retrieval: a version that uses no additional identifiers, sub-goal identifiers (denoted as -g), and whole sub-goal sequences as single identifiers (denoted as -τ ). Additionally, we include a Multi-task learning approach as an oracle baseline, which retains all incoming data at each stage and utilizes it for training in subsequent stages. For all baselines, we use the same pre-trained goal-conditioned policy and a diffusion model [36, 37] as the base policy architecture. A detailed description of the baselines and their hyperparameters are provided in Appendix B.2.Metrics. We use three metrics to report CiL performance: Forward Transfer (FWT), Backward Transfer (BWT), and Area Under Curve (AUC) [38, 7] . In our long-horizon tasks, these metrics rely on goal-conditioned success rates (GC), which measure the ratio of successfully completed sub-goals to the total sub-goals within each task [39] .• FWT (Forward Transfer): This evaluates the ability to learn tasks using previously learned knowledge. It is measured by the performance of a task when it occurs. • BWT (Backward Transfer): This evaluates the impact of each learning stage on the performance of tasks learned in previous stages. It measures the change in task performance from past stages observed in the current stage. • AUC (Area Under Curve): This represents the overall continual imitation learning performance in a scenario. It measures the average performance of tasks learned in the current stage over the remaining stages of the scenario.For all metrics, higher values indicate better performance, with details provided in Appendix B.3. Table 1 shows the CiL performance on Evolving Kitchen and Evolving World across three different scenarios (Complete, Semi, Incomplete). We compare the performance achieved by our framework IsCiL and other baselines (L2M, TAIL) with different conditioning values (g,τ ) for adapter retrieval. IsCiL consistently demonstrates superior performance in AUC across all scenarios, achieving between 84.5% and 97.2% of the oracle baseline (Multi-task learning). TAIL-τ shows the most competitive performance in the Complete CiL scenario across both environments. However, due to its isolated adapter for learning and evaluation, it fails to effectively utilize samples across stages.L2M and L2M-g exhibit relatively lower and less stable AUC in the Evolving Kitchen scenario. Conversely, in Evolving World-Semi, they surpass TAIL-τ in AUC. This demonstrates that they are capable of sharing different skills across stages. Despite this, they still struggle with accurately retrieving the correct skill or suffer from performance degradation due to knowledge overwriting. Unlike them, IsCiL effectively mitigates overwriting by maintaining distinct skill representations across stages. Both L2M-g and TAIL-g, which aim to leverage sub-goal labels for CiL, struggle to maintain performance due to skill distribution shifts, leading to catastrophic forgetting of skills for sub-goals. These challenges reveal that relying solely on sub-goal labels may not be sufficient to sustain and share skills effectively across different stages and tasks.Both Seq-FT and Seq-LoRA struggle with forgetting. This is evident in the Complete scenario, where Seq-FT achieves the highest FWT but shows the lowest BWT, leading to a decline in overall performance. EWC exhibits consistently lower performance, as the regularization used to preserve past knowledge significantly hinders learning on current tasks, leading to severe degradation in long-horizon tasks. Although EWC shows higher BWT compared to other sequential tuning baselines, its low FWT limits overall effectiveness. Table 2 shows the unseen task adaptation ability of IsCiL, where only the sub-goal sequence of novel task is provided without demonstrations. This scenario extends the existing Complete CiL scenarios by periodically introducing novel tasks. Metrics labeled with the suffix -A indicate results from adaptation tasks, whereas the other metrics reflect performance on all tasks. For this scenario, we exclude TAIL-τ from comparison, as it lacks the ability to adapt to novel tasks.IsCiL demonstrates superior task performance in both scenarios, which contributes to greater efficiency in task adaptation. Moreover, in Evolving Kitchen, IsCiL not only demonstrates task adaptation ability by achieving the highest FWT-A, but also significantly enhances its initial performance, raising FWT-A from 52.1 to an AUC-A of 72.8. TAIL-g shows comparable performance in FWT for the Evolving Kitchen. However, it struggles with catastrophic forgetting, leading to a -34.9 negative BWT when faced with significant distribution shifts in sub-goal demonstrations. In Evolving World, L2M, which actively learns to share skills during training, outperforms TAIL-g. L2M is the only baseline achieving performance improvement on unseen tasks through CiL.Table 3 measures CiL performance in scenario with task-level unlearning. For comparison, we use an adapter-based approach with parameter isolation-based continual learning private unlearning (CLPU) [3] , extending TAIL to TAIL-τ CLPU and IsCiL without skill adapter initialization. Similar to IsCiL, CLPU learns tasks in isolated models tagged with specific task identifiers and handles unlearning requests by removing the corresponding model parameters of the target task. Both TAIL-τ CLPU and IsCiL ensure output distribution equality between the unlearned model and the model trained with the retained dataset. Thus, their CiL performance remains largely unaffected by unlearning.Although IsCiL exhibits a slight performance degradation of 1.8% ∼ 5.2% after unlearning, as reported in Table 1 , it still demonstrates robustness by achieving a 115% higher AUC compared to TAIL-τ CLPU in incomplete scenarios.Rehearsal comparison. Figure 4 compares the sample efficiency to retain learned knowledge between IsCiL and a rehearsal-based continual imitation learning approach, Experience Replay (ER) [40] . For ER, we adjust the number of stored samples per learning stage, while IsCiL does not store rehearsals for training. IsCiL achieves the highest AUC in all environments and is the only approach where AUC surpasses FWT. ER shows comparable FWT in Complete, but as the number of stored samples increases, FWT decreases, indicating that more rehearsals actually reduce training sample efficiency.In Semi and Incomplete, using 250 rehearsals (approximately 5% of the stage dataset) yields FWT comparable to IsCiL but rarely improves AUC.Limited training resource. Figure 5 shows the computational efficiency of IsCiL in resourceconstrained training settings, as discussed in [38] . In this experiment, the training resources is limited to 1% to 50% of those used in Table 1 . IsCiL and TAIL-τ show robust performance for varied training resources. TAIL-g shows higher FWT, as it trains the same sub-goal data on the same adapter, which excels in learning new tasks, but it fails to retain that knowledge. However, using skill data from different stages to update the same adapter makes it vulnerable to skill distribution shifts in CiL; this ends up with significant AUC degradation.  
conclusion: 
 In this study, we presented the IsCiL framework to address key challenges in continual imitation learning (CiL). Our approach incorporates adapter-based skill learning, leveraging multifaceted skill prototypes and an adapter pool to effectively capture the distribution of skills for continual task adaptation. IsCiL specifies enhanced sample efficiency and robust task adaptation, effectively bridging the gap between adapter-based CiL approaches and the need for knowledge sharing across staged demonstrations. Comprehensive experiments demonstrate that IsCiL consistently outperforms other adapter-based continual learning approaches in various CiL scenarios.Limitations. Like other adapter-based CiL approaches, IsCiL requires extra computation for evaluation, which can create overhead, especially in resource-constrained environments. It also depends on sub-goal sequences for training and evaluation, adding complexity and resource demands. Another limitation is determining the appropriate size of the adapter parameters, which depends on the performance of the pre-trained base model and the degree of task shift, making optimal adaptation challenging. Moreover, balancing the stability of the embedding function with the prototype size remains an area that requires further refinement to achieve optimal performance.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Yes, we provide the codes for supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the papers for the datasets and assets used. [29, 30, 31] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Franka KitchenWe conduct experiments on the Franka kitchen environment [29, 30] . Each Franka kitchen task comprises of 4 sub-goals, from total pool of 7: microwave, kettle, bottom burner, top burner, light switch, slide cabinet, and hinge cabinet. Observation is a 60-dimensional vector, which is a combination of the positions and velocities of 7-DoF robot arm and interacting objects. We express sub-goal information using language embedding. The target sub-goal of the current state is acquired by a pre-defined environmental reward and task, and a sub-goal sequence to solve. We use 24 tasks in the 'mixed' dataset from the D4RL [29] . In the pre-training stage, we train the model only on tasks comprised of following four sub-goals: kettle, bottom burner, top burner, light switch. We conduct experiments on the multi-stage variation of the Meta-World environment [31, 34] . Each Meta-World task comprises of 4 sub-goals from total pool of 8: puck, box, handle, drawer, lever, button, door, and stick. The environments are divided into different scenarios based on which 4 out of 8 objects are placed on the table. For each environment, tasks are defined according to the sequence in which the 4 sub-goals must be achieved. Observation is a 140-dimensional vector, which contains the positions and velocities of the 4-DoF robot arm and all interacting objects in the environment. In this environment, sub-goal information is also expressed using language embedding. The expert dataset is collected using a heuristic expert policy provided by Meta-World [31] . In the pre-training stage of Meta World, we train the model on 24 tasks on a environment consisting of four objects: a puck, a drawer, a button, and a door.Evolving Kitchen Tables 5 and 6 display the detailed configurations of the Evolving Kitchen in our CiL scenario. Each task involves sequentially solving its respective sub-goals. The underlined sub-goals (e.g., kettle) are those missing in the Semi Complete and Incomplete scenarios.Evolving World Tables 7 and 8 display the detailed configurations of our Evolving World CiL scenario. Similarly, the Evolving World is also presented in the same way as the Evolving Kitchen.Unseen Task Adaptation Tables 9 and 10 show the detailed configurations of unseen tasks for our Evolving Kitchen-Complete Unseen and Evolving World-Complete Unseen. In the Evolving Kitchen, 2 new tasks appear every 5 stages. In the Evolving World, 4 new tasks appear every 4 stages. Each new task includes only the sequence of sub-goals that must be completed in order, without any demonstrations. Unlearning Scenario In the Unlearning Scenario, 1 learned task is unlearned every 5 learning stages.In the Evolving Kitchen Unlearning scenario, τ 4 , τ 8 , τ 13 , and τ 17 are sequentially unlearned.IsCiL consists of two modules: a skill retriever, π R , and a skill decoder, π D . The skill retriever π R includes three components: a state encoder f , skill prototypes X , and a skill adapter mapping function h. Each skill prototype χ z in X is composed of 20 bases b. To modulate skill decoder π D , we use Low Rank Adaptation(LoRA) [27] . In our experiment, we used 4-rank LoRA adapters for skill adapter. IsCiL training and evaluation process follows : for each sub-goal g in task dataset D i do 6: χ z ← Mode(X g ) // select the most frequently retrieved skill prototype from the set 10: θ z * ← h(χ z ) // map the selected skill prototype χ z to its skill adapter via h 11:D g i ← {(o, g ′ ) ∈ D i | g ′ = g} //Update θ z * using Eq. (3) // update the skill adapter based on task-specific learning 12:X ← X ∪ χ z * // append the new skill prototype to the skill set for future retrieval 13: Update the mapping function h to map χ z * to the updated adapter θ z * // update h with the new skill adapter θ z = π R (s t ) // retrieve skill 6:ât ∼ π D (o t , g t ; θ pre , θ z ) // decode the skill 7: end whileSeq-FT & Seq-LoRA Sequential Fine Tuning(Seq-FT) is a method that updates the entire model sequentially. The variation, Seq-LoRA, is used to determine how effectively the fixed pre-trained model can utilize its knowledge. Due to poor performance at very low ranks, Seq-LoRA was trained with a 64-rank adapter in our environment.EWC [41] Elastic Weight Consolidation (EWC) regularizes the weight update by using the Fisher information matrix for each network parameter. For our experiment, we adopted the online version of EWC, which updates the Fisher information at each stage by exponential moving average, following the methods in [38, 7] .We use the hyperparameter alpha, set 10, to determine the regularization strength. For updating the online Fisher information matrix Fi , we use the Fisher information matrix calculated at the current stage and apply the following formula for regularization: Fi = γF i-1 + (1 -γ)F i , where γ is set to 0.9.L2M [6] L2M is an adapter-based continual learning method consisting of keys and their corresponding adapters. When an input is provided, L2M uses a similarity function to search for the key corresponding to that input. Input is converted to a query and utilized to search for a key, where key is a vector with the same shape as the query. Each key is then updated to maximize its similarity to the data point associated with it. Finally, the data is used to update the adapter that corresponds to the key value found through that data. This method maximizes the diversity of key usage frequency by adjusting the similarity between keys and input values during the training phase for learning new tasks [18, 6] . In our implementation, we use the normalized state value as the input query to find the key in L2M. For L2M-g, we use the normalized embedding of the state concatenated with the given conditioned sub-goal information directly as a query. Our adapter pool consists of 100 adapters, each being a 4-rank LoRA adapter. TAIL [7] TAIL directly assigns an adapter to the given task using the task's identifier. We directly map the given identifier to the corresponding adapter. TAIL-g uses a 4-rank adapter, while TAIL-τ uses a 16-rank adapter.Multi-task At each stage, the model learns from the given data and stores all data for the next stage. The data stored in the buffer is mixed with the data from each stage in a 1:1 ratio for training the model.ER [40] Experience Replay (ER), similarly, retains knowledge by storing a subset of the current stage's data for the next stage. The data stored in the buffer is mixed with the data from each stage in a 1:1 ratio for training the model.CLPU [3] Continual Learning Private Unlearning (CLPU) [3] is a method for managing continual learning and unlearning. In a continual learning scenario, tasks that require maintenance are trained using existing models, while data that may require unlearning is trained on independent model parameters, tagged with when and through which task each model was trained. When an unlearning request for a specific task or training stage is received, the corresponding model parameters are completely removed to eliminate the influence of the target unlearning task from the model. CLPU provides highly efficient and powerful unlearning performance with a single delete operation for tasks learned in a continual learning scenario. In our experiment, we integrate the unlearning approach CLPU with TAIL-τ , which conducts training through task information-based searches, to handle unlearning requests in continual imitation learning as a comparative method.We report 3 metrics for CiL performance for tasks: Forward Transfer(FWT), Backward Transfer(BWT), Area Under Curve(AUC) [38, 7] . In multi-stage environment task, we report performance using the goal-conditioned success rates (GC), which evaluate the average success rate of successfully completed sub-goals out of N sub-goals in the task.• FWT:FWT τ = 1 |Iτ | i∈Iτ C τ,iwhere τ is task and C τ,i represents the GC score of task τ at stage i. and I τ is set of stage indices where task τ is trained in the CiL scenario.• BWT:BWT τ = 1 |Iτ | i∈Iτ 1 p-i-1 p j=i+1 (C τ,j -C τ,i ), where p is the final stage at which task τ is available. In the case where p = i BWT is 0.• AUC: AUC τ = 1 |Iτ | i∈Iτ 1 p-i p j=i C τ,j, represent the the overall performance of continual learning, internally including FWT and BWT. In the case where p = i, AUC τ is FWT τ .The final reported metric is the average across all tasks τ ∈ T . For all metrics, higher values indicate better performance.Pre-trained Base Model and Stage Settings Table 11 shows the hyperparameters and the architecture of the model we used as the base model for all baselines. Table 12 shows the common hyperparameters used to train the model for each stage in our experiments. Pre-trained model performance Table 13 shows the learning performance of the pre-trained model and its adaptation performance for tasks learned in scenarios without any prior training. In Evolving World, where new objects are added and the environment changes significantly, the pre-trained model failed to successfully complete any sub-goals of tasks. 8 provides a comprehensive visualization of the Skill Retriever in the LIBERO-goal scenario. The visualization highlights how the retriever successfully identifies and shares skills across different stages of CiL. This demonstrates its adaptability in handling varied states and tasks, showing its potential effectiveness even in complex LIBERO environments. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This submission introduces a method for continual imitation learning by prototype-based skill incremental learning. Low-level skills are represented by discrete prototypes (like basis functions), and their parameters, together with observations and goals, are passed into a decoder for action execution. The main contribution is that the proposed method eliminates the need for episodic replay, offering improved forward and backward transfer, and outperforms prior related works. The authors evaluated the method on Franka-Kitchen and Meta-World, and the method is shown to have strong adaptability to new tasks.

All reviewers agreed that the proposed method is interesting both method-wise and application-wise. The performance gain is strong as well. Thus, the meta reviewer recommends "accept" for this submission.

Reviews: review: paper_summary: paper_summary: Learn a two-layer hierarchy from a sequence of datasets, where the low-level skills are represented by a discrete set of prototypes: vectors that can be mapped to repeated patterns of actions represented by basis functions. The basis function parameters are then passed into a decoder function which takes actions based on the observation and goal. The prototypes are recovered by performing k-means to cluster the data of a particular skill. The frequency of skill selection overall is a score, and the decoder is trained with imitation learning.
strength_weakness: strength_weakness: It is not obvious how the skills might be entangled together post-hoc, since the reusability of a skill across tasks might make its unlearning impossible. The experimental results also seem cherry-picked to ensure that this is not an issue, which is probably disingenuous to the actual cause of privacy: whether a component be relearned without any of the information from a particular source.

The experiments appear to be convincing only in the semi and incomplete settings, but it is not entirely clear what the semi or incomplete settings are. Without a clear picture of how these components are defined, it is not clear whether the empirical results actually support the claims made in the introduction.
questions: questions: Why is Continual Imitation Learning abbreviated CiL? It seems like it should be CIL.

Are there clear ablations on how the many components contribute to the overall performance?

What metric can be used to evaluate the unlearning capability in the context of privacy? Can this be used to ensure particular data is not used? Was this evaluated?
limitations: limitations: see above.
paper_summary: paper_summary: The paper introduces a new adapter-based method for continual imitation learning that avoids episodic replay and exhibits better forward and backward transfer and overall performance as compared to prior work. The authors compare their method against baselines on a couple of simulation benchmarks and also provide ablation studies to motivate their design choices.
strength_weakness: strength_weakness: - The paper is a little hard to follow in certain parts. Figure 1 could be made clearer.
- The method assumes access to datasets labeled with sub-goals. This must be added to the limitations.
- Line 140 mentions that a fixed function f is used to encode the observation and goal into a state embedding. Is this fixed function either obtained from the pretraining phase or is a pre-trained encoder of some sort? How does this encoder deal with changes in a non-stationary environment that it might not have been trained on?
- It might be useful to also evaluate IsCiL on the LIBERO benchmark which is developed for such continual learning studies and also provides human-collected demonstrations. This would help highlight the efficacy of the proposed method further.
- Is the choice of using a diffusion base policy for a specific reason?
- The exact lifelong setting for experiments in Table 1 is unclear. From what I understand, the base model is pre-trained on a subset of tasks/objects and new tasks/objects are introduced during the lifelong learning stage. Assuming the table reports multitask performance for all baselines, this raises two questions - (1) Since FWT and BWT are only reported with a single value, does this mean that the training is only done in two stages - pretraining with limited objects and all new objects introduced together? In case the new objects are introduced incrementally, should these numbers be computed for each stage where a new object/task is introduced? (2) How does varying the task order and the initial pretraining set of tasks/objects affect the final performance?
- Lines 233-234 mention that IsCiL exhibits performance ranging between 84.5% and 97.2%. However, in Table 1, I do not see any number as high as 97% and I can see IsCiL performance as low as 68.9% in certain settings. Also, IsCiL does not seem to be “surpassing the oracle baseline for Multi-task learning” as mentioned for any of these cases. Some clarification about how to interpret these results would be helpful.
questions: questions: It would be great if the authors could address the “Weaknesses” listed above. Also, is there an expanded version of the name of the method - IsCiL?

I am willing to increase my score once these questions have been addressed.
limitations: limitations: The authors have addressed the limitations. I have suggested adding the requirement of sub-goal labeled datasets as a limitation under “Weaknesses”.
paper_summary: paper_summary: The paper presents IsCiL, an approach to continual learning that addresses the limitations of knowledge sharing in traditional Continual Imitation Learning methods. IsCiL uses a prototype-based skill incremental method where each skill is represented by prototype embeddings and skill adapter parameters for LoRA adaptation. The state encoder encodes observations and subgoals into state embeddings, while the skill retriever matches these to existing skill prototypes during inference. The method is evaluated in environments like Franka-Kitchen and Meta-World, demonstrating its ability to learn and adapt without needing complete expert demonstrations. The results show IsCiL's performance in task adaptation and its ability to perform task unlearning for privacy concerns.
strength_weakness: strength_weakness: 1. The retrieval and adaptation processes at every time step might lead to increased inference time and resources, which is not thoroughly analyzed in the paper.
2. Handling overlapping skills appears to not be handled or need manual intervention to unlearn, which is not ideal for maintaining performance across an increasing number of tasks.
3. The paper lacks a detailed analysis of the computational overheads and scalability issues associated with maintaining a prototype-based memory and multiple adapters, which is important for real-world applications.
4. A major assumption in the paper is the high costs and inefficiencies associated with comprehensive expert demonstrations. It is not well motivated why obtaining a single comprehensive demonstration would be more cost-effective than multiple incomplete demonstrations. This should be further motivated.
questions: questions: 1. Does retrieval and adaptation occur at every time step? How does this affect inference time and compute demands?
2. The method seems to require manual selection of the task identifier for task unlearning. How does the method handle conflicting skills that need to override or combine with existing prototypes? How are skills consolidated as the number and complexity of skills learns grows? 
3. Can the unlearning process be applied at different granularities (e.g., specific sub-tasks or stages within a task), or is it only applicable at the (sub-)task level?
limitations: limitations: The authors do address limitations but could include more transparency on the inference time and resource use, which the paper does not thoroughly analyze. Secondly, handling a large amount of skill prototypes without additional consolidation.
paper_summary: paper_summary: This paper tackles the problem of continual imitation learning, where an agent needs to continually adapt to new tasks through imitation learning. The paper proposed IsCiL, a method that utilizes prototype-based skill incremental learning to gradually grow a repository of skill prototypes that can be retrieved and adapted to new tasks. Evaluated on Franka-kitchen and meta world, IsCil showed superior performance over previous CiL methods.
strength_weakness: strength_weakness: Overall the contribution seems incremental. The idea of prototype-based skill learning, parameter-efficient adaptation via LoRA, as well as the idea of tackling lifelong learning through skills have already been explored by previous works, and it is not clear what the key innovation of this work is.

Some of the important concepts in this paper, such as skill prototype, base, and skill adaptor, are not properly defined, making it hard to comprehend how they are generated and used. For example, how exactly are the pre-trained base model and the adaptor weights combined? Where does the pre-trained base model come from? How are the skill prototypes initilized?

This paper assumes access to a set of subgoals. In practice, this seems like a strong assumption, as these subgoals essentially segment the demonstration trajectories and implicitly define a set of short-horizon skills. Considering that these sub-goals are also used for generating and selecting skill prototypes, it raises the concern that these sub-goal specifications are doing the heavy lifting.
questions: questions: Although not necessary, I’d be interested to see the performance of IsCiL on more challenging benchmarks such as LIBERO. Also, given that the metrics measured by this work are intuitively visible from the training curve, it would be great to show some training curves of IsCiL and baseline methods.

Minor questions:
- How many skills are updated for each demonstration transition? It seems that only the retrieved skill is updated for each transition. Wouldn’t this be an inefficient use of data? 
- How exactly does IsCiL determine when to add new skills? Do we need a human to manually identify new skills?
limitations: limitations: The limitations is adequately addressed.

Comments: title: Thank you!
Thank you for the comments
Thank you!
Thank you!
Response to Authors
Thank you!
Thank you for the rebuttal
comment: Thank you for raising your score from 5 to 6. We truly appreciate your consideration and constructive feedback. We will, of course, incorporate this analysis into the final version.

The discussion on how to effectively search for and consolidate the numerous learned skills will be a valuable future direction for IsCiL.
Thank you for your comments and for providing the additional analysis and clarifications on efficiency and scalability. As a result, I have increased my score.
Thank you for raising your score from 5 to 6. We truly appreciate your consideration and constructive feedback. We will, of course, incorporate this discussion into the final version.

Additionally, although IsCiL primarily focuses on task unlearning, researching how to maintain CiL performance while ensuring privacy at the skill level is a challenging but promising area for future work. The question of which metrics to use for measuring unlearning privacy was particularly insightful and will be invaluable for advancing this skill unlearning approach.
Thank you for raising your score from 4 to 5. We truly appreciate your consideration and constructive feedback. We will incorporate the clarifications and discussion into the final version.

The suggested training curve allowed us to intuitively demonstrate the performance of IsCiL, which is highly valuable. Additionally, the discussion about sub-goals helped us clarify the contributions of IsCiL, and we are pleased that this has further strengthened our work.
I thank the authors for the clarifications and the additional experiments, which strengthened the paper. I've therefore raised my score.
I appreciate the clarifications and believe that the additions will strengthen the paper. I am happy to raise my score.
Thank you for raising your score from 4 to 6. We truly appreciate your consideration and constructive feedback. We will incorporate the clarifications and feedback into the final version.

We are particularly pleased that the suggested experiments have allowed us to emphasize the robustness of IsCiL. We are very grateful to the reviewer for proposing these experiments through this discussion.
I thank the authors for the detailed clarifications and additional experiments. Taking the rebuttal into account, I am raising my score to 6.