Title: Policy Compatible Skill Incremental Learning via Lazy Learning Interface
Venue: NeurIPS 2025
Accepted: yes
Keywords: Skill Incremental Learning

Imitation Learning
Abstract: Skill Incremental Learning (SIL) is the process by which an embodied agent expands and refines its skill set over time by leveraging experience gained through interaction with its environment or by the integration of additional data. SIL facilitates efficient acquisition of hierarchical policies grounded in reusable skills for downstream tasks. However, as the skill repertoire evolves, it can disrupt compatibility with existing skill-based policies, limiting their reusability and generalization. In this work, we propose SIL-C, a novel framework that ensures skill-policy compatibility, allowing improvements in incrementally learned skills to enhance the performance of downstream policies without requiring policy re-training or structural adaptation. SIL-C employs a bilateral lazy learning-based mapping technique to dynamically align the subtask space referenced by policies with the skill space decoded into agent behaviors. This enables each subtask, derived from the policy's decomposition of a complex task, to be executed by selecting an appropriate skill based on trajectory distribution similarity. We evaluate SIL-C across diverse SIL scenarios and demonstrate that it maintains compatibility between evolving skills and downstream policies while ensuring efficiency throughout the learning process.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of Skill Incremental Learning, where an agent incrementally expands and refines its skill set over time through interaction or the addition of new data. The focus is on a particular challenge in this setting: skill compatibility—specifically within hierarchical architectures where the skill module and policy module are updated at different rates, leading to potential inconsistencies. The authors propose two mechanisms to maintain compatibility:\n- BwSC Ensures that newly updated skills remain usable by previously trained policies.\n- FwSC Allows previously trained skills and policies to utilize newly added skills effectively.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper clearly identifies a practical challenge in skill-based policy architectures and provides a thoughtful solution.\n- The proposed compatibility mechanisms are well-motivated and show promising results.\n- Extensive experiments in simulated environments are used to validate the effectiveness of the method.\n\n\nWeaknesses\n- The issue of skill-policy incompatibility appears to arise from the design choice of using a hierarchical architecture with asynchronous updates. However, the paper does not sufficiently motivate this design. Why are skills and policies trained separately or at different rates? Establishing this necessity upfront would better justify the need for compatibility mechanisms.\n\n-  There is no evaluation in more realistic settings, particularly under noisy or imperfect state/action representations. This limits confidence in the generalizability of the approach beyond clean simulations.\n\nQUESTIONS:\n- Clarify the motivation behind the assumed architectural setup. Why is a hierarchical skill-policy architecture necessary, and why are updates asynchronous? Is this common in practice or a design choice unique to this work?\n- Add evaluation or discussion on how the method would handle noisy inputs, domain shifts, or other real-world challenges. Even a small-scale real-world test or robustness analysis would significantly strengthen the contribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuild up an incremental skill library by applying append-only skill updates and versioning, as inspired by filesystems. This involves a high-level policy passing subgoals to a low-level policy according to a temporal distance estimator, where the low-level policy selects between skills. Each module uses memory as the parameterization, and consists of classifiers that output scores for different selections of skills and goals. The skills are added to by clustering the new data, and described using an energy-based function over the primitive actions. Recovering policies involves selecting the lowest-scoring low-level skill. This method is then evaluated on Franka Kitchen and meta-world environments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe challenge of building up skills without losing current capabilities is fundamental to lifelong learning.\n\nThe method draws interesting connections and evokes interesting ideas related to saving skills.\n\nThe evaluation environments are challenging\n\n**Weaknesses:**\n\nThere appears to be a typo in the intro line 48.\n\nWhile the method is a fairly sizable system, Figure 2 does not clearly articulate the core premise of the system, that is, what data it expects to receive and how it should transform that data into an appropriate set of skills. While this may be contained in the figure, it is obfuscated by the layers of complexity. \n\nEither the problem statement for Figure 2 is inadequate for describing how certain components are being used, especially how the memory system for both states and skills works, and what the inputs and outputs of the modules are. While it is possible to make assumptions from other work, it is not clear because of the way these components are expressed.\nIt is also not clear what is done if a query-label pair is rejected (line 140). This is because the concept of skill hooking has not been introduced yet. \n\nThroughout the paper, the ordering is confusing, since many of the terms used are not clearly explained, nor are they common practice. For example, lazy learning and skill hooking are terms used but not explained, even intuitively, let alone formally, and without any reference to when they will be described in detail.\n\nSeveral key issues with adding skills incrementally are not directly addressed: What happens to the skill space when there are too many skills, so that clustering and classification become infeasible? How do the classifiers both scale and handle errors? Is this method able to improve existing skills or transfer them, since this is another one of the core promises of incremental skill learning, and if not, is this a core issue with the philosophy of keeping old data? How does this method apply to settings where some of the data may be suboptimal? \nThe description of lower score being better is confusing because score is typically higher is better. \n\nIn like 182, it does not make sense to introduce new skills by training the skill decoder, since the skills are introduced by modifying the environment. There must be some sequence of skills to describe when the values will come out. \n\nAs far as I am aware, the baseline methods are much simpler. It seems important to guarantee that the core principles behind SIL-C are the source of its advantage, and not just the presence of more hyperparameter tuning. However, there are no clear descriptions of what the core principles of this method might be, and the ablative experiments removing different components are terse and do not provide much insight into how the results imply that the component adds benefit. For policy hooking, this is because it is not clear why random should do better than a policy, and varying the classifier does not help suggest why the classifier itself is useful.\n\nQUESTIONS:\nThe lack of clarity for the policy hooking, lazy learning, memory usage, classifiers and other components makes it hard to assess this work properly.\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce SIL-C, a novel framework for Skill Incremental Learning (SIL) that maintains compatibility between evolving skills and downstream hierarchical policies. That is, SIL-C addresses the challenge of ensuring that updates to low-level skills do not invalidate or require retraining of high-level policies. The proposed approach decouples policy learning from skill updates using a bilateral lazy learning interface, which dynamically maps high-level subtasks to low-level skills based on trajectory similarity. This interface allows for both forward and backward skill compatibility; that is, (1) the newly added skills will be used effectively when learning future policies, and (2) existing policies can, when applicable, use the newly added skills without re-training. The proposed method is evaluated on two different environments and compared to relevant existing techniques.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses an important and underexplored challenge in lifelong learning: how to maintain compatibility between evolving low-level skills and higher-level task policies. The ability to incrementally integrate skills without retraining existing policies is an important contribution with practical implications for robotics and hierarchical RL.\n\nThe paper is generally well-written, and the motivation is clearly laid out. Figure 2 is helpful in conveying the structure of the proposed system, but it is dense. I believe the paper could benefit from a high-level pseudocode or flow diagram that lays out the full execution and learning process (e.g., when are classifiers updated? how is skill hooking triggered?)\n\nThe architecture introduces significant complexity (e.g., bilateral lazy-learning interface, trajectory similarity computations, prototype memories, etc.), but the design choices are not clearly justified. There is very little intuition/high-level discussion and description of the elements, and there are no ablations or comparisons to simpler or parametric alternatives, making it unclear whether the full complexity is necessary or beneficial.\n\nQUESTIONS:\n1. What are possible alternatives to the lazy learning interface proposed in SIL-C? For example, could a learned (parametric) subtask-to-skill mapping perform similarly or better in practice? It would be helpful to understand the motivation for using a lazy, instance-based approach over other alternatives, and whether this choice contributes significantly to performance.\n2. How sensitive is SIL-C to the quality of the unsupervised clustering used? The performance of the method seems highly dependent on the clustering results, but there is little discussion or ablation on this. How were hyperparameters (e.g., number of clusters, clustering algorithm) selected, and how robust is SIL-C to suboptimal clustering outcomes?\n3. How many random seeds were used for the experiments presented in Tables 1 and 2? The reported standard deviations suggest significant variance, so knowing the number of seeds is important to assess statistical significance. Were statistical tests performed to validate the significance of SIL-C’s improvements over baselines?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method of Skill Incremental Learning by introducing a lazy learning interface that connects subtasks to skills and a policy integration mechanism that enables seamless integration of learned skills to existing downstream policies. The key problem that this method tries to tackle is the forgetting of past skills as well as the usefulness of newly added skills for downstream tasks. New skills at different phases is determined using clustering. The paper evaluates this method on both backwards and forwards compatibility; it outperforms the baselines in overall performance and the forward compatibility, and performs closely to methods with predefined skill labels.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Generally well-written and with reasonably clearly written motivations and method. See minor comments below for improvements with the exposition. \n- Good literature review.\n- The method is an interesting idea.\n- A good range of experiments were carried out with a good number of baselines.\n- Good ablation study on removing the lazy learning interface.\n## Weaknesses\n- I think that the paper could use another small iteration where some improvements to clarity are made (see suggestions below and Questions).\n#### Typos and suggestions\n- Line 27: \"Yet, a critical challenge remain**s**...\" (remain -> remains).\n- Line 95: \"... may be the identity...\" -> \"... may be the identity **map** ...\"; I think adding map makes it clearer.\n- Line 100: Please define what $\\tau$ is right next to $\\mathcal{D}\\_\\tau$ rather in line 102. E.g., \"Let $\\tau$ denote a task; we assume access to task-level expert demonstrations $\\mathcal{D}_\\tau$ ...\".\n\n- Line 103: Could you add \"Our objective is to find the collections of high- and low-level optimal parameters over all phases, $(\\Theta\\_h\\^\\*, \\Theta\\_l\\^\\*)$ ...\". This is so that it immediately clear what $(\\Theta_h^\\*, \\Theta_l^\\*)$ are without having to wait till the end of Equation 2.\n\n- Section 4 paragraph 1: Could the appropriate mathematical notation for terms such as \"subtask space\", \"skill space\", \"task space\", etc.\n\t- The same in Section 4.2 paragraph 1. \n- Section 5: Can the number of repetitions be added to the captions of tables with $\\pm$ values.\n\nQUESTIONS:\n- Line 95: In what cases would $\\psi$ not be the identity map and why?\n- Line 99-100: Would $\\mathcal{T}_p$ depend on the dataset $\\mathcal{D}_p$ that is presented to the agent? I.e., would the set of evaluation tasks be setup to best use the dataset, or would it be based on some other criterion (if any)?\n- Line 101 \"... assume access to task-level expert demonstrations...\" -> Does this mean that the agent gets examples of how to incorporate the newly acquired skills, rather than learning whether they are useful or not itself?\n- Line 125: Is there a difference between a \"subgoal\" and a \"subtask\" in the context of the paper?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}