Title: REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Resource Efficiency

On-device Learning

Continual Learning
Abstract: Recent rehearsal-free continual learning (CL) methods guided by prompts achieve strong performance on vision tasks with non-stationary data but remain resource-intensive, hindering real-world deployment. We introduce resource-efficient prompting (REP), which improves the computational and memory efficiency of prompt-based rehearsal-free methods while minimizing accuracy trade-offs. Our approach employs swift prompt selection to refine input data using a carefully provisioned model and introduces adaptive token merging  AToM and adaptive layer dropping ALD for efficient prompt updates. AToM and ALD selectively skip data and model layers while preserving task-specific features during the learning of new tasks. Extensive experiments on multiple image classification datasets demonstrate REP’s superior resource efficiency over state-of-the-art rehearsal-free CL methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel strategy that reduces training time and computational memory costs in rehearsal-free, prompt-based continual learning setup, to facilitate real-world scenario deployment. The proposed method primarily optimizes two stages: prompt selection and prompt update. To improve in both stages, the paper proposes three techniques which trade a slight accuracy drop for improved resource efficiency. Specifically, the authors introduce a lightweight model with reduced depth and width for prompt selection. This technique also requires a non-trainable random projection to map back to original dimensions. Empirical results shows that this technique preserves 97% of representational similarity compared to a larger model. \n\nThe two other main techniques introduced, Adaptive Token Merging (AToM) and Adaptive Layer Dropping (ALD), are based on the analysis of which backbone blocks contribute most to the final loss. By measuring the mean attention distance they found that shallower layers capture more meaningful representations of the input. These findings are incorporated into AToM, which employs a scheduler that dynamically adjust the number of tokens to merge based on layer depth. Merging occurs more aggressively in deeper layers than in shallow layers, preserving important task-related information. ADL, on the other hand, is regulated by a layer-keeping probability that is also influenced by the degree of token merging performed by AToM in each layer. Since deeper layers tend to merge more tokens, they are more likely to be dropped. \n\nThe experiment results indicate that with by accepting a small trade-off in accuracy performance, the method achieves faster training and reduced memory usage.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper introduces an innovative strategy designed to reduce computational costs in continual learning, thereby enhancing the applicability of these systems in real-world scenarios, particularly in resource-constrained environments. This is achieved through three well-conceived techniques: a lightweight model with random projection, Adaptive Token Merging (AToM), and Adaptive Layer Dropping (ALD). These techniques operate synergistically to optimize both the prompt selection and prompt update stages.\n\nWhile a trade-off analysis is included, understanding its granularity is crucial to allow users to effectively navigate this trade-off based on their specific requirements. Although the composition of the prompt pool has been analyzed, prompt selection relies on identifying the prompt with the highest similarity to a given input. This approach might lead to only a subset of prompts being frequently chosen and subsequently updated, perhaps because they are better suited to different input types. This behavior is partly justified by findings that increasing the overall prompt pool size does not yield significant improvements. However, this selection mechanism could become a limitation when dealing with data originating from highly diverse domains.\n\nQUESTIONS:\n- Could the authors provide a more detailed, quantitative exploration of this trade-off? Specifically, can they present tables or graphs that clearly map different levels of resource saving to their corresponding accuracy degradation on the tested datasets?\n\n\n- Do they have insights into how their current prompt selection mechanism would perform when encountering truly novel tasks or highly divergent domains that might not align well with the existing prompts?\n\nLIMITATIONS:\nThe stated \"curriculum limitation\" is a technical detail that requires further clarification. The authors should elaborate on its meaning and explain why the inability to determine the learning curriculum constitutes a significant limitation. Furthermore, it is crucial to provide more granular and practical guidance on the accuracy-resource trade-off. Similar to the curriculum learning constraint, the paper should investigate how the method performs when truly novel and unexpected tasks or domains are introduced.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a EFCIL method that based on prompt tuning. REP consists of two parts: The Adaptive Token Merging AToM and Adaptive Layer Dropping ALD.\nThe work first show that ViT behave in away that earlier layer focus on more local features while deeper layers focus on global information by mean attention distance analysis.\nAToM dynamically merge tokens based on task specific information and layer specific information to allow more aggressive merging in deeper layers. To improve computation efficiency, AToM relies on a smaller model i.e ViT tiny to compute query feature for prompt selection in large model i.e ViT Base\nALD meanwhile adaptively drop information in ViT layed based on number of merged tokens in AToM. This allows more aggressive dropping in later layers as earlier layer contains richer features information.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\nThis work show good motivational experiment by showing ViT layers behaviour in mean attention distances which justify the need of more aggressive AToM and ALD. The ablation studies and comparison to other dropping strategies also further solidifies the claim as ALD perform well. Proposed method also achieve at least 10% faster computational time and 10% less memory consumption. The proposed method is tested in large number of different CL methods which proven it's adaptability and modularity. This work shows that it is possible to use lightweight backbone to perform approximated query selection on huge model, which is potentially beneficial for future works.\nWeakness\nREP performance in average accuracy drops compared to vanilla methods. Only very small amount of cases where REP actually help the AA. This raises the question on whether the computational speed and GPU memory saving is significant enough to justify the drop in performance.\n\nQUESTIONS:\nwhat is the baseline method  used in Table 4? Is it possible to show graph of ACC and forgetting of different n and theta for different CL method when combined with REP?\nFor figure 4. What happen if Random drop have different aggressiveness? Will it have a more closer performance compared to ALD?\nFor AToM, as shown in Figure 3, gradient w.r.t prompt is significant and contribute essential part to CF? Is it possible to improve AToM by involving Fisher Information Matrix like in EwC to have a gradient informed prompt merging?\n\nLIMITATIONS:\nThe authors have thoroughly discussed limitations of REP in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces REP (Resource-Efficient Prompting), a framework designed to improve the computational and memory efficiency of prompt-based, rehearsal-free continual learning (CL) methods for vision tasks. The authors propose three complementary techniques: (1) a lightweight surrogate model for swift prompt selection, (2) Adaptive Token Merging (ATOM) to intelligently reduce the number of input tokens while preserving task-specific prompt information, and (3) Adaptive Layer Dropping (ALD), which dynamically skips transformer layers based on the token merging ratio from ATOM to reduce computational load. Through extensive experiments on various datasets and Vision Transformer (ViT) backbones, the paper demonstrates that REP can significantly reduce iteration time and memory usage (by up to 51% and 48% respectively) with only a minimal trade-off in model accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper addresses the critical and highly relevant challenge of deploying continual learning models on resource-constrained edge devices. \n2. The empirical evaluation is exceptionally thorough. The authors test REP across three datasets, three ViT backbone sizes, and integrate it into seven different state-of-the-art prompt-based CL methods. \n3. The framework delivers substantial improvements in resource efficiency for a negligible drop in accuracy. \n\nWeaknesses:\n1. The core technical components, while cleverly adapted, are not fundamentally new concepts. The methodology relies on adaptive versions of existing techniques like Token Merging and Progressive Layer Dropping. While the paper's contribution lies in the intelligent, adaptive scheduling and combination of these techniques for the CL context, the work could be perceived as more of an engineering or systems contribution rather than a fundamental algorithmic breakthrough.\n2. The proposed methods, particularly ALD, introduce new hyperparameters such as the merging threshold τ and the adjustment factor α. While the appendix provides a hyperparameter search, the paper lacks clear intuition or guidelines on how to set these parameters for new datasets or model architectures without performing expensive, task-specific tuning. This could potentially limit the \"plug-and-play\" applicability of the framework.\n3. The paper notes that in some cases (e.g., L2P with ViT-L), REP can even improve model accuracy. The authors hypothesize that this might be because ALD and ATOM help eliminate redundant computations, effectively acting as a form of regularization. This is a very interesting phenomenon, but the paper fails to provide a deeper analysis to support this hypothesis. It is unclear whether this effect is due to the suppression of noisy features or some other mechanism. Adding some visualizations or quantitative analysis to investigate this would have significantly enhanced the paper's depth.\n\nQUESTIONS:\nPlease address the concerns raised in the \"Weaknesses\" section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors addressed the resource efficiency in continual learning with strong pretrained backbones (e.g., ViT). The authors first analyze the potential computation redundancy in prompt selection and prompt updates, and then propose solutions for each of them. For prompt selection, the authors propose to use a lightweight surrogate model and a random projection to replace the full pass on the backbone model for obtaining the query feature. For prompt update, the authors first analyze the frozen backbone to highlight the richer information contained in shallow layers. Therefore, the proposed Adaptive token Merging (AToM) preserves prompt tokens and merges more aggressively in deep layers. Similarly, the proposed Adaptive layer dropping (ALD) also drops more aggressively in deep layers. Extensive experiments showed that the proposed method can effectively reduce the computation cost with a moderate performance drop.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well motivated and addresses an interesting problem of computation efficiency\n\n2. The paper is well-written and easy to follow \n\n3. The proposed method is well motivated and simple, but effective\n\nWeaknesses:\n\nI don't see many significant weaknesses in this work. There are a few minor remarks\n\n1. The hyperparameter comparison can be improved\n\n2. The presentation of ablation can be improved\n\nPlease see details in the Questions section.\n\nQUESTIONS:\n1. The hyperparameter comparison can be improved\n\nIn the hyperparameter comparison in the Appendix (which is not referred to in the main paper, I would encourage the author to add this reference), the parameter alpha varies between 10 and 16, which is a rather small range. I am wondering how it would impact the performance if we test with a wider range.\n\n2. The presentation of ablation can be improved\n\nThe ablation table is not easy to follow. I believe the row (1)-(6) shows the performance without the corresponding components. Please correct me if I'm understanding it wrong. First, there is no baseline performance as a reference. Second, it is unclear why (6) is more computationally efficient than (2) and (3).\n\n3. About the prompt selection\n\nDid the authors try to use the query feature from the intermediate layers of the f_query or f_efficient and project it with the random projection? Would this be an alternative to using a surrogate model?\n\nOverall, I believe this is a good paper with good motivation, interesting analysis, and effective method design. It would bring insights to the community for a more resource-efficient CL.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}