REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning
Authors: Sungho Jeon
Xinyue Ma
Kwang In Kim
Myeongjae Jeon
Venue: NeurIPS 2025 poster ()
Keywords: Resource Efficiency
On-device Learning
Continual Learning
Primary area: deep_learning

Abstract: Recent rehearsal-free continual learning (CL) methods guided by prompts achieve strong performance on vision tasks with non-stationary data but remain resource-intensive, hindering real-world deployment. We introduce resource-efficient prompting (REP), which improves the computational and memory efficiency of prompt-based rehearsal-free methods while minimizing accuracy trade-offs. Our approach employs swift prompt selection to refine input data using a carefully provisioned model and introduces adaptive token merging  AToM and adaptive layer dropping ALD for efficient prompt updates. AToM and ALD selectively skip data and model layers while preserving task-specific features during the learning of new tasks. Extensive experiments on multiple image classification datasets demonstrate REP’s superior resource efficiency over state-of-the-art rehearsal-free CL methods.

REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning
Sungho Jeon, Xinyue Ma, Kwang In Kim, Myeongjae Jeon
POSTECH Seunghojeon, xinnueaa,, .tt.aamnnnnnjjoooppppppootoo jeonneepstecchh..... kc.kr
Abstract
Recent rehearsal-free continual learning (CL) methods guided by prompts achieve strong performance on vision tasks with con-staiionay data but remain resource- intensive, hindering real-world edge deployment. We introduce resource-efficien prompting (REP), which improves the computational and memory efficiency of prompt-based rehearsal-free -free continual learning methods while minimizing accuracy trade-offs. Our approach employs swift prompt selection to refine input data using a carefully provisioned model and introduces adaptive token merging (AToM) and adaptive layer dropping (ALD) for efficient prompt updates. AToM and ALD se- lectively skip data and model layers while preserving task-specific features during the learning new tasks. Extensive experiments on multiple image classifica- tion datasets demonstrate REP's superior resource efficiency over state-of-the-an rehearsal-Fie free CL methods.
Introduction
Continual learning (CL) trains neural network models on multiple sequential tasks, where each task may include data distributions that diverge from previously encountered data. A crucial challenge for any CL algorithm is to effectively address catastrophic forgetting [23]. Severe forgetting occurs when model rapidly loses previously learned knowledge while adapting new tasks, significantly affecting the model's reliability and accuracy on earlier tasks.
Moreover, many of today's AI services designed for on-device scenarios to securely learn tasks locally [19 I0, 42). In on-device CL. improving computational efficiency is crucial, as it directly reduces energy usage and enhances the durability of edge devices. Mcanwhile, since device memory capacity often acts as a hard constraint, CL tasks that exhaust all available memory can cause system crashes due out-of-memory errors (9) Given typically limited memory sizes (1-8GB), achieving high memory efficiency is essential to enable diverse on-device deployment scenarios in practice.
In this paper, propose REP (Fig. TT. framework for resource efficiency in prompt-based based rehearsal-
free CL methods on frozen, pre-trained vision transformers (ViT's [37
38
35
31
[4]
13
30 33
Prompts are small set of parameters that progressively learn incoming tasks to combat forgetting. Updates to these compact prompts incur minimum data writes without significantly harming the lifespan of device storage, which typically sustains up 10K of writes per location. This makes prompt-based rehearsal- -free methods well-suited for on-device continual learning.
REP is built upon analysis of cost-accuracy trade-off's throughout the end-to-end learning process, with two key design insights. (1) The prompt selection stage, which constructs a prompt subset to augment input intermediate data is highly amenable numerous promising options with fast approximations. (2) In contrast, the prompt update stage, which involves forward-backwarrd passes the backbone model, presents a range of optimizations with vastly different cost-aceuracy trade-offs. Based on these insights, we develop three complementary techniques for both stages that trade negligible accuracy drop for high resource efficiency: (1) random projection-based lightweight
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompt pool
Unmerged token
Merged token
Gate function
Backbone
Adaptive Token Merging Adaptive Layer Dropping
Query
V
V
Prompt
K
V
Prompt Selection
Prompt Update
Figure 1: Overview of the proposed resource-efficien prompting (REP) algorithm for rehearsal freee CL. REP calculates query features from input samples using a lightweight surrogate model (c.g.. ViT-Ti) and random projections to swiftly extract prompts from the prompt pool. These prompts then inserted into a main backbone model (e.g., ViT-L) for training, which prioritizes model accuracy.
surrogate model for prompt selection, and (2) adaptive token merging (AToM) and (3) adaptive layer dropping (ALD) for prompt update. Notably, AToM and ALD unifurrfy skip parts of the data and model layers to reduce training costs while preserving critical task- knowledge. These approaches are inspired by our analysis (Fig. 2 and prior works (37 28 demonstrating that shallower layers in pre-trained ViTs
sensitive new tasks than deeper layers.
Experimental results demonstrate that REP considerably reduces training time (up to 51% ()) and memory usage (up 41%), with only marginal accuracy drop, when applied to seven state-of-the-an prompt-based CL methods across three backbone models of varying sizes. As the proposed techniques complementary in optimizing resource efficiency, we further evaluate them by integrating them into two distinct ViT-based, rehearsal-frea CL methods that do not rely on prompting. In these settings, REP achieves 37-48% reduction in training time and up to 48% decrease in memory usage, confirming that its efficacy can extend well beyond prompt-based CL scenarios.
2 Related work
Class-incremental learning. Our work focuses on a class caassinnreemetall learning (CIL) setting, where each task introduces classes within the same domain, but task IDs unavailable during inference [ In CNN-based approaches, rehearsal based methods that replay stored samples remain dominant due their strong performance [14 26 29) Recent methods such BudgerCI. [27] methods under computational constraints. MEMO 4 trades stored samples for task-specifie layers when beneficial. We compare our prompt-based approach with these methods and show superior performance under resource constraints.
and CarM 200 adopt data-driven strategies that outperform optimization-
Continual learning for the edge. Recent efforts in on-device learning have emphasized memory and energy efficiency, primarily outside the CI. domain p 34 33). A few studies have extended CL to edge devices: Hayes and Kanan [10] investigated online CL with CNNs for embedded systems; Kwon et al. [19] analyzed rehearsal vs. regularization methods in terms of storage, compute, and accuracy trade-offs; and Ma al. (22) introduced Miro. platform that dynamically configures CNN-based CL to reduce energy consumption within memory constraints. However, none of these works address the challenge rescurce-effcieet vision transformers (ViTs) for on-device CL, which is the primary focus of our work.
Prompting for continual learning. Prompting, which provides explicit instructions or queries to the model, has proven effective for ViTs CL by enabling adaptation to new tasks while preserving prior knowledge [3] 14
38 L2P (38 and DualPrompt [37] retrieve task-relevant prompts from shared pool using prompt tuning and prefix tuning, respectively. CODA-Prompt [i] improves prompting capacity with attention conditioned prompts, while OVOR [3] uses a single prompt to accelerate selection. HliDe-Promp (33 extends supervised prompt-based CL to self- supervised settings through hierarchical optimization. ConvPrompt 300 employs convolution-base layer-wise prompts to improve knowledge transfer with negligible overhead. LAE 7 interprets prompts form of parameter-efffcient fine-tuning, incorporating modules such adapters or
30
B1
33
35
37
attention-condittonee

---

LoRA. As shown in Section our techniques can be applied to these methods to enhance resource efficiency without significant accuracy loss.
3 REP: Resource-Effiiien Prompting
Prompt-based rehearsal-free CL achieves strong performance by mitigating forgetting without relying on replay. However, it remains resource-intensive during the prompt selection In particular, prompt updates require full forward and backward passes, which are costly for large backbones, limiting their applicability on resource-constrained devices.
and update stages.
3.1 Motivation: challenges in prompt selection and update
Prompt selection operates neural network fquery to compute query feature q(x)) R for given input af from task j. It then selects the prompt p" from the prompt pool P that maximizes cosine similarity with the query feature:
argmax (o(x)).
PLEP [|(x))
In practice, fquery is often the same as the main backbone fupdate- and each query adds an extra forward pass per sample. For large models such as ViT-L, this results in up to 28% increase in computation time (see Table
Prompt update refines learnable parameters for effective training and adaptation by combining the classification loss Lelass a prompt-specific loss Lprempt, and auxiliary loss Laux during task j:
L = T.class (fuptat((r (at)), v!) + Teprump (p", q(x))) + Laux:
where fupdate denotes the main backbone, and é₁ and 62 control the influence of the prompt-specific and auxiliary losses. Different methods instantiate this loss differently. For example, [37 38 omit
the auxiliary loss by setting e e, O, whereas 31 [
tune both coefficients.
Although most backbone layers remain frozen, each mini-batch still incurs full forward and backward passes through the network, requiring storage of intermediate activations. As a result, large backbones such ViT-L exhibit substantial memory usage and prolonged training time (see Table
3.2 Proposed solutions: REP
REP is built on two core insights:
1. The prompt selection stage can tolerate approximations: high retrieval quality can be achieved without relying on the full-capacity backbone.
2. Not all layers in a frozen backbone contribute equally to adaptation, allowing selective computa- tion during prompt updates.
To realize these insights, REP introduces three complementary techniques; a lightweight surrogate model for efficient prompt selection, and adaptive token merging (AToM) and adaptive layer dropping (ALD) for efficient prompt updates.
3.2.1 Guiding prompt selection with lightweight surrogate model
To reduce the computational cost of prompt selection, with reduced depth and width. Instead of reusing a large backbone such as ViT-L, we adopt a compact pre-trained ViT-Ti model, which effectively captures essential representations. Given an input a?
lightweight surrogate model feticient
fofficient produces a low-dimensional query feature Gefficient (I:) Rd (where d < D). We then apply fixed, non-trainable random projection á to map this feature back to the original D-dimensional space for prompt selection:
(0 (gfficient (x:)),
Pofficient argmax
P ||0(gemcieen (x:) (r)))
3

---

Empirically, this strategy preserves around 97% of the representational similarity to ViT-L-based query model, as measured by centered kernel alignment [17]. Despite residing dimensioni space, Pofficient effectively approximates the large model's representations, substantially reducing computational cost (see Table D
3.2.2
Analysis of the frozen backbone during prompt update
Mean Attention Dist
100
Mean Attention Dist
100
100
80
80
Mean Attention
80
60
60
Layer2
60
40
40
40
Lswe31
20
20
20
10
15
20
5
10
2
4
6
8
10
Layer ID
Attention Head
Attention Head
VIT-L by layer.
(b) VIT-L by attention head.
(c) by attention head.
Figure 2: Mcan attention distances for frozen blocks along (a) layers and (b/c) attention heads. We run the first task of Split ImageNet-R. (a/b) L2P with ViT-L, and (c) DualPrompt with ViT-B.
key insight of REP is that not all frozen blocks contribute equally to the final loss L. To validate this, we analyze the feature representations of pre-trained transformer blocks with prompts using mean attention distance, a metric previously used to study layer-wise representations in ViT models
during pre-training 6 28
For given attention head within frozen block. let (zg. 3q) denote the position of a query patch, and (r,, W) the positions of the patches the head attends to, with corresponding attention weights a, The distance d, between aatches-typicee
defined as Euclidean or pixel distance given by d. = (x: xx,)² + (y: )2 We then compute the weighted mean distance as 2. 2.ard Following 6 28 smaller distance indicates attention focused on narrow region (local information), while a larger distance suggests more distributed attention across the input (global information). We measure the average layer ID and attention head, during adaptation to new task in Split ImageNet-R (10 tasks).
attention distance of L2P with the ViT-L backbone along two dimensions:
As shown in Fig. a), shallower layers exhibit wider range of attention distances and tend to focus on both localized and global regions. In contrast, deeper layers consistently produce higher values, reflecting shift toward more global contextual information. This pattern aligns well with prior findings in [6 (28). A closer look at a few selected layers in Fig. ( further reveals that attention distances across heads vary more widely in shallow layers than in deeper ones, confirming that shallower layers capture more important representations of the input.
A similar trend is observed when using Prompt with ViT-B in Fig. ((c).. Unlike L2P, DualPrompt attaches prompts to the self attention layer of multiple transformer blocks, rather than only in the input sequence. This suggests that the feature representations of the pre-trained model, which balance local and global information, may not greatly affected by the prompt method in use.
3.2.3 Adaptive token merging (AToM)
We explain how we bring the above insights into practice through two compute-skipping techniques. We first consider the data-efficient compute-skipping method via token merging. Conventional token merging (ToMe) 2 reduces the number of tokens by uniformly merging n redundant tokens per layer, controlled by fixed scheduler The scheduler function r()) + applied to each layer I, and according to (2). it merges all tokens, including the prompts added by the token selection process. However, insights from Fig. and our additional analysis highlight two major problems.
First, there is a loss of task-specific information. The prompt tokens in CL carry essential task-related information. However, ToMe indiscriminately combines these prompt tokens with non- prompt tokens, diminishing their intrinsic value. According to our empirical data in Fig. 3 this approach can cause gradient explosions in the prompt tokens even with gradient clipping, leading to learning instability.
Second, there a lack of layer-specific adaptability. ToMe does not account for the disparity between shallow and deep layers, treating the importance of all layers uniformly. Therefore, there is risk of

---

excessive loss of valuable information in shallow layers, which are mainly responsible for adaptability diverse sequential tasks.
Our adaptive token merging (AToM; Algorithm I)
12
Original
addresses the loss of task-specific information by excluding prompt tokens during token merging, thereby maintaining their specificity. To enhance the effect of prompts and mitigate catastrophic for- getting, AToM uses a new progressive scheduler
10
ToMe
Gradient Norm
AToM
r' (1) + n', which dynamically adjusts the num-
ber of tokens to merge based on layer depth, as follows:
20
40
60
80
100
120
140
Iterations
(()) = min( x (1 - 1), Tmax).
(4)
Figure 3: The norm of gradient with respect to the prompt during training Split ImageNet-R (10 tasks) when AToM and ToMe (Conventional to- ken merging) are applied to L2P with ViT-L.
where denotes the layer index, change in the number of tokens to merge, defined as Tmas
the step
Z-T (with L being the number of layers), and Tmax is the maximum number of tokens to merge (by default, 2 x n). With this rl)), merging occurs more aggressively in deeper layers than in shallow ones, preserving important task-related representations.
3.2.4 Adaptive layer dropping (ALD)
Inspired by insights from Fig. 2 and prior work progressive layer dropping (PLD) 41 in the NLP domain, we propose adaptive layer dropping (ALD; Algorithm 2 with two key features: (1) the dropping schedule considers both temporal and spatial dimensions, and (2) it manages to drop layers non-uniformly to preserve critical representation features in shallower layers[] On the contrary, PLD considers only the temporal aspect and does not differentiate between layers when dropping. This results in poorer performance compared to ALD, as shown in Table
ALD prioritizes retaining shallow layers that contain richer information essential for model per- formance, especially after token merging. Thus, instead of operating on its schedule parameters, ALD leverages feedback from ATOM, specifically the count of merged tokens each layer, to guide layer-dropping decisions. The layer-keeping probability or,1 is defined as:
0,, = (a(l) x ((1 - f) exp( (- (t) +
(5)
where o.r1 is the probability of keeping layer I time step t, 0 is the minimum probability, y controls the decay rate, and (l) is the adjustment factor for layer 1, defined as:
Ja if (n(() n' n() 2 T if (n(1) n' r1)) <T.
all)
(6)
al)) quantifies the degree of token merging performed by AToM in layer I. nll) represents the original number of tokens, and n' (I) denotes the number of tokens remaining after merging. When the number of merged tokens surpasses the threshold r, ALD adjusts the layer-dropping probability based on a. Since deeper layers tend merge more tokens with ATOM, ALD is more likely to exceed 7 in deeper layers and drop more aggressively. These parameters should be tuned to balance hetween efficiency and the preservation of nuanced information contained in the merged tokens. We set 0.9, with set to 16 for ViT-L, 12 for ViT-B, and 8 for ViT-Ti, respectively.
Comparison with various layer-dropping strategies. We compare ALD against multiple layer- dropping strategies that skip computation across different portions of backbone layers. We report the training wall-clock GPU time and final average accuracy in Fig. 4 using L2P with the ViT-L backbone on Split ImageNet-R (10 tasks). Specifically, Top-layer Drop and Bottom-layer Drop statically drop the first and last 25% of layers, respectively. Random Drop randomly skips 25% of layers across the network, while Stochastic Depth applies linear decay schedule over Random Drop. For reference, we include No Drop, which performs no layer dropping and yields 75% accuracy. To account for the stochastic nature of compared layer dropping methods, such as Stochastic Depth, we
'The spatial dimension refers model layers, each processing input features at different levels of abstraction.
5

---

measure end-to-end wall-clock GPU time. Top-layer Drop performs the worst, achieving only 30% accuracy. Bottom-layer Drop (63%), Random Drop (70%). and Stochastic Depth (72%) also face noticeable accuracy degradation despite reduced GPU time.
In contrast, ALD maintains 75% accuracy- par with No Drop-while achieving GPU time savings comparable Random Drop. This high- lights that ALD's non-uniform strategy accounts for each layer's is essential spatial representations more effectively. REP triggers AToM and ALD for each task insertion. Much like our prompt- selection optimization, AToM consistently en- hances time and memory efficiency. contrast, ALD solely contributes to reducing time costs since operates with the layer-keeping probabil- ity O2 initialized at 1.0, i.e., no layer dropping.
ek:
Tac
20
Accunacy
(%)
2K
1K
36 10g Rayel 52.2 MVC Ramcom 50r
Figure 4: Comparing various layer- ayye-ddgpping strate- gies using L2P with the ViT-L backbone on Split ImageNet-R (10 tasks). The bar and marker are GPU time and final average accuracy, respectively.
3.2.5 Discussion
Alternative adaptation approaches. Prompt-based adaptation represents an effective paradigm for continual learning, particularly in resource-constrained settings, owing to its modularity and small parameter footprint. Unlike full fine-tuning, prompting keeps the backbone entirely frozen while allocating a small number of learnable parameters per task. This not only preserves prior knowledge and prevents catastrophic forgetting but also drastically reduces memory usage, which is critical factor for edge devices. In comparison to alternatives such as LoRA, RanPAC, and adapter tuning, prompting exhibits superior balance between efficiency and performance, demonstrated by our integration of REP into multiple state-of-the-art prompt-based CL. methods.
Compact backbone networks (e.g., CNNs). Although CNNs are generally more compact than vision transformers (ViTs), they consistently underperform in the CL setting, particularly under tight resource constraints. Our preliminary studies in Section D demonstrate that state-of- of-the-art CNN- based CL methods, such BudgetCL (27] and MEMO 43 achieve significantly lower accuracy compared to ViT-based approaches.
4 Experiments
We focus on the popular disjoint-CIL setup, where each task comprises a distinct set of non- overlapping classes and samples from old classes are not given in future tasks [29] 4 8). In this section, we demonstrate that (1) REP enhances resource efficiency in prompt-based rehearsal-free CL across various backbone model sizes and (2) its core components, ALD and AToM, are applicable even when CL methods do not employ prompting.
Data generation. To organize task streams, we use three image classification datasets: CIFAR- 100 (100 classes) [18), ImageNet-R (200 classes) [12,, and PlantDisease (38 classes) [25] Out of these datasets, ImageNet-R is known for exhibiting much higher intra-class variability and an uneven class-size distribution among images. We divide CIFAR- 100 and ImageNet-R into 10 tasks to create Split CIFAR-100 (i.e., classes per task) and Split ImageNet-R (i.e., classes per task), For PlantDisease, we drop 3 plant disease classes with very few images and organize the remaining 35 classes into 7 tasks to create Split PlantDisease (i.e., classes per task). We also present additional results with varying task lengths (e.g., 5 tasks and 20 tasks), and an additional Split CUB-200 dataset (32] in Section F and Section G respectively.
respectively 38 37 31
HiDe-Prompt 3 ConyPrompt 300, LAE 77, and OVOR [13 as representative prompting methods. These methods capitalize on ImageNet pre-trained models as backbones: ViT-L (307M), VIT-B (86M), and ViT-Ti (5.8M) -iT-Ti is one of the smallest vision transformer models to our knowledge. Detailed settings for all methods are provided in Section
Methods. We employ I.P
38 DualPrompt
CODA-Prompt
Hardware and metrics. We use an NVIDIA RTX 3090 to cover a wide range memory capacities while maintaining consistent computational power Specifically, we limit the GPU memory to emulate

---

Table I: Accuracy and computational cost of all competing methods three datasets. We report the final average accuracy, iteration time, and memory usage, both without and with REP. Iteration time and memory usage presented as absolute values. For the results without REP, parentheses indicate how many times higher the value is relative to its counterpart with REP.
Final average accuracy (1)
Ier time (ms)
Mem. (GB)
Model Method
Split CIFAR-100 100
Split ImageNet-R
Split PlantDesuuse
w/o REP
REP
w/o REP
REP
w/ REP
REP
w/o REP
REP
w/o REP
REP
L2P
88.2
87.011.3
75.61:1.
7..3+11.5
75.9:3.11
81.1
447(
240
1.4(
4.5
DualPrompt
86.3+0,3
85.1+1.3
71.240.6
70.6++0.9
75.1+1.1
78.2+2.9
424 x200)
208
5,9(x 1.4)
4.3
CODA-Prome
85.4:0.4
84.3111.7
74.4100.6
73.5::1.
75.0:220
74.1=2.7
x1.3)
441
13.2(x1.2)
11.0
VIT-L
HiDe-Promg
93.440.3
92.8+-111
78.7440.2
78.0++1.2
98.6-0.4
98.3-08
x13(x118)
227
7.2x1.2)
6.3
CouvPrompt
89.2:0.55
89.01:0.9
79.1:10.1
78.5:008
98.4:0.6
98.6
560X
x1.3
417
5.4(x13)
4.1
Are.Prefix
84,9+1.3
84.5+1155
70.5++1.9
70.2411.
73.3-0.5
72.9-0.9
x85(x1.1)
5.1(x11)
4.8
OVOR
86.240.6
85.810.9
75.440.77
74.3+1.
75.4=0.8
74.9=10
467( x.33)
362
7.5(x111)
6.8
UpperBound
94.1+40.1
85.240.2
99.6+0.33
427
9.8
L2P
84,5+08
83.41:8
59,440.8
58.7112
64.422.
64.0=2.44
143(x1.4)
102
2.3(x1.2)
2.0
DualPrompt
85.1+0.2
84.9+099
68.140.1
67.0+1.0
78.0+1.5
x33 7..2+1.9
101
2.1(x1.2)
1.8
CODA-Prompt
83.3+0.9
82.11199
59.040.3
58.341.8
7144008
70.9-2.7
164 x 1.1)
151
6.9(x1.)
5.7
HiDe: Prompt
88.5+0.7
89.24:1.6
64.4+100
1300 0.
76
4.10x17)
2.4
VIT-B
ConyPrompt
87,6+0,5 87.0+1.4
67.640.6
67.1+1.2
94,3-0.88
381(x1.4) 94
279
2.2(x11))
2.0
A.E-Prefix
83.5:008
84.2.11.3
61.340.6
61.0:10.9
70.::0.66
704
S70xL.1)
50
1.9XLD)
1.8
OVOR
84.2+00.7
83.94-1.0
61,040.6
6..6++1.
67.2-0.5
66.9-09
1340x1.1)
123
3.1(x.11)
3.0
UpperBound
92.0=0.2
81.410.1
99.6:20.4
143
3.2
L2P
60.3+0.2
59.3+:1.7
41.240.77
40.3++1.0
56.12299
55.242.9
34(x1.1)
30
0.5(x1.1)
0.5
DualPrompt
62.920.5
62.510.9
43.640.8
42.7+10
6..0=10
62.9=19
34(xL1)
31
0,4(x1))
0.4
CODA- Prompp
67.0+0.4 65.9+-1.8
49.340.8
48.242.0
6688-08
65.5+2.8
36(x1.2)
30
2.8(x11)
2.8
HiDe-Prompt
72.2:0.5 72.5+1.0
44.84088
45.2414
92.6=0.7
93.3=1.5
23(x1.2
20
0.5(xLD)
0.4
VIT-Ti
ConvPrompt
69.1+008
68.5+1.7
49.840.8
49.141.9
9244006
03
850x1.9)
45
0.5(x11)
0.5
LAE-Prefix 10
61.340.6
61.3114
49.7440.8
49.6#11.
66.8-0.7
66.0=1.5
15(x13)
12
0.4x1D
0.4
OVOR
67.041.3
49.3+1.22
70.3+40.6
71.6=1.7
35(xL1)
32
1.0xLI)
0.9
UpperBound
83.0=0.1
77.440.2
98.0=009
42
0.6
resource-constrainee devices, ranging from IGB (compatible with Raspberry Pi) to 16GB (compatible with NVIDIA Jetson AGX Xavier). For computational efficiency, we measure iteration time, which is linearly correlated with GPU energy usage (22). We adopt the final average accuracy (higher is better) and forgetting (lower is better) from various previous works [37 38 31 In Section M we further evaluate REP directly NVIDIA Jetson TX2 with 8GB of memory, although it is applicable beyond this specific device.
4.1 Main results
Table presents the efficacy of REP when integrated into prompt-based CL methods across various ViT backbones (ViT-I., ViT-B, and ViT-Ti) and datasets. For direct comparison of these methods under their original setups. please refer to Section C
Resource efficiency. Integrating REP into existing prompt-based CL methods substantially reduces computational overhead and memory usage across all model configurations. As shown in Table [ baseline methods without REP consume up to 1.4x, 1.7x, and 1.1x more GPU memory than those with REP for ViT-L, ViT-B, and ViT-Ti, respectively. Although the ViT backbones remain frozen, these models still perform backward passes to optimize prompts and classification heads, requiring memory to store large intermediate activations. REP mitigates these memory demands by selectively merging tokens (AToM) and adaptively dropping layers (ALD), thus reducing the memory footprint of intermediate data.
In addition, bascline methods without REP use up to 2.0x, 1.7x, and 1.9x training time than those with REP for ViT-L, VIT-B, and ViT-Ti, respectively. These improvements computational efficiency directly translate into proportionally lower energy consumption. The gains are more marked in VIT-L, as its larger size offers greater room for optimizing computation cost.
Accuracy. While all baseline methods were originally designed and evaluated on a ViT-B backbone, our study broadens their applicability by exploring cost-accuracy trade-off's in on-device CL across a diverse range of ViT architectures. When augmented with our techniques, these methods achieve notable gains in resource efficiency with only marginal accuracy degradation: 0.0-1.22 12% on Split CIFAR- 100, 0. 1.1.% 1 on Split ImageNet-R, and 0.0-0.8% on Split PlantDiscase. In some cases,
7

---

accuracy even improves with REP; for example, the accuracy of L2P with VIT-L on Split PlantDisease increases from 75.9% to 81.1% when combined with REP. For completeness, the forgetting metric reported in Section
4.2 Ablation studies
We validate our proposed techniques and the cho- sen hyperparameters, primarily using REP inte- grated into L2P with the ViT-L backbone (denoted as REP-L2P) on Split ImageNet-R (10 tasks).
Table 2: Ablation study of REP's components, demonstrating their contributions to both re- source efficiency and accuracy.
See Section E for additional results.
Ablated components
Acc. (f) Fgt. (1) lter. time (ms) Mem, (GR)
REP-L.P
75.3+11.5
3,6++03
4.5
Component ablation. In T ablate the
wio (ATaM + ALD)
74.8+-0.8
4.0+0888
349
5.5
components of REP to assess their individual con-
w/o (Jeeeen ALD)
74.9=008
3.5++08
419
5.6
tributions. Each row reports performance when the corresponding component is removed. All compo- nents contribute substantially to reducing computa- tion time and memory usage, with AL. affecting only computation time, as expected.
74.520.8
3.6:10.7
wia ALD
74.2::0.8
400.088
270
4.5
w/o AToM
74.5++0.4
3,4+005
303
3,5
fencees
74.620.4
2.6:0..5
Algorithm validation. Conventional token merg- ing (ToMe) [ and layer dropping (PLD) [4] are specifically designed to accelerate the training of transformer-based models across various domains. To validate the importance of incorporating our adaptive techniques instead in CL, we first evalu- ate how REP performs in case AToM and ALD are replaced with ToMe and PLD, respectively. The results are presented (1) of T Although applying ToMe PLD improves resouree effi- ciency over the native L2P baseline in Table [] it results in accuracy drop of 5.1 % or 2.0%, respectively, compared to using our techniques.
Table 3: Effect of using (1) conventional accel- eration methods, (2) intermediate layer-based prompt selection method, and (3) random drop with diverse drop ratios, compared to our pro- posed methods.
Method
Ace.
Fgt. (1) Iter. time (ms) Mem. (GB)
(1) Algorithm validation
REP-L2P w/ ToMe PLD
75.3++1.5 3.6
240
4.5
70,2:0.77 2.6:0.9
275
3.7
73.3+0.77 3.910.7
259
4.5
Surrogate prompt selection validation
Surrogate Interm. (4)
75.3++1.5 3.6+-113
240
4.5
74.0:10.8 3,9:0.9
264
4.3
Surrogate prompt selection validation. We fur- ther validate the effectiveness of our lightweight surrogate-base based prompt selection method by com- paring it with alternative that uses the CLSS token from intermediate layers of ViT-L (layers 4, 8, and 12). As shown in (2) of Table while using intermediate layers eliminates the need for a sep- arate surrogate, it substantially increases training time, since even a partial ViT-L forward pass is more computationally demanding than running the full surrogate model. Accuracy also drops, with shallower-layer cases showing greater degradation, likely due to their limited access to global context.
Interm. (8)
74.3+0.7 3.5+0.9
285
4.3
Interm. (12)
74.6:10.7 2.620.8
307
4.3
(3) Comparison with FIM based token morging
AToM
74.5:1..8 3.5=0088
419
FIM-hased
74.7+0.9 3.8+
264
6.2
(4) Comparison with random drop
ALD
75.8:10.5 3.5
401
6.5
Random-209
71.740.4 5.8+0.8
409
Random-254 70.6::0.4 6.41:.11
398
6.5
Random-309i
68.8:10.6 7.1
385
6.5
Random-509: 65.4-+0.8 9.4++1.8
Comparison with FIM-based token merging. We also investigate whether additional gradient information, e.g. through the Fisher Information Matrix (FIM) [16], could serve as an alternative to AToM. Specifically, we compare AToM with a FIM-based, using L2P with ViT-L on Split ImageNet-R (10 tasks), as shown in (3) of Table 3 Although this approach yields modest accuracy gains (0. 1.0.2%), it incurs roughly 10% increase in training time and memory usage. By contrast, AToM preserves prompt gradients without such overhead, offering more balanced trade-off between efficiency and forgetting mitigation. This result does not point to fundamental limitation of FIM-based merging, but rather underscores the difficulty of achieving practical efficiency improvements in its current form.
pradient-informer merging strategy
Comparison with random drop across varying aggressiveness levels. To evaluate ALD against Random Drop with varying aggressiveness, we applied dropping rates of 20%. 25%, 30%, and 50% using 1.2P with ViT-L. on on Split ImageNet-R (10 tasks). As shown in (4) of Table higher drop rates reduce training time below that of ALD but lead to substantial accuracy degradation (65-71%),
8

---

making them impractical for continual learning. In contrast, ALD dynamically adjusts the drop ratio, maintaining resource efficiency while minimizing accuracy loss.
AToM and ALD intensity. Table shows the effects of varying intensities of AToM and ALD, focusing on the number of merged tokens (n) in AToM and the keep ratio (0) in ALD. AToM appears to maintain stable accuracy trends even as more tokens are merged. In contrast, in the case of ALD, lower keep ratio appears improve resource efficiency by reducing training time. However, can markedly impair model accuracy if the ratio is too low. Overall, when used with carefully selected hyperparameters, AToM and ALD can effectively balance resource efficiency and accuracy. Notably, forgetting remains stable when using the default settings (n 8 and = 0.5).
Table 4: REP over varying # of merged tokens (n) and % of keep ratio (#).
(w/ 9=0.5) (1) Fgt. (4)
(ms) (GB)
75.5+0077 3.5+0.7
268
75.2 3.740.7
265
5.5
75.3+0.3 3.340.5
256
5.2
75.1 3.840.8
253
15.3+1.. 3.6+0..3
240
4.5
10
73.6#1.7
4,7:11.1
228
4.1
(w/ m=8)
Acc. (1) Fgt. (1) Iter. time (ms) Mem. (GB)
0.1
72.9114 4.4
217
4.5
73.241.2 4,4112
223
0.3
74,4 4.1+088
228
4.5
0.4
74.7+1.0 3.940.03.4005
3,9+0.5
235
4.5
0.5
75.3+15 3.6+0.3
240
In Section [l, we extend the above study to two additional prompt-based CL. methods, including HiDe-Prompt 333 and ConyPrompt [30].
74.8+0.9 3.6:0.9
255
0.7
74.440.6 3.5
270
4.5
0.8
74.2+007 384+09
277
74.320.5
3.620.8
282
4.3
Additional studies
We further assess the efficacy of REP by applying it to CL methods not covered by Table including two additional non promptinn rehearsal- methods: such as SLCA [40] and RanPAC [24]. Addi- tionally, we compare REP-L.2P with two adapter-based methods, including Online-LoRRA [39 and ADAM 15
More discussions are provided in Section H and Section]] respectively.
4.3.1 Applying REP to non-prompting rehearsal-free methods
SLCA fine-tunes the entire network using distinet learning rates for the representation and fully- connected layers, whereas RanPAC employs a non-trainable random projection to the activations from the pre-trained frozen backbone. For SLCA, we retain the original hyperparameters (e.g., the number of epochs per task), and integrate ConvPrompt to enable parameter-efficiet fine-tuning (PEFT), as SLCA lacks native PEFT support. We measure iteration time and memory usage, with ViT-L on Split ImageNet-R (10 tasks).
As shown in Table 5 SLCA incurs high memory overhead even with ConvPrompt, limiting its compatibility with 8GB edge devices. Applying REP reduces both training time and memory usage by up to 48%, making it operational such resource-constrained devices. Similarly, applying REP to RanPAC reduces iteration time by up to 37% (as shown in Table but does not lower memory usage, random projections from upsampling dominate the high memory cost.
Table REP on SLCA with ConvPrompt using
Table 6: REP on RanPAC using VIT-L backbone on Split ImageNet-R.
ViT-L backbone on Split ImageNet-R.
Method Acc. (t)
Iter. time (ms) Mem. (GB)
Method
Acc. (t)
Iter. time (ms) Mem. (GB)
SLCA 77.2
1,311
RanPAC
82.4-:0.6
332 210
8.2
SLCA 7..0
686
4.8
REP-RanPAC 82.0+11.7
8.2
4.3.2
Comparison with adapter-based methods
LoRA-based method. We analyze Online-LoRA using the ViT-L backbone on Split ImageNet-R (10 tasks). As shown Table 7 this method in- troduces substantial memory overhead (>20GBB), making it incompatible with the memory specifi- cations of most edge devices. This overhead arises
Table 7: Comparison of REP with Online-LoRA using ViT-L backbone on Split ImageNet-R.
Method
Acc. (t) Fgt. (1)
Iter. time (ms) Mem. (GB)
REP-L2P Online-LoRA 52.6+:188 21.242..6
75.3+1.5 2,8+0.2
240
4,5
1.739
20.9

---

We focus standard continual learning setting where task arrivals predefined part of the problem formulation. In this setting, REP is designed to operate under the given task order. Curriculum learning, which explicitly determines optimizes the task sequence, addresses related but distinct problem. Since our study does not include curriculum-bast been evaluated in scenarios where the task order can be deliberately arranged. Exploring how REP could be combined with curriculum learning strategies is an interesting direction for future research. Moreover, although using a replay buffer may be considered infeasible in real-world scenarios with approach, from using replay buffers and multiple forward/backward passes, which significantly inflate both training time and memory usage.
Other adapter-based methods. Next, we look into ADAM, a representative adapter-based CL method. Table presents results using the ViT-L backbone on Split ImageNet-R (20 tasks). While ADAM variants can reduce training costs through lightweight adapters, they often suffer from large memory footprints that exceed the device limits (e.g., 8GB) from significantly lower accuracy compared to REP-L2P.
Table 8: Comparison REP with ADAM vari- ants using ViT-L backbone Split ImageNet-R, R.
Method
Acc. (1)
Iter. time (ms) Mem. (GB)
REP-L.2P
72.6+-0.6
240
4.50
VPT-Deep
66.7=0,4
136
12.6
VPT-Shallow
64.5
125
6.9
SSF
72.2=0.2
147
22.1
Adapter
62.1=0011
12.0
5 Conclusions
In this work, we introduce Eessurce-Efficient Prompting (REP), framework designed to enhance the computational and memory efficiency of prompt-based rehearsal continual learning methods. By incorporating surrogate prompt selection, adaptive token merging and adaptive layer dropping, REP selectively streamlines the learning process while preserving task-specific features, significantly reducing resource consumption without compromising accuracy. Experiments on multiple image classification datasets demonstrate that REP achieves substantial efficiency gains over state-of-the-an ViT-based rehearsal-frre methods, making it a practical solution for on-device continual learning.
task scheduling, REP has not
strict data privacy constraints, on-device CL is already viewed as making rehearsal-basee REP a promising direction for future investigation.
One limitation of our work is hyperparameter tuning. Following prior works, we tuned the hyper- parameters through grid search with 5-fold cross-validation for each task. While this provides an automatic tuning procedure, may limit the "plug-and-play" applicability of REP. Empirically, we observed that performance is relatively insensitive to variations in a, making it reasonable to fix 0.9. For the impact is more dataset- and model-dependent, but we found a consistent positive correlation between backbone size and the effective value. As future work, plan conduct more detailed study on hyperparameter tuning.
10

---

Acknowledgements
This work was supported by the Institute of Information & Communications Technology Plan- ning & Evaluation (IITP) grants (No. 18S-201-- Artificial Intelligence Graduate School Program (POSTECH); No. 05-2024-00459797, Development of ML Compiler Framework for On-device AI; No. RS-2025-02304554, Efficient and Scalable Framework for Al Heterogeneous Cluster Systems; No. 25-2002--1120020, Visual Intelligence for Space-Time Understanding and Generation), the National Research Foundation of Korea (NRF) grants (No. RS-2024-00337559; No. RS- 2024-00354947), and the Electronics and Telccommunications Research Institute (ETRI) grant (No. 25ZS1100, Research on Performrrfo Computing to Overcome Limitations of AI),
all funded by the Korean government (MSIT).
References
[1] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Wos Woo Ha, and Jonghyun Choi. Rainbow memory:
Continual learning with memory of diverse samples. In CVPR, 2021.
[2] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. ECCV, 2018.
[5] Arslan Chaudhry, Puncet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[7] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. In ICCV, 2023.
[8] Alexander Gepperth and Barbara Hammer. Incremental learning algorithms and applications. In ESANN. 2016.
[9] In Gim and JeongGil Ko. Memory-efficiees dnn training on mobile devices. In MobiSys, 2022.
[10] Tyler L. Hayes and Christopher Kanan, Online continual learning for embedded devices. In CoLLAs, 2022.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of- citto--Diitribution Generalization. In ICCV, 2021.
[13] Wei-Cheng Huang, Chun-Fu Chen, and Hsiang Hsu. Ovor: Oneprompt with virtual outlier regularization for rehearsal-free class-incremental learning. In ICLR, 2024.
[14] Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. ICCV, 2023.
[15] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2025.
[16] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkk--ar,,nkka. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 2017.
[17] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.

---

[18] Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[19] Young D. Kwon, Jagmohan Chauhan, Abhishek Kumar, Pan Hui Hkust, and Cecilia Mascolo. Exploring system performance of continual learning for mobile and embedded sensing applications. In JEEF/ACM SEC, 2021.
[20] Soobee Lee, Minindu Weerakoon, Jonghyun Choi, Minjia Zhang, Di Wang, and Myeongjae Jeon. Carm: Hierarchical episodic memory for continual learning. In DAC, 2022.
[21] Gangmuk Lim, Jeongscob Ahn, Wencong Xiao, Youngjin Kwon, and Myeongjae Jcon. Zico: Efficient gpu memory sharing for concurrent dnn training. In USENIX ATC, 2021.
[22] Xinyue Ma, Suyeon Jeong, Minjia Zhang, Di Wang, Jonghyun Choi, and Myeongjac Jeon. Cost-effective on-device continual learning over memory hierarchy with miro. In MobiCom, 2023.
[23] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, 1989.
[24] Mark D McDonnell, Dong Gong, Amin Parvanch, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. In NewrlPS, 2023.
[25] Sharada Prasanna Mohanty, David Hughes, and Mareel Salathe. Using Deep Learning for Image-Based -ased Plant Disease Detection. arXiv:1604.03 2016.
[26] Ameya Prabhu. Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress continual learning. In ECCV, 2020.
[27] Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In CVPR, 2023.
[28] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurlPS, 2021.
[29] Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. CVPR, 2017.
[30] Anurag Roy. Riddhiman Moulick, Vinay K Verma, Saptarshi Ghosh, and Abir Das. Convolutional prompting meets language models for continual learning. CVPR, 2024.
[31] James Scale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilee Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. CVPR, 2023.
[32] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-UCSI Birds 200. Technical Report CNS-TE T0-2011-2110000 Caltech, 2011.
[33] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decompo- sition of prompt-based continual learning: Rethinking obscured sub-optimality. In NewrlPS, 2023.
[34] Qipeng Wang. Mengwei Xu, Chao Jin, Xinran Dong, Jinliang Yuan, Xin Jin, Gang Huang, Yunxin Liu, and Xuanzhe Liu. Melon: Breaking the memory wall for resource-efficient on-device machine learning. In MobiSys, 2022.
[35] Yahin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. In NeurlPS, 2022.
[36] Yue Wang. Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2 train: Training of-h-of-theeart cnns with over 80% energy savings. In NewriPS, 2019.
[37] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Ye Yu Lec, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Dualprompt: Complementary prompting for rehearsal- free continual learning. In ECCV, 2022.
[38] Zifeng Wang, Zizhao Zhang, Chen-Ye Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022.
[39] Xiwen Wei, Guihong Li. and Radu Marculescu. Online-lora: Task- free online continual learning via low rank adaptation. In WACV 2025.
12

---

[40] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning pre-trained model. In ICCV, 2023.
[41] Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progres- sive layer dropping. In NeurlPS, 2020.
[42] Yuqing Zhao, Divya Saxena, and Jiannong Cao. Memory- efficcent domain incremental learning intemet of things. In SenSys, 2022.
[43] Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. model 603 exemplars: Towards memory-efficient class-incemmntte learning. In ICLR, 2023.
[44] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 21Xiv:211..0788 2021.
[45] Huiping Zhuang, Yuchen Liu, Run He, Kai Tong, Ziqian Zeng. Cen Chen, Yi Wang. and Lap-Pui Chau. oal: Forward-only online analytic learning with fast training and low memory footprint in class incremental learning. In NeurIPS, 2024.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We ensure that the claims made in the abstract and introduction accurately reflect our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method in Section 5 including specific learning environments and learning algorithms that affect the broader applicability of this work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limiittions"" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these as- sumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
13

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our method does not require the full set of assumptions and a complete proof. Our method introduces two scheduling algorithms that are intuitive understand.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include detailed explanations of our methods along with pseudo code, how to set hyperparameters, and how to split datasets to create continuous tasks to train in the paper. Nonetheless, we attach the experimental code as a supplementary file for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this. but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
14

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Reference pre-trained models, learning methods, and datasets are publicly available. Our code is zipped and attached supplementary file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/nipsscccppllll
quddes//odd@ubbbissiiooooPo.
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
"No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https://nis) httpe///iiss..ccpul...eee cc/publicc Saadde//CodeSuubnisiiooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe basic experimental methodology in Section 4 in the main paper, and discuss additional details on implementations and hyperparameters in Section B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
15

---

Justification: We calculated the performance for all experiment 5 times, and reported the mean and standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
We discuss relevant details in Section andd Section M.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httt:p///eeurrrpp..ccpp/////hess
Answer: [Ys]]
Justification: We comprehensively read the ethics guidelines and confirm that we do not violate anonymity of the review process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
16

---

Answer: [Yes]
Justification:
We discuss about societal impacts in Section N
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considera- tions, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any new datasets models. We provide simple compute-skipping techniques for existing continual learning methods using pre-trainined models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should
describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators of the publicly available models and datasets by citing the direct links to the websites or original papers where possible.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswi sweorswwthcode...m wtw.odddecoceeeooooooooeoeeeeeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We include our code as an asset in its original form along with an appropriate level of documentation.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submis- sions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not utilize crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not utilize crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not include LLM for developing our core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http://arrpp..coooooeeeeeeeeeeep for what should or should not be described.
19

---

A
Algorithm details
Algorithm 1 Adaptive Token Merging (AToM)
Input: Initial set of all tokens T'; Set of prompt tokens P: Number of model layers L; Maximum number of tokens to merge Tmax Initialize: Timal f- T;
1: for E (1,2.... 11.2....11. _1] do
2: Tatt t (ima)))))))
3:
Teligiblc + Tottn
4:
8 T.
5:
n' min (6 x (1 1), Tmax)
6: T margod + Merge (Talggble: n'
7: Teoncat Concat (T (T mergeas P)
8: T iinal MLP((T concuty 1)
9: end for
10: return TV linal
Algorithm 2 Adaptive Layer Dropping (ALD)
Input: Input tensor X; Keep ratio of layer 0,, Number of layers L; Minimum ratio 8:
Decay rate y, Spatial threshold Adjustment factor
1: for 1 E {1, 2,
L} do
2: if (n( n/ (1) > then
3: (l) + a
4:
else
5:
(1) +-
6: end if
7:
l2t +- all) x ((1 0) exp(-x-
8:
if (BBemoull((f..)) = 1 then
9:
Xou Exec(1, X
10: else
11: Xout Xaut
12:
end if
end for
14:
return Xour
B Implementation details
Unless otherwise stated, we use the term REP-L2P to denote REP applied to L2P with a ViT-L backbone. Similarly, the notations L, B, and Ti used to generally denote ViT-L, VTT-B, and ViT-Ti backbones, respectively, in the prompt update the stage of any prompt-based CL. method.
B.1 Prompt-based methods for REP
We present details on the multiple prompt-based rehearsal-free CL methods to which REP is integrated for enhancing resource efficiency.
L2P positions prompts at the first layer of the transformer architecture [38]. These prompts are learnable parameters that dynamically evolve with the training process. The mechanism begins with a prompt pool P = (p1,20 P2, ..., P.. C DD
For given input oa task T. L.2 computes query feature ((a{) E TR. to select the corresponding prompt. The prompt p" is selected based on maximizing the cosine similarity with respect to the
20

---

query i.e. input data:
(g(x²)) ( [PR]Ten
p argmax Pc PEEP
where (pi](e.)) is the c-th row of pk.
The selected prompt p" is concatenated with the input embedding =(x;) form the prompt- augmented input = 2((x)) = Ip"; =<x))]. The training objective of L2P balances classification loss
Letass and prompt-adjustment loss Lprompt:
L = Letass (2' (x))..) + Lerompe ((nn)()))))))))
(8)
DualPrompt leverages prompts multiple layers of the transformer architecture [37. It introduces a general prompt g E R'ux and a set of task-specific prompts E {ei, 62, 62, er} C R1xx0 These prompts are incorporated at specified layers in the transformer model.
For given input x² in task Ti the model's transformer layers are modified by attaching g and e; the layers, resulting in prompted architecture Jg.e.- The feature transformation lG for the input sample xf is then obtained as:
h2 Sa.e. (x/))-
(9)
Similar to L2P, DualPrompt optimizes Lelsss and Lrrorpni
L = Lelass (hi, v + Lerompp (g. es).
(10)
CODA-Prompt decomposes learnable prompts into components and uses attention mechanism from pre-trained ViT model to select relevant prompts (3I. Instead of a single prompt, CODA-
Prompt learns a of prompt components P (P,, P2, ... PM3. The final prompt p is calculated as weighted sum:
p = Σ am ampms
(11)
m
where weights determined based the query ((x) and keys K RDxM
a= ==>({(x),K)) K).
(12)
When the task changes, the current components are frozen, and new one is added, promoting orthogonality, denoted as:
Tronto B) = BBB -113,
(13)
where represents P, K, or A (A represents the attention vector). The full optimization target is:
min Pa, An an
(ffolfo... + (Lorto P) + Lortho (K) + Lortho (A))
(14)
where Pa K" and A are new components, and À balances the orthogonality loss BI
HiDe--rompt decomposes the CL objective into three parts wiitts-wtthi-ttskk prediction (WTP), task- identity inference (TII), and task-adaetive prediction (TAP) [33] It manages uninstructed/instrueted representations by modeling each class with approximate distributions (e.g., Gaussian means μ.)).
The overall combined loss is composed with WTP loss (Cwre). THI loss (Crn), TAP loss (CtAp) and contrastive regularization loss (CCR):
LIiDe LWTP (p:: + LTm(w;; LTAP Go) + À LCR (pr: (Ma)),
(15)
where p, is the current prompt, th are auxiliary heads, and G... G. denote distributions of unin- structed/instruetedd representations for old classes. This hierarchical approach excels under self- supervised pre-training.
21

---

ConyPrompt generates laser-wise prompts via convolution on a shared embedding matrix SE [30]. For a layer f and head h. it produces M prompt components PC P. by convolving SE and SEy, with mth prompt generator Gt..... The final prompt is computed by weighting these components via projection network and prompt keys:
M
M Σ
P(K) 2+1A
P. E,h,m
P(V 2+1,A
(16)
m=1
m=1
Here, Sgm are similarity scores derived from [CLS] embeddings, enabling a dynamic prompt compo- sition with small overhead.
OVOR employs single prompt throughout all tasks, in contrast to methods that maintain large prompt pool [33] To further refine decision boundaries, introduces virtual outlier regularization (VOR), which generates synthetic outliers Doutlier to penalize overconfident predictions outside the training distribution. OVOR's loss is described as:
LVOr Ex (((5(x) .y) + A AB Doustior hinge( Touttier - -E(v))
(17)
where Dt is the training set for task I, H is a cross-entropy or similar classification loss, ( the model prediction, and y the ground-truth label for x. The term E(v) is an energy- confidence measure, while penalizing overconfidence for outliers. Despite using only one prompt for all tasks, OVOR reports strong results by restricting overconfident predictions and minimizing overhead.
LAE reframes prompts as an instance of parameter-efficient fine-tuning (PEFT), incorporating addi- tional modules (Adapters or LoRA) to mitigate forgetting 7 It accumulates multi-task knowledge through online/offline PET modules and ensembles them at inference. Concretely, it defines an online PET 000 for new tasks and an offline PET pet for older tasks:
pet
goff
a gott
+ (1 - ax) 0ee:
(18)
pet
pet
where is close to 1 (EMA update). During inference, LAE combines the outputs of these two experts to better handle both novel and historical tasks.
B.2 Hyperparameters and configurations
For all prompt-based CL. methods, we employ the ADAM optimizer [1] with hyperparameters 31 = 0.990 and 32 = 0.999, following their original implementation. For prompt selection, we set the prompt pool size to 10 and the prompt length to 5, following the original implementations of L2P, DualPrompt, and HiDe-Prompt. For CODA-Prompt, we implement a cosine-decay learning rate strategy described in [1]. while other prompt- based baselines use a fixed learning rate of 1.875 10-3 by default.
We consistently use the mini-batch size of 16 for all methods to maintain a uniform computational load for each training iteration. For fair comparisons, each method performs 1875, 1080, and 2583 iterations per task for Split CIFAR-100 18), Split ImageNet-R [12), and Split PlantDisease (25), respectively. This setup ensures that among prompt-based methods, iteration time itself correlates linearly with energy usage, as training wall-clock time similarly reflects energy consumption, as discussed in the main paper.
C Competing methods with original setups
For the evaluation of baseline methods in the main paper, we adhere to the hyperparameters specified in the original studies, except for the number of iterations per task and batch size. We adopt smaller batch size to account for on-device memory limitations, which prevent the use of the original batch sizes. Nonetheless, we present the results of REP-L2P, comparing with L2P, DualPrompt, CODA-Prompt, HiDe-Prompt, ConvPrompt, LAE, and OVOR using their original hyperparameters Table and Table respectively. In these experiments, we measure wall-clock GPU time rather than iteration time to ensure a fair comparison across diverse original settings. The results demonstrate that although these methods are based on the ViT-B backbone, most of them consume significantly more memory than REP-L2P, primarily due to their originally large batch sizes.
22

---

Table 9: Results on Split CIFAR-100, with ViT-B as the backbone (following their original setups).
Method
Acc. (1)
Egt. (4)
GPU time (s)
Mem. (GB)
REP-I.2P
87.0+1..3
4.5++0.1
4,421
L2P-B
84.410.7
7.410.44
2,803
14.2
DualPrompt-B
85.340.8
5.210.1
2,624
12.3
CODA-Prompt-BB
86.120.77
1.71:.33
3,227
18.8
BB
92.2+0.4
3.2+0.44
20,169
5.3
ConyPrompt-B
88.6+0.6
3.4+0.44
6,544
14.8
LAE-Pre 10-B
84.8+0.6
4.9
1125
2.8
OVOR-B
85.310.6
4.8.1..8
2,637
8.4
Table 10: Results on Split ImageNet-R, with VIT-B as the backbone (following their original setups).
Method
Acc. (1)
Fgt. (4)
GPU time
Mem. (GB)
REP-L2P
7..
3.6:H..3
2,542
4.5
L2P-B
59.7=0.8
9.7
1,610
14.2
DualPrompt-B
67.5=1.9
4.7:0.22
1,508
12.3
CODA-Prompt-B
70.5++0.7
1.6+0.11
1,854
18.8
HiDe-Prompt-B
75.0++0.5
2.4+0.66
11,588
5.3
ConyPrompt-B
76.3+00.5
3.6+0..5
3,760
14.8
IO-
70.810.8
8.810.9
646
14.8
OVOR-B
73.210.8
4.510.9
1,516
8.4
D Preliminary empirical studies
RN10
DusiP
CODA
MEMO
BudgetOL
Chocipo nts
Maps
Memory (GB)
Up to 1GB
1-GB
4-8GB
1.0
THBLIH
4.0
3.0
Li
0.5
2.0
CIFAR-10 100
100
100
100
Sain
Accuracy (%)
6K
OK
Spit ImageNet/
Wide
IK
5k
10c
CK
10K
204
[K
23k
404
Spint PlamDiseasa
100
100
GPU Time (s)
Figure 5: Cost-accuracy trade-offs various VIT- and CNN-based methods over three different memory budgets: up to 1GB, 1-4GB, and 4-8GB. The memory breakdown of each method is in first row. Experiments on Split CIFAR-100, Split ImageNet-R, and Split PlantDisease are on second, third, and fourth respectively. ViT-based methods consistently outperform CNN-based methods by a large margin.
In this studies, we uncover the cost-accuracy trade-offs to highlight the impact of each baseline method on the accuracy gained relative to its training cost. Here, we limit the memory a maximum of 8GB. We employ L2P (38), DualPrompt [7], and CODA-Promp [11] as representative ViT-based prompting methods. demonstrate the effectiveness of prompting methods, we also incorporate two state-of-the-ard CNN-based methods, such BudgetCI. 27 and MEMO 43 Both BudgetCI. and MEMO improve model accuracy by leveraging spare memory to store replay past samples, with MEMO additionally storing model checkpoints from history. These methods also use ImageNet
23

---

pre-trained models as backbones: ResNet- 10 (RN10; 5M), ResNet- 18(N188; 11M), ResNet-26 (RN26; 14M), ResNet-34 (RN34:22M), and ResNet-101 (RN101; 43M) models.
For BudgetCI. and MEMO, we adopt the SGD optimizer with a learning rate scheduling and a weight decay, following their original papers 27 43 Both methods construct mini-batches of 16 samples, with 8 samples from the new task and 8 from prior tasks in the replay buffer. We adopt the setup in [43] and train for 200 epochs across all datasets. Thus, BudgetCL and MEMO perform more iterations per epoch with a larger replay buffer, resulting in longer training time and higher energy usage. For fair comparisons, we group methods by memory usage.
Figure BJ visually represents the comparison results across device memory capacities and datasets. In each graph, the x-axis indicates GPU wall-clock time, which corresponds to energy cost (lower is better), while the y-axis indicates final average accuracy (higher is hetter). Thus, a more cost-effective method appears closer to the upper-left corner of the graph. Overall, ViT-hased methods outperform CNN-based methods by a large margin, achieving 26-36% higher accuracy with 45-90% less time and energy spent under the same memory budget. We elaborate on two key characteristics that explain these results:
Scaling across varying backhone networks. As we can see in Fig. 5 ViT-based methods scale well, with larger backbone networks overall yielding higher accuracy. In contrast, CNN-based methods scale poorly with increased backbone size. Specifically, on Split ImageNet-R, a more challenging dataset, MEMO/RN34 is only 3.0% better than MEMO/RN18. despite being around twice larger. On Split CI0, increasing the backbone size yields only limited gains.
CNN-based methods are often considered more memory-efficient because they use backbones magnitude smaller than ViT models. Thus, it is commonly believed that ample memory can be allocated to replay samples or the past model checkpoints [43), which help improve model performance. However, in CL, which involves typical training iterations, a substantial amount of memory must be allocated for feature maps (21]. leaving little room for memory buffers. For instance, for device memory range of 1-4GB Table IT the backbone of MEMO/RN18 consumes only 43MB of memory (6.5% of ViT-L), but feature maps occupy as much as 1.7GB (more than ViT- based methods). This causes MEMO/RN18 degrading model accuracy, as observed in Fig. 5
Memory efficiency.
be short on memory for the replay buffer, dramatically
Table 11: ViT- and CNN-based methods with specific backbone models are mapped to the memory capacity they can fit into. DualP DualPrompt, and CODA-P is CODA-Prompt.
Memory capacity
Edge device
VIT- methods
CNN-based methods
4-8GB
Jetson Xavier NX L2P (VTT-L), DualP (ViT-L)
MEMO (RN26, RN34) BudgetCL (RN101)
CODA-P (VIT-B)
1-4BB
Jetson Nano
L2P (VTT-B), DualP (ViT-B) BudgerCI. (RN34)
MEMO (RNIO, RN18)
Up to 1GB
Raspbenyy Pi
L2P (VTT-ii, DualP (ViT-Ti) BudgetCL. (RN10. RN18)
E Extended ablation studies
As the paper mainly discusses the ablation study based on Split ImageNet-R, we here present an ex- tended ablation studies of REP using Split CIFAR- 100 (10 tasks). The results are shown in Table 12 Each row reflects performance with the corre- sponding component removed. Similar the find- ings from Split ImageNet-R, the ablation of any component within REP for the Split CIFAR-100 100 dataset impacts resource efficiency. All compo- nents contribute the reduction of computation time and memory usage, with ATOM particularly standing out in both aspects of resource efficiency.
Table 12: Component ablation for Split CIFAR- 100. Ablating any component of REP results in lower resource efficiency by increasing training time and memory consumption.
Ablated components
Acc. (?) Fg. (1) (ms) Mcm. (GB)
REP-L2P
87.0:1133
4.5200.1
we (AToM ALD)
86.410..4
5.8
349
5.5
(fincen ALD)
86.4-..22
49:08
419
who Urncan AToM)
86.8:0077
5.02004
401
6.5
wio ALD
85.9:10.5
5,0:10.7
270
4.5
AToM
86.1-:0.2
4.7
weo fatean
$6.1
4.0
326
4.8
24

---

Next, we perform algorithm validation on Split CIFAR-100 (10 tasks), for which we present the results in Table 13 Consistent with the Split ImageNet-R results, using static token merg- ing (ToMe) 2 progressive layer dropping (PLD) 41 instead of adaptive token merging (AToM) adaptive layer dropping (ALD) neg- atively affects model accuracy. This experiment further corroborates the efficacy of our proposed algorithms for optimizing resource usage without compromising the model accuracy in CL setup. We further extend analysis from Table 4yy evalu- ating two additional prompt-based CL. methods, including HilDe-rrompl (denoted as REP-HiDeP) and ConyPrompt (denoted as REP-ConvP), with results detailed in Table 14 These results show trend consistent with Table
Table 13: Algorithm validation for Split CIFAR- 100. Changing our algorithms conventional token merging (ToMe) or progressive layer drop- ping (PLD) harms model accuracy.
Acc. (1) Fgt. (U) Iter. time (ms) Mem. (GB)
REP-L2P 87.011.3 4.5
240
4.5
ToMe 83.64:0.1 3.740.66
275
3.7
w/ PLD 84.4+00 4.9.0.88
259
4.5
Table 14: REP over diverse # of merged tokens (n)) and % of keep ratio (0), using HiDe- HiDe-Prompt and ConvPrompt.
Method (w/ 0=0.5)
Acc. (t)
Fgt. (1)
Iter. time (ms)
Mem. (GB)
REP-HiDeP
2
78.6 1.5
2.0+
251
7.3
REP-HiDeP
4
1.1
2.1
245
7.0
REP-HiDeP
8
1.2
2.010.9
227
6.3
REP-ConvP
1.
3.410..5
460
5.5
REP-ConvP
4
7..8=113
3.5+0.66
450
REP-ConvP
8
7..5
3.7+1.11
417
4.1
Method
(w/ n=8)
Acc. (t)
Fgt. (4)
Iter. time (ms)
Mem. (GB)
REP-HiDeP
0.25
77.8
2.410.8
REP-HiDeP
0.50
78.0=112
2.010.9
227
6.3
REP-HiDeP
0.75
77.9+0.9
2.3+0.7
255
6.3
REP-ConvP
0.25
78.2+1.11
3.9+0.4
387
4.1
REP-ConvP
0.50
78.5+0.8
3.7+1.1
417
4.1
REP-ConvP 0.75
78.3
4.0.1..2
469
F
Diverse task sequence lengths
We examine the scalability and robustness of our approach by employing two task sequence lengths, including 5 and 20 tasks, on the Split CIFAR- 100 and Split ImageNet-R datasets, respectively. Longer task sequences simulate real-world scenarios more accurately, where CL methods require frequent model updates. The results are shown in Table 15 and T6ble[6] respectively. Across all scenarios, method shows consistent preservation of both model accuracy and resource efficiency.
Table 15: Results on Split CIFAR- organized as tasks (5T) and 20 tasks (20T), respectively.
Accuracy (?)
Forgetting ())
time
Memn (GB)
CIFAR-IOU
Splin
(20T)
Split 100AR-100 (ST)
Split CIFAR-100 (20T)
REP
w/o REP
w/ REP
w/o REP
w/ V/REP
REP
w/ REP
w/o REP
w/ REP
w/o REP
REP
L2P-L
88.72033
88.34009
84.210.6
83.9-0.9
4.4+0.3
4.8:00.7
6.0:1:.11
6.0s:0.2
447
240
6.5
4.5
DusiPromge
87.910.5
87.210.9
83.710.7
83.0..0.9
4.410.2
4.710.8
6.3110.6
6.810.99
424
208
CODA-Prompe
87.911.1
839910.
83.0.0.88
6.011.22
Prompt-
94,1209
93.741.4
91.910.9
91.4-1.2
15440.6
188110
1.940.88
2.12099
413
227
CowyPrompt
$9.910.7
$9.7110
88.110.6
87.611.22
2210.77
2.410.9
2.7.1..7
3.0.0.88
560
417
5.4
4.1
LAE LAE-Prefix
10-L
85.610.6
85.3109
832110.5
82.11.9
1.510.7
4.71100
6.210.33
6.510.99
185
170
3.611.0
3.290.9
G Additional datasets
We also expand our experiments on REP by incorporating an additional dataset: Split CUB-200 32
which is designed for fine-grained image classification. Split CUB-200 contains 5,994 bird images
25

---

Table 16: Results Split ImageNet-R organized as 5 tasks (5T) and 20 tasks (20T), respectively.
Method
Accuracy
Forgetting ())
Iter. time
Mem. (GB)
Split ImageNes-R (ST)
Split ImageNet-R (20T)
Split ImageNes-R (ST)
Split ImageNer-RR (20T)
wio REP
REP
wio REP
REP
REP
REP
728-00
4.7+0.55
DralPromptt
73.1400.
72.5+0.9
09:5-04
090011
2.2+0.5
2.6:13
5.0:0.9
5.8=0..9
424
5.9
CODA Prompx L
76.820.66
76.021.1
73.4=02
72.7=111
1120.11
1.2009
5.24:.33
5.9.0.99
568
441
13.2
11.0
HiDe Prompe
79.110.5
76.611.2
78.0.0.6
77.5-1.3
1.710.5
2,120.9
2.310.6
2.611.1
413
227
7.3
6.3
ComPPomp--
79.5110
78.3-055
778-1.1
30109
2.910.9
71,7408
71441.
698-007
69.3-09
41203
43:009
50::04
$4:009
OVOR-I.
76.7408
75.9+1.3
74.6-05
73.9-0.
5.1+07
3.+110
4.6+0.6
5.1+0.9
467
362
7.5
6.8
categorized into 200 classes, which are split into 5 tasks, each having 40 classes. As shown in Table 17 integrating REP achieves resource efficiency by bringing significant reductions in both training time and memory usage.
Table 17: Results on Split CUB-200, using ViT-L as the backbone model.
Method
Acc. (t)
Fgt.
Iter. time (ms)
Mem. (GB)
w/o REP
w/ REP
w/o REP
w/ REP
w/o REP
w/ REP
w/o REP
w/ REP
L2P-L
7..7-0.9
74.7+110
6.7.1144
6.7.10.3
447(x1.9)
240
6.5(x
4.5
DualPrompt-LL
7.94=0.9
72.1
7.8-
7.9.:0.8
424(x2.0)
208
5.9(x1.4)
4.3
CODA-Prompt-LL
7.8
7..0
5.8:1099
6.0:10.7
568(x
441
13.2(x
11.0
HiDeePrompt-L
86.8=0.5
86.6=110
1.8+0.8
1.8+1.3
413(x188)
227
7.3(x1.2)
6.3
Conv Prompt-L
82.7-
82.2.10.9
5.3:1088
5.7.10.9
5.0(x
417
5.4(x13)
4.1
LAE-Prefix10-L
0374=007
73.0+112
8.5:008
8.8:10.7
185(x11))
170
5.1(x1.1)
4.8
OVOR-L
7..6
78.0
5.9:0.99
6.3:10.9
467(x1..)
362
7.5(x1.1)
6.8
H Supplementary evaluations
Self-supervised pre-training, We evaluate REP with HiDe-Prome Prompt (33 under self-supervised
Table 18: Effect of REP with HiDe-Prompt under
self-supervised pretraining paradigms.
pre-training paradigms, including iBOT21K [44]
and DINO B1, using the VIT-B backbone Split ImageNet-R (10 tasks). As shown in Table [8]
Method
Acc. (t) Fgt. (L)
Iter. time (ms) Mem, (GBB
HiDe-iBOT 71.5::0.3
1.40.4
130
4.1
REP reduces training time and memory usage by 14-18% with only 0.6-0.8% accuracy drop while forgetting remains improved.
REP-iBOT 70.7++0.9
0.7-0.7
76
2.4
HiDe-DINO 68.6+:022 1.8=0.2
130
4.1
REP-DINO 68.0++0.8 0.7-0.7
2.4
Comparison with analytic learning- beaseing-based CL method. We evaluate F-OAL [45 as the represen- tative analytic learning-based CL. method Split CIFAR- 100 (10 tasks). Table 19 demonstrates that REP-L2P can achieve compatible accuracy, with shorter training time. Since F-OAL is an on- line method, a perfect 1:1 comparison is tricky. Nonetheless, these results suggest that REP can be competitive with F-OAL's energy efficiency while maintaining similar accuracy. Each method has distinct advantages and disadvantages depending on the context.
Table 19: Comparison REP-L2P with F-OAL
on Split CIFAR-100.
Method
Acc. (f) Fgt. (1) Iter. time (ms) Mem. (GB)
REP-L2P 87.0+1.3 4.5+0..1
240
4.5
F-OAL 87.6.11.4 5.1.0.3
359
1.6
I Extended main results including forgetting metric
We present the extended results in Table including the forgetting metric. The result is shown in Table 20
26

---

Table 20: Forgetting and computational cost of all competing methods Split CIFAR-100 100, Split ImageNet-R, and Split PlantDisease datasets, We report forgetting, iteration time, and memory usage both without and with REP. Iteration time and memory usage are also shown absolute values, with their multiples indicating how many times larger than REP.
Forgetting (1)
Ier. time (ms)
Mem. (GB)
Model Method
Split CIFAR- 100
Split ImageNet-R
Split PlantDe
w/o REP
REP
w/o REP
REP
weo REP
REP
w/o REP
REP
w/o REP
REP
L.P
4.740.11
4.5:1.11
2.8
3.4
13.24338
8.74:2.1
447(
240
6.5(x
4.5
DualPrompt
4,5+00.7
4.9+0.99
3.8=0.4
4.1-0.9
14.8+3.0
10.1+3.7
424 x2.(x2.0)
208
5,9(x 1.4)
4.3
CODA-Prome
4.310..1
4.9:10.8
3.7=0.8
4.3-18
14.92299
15.1=3.0
x1.3)
441
13.2(x1.2)
11.0
HiDe-Promp
1.740.6
1.8+111
2.0-0.5
20-009
1.0+088
1,141.3
x13(x1.8)
227
7.3xx12)
6.3
VIT-L
CouvPrompt
3.0:10.7
3.141.9
3.420.8
3.7=1.1
1.240.5
1.241.0
560(
417
5.4(x13)
4.1
AAE.Prefix
5.5++1.0
15.9+1.4
x85(x1.1)
5.1(x11)
4.8
OVOR
2.9:0.8
3.14:14
4.6=0.9
4.9-18
14.5:2.5
14.9=3.1
467( x..3)
362
7.50x11))
6.8
UpperBound
427
9.8
L2P
6.640.8
6.740.9
6.6=1.2
6.1=1.2
25.942.8
23.5=2.9
143(x
102
2.3(x1.2)
2.0
DualProopt
5.6+0.3
5.640.7
3.7+0.3
4.3-1.1
10.5+119
x33 10.9+2.0
101
2.1(x1.2)
1.8
CODA-Prompt
6.040.22
6.541.5
6.1=0.8
6.9-09
18841111
19.1=2.9
164 1.1)
151
6.9(x12)
5.7
HiDe Prompt
1.640.8
1.6114
1.91.3
1.2+10
76
4.10x17)
2.4
VIT-B
ConyPrompt
3.740..6
3,9+166
5.5-0.7
5.9-14
1.6+088
18+1.5
381(x1.4)
279
2.2(x111)
2.0
AAE-Prefix
6.8:1.99
5,910.7
9.3:09
9.2-1.0
1699:88
16.9=1.2
570xL.1)
50
1.9XLD)
1.8
OVOR
5.740.6
5.9+1.5
4.9-0.7
5.5-144
208+009
21.4-166
1340x1.1)
123
3.1(x.11)
3.0
UpperBound
143
3.2
L2P
16.3409
168+1.11
9.-009
10.5+11
31.243.1
29.9-441
34(x1.1)
30
0.5(x1.))
0.5
DualPrompt
15.742.
15.843.0
8.920.8
9.2-100
25.0=2.2
26.12288
34(xL1)
31
0,4(x1)
0.4
CODA-Promp
12.9+1.6
13.2+2.3
13.440,7
14.0+0.8
23.943.3
24.0
16(x1.2)
30
2.8(x11)
2.8
HiDe-Prompt
6.840.77
6.641.3
2.2-0.5
19=1.
1,0008
1.0004
23(x1.2
20
0.5(xLD)
0.4
VIT-Ti
CommPrompt
10.3+0.77
11,14188
14,14088
15.0+1.
1.5+09
1.000.7
850x1.9
45
0.5(x1.1)
0.5
LAE-Prefix-D 10
17.1+0.3
17.1+09
17.040.5
1788:0.9
27.6:08
27.9114
15(x13)
12
0.(x1D
0.4
OVOR
12.0#1.5 129427
128408
13.141.5
188+39
35(x11)
32
1.0XLI)
UpperBound
42
0.6
J Additional related works
J.1 Online or few-shot CL
In on-device CL. the memory capacity for storing data from new tasks may be limited. Thus, one might consider adopting online or few-shot CL methods save memory costs, as they require fewer new-task samples to be maintained in memory. Offline CL (our setting) typically yields better accuracy with higher resource usage. However, memory allocation for new samples not major constraint in offline CL. All prompt-based methods require 355-269MB to store new samples for both Split CIFAR- 100 and Split ImageNet-R, accounting for only 3.4-5.96 of REP-L2P's memory usage. Moreover, modern DL frameworks can store data in storage and proactively prefetch upcoming mini-batches for training, virtually eliminating memory concerns for new samples.
J.2 Advantages of REP over NAS
While NAS (Neural Architecture Search) can be integrated into REP to drop some layers, usually involves searching the model from scratch, which is time- and computation-consuiingg In contrast, our method applies adaptive schedules directly to an existing pre-trained model, significantly saving computation time. So, it can naturally adapt to device-internal resource conditions without relying on external server resources.
K
Impact of varying hyperparameter values
Guided by the hyperparameter tuning strategies used in DualPrompt B7 and CODA-Prompt B1] we fine-tuned key hyperparameters for our ALD approach using REP-L.P. Our focus is primarily on optimizing the threshold parameter (a) and the adjustment factor (r), which are crucial for ALD's adaptability and efficiency. We conducted a hyperparameter search using 5-fold cross-validation. for which we designate 20% of our training dataset validation set. Table 21 summarizes the results.
27

---

We further evaluate the impact of varying prompt-selection-rlane hyperparameters using REP-L2P on subsets of 5-datasets dataset introduced [8]]. Table 22 shows that the primary factor driving computation costs is prompt length, as it introduces more learnable parameters (e-g., with pool size of 10, increasing prompt length from time by 14%). In contrast, increasing the prompt pool size has almost no impact on training time and memory usage.
increases memory usage by 500 MB and training
Table 21: Results of ALD's hyperparameter tuning via 5-fold cross-validation.
T
10
14
15
17
0.6
74.32
74.62
74.75
74.45
74.87
74.99
74.93
75.12
74.82
74.82
0.7
74.17
74.38
74.45
74.98
74.18
75.06
75.15
75.26
74.43
74.31
0.8
74.48
74.50
74.41
74.36
74.87
75.29
74.91
75.20
74.99
74,39
0.9
74.49
74.36
74.77
74.38
74.58
75.33
75.34
75.42
74.88
74.96
0.95
74.45
74.76
74.39
74.54
74.69
75.39
75.32
75.38
74.76
75.13
Table 22: Effect of prompt pool size on subsets of 5-datasets. The prompt length is 10 by default.
Pool size Acc. (f) Iter. time (ms)
Mem. (GB)
10
62.73
240
4.5
15
63.07
240
4.5
20
63.40
240
4.5
63.78
241
4.5
L REP under varying computation budgets
We compare REP-L.2P across various computation budgets defined by the number of iterations per task. The comparison results using Split CIFAR-100 (10 tasks) are shown in Table 23 Evidently, the accuracy of REP-L2P is comparable to that of L.2P-L across six computation budgets, with the performance gap getting increasingly marginal lower compute budgets. We observe that the overall accuracy for both methods does not drop significantly with reduced compute budgets. This attributed to the fact that Split CIFAR-100 is considered a less complex CL benchmark.
Table 23: Impact of REP under various computation budgets (i.e., number of iterations per task).
Method
313
625
938
1250
1563
1875
REP-L2P
84.1
85.5
86.6
86.7
86.6
86.9
L2P-L.
84.4
86.3
87.3
87.8
87.8
88.2
M REP on edge devices
Miro [22 introduces dynamic approach for fine-tuning key design parameters of rehearsal- based CL methods to achieve high model accuracy while simultaneously reducing energy costs, i.e., high cont-effectiveness. Miro's 's methodology centers identifying the suitable memory size within the device's s capacity to accommodate both old and new samples for training. We compare REP-L.2P with Miro directly reference edge device, NVIDIA Jetson TX2. This device is equipped with 256-core NVIDIA Pascal GPU, 8GB of RAM, and a 32GB eMMC 5. drive, The RAM is a single unified memory shared by the CPU and GPU. We report energy usage as joules (J) obtained by multiplying power by time. Power usage is measured by reading the built-in sensor values provided by Jetson devices for the GPU, RAM, CPU, and I/O connection.
M.1 Power usage breakdown
When implementing CL. on edge devices, the majority of power consumption that influences energy usage during the training of a new task comes from GPU operations. Figure shows power con-
28

---

sumption on NVIDIA Jetson TX2 across individual system components. Static power is measured when the system inactive, while dynamic power measured during the training of a ViT-B model. During the training process, power usage surges up to 6.5x, with the GPU contributing 60% to the dynamic power.
9
Power (W)
Static
Dynamic
6
3
Total
GPU
MEM
CPU
Other
Figure 6: Power breakdown for training VYT-B model on NVIDIA Jetson TX2.
M.2
REP vs Miro
In this experiment, we explore enery--accurae trade-offs REP-L2P and Miro. Figure visually represents the comparison results. In each graph, the x-axis signifies total energy usage (lower is better), while the y-axis signifies final average accuracy (higher is better). Hence, a more cost-effective strategy is positioned closer to the upper-left corner of the graph. Regarding Miro, we maintain the hyperparameters as suggested in the original work [22] but incorporate the use of pre-trained ResNet- 18 (RNI8; IIM), ResNet-34 (RN34; 22M), and ResNet-50 (RN50; 25M) models instead of non-pre-trained ones to enhance overall performance. For REP-L2P, we vary the number of training iterations per task insertion to match the energy usage of Miro variants with different ResNet models.
When operating within the same energy budget, REP-L2P consistently outperforms Miro variants, achieving 22-33% higher accuracy across datasets. REP-L2P demonstrates superior cost-effectivenness especially on Split ImageNet-R (10 tasks) compared to Split CIFAR- 100 (10 tasks). Both REP-1.2P and Miro prove to be memory-efficient to some extent as they fit comfortably within the on-device memory capacity. Although we explore larger ResNet models for Miro, we do not observe the accuracy levels comparable to REP-L2P for either dataset. This experiment also underscores the importance specifically optimizing vision transformers for on-device CL scenarios advance performance boundaries.
MirD
REP Split ImageNet-R
Split CIFAR-100
90
Accuracy (96)
6.-
RN34
RN50
40
40
150
250
350
450
200
350
500
(8)
Energy (k))
Figure Energy-accuracy trade-offs between REP-L2P and Miro on Split CIFAR-100 (10 tasks) and Split ImageNet-R (10 tasks) To provide a spectrum of mergy-accuracy trade-offs, we use pre-trained ResNet-18 (RN18), ResNet-34 (RN34), and ResNet-50 (RN50) for Miro.
N Societal impacts
The proposed method widely promotes AI for robotics agents and surveillance systems with high efficiency. Once the edge AI is easily deployable by the proposed method, the system may be used for monitoring unwanted mass populations. Then, private information such as identity, clothing information, and personal attributes (e.g., age, gender, etc.) could be obtained by adversaries. Although our method has no intention foster such problematic cases, we not currently propose secure solutions to prevent them from happening.
29


Reviews:
[{"review_id": "yQJOsgBDyH", "reviewer": "Reviewer_mcMq", "summary": "The authors addressed the resource efficiency in continual learning with strong pretrained backbones (e.g., ViT). The authors first analyze the potential computation redundancy in prompt selection and prompt updates, and then propose solutions for each of them. For prompt selection, the authors propose to use a lightweight surrogate model and a random projection to replace the full pass on the backbone model for obtaining the query feature. For prompt update, the authors first analyze the frozen backbone to highlight the richer information contained in shallow layers. Therefore, the proposed Adaptive token Merging (AToM) preserves prompt tokens and merges more aggressively in deep layers. Similarly, the proposed Adaptive layer dropping (ALD) also drops more aggressively in deep layers. Extensive experiments showed that the proposed method can effectively reduce the computation cost with a moderate performance drop.", "questions": "1. The hyperparameter comparison can be improved\n\nIn the hyperparameter comparison in the Appendix (which is not referred to in the main paper, I would encourage the author to add this reference), the parameter alpha varies between 10 and 16, which is a rather small range. I am wondering how it would impact the performance if we test with a wider range.\n\n2. The presentation of ablation can be improved\n\nThe ablation table is not easy to follow. I believe the row (1)-(6) shows the performance without the corresponding components. Please correct me if I'm understanding it wrong. First, there is no baseline performance as a reference. Second, it is unclear why (6) is more computationally efficient than (2) and (3).\n\n3. About the prompt selection\n\nDid the authors try to use the query feature from the intermediate layers of the f_query or f_efficient and project it with the random projection? Would this be an alternative to using a surrogate model?\n\nOverall, I believe this is a good paper with good motivation, interesting analysis, and effective method design. It would bring insights to the community for a more resource-efficient CL.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well motivated and addresses an interesting problem of computation efficiency\n\n2. The paper is well-written and easy to follow \n\n3. The proposed method is well motivated and simple, but effective\n\nWeaknesses:\n\nI don't see many significant weaknesses in this work. There are a few minor remarks\n\n1. The hyperparameter comparison can be improved\n\n2. The presentation of ablation can be improved\n\nPlease see details in the Questions section.", "was_revised": true, "final_justification": "I thank the authors for the detailed rebuttal and believe that they have properly addressed all my concerns. Therefore, I maintain my initial positive rating.", "rebuttal": "We appreciate your detailed review and constructive comments, which are addressed below.\n\n> ‘Q1: The hyperparameter comparison can be improved’\n\n**[Response]**\nThank you. We conducted experiments with a broader range of parameter values (alpha=0.6--0.95 and tau=6--20) and observed similar trends as before, as shown in the table below. The notation for alpha and tau was inadvertently swapped in Supplemental Figure 8 of our submission. We apologize for the mistake. As noted in line 682, hyperparameters were automatically tuned via five-fold cross-validation on the training set. The complete set of results will be included in the final version.\n\n| α \\\\ τ | 6 | 8 | 10 | 12 | 14 | 15 | 16 | 17 | 18 | 20 |\n|-------:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n| 0.6  | 74.32 | 74.62 | 74.75 | 74.45 | 74.87 | 74.99 | 74.93 | 75.12 | 74.82 | 74.82 |\n| 0.7  | 74.17 | 74.38 | 74.45 | 74.98 | 74.18 | 75.06 | 75.15 | 75.26 | 74.43 | 74.31 |\n| 0.8  | 74.48 | 74.50 | 74.41 | 74.36 | 74.87 | 75.29 | 74.91 | 75.20 | 74.99 | 74.39 |\n| 0.9  | 74.49 | 74.36 | 74.77 | 74.38 | 74.58 | 75.33 | 75.34 | 75.42 | 74.88 | 74.96 |\n| 0.95 | 74.45 | 74.76 | 74.39 | 74.54 | 74.69 | 75.39 | 75.32 | 75.38 | 74.76 | 75.13 |\n\n> ‘Q2: The presentation of ablation can be improved’\n\n**[Response]**\nThank you. The ablation results in Table 2 were reported additively from the baseline L2P-L, rather than subtractively from our final design. Our algorithm integrates three key components: (a) surrogate model f_efficient, (b) adaptive token merging (AToM), and (c) adaptive layer dropping (ALD). In Table 2, entry `(2) AToM' indicates that only AToM was used among the three components, which explains why configuration (6) is more computationally efficient. We will revise the table so that each row reflects performance with the corresponding component removed. The updated table will be as shown below. \n\n| Ablated components          | Acc. (↑) | Fgt. (↓) | GPU Time (s) | Mem. (GB) |\n|-----------------------------|---------:|---------:|-------------:|----------:|\n| Final design (REP-L2P) | 75.3±1.5 | 3.6±0.3  | 2542.4       | 4.5       |\n| (1) w/o (AToM+ALD)   | 74.8±0.8 | 4.0±0.8  | 3698.8       | 5.5       |\n| (2) w/o (f_efficient + ALD) | 74.9±0.8 | 3.5±0.8  | 4442.0       | 5.6       |\n| (3) w/o (f_efficient + AToM)  | 74.5±0.8 | 3.6±0.7  | 4244.0       | 6.7       |\n| (4) w/o  ALD | 74.2±0.8 | 4.0±0.8  | 2861.4       | 4.5       |\n| (5) w/o  AToM | 74.5±0.4 | 3.4±0.5  | 3206.7       | 5.5       |\n| (6) w/o f_efficient | 74.6±0.4 | 2.6±0.5  | 3448.8       | 4.8       |\n\n\n> ‘Q3: About the Prompt Selection’\n\n**[Response]**\nThank you for the suggestion. We had not initially tested this approach, but your question prompted us to conduct experiments, with results shown in the table below. Specifically, we evaluated prompt selection using CLS tokens from ViT-L’s intermediate layers (l = 4, 8, 12). While using intermediate layers reduces the need for a separate surrogate, it increases GPU time since even a partial ViT-L forward pass (to l=8, covering ~67% of layers) is more computationally expensive than running the full ViT-Ti (5M vs. 307M params). Accuracy also drops for shallower layers (e.g., l=4: -1.3%) due to reduced global context. These findings demonstrate the efficiency advantage of our surrogate model. We will include this analysis in the final version.\n\n| Method | Acc | Fgt | GPU Time (s) | Mem (GB) |\n|------------|------------|------------|------------|------------|\n| Final design (surrogate model) | 75.3±1.5 | 3.6±0.3 | 2542.4 | 4.5 |\n| ViT-L interm. layers (l=4) | 74.0±0.8 | 3.9±0.9 | 2792.9 | 4.8 |\n| ViT-L interm. layers (l=8) | 74.3±0.7 | 3.5±0.9 | 3022.8 | 4.8 |\n| ViT-L interm. layers (l=12) | 74.6±0.7 |2.6±0.8 | 3253.4 | 4.8 |"}, {"review_id": "GSKAHYwSnj", "reviewer": "Reviewer_tV8e", "summary": "This paper introduces REP (Resource-Efficient Prompting), a framework designed to improve the computational and memory efficiency of prompt-based, rehearsal-free continual learning (CL) methods for vision tasks. The authors propose three complementary techniques: (1) a lightweight surrogate model for swift prompt selection, (2) Adaptive Token Merging (ATOM) to intelligently reduce the number of input tokens while preserving task-specific prompt information, and (3) Adaptive Layer Dropping (ALD), which dynamically skips transformer layers based on the token merging ratio from ATOM to reduce computational load. Through extensive experiments on various datasets and Vision Transformer (ViT) backbones, the paper demonstrates that REP can significantly reduce iteration time and memory usage (by up to 51% and 48% respectively) with only a minimal trade-off in model accuracy.", "questions": "Please address the concerns raised in the \"Weaknesses\" section.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses the critical and highly relevant challenge of deploying continual learning models on resource-constrained edge devices. \n2. The empirical evaluation is exceptionally thorough. The authors test REP across three datasets, three ViT backbone sizes, and integrate it into seven different state-of-the-art prompt-based CL methods. \n3. The framework delivers substantial improvements in resource efficiency for a negligible drop in accuracy. \n\nWeaknesses:\n1. The core technical components, while cleverly adapted, are not fundamentally new concepts. The methodology relies on adaptive versions of existing techniques like Token Merging and Progressive Layer Dropping. While the paper's contribution lies in the intelligent, adaptive scheduling and combination of these techniques for the CL context, the work could be perceived as more of an engineering or systems contribution rather than a fundamental algorithmic breakthrough.\n2. The proposed methods, particularly ALD, introduce new hyperparameters such as the merging threshold τ and the adjustment factor α. While the appendix provides a hyperparameter search, the paper lacks clear intuition or guidelines on how to set these parameters for new datasets or model architectures without performing expensive, task-specific tuning. This could potentially limit the \"plug-and-play\" applicability of the framework.\n3. The paper notes that in some cases (e.g., L2P with ViT-L), REP can even improve model accuracy. The authors hypothesize that this might be because ALD and ATOM help eliminate redundant computations, effectively acting as a form of regularization. This is a very interesting phenomenon, but the paper fails to provide a deeper analysis to support this hypothesis. It is unclear whether this effect is due to the suppression of noisy features or some other mechanism. Adding some visualizations or quantitative analysis to investigate this would have significantly enhanced the paper's depth.", "was_revised": true, "final_justification": "I appreciate the authors’ thorough rebuttal and find that they have satisfactorily addressed all of my concerns. I therefore uphold my original positive evaluation.", "rebuttal": "Thank you for your detailed review and constructive comments. Here we address your initial concerns below.\n\n> ‘Q1: The core technical components, while cleverly adapted, are not fundamentally new concepts. The methodology relies on adaptive versions of existing techniques like Token Merging and Progressive Layer Dropping. While the paper's contribution lies in the intelligent, adaptive scheduling and combination of these techniques for the CL context…’\n\n**[Response]**\nWe appreciate the reviewer's observation that REP incorporates ideas related to Token Merging and Progressive Layer Dropping. However, REP introduces new algorithmic elements that go beyond simply reusing these techniques: REP introduces adaptive mechanisms for both token merging and layer dropping that are specifically designed for rehearsal-free continual learning. AToM selectively excludes prompt tokens and progressively merges tokens more aggressively in deeper layers to preserve task-specific features. Similarly, ALD dynamically adjusts layer-dropping probabilities based on merged token counts and temporal schedules. Naively applying prior static approaches results in significant accuracy drops (Table 3), whereas these tailored mechanisms maintain accuracy while improving efficiency.\n\nIn addition to these algorithmic components, REP offers a framework-level contribution by integrating them into a unified system applicable to diverse scenarios. It generalizes beyond prompting methods to non-prompting and self-supervised continual learning approaches (Sec. 4.3), consistently reducing computation and memory usage (up to 48% and 31%, respectively) with only minor accuracy degradation. REP is also instantiated on a real edge device (Appendix K), demonstrating its practical viability for on-device continual learning.\n\nWe believe that by combining these algorithmic-level components with a framework-level contribution that is both generalizable and practically validated, REP represents a novel and comprehensive solution for resource-efficient continual learning.\n\n> ‘Q2: … introduce new hyperparameters …clear intuition or guidelines on how to set these parameters …without performing expensive, task-specific tuning… limit the \"plug-and-play\" applicability of the framework.\n\n**[Response]**\nThank you for raising this point. Following prior work (e.g., [1]), we tuned the hyperparameters through grid search with 5-fold cross-validation for each task. While this provides an automatic tuning procedure, we agree with the reviewer that it may limit the “plug-and-play” applicability of REP. This limitation is not unique to our method and is shared by many existing continual learning approaches.\n\nEmpirically, we observed that performance is relatively insensitive to variations in alpha, making it reasonable to fix alpha=0.9. For tau, the impact is more dataset- and model-dependent, but we found a consistent positive correlation between backbone size and the optimal tau value (see also our response to Reviewer mcMq, where we have a broader range of experiments). We will explicitly state this limitation in the final version and suggest a more systematic analysis of hyperparameter tuning as a future work direction.\n\n\n> ‘Q3: The paper notes that in some cases (e.g., L2P with ViT-L), REP can even improve model accuracy. The authors hypothesize that this might be because ALD and ATOM help eliminate redundant computations, effectively acting as a form of regularization. This is a very interesting phenomenon, but the paper fails to provide a deeper analysis to support this hypothesis. It is unclear whether this effect is due to the suppression of noisy features or some other mechanism. Adding some visualizations or quantitative analysis to investigate this would have significantly enhanced the paper's depth.’\n\n**[Response]**\nThank you for noting this phenomenon. We agree that a deeper analysis would strengthen the paper. To this end, we conducted additional experiments analyzing activation variance as a proxy for noisy features in the backbone model's representations. Specifically, we measured the normalized variance of token norms in the last layer. Using Split ImageNet-R (10 tasks) with ViT-L as the backbone, we compared REP-L2P with the baseline L2P-L.\n\n| Method | Acc | Fgt | Norm. Variance |\n|------------|------------|------------|------------|\n| L2P-L | 75.6±1.0 | 2.8±0.2 | 0.29  |\n| REP-L2P | 75.3±1.5 |3.4±0.7 | 0.20 (-30%) |\n\nREP reduces average activation variance by 30%, suggesting that AToM and ALD suppress noisy features, which likely contributes to the observed accuracy improvements in certain settings. We propose to include this preliminary analysis, supporting the regularization effect introduced by REP, in the supplemental material.\n\n\n\nReference\n\n[1] Veniat et al., Efficient continual learning with modular networks and task-driven priors, ICLR 2021."}, {"review_id": "2uflUqfzDC", "reviewer": "Reviewer_jSzb", "summary": "This paper proposed a EFCIL method that based on prompt tuning. REP consists of two parts: The Adaptive Token Merging AToM and Adaptive Layer Dropping ALD.\nThe work first show that ViT behave in away that earlier layer focus on more local features while deeper layers focus on global information by mean attention distance analysis.\nAToM dynamically merge tokens based on task specific information and layer specific information to allow more aggressive merging in deeper layers. To improve computation efficiency, AToM relies on a smaller model i.e ViT tiny to compute query feature for prompt selection in large model i.e ViT Base\nALD meanwhile adaptively drop information in ViT layed based on number of merged tokens in AToM. This allows more aggressive dropping in later layers as earlier layer contains richer features information.", "questions": "what is the baseline method  used in Table 4? Is it possible to show graph of ACC and forgetting of different n and theta for different CL method when combined with REP?\nFor figure 4. What happen if Random drop have different aggressiveness? Will it have a more closer performance compared to ALD?\nFor AToM, as shown in Figure 3, gradient w.r.t prompt is significant and contribute essential part to CF? Is it possible to improve AToM by involving Fisher Information Matrix like in EwC to have a gradient informed prompt merging?", "limitations": "The authors have thoroughly discussed limitations of REP in Section 5.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\nThis work show good motivational experiment by showing ViT layers behaviour in mean attention distances which justify the need of more aggressive AToM and ALD. The ablation studies and comparison to other dropping strategies also further solidifies the claim as ALD perform well. Proposed method also achieve at least 10% faster computational time and 10% less memory consumption. The proposed method is tested in large number of different CL methods which proven it's adaptability and modularity. This work shows that it is possible to use lightweight backbone to perform approximated query selection on huge model, which is potentially beneficial for future works.\nWeakness\nREP performance in average accuracy drops compared to vanilla methods. Only very small amount of cases where REP actually help the AA. This raises the question on whether the computational speed and GPU memory saving is significant enough to justify the drop in performance.", "was_revised": true, "final_justification": "Dear Authors, thanks for your response and updates. It's not fully clear if the trade-off between performance drop and computational gain is worthwhile across all settings. so I will keep my score as borderline reject.", "rebuttal": "Thank you for your detailed review and highlighting both the strengths of REP and areas for improvement.\n> This raises the question on whether the computational speed and GPU memory saving is significant enough to justify the drop in performance.\n\n\n**[Response]**\nWe acknowledge that in specific setups (e.g., CODA-Prompt), REP might cause a 1–2% accuracy drop in exchange for noticeable efficiency gains. In many other cases, however, the drop is negligible; for instance, in Split PlantDisease, REP even outperformed the baseline. This shows that the performance impact varies across tasks and baseline methods. Importantly, in edge-centric contexts, a small accuracy gap is often outweighed by the substantial energy and memory benefits, making the trade-off justifiable.\n\n> ‘Q1: What is the baseline method used in Table 4? Is it possible to show graph of ACC and forgetting of different n and theta for different CL method when combined with REP?’\n\n**[Response]**\nIn the main paper, our ablation study primarily uses L2P without REP as the baseline CL method on Split ImageNet-R with a ViT-L backbone (additional results on Split CIFAR-100 are provided in Appendix H). Below, we present extended results for Table 4, including other CL methods such as HiDe-Prompt (‘HiDeP’) and ConvPrompt (‘ConvP’). These results demonstrate REP’s effectiveness across diverse CL methods: accuracy drops by less than 1% for moderate settings (n=4-8, θ=0.5-0.75), while forgetting remains stable or improves thanks to adaptive merging and dropping that preserve task-specific features. See Section 4.3 of the main paper for further studies, including non-prompting and self-supervised CL methods for REP’s generalizability.\n\n| Method | n (w/ theta=0.5) | Acc | Fgt | GPU Time (s) | Mem (GB) |\n|------------|------------|------------|------------|------------|------------|\n| HiDeP | 2 | 78.6±1.5 | 2.0±0.3 | 2654.8 | 7.3 |\n| HiDeP | 4 | 78.3±1.1 | 2.1±0.8 | 2597.1 | 7.0 |\n| HiDeP | 8 | 78.0±1.2 | 2.0±0.9 | 2404.7 | 6.3 |\n| ConvP | 2 | 79.1±1.0 | 3.4±0.5 | 4876.8 | 5.5 |\n| ConvP | 4 | 78.8±1.3 | 3.5±0.6 | 4770.8 | 5.1 |\n| ConvP | 8 | 78.5±0.8 | 3.7±1.1 | 4417.4 | 4.1 |\n\n| Method | theta (w/ n=8) | Acc | Fgt | GPU Time (s) | Mem (GB) |\n|------------|------------|------------|------------|------------|------------|\n| HiDeP | 0.25 | 77.8±1.0 | 2.4±0.8 | 2234.0 | 6.3 |\n| HiDeP | 0.5 | 78.0±1.2 | 2.0±0.9 | 2404.7 | 6.3 |\n| HiDeP | 0.75 | 77.9±0.9 | 2.3±0.7 | 2705.3 | 6.3 |\n| ConvP | 0.25 | 78.2±1.1 | 3.9±0.4 | 4103.8 | 4.1 |\n| ConvP | 0.5 | 78.5±0.8 | 3.7±1.1 | 4417.4 | 4.1 |\n| ConvP | 0.75 | 78.3±0.4 | 4.0±1.2 | 4969.6 | 4.1 |\n\nThese results reveal REP's efficacy across diverse CL methods: Acc drops <1% for moderate n=4-8/theta=0.5-0.75, with Fgt stable or improved due to adaptive merging/dropping preserving task features. (Also, please refer to section 4.3 in our main paper for additional study, including non-prompting CL methods and self-supervised CL methods for REP’s generalizability.)\n\n\n> ‘Q2: For figure 4. What happen if Random drop have different aggressiveness? Will it have a more closer performance compared to ALD?’\n\n**[Response]**\nIn Figure 4, Random Drop showed lower performance than ALD with a 25% drop. We further examined how varying aggressiveness (drop rate) affects the trade-off between accuracy and GPU time: We experimented with drop rates of 20%, 25%, 30%, and 50% on Split ImageNet-R using the ViT-L backbone, summarized in the table below. The results show that increasing the aggressiveness of Random Drop (30–50%) reduces GPU time below ALD but causes a significant accuracy drop (65–71%), making it less practical for CL. In contrast, ALD adaptively adjusts the drop ratio, maintaining resource efficiency with minimal accuracy loss.\n\n| Method   | Acc | GPU Time (s) |\n|-----------------|-------|-------|\n | ALD | 75.8±0.5 | 4130 |\n | Random-20% | 71.7±0.4 |4209 |\n | Random-25% | 70.6±0.7 | 4014 |\n | Random-30% | 68.8±0.6 | 3964 |\n | Random-50% | 65.4±0.8 | 3743 | \n\n> ‘Q3: For AToM, as shown in Figure 3, gradient w.r.t prompt is significant and contribute essential part to CF? Is it possible to improve AToM by involving Fisher Information Matrix like in EwC to have a gradient informed prompt merging?’\n\n**[Response]**\nYes, Fig. 3 shows that prompt gradients play a key role in mitigating CF, which motivates AToM’s prompt-preserving merging strategy. Incorporating the Fisher Information Matrix (FIM) for gradient-informed merging is indeed an interesting idea. We tested this approach using L2P-L (ViT-L backbone) on Split ImageNet-R. The performance improvement was limited (+0.1–0.2% in accuracy), while GPU time and memory usage increased by about 10% (e.g., baseline REP GPU time: 2542.4s → 2796.6s, memory: 4.5GB → 4.95GB). The additional computational cost of FIM reduced overall resource efficiency.\n\nWe do not consider this a fundamental limitation but note that it is nontrivial to achieve practical gains. With sufficient optimization, gradient-informed merging remains a promising direction for future work."}, {"review_id": "T6IeDowush", "reviewer": "Reviewer_W2oY", "summary": "The paper introduces a novel strategy that reduces training time and computational memory costs in rehearsal-free, prompt-based continual learning setup, to facilitate real-world scenario deployment. The proposed method primarily optimizes two stages: prompt selection and prompt update. To improve in both stages, the paper proposes three techniques which trade a slight accuracy drop for improved resource efficiency. Specifically, the authors introduce a lightweight model with reduced depth and width for prompt selection. This technique also requires a non-trainable random projection to map back to original dimensions. Empirical results shows that this technique preserves 97% of representational similarity compared to a larger model. \n\nThe two other main techniques introduced, Adaptive Token Merging (AToM) and Adaptive Layer Dropping (ALD), are based on the analysis of which backbone blocks contribute most to the final loss. By measuring the mean attention distance they found that shallower layers capture more meaningful representations of the input. These findings are incorporated into AToM, which employs a scheduler that dynamically adjust the number of tokens to merge based on layer depth. Merging occurs more aggressively in deeper layers than in shallow layers, preserving important task-related information. ADL, on the other hand, is regulated by a layer-keeping probability that is also influenced by the degree of token merging performed by AToM in each layer. Since deeper layers tend to merge more tokens, they are more likely to be dropped. \n\nThe experiment results indicate that with by accepting a small trade-off in accuracy performance, the method achieves faster training and reduced memory usage.", "questions": "- Could the authors provide a more detailed, quantitative exploration of this trade-off? Specifically, can they present tables or graphs that clearly map different levels of resource saving to their corresponding accuracy degradation on the tested datasets?\n\n\n- Do they have insights into how their current prompt selection mechanism would perform when encountering truly novel tasks or highly divergent domains that might not align well with the existing prompts?", "limitations": "The stated \"curriculum limitation\" is a technical detail that requires further clarification. The authors should elaborate on its meaning and explain why the inability to determine the learning curriculum constitutes a significant limitation. Furthermore, it is crucial to provide more granular and practical guidance on the accuracy-resource trade-off. Similar to the curriculum learning constraint, the paper should investigate how the method performs when truly novel and unexpected tasks or domains are introduced.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper introduces an innovative strategy designed to reduce computational costs in continual learning, thereby enhancing the applicability of these systems in real-world scenarios, particularly in resource-constrained environments. This is achieved through three well-conceived techniques: a lightweight model with random projection, Adaptive Token Merging (AToM), and Adaptive Layer Dropping (ALD). These techniques operate synergistically to optimize both the prompt selection and prompt update stages.\n\nWhile a trade-off analysis is included, understanding its granularity is crucial to allow users to effectively navigate this trade-off based on their specific requirements. Although the composition of the prompt pool has been analyzed, prompt selection relies on identifying the prompt with the highest similarity to a given input. This approach might lead to only a subset of prompts being frequently chosen and subsequently updated, perhaps because they are better suited to different input types. This behavior is partly justified by findings that increasing the overall prompt pool size does not yield significant improvements. However, this selection mechanism could become a limitation when dealing with data originating from highly diverse domains.", "was_revised": true, "final_justification": "After reading the rebuttal, the additional experiments, and the discussion with the AC and other reviewers, I find that the paper makes a practical, well-argued contribution to resource-efficient rehearsal-free prompt CL. While some concerns remain, the overall evidence supports the claim of meaningful efficiency gains with small accuracy trade-offs. On balance, reasons to accept outweigh the limitations, so I keep a borderline-accept recommendation.", "rebuttal": "Thank you for your review and constructive comments. \n\n> ‘Q1: Could the authors provide a more detailed, quantitative exploration of this trade-off? Specifically, can they present tables or graphs that clearly map different levels of resource saving to their corresponding accuracy degradation on the tested datasets?’\n\n**[Response]**\nThank you. The tables below extend Table 4 of our main paper, showing how varying the intensity of AToM and ALD affects resource savings alongside accuracy drop and forgetting increase. REP provides flexible resource savings under different computational budgets while incurring only moderate accuracy loss. Nonetheless, there is a clear trade-off: more aggressive computation skipping leads to greater accuracy degradation. We will include these results in the final version.\n\n| N (w/ θ = 0.5) | Acc | GPU Time (s) | Memory (GB) |\n|---------------:|---------:|-------------:|------------:|\n| Baseline | 75.6 | 4735.2 | 6.5 |\n| 1 | 75.5 (-0.1 %) | 2833.6 (-40.1 %) | 5.5 (-15.4 %) |\n| 2 | 75.2 (-0.4 %) | 2808.4 (-40.7 %) | 5.5 (-15.4 %) |\n| 4 | 75.3 (-0.3 %) | 2714.0 (-42.7 %) | 5.2 (-20.0 %) |\n| 6 | 75.1 (-0.5 %) | 2667.5 (-43.7 %) | 4.8 (-26.2 %) |\n| 8 | 75.3 (-0.3 %) | 2542.4 (-46.3 %) | 4.5 (-30.8 %) |\n| 10 | 73.6 (-2.0 %) | 2413.8 (-49.0 %) | 4.1 (-36.9 %) |\n\n| θ (w/ n = 8) | Acc | GPU Time (s) | Memory (GB) |\n|-------------:|---------:|-------------:|------------:|\n| Baseline | 75.6 | 4735.2 | 6.5 |\n| 0.9 | 74.3 (-1.3 %) | 2987.8 (-38.1 %) | 4.5 (-30.8 %) |\n| 0.8 | 74.2 (-1.4 %) | 2935.5 (-38.2 %) | 4.5 (-30.8 %) |\n| 0.7 | 74.4 (-1.2 %) | 2861.4 (-39.6 %) | 4.5 (-30.8 %) |\n| 0.6 | 74.8 (-0.8 %) | 2705.6 (-42.9 %) | 4.5 (-30.8 %) |\n| 0.5 | 75.3 (-0.3 %) | 2542.4 (-46.3 %) | 4.5 (-30.8 %) |\n| 0.4 | 74.7 (-0.9 %) | 2489.8 (-47.4 %) | 4.5 (-30.8 %) |\n| 0.3 | 74.4 (-1.2 %) | 2413.3 (-49.0 %) | 4.5 (-30.8 %) |\n| 0.2 | 73.2 (-2.4 %) | 2362.4 (-50.1 %) | 4.5 (-30.8 %) |\n| 0.1 | 72.9 (-2.7 %) | 2301.7 (-51.4 %) | 4.5 (-30.8 %) |\n\n\n\n\n\n\n\n\n\n\n\n\n> ‘Q2: The stated \"curriculum limitation\" is a technical detail that requires further clarification. The authors should elaborate on its meaning and explain why the inability to determine the learning curriculum constitutes a significant limitation.’\n\n**[Response]**\nThank you for asking for clarification. In Section 5, we intended to clarify the scope of our work: we focus on a standard continual learning setting where task arrivals are predefined as part of the problem formulation. In this setting, REP is designed to operate under the given task order.\n\nCurriculum learning, which explicitly determines or optimizes the task sequence, addresses a related but distinct problem. Since our study does not include curriculum-based task scheduling, REP has not been evaluated in scenarios where the task order can be deliberately arranged. Investigating how REP could be combined with curriculum learning strategies is an interesting direction for future research.\n\n> ‘Q3: Do they have insights into how their current prompt selection mechanism would perform when encountering truly novel tasks or highly divergent domains that might not align well with the existing prompts?’\n\n**[Response]**\nThank you for the comment. To clarify, the similarity-based prompt selection mechanism is part of the baseline CL methods rather than REP itself. To simulate more diverse domains, we conducted preliminary experiments with subsets of the 5-datasets benchmark, introduced in [3]. The results are reported below: \n\n\n\n| Pool size  | Acc | Memory |\n|-----------------|-------|-------|\n | 10 | 62.73 | 4.95 |\n | 15 | 63.07 | 4.95 |\n | 20 | 63.40 | 4.95 |\n | 30 | 63.78 | 4.95 |\n\nThe average accuracy is lower due to the stronger domain drifts, and the optimal pool size shifted to larger values. Given the results, we agree that for tasks that cover more diverse domains, similarity-based prompt-selection mechanisms can become a limitation, as also discussed in L2P [3]. When testing L2P on a collection of 5 commonly used datasets, increasing the pool size leads to proportional improvements in accuracy. \nNonetheless, as shown in the table above, the pool size is not a dominant factor for memory usage or computation time. For tasks that contain higher drifts or cover more domains, we can adopt a larger pool size without compromising the cost-efficiency that REP brings. \n\nIf we have misunderstood the reviewer’s comment, please advise us.\n\nReference\n\n[2] Koh et al, WILDS: a benchmark of in-the-wild distribution shifts, ICML 2021.\n\n[3] Wang et al, Learning to Prompt for Continual Learning, CVPR 2022."}]

Author rebuttal:

Meta-review: This paper aims to improve the computational and memory efficiency of rehearsal-free continual learning. In particular, it targets the prompt-based rehearsal-free continual learning paradigm. The proposed approach is based on insights made in the prompt selection and the prompt update stages. This paper then proposes techniques to improve the prompt selection process and also adaptive layer dropping and token merging to improve the prompt update process.

During the review process, the strengths of the paper were identified as: exceptionally thorough evaluation, a framework that delivers substantial resource-efficiency, being adaptable and modular (owing to the diversity of the tasks, models, datasets considered), etc.

After the reviewer-author and the reviewer-AC discussion phases, during which all the reviewers participated actively, the reviewers acknowledged that the questions in their initial reviews were well-addressed. The only major point was reviewer jSzb's comment on whether the trade-off between performance drop and resource efficiency is worthwhile. The AC notes that this point was well treated in the author responses, which clearly show both the positive and the negatives sides to this trade-off. Given this balanced presentation and the overwhelming positive evaluations of all the reviewers (including reviewer jSzb), the AC recommends the paper for acceptance.

The final version should include all the clarifications provided in the rebuttal and during the discussion phase.
Decision: Accept (poster)