Componential Prompt-Knowledge Alignment for Domain Incremental Learning
Authors: Kunlun Xu
Xu Zou
Gang Hua
Jiahuan Zhou
Venue: ICML 2025 poster (2025)
Keywords: Domain Incremental learning; Image Classification
Primary area: applications->computer_vision

Abstract: Domain Incremental Learning (DIL) aims to learn from non-stationary data streams across domains while retaining and utilizing past knowledge. Although prompt-based methods effectively store multi-domain knowledge in prompt parameters and obtain advanced performance through cross-domain prompt fusion, we reveal an intrinsic limitation: component-wise misalignment between domain-specific prompts leads to conflicting knowledge integration and degraded predictions. This arises from the random positioning of knowledge components within prompts, where irrelevant component fusion introduces interference.
To address this, we propose Componential Prompt-Knowledge Alignment (KA-Prompt), a novel prompt-based DIL method that introduces component-aware prompt-knowledge alignment during training, significantly improving both the learning and inference capacity of the model. KA-Prompt operates in two phases: (1) Initial Componential Structure Configuring, where a set of old prompts containing knowledge relevant to the new domain are mined via greedy search, which is then exploited to initialize new prompts to achieve reusable knowledge transfer and establish intrinsic alignment between new and old prompts. (2) Online Alignment Preservation, which dynamically identifies the target old prompts and applies adaptive componential consistency constraints as new prompts evolve. Extensive experiments on DIL benchmarks demonstrate the effectiveness of our KA-Prompt.
Our source code is available at https://github.com/zhoujiahuan1991/ICML2025-KA-Prompt.

Componential Pronpt-Knowledge Alignment for Domain Incremental Learning
Kunlun Xu
Xu Zou
Gang Hua?
Jiahuan Zhou"
Abstract
Domain 2-2
Domain
Domain
Data
Time
Domain Incremental Learning (DIL) aims to learn from non-stationary data streams across domains while retaining and utilizing past knowledge. Although prompt-based methods effectively store multi-domain knowledge in prompt parameters and obtain advanced perfor- mance through cross-domain prompt fusion, we reveal intrinsic limitation: component-wii misalignment between domain-specific prompts leads to conflicting knowledge integration and degraded predictions. This arises from the random positioning of knowledge components within prompts, where irrelevant component fusion introduces interference. To address this, we propose Componential Prompt-Knowlede Alignment (KA-Prompt), novel prompt-based based DIL method that introduces component-aware hompt-knnwledge alignment during training. significantly improving both the learning and inference capacity of the model. KA-Prompt operates two phases: (I) Initial Componential Structure Configuring, where containing knowledge relevant to the new domain are mined via greedy search, which is then exploited to initialize new prompts to achieve reusable knowledge transfer and establish intrinsic alignment between new and old prompts. (2) Online Alignment Preservation, which dynam- ically identifies the target old prompts and applies adaptive componential consistency constraints prompts evolve. Extensive experiments on DIL benchmarks demonstrate the effectiveness of our KA-Prompt Our source code is available at http:///itub..com/hhojjjaaaa 191//CMM.2025 KA-Prompt.
G-Prompt (Existing)
PE PP-1
PC
1
Prumpt Leerning
Align
KA-Prompt (Owrs)
P P?-1
Pf
(a) Comparison of prompt learning methods
Meiched Pronel
Qaas
VIT
Class
Inference
Fused Prompt
Image
Prempt Malching
C-Prompt (Existing)
An D_
Image
Single Prompt A Atention
Posed Pringt
CLaSS
XA-Prrmpt (Ours)
Image
class
(b) Comparison of prompt- based inference process
Figure 1: (a) DIL aims to learn with a stream of data from different domains. Statc-of-the-an method C-Prompt learns domain sspecffc prompts independently. leading to component-aware (e.g.. object part orders) misalignment. To settle this, our KA-Prompt introduces cross-domain align- ment constraints. (b) During inference, by fusing prompts from different domains, our KA- KA-Prompt outperforms C- Prompt, benefiting from improved prompt learning capacity and enhanced cross-domain knowledge compatibility.
old prompts
1. Introduction
Domain Incremental Learning (DIL) aims to adapt to con- tinuous data streams with substantial domain shifts caused by variations in style, data quality degradation, and environ- mental changes (Song al., 2024; Liu et al., 2024a; Gong et al., 2022: Shi al.. 2024a: Xu et al., 2025). The key difficulty of DIL lies in balancing the plasticity learn new domains with the stability to retain and utilize historical knowledge, trade-off known as the plasticity-stabiine or acquisition-forgettin dilemma (Li et al., 2024c).
Wangxuan Institute of Computer Technology. Peking Univer- sity, Beijing, China "School of Artificial Intelligence and Automa- tion. Huazhong University of Science and Technology. Wuhan, China AAmazonnccm com, Inc, Bellevue, WA 98004, USA. Correspon- dence to: Jiahuan Zhou <jiahuanzhou @ pku.edu.en
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Existing approaches to DIL primarily rely on data replay or regularization. Replay-based methods (Shi & Wang,

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
2024; Jeeveswaran et al., 2024) retain learned knowledge by storing historical samples, but they raise privacy concerns and require growing storage overhead. Regularization-ban techniques (Wang et al., 2023b; Li & Hoiem, 2017) alleviate forgetting by constraining parameter updates, yet the strict constraints often hinder new knowledge acquisition.
(2) Methodological Innovation: KA-Prompt introduces a component-aware alignment framework for prompt learn- ing, proposing initial componential structure configuring to comprehensively mine reusable knowledge to obtain high-quality intrinsic alignment. Besides, online align- ment preservation is developed to dynamically maintain the component-wis alignment as the prompts evolve.
Prompt-based methods have recently emerged as a promis- ing alternative, where domain-specific prompts containing task knowledge can be stored in isolation to mitigate for- getting (Wang et al., 2022a). State-of-the-ar approaches like C-Prompt (Liu et al., 2024a) underscored that fusing prompts across domains can improve model performance since cross-domale shared knowledge can be exploited to facilitate inference. However, as shown in Fig. (a), since these methods learn prompts independently per domain, the prompts containing shared knowledge (e.g., object parts knowledge) are randomly positioned within the componen- tial level of the prompt. Therefore, the inter-domain prompts typically exhibit component-wise misalignment. This mis- alignment causes interference during cross cross-domain prompt fusion, limiting their ability to leverage shared knowledge.
(3) Extensive experiments conducted on four DIL bench- marks demonstrate the significant superiority of the pro- posed KA-Prompt over the state-of-the-art approaches.
2. Related Work
2.1. Incremental Learning
Incremental learning (IL) aims to continuously acquire new knowledge while retaining the old knowledge of historical tasks. Its primary challenge lies in balancing new knowledge acquisition with mitigating catastrophic forgetting (Li et al., 2024b; Chen et al., 2024; Kang & Choi, 2024). IL is com- monly categorized into three scenarios. Task-ineremental learning (TIL) assumes that task labels are available during inference (Oren & Wolf, 2021; Van de Ven et al., 2022). Class-incremental learning (CIL.) sequentially learns new classes with task labels not provided during inference (Zhou et al., 2024; Li et al., 2024a), typically within single do- main. Domain-incremente learning (DIL.) presents more challenging setting where the class set remains unchanged, but each task introduces a new domain, and task labels are not available during inference (Mirza et al., 2022; Lamers et al., 2023). Larger domain shifts generally exacerbate the forgetting problem (Xu et al., 2024a; Cui et al., 2024). Ex- isting DIL methods can be broadly categorized into replay- based, regularization-basee and prompt-based approaches.
To settle these problems, we propose KA-Prompt, a prompt- based framework that enforces component wise knowledge alignment across domains through two key designs: (1) Initial Componential Structure Configuring: Instead of randomly initializing the components of new prompts, we actively mine a set of old prompts that contains shared knowledge with the new domain. By integrating their com- ponential structures into the new prompts during initializa- tion, intrinsic cross-domain alignment is established. To achieve this, a Reusable Knowledge Mining mechanism is developed where greedy prompt search algorithm is in- troduced to maximize the reusable knowledge within the selected prompt set. (2) Online Alignment Preservation: As the new prompts update, the componential structures within prompts typically drift and induce misalignment in prompts. To overcome this, we introduce dynamic align- ment mechanism to maintain the componential structure consistency between prompts. Specifically, an Aligning- guided New Prompt Learning scheme is exploited, where the target old prompts are dynamically identified to conduct adaptive alignment constraints as the new prompts evolve. Experimental results (e.g., Fig. 1 (b)) verify that our method effectively enhances the knowledge compatibility between prompts, improving both the acquisition and inference pacity. Extensive experiments on DII. benchmarks show that our KA-Prompt outperforms the existing methods by large margins. To sum up, the contributions of the paper are as follows:
Replay-based methods store a subset of historical exemplars and replay them during new task learning (Jeeveswaran et al., 2024; Shi & Wang. 2024; Xic et al., 2022). However, their reliance on raw data storage raises privacy risks and scalability concerns, especially in resource-constrained sce- narios (Xu et al., 2024c) Regularization- based approaches impose constraints on model features or parameters to miti- gate forgetting (Asadi et al., 2023; Bonato et al., 2024; Li et al., 2025), but such constraints can significantly hinder the acquisition of new domain knowledge due to the substantial representational gaps between domains (Xu et al., 2024b).
Recently, prompt-based learning has emerged as an effec- tive solution for DIL (Wang et al., 2022a; 2023b; Liu et al., 2024a). By maintaining a pool of domain-specific prompts, these methods effectively preserve historical knowledge and mitigate catastrophic forgetting. Most prompt- based DIL approaches are implemented using pre-trained Vision Trans- formers (ViT) et al., 2021; Shi et al., 2024b; Zhang et al., 2025), and recent works (Feng et al., 2024; Wang
(1) Problem Identification: We reveal that component-win wise misalignment in prompts limits their cross-domain knowl- edge integration and utilization capacity.
2

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
al., 2025) have also explored multi-modal pre-trained models such as CLIP (Radford et al., 2021; Liu et al., 2025). This study focuses on ViT-based prompt to ensure fair comparison with recent ffute-of-the-ar approaches.
Knowledge Misalignment
wing
2.2. Prompt Composition in Incremental Learning
Recent studies (Smith et al., 2023; Liu et al., 2024a) have demonstrated that fusing prompts from different domains can enhance model robustness during inference. CODA- Prompt (Smith et al., 2023) Iearns soft weights for each prompt and combines all task prompts for inference. As do- main knowledge varies significantly across prompts, fusing all prompts can lead to knowledge conflicts and suboptimal performance. To address this, C-Prompt (Liu al., 2024a) selects and fuses only the most relevant prompts for each input sample, achieving state-of-the-an performance. How- ever, since C-Prompt learns prompts independently for each domain, the misalignment of knowledge across prompts hin- ders their effective integration, thereby limiting performance gains from prompt composition. To overcome these limi- tations, this paper focuses enhancing the utilization of historical knowledge and improving cross-domain prompt fusion compatibility to optimize inference performance.
Prompt of Domain t-1
Prompt of Domain t
Fused Prompt
(a) Knowledge misalignment in C-Prompt (Liu et al., 2024a)
Shuffle
AOC
64.
Fused
Shuffle
Prompt
Non A
C
Shuffle
(b) Evaluation by shuffling prompt components before fusion
Figure 2: (a) Within a prompt, different components typi- cally encode distinct types of knowledge. In C-Prompt, in- dependently learned prompts exhibit misalignment in com- ponential knowledge, leading to the fusion of irrelevant knowledge during inference. (b) By shuffling the compo- nential positions of different domains before fusion, some orders with better knowledge alignment can be generated.
3. The Proposed Method
This section first presents the preliminaries in Sec. 3.1, in- cluding the problem formulation of DIL and overview of the C-Prompt baseline. Then, a motivation study is pro- vided in Sec. 3.2 to analyze the phenomenon of prompt misalignment. Finally, the proposed KA-Prompt framework is detailed in Sec. 3.3.
The key kj is learnable and used for prompt matching.
Given an input image E #xxx where H, W, and C denote the height, width, and number of channels, re- spectively, it is first tokenized (Ronen et al., 2023) into sequence representation h E RR where I,, is the sequence length. Then, the following process is adopted to obtain model prediction according to Pe:
3.1. Preliminary
Problem Formulation: In DIL, the model is expected to train with a stream of domains to become a universal ex- pert for all the domains. Formally, a stream of T datasets D {D.) is given step by step during training, where D. [(x, ) 2 contains N₂ pairs of image æ; and cor- responding label y4. When D, is given, the previous t datasets are inaccessible (Liu et al., 2024a). Each domain is collected with a test dataset D for model evaluation.
Firstly, query function q(-), using fo as feature extractor, is exploited encode the image into a D-dimensional query vector, i.e., q(x) E IR,, Next, the similarity between the input sample and given prompt key ky is computed as:
S(g(=), kt) = (1 + cos(g(a)) k()) /2,
(2)
where cos(.) denotes the cosine similarity function. ((((x), k)) E [0, 1] and a higher S(q(x), kt) value indi- cates stronger relevance between I and pi.
Compositional Prompting (C-Prompt): A pre-trained Vi- sion Transformer (VTT) model fo employed as the back- bone, with its parameters 0 kept frozen throughout training. To adapt each domain, a learnable prompt set P, is intro- duced, defined as:
The top-K relevant prompts, represented as K, KK (p) P/ p....., . P are selected and then fused into a compo- sitional prompt p', via linear combination gc(-) process:
P. {(p). k.), (p, k.)...............
(p. No k ))))
(1)
K
1
p'= - -.
p
(3)
where each tuple (pi. k:) consists of a prompt embedding pi E xD and key ky E R.,. Here, Lp denotes the prompt length, and D represents the embedding dimension.
K
Then, pt, is concatenated with a class token [cls] RRID
3

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
and heg forming the final input representation hi
the size of single prompt set [Psl Np an appropri- ate selection of Np historical prompts is required. While certain approaches initialize new prompts using those from the preceding stage (Wang et al.. 2023a), they are limited in their ability to capture the shared knowledge between old and new domains, as the most relevant prior knowledge may be dispersed across multiple domains. For example, in some Cli-
[[cls]; h P E [(1++=+L))> This representation is subsequently processed by the self-attention layers. A clas- sification head is ultimately used to get the class prediction.
3.2. Motivational Study
Prompts are few mounts of additional learnable parame- ters that guide the model to focus on the discriminative fea- tures (Pu et al., 2024; Liu et al., 2024b; Yao et al., 2025). As illustrated in Fig. 2 (a), within a prompt. Pi E xd, its Lp components can capture distinct discriminative knowledge, such as object parts (head, wing, tail, ...). Since C-Prompt learns each prompt set 'e independently during training. the corresponding prompt components across different domains become misaligned. Consequently, during prompt fusion, unrelated knowledge components are merged, leading to knowledge conflicts and suboptimal model performance.
domain order of Real -uae--uukukkwww-iiiiiee part images may exhibit similarities to the Quickdraw style, whereas others may be more closely related to the Real style. Hence, two key principles guide the selection of reusable prompts:
The selected prompts should contain knowledge that is highly relevant to the new domain.
The selected prompts should collectively cover as much domain knowledge as possible.
A direct method to fulfill these principles is exhaustive grid search, which requires Cli- (1-1)xx = Onn*) iterations. Con- sequently, this approach is NP-hard and computationally prohibitive. To address this, a Reusable Knowledge Min- ing module is proposed, employing greedy search strat- egy (Flamich, 2024; Yang al., 2024).
To validate this issue, we conduct an experiment on C- Prompt by randomly shuffling prompt components across different domains before fusion during evaluation. The re- sults, shown in Fig 2 (b), are obtained by training on the Do- mainNet benchmark and evaluating accuracy (ACC), where the last domain is adopted for analysis. "Non" represents the original evaluation results, while "B," and "C" cor- respond to three rounds of random shuffling. The observed fluctuations in accuracy suggest that the original knowledge component order learned by C-Prompt is far from optimal. Therefore, one of our objectives is to improve cross-domain knowledge alignment to boost model inference.
Prompt-Data Relation Extraction (f R): Given a new do- main dataset D., the relationship between historical prompts and new domain samples is first assessed. Specifically, the pre-trained ViT fe is used to extract image features from Dr, forming a feature pool F, E R. Meanwhile, the keys of all historical prompts are arranged into a matrix K,-1 = [k]; k?:...: kj"; kyj . k E 1lx An base relation matrix S E R(((-I)xN,IxNN, is then com- puted as
Furthermore, since the domain-specific knowledge is learned independently, the generalizable knowledge ac- quired from previous domains remains underutilized when C-Prompt learns new domains. This limitation hinders C- Prompt's ability adapt to new domains. Hence, another key objective of this work is to enhance the utilization of historical knowledge to improve new domain learning.
S = 1fh (F₂) + 1//2,
(4)
where f-(-) denotes L2 normalization applied row-wise to the matrix. Each element 5 E [0. 1] represents the correlation score between a historical prompt and a new domain sample.
3.3. KA-Promph Approach
Greedy Prompt Search (fa): A reusable prompt mem- ory M, is initialized as empty set, which progressively includes the historical prompts most relevant to the new domain. At each step. the prompt containing the highest amount of unique new domain-relevant knowledge (absent from M,) is identified and added to M, This process is repeated until M. = MN
The detailed design of our KA-Prompt is illustrated in Fig. 3. It consists of Reusable Knowledge Mining mecha- nism (VPM) and an Aligning-Guided New Prompt Learning scheme (L) which serve to extract generalizable knowl- edge from previous domains to facilitate new prompt learn- ing and to ensure cross-domale prompt alignment, respec- tively.
Specifically, at each search step, we construct a memory relation matrix S.M M.MM by selecting rows from the base matrix S according to the indices of filtered reusable historical prompts. A sample effect vector v E NR is then computed by selecting the maximum value from each column of M representing the degree to which each new sample has already been assigned relevant knowledge. This
Reusable Knowledge Ming
A straightforward approach to fully utilize historical knowl- edge and ensure cross-domain knowledge alignment is to directly transfer historical prompts to new prompts. However, since the total size of the historical prompt set, [Pi ....U...PPPP.. UP grows proportionally to times
4

---

Componential Prompt-Knowledge Aligument for Domain Incremental Learning
Domain
Domain
Domain
D
Dest
(a) Reusable Knowledge Mining (Pxx
O Image Feature
Prompt
Prompi-Data Relation Extraction (fn)
Greedy Prompt Search (fc)
(y-max(SS, k)
Selected Prompt
Pi-2
Si
Top1
Breakknn
Prompt set.P Pa- Prompt sel. P---
Add to Memory
Prompt Pool
Base Matrix 5
Difference Matrix 5
Cumulativ Score Histogram
Eg (4)
+
$000
Caloree Expand
Column wise Max
VIT
Simulanby
When JM M
O
Reusable Prompt emory MM
New dataset D.
Festure Pool
Assignment Matrx
Memory Moitixx S
(b) A ligning-guided New Prompt I. Leanning (
Image Feature
Multilication
Prompt Fusion
[iittorical Prompt O Online A ligning (fa)
ligned Prompt sage
Becotoon Nar.
VI
(xx)
P6-2
12
PE-
Topk
min(s-a.3)
P2-2
P
400
Imput x
Prompt
Prompt weighting
( New Prompi Training (
Min
Lure Lagnn
Knowledge Intitalization
RC
Topk
Aape MY
ViT
R Reusable Prompt Memory M,
New promptset Set.P.
P2
Similanty
Matched Promat
Prompe
Fused
Feed with Input
Figure 3: The illustration of our KA-Prompt method. When the new domain data D, is given, the Reusable Knowledge Mining mechanism constructs a reusable prompt memory that contains the shared knowledge between the old and new domains. The reusable prompt memory is then utilized to initiate new prompts. Next, Aligning-guide New Prompt Learning scheme is conducted for Nater iterations learn the knowledge of the new domain, where the prompt training and historical prompt online aligning ensure new knowledge acquisition and cross-domain aligning, respectively.
vector then expanded into S E 1 which denoted assignment matrix.. Then, difference matrix
New Prompt Training (fr): Given an input image æ, the pre-trained ViT-based query function g(x=) is employed to extract image query. The prompt matching and fusing process of C-Prompt, introduced in Sec. 3.1, is then applied to obtain the fused prompt P' is obtained, which is fed to the pre-trained ViT with the input image æ. A classification loss is used facilitate prompt learning:
S = [S S S 0}.
(5)
is computed to quantify the additional unique knowledge that each historical prompt contributes. The overall unique relevant knowledge of each prompt is then estimated via a cumulative score histogram
Lnew CE(fe(x, p')).))
N Σ
H:
S
(6)
where CE represents the cross-entropy loss, and y is the label of æ. Besides, as shown in Fig. 3 (b), the Top-K matched similarity score set S. = {8} ......,
are extracted in frr module, and the minimal matched similarity a min (S.} retained for subsequent process.
The prompt corresponding to the top-1 value in H added to Mr. The above process repeated until M, complete. Aligning-guided New Prompt Learning
Historical Prompt Online Aligning (f Given the historical prompt pool P Pi J P2 .... U P the prompt matching process of C-Prompt, as introduced in Sec. 3.1, is applied. The resulting matched scores and prompts are denoted as {s|_1; & & and (p]11p?_11 ........ .........
Once M2 is obtained, initialize new domain prompts using M, to achieve a strict componential structure align- ment between new and old prompts. However, despite the initial alignment, componential structure drift can occur as new prompts update during training. To mitigate this, we introduce the following learning and alignment constraints.
,,K})> respectively. Each matched histor- ical prompt pi_1 is assigned a prompt weight w com-
5

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
Table 1:
The Avg-ACC performance comparison across four DIL benchmarks.
Methods
Publication
DomainNet
ImageNet-R
ImageNet-C
ImageNet-Mix
Average
EWC
NAS 2017
47.10 100.58
51.83+0.33
63.20+1.23
57.82+0.90
55.49=0.42
LwF
T-PAMI 2017
54.8520.06
57.1
69.01
65.2720.93
61.5620.45
L2P
CVPR
53.7440.04
56.55+0.33
77.86#0.44
64.20+0.34
63.09
S s-Prompts
NewriPS 2022
44.08a0.05
27.23
60.25
24.25s0.19
38.95
DualPrompt
FCCV 2022
55. 8+0.02
59.47+1.00
78.52+0.30
63.57+0.17
64. 19
ESN
AAAI 2023
45,7420.28
16.39x0.23
68. 28.34
14.9920.53
36.37
CODA-Promps
CVPR 2023
55,81 20.03
20.33
16.25
64.92a0.04
63.55
InfL.RAA
CVPR 2024
48.76+
41.20+1.65
2
35.38+0.27
44.6 44.62+0.43
Cprompt
CVPR 2024
52.88+0.12
59.48
70.08
63.6410.74
61.52+0.33
C-Prompt
ICV 2024
58.66:0.05
62.43++.49
79. 79.8440.38
65.35+0.52
66.57=0.20
KA-Prompt
Paper
62.91 14
60.51
85.43+0.56
70.35+0.32
71.30+0.19
puted as follows:
top-K matched prompts are then selected from P, and the same pipeline as in new prompt training is followed to gen- erate classification results.
w/ = e mi
(8)
where 7 is a hyperparameter used to scale the weight. If an old prompt exhibits greater similarity to the query than the matched new prompt, its weight wp-1 is set 1. Con- versely, when old prompt is less similar to the query than the matched new prompt, its weight w decreases proportionally to the similarity reduction.
4. Experiments
4.1. Experimental Settings
Datasets:
Our experiments conducted on four multi- domain benchmarks including DomainNet, ImageNet-R, ImageNet-C, and ImageNet-Mix. Within each benchmark, the domains are sorted by decreasing image counts to simu- late a challenging DIL scenario, following C-Prompt (Liu et al., 2024a) and CaSSLe (Fini al., 2022).
To maintain the componential structure alignment of cross- domain prompts, the prompts are fused as follows:
K Σ (wi-IPi D.
K
P
K+
RR
vf
K
ER
wi
Pmr
DomainNet (Peng al., 2019) contains 345 classes and 586,575 images which are collected from six different style domains, i.e. Real, Quickdraw, Sketch, Painting, Infograph, and Clipart. The training data across classes and domains are imbalanced, making this dataset more challenging.
mm1
(9)
where the term K+ΣK, wi- is employed to normalize the overall weights.
Finally, T is fed to the pre-trained ViT with the input prompt knowledge align-
ImageNet-R (Hendrycks et al., 2021) contains 30,000 im- ages of 200 categories. All images are split into 15 different style domains. The images in each domain are divided into training and testing sets with 7:3 ratio.
image æ. cross-entropy-baseee ment loss is introduced to ensure the componential fusion compatibility between and old prompts:
Lagn CF. ).
(10)
ImageNet-C (Hendrycks & Dietterich, 2018) contains 1000 categories covering 15 quality corruptions and environmen- tal changes. Following (Liu ct al., 2024a), 200 categories identical to ImageNet-R in ImageNet-CC are used to form a DIL benchmark, where each category contains 7,000 images for training and 3,000 images for testing.
Training and Inference
During training, once D. is obtained, V M (as shown in Fig. 3 (a)) is executed for one time. Then, V is applied
Niter iterations under the supervision of the overall loss:
ImageNet-Mix (Liu et al., 2024a) is built by fusing ImageNet-C and ImageNet-R, which comprises total of 30 domains, incorporating various image styles, qualities, and environmental variations.
L = Lnew + Xagn:
(11)
where À is a hyperparameter to balance the new knowledge learning and cross-domain prompt alignment.
Evaluation Metrics:
Following previous works (Liu et al., 2024a; Wang et al., 2022a), given DIL dataset with T do-
During inference, the prompt sets of all encountered do- mains are gathered, i.e, P Pi U P2 .... U Pr. The
6

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
Table 2: The per-domain performance comparison on DomainNet. The domain order during training is Real->Qucckdran Painting-+ - Clipant.
Method
Publication
Real
Quickdraw
Painting
Sketch
Infograph
Clipart
Avg-ACC
EWC
2017
60.5720.56
25.4321.5
44.55a1.16
50.062v.18
25.6920.56
76.29
57.1020.58
LwF
T-PAMI 2017
68.52.0.05
33.10a0.4
54.80a0.32
58.54
35.45
78.67a0.20
54.85.0.06
L2P
CVPR 2022
77.9620.12
19.09
59.75
56.3820.04
30.5220.13
78.74
53, 33.744
S S-Prompts
NewriPS 2022
65.5 25.04440.8
8.42++038
47.24
43.96=n.03
20.99 20.99+0.02
78.35+0.09
44. 14.08+0.05
DualPrompt
ECCV 2022
78. 78.11 10.01
24.36s0.12
60.67=0.11
57.85=0.14
30. 20.8820.07
79.23+0.11
55, 1820.02
ESN
AAAI 2023
15.95
9.87 9.879
48.29
48. 48.53
22.17
79.2420.13
45.74
CODA Prommp
CVPR 2023
+8.37+0.03
23. +0.03
60.33+0003
59.98=0.07
31.82+0.01
80.46+0.08
10.80
InfLoRA
CVPR 2024
66.36+0.32
18. 18.5440064
49.17
52.47=0.27
26. 26.18+0.20 1
79.83+0.24
48. 48.76+0.30
Cprompt
CVPR 2024
83.47
26.2440046
67.30=0.26
45.78=0.29
28.90 28.90=0.16 16
65,58+0.19
52. 92.88#0.1
C-Prompt UCV
83.3420.11
49.1 17.0.30
64.5520.23
57.20a0.22
32.29
65.4420.66
58.6620.05
KA-Prompt
This Puper
82.3020.07
52.12:0.26
68.4220.15
62.4320.23
38.25
72.9720.8
62.91
mains, the average accuracy (Avg-ACC) is used to evaluate the model. It is calculated as:
4.2. Comparison with State-of-the-Art
The comparison results across four DIL datasets, along with the average performance, are presented in Tab. 1. Specifically, our KA-Prompt outperforms the state-of oftte-of-iee oft---ffthhheee ar- C-Prompt, achieving improvements of 4.25 4..5%/44. /4.///4.88/555 %%5000% on DomainNet, ImageNet-R, ImageNet-C. and ImageNet-Mix, respectively. Overall, KA- Prompt surpasses C-Prompt with an average improvement of 4.73% % across the four datasets. These results under- score the robustness of KA- to variations in style and data quality, attributed to its enhanced utilization of histori- cal knowledge during training and improved cross-domain knowledge compatibility that boosts discriminative feature utilization during inference.
T
AT
Σ
TT
(12)
T
where ar, represents the classification accuracy on the é- th domain testing with the model trained on T-th domain (Mr), and Ar represents the overall performance of Mr across different domains.
Additionally, the Average performance across 4 DII bench- marks is reported to assess the overall DIL. capacity of dif- ferent models in diverse scenarios.
We compare our KA-Prompt with the regularization- based incremental learning methods LwF (Li & Hoiem, 2017), EWC (Kirkpatrick al., 2017), ESN (Wang et al., 2023b) and prompt- based method L2P (Wang et al., 2022c), S-Prompts (Wang et al., 2022a), DualPrompt (Wang et al., 2022b), CODA-Prompy (Smith et al., 2023), CPrompt (Gao et al., 2024) and C-Prompt (Liu et al., 2024a). Besides, we also compare with the parameter- efficient tuning method InfLoRA (Liang & Li, 2024). All experiments conducted under the official codes with VIT (ViT-B/16) pre-trained on ImageNet-211 22k as the backbone.
Compared Methods:
Additionally, Tab. 2 reports the detailed per-domain perfor- mance on DomainNet, highlighting KAA-Prompt's superior performance on intermediate domains such as Quickdraw, Painting, Sketch, and Infograph. We also observe that C- Prompt and CODA-Prompt outperform KA-Prompt on the first and last domains, respectively. This discrepancy can be attributed to C-Prompt's emphasis on domain-specine knowledge isolation during training, which reduces forget- ting but limits the model's acquisition capacity of new do- mains. On the other hand, other prompt- prompt-based methods, e.g., CODA-Prompt, L2P, S- S-Prompts, and DualPrompt, priori- tize new knowledge acquisition but suffer from catastrophic forgetting caused by prompt-classifie misalignment. In con- trast. our reusable prompt mining and initialization design effectively utilize generalizable knowledge of historical do- mains to enhance new domain learning. Besides, the online aligning design consolidates the componential knowledge alignment as new prompts evolve, improving the robustness of cross- domain knowledge utilization during inference.
We follow the prompt and clas- sifier configuration of C-Prompt (Liu et al.. 2024a). e.g., prompt length L,p and prompt number Np of each domain, the shared classifier across all domains. The Adam opti- mizer (31 = 0.9, 32- 0.999) adopted to train the model. The default batch size and learning rate for all benchmarks set to 128 and 0.005 respectively, except for DomainNet where the learning rate is set to 0.0006. The default training epochs are set to 5 except for NomainNet (10 epochs). The training images are resized to 224x224. The hypermeters T and are set to 0.01 and 0.1 by default, respectively. All experiments are conducted on single Nvidia 4090 GPU.
Implementation Details:
To further analyze the model learning process, we visualize the Avg-ACC performance across seen domains through-
7

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
70
65
80
60
Avg-ACC
Avg-ACC
55
60
50
45
KA-Proeupt (Ours)
CPProppi
Luf
KA-Prompi (Ours
DuilProppt CPrroeg
Cprcnne
Luf
40
COD4-Prumpi
Cproeupt BWC
12P
40
CODA-Pronge
BWC
DasiProope
35
20
10
12
13
10
11
13
14
15
(a) ImageNct-R averuge accuracy domains
(b) ImageNet-C average accuracy on domains
Figure 4: The seen domain performance tendency along domain incremental learning process.
(Bassiine)
62,43
66.5
Avg-ACC (%)
Avg-ACCPP (%)
63,55
+Wang ol. 2023a
65,36
+ffx* + fa)
66.0
64.71
+f.
66.51
+(x
fa) fa
63.5
Avg-ACC (%)
Figure 6: Ablation studies on the hyper- paper-pprameters under ImageNet-R dataset.
Figure 5: Ablation on the model components.
edge from old domains. Besides, fT + A. i.e., V obtains 2.28% improvement compared to the baseline, verifying the effectiveness of online alignment design. Finally, when all our modules are used together, the model performance fur- ther improved since the training-stage historical knowledge utilization and the inference stage knowledge compatibility is complementary to each other.
the continual learning procedure in Fig. 4. The results show that method obtains comparable performance with the existing methods in the initial training stages. This is because the cross-domain shared knowledge is relatively inadequate the early periods. After learning more than 4 domains, our KA-Promph consistently outperforms the com- petitors, which attributed our componential prompt- knowledge alignment designs that effectively utilized the learned knowledge to boost both training and inference along the continual learning process.
Analysis on the the performance of KA- Prompt under different values of the hyperparameters and X. The parameter T controls the weights of old prompts that are distant from new samples during prompt fusion, where a larger T increases their im- pact. Meanwhile, serves the weight for the alignment loss, balancing knowledge learning and componential alignment to the historical prompts. Based on empirical anal- ysis, the optimal hyperparameter values are set to r = 0.01 and = 0.1 as the default configuration.
In Fig. 6, we evaluate
4.3. Ablation Studies
Analysis on the proposed modules. In Fig. 5, duct ablation studies on the modules of our KA-Prompt. When using the fo module alone, our method degrades as C-Prompt (Liu et al., 2024a) baseline. fR and fa do not influence the model when used alone, and they form further verify the historical knowledge utilization capacity of L M. we additionally com- pare with (Wang al., 2023a) that provides simple initial prompt initialization strategy, i.e., using the prompt of the previous stage P.-1 to initialize 'Pt-
Ablation on prompt shuffle. demonstrate that our componential prompt-knowledge alignment reduces knowl- edge conflicts, we conduct ablation study by shuffling prompt components under varying conditions. As shown in Fig. 7 (a), C-Prompt (Liu et al., 2024a) experiences per- formance improvement in Shuffle-B/C/DAN due to its ran- domly learned componential structure, making the original componential structure tend to be suboptimal. contrast, KA- KA-Prompt consistently exhibits degradation when prompt
M when used together.
The experimental results show that fr (f ff achieves 1.81% % improvement compared to f et al., 2023a), demonstrating effectively mine the generalizable knowl-
8

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
components perturbed. KA-Prompp's sensitivity to com- ponent structures confirms its success in learning an intrin- sically aligned structure.
Impact Statement
This paper presents work whose goal is to advance the inves- tigation of domain incremental learning in Computer Vision. There are many potential societal consequences of our work, including the continuously evolving Al agents, the multi- domain downstream stak adaption of large vision models, and the lifelong environment adaption of autonomous driv- ing systems.
Besides, although random shuffling introduces misalign- ment noise that disrupts KA-Prompt's carefully orga- nized components, the worst performance of KA-Prompt (Fig. 7 (b) Shuffle-E) is 6.4% superior the best condition of C-Prompt (Fig. 7 (a) Shuffle-B). This is attributed to our greedy prompt search algorithm effectively collecting the generalizable knowledge from all historical domains, signif- icantly improving the new domain adaptation capacity of the model.
References
Asadi, N., Davari, M., Mudur, S., Aljundi, R., and Belilovsky, E. Prototype-sampee relation distillation: towards replay-free continual learning. In Interna- tional Conference on Machine Learning, pp. 1093-1106. PMLR, 2023.
67
66
06.30
ACC
65
66.40
66.85
65.79
ACC
22.89
12.93
72
7223
Bonato, J., Pelosin, F., Sabetta, L., and Nicolosi, A. Mind: Multi-task incremental network distillation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11105-11113, 2024.
63
Non
C
E
70
Non
A
C
D
Shatfle
Shuffie
(a) C-Prompt (Liu et al., 2024a)
(a) KA- KA-Prompt (Ours)
Chen, Q.. Shui, C., Han, L., and Marchand, M. On the dilemma in continual meta-learning: theory and algorithm. Advances in Neural Information Processing Systems, 36, 2024.
Figure 7: Performance comparison on the last domain of DomainNet under different componential position shuffling conditions.
Cui, Z., Peng, Y., Wang, X., Zhu, M., and Zhou, Con- tinual vision- languge retrieval via dynamic knowledge rectification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11704-11712, 2024.
5. Conclusion
In this paper, we investigate the practical and challeng- ing domain incremental learning problem and propose a novel method KA-Prompt. Based the observation that component-we wise misalignment between domain specific prompts leads to conflicting knowledge integration and de- graded predictions, we introduce a component- wise knowl- edge alignment paradigm with two complementary designs: (I) Initial Componential Structure Configuring exploits a novel greedy prompt search algorithm to comprehensively mine the historical prompts containing reusable knowledge, which utilized provide effective knowledge transfer and intrinsic alignment. (2) Online Alignment Preservation dynamically identifies the target old prompts and applies adaptive componential consistency constraints along the new prompt learning procedure. Extensive experimental results underscore our component-wis alignment paradigm effectively improves the acquisition and inference capacity simultaneously.
Dong. L., Li, Z., Xu, K., Zhang, Z., Yan, L.,, Zhong, S., and Zou, Category-aware transformer network for better human-object interaction detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19538-19999 19547, 2022.
Feng, Y., Tian, 7., Zhu, Y., Han, Z., Luo, H., Zhang, G., and Song, M. Cp-prompt: Composition- cross-modal prompting for conain-incremennal continual learning. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 2729-2738, 2024.
Fini, E., Da Costa, V. G. T., Alameda Pineeda X., Ricci, E., Alahari, K., and Mairal, J. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9621-9630, 2022.
Acknowledgements
Flamich, G. Greedy poisson rejection sampling. Advances in Neural Information Processing Systems, 36, 2024.
This work was supported by the National Natural Science Foundation of China (62376011) and the National Key R&D Program of China (2024YFA) 1410000).
Gao, Z., Cen, J., and Chang, X. Consistent prompting for rehearsal-free continual learning. In Proceedings of the

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 28463-28473, 2024.
Li, Y., Xu, W., Wang, H., Qi, Y., Guo, J., and Li, R. Person- alized federated bomman-inncrementaa learning based on adaptive knowledge matching. In European Conference on Computer Vision, pp. 127-144. Springer, 2025.
Gong, Y., Huang, L.,, and Chen, L.. Person re-identification method based on color attack and joint defence. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4313 432222222222 2022.
Li, 7. and Hoiem, D. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 40(12):2935-2940 2017.
Hendrycks, D. and Dietterich, G. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint
Liang, Y-S. and Li, W.-J Inflora: Interference-fret low- rank adaptation for continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 066638-3367, 2024.
2018.
Hendryeks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of nut-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021.
Liu, Z.,, Peng. Y., and Zhou, J. Compositional prompting for anti-forgetting in domain incremental learning. Inter- national Journal of Computer Vision, pp. 1-18, 2024a.
Jeeveswaran, K., Arani, E., and Zonooz, B. Gradual di- vergence for seamless adaptation: A novel domain in- cremental learning method. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 21486-21501. PMLR, 2024.
Liu, Z.. Peng, Y., and Zhou, J. Insvp: Efficient instance visual prompting from image itself. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 6443- 6232,,65,2220 2024b.
Liu, 7., Xu, K., Su, B., Zou, X., Peng, Y., and Zhou, J. Stop: Integrated spatial-temporal dynamic prompting for video understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
Kang, H. and Choi, D.-W. Recall-orientdd continual learn- ing with generative adversarial meta-model. In Pruceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13040-13048, 2024.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3351-335 -3261, 2017.
Mirza, M. J., Masana, M., Possegger, H., and Bischof, H. An efficient comain-incremental learning approach to drive in all weather conditions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3001-3011, 2022.
Lamers, C., Vidal, R., Belbachir, N., van Stein, N., Bäeck, T., and Giampouras, P. Clustering based domain- incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3384- 2023.
Oren, G. and Wolf, L. In defense of the learning without forgetting for task incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2209-22111 2021.
Peng, X., Bai, Q. Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 1406-1415, 2019.
Li Q., Peng, Y., and Zhou, J. Fes: Feature calibration and separation for non-exemplar class incremental learning. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 28495-28504, 2024a.
Li, Q., Peng, Y., and Zhou, J. Progressive prototype evoly- ing for dual- dorg-fogg mitigation in non-exemplar online continual learning. In Proceedings of the 32nd ACM In- ternational Conference on Multimedia, pp. 2477-2486, 2024b.
Pu, Y., Wu, X., Yang, L., and Wang, S. Learning prompt- enhanced context features for weakly-supervised video anomaly detection. IEEE Transactions on Image Process- ing. 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp.
Li, Q. Xu, K., Peng, Y., and Zhou, J. Exemplar-free life- long person re- re-identificatinn via prompt-guided adaptive knowledge consolidation. International Journal of Com- puter Vision, pp. 1-16, 2024c.
PMIR, 2021.
10

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
Ronen, T., Levy, O., and Golbert, A. Vision transformers with mixed-resolution tokenization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4613-4622, 2023.
Wang, Y., Ma, Z.,, Huang. Z., Wang, Y., Su, Z., and Hong, X. Isolation and impartial aggregation: A paradigm of incremental learning without interference. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10209-10217, 2023b.
Shi, H. and Wang. H. A unified approach to domain in- cremental learning with memory: Theory and algorithm. Advances in Neural Information Processing Systems, 36, 2024.
Wang. Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H.,, Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022b.
Shi, J., Yin, X., Zhang, Y., Xie, Y., Qu, Y., et al. Learn- ing commonality, divergence and variety for unsupervised visible-infrared person re- re-identification. Advances in Neu- ral Information Processing Systems, 37:99715-99777 2024a.
Wang. Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 139-149, 2022c.
Shi, J., Zheng, S., Yin, X., Lu, Y., Xie, Y., and Qu, Y. Clip-guided federated learning on heterogeneity and long- tailed data. In Proceedings of the AAAI Conference on Ar-
Xie, J., Yan, S., and He, X. General incremental learn- ing with domain-aware categorical representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14351-14366 2022.
rificial Intelligence, volume 38, pp. 14955 11963, 2024b.
Smith, J. S., Karlinsky, L.. Gutta, V., Bascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11909-11919, 2023.
Xu, K., Zhang, H., Li, Y., Peng, Y., and Zhou, J. Mitigate catastrophic remembering via continual knowledge pu- rification for noisy lifelong person re-identification. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 5790-5799, 2024a.
Song. X., He, Y., Dong, S., and Gong. Y. Non-exemplar domain incremental object detection via learning domain bias. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 15056-15505 15065, 2024.
Xu, K., Zou, X., Peng, Y., and Zhou, J. Distribution-aware knowledge prototyping for non-exemplar lifelong person re-identification. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 16604-16613, 2024b.
Van de Ven, G. M., Tuytelaars, T., and Tolias, S. Three types of incremental learning. Nature Machine Intelli- gence, 4(12):
2022.
Xu, K.,, Zou, X., and Zhou, J. Lstke: Long short-term knowl- edge consolidation for lifelong person re-identification. In Proceedings of the AAAI Conference Artificial In- telligence, volume 38, pp. 16202-16210, 2024c.
Wang, L., Xie, J., Zhang, X., Huang, M., Su, H., and Zhu, J. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2023a.
Xu, K., Jiang, C., Xiong, P., Peng, Y., and Zhou, J. Dask: Distribution rehearsing via adaptive style kernel learning for exemplar free lifelong person re-identification. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 8915-88188888 8923, 2025.
Wang, Q., He, Y., Dong, S., Gao, X., Wang, S., and Gong, Y. Non-exemplar domain incremental learning via cross- domain concept integration. In European Conference Computer Vision, pp. 1441662. Springer, 2025.
Wang, Y., Huang, R., Song, S., Huang, Z., and Huang, G. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in Neural Information Processing Systems, 34:1 14:11960- 11973,200 2021.
Yang, Y.,. Gao, R., Wang, X., Ho, T.-Y., Xu, N., and Xu, Q. Mma-diffusion: Multimodal attack on diffusion models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 7737-7746. 2024.
Wang. Y., Huang, Z., and Hong, X. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022a.
Yao, Y., Liu, Z., Cui, Z., Peng. Y., and Zhou, J. Selective visual prompting in vision mamba. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025.
pp.

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
Zhang, C., Xu, K., Liu, Z., Peng, Y., and Zhou, Scap: Transductive test-time adaptation via supportive clique-based attribute prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
Zhou, D.-W., Cai, Z.-W., Ye, H.-J., Zhan, D.-C., and Liu, Z. Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, pp. 1-21, 2024.
12

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
A. Algorithm.
The overall process of our key phases VA M and Ve are shown Alg. I and Alg. 2, respectively.
Algorithm Reusable Knowledge Mining (V-)
Input: Data Dr (( y:) E Prompt P = u...uuuu...PPPP
Output: Reusable prompt memory MM
Prompt-Data Relation Extraction module (f))
Extract image features F, E R x from D; via pre-trained ViT fo; Gather the keys of prompts in P: K,-1
[ ki; = k2: ......... MM
Base relation matrix S S (K,- (F+) 11/2, Eq. 4;
# Greedy Prompt Search module (fa) Initialize M, = 0;
for [M&] < Np do
Extract relation matrix S S E IR|_MsXx.e according to M, and 8
Obtain sample effect vector v E R.V. by keeping column-wise maximum value of S S.M;
Obtain S E N),,,,, by column-wise expansion of v;
Difference matrix S:: 5 = max [5) - 0). Eq. 5;
Cumulative score histogram H: H, Σラ S Eq. 6; Top-1 element &- arg max 11; Update M, - Mr U (Ppk. k,),, kk) P: end for Return M₁
Algorithm 2 Aligning-guides New Prompt Learning (Vrc)
Input: Data D, (<x, yi) ) Prompt pool Pt-1 = Pi UP2 U .... UP-1 U....UPP.
Output: New prompt set P. Initialize P.- = MM
for m to Nater do
Simple (x. y) from D;;
# New Prompt Training module (fr)
Obtain similar scores over Pi: (S(g(z), k() = (1 + cos(g(2))) k:)) /2) Eq. 2:
Obtain top-K K scores S = {&|, s? ..... $ }:
Obtain minimum matched score a min St.
Obtain top-K K prompts K [p] p.
P. };
Obtain compositional prompt p'x 9r (p. P. ....... )
K Σ Pi Eq. 3;
New data learning loss Lnew CE(fe(æ, p'a) y). Eq. 7;
# Historical Prompt Online Aligning module (f A)
Obtain similar scores over Pi-1: (S(g(z), k;) = (1 cos(g(x), k()) /2] /2) ((-()) Bq. 2:
Obtain top- K scores St = [s]
8/
Obtain top-K K prompts Kt-1 = (p: Pt
K }:
Obtain old prompt weight wi-1 e min
-0,03/7 Eq. 8:
Obtain alignment promptp = K+ΣR w
Σk [ 1 + K+ER, wi_, D' Eq. 9;
Calculate knowledge alignment loss Lagn = CF.So( ) v). Eq. 10;
Optimize C = Lnew + >aagr Eq. 11;
end for Return P.
13

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
Note that in Alg. 1, the beginning of the Greedy Prompt Search module, the Reusable Prompt Memory is empty, and the Memory Matrix S TRO0N is empty matrix. Then, the column-wise
process is conducted to obtain x N₂ null matrix. Consequently, the Assignment Matrix S* become (((t - 1) Np] N₂) null matrix. Besides, the Cumulative score histogram H could be zero-vector. This phenomenon arises since the new domain is highly relevant to small subset of historical prompts, and the remaining historical prompts are considered useless. In such conditions, we randomly select R 2 prompts in M2 and generate a prompt (Pk. kx) by through their interpolation. Then (P k&) added to Mc.
B. Visualization Results.
Seen-Domain Evaluation Curve In addition to main paper, also provide the seen domain performance tendency the other two benchmarks, i.e., DomainNet and ImageNet-Mix, which are illustrated in Fig. 8 and Fig. 9, respectively. The results show that although our method obtains comparable performance with the existing methods in the initial training stages, consistently outperforms the competitors after learning from 3-th and 10-th domains on DomainNet and ImageNet- Mix respectively, attributed to our componential prompt-knowledge alignment designs that effectively utilized the learned knowledge boost both training and inference along the continual learning process.
90
KA-Prompl (Ours)
C-Prompt
LwF
80
CODA- P-DA-rompl
Cprompt
L2P
DualPrompt
FWC
Avg-ACC
70
60
50
40
2
3
5
Figure 8: The seen domain performance tendency on DomainNet.
80
60
Avg-ACC
50
40
30
KA-Prompt (Ours)
C-Prompt
1wF
CODA-Prompt
Cprompt
L2P
20
DualPrompt
EWC
10
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Figure 9:
The seen domain performance tendency on ImageNet-Mix.
14

---

Componential Prompt-Knowledge Alignment for Domain Incremental Learning
C. Parameters and Overhead Comparison
also compare the model parameters, training overhead, and performance with state-of-the-ar prompt-based methods in Tab. All experiments are conducted on the ImageNet-R benchmark. The numbers total parameters and trainable parameters in KA-Prompt are identical to the baseline C-Prompt (Liu al., 2024a) since no extra learnable parameter introduced in method. The number of trainable parameters is larger than other methods, e.g.. CPrompt, CODA-Prompt, and DUalPrompt. This is because the competitors learn one long prompt for each domain, while C-Prompt and our KA-Prompt learn a set of short prompts per domain. Our GPU Memory cost and training Batch Time are comparable to most prompt-based methods. The GPU Memory cost and training Batch Time improvement compared to the baseline is due to the Historical Prompt Online Aligning module (f A) which introduces extra computing.
As for inference, our method adopts the same prompt matching. fusion, and utilization process as C-Prompt without introducing extra operation. Given the comparable training and inference overhead, our method outperforms the existing methods by 4.08 -39.28%% highlighting the effectiveness and feasibility of our method.
Table Comparison of the number of parameters, overhead and performance with the state-of-the-arl on ImageNet-R.
Methods
Total Params
Trainable Params GPU Memory (GB)
Batch time (S)
Avg-ACC
L2P (Wang et al., 2022c)
86,263,843
311,387
19.33
0.66
56.55
S-Prompts (Wang et al., 2022a)
92,964,094
1,637,910
14.90
0.39
27.23+0.16
DualPrompt (Wang et al., 2022b)
86,514,211
561,755
19,45
0.66
59.47=1.00
CODA-Prompt (Smith et al., 2023)
88,437,580
2,638,926
19.18
0.80
55.21
CPrompt (Gao et al., 2024)
92,485,224
2,689,168
41.64
1.18
59.48=1.09
C-Prompt (Liu et al., 2024a)
89,180,505
3,381,851
11.99
0.42
62.43+0.99
KA-Prompt (Ours)
19,180,505
3,381,851
19.28
0.72
15


Reviews:
[{"review_id": "rIEmZczOE2", "reviewer": "Reviewer_W2NY", "summary": "The paper addresses the challenge of DIL. The authors identify a limitation in existing prompt-based methods: component-wise misalignment between domain-specific prompts leads to conflicting knowledge integration and degraded predictions. To address this, they propose ​KA-Prompt, a method that enforces component-wise knowledge alignment across domains. KA-Prompt operates in two phases: (1) ​Initial Componential Structure Configuring, where a set of old prompts containing relevant knowledge is mined via greedy search to initialize new prompts, ensuring reusable knowledge transfer and intrinsic alignment; and (2) ​Online Alignment Preservation, which dynamically identifies target old prompts and applies adaptive componential consistency constraints as new prompts evolve.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. We hope our responses address your concerns.\n\n**W1: Misalignment's occurring and definition**\n\n(1) Misalignment occurs during cross-stage prompt fusion in DIL. In Fig. 2, each prompt (e.g., $p_t^1$) consists of 4 tokens, each encoding distinct partial knowledge of objects. *Misalignment* refers to the disorder of partial knowledge within prompt tokens of different stages. When fusing prompts from multiple stages, this cross-stage token-level misalignment introduces semantic conflicts in the fused prompt. Then, the sub-optimal fused prompt is fed to the Attention layer, leading to degraded performance.\n\n(2) Misalignment does not occur in VPT because it is designed for static training data, where all prompts are optimized jointly rather than incrementally.\n\n(3) Our method explicitly enhances cross-stage prompt alignment, ensuring that the fused prompt retains a coherent representation of semantics. These high-quality fused prompts thereby lead to improved test performance.\n\n**Q1: Measuring alignment**\n\n(1) To measure alignment, we have traversed different token orders across stages to identify the configuration that yields the highest model performance, which we define as the optimal alignment, as shown in Fig. 7.\n\n(2) The results show that the learned prompt token order in C-Prompt baseline exhibits a performance degradation of 0.64% compared to the optimal alignment, verifying the presence of misalignment.\n\n(3) In contrast, the prompt token order learned by us consistently outperforms alternative orders by 0.57–0.79%, demonstrating that our approach effectively achieves optimal alignment.\n\n\n**W2: Validation of misalignment**\n\nWe have visualized the attention maps of prompt tokens at different stages, along with the fused tokens, in  Fig. A of https://anonymous.4open.science/r/ICML-31/FigA-Heatmap.png. The results show that each domain-specific prompt token learns a semantic part of objects. Compared to C-Prompt which introduces semantic misalignment in prompt tokens from distinct domains, the fused prompt tokens in our method effectively preserve part-level information, enabling the model to fully exploit discriminative features.\n\n**W3: Necessity of the Greedy Prompt Search (GPS)**\n\n(1) GPS is necessary because (a) the shared knowledge between new and old domains is distributed across various old prompts, and (b) the knowledge of some old prompts overlaps significantly. A naive selection based solely on high similarity scores in a single search would often lead to redundant prompt selection while overlooking relevant knowledge present in lower-similarity prompts. This results in insufficient utilization of old knowledge. Please refer to Reviewer **8nTg-W3, Q4** for more experimental analyses.\n\n(2) The computational cost of GPS is significantly less than performing a single forward pass on the training set since it primarily involves simple matrix addition and subtraction.\n\n**W4,Q3,Q4: Catastrophic forgetting**\n\nOur method inherits the anti-forgetting capacity of prompt learning and does not significantly risk forgetting: \n\n(1) In our Historical Prompt Online Aligning (HPOA) module, old prompts are frozen, ensuring that knowledge from previous tasks is not overwritten or forgotten.\n\n(2) The prompt selection for old tasks is minimally influenced by HPOA. This is because prompt selection relies on prompt keys, and new prompt keys are only trained by minimizing their distance to new data features.\n\n(3) In Tab. 2, domains are trained sequentially from left to right. Our KA-Prompt performs 1.04% inferiorly to the C-Prompt baseline, while we outperform the C-Prompt from the second domain and obtain **4.25%** Average accuracy improvement across all domains. These results show that our approach achieves a better balance between acquisition and forgetting.\n\n**W5: Metrics setup**\n\n$a_{T,i}$ is evaluated on the test data of the i-th domain. Eq. 12 measures the final model’s performance across all previous domains by computing the average performance over them. We have carefully clarified these details in the revised version.\n\n**Q2: Classifier setup**\n\nThe final classifier is shared across all domains, consistent with most prompt-based methods. We have clarified this in the revised version.\n\n**Q5: Visualization of S-Prompt**\n\n(1) To maintain consistency in the color-method pairing relations across Fig. 4 (a)(b), Fig. 8, and Fig. 9, only top-8 methods ranked by Average accuracy are chosen for performance visualization, where S-Prompt is not included.\n\n(2) In Fig. B-1, and Fig. B-2 of https://anonymous.4open.science/r/ICML-31/FigB-S-Prompts.png, we have added the performance curves of S-Prompt. The results show that our KA-Prompt effectively outperforms existing methods during long-term learning.\n\n**Comments: Simplify Fig. 2**\n\nTo highlight the misalignment phenomenon, a simplified illustration of Fig. 2 is shown in Fig. C of https://anonymous.4open.science/r/ICML-31/FigC-Simplify.png.", "claims_and_evidence": "No. Please refer to \"other strengths and weaknesses\" for detail.", "methods_and_evaluation_criteria": "No. Please refer to \"other strengths and weaknesses\" for detail.", "theoretical_claims": "No proofs.", "experimental_designs_or_analyses": "Yes. Please refer to \"other strengths and weaknesses\" for detail.", "relation_to_broader_scientific_literature": "The authors effectively build on prior work in prompt-based learning, such as C-Prompt and CODA-Prompt, while addressing a critical limitation (component-wise misalignment) that has not been previously explored in depth.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths \n\n1. Extensive experiments on multiple benchmarks demonstrate the superiority of KA-Prompt over state-of-the-art methods.\n2. Although I am not entirely convinced by the motivation behind this paper, the issue of how to share knowledge across different domains is indeed a critical problem in DIL. This paper offers a new approach to addressing this challenge.\n\nWeakness:\n\n1. The authors claim in their contributions that \"We reveal that component-wise misalignment in prompts limits their cross-domain knowledge integration and utilization capacity.\" However, the paper does not provide any experimental or theoretical evidence to support this claim. Attention mechanisms in Transformers are inherently permutation-invariant, and methods like Visual Prompt Tuning (VPT) can operate without positional encoding. Therefore, it is unclear why misalignment would occur in the first place. The authors need to provide a more rigorous justification for this claim.\n2. The motivation for the paper, as illustrated in Figure 2, is based on two assumptions: (1) different components typically encode distinct types of knowledge, and (2) independently learned prompts exhibit misalignment in componential knowledge, leading to the fusion of irrelevant knowledge during inference. However, these assumptions are not supported by any experiments or references to prior work. Specifically, the depiction of different tokens representing different parts of an airplane in Figure 2 is confusing and lacks empirical validation.\n3. The necessity of the Greedy Prompt Search module is questionable. As I understand it, this module computes the similarity between training samples and all prompt keys, then selects the most similar prompts for initialization. Even if the similarity is computed across all training samples, the computational cost would still be less than performing a single forward pass on the training set. The authors should justify the need for this module more clearly.\n4. The paper does not adequately address the issue of catastrophic forgetting in DIL. In fact, the authors' approach may increase the risk of forgetting. By using prompts from old tasks to initialize new tasks and encouraging similarity between new and old prompts during Historical Prompt Online Aligning, the model may incorrectly select new prompts for old task data during inference, exacerbating forgetting.\n5. The experimental setup is not clearly defined, particularly in Equation 12. It is unclear whether $a_{T,i}$ is evaluated on the test data of the i-th domain only or on the test data of all previous domains. This ambiguity needs to be clarified to ensure the reproducibility and validity of the results.", "other_comments_or_suggestions": "Figure 2 is somewhat overly cluttered, making it difficult to grasp the key points.", "questions_for_authors": "1. Compared to C-Prompt, the motivation for this paper is based on the concept of \"misalignment.\" How is misalignment defined, and how is alignment measured and evaluated? What metrics or experiments are used to determine whether alignment has been achieved?\n2. How is the final classifier set up? Is there a shared classifier across all domains, or does each domain have its own separate classifier? This distinction is crucial for understanding the model's ability to generalize across domains.\n3. Given the weaknesses identified, how does the proposed method address the issue of catastrophic forgetting in incremental learning? Specifically, how does the method ensure that knowledge from old tasks is not overwritten or forgotten when learning new tasks?\n4. Are the prompts from old tasks updated during the Online Aligning process? If so, how does the method ensure that the updated prompts perform better on old tasks compared to the original prompts?\n5. Why are the results for S-Prompt missing in Figures 8 and 9? Based on my experimental experience, S-Prompt is effective in reducing forgetting, yet it is notably absent from these comparative results. Could the authors explain this omission and provide the missing results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QW1Z9HGKmC", "reviewer": "Reviewer_8nTg", "summary": "This paper introduces a novel component-based prompt knowledge alignment method, KA-Prompt, for Domain Incremental Learning (DIL). Its key contribution lies in addressing the cross-domain prompt misalignment problem, which is claimed to be a major limitation of existing prompt-based DIL methods, such as C-Prompt. The proposed framework consists of two core mechanisms: Reusable Knowledge Mining (ΨM), which selects and initializes new prompts based on relevant past knowledge, and Aligning-guided New Prompt Learning (ΨL), which dynamically maintains component alignment across domains. Experimental results on four benchmarks (DomainNet, ImageNet-R, ImageNet-C, ImageNet-Mix) demonstrate that the proposed method outperforms state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback and comments. We hope the following responses address your concerns.\n\n**W1: Contributions on framework**\n\nThe C-Prompt baseline corresponds to our New Prompt Training branch. We have made two key designs to form a brand-new DIL framework:\n\n(1) Reusable Knowledge Mining (RKM) mechanism. Unlike existing methods that randomly initialize new prompts, RKM actively searches for old prompts containing shared knowledge between the new and all old domains, significantly improving new domain adaptation and cross-stage knowledge alignment.\n\n(2) Historical Prompt Online Aligning (HPOA) branch. HPOA introduces an online search and re-weighting based prompt fusion strategy to mitigate cross-stage knowledge drift during training, effectively improving the utilization of multi-domain knowledge.\n\n**W2: Quantization of misalignment**\n\nTo quantify misalignment, we have traversed the cross-stage prompt token orders to obtain the performance of the optimal alignment. As shown in Fig. 7, C-Prompt exhibits a 0.64% degradation compared to the optimal alignment, indicating suboptimal order learning. In contrast, our method achieves **0.57-0.79%** higher performance than alternative orderings, verifying that it successfully attains optimal alignment.\n\n**W3, Q4: Complexity evaluation and theoretical analyses**\n\nThe objective of RKM module at stage $t$ is formulated as follows: given $m$ training samples and $n=(t-1)×L_s$ old prompts, obtain $k=L_s$ old prompts that minimize $\\sum_{i=1}^{m}\\{\\max_{1≤j≤n} s(x_i,p_j)\\}$ where $s(x_i,p_j)$ denotes the similarity between a training sample $x_i$ and a prmpt $p_j$.\n\n(1) The complexity of our greedy search algorithm is $mnk=O(nk)$, more efficient than exhaustive grid search, which requires $mC_n^k=O(n^k)$ operations.\n\n(2) The optimization of RKM can be approximated as a k-medoids problem [1] by considering old prompts as special training samples (since m>>n, this approximation does not affect the theoretical conclusion). According to Theorem 4.4 of [1], in a one-way search setting, the error bound of our solution $E(θ^*)$ relative to the optimal solution $E(θ)$ satisfies:\n$E(θ^*)≤(1+\\frac{2k}{n+m})E(θ)$. Since m+n>>k, our greedy search ensures globally optimal solutions under different conditions and leads to stable performance.\n\n[1] PAMAE: Parallel k-Medoids clustering with high accuracy and efficiency. SIGKDD, 2017.\n\n(3) We have conducted experiments on ImageNet-R to compare our method with existing approaches:\n\n|Method|Previous-Domain [a]|Most-Similar [b]|Greedy (Ours)|\n|-|-|-|-|\n|Avg-ACC|63.55±0.46|64.32±0.48|**65.36**±0.52|\n\n[a] initializes new prompts using those from the previous domain.\n\n[b] selects prompts with the highest similarity scores in a single search step for initialization.\n\nThe results show that our greedy search-based reusable knowledge mining strategy (Greedy) consistently outperforms [a] and [b] with improvements of **1.81%** and **1.04%**, respectively.\n\nBoth [a] and [b] suffer from insufficient utilization of old knowledge due to limited knowledge relevance within adjacent domains and overlooking of knowledge in lower-similarity prompts, respectively.\n\nIn contrast, our approach evaluates the unique knowledge that has not been included in the selected prompts iteratively, effectively improving the utilization of old knowledge.\n\n**Q1: Definition of components and semantic consistency**\n\n(1) The components refer to the tokens of each prompt. As shown in Fig. 2, each prompt, e.g., $p_t^1$, contains 4 tokens (i.e., components), each encoding a distinct aspect of object knowledge.\n\n(2) In DIL, the categories of different domains are identical, thus the object parts are semantically consistent across domains.\n\n**Q2, Q3: Do alignment alleviate conflicts**\n\nWe have visualized the attention maps of prompt tokens at different stages, along with the fused tokens, in  Fig. A of https://anonymous.4open.science/r/ICML-31/FigA-Heatmap.png. The results show that each prompt token (component) captures a semantic part of objects.\n \n(1) Due to semantic misalignment among cross-domain prompt tokens, the fused prompt tokens in C-Prompt fail to precisely capture object-specific information.\n \n(2) Our method improves component-wise alignment, enabling the fused prompt token to effectively encode discriminative features of objects. This results in a **4.73%** increase in Average accuracy, as reported in Tab. 1.\n\n**Q5: Influence of prompt fusion**\n\nFig. D in https://anonymous.4open.science/r/ICML-31/FigD-Pos-Transfer.png shows the testing performance trends for each domain. The results indicate that 2 out of 14 old domains exhibit performance reduction after continual training. This arises due to data imbalance, where some domains contain a few training samples, leading to biased knowledge during fusion. Nevertheless, our method achieves significant improvements on 12 out of 14 old domains, indicating that it effectively achieves positive knowledge transfer.", "claims_and_evidence": "In this paper, the authors propose that changes in the order of prompts lead to significant variations in accuracy, which leads to the issue of prompt misalignment. Although the ablation experiment based on shuffling (Fig. 7) provides some preliminary insights, the paper does not offer sufficient theoretical analysis to explain the importance of prompt order and the negative impact of prompt misalignment on model performance. Therefore, more experiments are needed to further validate the actual impact of prompt misalignment and eliminate potential experimental biases. To strengthen the credibility of this claim, it is recommended that the authors add more experiments, conduct broader validations, and quantify the specific effects of prompt misalignment on model performance.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria, including the use of benchmark datasets like DomainNet and ImageNet-Mix, are meaningful for addressing the challenges in Domain Incremental Learning (DIL) and enhancing cross-domain knowledge transfer.", "theoretical_claims": "No, the paper does not involve theoretical claim.", "experimental_designs_or_analyses": "The experimental design of this paper aims to verify the effectiveness of KA-Prompt in Domain Incremental Learning (DIL) tasks through comparisons with multiple baseline methods, ablation studies, and hyperparameter analyses. Overall, the experimental design is relatively comprehensive. However, there are some potential shortcomings that require further investigation: The motivation section suggests that prompt misalignment is a major performance bottleneck of C-Prompt, but the ablation study (Fig. 7) only preliminarily validates this hypothesis by shuffling prompt components, without further quantifying the specific impact of misalignment on model representations. Additionally, the scale of the experiments is not sufficient to rule out potential experimental biases.", "relation_to_broader_scientific_literature": "The core contribution of this paper lies in optimizing C-Prompt, with the main contribution being the resolution of prompt misalignment in cross-domain learning to enhance cross-domain prompt fusion. C-Prompt (Compositional Prompting) is a prompt-based Domain Incremental Learning (DIL) method that aims to adapt to cross-domain tasks by learning compositional prompts.", "essential_references_not_discussed": "The authors discuss and compare a wide range of related methods.", "other_strengths_and_weaknesses": "One of the main contributions of this paper is to explicitly identify component-level misalignment as a key issue in prompt-based Domain Incremental Learning (DIL). Previous works, such as C-Prompt and CODA-Prompt, primarily focused on selecting and fusing relevant prompts but largely overlooked the potential knowledge interference caused by the random positioning of prompt components across different domains.\n\nThe main weaknesses of this paper are as follows:\n\n1. Overall, the core contribution of this paper lies in optimizing C-Prompt rather than proposing an entirely new framework, making it more of an incremental improvement with limited novelty. The primary contribution is addressing the prompt misalignment issue in cross-domain learning to enhance prompt fusion across different domains. However, it remains based on the existing C-Prompt structure, with its methodological innovation mainly reflected in localized improvements rather than introducing a new paradigm for incremental learning.\n\n2. The authors propose that different prompt orders lead to significant variations in accuracy, thereby introducing the issue of prompt misalignment. While the shuffle-based ablation study (Fig. 7) provides some preliminary insights, the paper lacks sufficient theoretical analysis to explain the importance of prompt order and the negative effects of prompt misalignment. More experiments are needed to further validate the actual impact of prompt misalignment and rule out potential experimental biases. To enhance the credibility of this claim, the authors are encouraged to conduct broader experiments and quantify the specific impact of prompt misalignment on model performance.\n\n3. Additionally, the paper does not provide a formal analysis or complexity evaluation of the greedy search for reusable knowledge (ΨM). Since greedy algorithms often lead to suboptimal solutions, analyzing its approximation guarantees would strengthen the theoretical depth of the proposed method.", "other_comments_or_suggestions": "Please refer to the weaknesses.", "questions_for_authors": "1.What constitutes a \"component\" within a prompt? How is semantic consistency ensured across components in different domains? If the definition of components within prompts is vague or semantically unclear (e.g., if the components are simply vector representations without strict semantic binding), then the effectiveness of the proposed component-level alignment may face fundamental issues.\n\n2.Does component alignment really alleviate conflicts? The paper assumes that aligning prompt components will inevitably reduce knowledge conflicts; however, this assumption has flaws: does the alignment of components necessarily lead to alignment at the semantic or knowledge level? Is it possible that structural alignment does not result in actual semantic alignment?\n\n3.The authors did not provide sufficient qualitative or visual evidence to demonstrate that component-level alignment achieves the expected semantic fusion effect.\n\n4.How does the greedy search ensure that the selected set of prompts is truly globally optimal? The paper fails to provide enough evidence to verify whether such locally optimal solutions are sufficient, or whether they might lead to unstable performance or degradation in large-scale, multi-domain conditions. The paper does not explore whether using a greedy algorithm could introduce significant performance fluctuations, nor does it provide comparisons with other heuristic algorithms to justify the choice of a greedy approach.\n\n5.Could the fusion of multiple prompts cause negative transfer across domains? In cases where there are significant differences in prompt knowledge between domains, does this fusion always lead to positive gains? The paper lacks an analysis and discussion of how the fusion mechanism might trigger negative transfer.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "7SVcp4qv6h", "reviewer": "Reviewer_AMjb", "summary": "This paper focuses on the domain incremental learning (DIL) task and identifies component-wise misalignment between domain-specific prompts as a key factor that leads to conflicting knowledge integration and degraded predictions in prompt-based DIL methods. To address this issue, the authors propose the Componential Prompt-Knowledge Alignment (KA-Prompt) approach, which introduces a dual-phase framework to enhance component-wise alignment, thereby improving knowledge utilization during both training and inference. Extensive experimental results on four DIL benchmarks demonstrate that KA-Prompt achieves promising improvements compared to state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s constructive feedback and recognition. We hope the following responses effectively address your concerns.\n\n**W1: Performance analysis**\n\n(1) Both KA-Prompt and the C-Prompt baseline exhibit lower performance on the initial domain compared to other prompt-based methods due to the BEMA design proposed in C-Prompt. BEMA is a batch-wise antiforgetting strategy that prevents the forgetting of cross-domain shared parameters (e.g., classifier). Since BEMA constrains new knowledge learning, and the randomly initialized shared parameters lack semantic knowledge, it limits the initial performance of both KA-Prompt and C-Prompt, particularly on small-scale datasets like ImageNet-R.\n\n(2) To further analyze this effect, we removed BEMA during the first domain learning, and the results are presented in Fig. E of https://anonymous.4open.science/r/ICML-31/FigE-ImageNet-R.png. The findings show that: (a) Without BEMA, our model achieves performance comparable to state-of-the-art methods on the first domain, confirming that BEMA is the primary factor contributing to the degraded initial performance. (b) After learning from 15 domains, our KA-Prompt (w/o initial BEMA) achieves **66.54±0.68**, which is 0.03% higher than our results in the paper (w/ initial BEMA). This marginal performance improvement occurs because some discriminative knowledge is shared across domains. Thus, even if the initial domain is insufficiently trained, knowledge from later domains can still enhance its performance within our framework. (c) Overall, KA-Prompt achieves **4.08%** and **4.11%** improvements on ImageNet-R with and without initial BEMA, respectively, verifying its effectiveness in consolidating long-term knowledge. These improvements stem from our knowledge alignment design, which enhances cross-domain prompt compatibility, enabling more efficient utilization of learned knowledge and facilitating both positive forward and backward transfer.\n\n**W2: Methodological details**\n\nYour understanding is correct. \n\n(1) At the beginning of Greedy Prompt Search, the Reusable Prompt Memory is empty, and the Memory Matrix $S^M\\in \\mathbb{R}^{0\\times N_t}$ is an empty matrix. Then, the column–wise max process is conducted to obtain a $1\\times N_t$ null matrix. Consequently, the Assignment Matrix $S^*$ become a ($[(t−1)×N_p]×N_t$) null matrix. \n\n(2) We have incorporated these methodological details into our paper for improved clarity.\n\n**W3: Unify the component shapes**\n\n Thank you for this valuable suggestion. We have modified the shape of the ViT model in Fig. 1(b) to match Fig. 3, ensuring overall consistency. The revised version is illustrated in Fig. F in https://anonymous.4open.science/r/ICML-31/FigF-Modify.png.\n\n\n**Comments: Moving Figure 7 to main paper**\n\nThanks for the valuable suggestions and appreciation of our experimental designs. We have moved Fig. 7 to our main paper. We have moved Fig. 7 to the main paper. Additionally, the following experimental settings and quantitative analyses are included:\n\nTo demonstrate that componential prompt-knowledge alignment mitigates knowledge conflicts, we conducted an ablation study by shuffling prompt components under different conditions. The experiments were performed on the final domain of the DomainNet benchmark, where all learned domain prompts were used for prompt matching.\nAs shown in Fig. 7 (a), C-Prompt (Liu et al., 2024a) experiences performance improvement in Shuffle-B/C/D/E due to its randomly learned componential structure, which tends to be suboptimal in its original form. In contrast, KA-Prompt consistently degrades when prompt components are perturbed, indicating its intrinsic alignment structure. Despite the misalignment noise introduced by random shuffling, the worst performance of KA-Prompt (Fig. 7 (b) Shuffle-E) remains **6.4%** higher than the best performance of C-Prompt (Fig. 7 (a) Shuffle-B). This demonstrates that our greedy prompt search algorithm effectively extracts generalizable knowledge across domains, significantly enhancing adaptation to new domains.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "This paper does not provide a theoretical claim or proof. However, the challenges identified in existing methods are intuitive and are effectively demonstrated through well-designed experiments.", "experimental_designs_or_analyses": "Yes. I have checked the experimental designs and analyses in Figures 1, 2, 4, 5, 6, 7, 8, 9, and Tables 1, 2, 3. These experiments are sound and comprehensively demonstrate the motivation, effectiveness, and efficiency of the proposed method.", "relation_to_broader_scientific_literature": "This paper provides a novel solution to enhance cross-domain knowledge utilization during both testing and sequential training. The proposed ideas have the potential to inspire further research in areas involving domain shifts and incremental learning, such as transfer learning and class/task incremental learning.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Paper strength:\n1. The manuscript is well-organized and clearly written, making complex concepts accessible to a broad audience. The visual representations are well-designed and effectively illustrate the motivation, methodological designs, and effectiveness of the proposed method.\n2. A deep understanding of prompt-based DIL technologies is demonstrated through a comprehensive literature review. The identification of component-wise misalignment between domain-specific prompts, supported by well-designed experiments, is insightful and offers inspiration for further research in handling domain shifts.\n3. The proposed method is sound and innovative. First, the Greedy Prompt Search module provides a novel, low-cost solution to improve the utilization of historical prompts, which could be highly beneficial to incremental learning and transfer learning communities. Second, the Historical Prompt Online Aligning module dynamically matches historical prompts and adopts an adaptive strategy to constrain the prompt alignment, effectively addressing the componential knowledge misalignment identified in the paper.\n4. Extensive experiments on multiple benchmarks are conducted, with the proposed method demonstrating notable improvements in the DIL task. Additionally, sufficient ablation studies are included, verifying that the proposed designs effectively achieve the claimed objectives.\n\nPaper weakness:\n1. Some experimental results require further analyses. For instance, in Figure 4(a), KA-Prompt and the C-Prompt baseline underperform Dual-Prompt, CODA-Prompt, and CPrompt in the first domain. The reasons for this phenomenon should be discussed.\n2. Certain methodological details are not sufficiently introduced. Specifically, if the Reusable Prompt Memory is empty, the Memory Matrix should also be empty. In this case, does the Assignment Matrix S^* become a ( [(t−1)×N_p]×N_t) null matrix? The authors should clarify how such a case is handled, either in the main paper or in the appendix.\n3. The ViT model in Figures 1(b) and 3 should be illustrated with a consistent shape to improve clarity.", "other_comments_or_suggestions": "Figure 7 is a crucial experiment that verifies the effectiveness of the proposed approach in addressing knowledge misalignment compared to the existing methods. It is recommended that this figure be moved to the main paper.", "questions_for_authors": "This paper presents an new DIL approach with extensive evaluations. The presentation and experiments are comprehensive, I have a few minor concerns regarding the experimental analyses and implementation details for special cases. Please address these issues, as outlined in the paper weaknesses section, during the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JY1oxgvCp0", "reviewer": "Reviewer_3qyA", "summary": "Domain Incremental Learning (DIL) is crucial for processing data across different domains while maintaining previously acquired knowledge, but current prompt-based methods suffer from misalignment issues when integrating knowledge from different domains. The authors identify that this problem stems from random positioning of knowledge components within prompts, which can lead to interference when irrelevant components are combined. To address this, they introduce KA-Prompt, a novel method that focuses on component-aware prompt-knowledge alignment during the training process. The approach works in two phases: first establishing alignment between new and old prompts through initial structure configuration, and then preserving this alignment through dynamic identification of relevant prompts and adaptive consistency constraints. Through extensive testing on DIL benchmarks, KA-Prompt demonstrates significant improvements over existing methods, showing the effectiveness of their component-aligned approach.\n\n## update after rebuttal\n\nThe experiments provided in rebuttal demonstrate that reusable knowledge mining can capture richer and more accurate semantic representations. However, it seems that the observation that “the semantic partial knowledge of objects in the new prompts is continuously reinforced during incremental learning” has not been observed. Moreover, the contribution of the proposed prompt fusion mechanism is ambiguous due to the superiority of individual prompts. Therefore, I decided to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and recognition. Below are our responses, which we hope effectively address your concerns.\n\n**Q1-1: Requirement of maintaining a prompt pool.**\n\n(1) Maintaining a prompt pool is not a specific requirement of KA-Prompt but is already present in the baseline method.\n\n(2) A prompt pool is fundamental to prompt-based continual learning methods, enabling the retention of historical knowledge with negligible storage overhead. All compared state-of-the-art prompt-based approaches, including L2P, S-Prompts, DualPrompt, CODA-Prompt, CPrompt, and C-Prompt, incorporate a prompt pool.\n\n**Q1-2: Additional training overhead**\n\n(1) Maintaining a prompt pool does not introduce additional training overhead since stored prompt sets are frozen after being added and are not further optimized. \n\n(2) In our KA-Prompt, frozen prompts from the pool are selected to guide cross-domain componential knowledge alignment in the Historical Prompt Online Aligning (HPOA) branch. This branch shares nearly the same computational cost as the New Prompt Training branch, introducing only an additional forward pass. However, the HPOA branch contributes a **2.28%** improvement when applied to the baseline model on the ImageNet-R benchmark, as shown in Fig. 5 of our main paper.\n\n**Q2: Aligned prompt update**\n\n (1) The aligned prompts are updated during training to encode knowledge from new domains.\n\n (2) The prompt parameters in the Reusable Prompt Memory remain frozen during the aligned prompt update, ensuring they are unaffected during training.\n\n**Q3: Purpose of Historical Prompt Online Aligning module**\n\n The Historical Prompt Online Aligning (HPOA) module aims to align the new prompts to parts of historical prompts in the prompt pool. Specifically, as new prompts are updated, HPOA dynamically matches them to the closest historical prompts, assigns fusion weights, and fuses them. The fused prompts are then fed into the ViT model, ensuring cross-domain componential alignment throughout training.\n\n**Q4: Sblation study**\n\n(1) By default, module ablation studies in the main paper are conducted on ImageNet-R. This has been explicitly stated in the revised version.\n\n (2) Additional ablation studies on ImageNet-Mix and DomainNet are provided in Fig. G of https://anonymous.4open.science/r/ICML-31/FigG-Ablation.png. \n Specifically, the results show that: \n \n (a) Our prompt initialization strategy (Reusable Knowledge Mining, $\\boldsymbol{f}_R + \\boldsymbol{f}_G$) significantly outperforms the existing methods (Wang et al., 2023a), due to the improved historical knowledge utilization capacity.\n \n (b) Our online alignment design ($\\boldsymbol{f}_A$) consistently improves model performance by strengthening knowledge alignment during training, thereby enhancing cross-domain fusion compatibility at test time.\n \n (c) When all our modules are used together, the model performance is further improved since the historical knowledge utilization is improved during both training and testing.\n\n**Q5: Heat map visualization**\n\n Thanks for your suggestion! We have visualized the attention maps of prompt tokens at different stages, along with the fused tokens, in  Fig. A of https://anonymous.4open.science/r/ICML-31/FigA-Heatmap.png. The results show that each prompt token (component) captures a semantic part of objects.\n \n(1) Due to semantic misalignment among tokens, the fused prompt tokens in C-Prompt fail to precisely capture object-specific information, preventing the model from fully leveraging discriminative features.\n \n(2) Our method improves component-wise alignment, enabling the fused prompt token to effectively capture object features. This results in a **4.73%** Average accuracy improvement in Tab. 1 of our main paper.", "claims_and_evidence": "Yes. This paper randomly shuffles different components of prompt during prompt fusion, and the resulting fluctuations demonstrate that the previous fusion method did not achieve knowledge alignment.", "methods_and_evaluation_criteria": "The proposed method is meaningful for the domain incremental learning it focuses on. The paper provides evidence that KA-Prompt enhances the ability to integrate cross-domain knowledge in continuous learning. However, further analysis is needed regarding the mitigation of knowledge mismatch in the prompt component.", "theoretical_claims": "This article does not involve theoretical proofs.", "experimental_designs_or_analyses": "The authors conducted experiments on four benchmark datasets and compared state-of-the-art cue-based incremental learning methods. The overall experimental design is sound.", "relation_to_broader_scientific_literature": "This paper improves on previous work in two ways:\n\n1. It optimizes the initialization mechanism of the newly introduced prompt. Instead of directly using the prompt obtained from the previous task as the initialisation, a suitable prompt is searched from the existing prompts as the initialisation of the new prompt.\n\n2. Different from previous work that directly merges the components in the corresponding positions in the prompt, this article aligns knowledge before merging, thereby reducing the ineffective merging of unrelated knowledge.", "essential_references_not_discussed": "In my understanding, the important references have already been discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The figures and tables in this paper are clearly represented.\n2. The paper provides a detailed and clear description of the limitations of the previous work and the motivation for the proposed methodology.\n\nWeaknesses:\n\n1. Insufficient analysis of experiments.\n2. Some parts of the proposed method lack definition.", "other_comments_or_suggestions": "None.", "questions_for_authors": "I have some concerns about this paper:\n\n1. As shown in Figure 3, KA-Prompt needs to maintain a prompt pool in addition to the prompt set. Is this a requirement specific to KA-prompt or is it already present in the baseline method? The additional training overhead associated with prompt pools should be discussed.\n2. Does the aligned prompt get updated during training? If so will these updates have an effect on the prompt parameter in Reusable Prompt Memory?\n3. Following up on the previous question, is the purpose of Historical Prompt Online Aligning to align the new prompt to the historical prompt pool, or vice versa?\n4. The ablation study does not seem to indicate on which dataset it was performed. Ablation studies on more datasets can further demonstrate the contribution of the proposed module.\n5. Can the misalignment illustrated in Figure 2 and the optimization resulting from the proposed method be further analyzed in the form of a heat map visualization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews: three positive, one negative.
It presents the Componential Prompt-Knowledge Alignment (KA-Prompt) for Domain Incremental Learning (DIL), aiming to address the knowledge conflicts and interference caused by the misalignment of components between domain-specific prompts. The positive reviews praise the methodology’s feasibility and novelty, the insightful perspective on prompt-based DIL researches, and the clear, and well-organized writing. The authors conduct a sufficient ablation study to verify the performance improvement achieved by the proposed designs. However, concerns were raised about the lack of experimental support for the motivation and contribution of this paper beyond accuracy. The reviewers urged the authors to explain how “alignment” was measured and evaluated. Detailed settings for classifiers and a comparison with the important SOTA method (S-Prompt) also need to be clarified.
The authors made significant efforts in their response, the Area Chair (AC) feels that the idea of identifying the component-wise misalignment between domain-specific prompts is interesting and inspiring for DIL. The AC recommends accepting the paper, but the concerns raised in the comments should be addressed when preparing the final version.
Decision: Accept (poster)