Not Just Object, But State: Compositional Incremental Learning without Forgetting
Authors: Yanyi Zhang
Binglin Qiu
Qi Jia
Yu Liu
Ran He
Venue: NeurIPS 2024 poster (2024)
Keywords: Incremental Learning
Compositional Learning
Prompting
Primary area: machine_vision

Abstract: Most incremental learners excessively prioritize object classes while neglecting various kinds of states (e.g. color and material) attached to the objects. As a result, they are limited in the ability to model state-object compositionality accurately. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL), which enables the model to recognize a variety of state-object compositions in an incremental learning fashion. Since the lack of suitable datasets, we re-organize two existing datasets and make them tailored for composition-IL. Then, we propose a prompt-based Composition Incremental Learner (CompILer), to overcome the ambiguous composition boundary. Specifically, we exploit multi-pool prompt learning, and ensure the inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting which injects object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-art performance achieved by CompILer. Code and datasets are available at: https://github.com/Yanyi-Zhang/CompILer.

Not Just Object, But State: Compositional Incremental Learning without Forgetting
Yanyi Zhang ' Binglin Qiu / Qi Jia Yu Liu 1| Ran He
1 International School of Information Science & Engineering, Dalian University of Technology 2 MAIS&CRIPAC, Institute of Automation, Chinese Academy of Sciences yanyi. Saany.iaang@mail.... wwwttailllll......mmmma dlut. edu. en, mlandy@mail dlut. edu. jiaqiedlut edu. cn, 1iuyu8824ddlut. edu. cn, rhe@nlpr ac. ac..c..c.c
Abstract
Most incremental learners excessively prioritize coarse classes of objects while neglecting various kinds of states (e.g. color and material) attached to the objects. As a result, they are limited in the ability to reason fine fine-grained compositionality of state-object pairs. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL) enabling the model recognize state-object compositions as a whole an incremental learning fash- ion. Since the lack of suitable benchmarks, we re-organize two existing datasets and make them tailored for composition-Il. Then, we propose a prompt-based Composition Incremental Learner (ComplLer), overcome the ambiguous com- position boundary problem which challenges composition-IL largely. Specifically, we exploit multi-pool prompt learning, which is regularized by inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting by using object prompts guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-an performance achieved by ComplLer. Code and datasets are available https:/ htt///gttttbb.t/// wtt.hu..cco//// wwt.cco////aaaaa/ com/Yanyi- Sairyy--Zhange -Zhang/ CompILer
Introduction
gathers increasing attention due its ability to make the models learn new tasks rapidly, without forgetting previously acquired knowledge. Yet, traditional class-IL sets strict limit on the old classes such that they should not recur in newly incoming tasks. To break such strict limitation, recent studies develop a new setting mostly called Blurry Incremental Learning (blur-IL) 24 11. where the incremental sessions allow the recurrence of previous classes, resulting in more realistic and flexible scenario. Despite such empirical progresses on incremental learning, they aim to improve object classification only, while overlooking fine-grained states attached to the objects. For instance, analyzing how the clothing styles (akin to states) have changed over time is important for forecasting the future trends that will emerge.
Class Incremental Learning (class-IL) 37 22 16
10
To simultaneously model objects and their states, some efforts are dedicated to Compositional Learning whose aim is how to equip the models with compositionality 44] 31 45). The core of compositional learning lies in the structure of class labels, which conceptualizes a state-object pair (e.g. "Brown Pants" and "Yellow Dress") as whole, rather than a lonely object label. this way, the model can dissect and reassemble learned knowledge, achieving more fine- tne-ggained understanding about the objects. However, existing works are mainly focused on zero-shot generalization from seen
corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Class Incremental Learning
(b) Blurry Incremental Learning
(c) Compositional Incremental Leurning
T
Data
T1
Data
Ti
Data
Boswn
Yellow
Parti
Diess
Suil
Pusts
Dress
Sui
Dicie
X
No Recurrence
Class Recumence
Primitives Recurrence
T2
Data
T2
Data
T2
Data
Sliver
Yellows
Shoes
Shoors
Class Label: Object
Class Label: Object
Class Label: Stute-Objeet
Figure I: Differences between Class Incremental Learning (class-IL), Blurry Incremental Learning (blur-IL), and Compositional Incremental Learning (composition-ll), The object classes are not allowed to recur in the class-IL scenario, whereas they may recur randomly in the blur-IL scenario. Different from them, the classes in composition-Il involve state-object compositions apart from the object classes. Besides, the compositions do not reoccur, but the primitives (states objects) may randomly reappear across incremental sessions.
whereas none of them consider the challenging fact that the model must deal with a significantly larger number of composition classes than object classes. As result, it is hardly feasible to learn all compositions by training the model once.
compositions to unseen ones 26 27
To remedy the limitations inherent in incremental learning and compositional learning, conceive novel task named Compositional Incremental Learning (composition-Il)) enabling the model to continually learn new state-object compositions in incremental fashion. As compared in Fig. [ we can see that composition-ll integrates the characteristics of class-IL and blur-IL. Although the composition classes are disjoint across incremental tasks, the primitive classes (i.e. objects and states) encountered in old tasks are allowed to reappear in new tasks. Unfortunately, existing incre- mental learning approaches are challenged by such a compositional scenario, because their models excessively prioritize the object primitives while neglecting the state primitives. Consequently, the compositions with the same object but with different states become ambiguous and indistinguishable.
To tackle the problem, we propose a rehearsal- and prompt- based Compositional Incremental Learner (ComplLer). Specifically, prompt learning, object-injected state prompting, and generalized-men construct three prompt pools for learning the states, objects and compositions individually. Upon that, we add extra restrictions to regularize the inter-pool prompt discrepancy and intra-pool prompt diversity. This multi-pool prompt learning paradigm strengthens the fine-grained understanding and reasoning towards primitive concepts and their compositions. In addition. as the state classes are more difficult distinguish than the object ones, we propose object-injected state prompting which incorporates object prompts to guide the selection of state prompts. Furthermore, we fuse the fusion manner, which helps to adaptively eliminate irrelevant
model comprises of three primary components: multi-pool prompt fusion. Firstly, we
selected prompts by a generalized-ment information learned in the prompts. Last but not least, we also leverage symmetric cross-entropy loss alleviate the impact of noisy data during training.
In summary, the main contributions in this work are encapsulated as follows: (1) We devise new task coined compositional incremental learning (composition-IIL) enables learning fine-grained state-object compositions continually while the isolated primitive concepts can randomly recur in incremental tasks. (2) To address the lack of datasets, we re-organize two existing datasets such that they are tailored specifically for composition-Il IL. For the two new datasets (Split-Cl) Siliit-Cllhingg and Split-UT-Zappos), we split them into and 10 incremental tasks for evaluating the methods. (3) We propose novel learning-tor toppiinttttppppppp model for composition-IL, namely ComplLer. Our state- of-the-art results on pplit-Clothing and Split-UT-Zapp validate the effectiveness of ComplLer for incrementally learning new compositions without forgetting old ones.
Related Work
Incremental Learning. The approaches to addressing catastrophic forgetting for incremental learning can be broadly grouped into four categories: regularization based methods [6] 39 aim to protect

---

Split-Clothing (5 Tasks)
Split-UT-Zappos (5 tasks)
Split-UT-Zappos (10 tusks)
Task
Task
Task
Task
Task
Task 6
Task
Task 8
Task
Task 10
Figure 2: Data Statistics of Cloth-Cl and Split- for tasking composition-Ile Split- includes both 5-task and 10-task
Clothing is divided into a 5-task scenario, while scenarios. In all settings, the number of images per task has been balanced properly.
influential weights of old experiences from updating; knowledge distillation based methods [16 distill knowledge from the model trained on the previous tasks and adapt it to new tasks; rehearsal require memory buffer to store some old data, so to make the network remember previous tasks; parameter isolation methods allocates different model parameters each task, prevent any possible interference. Different from the methods, L2P [43] proposes an innovative learning-to- creeing-tt-pooon pororippppppppppp paradigm, which incorporates plasticity and stability through adapting of learnable prompt tokens top of frozen pre-trained backbone. Inspired by L.2P 43], more recent works [42 36, (2, 28 take full advantage of various prompt tuning strategies, achieving new tfate-of-the-ars performance for incremental learning. However, such methods take into account object classes solely, while neglecting various kinds of state classes associated with the objects. To this end, our work proposes compositional incremental learning with the purpose continually identifying the composition classes of state-object pairs. Note that, Liao, et al 18 conduct an initial study toward the compositionality in incremental learning, whereas their attention is on the composition of multiple object classes (e.g. "Car" and "Person"") in one image, rather than the state-object compositions in this work.
based methods 32
48
20
Compositional Learning. A major line of compositional learning research focuses on Compositional Zero-Shot Learning (CZSL) [23]. which aims infer unseen state-object compositions by acquiring knowledge from seen ones. Subsequent approaches building upon the CZSL setting further incor- porate graph neural networks to model the dependency between primitives and compositions (25), and employ cosine classifiers to avoid being overly biased toward seen compositions (2]] Other approaches L3
33
14
12
propose training two classifiers to identify states and objects separately. The
477 model both composition and primitives simultaneously. achieving state-of-the-art results. Albeit the numerous attempts made in compositional learning, they fail to consider an incremental learning paradigm given the increasing number of composition classes in open-world scenarios. Besides, directly applying CZSL methods to composition-ll might lead to stale and decaying performance on forgetting. By contrast, our proposed Compller markedly bypasses catastrophic forgetting with the help multi-pool prompt learning.
latest works [19
3
38
8
3
Preliminaries
In this section, we firstly define the task of composition-IL, and then introduce two datasets we construct for the task, followed by revealing the ambiguous composition boundary problem.
3.1 Problem Definition
For composition-ll a model sequentially learns N tasks T (Ti, -.,.--TT'' corresponding to set of composition classes C {C1, C.. C. We note that the composition classes between incremental tasks are always disjoint, which means C, ne; 0 for any i * j. Different from the composition classes, the primitive classes are allowed to recur in different tasks. That means allows the tasks to share some primitive concepts of objects and states. Therefore, we can define the set {or, 02, ... .0,, respectively. Given each image it has composition label which is constructed with state label s and an object label o, i.e. e =< 8, 0,02 >, where E C, 8 E S and E O. We take the example of "red shirt", where "red" is denoted with "shirt" corresponds to and "red shirt" is expressed with
of all state and object classes with = (81, 82. 88) and

---

Red Shoes
White Shoes
Pink Skirt
White Dress
Blue Dress
Black Dress
Yellow Dress
(a) L2P
(b) Compiler
Figure 3: r-SNE feature distributions of seven compositions from the benchmark. For the compositions with the same object but with different states, our Compll er achieves more distinguishable boundaries than the 1.2P baseline.
3.2 Dataset Construction
As there are no existing datasets suitable for composition-IL. we re-organize the data in Cloth- ing16K [46] and UT-Zappos50K 44. and construct two new datasets tailored for composition-IL. namely Split-Clothing and Split-UT-Zappos. To be more specific, we firstly sort the composition classes based on the number of their images, and then select the foremost 35 compositions from Cloth- ing16K and the top 80 from UT-Zappos50K, so as to construct Split-Clothing and Silit-UT-Zappos. respectively. In this way, Split-Clothing encompasses 9 states and 8 objects while Split-UT-Zappos consists of 15 states and 12 objects in total. For Split-Clothing, we randomly partition the composi- tions into tasks. Regarding Split- (T--appos, the compositions are sorted by count and are evenly divided into 5 and 10 tasks. The image distribution for each task is shown in Fig. 2 Note that, elaborate more details on both datasets the following technical appendix.
3.3 Revealing the Ambiguous Composition Boundary
The main stumbling block in composition-ll IL is the ambiguous composition boundary. Although the composition label consists of two primitives (i.e. object and state), we note that the model excessively prioritizes the object primitive while neglecting the state primitive. Consequently, the compositions with the same object but with different states become ambiguous and indistinguishable. To prove that, apply L2P 4 in composition classification. As illustrated in Fig. 3 (a), the i-SNE visualization showcases the entanglement among the compositions like "white dress", "black dress" and "blue dress". We conjecture that this ambiguous problem tends to become more severe when more tasks are arriving incrementally. To address it, we propose new model namely ComplLer, which disentangles compositions and primitives via a multi-pool prompt learning. Advantageously. our method promotes the learning on the states and establishes clearer composition boundaries, as shown in Fig. (b).
composition-IL, whereas it is challenged by significant ambiguities
Methodology
Overview. We leverage the marning-to-promen paradigm 4 and develop novel compositional incremental learner (ComplLer) tailored specifically for composition-IL. As depicted in Fig. 4 ComplLer comprises three primary components: multi-pool prompt learning, object-guided state prompting, and generalized-mean prompt fusion. Firstly, we initialize three prompt pools dedicated learning and storing visual information related states, objects and their compositions. In order to
differentiate the knowledge learned across and within prompt pools, we define inter-pool discrepant loss and intra-pool diversified loss jointly. We then employ object prompts guide the selection of state prompts, thereby improving the state representation learning. Moreover, we utilize a generalized- mean fusion to integrate the selected prompts learnable manner. Ultimately, we optimize the classification objective with symmetric cross-entropy loss, to alleviate the effect of noisy data.
4.1 Multi-pool Prompt Learning
The marming-to-promy paradigm 35
especially suitable for large pre-trained backbones, has opened up new path for incremental learning. It has proven to incorporate plasticity and stability

---

Selection
Kg
Fratun
Objjcct Pronze Pool
(P Ri-1
Embedding Layer f
Orject injected
Selection
Gebc Finisr
[6]
Stato Prompong
K
Yallw Decw
Prompt
[P Pim1
Transformer Encoder f.
Selection
GeM P..
Composition Prompt Pool
(P
AScv
Lomabli:
Frezen
Image Tokee
Pair
Classifee
Composition
Object
Figure 4: Overall architecture our composition incremental learner (ComplLer), which comprises multi-pool prompt learning, object-injected state prompting, and generalized- prompt fusion. The multi-poo prompt learning mechanism captures information related to states, objects, and their compositions, each through dedicated pool. The object-injected state prompting utilizes the object prompt to promote the state representation learning. Moreover, the generalized -eenn -mean prompt fusion used to prioritize the useful prompts and diminish the irrelevant ones.
better through adapting a set of learnable tokens in a prompt pool to a frozen pre-trained backbone. Nevertheless, existing prompt-based approaches are initially designed for class-IL, thereby building a single prompt pool for object classification solely. when dealing with state-object composition classification, they tend excessively prioritize the object primitive while neglecting the state primitive. To this end, we propose to construct three discrepant and diversified prompt pools P.,. F. and P.,, which serve learn visual information related states, objects and their compositions. respectively. Besides, each pool is associated with a set of learnable keys i for query-key prompt selection. The three prompt pools and their keys are defined as:
Po = (P). Pa pM
KK = { [kg K
K w (s,0,c),
(1)
where pi E Lx is a single prompt with token length L and embedding dimension D. KJ E R, the key of P learnable token with the same size. M is the number of prompts in each pool.
One important concern in such multi-por prompt learning is how to enrich the prompts with the avoidance of identical pools. To achieve it, we consider integrating inter-pool prompt discrepancy and nntra-pool prompt diversity jointly. On the one hand, the inter-pool prompts should be discrepant the visual information about states, objects, and compositions should be different. One the other hand, within each pool, the intra-pool prompts should be diversified so to capture more comprehensive features from all the classes.
In practice, we formulate a unified objective to regularize both inter-pool discrepancy and intra-pool diversity, by leveraging simple and effective directional decoupled loss used in [17). The directional decoupled (dd) loss between any two pools (e.g. P. and P5) is formulated as:
M
M
(1,)
2
max (0, Othrc - 0mm))
(2)
ddl
M(M 1)
m=
( PM
0 0um = cos
max PM é) max PTm
where fmm measures the angle between any two prompts, n and m; is scalar to avoid division by zero. Note that, ((ij)) encourages the angles between each prompt to at least Ottre degrees. Since (i,j) is unordered Cartesian product of w, i.e. (i,j) E {(i,5) i i w A.j E w/. the inter-pool prompt discrepancy loss for the three pools can be expressed with Linter = dd + C + cla.)) and the intra-pool prompt diversity loss becomes Lintre ((s.0) + c(o,0) + (c,c) As opposed to Linters Lintra computes the angle between any two prompts within the same pool. Thus, contains the case when n m, for which we set 6 0nm 0.
((s.c)
5

---

4.2
Object-injected State Prompting
Akin to the query-ken matching mechanism in other work [43] 42 40), we utilize fixed fea- ture extractor f to obtain a query feature ((x) [x)) (0, :). determining which prompts the pool to be selected. However, pre-trained backbones are typically trained for object clas- sification, thus under-performing for state rep- resentation learning. In addition, it is more dif- ficult to predict the state classes due to their more abstract and fine-grained characteristics. To tackle this problem. we strategically inject object prompts to guide the selection of state prompts. Intuitively, once we have learned knowledge about the object class, it may be eas- ier to predict the correct state class and avoid mistaken results. For instance, given object is "heels", we can expect that the corresponding state is unlikely be "canvas" or "plastic". To summarize, select object and composition prompts in each pool based on the original query feature, which means qo (x) = (e(x)) - q(z); but for the selection of state prompts, we propose object-injected state prompting to ameliorate the query feature as shown in Fig. 5
Fused Object Prompt Po
Query Feature (x)
Dot Product
Token Token
Token
Injected Query Feature (x)
Figure 5: Architecture of object-injected state prompting. Query feature serves as Q. while fused object prompt serves both K and V.
Specifically, employ the fused object prompt Po (see Sec. 4.3 to perform cross attention on the query feature ((x), resulting in object-injected query feature (x()) for the state prompt selection:
4s (I) Cros)Autn(g()). P.) Softmax ((=) P.Wk
P.WV,
VD
where Wº. W and WV are learnable projections. To establish alignment between the query and the selected prompts, we optimize a surrogate loss for state, object and composition prompting jointly:
Cour Cos(sfulee Kt), w E {s,o,c),
(5)
O
where ..S(..)) denotes cosine similarity, K., represents the subset of top-k keys selected from K and { is subset of top-k indices from [1, M] (prompt number). Despite the simplicity of the judicious prompt selection within the state prompt
object-injected state prompting, facilitates pool, alleviating the hurdles posed by state learning.
Generalized-men -men Prompt Fusion
After obtaining the selected top-k prompts {P*)) /==1. the next step is fusing these prompts into single prompt. It is general to utilize simple mean pooling whereas it overlooks the relative importance of each prompt. Besides, when the prompts contain information that is unrelated or contradictory current task, it is critical to strengthen useful prompts and eliminate irrelevant ones. To this end, we draw inspiration from generalized-menn pooling (2 and exploit generalized-mean (GeM) prompt fusion which given by:
Ps = GeMu P
P.*)
Ps.
w E (s,
k
where 17 is learnable parameter. When 1 GeM becomes mean pooling; as r) approaches infinity (n + (o), it converges to max pooling. By taking over mean and max pooling, GeM learns to achieve an optimal fusion, mitigating the influence of irrelevant information present in the prompts.
4.4 Training and Inference
Classification Objective. We prepend three fused prompts (i.e. P., P. and P.) with va, which the output from ViT embedding layer f-). The extended token sequence is Zp [P: P. Poi Po]

---

Table Avg Ace and FTT results on Split- Clothing tasks) and Split-UT-Zappos (5 and tasks).
The best results are marked in bold. All results with standard deviations are averaged over three runs.
Datasets Metrics
Split-Clothing (5 tasks)
Split-UT-Zappos tasks)
Splic-UT-Zappos (10 tasks)
Avg Acc(f)
FTT())
Avg Ace(())
FTT())
Avg Ace(())
FTT())
Upper Bound
97.02=0.10
68.71+0.41
68.71+0.41
BWC [00
47.8
32.1520.44
37.3942.06
55.7042.76
24.63300994
61.31+2.29
LwF [6
49.96=0.68
44.22=0.53
40.15+0.43
49.6140.68
30.38+1.41
58.15
iCaRL [32
68,65=0,41
31.74=1.89
37.7842.14
55.0643.50
11.40+1.96
59.6542.44
T.2P [43
80.22=0.4
14.232044
42.20+2.18
20.41+2.76
31.6540.166
31.0241.66
Deep 1-2++ 43 33
80.55=0.45
12.60=1.90
42.37+0.65
30.10+1.56
30.68+0.35
32.20+1.90
Dual-Prompt
42
87.87=0.63
7.71=0.25
43.30=0.19
19.41+2.80
33.01#1.65
24.61=1.11
CODA- Prompt 33
86.35=0.20
8.99=0.71
43.3540.29
21.76+2.45
31.40+0.36
30,54+2.63
LGCL 7
87.32=0.10
7.58=0.06
33.5640.31
24.3740.56
Sim- ComplLer
88.38+0.08
8.01=0.42
45.7040.68
20.0640.62
33.30+0.10
30.31+0.03
ComplLer
89.21=0.24
7.2644.60
46.4820.26
19.2740.75
34.43+0.07
28.69
Then, feed Vp transformer encoder layer f+(+) and achieve P P", and P for classifying state, object and composition classes, respectively. We estimate the probability via a classifier (-)
p(w .r) f (P.). For each image I, denote its ground-truth distribution over labels with
q(w r)) When w is consistent with the ground truth, then g(w x) = otherwise, q(w x)) = 0.
As result, the cross entropy (CE) loss used for classification objective is:
f2 CC Σ q(w x)) log x)) 22 E [|S 10), [C]]
where Ω represents the number of classes. However, the model optimized with standard CE loss is easily affected by noisy samples during training. Instead, we advocate using symmetric cross entropy loss (SCE) [I]. which incorporates an additional term called reverse cross entropy (RCE), to mitigate the impact of noisy data. Contrary to CE, the formula for RCE loss is defined as:
Ω RCC Σ plw æ)) log g(w x)) E (s,0,c}.
(8)
Then, the SCE loss combines two loss terms by SsC LGE + a aL where a hyper- parameter that controls the weight of the RCE term. As a result, the whole SCE loss becomes LSCE CACE + ((CSCE + CSCE), where adjusts the weights between primitives and compositions.
Total Loss. The total loss for training the whole ComplLer model is:
Liotal = Cinter + Xe + Xg + LSCE,
(9)
where Å1, Ag. 03 are hyper-parameters balancing different terms.
Inference. During inference, we incorporate the primitive probabilities to aid the composition probability. Hence, the final probability for composition classification is expressed with:
Plinal (c a:) p(c] x) + p(p(s x)) + p(o x)),
(10)
where x adjusts the probabilities.
Experiments
5.1 Datasets and Metrics
We conduct experiments on two newly split datasets: Split-Clothing and Split-UT-Zappos as eluci- dated in Section 3.2 We assess the overall performance on compositions using Average Accuracy (Avg Acc) and Forgetting (FTT). A higher Avg Acc signifies stronger recognition abilities, while lower FTT indicates improved resilience against forgetting. Additionally, we provide individual Aver- age Accuracy scores on states and objects, denoted as State and Object for simplicity. These metrics imply the ability to recognize fine-grained primitives. Furthermore, calculate the Harmonic Mean (HM) between State and Object, i.e. HM 2 SstatexObjeca We provide more emphasis to Avg [State++ Ace and HM due their more comprehensive assessment. Avg Acc encompasses the plasticity and stability B3 2 and HM provides a holistic evaluation both state and object.
7

---

5.2 Implementation Details
For a fair comparison with previous works 443 42 [2 we also employ ViT B/16 I pretrained on the Ima- geNet 1K dataset the feature ex- tractor and backbone. For multi-pool prompt learning, the size of each pool set to 20, and each prompt has tokens. We select top-5 prompts from each pool and generate a fused prompt. During training, we utilize the Adam optimizer [9] with a batch size of 16. The whole ComplLer undergoes training for 25 epochs on the Split- Clothing, for 10 epochs the 5-task Split-UT-Zappos, and for 3 epochs on the 10-task Split-UT-Zappor learning rate to 0.03, while we use learning rate of 0.02 for the 5-task Split-UT-Zappos. Note that, for all the methods, their results are averaged over three runs with the corresponding standard deviations reported to mitigate the influence of random factors.
Table State, Object and HM results on Split-Clothing The best results are marked in bold.
Datasets
Split-Clothing (5 tasks)
Metrics
State
Object
HM
Upper Bound
97.4420.08
97.09=0.10
97.26+0.08
EWC [0
86.49+0.97
52.72+1.30
675.50=0.97
LwF [6
87.11=0.66
54.57=0.69
67.10+0.33
iCaRL [32
91.2111105
71.70=0.99
80.28+0. 74
L2P 43
83.03=0.42
95.5620.57
88.85+0.1 16
Dual-Prompt 42
90.77=0.25
94.18+0.31
92.45+0.20
LGCL 7
91.45=0.20
94.87=0.33
93.1340.
CCmplLeer
91.1520.10
96.3210.02
93.66+0.02
Compll.rr
91.81=0.23
96.67=0.01
94.18:0.06
For the Split-Clothing and the 10-task Split-UT-Zappos, we set the
As there few hyper- parameters in the model, conduct rigorous tuning them. For instance, we set Oor to 2
T. for all settings. For Split-Clothing the loss weights x1 and A3 are set to 0.1; A2 is set to 10-7: a and / for SCE loss are 0.006 and 0.3, and the parameter je during inference is 0.5. For 5-task Split-UT-Zappaa respectively. For 10-task Split-UTTZZppaar ' 22, A3, a, 3 and j are set to 0.5, 10-?, 7 0.1, 0.05, 0.4 and 0.03. We elaborate more details on hyper-parameter analysis in the appendix.
A1, Ag As a, B and fu are to 1.0, 3 x 10-6, 6, 0.7, 0.01, 0.7 and 0.02,
5.3 Compared Baselines
To demonstrate the effectiveness of the proposed method, we compare ComplLer with state-of-the-a incremental learning methods, including prompt-free approaches [10
16
32 and prompt-base
methods 43 42
Note that, due to LGCL 7 relying on CLIP 30 to achieve language guidance at the task level, it is limited by the length of class names per task. Thereby, LGCL fails to operate on the 5-task Split-UT-Zapoo since the total length of class names exceeds the limitation.
33
Z]. All the methods are rehearsal-free except iCaRL 32
To streamline our ComplLer, we further implement simplified version named Sim-ComplLeer and report its results. Sim-ComplLer is optimized using cross entropy loss and is comprised solely of multi-pool prompt learning and generalized mean prompt fusion. In other words, we exclude the object-injected prompting, directional decoupled loss, and reverse cross entropy loss, resulting in large reduction of hyperparameters only B, X3. and p.
5.4 Comparison with the State-of-the-arts
The compared results on Avg Acc and FTT are reported in Table I Overall, ComplLer consistently outperforms all competitors on Avg Acc by a significant margin. For FTT scores, ComplLer excels previous methods with 0.32% on the 5-task Split-Clothng and with 0. 1.14% on the 5-task Split-UT-- Zappos, while falling behind Dual-Prompp [42 and I.GCI. [7] for the 10-task Split-UT-Zappos. We notice that, the main reason is these methods sacrifice more plasticity for lower forgetting rates. Besides, the number of model parameters in these methods dynamically increases along with more incremental tasks arriving, whereas our ComplLer does not rely on imposing task-specific parameters to reduce the forgetting.
We also report the primitives accuracy and their HM in Table 2 and Table Likewise, our method surpasses other methods considerably in terms of State and HM. Interestingly. the prompt-free methods 16 10 32) achieve higher accuracy in state prediction than object prediction for Split- Clothing, which is contrary to other results. This is because the states in Split-Clothing are color- related deseriptions, which are easier to capture with the help of parameter fine-tuning. The prompt- based methods do not exhibit this phenomenon because their pre-trained backbones are initially
8

---

Table State, Object and HM results on Split-UT-Zappon (5 tasks) and SpaipoUT (10 tasks).
Datascts
Split-UT-Zappos (5 tasks)
Split- UT-appo- (10 tasks)
Metrics
State
Object
11M
State
Object
11M
Upper Bound
15.10=0.10
88.1330.03
81.90#0.06
75.10=0.10
88.1340.03
81.90+0.06
EWC [00
47.95=1.26
76.53+0.91
58.90=0.53
39.29+2.69
67 67.64+1.99
49.69=2.30
LwF [16
53.13=1.08
75.48+0.82
62.35+0.31
38,70+2.33
68.90+1.97
49.54=1.30
iCaRL 32
51.71=0.95
75.03+0.49
61.22+0.78
38,9442.01
67.10+1005
49.27=1.58
L2P]43]
$2.20=2.32
79:05=0:0T
62.875161
42.6640087
76.60+0.03
54.80=00:35
Dual-Prompt 42
52.25=0.77
77.46+0.05
62.40+0.34
44.34+1.61
77.92+0.37
56.51+1.11
LGCL 7
43,44=0.79
78.64+0.64
55,96#0,43
Sim-Compie Can-ComplLeer
55.93=1.23 56.85=0.34
79.6940.06 79.5620.04
65.7240.53
45.88+0.38
75.72+0.67
57.1420.06
ComplLer
66.31+0.15
46.2741.56
76.65+1.19
57.69=0.42
Table 5: Ablative experiments for (a) object-injected state prompting, (b) prompt fusion method. (a) Object-injected state prompting.
(b) Prompt fusion method.
Dataset
Split-Clothing (5 tasks)
Dataset
tasks)
Metrics
Avg Ace
FTT())
HM
Metrics
Avg Acc
FTT())
HM
None
88.45+0.100
7.93#0.11
93.70=0.03
Max
84.7020.64
12.24+2.25
91.3420.30
S~O
88.27=0.02
7.99+0.05
93.67+0.01
Mean
87.80=0.12
7.82=0.01
93.38+0.03
0-S
89.21=0.24
7.26+0.60
94.18+0.06
GeM
89.21=0.24
7.26=0.60
94.18+0.06
trained for object classification, and are frozen across incremental sessions. As the performance improvements are mainly attributed to the accuracy state recognition, it suggests that our model enhances the understanding fine-grained compositionality.
5.5
Ablation Study and Analysis
Effect of multi-pool prompt learn- ing. This experiment aims to delineate the contribution of three pools in Com- plLer. We firstly implement baseline model with composition prompt pool only. Building upon the baseline, develop two additional models, which incorporate either object state prompt pool. As reported in Table the in- clusion of primitive prompt pool yields consistent gains over the baseline. Fur- thermore, the best results are achieved when the model integrates all three pools simultaneously. This experiment signifies the sigmificant necessity of exploiting multiple prompt pools for composition-IL.
Table 4: Ablation study on multi-pool prompt learning with Split- clott-Clo dataset.
Prompt Pool
Split- (5 tasks)
S
0
Avg Acc
FTT())
HM
80.22+0.41
14.23+0.44
88.85+0. 16
88. 18.00...11
7.79#0.04
93.55+0.04
/
88. 88.09+0.50
7.2640.54
93.52+0.15 13
/
88.38+0.08
8.0140.42
93.6610.02
Effect of object-injected state prompting. To provide insights into object-injected state prompting. we compare three models: None (vanilla model), S +0 (state-injected object prompting) and O-S (object-injected state prompting). As shown in Table 5a compared to the None model, the S= exhibits decrease in all metrics, implying that state prompts may interfere with the selection of object prompts. On the contrary, O-S outperforms the None model as we expect. This phenomenon validates our motivation that state recognition harder than object recognition, and thereby the former cannot help the latter easily. Yet, it promising direction for future research.
Effect of generalized-men mean prompt fusion. This study aims to study the impact of prompt fusion on ComplLer. As shown in Table 5b) GeM performs better than both max and mean pooling across various metrics. It validates the benefit of GeM on mitigating irrelevant information in the selected prompts. as it may hamper the model's attention on image tokens.
Effect of loss functions. As shown in Table we investigate the influence of loss functions used in our model, including directional decoupled Toss (Linter and Lintra) and symmetric cross entropy loss (Log and LRCF). The baseline model (the first row) includes all modules but trained by cross entropy loss only. By adding the RCE loss, the model is equivalent to training with the SCE loss, which help to improve the robustness to noisy labels. The use of either Linter or Lintra improves the performance on both datasets, and synchronously applying them witnesses all-around improvements

---

Table 6: Ablate the loss functions on Split-Clothing and Split- Split-UT-Zappos.
Loss function
Split-Clothing (5 tasks)
SplitUUT-Zappos (5 tasks)
LOE
LRCE
Linter
Lintra
Avg Acc
FTT())
Avg Acc
FTT())
88.17=0.08
8.08+0.27
44.8340.15
19.49#2.93
88.36200377
8.33+0.11
45,47=0.07
20.1420.43
88.3240.56
7.82+0.64
45.58+0.04
19.6420.37
88.42+0.30
8.23+0.06
45.6240.13
20.13 14
88.61=0.61
7.72+0.87
46.01=0.69
19.50=0.50 86
89.21=0.24
7.26#0.60
46.48+0.26
19.27=0.75
Yallon Skin
Lesther Antle
T1
96.6
T1
99.83
Yellow Skin
Lextter Aokle
Yellow Diea
Stede Ankle
12
94.44
12
72-
Browa Hoodic
17.08
73-
60.32
96.25
Beown Hloodie
Stade
Ralll
T4-
V2.91
8595
83,00
92.24
14
H:.74
74- 96.06
Black Sait
Carvas Sacakers
15
92.50
00
24
10
Bluck Sait
Cuarvas Sacalers Sucde Slippes
12
14
15
Grees Suit
(a) Composition Accuracy
(b) State Accuracy
(c) Objoct Accuracy
(d) Prediction Results
Figure 6: Results and analysis. (a) to (c) show accuracy of Compll on composition, state, and object for each task in Split-Clothing The x-axis represents the test stream, and the y-axis denotes the status after training the Ti task. Darker background color indicates higher accuracy. (d) displays some images and their predictions: top row is GT, middle row is ComplLer prediction, and bottom row is L2P 433 prediction. Green indicates correct predictions, while red indicates incorrect predictions.
compared the baseline. Eventually, achieve the best results when combing all the loss terms during training.
5.6 Additional Results and Analysis
In order to study the repeatability characteristic in composition- we exhibit more results on Split-Clothing in Fig. [ in (a), it shows decreasing trend composition accuracy along with the introduction of new tasks; however, the green rectangles in (b) and (c) showcase that the accuracy occasionally increases as more tasks are learned. We conjecture the reason is mostly attributed to the re-occurrence of primitive concepts. This forward transfer is critical for incremental learners. We compare the composition predictions between ComplLer and L2P (43 in Fig. predicts all the images correctly, while L2P makes limitation arises from an excessive focus on the dominant object primitive, while weakening the attention toward state primitive. Fortunately, ComplLer relieves the bias toward object classes, and enhances the perception on state classes.
6 (d). ComplLer mistakes, particularly for state labels. This
Conclusion
In this paper, we have proposed a novel task coined compositional incremental learning (compostion- IL), which is stumbled by ambiguous composition boundary. To tackle it, we develop learning- to-prompt model, namely ComplLer. Our model exploits multi-pool prompt learning to model composition and primitive concepts, object-injected state prompting to improve the selection of state prompts, and generalized mean prompt fusion to eliminate irrelevant information. Extensive experiments on two tailored datasets show that ComplLer achieves state-of-the-ud performance. In the future, is challenging yet potential to consider reasoning multiple state classes per object.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant Numbers 62102061, 62272083 and 62472066, and part by the Open Projects Program of State Key Laboratory of Multimodal Artificial Intelligence Systems.
10

---

References
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2020.
[2] Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal-free continual learning. In IEEE Conf. Comput. Vis. Pattern Recog.. 2024.
[3] Shaozhe Hao, Kai Han, and Kwan- Yea K Wong. Learning attention disentangler for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 15315- 15324, 2023.
[4] Geoffrey Hinton. Some demonstrations of the effects of structural descriptions in mental imagery. Cognitive Science, 3(3):231-250, 1979.
[5] Chenchen Jing, Yukun Li, Hao Chen, and Chunhua Shen. Retrieval- aaggmented primitive representations for compositional zero-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2652-2660, 2024.
[6] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. Adv. Neural Inform. Process. Syst., 33:3647-3658, 2020.
[7] Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt- based continual learning. In Int. Conf. Comput. Vis., pages 11429-1144 11439, 2023.
[8] Hanjae Kim, Jiyoung Lee, Seongheon Park, and Kwanghoon Sohn. Hierarchical visual primitive experts for compositional zero-shot learning. In Int. Conf. Comput. Vis., pages 5675-5683 2023.
[9] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 1L<(13):3511-352, 2017.
et al.
[11] Hyunseo Koh, Dahyun Kim, Jung-Wor Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. International Conference on Learning Representations, 2022.
[12] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng. and Muli Yang. Siamese contrastive embedding network for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 9326-9335, 2022.
[13] Yong- Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-obeect compositions. In IEEE Conf. Comput. Vis. Pattern Recog., pages 11316-11321 2020.
[14] Yong-Lu Li, Yue Xu, Xinyu Xu, Xiaohan Mao, and Cewu Lu. Learning single/multi-atrrbutee of object with symmetry and group. IEEE Trans. Pattern Anal. Much. Intell., 44(12):943399055 2021.
[15] Yun Li, Zhe Liu, Hang Chen, and Lina Yao. Context-based and diversi specificity in compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., 2024.
[16] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2955-2947, 2017.
[17] Zilong Li, Yiming Lei, Chenglong Ma, Junping Zhang, and Hongming Shan. Prompt-in-prompt learning for universal image restoration. arXiv preprint 231.0223, 2023.
11

---

[18] Weiduo Liao, Ying Wei, Mingchen Jiang, Qingfu Zhang, and Hisao Ishibuchi. Does continual learning meet compositionality? new benchmarks and an evaluation framework. Adv. Neural Inform. Process. Syst., 2023.
[19] Yu Liu, Jianghao Li, Yanyi Zhang, Qi Jia, Weimin Wang, Nan Pu, and Nicu Sebe. Pmgnet: Disentanglement and entanglement benefit mutually for compositional zero-shot learning. Computer Vision and Image Understanding. 2024.
[20] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Adv. Neural Inform. Process. Syst., 30, 2017.
[21] Massimiliano Mancini, Muhammad Ferjad Naeem, Yonggin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5230, 2021.
[22] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Trans. Pattern Anal. Mach. Intell., 45(5):5513-5553 2022.
[23] Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1992-1801,2201 2017.
[24] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 11731-11741, 2023.
[25] Muhammad Ferjad Nacem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 953-962, 2021.
[26] Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute- object compositions. In Eur. Conf. Comput. Vis., pages 169-185, 2018
[27] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task- driven modular networks for zero-shot compositional learning. In Int. Conf. Comput. Vis., pages 3593-3602, 2019.
[28] Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, Yuan Xie, et al. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2023.
[29] Filip Radenović, Giorgos Tolias, and Ondfej Chum. Fine-tuning enn image retrieval with no human annotation. IEEE transactions on patlern analysis and machine intelligence, 41(7):1655- 1668, 2018
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, pages 8748-8763, 2021.
[31] Pasko Rakic, Jean-Pierre Bourgeois, and Patricia Goldman- Rakic. The self-organizing brain: from growth cones to functional networks. Elsevier, 1994.
[32] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In IEEE Conf. Compul. Vis. Pattern Recog.. pages 2001-2010, 2017.
[33] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogério Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention prompting for rehearsal-free continual learning. IEEE Conf. Comput. Vis. Pattern Recog., pages 11909-11919, 2023.
12

---

[34] Filip Szatkowski, Mateusz Pyla, Marcin Przewigžlikowski. Sebastian Cygert, Bartlomiej Twar- dowski, and Tomasz Trzciński. Adapt your teacher: Improving knowledge distillation for exemplar-free continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1977-1987, 2024.
[35] Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When prompt-based incremental learning does not meet strong pretraining. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1706-1716, 2023.
[36] Liyun Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Adv. Neural Inform. Process. Syst. 36, 2024.
[37] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Trans. Pattern Anal. Mach. Intell., 2024.
[38] Qingsheng Wang, Lingqiao Liu, Chenchen Jing, Hao Chen, Guoqiang Liang, Peng Wang, and Chunhua Shen. Learning conditional attributes for compositional zero-shot learning, In IEEE Conf. Comput. Vis. Pattern Recog,, pages 1199-11206, 2023.
[39] Wenjin Wang, Yunqing Hu, Qianglong Chen, and Yin Zhang. Task difficulty aware parameter allocation & regularization for lifelong learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 7776-7785, 2023.
[40] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's is razor for domain incremental learning, Adv. Neural Inform. Process. Syst., 35:5682-5695, 2022.
[41] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Int. Conf. Comput. Vis., pages 322-330, 2019.
[42] Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Dualprompt: Complementary prompting for rehearsal-fri -ree continual learning. In Eur. Conf. Comput. Vis., pages 63 631-648, 2022.
[43] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Learning prompt for continual learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 139-149, 2022.
[44] Aron Yu and Kristen Grauman. Fine- tieeggranned visual comparisons with local learning. In IEEE
Conf. Comput. Vis. Pattern Recog., pages 192-199, 2014.
[45] Alan L Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? Int. J. Comput. Vis., 129(3):7818802, 2021.
[46] Tian Zhang, Kongming Liang. Ruoyi Du, Xian Sun, Zhanyu Ma, and Jun Guo. Learning invariant visual representations for compositional zero-shot learning. In Eur. Conf. Comput. Vis., pages 339-355, 2022.
[47] Yanyi Zhang, Qi Jia, Xin Fan, Yu Liu, and Ran He. Csenet: Class-specified cascaded network for compositional zero-shot learning. In IEEE International Conference on Acoustics, Speech and Signal Processing. pages 3705-3709, 2024.
[48] Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Sean Lim, and Yunzhe Jia. A simple but strong baseline for online continual learning: Repeated augmented rehearsal. Adv. Neural Inform. Process. Syst. 35:14771 2022.
13

---

A Appendix
In addition to the content in the main paper. this appendix elaborates more details on the datasets, algorithm procedure, empirical analysis. nyper-parameter analysis, and quantitative and qualitative experiments. Meanwhile, please refer to the source code and some data samples included in the supplementary material we submit.
3000
5000
3000
2300
4030
2000
1000
(i Split-Clothing (5 tusks)
(ii Split-UT-Zappr (5 tasks)
(iii) Spli-UT--appon (10 tasks)
Figure 7: Number of images per task. The horizontal axis indicates the task ID, while the vertical axis represents the corresponding number of images. The colors pink and blue denote training and testing datasets, respectively. Overall, the data distributions are relatively balanced across tasks.
A.1 More on Dataset Details
We provide more information about two newly tailored datasets (Split-Clothing and Split- -a-apo)T- and extensively discuss the limitations of Splii-UT-Zappos. The number of images in each task is shown in Fig.
Split-Clothing is derived from Clothing 166, which is originally used for multi-label classification tasks and consists of 37 state-object compositions and 16,170 images scraped from Google, Bing. and DuckDuckGo. We sorted these compositions by the number of images and selected the top 35 compositions, resulting in a total of 15,915 images in the Split-Clothing dataset. Specifically, the object in Split-Clothing includes various types of garments such as "dresses" and "hoodies", while the state delineates the color of the clothing, such as "silver" and "yellow"". We allocate 80% of the dataset for training and the remaining 20% for testing. We split this dataset for 5 incremental tasks.
Sppppo a subset of UT-Zappps500K, which is large shoe dataset consisting of 50,025 catalog images collected from Zappos.com. This dataset is originally created for online shopping task and we selected a total of 28,497 images to comprise the Uplit-UT--Zppos.. The state classes in Split-UT-Zappe
describe different materials (e.g. canvas and leather) and the object classes related to footwear (e.g. heels and slippers). Consistent with images for training and 20% for testing. We split this dataset for either 5 or 10 incremental tasks.
we employ 80% of the
is noteworthy that, even for the upper bound model (i.e. supervised learning with all the data), satisfactory performance remains elusive on Sp-ia-UTs as shown in Table [ and Table 3 This deficiency can be attributed to the pronounced long-tail distributions and invisible states inherent in UT-Zappos50K. For example, the composition "Faux Fur Slippers" comprises mere 25 training images, whereas "Leather Sandals" accounts for 1783. This disproportionate distribution predisposes the severe bias towards the compositions prevalent in the head of the distributions, while inadequately capturing those in the tail. Besides, some states in Aplit-UT-Zappus lik.e "Leather" "Synthetic Leather", always visible as visual transformations. The deficient state description poses a greater challenge for classification, thereby resulting in suboptimal upper bound results. We contend that tackling the long-tail distribution challenge within Split-UT-Zappe and constructing more balanced datasets conducive composition-ll represent promising avenues for future research.
Limitations Split-UT-Zappos.
material differences that are not
A.2 More on Training Procedure
We summarize the training procedure of the proposed ComplLer in Algorithm I
14

---

Algorithm 1:
Training Procedure of ComplLer for composition-IL
Input: Training data for tasks, where each image has a of three labels: c, 8 and o. Three prompt pools P. with corresponding keys K.,, where E {8, c). Pre-trianed feature extractor f()). pre-trained input embedding layer fe(-), pre-trained transformer encoder layer
fr(-), cross attention layer CrossAtt[(:: .), classifiers (p(()) GeM fusion 2 Initialize: P... K.,, CrossAut[(;) :). (.). (........)
for t 1.... do for each sample in the batch do loss Cp Cinnne A2Cintrar Calculate query feature of object and composition go go(2) = (x(x) [
Estimate the prompt-specties
Lookup top-k object keys K, and composition keys Ka
Select top-k prompts associated with the keys in P. and Pe:
Fuse the selected prompts by P. GeM, P Pos.... Po and
Pe GeM. Pet ,es,.... ,..,, );
10
Perform object-injected state prompting by: (a(x) CrossAtn(fce)00:,] Lookup top-k state keys K.;
11
12
Select top-k state prompts associated with the keys in Ps;
13
Fuse the selected state prompts by P. = GeM, p P P."): Calculate the input embedding sequence Ie (z);
14
15
Prepending with fused prompts by Tp = P:: Po:
16
Feed Tp to transformer encoder layer fr()) and achieve P P and P. Calculate the probability by p(w x) = a (P"):
17
18
Estimate per sample loss L2 LSC + Sa end
19
20
Calculate per batch loss LB by accumulating (C, + Ep).
21
end
Output: The network and its network parameters tb.
A.3
Empirical Analysis on Learned Prompts
In Section 4.1 we design multi-pool prompt learning to learn visual representations state, object, and composition, respectively. By applying inter-pool discrepant loss and intra-pool diversified loss, we ensure discrepancy across pools and diversity within each pool. To validate the effectiveness of the approach, we conduct 1-SNE visualization on the prompt pools within the three experiment settings. Figure [ iillustrates that the learned knowledge across the three pools is remarkably discriminative, suggesting that each pool has effectively captured unique features. Mcanwhile, the prompts within each pool exhibit relatively wide dispersion demonstrating that the learned information within each pool is diversified.
A.4 Empirical Analysis of Generalized- Prompt Fusion
The detailed operation of Generalized-menn Prompt Fusion is elaborated on Section 4.3 The parameter 1) in Eq. [ can be learnable as this operation is differentiable, allowing it to be included in the whole propagarpnnn process. To prove that, the corresponding derivatives of Eq. 6 are given by:
dP.
1
P.
1 1-1 1-pp-11
0P
k
(11)
aP
P
>b 1==
Pe Ps.
log
On
n
1=
p
Ei=1
kk
P...1
A.5 Empirical Analysis of Symmetric Cross Entropy Loss
In this section, we delve into why symmetric cross entropy (SCE) loss can effectively mitigate noisy data in the dataset. We define two distributions, q and p, where q = (I: x)) represents the ground truth class distribution for sample and (k represents the predicted class distribution. The
15

---

(a) Split-llohing (5 tasks)
(b) Split-UT-Zappos tasks)
(c) Split-UT-Zappos (10 tasks)
Figure 8: The 1-SNE visualization of learned prompts on Split-Clothing (5 tasks), Split-UT-Zappos{5 tasks), and Split-UT-Zappos (10 tasks). The composition prompts are colored in yellow, the state prompts in green, and the object prompts blue.
47.2
47.4
47.4
Avg Acc (%)
Ace Ace (%)
Avg Acc (%)
46.8
46.4
45.8
45.8
45
45
1.2
3e-7
1e-
36
l06
30-5
0.9
(a) Impact of 21
(b) Impact of 22
(b) Impact of 2g
Figure 9: Impact of parammtersseterss on average accuracy in Split-UT-Zappos (5 tasks).
relation between the cross entropy (gg p) and KL-divergence KL(q p) is expressed as:
KL || p) H(g.p) H(4):
(12)
in which H (g) denotes the entropy of g. From the perspective of KL divergence, the optimization objective is to make the prediction distribution closely resemble the ground truth distribution essentially minimizing their KL. divergence. However, in the presence of noise within the dataset, may not accurately reflect the ground-truie class distribution, whereas might reveal the real distribution. Therefore, it becomes necessary to consider the reverse direction of KL divergence, i.e. K LL || v) and construct symmetric KL divergence represented by:
SKL(q p) = KL || p) + KL 4).
(13)
By transferring this idea from KI. divergence to cross entropy, the formula for symmetric cross entropy is given by:
SCE = CE + RCF = H(g. p) + H(p,)).
(14)
where RCE = H(p q) is the reverse counterpart of H H(g p), known as reverse cross entropy.
A.6 More on Hyper-paramedt Analysis
We investigate the impact of several hyper-parameters as depicted in Fig. Specifically, we explore the influence of weights in the total loss. We present the results of tuning the weight Å1. A2, and A on the Split-U'T-Zappos (5 tasks). The average accuracy shown in Fig. a)), (b), and (c) initially exhibits an increasing trend, followed by overall decreasing trend. Complleer reaches its peak performance when Z1 1.0, X2 3e and A 0.7.
A.7 More on Ablation Studies
Effect of multi-pool prompt learning. In addition to presenting the results of Avg ACC, FTT, and HM on Split-Clothing (5 tasks) for multi-pool prompt learning, we also include remaining results for State and Object in Table 7 An intriguing observation is that the model achieves the best
16

---

Table Ablate the multi-pool prompt learning on Split-Clothing (5 tasks). C. S, and O denote the composition pool, state pool, and object pool, respectively.
Prompt Pool
Split- Clothing (5 tasks)
S 0
Avg Acc
FTT())
State
Object
HM
80.01
12.85
83.25
94.45
88.50
88. 08.10+0.11
7.79+0.04
90.42+0.02
96.91+0.10
93.55+0.04
88.09+0.50
7.26+0.54
91.67=0019
96.00=0.03
93.5240.15 13
88.38+0.08
8.01+0.42
91.15+0.10
96.32+0.02
93.66#0.02
Table 8: Ablate the guidance on Split-Clothing tasks).
Dataset
(5 tasks)
Guidance
Avg Acc
FTT())
State
Object
HM
None
88.45=0.10
7.93=0.11
91.26#0.11
96.2840.01
93.7021003
S-O
88.27+0.02
7.99=0.05
90.99=0.22
96.5340.19
93.67=0.01
0
89.21=0.24
7.26+0.60
91.81=0.23
96.6740.01
94.18+0.06
results for Object when the object pool is used exclusively, while it reaches the peak performance for State when only the state pool introduced. This phenomenon suggests that incorporating the primitive prompt pools significantly aids in learning the representations of the corresponding primitives. Furthermore, when the model integrates the state prompt pool alone, not only does the accuracy of state is improved, but the accuracy of object also experiences significant gains, and vice versa. This implies that the learning of one type of primitive representation is crucial for the learning of another. Only when all three pools are integrated does the model achieve state-of-the-art the-art performance in the most important metrics, namely average accuracy and HM.
Effect of object-injected state prompting. In this section, report the remaining results for object-injected state prompting on Split-Clothing (5 tasks) as shown in Table g State-injected object prompting (S-+) introduces interference information for composition learning. resulting in decline across all metrics expect Object. By contrast, object-injected state prompting (0+S) leads to a significant improvement across all metrics, demonstrating the effectiveness of the proposed method.
Effect of generalized-meal prompt fusion. We show the intact results of the ablation study on pooling methods on result, while integrating mean pooling leads to stable growth. Only deploying Generalized-men pooling achieves the state-of-the-ee results, surging from 84.70 to 89.21 in Avg Acc and plunging to 7.26 in FTT. Consequently, the theory that GeM pooling can diminish the irrelevant information and emphasize the crucial aspects is proved.
(5 tasks) in Fig. [ It is obvious that max pooling hits the lowest
Effect loss functions. Eventually, we present supplementary ablation experiments on loss functions in Table 10 and IT The first row indicates that the model is optimized using cross entropy loss exclusively which achieves the poorest results. Furthermore, integrating either reverse cross entropy loss, inter-pool prompt discrepancy loss, or intra-pool diversity loss contributes to improvements across all metrics. Notably, the effectiveness of reverse cross entropy loss is higher on Split-UT-- Zappos (5 tasks) compared to that on Split-Clothing (5 tasks). To conclude, the best performance is achieved when all loss functions are simultaneously employed.
A.8 More on Qualitative Results
More qualitative results are presented to demonstrate the effectiveness of ComplLer. We conducted a comparative analysis of the prediction results between ComplLer and L2P (4 on Split-Clothing (5 tasks), Split-oTT-Zapps (5 tasks) and Split-UT-Zaapps (10 tasks). The ground truth is presented in the first row, followed by the prediction results of ComplLer and L2P the second and third rows, respectively. As illustrated in the Fig. I0 ComplLer achieves correct predictions for all the images, while 12P consistently yields wrong classification results due to ambiguous composition boundary. Specifically, L2P exhibits a higher occurrence of misclassification which can be attributed to the disproportionate emphasis placed on objects, diminishing the model's focus on states. This experiment proves that ComplLer effectively enhances the model's attention towards states, resulting in an improved overall fine- fine-grained perception capability.
states compared to objects,
17

---

Table 9: Ablate the pooling Split-Clohhing (5 tasks).
Dataset
Split-Clothing (5 tasks)
Pooling
Avg Acc
FTT())
State
Object
HM
Max
84.70#0.64
12.24#2225
86. 79.99
968440.01
91.54=0.30
Mean
87.80#0.12 12
7.82+0.01
90.78+0.12
16.14+0.01
93.38=0.03
GeM
89.21#0.24
7.26+0.60
91.81+0.23
96.67=0.01
94.18=0.06
Table 10: Ablate the loss function on Split-Clothing (5 tasks).
Loss function
Split-Clothing (5 tasks)
LoE
LRCE
Linter
Lentra
Avg Acc
FTT())
State
Object
HM
88.17+0.08
8.08=0.27
90.9910.21
96,41+0,08
93.62+0.03
88. 88.36+0.37
8.33=0.11
90.88+0.30
96.64+0.05
93.67+0.06
88.32+0.56
7.82+0.64
90.85+0.57
96.61+0.07
93.66+0.12
88 88.42+0.30
8.23#0.06
91.18+0.04
96.44+0.10
93.73+0.06
88 88.61
7.72=0.87
90.94+0.68
96.85+0.02
93.81
V
/
89.21=0.24
7.26+0.60
91.81+0.23
96.67+0.01
94.18+0.06
Table 11: Ablate the loss function on Split- Split-UT-Zappos (5 tasks).
I
Loss function
Split-UT-Zappos (5 tasks)
Loe
LRCE
Linter
Lintra
Avg Ace
FTT())
State
Object
HM
44.83+0.15
19.4912.93
55.07=0.25
79,0620.06
64.9220.18
45.47+0.07
20.1440.44
55.92+0.05
79.1420.13
65.47=0.03
45.58+0.04
19.64+0.37
56.02+0.04
79.25=0.01
65.64=0.01
4
45.6240.13
20.13+0.14
55.98+0.24
79.45=0.20
65.6820.08
46.01+0.69
19.50+0.86
56.31+0.72
79.53=0.05
65.9420.40
06.48+0.26
19.27+0.75
56.85+0.34
79.56*0.04
66.31=0.15
Green Shoes
Blue Shocs
Wool Slippers
Faux Leather Sandals
Green Shoes
Blue Shoes
Wool Slippers Suede Slippers
Faux Leather Sandals Full-grain Sandals
Brown Shoes
White Shocs
Pink Skirt
Silver Skirt
Leather Flats
Canvas Sneakers
Pink Skirt
Silver Skirt
Leather Flats
Canvas Sneakers Synthctic Sneakers
Yellow Dress
Pink Skirt
Canvas Boat
Yellow Skirt
Blue Shirt
Rubber Sandals
Leather Ankle
Yellow Skirt
Blue Shirt
Rubber Sandals
Leather Ankle Faux Leather Ankle
Yellow Dress
Blue Shorts
Synthetic Sandals
Black Shoes
Black
Leather Heels
Leather Oxfords
Black Shoes
Black
Leather Hoels
Leather Oxfords Full-grain Oxfords
White Shoes
Green Suit
Satin Heels
(a) Split-Cog (S tasks)
(b) Split-UT-Zappos (5 tasks)
Satin Sandals
Rubber Sandals
Full-grain Leather Loafers
Satin Sandals Synthetic Sandals
Rubber Sandals Faux Leather Sandals
Full-grain Leather Loafers Leather Loafers
Suede Ankle
Suede Hecls
Synthetic Mid-Calf Boots
Suede Ankle
Suode Hocls
Synthetic Mid-Calf Boots Nylon Mid-Calf Boots
Leather Ankle
Patent Leather Heels
(c) Split-UT-Zappos (10 tasks)
Figure 10: More qualitative results. For each sample. top row is ground-truth label (in black), middle row is ComplLer prediction, and bottom row is L2P 43 prediction. The primitives in green and red refer correct and incorrect predictions.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction effectively summarize the paper's contributions and scope, outlining the key claims and objectives.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in both the paper and the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We provide the full set of assumptions and complete proof for each theoretical
result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive implementation details and code to ensure the experiments are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides open access the data and code, along with sufficient instructions to faithfully reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
coblic//uuide//Coeeetuiiiss wtw.oeeeSuubiissiooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttp:/cccppplclguuaieeaeaeccuuea
Saaeee//CoSSSubisssS.S.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details, including but not limited to: data splits, hyperparameters, how they were chosen, type of optimizer, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper appropriately reports the relevant information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resourees in implementation details section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https: http::////eaaessscc/pp//c//cchct
Answer: [Yes]
Justification:
The research conducted in the paper aligns with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited the creators and original owners of assets used our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the paper are well documented and the documen- tation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "x48IMSe78H", "reviewer": "Reviewer_x9g2", "summary": "The paper propose compositional Incremental Learning to enable models to recognize state-object compositions incrementally. The paper provides two tailored datasets for composition-IL by modifying two existing datasets in the fashion domain. The paper propose a new prompt-based model comprising of multi-pool prompt learning, object-injected state prompting and generalized-mean prompt fusion. The method achieves competitive performance on the two proposed datasets.", "questions": "1. I am curious what is the difference between object-state pairs and a group of concepts describing a class? Like why limit the concepts to just object and state, there can be more concepts attached to a class. So, a class can also be described as a group of concepts and classes can then be learned incrementally with overlapping concepts from old classes.", "limitations": "Limitations of this work is not explicitly addressed in paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper discusses a new direction which aims to exploit state primitives of objects for incremental classification. The authors propose a new incremental setting where objects or states can reappear in new classes in new tasks.\n2. The paper provides two curated datasets in the fashion domain to study composition-IL.\n3. The paper propose a new prompt-based model for composition-IL. It is nice to report HM accuracy for better evaluation.", "weaknesses": "I have major concerns with the experimental part of the paper.\n1. Hyperparameters: It is very concerning that six hyperparameters are tuned on the testing set. The hyperparameter values even differ for different task splits on the same dataset. It is weird that the authors use 25 epochs for one dataset, while for the other dataset, 10 epochs for 5-task setting while 3 epochs for the 10 task setting. Even different learning rates are used for different settings on same datasets. How are these decided? It looks like everything is optimized for the test sets in all settings. This is not a fair way of doing experiments. It is acceptable if the authors fine-tune the model for one dataset and use the same parameters for all settings across different datasets (I think this is commonly done in continual learning domain). How can the proposed method be useful/practical if it needs to fine-tune so many hyper parameters on every setting using the test set to get good results?\n2. Lack of experiments with random seeds: The experiments are conducted using a single random seed. The proposed method has improvements of 1% or even less in some settings. It is standard practice in CL to report results with multiple random seeds for fair evaluation and to establish the robustness of the model.\n3. Competitive recent baselines like HiDe-prompt [1] are not included in the comparison.\n4. Simple methods like NCM classifier [2] and Mahalanobis-distance based classifier [3] outperforms prompt methods like L2P on several datasets with first-task adaptation and no training in new tasks (using frozen model after the first task and doing continual evaluation). It would be interesting to see how these strong baseline methods work in the proposed settings.\n\n[1] Wang, Liyuan, et al. \"Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[2] Paul Janson,  et al, A simple baseline that questions the use of pretrained models in continual learning. arXiv preprint arXiv:2210.04428, 2022. \n\n[3] Dipam Goswami, et al, Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: About hyperparameter tuning\n\nA1: Thanks for your question. Firstly, we claim that finding the optimal balance between hyperparameters is important yet challenging for continual learning. Since the feature distributions within each incremental session may vary considerably, the optimal hyperparameters for each session are often different. Fortunately, our work focuses on using a set of hyperparameters across all sessions, rather than tuning different hyperparameters for each individual session.\n\nMoreover, we state that due to the complexity of the dataset and the differences in feature distributions, different hyperparameters are often assigned to different datasets. We find this phenomenon is quite common in prompt-based continual learning. For example, Dual-Prompt trains CIFAR-100 with a learning rate of 0.03 for 5 training epochs, whereas it trains ImageNet-R with a learning rate of 0.005 for 50 epochs. Likewise, LGCL's class-level and task-level objective weights on CIFAR-100 are 0.137 and 0.323, respectively, accurate to even three decimal places.\n\nFurthermore, for the same dataset, different settings are often assigned with different hyperparameters to ensure the optimal performance. It is a common strategy in compositional learning. For example, in the milestone work Co-CGE [1] in Compositional Zero Shot Learning (CZSL), the method trains for 300 epochs in a closed-world scenario on the CGQA dataset but only for 200 epochs in an open-world scenario. We emphasize that in CZSL, the training data for both open-world and closed-world scenarios is exactly the same, which is essentially no different from our case of hyperparameter setting. In compositional filed, the reason that different hyperparameters can be assigned for the same dataset under different settings is due to the complex interplay of the feature space resulting from compositionality; without appropriate hyperparameters, the model’s potential may be severely limited. Our empirical choice of different hyperparameters is based on: for Split-Clothing, with its simpler primitive description, a larger number of epochs should be chosen to ensure learning a more accurate representation, whereas due to the visual invisibility of the state in Split-UT, excessively large epochs may lead to overfitting on some irrelevant information. For example, the model might mistakenly associate “Suede” with “black” because many “Suede shoes” in the dataset appear in black.\n\nFinally, the core contributions and innovations of our work lie in the novel task setting, two corresponding datasets, and the method for solving ambiguous composition boundaries. We acknowledge that setting the same parameters across all datasets is an optimal setup as it demonstrates a model's generalizability, but this is not the central problem our work addresses. However, we will also consider the reviewers' feedback and explore a more robust model in future work, which will be discussed in the next version.\n\nQ2: Lack of experiments with random seeds\n\nA2: We greatly appreciate the reviewer's reminder. We conducted experiments with 4 different random seeds as shown in Tab 1 of the PDF. Although there are slight fluctuations in the results, it consistently achieves SOTA performance across all metrics. Although single seed experiments are acceptable in recent continual learning works [2][3], we will use multiple random seeds for a more thorough comparison.\n\nAdditionally, we clarify that CompILer does not use any extra semantic knowledge, does not store any information of old session, does not allocate any task-specific parameters, and does not dynamically increase network parameters. We believe that these techniques (which are commonly used by prompt-based methods except L2P) undoubtedly contribute significantly to performance improvements, but they are contrary to the principles of continual learning, as noted in line 263 of our manuscript.\n\nQ3: Lack of methods\n\nA3: We have conducted comparisons with the methods you suggested, and the results are shown in Tab 2 of the PDF. Due to limited rebuttal period, we mainly compare Compiler with the methods on Split-UT-Zappos. HiDe-Prompt surpasses CompILer due to the task-specific prompt settings and the storage of statistics of old classes which seem to contradict the principles of continual learning to some extent.\nFor NCM and Fecam, unfortunately, although this first-task adaptation approaches achieve a lower forgetting rate, their plasticity on incremental tasks is quite poor because they only work well when the first task contains the most classes. We will include these methods as baselines in the next version and provide proper citations.\n\nQ4: The relationship with a group of concepts.\n\nA4: Some datasets associate classes with a group of concepts. For example, in the CUB dataset, a number of attribute concepts are provided for classes, such as the color of the leg/back, the pattern of the head/breast, etc. However, these attribute information is locally semantic, and cannot encapsulate the global state features of a class. As a result, it limits the attribute information forming a pair with the object. In contrast, the composition-IL task combines objects with global states, so that objects and states are considered on an equal status and exhibit compositionality. Notably, the original datasets Clothing16K and UT-Zappos50K only provide a single state description for each class which prevents Split-Clothing and Split-UT from forming an object with a group of global concepts. Nevertheless, we also acknowledge your advice and it is potential to explore a new benchmark with multiple global states further.\n\n[1] Learning Graph Embeddings for Open World Compositional Zero-shot Learning (TPAMI 2022)\n\n[2] When Prompt-based Incremental Learning Does Not Meet Strong Pretraining (ICCV 2023)\n\n[3] Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning (NeurIPS 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AEA6N7S71V", "reviewer": "Reviewer_B271", "summary": "This paper introduces a novel task termed Compositional Incremental Learning (composition-IL), which aims to enable models to recognize a variety of state-object compositions incrementally. The authors propose a new model called CompILer, which employs multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to overcome the challenges in this task. The study utilizes two newly tailored datasets, Split-Clothing and Split-UT-Zappos, and demonstrates state-of-the-art performance through extensive experiments.", "questions": "Model Efficiency: How does the size of the learnt prompts in CompILer compare to current prompt-based methods? Please provide a detailed analysis of model efficiency.\n\nHyperparameter Analysis: Can you provide a systematic analysis of the hyperparameters $\\lambda_1$ and $\\lambda_2$ in Equation 9? How do variations in these values impact the model's performance?\n\nPerformance Comparison: Why do LGCL and CODA-Prompt perform well on the Split-Clothing and Split-UT-Zappos tasks, and what factors limit the performance improvements of CompILer over these methods? A detailed comparative analysis would be beneficial.", "limitations": "The authors have acknowledged the limitations of their work, particularly concerning the performance challenges on the Split-UT-Zappos dataset due to long-tail distributions and the difficulty in distinguishing state classes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of composition-IL addresses a critical gap in incremental learning by focusing on the recognition of state-object compositions, which traditional class-IL and blur-IL approaches overlook\n2.  The proposed CompILer model is well-conceived, leveraging multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to enhance the learning of compositions.", "weaknesses": "1. The paper shows that CompILer only slightly outperforms LGCL and CODA-Prompt on the Split-Clothing and Split-UT-Zappos datasets. However, the authors do not detailedly analyze why LGCL and CODA-Prompt perform well in these tasks or why CompILer's improvements are limited. A detailed comparative analysis is needed.\n\n2. Lack of analysis on model efficiency, especially on the size of learnt prompts comparing to current prompt-based methods\n\n3. The paper lacks a systematic analysis of the hyperparameters $\\lambda_1$ and $\\lambda_2$ in Equation 9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: About performance comparison.\n\nA1: Thanks for your question. We are glad to answer it. First, we would like to emphasize that CompILer has achieved SOTA performance across all settings. Notably, on the Split-Clothing dataset, CompILer beats LGCL and CODA-Prompt with an improved average accuracy of 1.55% and 1.63%, respectively. The reason why the performance improvement might not meet the reviewer's expectations is that both LGCL and CODA-Prompt rely on parameter expansion training paradigms. As what we have mentioned in line 263 of our paper, the parameters of these two methods increase significantly when more new tasks are incoming. Although introducing task-specific parameters undoubtedly boosts the performance, this approach contradicts the core principle of continual learning, where the network parameters should not increase unlimitedly as the number of tasks grows. Additionally, as noted in line 255 of our paper, LGCL introduces external semantic priors to achieve language guidance. Although this trick can further enhance the performance, it also limits the model's applicability. For instance, LGCL fails to operate on the 5-task Split-UT-Zappos since the total length of class names exceeds the limitation. In contrast, CompILer does not employ any parameter-increasing techniques or additional semantic knowledge. Compared to our baseline L2P, which does not use parameter expansion, CompILer shows a significant improvement, with 8.73%, 4.91%, and 3.13% across the three settings. We will shed more light on the comparison in the next version. We will emphasize this point again in the camera-ready version and expand on this section.\n\nQ2: About model efficiency.\n\nA2: We have followed your suggestion and further compared the size of prompts used in CompILer and other prompt-based methods. As shown in Tab. 3 of the attached PDF, the size of the prompt pool in CompILer is smaller than that in CODA-Prompt but is larger than that in Dual-Prompt and LGCL. As anticipated, CompILer demonstrates higher model efficiency for a longer sequence of tasks, because we will not dynamically increase the model parameters when we have pre-define the size of prompt pool. Consequently, in such scenarios, we can conjecture that CompILer’s memory footprint and inference speed are more efficient compared to Dual-Prompt and CODA-Prompt.\n\nQ3: Lack of hyperparameter analysis\n\nA3: We have added more experiments regarding the parameters $\\lambda_1$ and $\\lambda_2$, as shown in Fig. 1 of the attached PDF. The model's average accuracy shows an initial increase followed by a decrease within the interval, reaching a peak at $\\lambda_1=1.0$ and $\\lambda_2=3e-5$. Therefore, we consider the model to be at a local optimum in this setting and choose these parameters as the final setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhmHZba41A", "reviewer": "Reviewer_KbBF", "summary": "This work presents a new task called Compositional Incremental Learning (composition-IL).\nThis new task extends the existing class incremental learning to a more fine-grained scenario for more realistic applications.\nThis work formulates and designs a new composition-IL benchmark based on Clothing16K and UT-Zappos50K datasets.\nTechnically, the authors propose several novel strategies based on prompting on pretrained ViT.\nSpecifically, this work suggests using three prompt pools for states, objects, and their compositions respectively.\nTo overcome the gap between the pretrained task and the composition task, the object prompts are further injected to guide the selection of state prompts.\nThe authors also suggest a learnable generalized-mean prompt fusion scheme for prompt reducing.\nExtensive experiments are provided to evaluate the models.", "questions": "Please see the 'Weakness' section for detailed questions.\nIn conclusion, I expect the authors to provide their thoughts about the necessity of the proposed composition-IL setting, as well as some explanations of the small performance gap when equipped with the proposed modules.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper has done a great job in presentation, with clear wiring and figuring, enabling the fast understanding of reviewers familiar with continual learning.\n2. The contribution of proposing novel and more realistic tasks is always welcomed. And this work has done some essential initial jobs such as creating a benchmark for composition-IL.\n3. The proposed method looks convincing to me.", "weaknesses": "1. The main concern for me is the proposed composition-IL itself. From the experimental results (Tables 1 and 2), it seems that the previous prompt-based works can also handle this task with comparable performance. So my double is that if this work overclaims the difficulty of composition-IL? As someone working in the continual learning field, I think new settings and tasks are always welcomed only if the new task is challenging and practical enough.\n2. The effectiveness of the proposed components. As shown in Table 3, I think the performance gap between C+S+O and C+S/C+O is small, so the effectiveness of multi-pool prompt learning is not that significant. Similar results are also in Table 4. In Table 4(a), there is only a 0.32% improvement in HM when introducing the object injection. In Table 4(b), there is also only a 0.3% improvement when using the GeM instead of the naive mean pooling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: About the composition-IL\n\nA1: Thank you for your great comment. Firstly, we would like to emphasize that previous prompt-based works have achieved satisfactory results by employing strategies that contradict the paradigm of continual learning. The compared prompt-based methods all involve task-specific parameters except L2P, which means that their network parameters increase gradually with the introduction of more incremental tasks. This approach effectively enhances the performance but goes against the fundamental principles of continual learning, which aims for a fixed model to incrementally learn new tasks while overcoming forgetting of old tasks. The training paradigm of increasing parameters clearly contradicts this requirement. These competitive results achieved by network expansion should not decrease the awareness of significance of composition-IL.\n\nInstead of network expansion, L2P shows a large gap compared to the Upper Bound, as evidenced in Table 1 of the paper, demonstrating the significant necessity for further research. L2P is the only method in previous work that does not employ the aforementioned tricks. It shows a considerable performance collapse across three settings compared to One-batch learning, failing to even reach 50% of the Upper Bound. This significant degradation underscores the challenges faced by composition-IL due to ambiguous composition boundaries, highlighting the continued necessity for research. Hence, exploring methods that strictly adhere to the continual learning paradigm and achieve satisfactory results remains highly demanded and challenging.\n\nFurthermore, we underscore the research significance of composition-IL from a theoretical perspective. The hallmark of the human cognitive system is compositionality, allowing people to decompose and recombine knowledge to better understand and facilitate learning of new knowledge. Therefore, much scholarly attention has focused on how to equipe the model with compositionality. Unfortunately, there is little focus on studying compositionality in incremental learning scenarios. We recognize that in many real-world scenarios, decisions often rely on state-object pairs. Therefore, we propose Composition-IL to address the gap in modeling states within incremental learning. For instance, in autonomous driving systems, decisions often hinge not just on detecting pedestrians but also on understanding their state to determine the vehicle's next actions. Hence, this task is highly practical and worthy of pursuit.\n\nQ2: The effectiveness of the proposed components.\n\nA2: About C+S+O: The purpose of the multi-prompt pool can be seen as a decoupling mechanism. Compared to a single prompt pool, the three-prompt pool exhibits a significant improvement, increasing performance by 8.73%. Introducing an additional pool (decoupling a single primitive), can lead to substantial performance gains. This is because arbitrary primitive disentanglement helps to obtain clean, independent primitive features while implicitly decoupling another one and aiding compositional learning. For example, when using two prompt pools to separately learn composition and state, the state prompt and composition prompt are concatenated together with feature sequence and processed through MHA. During this process, the tokens used for composition classification interact with the state prompt, allowing the composition to understand what clean state information looks like, which aids the model's learning of state in the composition branch. Simultaneously, the model recognizes information irrelevant to the state prompt as object information. As a result, during classification, the classifier performs better at distinguishing which information is useful for state classification and which is for object classification, achieving a 1+1>2 effect. Therefore, using two prompt pools yields impressive experimental results. However, this does not imply that two pools are necessarily optimal. Only by explicitly incorporating two prompt pools for state and object can they complement each other better to achieve optimal results, as shown in manuscript Table 3.\n\nAbout Object-injected State Prompting: The expected improvement from Object-injected State Prompting is not as apparent due to limitations in the prompt selection mechanism. Like L2P, prompt selection follows the key-value mechanism. A drawback of this paradigm is that it cannot be optimized in an end-to-end fashion because it uses keys and queries to select a prompt index from the pool. This reliance on a second, localized optimization to learn the keys means that the model gradient cannot backpropagate through the key/query index selection, leading to only minor improvements in the key. Unfortunately, injected state prompting also suffers from this limitation, as it is subject to separate optimization without model gradient improvements. This is a common issue in prompt-based continual learning and is worth exploring further. We plan to investigate alternative methods in future work.\n\nAbout Generalized Mean Pooling: GeM can be seen as a balance between Max Pooling and Mean Pooling. We have analyzed it from the derivatives view in our appendix. We leverage GeM to filter out irrelevant information from the selected prompts. As a result, the fewer prompts selected, the higher relevance between these prompts and the images. Therefore, the effectiveness of GeM is linearly related to the amount of information to be integrated. The more information that needs to be fused, the more pronounced the effect of GeM Pooling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONDWqTlGLN", "reviewer": "Reviewer_EVGX", "summary": "This paper presents a novel setting of incremental learning, named Compositional Incremental Learning (composition-IL). This setting differs from existing ones as it involves recognizing not only new objects (e.g., a shirt), but also their states (e.g., red) and the resulting compositions (e.g., a red shirt). To address this novel setting, the authors present a prompting-based approach that leverages a pre-trained transformer network to learn the tasks incrementally, thus sharing some similarities with existing methods such as L2P, Dual-Prompt, and CODA-Prompt. Additionally, the authors devise a three-way retrieval mechanism, devoting tailored prompt pools to state, object, and composition-level representation learning. In the experimental section, the authors assess several aspects: the comparison with the state of the art, the benefits of the three-way prompting strategy, the impact of injecting object-level information into the state-level prompt, and the contributions of each involved loss function.", "questions": "I have no questions.", "limitations": "The limitations are briefly discussed in the supplementary materials.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is clear and fluent. Both the dataset and the model are clearly described.\n2) Novelty. To the best of my knowledge, the experimental setting is novel and deserving. \n3) While existing prompting strategies often organize the prompt pool in a flat structure, this paper advances the field by leveraging on two concepts: multiple prompts pool related to different concepts/tasks, and possible dependencies between prompts of different concepts.", "weaknesses": "**Many hyperparameters/tuning complexity (major)**. From a methodological perspective, the model incorporates several loss terms, each requiring a tailored balancing hyperparameter. Additionally, other factors such as the length of each prompt and the size of the prompt pool must be considered. In truth, the authors report the chosen configuration in their experiments in the second paragraph of Sec. 5.2. As can be observed, there are many hyperparameters, which could complicate the model’s application in real-life scenarios. In this respect, the authors should provide a simplified (yet still reliable) version of their approach, pruning some loss terms and configurations that have a negligible impact on results. Indeed, by examining the results of the ablation studies, it appears that there is room for simplifying the training stage.\nStatistical significance (major). Concerning Tables 3 and 4, the results achieved by the final configuration of the Compiler are sometimes very close to the simpler, ablated versions. For example, consider Table 4(a) where 'no injection' is close to 'O->S', and Table 3 where 'C+S' is close to 'C+S+O'. In these cases, it is difficult to conclude whether the benefits are due to effective technical advances or should be ascribed to statistical noise. In this respect, I strongly encourage the authors to perform multiple runs for each experiment and average the final results. This approach would render the comparisons more significant.\n\n**More rehearsal baseline (major)**. I would suggest including in the experimental section the results for DER++ [b] and ER-ACE [c], two straightforward baselines based on rehearsal. As the main contribution of this paper concerns the introduction of a new experimental setting, I believe it is important to provide an extensive evaluation of the existing literature. Moreover, the authors should compare their approach with SLCA [d], a recently introduced baseline for the continual fine-tuning of pre-trained models.\nRelation with existing works (major). There is a recent research line on multi-label incremental learning; see [a], one of the most cited papers in the field. In this respect, how does the proposed setting deviate from the multi-label scenario? To justify the introduction of a novel setting, the authors should discuss potential overlaps with existing benchmarks more thoroughly. From a more methodological perspective, instead, I would encourage the authors to provide more references while introducing the learning objectives used in their approach. For example, CODA-Prompt devises orthogonality constraints between prompts, sharing strong similarities with the objectives outlined by Eq. 2 and 3. Moreover, aligning the query and the selected prompts is common in prompting-based approaches (see L2P). \n\n**Lacking ablation studies (minor)**. In Tab. 3, it would be important to see the results of single-pool prompt learning, especially for ‘S’ and ‘O’ (‘C’ has instead been already provided). Indeed, I would guess that a simple strategy based only on object-level prompts would be enough to reach satisfactory performance. \n\n**Clarification requested (minor)**. In Sec. 4.2, the authors discuss an attention-based mechanism to inject object-level cues into the state-level prompt. In this respect, they use three learnable projections, indicated by W_Q, W_K, and W_V. However, as these transformations are continuously updated while tasks progress, did the authors consider that these projections could suffer from catastrophic forgetting? It seems that there are no explicit countermeasures against forgetting regarding these variables, which could hinder the efficacy of the retrieval stage.\n\n**Formatting issue (minor)**. Table 4 appears before Table 3.\n\n[a] Kim, C. D., Jeong, J., & Kim, G. (2020). Imbalanced continual learning with partitioning reservoir sampling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16 (pp. 411-428). Springer International Publishing. \n\n[b] Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020). Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33, 15920-15930.\n\n[c] Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., & Belilovsky, E. (2021). New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025.\n\n[d] Zhang, G., Wang, L., Kang, G., Chen, L., & Wei, Y. (2023). Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19148-19158).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Many hyperparameters/tuning complexity\n\nA1: Thank you for your constructive suggestions. We consider that assigning weights to different loss terms is a common practice. We highly agree with your suggestion to provide a simplified version of the model. As you suggested, the 'C+S' and 'C+O' in Tab 3 can be viewed as simplified CompILer which are denoted as CompILer-S and CompILer-O for clarity. The whole results are shown in the manuscript appendix Tab 6. Note that the all results are computed by composition pool for fair. The methods significantly lag behind the full CompILer on Split-UT-Zappos because when state features have higher semantics or are visually less apparent, the impact of ambiguous composition boundaries becomes more pronounced. Thus, primitive scores are particularly crucial in such cases. We will continue to explore more simplified models to reduce the number of hyperparameters and present the results in the next version.\n\nQ2: Statistical significance\n\nA2: We conducted the multiple runs with different random seeds as shown in PDF Tab 1. CompILer achieves SOTA results across different random seeds, indicating that the performance improvement is not due to statistical noise. We will extend this setting to all methods and experiments in the next version.\n\nAdditionally, we want to elaborate on why 'C+S' is close to 'C+S+O'. The purpose of the multi-prompt pool is to be viewed as a decoupling mechanism. The state disentanglement helps obtain clean state features while implicitly decoupling object and aiding compositional learning. Specifically, the tokens used for composition classification interact with the state prompt during MHA operation, allowing the composition to understand what clean state information looks like, which aids the model's learning of state in the composition branch. Meanwhile, the model realizes information irrelevant to the state prompt is object information. Thus, the classifier becomes better at distinguishing which information is useful for state classification and which is for object classification during classification, achieving a 1+1>2 effect. Thus, using two prompt pools yields impressive experimental results. However, this does not imply that C+S are optimal. Only by explicitly incorporating two prompt pools for state and object can they complement each other better to achieve optimal results.\n\nMoreover, the reason 'no injection' is close to 'O->S' is due to the limitations in prompt selection. Prompt selection follows the key-value mechanism, but one drawback of this paradigm is that it cannot be optimized end-to-end. The key and query are used to select a prompt index from the pool, relying on a second, localized optimization to learn the keys because the model gradient cannot backpropagate through the key/query index selection. Unfortunately, object-injected state prompting also suffers from this separate optimization issue, as it is constrained by the same drawback. This is a common problem in prompt-based continual learning. We plan to explore alternative methods in future work.\n\nQ3: More rehearsal baseline\n\nA3: We have included these baselines in Tab 4 of the PDF, where we demonstrate their performance alongside CompILer. For each method, we allocated a memory buffer size of 800, meaning that each class has 10 samples stored on average. As the results show, CompILer achieves the best performance without any rehearsal.\n\nQ4: Relation with existing works\n\nA4: We recognize that there are significant differences between multi-label IL and composition-IL. In a multi-label scenario, ground truth labels correspond to multiple independent entities within an image, where these labels are independent of each other and lack semantic interactions or fine-grained information. In contrast, the labels in composition-IL consist of a single object combined with its descriptive state. Therefore, composition-IL labels describe a single entity in a more fine-grained fashion rather than multiple entities. To better illustrate the difference, we present a figure in Fig 2 of the PDF. Additionally, we will include references to CODA-Prompt and L2P in the methods section of the next version.\n\nQ5: Lacking ablation studies\n\nA5: We have taken your suggestion, and the results are shown in Tab 5 of the PDF. The first and second rows report the results for state and object classification. An interesting finding is that the accuracy of single primitive classification does not even reach that of using a single pool for composition classification. This is because the experimental setup transitions from the traditional Class Incremental Scenario to a Stochastic Incremental Blurry Task Boundary Scenario (Si-blurry) [1]. Si-blurry faces continuous change of classes between batches leads to intra- and inter-task forgettings, making it difficult for the model to retain previously learned knowledge.\n\nQ6: Clarification requested\n\nA6: The object-injected state prompting module primarily focuses on selecting state prompts to enhance network plasticity and achieve Complementary Learning Systems (CLS). CLS posits that humans continually learn through the synergy of two learning systems: the hippocampus, which specializes in learning pattern-separated representations from specific experiences, and the neocortex, which focuses on acquiring more general and transferable representations from sequences of past experiences. The continuously updated weights Q, K, and V act as a global knowledge transmitter, propagating knowledge from old to new, thereby sharing the knowledge from old to new. Additionally, we would like to emphasize that CompILer addresses catastrophic forgetting by freezing the pre-trained ViT which aligns with findings in [2]. \n\nQ7: Formatting issue\n\nA7: Thanks a lot. We will revise their orders.\n\n[1] Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning\n\n[2] Consistent Prompting for Rehearsal-Free Continual Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers x9g2 (R1), B271 (R2), KbBF (R3), and EVGX (R4) for their constructive comments and acknowledgements: “the proposed new task composition-IL is novel and welcome”(R1, R2, R3, R4); “the proposed benchmarks are well-constructed”(R1, R3); “CompILer is well-conceived”(R2, R3); “the paper is good presentation”(R3, R4); “the metric is meaningful”(R1).

Meta-review: After the rebuttal and the authors-reviewers discussion, all reviewers recommended acceptance of the work. After reading the paper, the reviews, and the discussion, I see no reason to overturn their assessment, as the proposed approach and problem formulation can inspire future research in the intersection between compositionality and continual learning. The authors are encouraged to include and discuss the promised baselines and results (e.g., multiple seed runs, Sim-CompILer, additional baselines from the pdf), and follow x9g2's suggestions on captions and hyperparameters discussion.
Decision: Accept (poster)