Title: Not Just Object, But State: Compositional Incremental Learning without Forgetting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Incremental Learning

Compositional Learning

Prompting
Abstract: Most incremental learners excessively prioritize object classes while neglecting various kinds of states (e.g. color and material) attached to the objects. As a result, they are limited in the ability to model state-object compositionality accurately. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL), which enables the model to recognize a variety of state-object compositions in an incremental learning fashion. Since the lack of suitable datasets, we re-organize two existing datasets and make them tailored for composition-IL. Then, we propose a prompt-based Composition Incremental Learner (CompILer), to overcome the ambiguous composition boundary. Specifically, we exploit multi-pool prompt learning, and ensure the inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting which injects object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-art performance achieved by CompILer. Code and datasets are available at: https://github.com/Yanyi-Zhang/CompILer.

Not Just Object, But State: Compositional Incremental Learning without Forgetting

Abstract

Most incremental learners excessively prioritize coarse classes of objects while neglecting various kinds of states ( e . g . color and material) attached to the objects. As a result, they are limited in the ability to reason fine-grained compositionality of state-object pairs. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL), enabling the model to recognize state-object compositions as a whole in an incremental learning fashion. Since the lack of suitable benchmarks, we re-organize two existing datasets and make them tailored for composition-IL. Then, we propose a prompt-based Comp osition I ncremental L earn er ( CompILer ), to overcome the ambiguous composition boundary problem which challenges composition-IL largely. Specifically, we exploit multi-pool prompt learning, which is regularized by inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting by using object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-art performance achieved by CompILer. Code and datasets are available at: https://github.com/Yanyi-Zhang/CompILer .

1 Introduction

Class Incremental Learning (class-IL) [37, 22, 16, 10] gathers increasing attention due to its ability to make the models learn new tasks rapidly, without forgetting previously acquired knowledge. Yet, traditional class-IL sets a strict limit on the old classes such that they should not recur in newly incoming tasks. To break such a strict limitation, recent studies develop a new setting mostly called Blurry Incremental Learning (blur-IL) [24, 11], where the incremental sessions allow the recurrence of previous classes, resulting in a more realistic and flexible scenario. Despite such empirical progresses on incremental learning, they aim to improve object classification only, while overlooking fine-grained states attached to the objects. For instance, analyzing how the clothing styles (akin to states) have changed over time is important for forecasting the future trends that will emerge.

To simultaneously model objects and their states, some efforts are dedicated to Compositional Learning whose aim is how to equip the models with compositionality [4, 31, 45]. The core of compositional learning lies in the structure of class labels, which conceptualizes a state-object pair ( e . g . 'Brown Pants' and 'Yellow Dress') as a whole, rather than a lonely object label. In this way, the model can dissect and reassemble learned knowledge, achieving a more fine-grained understanding about the objects. However, existing works are mainly focused on zero-shot generalization from seen

(a) Class Incremental Learning

(c) Compositional Incremental Learning

Data 1

𝑇

!

𝑇

"

⋯

Data 2

Class Label: Object

⋯

Data 1

𝑇

!

𝑇

"

⋯

Data 2

No Recurrence

Class Label: Object

⋯

Data 1

𝑇

!

𝑇

"

⋯

Data 2

⋯

Class Label: State-Object

Primitives Recurrence

Pants

Suit

Shorts

Dress

Skirt

Shoes

Yellow

Shorts

Black

Suit

Pants

White

Suit

Yellow

Dress

Sliver

Skirt

Pants

Suit

Suit

Dress

Skirt

Shoes

compositions to unseen ones [26, 27, 5, 15], whereas none of them consider the challenging fact that the model must deal with a significantly larger number of composition classes than object classes. As a result, it is hardly feasible to learn all compositions by training the model once.

To remedy the limitations inherent in incremental learning and compositional learning, we conceive a novel task named Compositional Incremental Learning (composition-IL), enabling the model to continually learn new state-object compositions in an incremental fashion. As compared in Fig. 1, we can see that composition-IL integrates the characteristics of class-IL and blur-IL. Although the composition classes are disjoint across incremental tasks, the primitive classes ( i . e . objects and states) encountered in old tasks are allowed to reappear in new tasks. Unfortunately, existing incremental learning approaches are challenged by such a compositional scenario, because their models excessively prioritize the object primitives while neglecting the state primitives. Consequently, the compositions with the same object but with different states become ambiguous and indistinguishable.

To tackle the problem, we propose a rehearsal-free and prompt-based Comp ositional I ncremental L earn er ( CompILer ). Specifically, our model comprises of three primary components: multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion. Firstly, we construct three prompt pools for learning the states, objects and compositions individually. Upon that, we add extra restrictions to regularize the inter-pool prompt discrepancy and intra-pool prompt diversity. This multi-pool prompt learning paradigm strengthens the fine-grained understanding and reasoning towards primitive concepts and their compositions. In addition, as the state classes are more difficult to distinguish than the object ones, we propose object-injected state prompting which incorporates object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean fusion manner, which helps to adaptively eliminate irrelevant information learned in the prompts. Last but not least, we also leverage symmetric cross-entropy loss to alleviate the impact of noisy data during training.

In summary, the main contributions in this work are encapsulated as follows: (1) We devise a new task coined compositional incremental learning (composition-IL). It enables learning fine-grained state-object compositions continually while the isolated primitive concepts can randomly recur in incremental tasks. (2) To address the lack of datasets, we re-organize two existing datasets such that they are tailored specifically for composition-IL. For the two new datasets (Split-Clothing and Split-UT-Zappos), we split them into 5 and 10 incremental tasks for evaluating the methods. (3) We propose a novel learning-to-prompt model for composition-IL, namely CompILer. Our stateof-the-art results on Split-Clothing and Split-UT-Zappos validate the effectiveness of CompILer for incrementally learning new compositions without forgetting old ones.

2 Related Work

Incremental Learning. The approaches to addressing catastrophic forgetting for incremental learning can be broadly grouped into four categories: regularization based methods [6, 39] aim to protect

Split-Clothing(5Tasks)

Split-UT-Zappos(5 tasks)

Split-UT-Zappos(10 tasks)

Task1

Task 2

Task3

Task 4

Task 5

Task6

Task 7

Task8

Task 9

Task10

influential weights of old experiences from updating; knowledge distillation based methods [16, 34] distill knowledge from the model trained on the previous tasks and adapt it to new tasks; rehearsal based methods [32, 48, 20] require a memory buffer to store some old data, so as to make the network remember previous tasks; parameter isolation methods allocates different model parameters to each task, to prevent any possible interference. Different from the methods, L2P [43] proposes an innovative learning-to-prompt paradigm, which incorporates plasticity and stability through adapting a set of learnable prompt tokens on top of a frozen pre-trained backbone. Inspired by L2P [43], more recent works [42, 36, 2, 28] take full advantage of various prompt tuning strategies, achieving new state-of-the-art performance for incremental learning. However, such methods take into account object classes solely, while neglecting various kinds of state classes associated with the objects. To this end, our work proposes compositional incremental learning with the purpose to continually identifying the composition classes of state-object pairs. Note that, Liao, et al [18] conduct an initial study toward the compositionality in incremental learning, whereas their attention is on the composition of multiple object classes ( e . g . 'Car' and 'Person') in one image, rather than the state-object compositions in this work.

Compositional Learning. Amajor line of compositional learning research focuses on Compositional Zero-Shot Learning (CZSL) [23], which aims to infer unseen state-object compositions by acquiring knowledge from seen ones. Subsequent approaches building upon the CZSL setting further incorporate graph neural networks to model the dependency between primitives and compositions [25], and employ cosine classifiers to avoid being overly biased toward seen compositions [21]. Other approaches [13, 14, 12] propose training two classifiers to identify states and objects separately. The latest works [19, 3, 38, 8, 47] model both composition and primitives simultaneously, achieving state-of-the-art results. Albeit the numerous attempts made in compositional learning, they fail to consider an incremental learning paradigm given the increasing number of composition classes in open-world scenarios. Besides, directly applying CZSL methods to composition-IL might lead to a stale and decaying performance on forgetting. By contrast, our proposed CompILer markedly bypasses catastrophic forgetting with the help of multi-pool prompt learning.

3 Preliminaries

In this section, we firstly define the task of composition-IL, and then introduce two datasets we construct for the task, followed by revealing the ambiguous composition boundary problem.

3.1 Problem Definition

For composition-IL, a model sequentially learns N tasks T = {T 1 , T 2 , · · · T N } corresponding to a set of composition classes C = {C 1 , C 2 , · · · C N } . We note that the composition classes between incremental tasks are always disjoint, which means C i ∩ C j = ∅ for any i = j . Different from the composition classes, the primitive classes are allowed to recur in different tasks. That means it allows the tasks to share some primitive concepts of objects and states. Therefore, we can define the set of all state and object classes with S = { s 1 , s 2 , · · · , s n } and O = { o 1 , o 2 , · · · , o m } , respectively. Given each image x , it has a composition label c which is constructed with a state label s and an object label o , i . e . c = < s, o > , where c ∈ C , s ∈ S and o ∈ O . We take the example of 'red shirt', where 'red' is denoted with s , 'shirt' corresponds to o , and 'red shirt' is expressed with c .

̸

Red Shoes

WhiteShoes

Pink Skirt

White Dress

Blue Dress

BlackDress

YellowDress

(a) L2P

(b) Compiler

3.2 Dataset Construction

As there are no existing datasets suitable for composition-IL, we re-organize the data in Clothing16K [46] and UT-Zappos50K [44], and construct two new datasets tailored for composition-IL, namely Split-Clothing and Split-UT-Zappos . To be more specific, we firstly sort the composition classes based on the number of their images, and then select the foremost 35 compositions from Clothing16K and the top 80 from UT-Zappos50K, so as to construct Split-Clothing and Split-UT-Zappos, respectively. In this way, Split-Clothing encompasses 9 states and 8 objects while Split-UT-Zappos consists of 15 states and 12 objects in total. For Split-Clothing, we randomly partition the compositions into 5 tasks. Regarding Split-UT-Zappos, the compositions are sorted by count and are evenly divided into 5 and 10 tasks. The image distribution for each task is shown in Fig. 2. Note that, we elaborate more details on both datasets in the following technical appendix.

3.3 Revealing the Ambiguous Composition Boundary

The main stumbling block in composition-IL is the ambiguous composition boundary. Although the composition label consists of two primitives ( i . e . object and state), we note that the model excessively prioritizes the object primitive while neglecting the state primitive. Consequently, the compositions with the same object but with different states become ambiguous and indistinguishable. To prove that, we apply L2P [43] to composition-IL, whereas it is challenged by significant ambiguities in composition classification. As illustrated in Fig. 3 (a), the t -SNE visualization showcases the entanglement among the compositions like 'white dress', 'black dress' and 'blue dress'. We conjecture that this ambiguous problem tends to become more severe when more tasks are arriving incrementally. To address it, we propose a new model namely CompILer, which disentangles compositions and primitives via a multi-pool prompt learning. Advantageously, our method promotes the learning on the states and establishes clearer composition boundaries, as shown in Fig. 3 (b).

4 Methodology

Overview. We leverage the learning-to-prompt paradigm [43] and develop a novel compositional incremental learner (CompILer) tailored specifically for composition-IL. As depicted in Fig. 4, CompILer comprises three primary components: multi-pool prompt learning, object-guided state prompting, and generalized-mean prompt fusion. Firstly, we initialize three prompt pools dedicated to learning and storing visual information related to states, objects and their compositions. In order to differentiate the knowledge learned across and within prompt pools, we define inter-pool discrepant loss and intra-pool diversified loss jointly. We then employ object prompts to guide the selection of state prompts, thereby improving the state representation learning. Moreover, we utilize a generalizedmean fusion to integrate the selected prompts in a learnable manner. Ultimately, we optimize the classification objective with symmetric cross-entropy loss, to alleviate the effect of noisy data.

4.1 Multi-pool Prompt Learning

The learning-to-prompt paradigm [43, 35], especially suitable for large pre-trained backbones, has opened up a new path for incremental learning. It has proven to incorporate plasticity and stability

ℒ

!"#

$

Object Prompt Pool

State Prompt Pool

Composition Prompt Pool

𝑓

!

(#)

Transformer Encoder

𝑓

"

(#)

𝜑

𝑐

(#) 𝜑

𝑠

(#) 𝜑

𝑜

(#)

ℒ

!"#

(

ℒ

!"#

)

❄

🔥

···

Selection

𝑲

#

Selection

𝑲

$

Selection

𝑲

%

GeM

Fusion

Object-injected

State Prompting

Feature

Extractor

𝑓($)

❄

❄

GeM

Fusion

GeM

Fusion

❄

{𝑃

#

$

!

}

&'(

)

{𝑃

$

$

!

}

&'(

)

{𝑃

%

$

!

}

&'(

)

𝑷

#

𝑷

$

𝑷

%

Prompt

Image Token

Prompt-Key Pair

𝜑

𝜔

(#)

Classifier

Learnable

❄

Frozen

Composition

State

Object

Yellow Dress

better through adapting a set of learnable tokens in a prompt pool to a frozen pre-trained backbone. Nevertheless, existing prompt-based approaches are initially designed for class-IL, thereby building a single prompt pool for object classification solely. when dealing with state-object composition classification, they tend to excessively prioritize the object primitive while neglecting the state primitive. To this end, we propose to construct three discrepant and diversified prompt pools P s , P o and P c , which serve to learn visual information related to states, objects and their compositions, respectively. Besides, each pool is associated with a set of learnable keys K ω for query-key prompt selection. The three prompt pools and their keys are defined as:

where P i ω ∈ R L × D is a single prompt with token length L and embedding dimension D . K i ω ∈ R D , the key of P i ω , is a learnable token with the same size. M is the number of prompts in each pool.

One important concern in such multi-pool prompt learning is how to enrich the prompts with the avoidance of identical pools. To achieve it, we consider integrating inter-pool prompt discrepancy and intra-pool prompt diversity jointly. On the one hand, the inter-pool prompts should be discrepant as the visual information about states, objects, and compositions should be different. One the other hand, within each pool, the intra-pool prompts should be diversified so to capture more comprehensive features from all the classes.

In practice, we formulate a unified objective to regularize both inter-pool discrepancy and intra-pool diversity, by leveraging a simple and effective directional decoupled loss used in [17]. The directional decoupled (dd) loss between any two pools ( e . g . P i and P j ) is formulated as:

where θ nm measures the angle between any two prompts, n and m ; ϵ is a scalar to avoid division by zero. Note that, L ( i,j ) dd encourages the angles between each prompt to be at least θ thre degrees. Since ( i, j ) is unordered Cartesian product of ω , i . e . ( i, j ) ∈ { ( i, j ) | i ∈ ω ∧ j ∈ ω } , the inter-pool prompt discrepancy loss for the three pools can be expressed with L inter = L ( s,o ) dd + L ( s,c ) dd + L ( o,c ) dd , and the intra-pool prompt diversity loss becomes L intra = L ( s,s ) dd + L ( o,o ) dd + L ( c,c ) dd . As opposed to L inter , L intra computes the angle between any two prompts within the same pool. Thus, it contains the case when n = m , for which we set θ thre -θ nm = 0 .

4.2 Object-injected State Prompting

Akin to the query-key matching mechanism in other work [43, 42, 40], we utilize a fixed feature extractor f ( · ) to obtain a query feature q ( x ) = f ( x )[0 , :] , determining which prompts in the pool to be selected. However, pre-trained backbones are typically trained for object classification, thus under-performing for state representation learning. In addition, it is more difficult to predict the state classes due to their more abstract and fine-grained characteristics. To tackle this problem, we strategically inject object prompts to guide the selection of state prompts. Intuitively, once we have learned knowledge about the object class, it may be easier to predict the correct state class and avoid mistaken results. For instance, given an object is 'heels', we can expect that the corresponding

···

···

···

···

···

Query Feature

Q

K

V

Fused Object Prompt

𝑷

!

Injected Query Feature

𝑞

"

(𝑥)

···

Token 1 Token 2

Token N

Dot

Product

state is unlikely to be 'canvas' or 'plastic'. To summarize, we select object and composition prompts in each pool based on the original query feature, which means q o ( x ) = q c ( x ) = q ( x ) ; but for the selection of state prompts, we propose object-injected state prompting to ameliorate the query feature as shown in Fig. 5.

Specifically, we employ the fused object prompt P o (see Sec. 4.3) to perform cross attention on the query feature q ( x ) , resulting in object-injected query feature q s ( x ) for the state prompt selection:

where W Q , W K and W V are learnable projections. To establish alignment between the query and the selected prompts, we optimize a surrogate loss for state, object and composition prompting jointly:

where COS ( · , · ) denotes cosine similarity, K ω represents the subset of top-k keys selected from K ω , and { s i } k i =1 is a subset of top-k indices from [1 , M ] (prompt number). Despite the simplicity of the object-injected state prompting, it facilitates more judicious prompt selection within the state prompt pool, alleviating the hurdles posed by state learning.

4.3 Generalized-mean Prompt Fusion

After obtaining the selected top-k prompts { P s i ω } k i =1 , the next step is fusing these prompts into a single prompt. It is general to utilize a simple mean pooling whereas it overlooks the relative importance of each prompt. Besides, when the prompts contain information that is unrelated or contradictory to current task, it is critical to strengthen useful prompts and eliminate irrelevant ones. To this end, we draw inspiration from generalized-mean pooling [29] and exploit generalized-mean (GeM) prompt fusion which is given by:

where η is a learnable parameter. When η = 1 , GeM becomes mean pooling; as η approaches infinity ( η →∞ ), it converges to max pooling. By taking over mean and max pooling, GeM learns to achieve an optimal fusion, mitigating the influence of irrelevant information present in the prompts.

4.4 Training and Inference

Classification Objective. We prepend three fused prompts ( i . e . P s , P o and P c ) with x e , which is the output from a ViT embedding layer f e ( · ) . The extended token sequence is x p = [ P c ; P s ; P o ; x e ] .

Then, we feed x p to a transformer encoder layer f r ( · ) and achieve P r s , P r o and P r c for classifying state, object and composition classes, respectively. We estimate the probability via a classifier φ ω ( · ) : p ( ω | x ) = φ ω ( P r ω ) . For each image x , we denote its ground-truth distribution over labels with q ( ω | x ) . When ω is consistent with the ground truth, then q ( ω | x ) = 1 ; otherwise, q ( ω | x ) = 0 . As a result, the cross entropy (CE) loss used for classification objective is:

where Ω represents the number of classes. However, the model optimized with a standard CE loss is easily affected by noisy samples during training. Instead, we advocate using a symmetric cross entropy loss (SCE) [41], which incorporates an additional term called reverse cross entropy (RCE), to mitigate the impact of noisy data. Contrary to CE, the formula for RCE loss is defined as:

Then, the SCE loss combines two loss terms by L ω SCE = L ω CE + α L ω RCE , where α is a hyperparameter that controls the weight of the RCE term. As a result, the whole SCE loss becomes L SCE = L c SCE + β ( L s SCE + L o SCE ) , where β adjusts the weights between primitives and compositions.

Total Loss. The total loss for training the whole CompILer model is:

where λ 1 , λ 2 , λ 3 are hyper-parameters balancing different terms.

Inference. During inference, we incorporate the primitive probabilities to aid the composition probability. Hence, the final probability for composition classification is expressed with:

where µ adjusts the probabilities.

5 Experiments

5.1 Datasets and Metrics

We conduct experiments on two newly split datasets: Split-Clothing and Split-UT-Zappos as elucidated in Section 3.2. We assess the overall performance on compositions using Average Accuracy ( Avg Acc ) and Forgetting ( FTT ). A higher Avg Acc signifies stronger recognition abilities, while a lower FTT indicates improved resilience against forgetting. Additionally, we provide individual Average Accuracy scores on states and objects, denoted as State and Object for simplicity. These metrics imply the ability to recognize fine-grained primitives. Furthermore, we calculate the Harmonic Mean ( HM ) between State and Object, i . e . HM = 2 × ( State × Object ) ( State + Object ) . We provide more emphasis to Avg Acc and HM due to their more comprehensive assessment. Avg Acc encompasses the plasticity and stability [33, 2] and HM provides a holistic evaluation on both state and object.

5.2 Implementation Details

For a fair comparison with previous works [43, 42, 2, 33], we also employ ViT B/16 [1] pretrained on the ImageNet 1K dataset as the feature extractor and backbone. For multi-pool prompt learning, the size of each pool is set to 20 , and each prompt has 5 tokens. We select top-5 prompts from each pool and generate a fused prompt. During training, we utilize the Adam optimizer [9] with a batch size of 16 . The whole CompILer undergoes training for 25 epochs on the SplitClothing, for 10 epochs on the 5-task Split-UT-Zappos, and for 3 epochs on

the 10-task Split-UT-Zappos. For the Split-Clothing and the 10-task Split-UT-Zappos, we set the learning rate to 0 . 03 , while we use a learning rate of 0 . 02 for the 5-task Split-UT-Zappos. Note that, for all the methods, their results are averaged over three runs with the corresponding standard deviations reported to mitigate the influence of random factors.

As there are a few hyper-parameters in the model, we conduct a rigorous tuning on them. For instance, we set θ thre to π 2 for all settings. For Split-Clothing, the loss weights λ 1 and λ 3 are set to 0 . 1 ; λ 2 is set to 10 -7 ; α and β for SCE loss are 0 . 006 and 0 . 3 , and the parameter µ during inference is 0 . 5 . For 5-task Split-UT-Zappos, λ 1 , λ 2 , λ 3 , α , β and µ are set to 1 . 0 , 3 × 10 -6 , 0 . 7 , 0 . 01 , 0 . 7 and 0 . 02 , respectively. For 10-task Split-UT-Zappos, λ 1 , λ 2 , λ 3 , α , β and µ are set to 0 . 5 , 10 -7 , 0 . 1 , 0 . 05 , 0 . 4 and 0 . 03 . We elaborate more details on hyper-parameter analysis in the appendix.

5.3 Compared Baselines

To demonstrate the effectiveness of the proposed method, we compare CompILer with state-of-the-art incremental learning methods, including prompt-free approaches [10, 16, 32] and prompt-based methods [43, 42, 33, 7]. All the methods are rehearsal-free except iCaRL [32]. Note that, due to LGCL [7] relying on CLIP [30] to achieve language guidance at the task level, it is limited by the length of class names per task. Thereby, LGCL fails to operate on the 5-task Split-UT-Zappos since the total length of class names exceeds the limitation.

To streamline our CompILer, we further implement a simplified version named Sim-CompILer and report its results. Sim-CompILer is optimized using cross entropy loss and is comprised solely of multi-pool prompt learning and generalized-mean prompt fusion. In other words, we exclude the object-injected prompting, directional decoupled loss, and reverse cross entropy loss, resulting in a large reduction of hyperparameters to only β , λ 3 , and µ .

5.4 Comparison with the State-of-the-arts

The compared results on Avg Acc and FTT are reported in Table 1. Overall, CompILer consistently outperforms all competitors on Avg Acc by a significant margin. For FTT scores, CompILer excels previous methods with 0.32% on the 5-task Split-Clothing and with 0.14% on the 5-task Split-UTZappos, while falling behind Dual-Prompt [42] and LGCL [7] for the 10-task Split-UT-Zappos. We notice that, the main reason is these methods sacrifice more plasticity for lower forgetting rates. Besides, the number of model parameters in these methods dynamically increases along with more incremental tasks arriving, whereas our CompILer does not rely on imposing task-specific parameters to reduce the forgetting.

We also report the primitives accuracy and their HM in Table 2 and Table 3. Likewise, our method surpasses other methods considerably in terms of State and HM. Interestingly, the prompt-free methods [16, 10, 32] achieve higher accuracy in state prediction than object prediction for SplitClothing, which is contrary to other results. This is because the states in Split-Clothing are colorrelated descriptions, which are easier to capture with the help of parameter fine-tuning. The promptbased methods do not exhibit this phenomenon because their pre-trained backbones are initially

trained for object classification, and are frozen across incremental sessions. As the performance improvements are mainly attributed to the accuracy of state recognition, it suggests that our model enhances the understanding on fine-grained compositionality.

5.5 Ablation Study and Analysis

Effect of multi-pool prompt learning. This experiment aims to delineate the contribution of three pools in CompILer. We firstly implement a baseline model with composition prompt pool only. Building upon the baseline, we develop two additional models, which incorporate either object or state prompt pool. As reported in Table 4, the inclusion of primitive prompt pool yields consistent gains over the baseline. Fur-

thermore, the best results are achieved when the model integrates all three pools simultaneously. This experiment signifies the significant necessity of exploiting multiple prompt pools for composition-IL.

Effect of object-injected state prompting. To provide insights into object-injected state prompting, we compare three models: None (vanilla model), S → O (state-injected object prompting) and O → S (object-injected state prompting). As shown in Table 5a, compared to the None model, the S → O exhibits a decrease in all metrics, implying that state prompts may interfere with the selection of object prompts. On the contrary, O → S outperforms the None model as we expect. This phenomenon validates our motivation that state recognition is harder than object recognition, and thereby the former cannot help the latter easily. Yet, it is a promising direction for future research.

Effect of generalized-mean prompt fusion. This study aims to study the impact of prompt fusion on CompILer. As shown in Table 5b, GeM performs better than both max and mean pooling across various metrics. It validates the benefit of GeM on mitigating irrelevant information in the selected prompts. as it may hamper the model's attention on image tokens.

Effect of loss functions. As shown in Table 6, we investigate the influence of loss functions used in our model, including directional decoupled loss ( L inter and L intra ) and symmetric cross entropy loss ( L CE and L RCE ). The baseline model (the first row) includes all modules but is trained by cross entropy loss only. By adding the RCE loss, the model is equivalent to training with the SCE loss, which help to improve the robustness to noisy labels. The use of either L inter or L intra improves the performance on both datasets, and synchronously applying them witnesses all-around improvements

(d) Prediction Results

Suede Heels

Suede Heels

Red Hoodie

Leather Ankle

Leather Ankle

Suede Ankle

Yellow Dress

Canvas Sneakers

Canvas Sneakers

Suede Slippers

Black Suit

Black Suit

Green Suit

(a) Composition Accuracy

99.49

97.77

94.17

92.97

92.80

96.34

92.98

85.95

83.66

92.37

83.80

83.80

93.24

93.24

90.20

T1

T2

T3

T4

T5

After Tk

T1

T2

T3

T4

T5

(b) State Accuracy

99.68

98.46

97.08

96.74

96.57

97.25

93.44

86.40

85.19

95.33

86.16

86.60

96.14

96.62

93.08

T1

T2

T3

T4

T5

After Tk

T1

T2

T3

T4

T5

(c) Object Accuracy

99.83

98.97

96.47

96.06

96.06

98.32

98.32

98.60

97.25

96.26

97.04

96.10

95.81

95.33

96.97

T1

T2

T3

T4

T5

After Tk

T1

T2

T3

T4

T5

compared to the baseline. Eventually, we achieve the best results when combing all the loss terms during training.

5.6 Additional Results and Analysis

In order to study the repeatability characteristic in composition-IL, we exhibit more results on Split-Clothing in Fig. 6: in (a), it shows a decreasing trend in composition accuracy along with the introduction of new tasks; however, the green rectangles in (b) and (c) showcase that the accuracy occasionally increases as more tasks are learned. We conjecture the reason is mostly attributed to the re-occurrence of primitive concepts. This forward transfer is critical for incremental learners. We compare the composition predictions between CompILer and L2P [43] in Fig. 6 (d). CompILer predicts all the images correctly, while L2P makes some mistakes, particularly for state labels. This limitation arises from an excessive focus on the dominant object primitive, while weakening the attention toward state primitive. Fortunately, CompILer relieves the bias toward object classes, and enhances the perception on state classes.

6 Conclusion

In this paper, we have proposed a novel task coined compositional incremental learning (compostionIL), which is stumbled by ambiguous composition boundary. To tackle it, we develop a learningto-prompt model, namely CompILer. Our model exploits multi-pool prompt learning to model composition and primitive concepts, object-injected state prompting to improve the selection of state prompts, and generalized-mean prompt fusion to eliminate irrelevant information. Extensive experiments on two tailored datasets show that CompILer achieves state-of-the-art performance. In the future, it is challenging yet potential to consider reasoning multiple state classes per object.

7 Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant Numbers 62102061, 62272083 and 62472066, and in part by the Open Projects Program of State Key Laboratory of Multimodal Artificial Intelligence Systems.

References

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal-free continual learning. In IEEE Conf. Comput. Vis. Pattern Recog. , 2024.

Shaozhe Hao, Kai Han, and Kwan-Yee K Wong. Learning attention as disentangler for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 1531515324, 2023.

Geoffrey Hinton. Some demonstrations of the effects of structural descriptions in mental imagery. Cognitive Science , 3(3):231-250, 1979.

Chenchen Jing, Yukun Li, Hao Chen, and Chunhua Shen. Retrieval-augmented primitive representations for compositional zero-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2652-2660, 2024.

Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. Adv. Neural Inform. Process. Syst. , 33:3647-3658, 2020.

Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in promptbased continual learning. In Int. Conf. Comput. Vis. , pages 11429-11439, 2023.

Hanjae Kim, Jiyoung Lee, Seongheon Park, and Kwanghoon Sohn. Hierarchical visual primitive experts for compositional zero-shot learning. In Int. Conf. Comput. Vis. , pages 5675-5685, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In International Conference on Learning Representations , 2022.

Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang. Siamese contrastive embedding network for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 9326-9335, 2022.

Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object compositions. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 11316-11325, 2020.

Yong-Lu Li, Yue Xu, Xinyu Xu, Xiaohan Mao, and Cewu Lu. Learning single/multi-attribute of object with symmetry and group. IEEE Trans. Pattern Anal. Mach. Intell. , 44(12):9043-9055, 2021.

Yun Li, Zhe Liu, Hang Chen, and Lina Yao. Context-based and diversity-driven specificity in compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog. , 2024.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. , 40(12):2935-2947, 2017.

Zilong Li, Yiming Lei, Chenglong Ma, Junping Zhang, and Hongming Shan. Prompt-in-prompt learning for universal image restoration. arXiv preprint arXiv:2312.05038 , 2023.

Weiduo Liao, Ying Wei, Mingchen Jiang, Qingfu Zhang, and Hisao Ishibuchi. Does continual learning meet compositionality? new benchmarks and an evaluation framework. Adv. Neural Inform. Process. Syst. , 2023.

Yu Liu, Jianghao Li, Yanyi Zhang, Qi Jia, Weimin Wang, Nan Pu, and Nicu Sebe. Pmgnet: Disentanglement and entanglement benefit mutually for compositional zero-shot learning. Computer Vision and Image Understanding , 2024.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Adv. Neural Inform. Process. Syst. , 30, 2017.

Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 5222-5230, 2021.

Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Trans. Pattern Anal. Mach. Intell. , 45(5):5513-5533, 2022.

Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 1792-1801, 2017.

Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 11731-11741, 2023.

Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 953-962, 2021.

Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attributeobject compositions. In Eur. Conf. Comput. Vis. , pages 169-185, 2018.

Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc'Aurelio Ranzato. Taskdriven modular networks for zero-shot compositional learning. In Int. Conf. Comput. Vis. , pages 3593-3602, 2019.

Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, Yuan Xie, et al. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations , 2023.

Filip Radenovi´ c, Giorgos Tolias, and Ondˇ rej Chum. Fine-tuning cnn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence , 41(7):16551668, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763, 2021.

Pasko Rakic, Jean-Pierre Bourgeois, and Patricia S Goldman-Rakic. The self-organizing brain: from growth cones to functional networks . Elsevier, 1994.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 2001-2010, 2017.

James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogério Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 11909-11919, 2023.

Filip Szatkowski, Mateusz Pyla, Marcin Przewi˛ e´ zlikowski, Sebastian Cygert, Bartłomiej Twardowski, and Tomasz Trzci´ nski. Adapt your teacher: Improving knowledge distillation for exemplar-free continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1977-1987, 2024.

Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When prompt-based incremental learning does not meet strong pretraining. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 1706-1716, 2023.

Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Adv. Neural Inform. Process. Syst. , 36, 2024.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Trans. Pattern Anal. Mach. Intell. , 2024.

Qingsheng Wang, Lingqiao Liu, Chenchen Jing, Hao Chen, Guoqiang Liang, Peng Wang, and Chunhua Shen. Learning conditional attributes for compositional zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 11197-11206, 2023.

Wenjin Wang, Yunqing Hu, Qianglong Chen, and Yin Zhang. Task difficulty aware parameter allocation & regularization for lifelong learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 7776-7785, 2023.

Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Adv. Neural Inform. Process. Syst. , 35:5682-5695, 2022.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Int. Conf. Comput. Vis. , pages 322-330, 2019.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Eur. Conf. Comput. Vis. , pages 631-648, 2022.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Learning to prompt for continual learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 139-149, 2022.

Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 192-199, 2014.

Alan L Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? Int. J. Comput. Vis. , 129(3):781-802, 2021.

Tian Zhang, Kongming Liang, Ruoyi Du, Xian Sun, Zhanyu Ma, and Jun Guo. Learning invariant visual representations for compositional zero-shot learning. In Eur. Conf. Comput. Vis. , pages 339-355, 2022.

Yanyi Zhang, Qi Jia, Xin Fan, Yu Liu, and Ran He. Cscnet: Class-specified cascaded network for compositional zero-shot learning. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 3705-3709, 2024.

Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Sean Lim, and Yunzhe Jia. A simple but strong baseline for online continual learning: Repeated augmented rehearsal. Adv. Neural Inform. Process. Syst. , 35:14771-14783, 2022.

A Appendix

In addition to the content in the main paper, this appendix elaborates more details on the datasets, algorithm procedure, empirical analysis, hyper-parameter analysis, and quantitative and qualitative experiments. Meanwhile, please refer to the source code and some data samples included in the supplementary material we submit.

0

1000

2000

3000

T1

T2

T3

T4

T5

(ⅰ) Split-Clothing (5 tasks)

0

2000

4000

6000

T1

T2

T3

T4

T5

(ii) Split-UT-Zappos (5 tasks)

0

1000

2000

3000

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

A.1 More on Dataset Details

We provide more information about two newly tailored datasets (Split-Clothing and Split-UT-Zappos) and extensively discuss the limitations of Split-UT-Zappos. The number of images in each task is shown in Fig. 7.

Split-Clothing is derived from Clothing16K, which is originally used for multi-label classification tasks and consists of 37 state-object compositions and 16,170 images scraped from Google, Bing, and DuckDuckGo. We sorted these compositions by the number of images and selected the top 35 compositions, resulting in a total of 15,915 images in the Split-Clothing dataset. Specifically, the object in Split-Clothing includes various types of garments such as 'dresses' and 'hoodies', while the state delineates the color of the clothing, such as 'silver' and 'yellow'. We allocate 80% of the dataset for training and the remaining 20% for testing. We split this dataset for 5 incremental tasks.

Split-UT-Zappos is a subset of UT-Zappos50K, which is a large shoe dataset consisting of 50,025 catalog images collected from Zappos.com. This dataset is originally created for an online shopping task and we selected a total of 28,497 images to comprise the Split-UT-Zappos. The state classes in Split-UT-Zappos describe different materials ( e . g . canvas and leather) and the object classes are related to footwear ( e . g . heels and slippers). Consistent with Split-Clothing, we employ 80% of the images for training and 20% for testing. We split this dataset for either 5 or 10 incremental tasks.

Limitations of Split-UT-Zappos. It is noteworthy that, even for the upper bound model ( i . e . supervised learning with all the data), satisfactory performance remains elusive on Split-UT-Zappos as shown in Table 1 and Table 3. This deficiency can be attributed to the pronounced long-tail distributions and invisible states inherent in UT-Zappos50K. For example, the composition 'Faux Fur Slippers' comprises a mere 25 training images, whereas 'Leather Sandals' accounts for 1783. This disproportionate distribution predisposes the severe bias towards the compositions prevalent in the head of the distributions, while inadequately capturing those in the tail. Besides, some states in Split-UT-Zappos lik,e 'Leather' vs 'Synthetic Leather', are material differences that are not always visible as visual transformations. The deficient state description poses a greater challenge for classification, thereby resulting in suboptimal upper bound results. We contend that tackling the long-tail distribution challenge within Split-UT-Zappos and constructing more balanced datasets conducive to composition-IL represent promising avenues for future research.

A.2 More on Training Procedure

We summarize the training procedure of the proposed CompILer in Algorithm 1.

Algorithm 1: Training Procedure of CompILer for composition-IL

1 Input: Training data for T tasks, where each image x has a set of three labels: c , s and o . Three prompt pools P ω with corresponding keys K ω , where ω ∈ { s, o, c } . Pre-trianed feature extractor f ( · ) , pre-trained input embedding layer f e ( · ) , pre-trained transformer encoder layer f ( · ) , cross attention layer CrossAttn ( · , · ) , classifiers φ ( · ) , GeM fusion GeM ( · · · ) .

A.3 Empirical Analysis on Learned Prompts

In Section 4.1, we design multi-pool prompt learning to learn visual representations of state, object, and composition, respectively. By applying inter-pool discrepant loss and intra-pool diversified loss, we ensure discrepancy across pools and diversity within each pool. To validate the effectiveness of the approach, we conduct t -SNE visualization on the prompt pools within the three experiment settings. Figure 8 illustrates that the learned knowledge across the three pools is remarkably discriminative, suggesting that each pool has effectively captured unique features. Meanwhile, the prompts within each pool exhibit a relatively wide dispersion demonstrating that the learned information within each pool is diversified.

A.4 Empirical Analysis of Generalized-mean Prompt Fusion

The detailed operation of Generalized-mean Prompt Fusion is elaborated on Section 4.3. The parameter η in Eq. 6 can be learnable as this operation is differentiable, allowing it to be included in the whole back-propagation process. To prove that, the corresponding derivatives of Eq. 6 are given by:

A.5 Empirical Analysis of Symmetric Cross Entropy Loss

In this section, we delve into why symmetric cross entropy (SCE) loss can effectively mitigate noisy data in the dataset. We define two distributions, q and p , where q = ( k | x ) represents the ground truth class distribution for sample x and p = ( k | x ) represents the predicted class distribution. The

(a) Split-Clothing (5 tasks)

46.0

46.4

46.8

47.2

0.8

0.9

1.0

1.1

1.2

Avg Acc (%)

45

45.8

46.6

47.4

3e-6

1e-5

3e-5

1e-4

3e-4

Avg Acc (%)

(a) Impact of

𝜆

!

(b) Impact of

𝜆

"

45

45.8

46.6

47.4

0.5

0.6

0.7

0.8

0.9

Avg Acc (%)

(b) Impact of

𝜆

#

7

7

6

6

5

relation between the cross entropy H ( q, p ) and KL-divergence KL ( q ∥ p ) is expressed as:

in which H ( q ) denotes the entropy of q . From the perspective of KL divergence, the optimization objective is to make the prediction distribution p closely resemble the ground truth distribution q , essentially minimizing their KL divergence. However, in the presence of noise within the dataset, q may not accurately reflect the ground-truth class distribution, whereas p might reveal the real distribution. Therefore, it becomes necessary to consider the reverse direction of KL divergence, i . e . KL ( p ∥ q ) and construct symmetric KL divergence represented by:

By transferring this idea from KL divergence to cross entropy, the formula for symmetric cross entropy is given by:

where RCE = H ( p, q ) is the reverse counterpart of H ( q, p ) , known as reverse cross entropy.

A.6 More on Hyper-parameter Analysis

We investigate the impact of several hyper-parameters, as depicted in Fig. 9. Specifically, we explore the influence of weights in the total loss. We present the results of tuning the weight λ 1 , λ 2 , and λ 3 on the Split-UT-Zappos (5 tasks). The average accuracy shown in Fig. 9(a), (b), and (c) initially exhibits an increasing trend, followed by an overall decreasing trend. CompILer reaches its peak performance when λ 1 = 1 . 0 , λ 2 = 3 e -6 , and λ 3 = 0 . 7 .

A.7 More on Ablation Studies

Effect of multi-pool prompt learning. In addition to presenting the results of Avg ACC, FTT, and HM on Split-Clothing (5 tasks) for multi-pool prompt learning, we also include the remaining results for State and Object in Table 7. An intriguing observation is that the model achieves the best

results for Object when the object pool is used exclusively, while it reaches the peak performance for State when only the state pool is introduced. This phenomenon suggests that incorporating the primitive prompt pools significantly aids in learning the representations of the corresponding primitives. Furthermore, when the model integrates the state prompt pool alone, not only does the accuracy of state is improved, but the accuracy of object also experiences significant gains, and vice versa. This implies that the learning of one type of primitive representation is crucial for the learning of another. Only when all three pools are integrated does the model achieve state-of-the-art performance in the most important metrics, namely average accuracy and HM.

Effect of object-injected state prompting. In this section, we report the remaining results for object-injected state prompting on Split-Clothing (5 tasks) as shown in Table 8. State-injected object prompting (S → O) introduces interference information for composition learning, resulting in a decline across all metrics expect Object. By contrast, object-injected state prompting (O → S) leads to a significant improvement across all metrics, demonstrating the effectiveness of the proposed method.

Effect of generalized-mean prompt fusion. We show the intact results of the ablation study on pooling methods on Split-Clothing (5 tasks) in Fig. 9. It is obvious that max pooling hits the lowest result, while integrating mean pooling leads to a stable growth. Only deploying Generalized-mean pooling achieves the state-of-the-art results, surging from 84.70 to 89.21 in Avg Acc and plunging to 7.26 in FTT. Consequently, the theory that GeM pooling can diminish the irrelevant information and emphasize the crucial aspects is proved.

Effect of loss functions. Eventually, we present supplementary ablation experiments on loss functions in Table 10 and 11. The first row indicates that the model is optimized using cross entropy loss exclusively which achieves the poorest results. Furthermore, integrating either reverse cross entropy loss, inter-pool prompt discrepancy loss, or intra-pool diversity loss contributes to improvements across all metrics. Notably, the effectiveness of reverse cross entropy loss is higher on Split-UTZappos (5 tasks) compared to that on Split-Clothing (5 tasks). To conclude, the best performance is achieved when all loss functions are simultaneously employed.

A.8 More on Qualitative Results

More qualitative results are presented to demonstrate the effectiveness of CompILer. We conducted a comparative analysis of the prediction results between CompILer and L2P [43] on Split-Clothing (5 tasks), Split-UT-Zappos (5 tasks) and Split-UT-Zappos (10 tasks). The ground truth is presented in the first row, followed by the prediction results of CompILer and L2P in the second and third rows, respectively. As illustrated in the Fig. 10, CompILer achieves correct predictions for all the images, while L2P consistently yields wrong classification results due to ambiguous composition boundary. Specifically, L2P exhibits a higher occurrence of misclassification on states compared to objects, which can be attributed to the disproportionate emphasis placed on objects, diminishing the model's focus on states. This experiment proves that CompILer effectively enhances the model's attention towards states, resulting in an improved overall fine-grained perception capability.

Green Shoes

Green Shoes

Brown Shoes

Yellow Dress

Black Shoes

Black Shoes

Blue Shoes

Blue Shoes

White Shoes

Silver Skirt

Silver Skirt

Pink Skirt

Blue Shirt

Blue Shirt

Blue Shorts

Black Suit

Black Suit

White Shoes

Green Suit

Synthetic Sandals

Suede Ankle

Suede Ankle Leather Ankle

Rubber Sandals

Rubber Sandals Faux Leather Sandals

Faux Leather Sandals

Faux Leather Sandals Full-grain Sandals

Canvas Sneakers

Canvas Sneakers

Synthetic Sneakers

Leather Ankle

Leather Ankle

Faux Leather Ankle

Wool Slippers

Wool Slippers Suede Slippers

Leather Flats

Leather Flats

Canvas Boat

Rubber Sandals

Rubber Sandals

Synthetic Sandals

Leather Heels

Leather Oxfords

Suede Heels

Suede Heels Patent Leather Heels

Full-grain Leather Loafers

Full-grain Leather Loafers Leather Loafers

Synthetic Mid-Calf Boots Nylon Mid-Calf Boots

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction effectively summarize the paper's contributions and scope, outlining the key claims and objectives.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in both the paper and the appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and a complete proof for each theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide comprehensive implementation details and code to ensure the experiments are reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides open access to the data and code, along with sufficient instructions to faithfully reproduce the experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the training and test details, including but not limited to: data splits, hyperparameters, how they were chosen, type of optimizer, etc.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper appropriately reports the relevant information about the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computer resources in implementation details section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper aligns with the NeurIPS Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss potential societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credited the creators and original owners of assets used our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets introduced in the paper are well documented and the documentation is provided alongside the assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel setting of incremental learning, named Compositional Incremental Learning (composition-IL). This setting differs from existing ones as it involves recognizing not only new objects (e.g., a shirt), but also their states (e.g., red) and the resulting compositions (e.g., a red shirt). To address this novel setting, the authors present a prompting-based approach that leverages a pre-trained transformer network to learn the tasks incrementally, thus sharing some similarities with existing methods such as L2P, Dual-Prompt, and CODA-Prompt. Additionally, the authors devise a three-way retrieval mechanism, devoting tailored prompt pools to state, object, and composition-level representation learning. In the experimental section, the authors assess several aspects: the comparison with the state of the art, the benefits of the three-way prompting strategy, the impact of injecting object-level information into the state-level prompt, and the contributions of each involved loss function.\n\nSTRENGTHS:\n1) The paper is clear and fluent. Both the dataset and the model are clearly described.\n2) Novelty. To the best of my knowledge, the experimental setting is novel and deserving. \n3) While existing prompting strategies often organize the prompt pool in a flat structure, this paper advances the field by leveraging on two concepts: multiple prompts pool related to different concepts/tasks, and possible dependencies between prompts of different concepts.\n\nWEAKNESSES:\n**Many hyperparameters/tuning complexity (major)**. From a methodological perspective, the model incorporates several loss terms, each requiring a tailored balancing hyperparameter. Additionally, other factors such as the length of each prompt and the size of the prompt pool must be considered. In truth, the authors report the chosen configuration in their experiments in the second paragraph of Sec. 5.2. As can be observed, there are many hyperparameters, which could complicate the model’s application in real-life scenarios. In this respect, the authors should provide a simplified (yet still reliable) version of their approach, pruning some loss terms and configurations that have a negligible impact on results. Indeed, by examining the results of the ablation studies, it appears that there is room for simplifying the training stage.\nStatistical significance (major). Concerning Tables 3 and 4, the results achieved by the final configuration of the Compiler are sometimes very close to the simpler, ablated versions. For example, consider Table 4(a) where 'no injection' is close to 'O->S', and Table 3 where 'C+S' is close to 'C+S+O'. In these cases, it is difficult to conclude whether the benefits are due to effective technical advances or should be ascribed to statistical noise. In this respect, I strongly encourage the authors to perform multiple runs for each experiment and average the final results. This approach would render the comparisons more significant.\n\n**More rehearsal baseline (major)**. I would suggest including in the experimental section the results for DER++ [b] and ER-ACE [c], two straightforward baselines based on rehearsal. As the main contribution of this paper concerns the introduction of a new experimental setting, I believe it is important to provide an extensive evaluation of the existing literature. Moreover, the authors should compare their approach with SLCA [d], a recently introduced baseline for the continual fine-tuning of pre-trained models.\nRelation with existing works (major). There is a recent research line on multi-label incremental learning; see [a], one of the most cited papers in the field. In this respect, how does the proposed setting deviate from the multi-label scenario? To justify the introduction of a novel setting, the authors should discuss potential overlaps with existing benchmarks more thoroughly. From a more methodological perspective, instead, I would encourage the authors to provide more references while introducing the learning objectives used in their approach. For example, CODA-Prompt devises orthogonality constraints between prompts, sharing strong similarities with the objectives outlined by Eq. 2 and 3. Moreover, aligning the query and the selected prompts is common in prompting-based approaches (see L2P). \n\n**Lacking ablation studies (minor)**. In Tab. 3, it would be important to see the results of single-pool prompt learning, especially for ‘S’ and ‘O’ (‘C’ has instead been already provided). Indeed, I would guess that a simple strategy based only on object-level prompts would be enough to reach satisfactory performance. \n\n**Clarification requested (minor)**. In Sec. 4.2, the authors discuss an attention-based mechanism to inject object-level cues into the state-level prompt. In this respect, they use three learnable projections, indicated by W_Q, W_K, and W_V. However, as these transformations are continuously updated while tasks progress, did the authors consider that these projections could suffer from catastrophic forgetting? It seems that there are no explicit countermeasures against forgetting regarding these variables, which could hinder the efficacy of the retrieval stage.\n\n**Formatting issue (minor)**. Table 4 appears before Table 3.\n\n[a] Kim, C. D., Jeong, J., & Kim, G. (2020). Imbalanced continual learning with partitioning reservoir sampling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16 (pp. 411-428). Springer International Publishing. \n\n[b] Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020). Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33, 15920-15930.\n\n[c] Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., & Belilovsky, E. (2021). New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025.\n\n[d] Zhang, G., Wang, L., Kang, G., Chen, L., & Wei, Y. (2023). Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19148-19158).\n\nQUESTIONS:\nI have no questions.\n\nLIMITATIONS:\nThe limitations are briefly discussed in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new task called Compositional Incremental Learning (composition-IL).\nThis new task extends the existing class incremental learning to a more fine-grained scenario for more realistic applications.\nThis work formulates and designs a new composition-IL benchmark based on Clothing16K and UT-Zappos50K datasets.\nTechnically, the authors propose several novel strategies based on prompting on pretrained ViT.\nSpecifically, this work suggests using three prompt pools for states, objects, and their compositions respectively.\nTo overcome the gap between the pretrained task and the composition task, the object prompts are further injected to guide the selection of state prompts.\nThe authors also suggest a learnable generalized-mean prompt fusion scheme for prompt reducing.\nExtensive experiments are provided to evaluate the models.\n\nSTRENGTHS:\n1. This paper has done a great job in presentation, with clear wiring and figuring, enabling the fast understanding of reviewers familiar with continual learning.\n2. The contribution of proposing novel and more realistic tasks is always welcomed. And this work has done some essential initial jobs such as creating a benchmark for composition-IL.\n3. The proposed method looks convincing to me.\n\nWEAKNESSES:\n1. The main concern for me is the proposed composition-IL itself. From the experimental results (Tables 1 and 2), it seems that the previous prompt-based works can also handle this task with comparable performance. So my double is that if this work overclaims the difficulty of composition-IL? As someone working in the continual learning field, I think new settings and tasks are always welcomed only if the new task is challenging and practical enough.\n2. The effectiveness of the proposed components. As shown in Table 3, I think the performance gap between C+S+O and C+S/C+O is small, so the effectiveness of multi-pool prompt learning is not that significant. Similar results are also in Table 4. In Table 4(a), there is only a 0.32% improvement in HM when introducing the object injection. In Table 4(b), there is also only a 0.3% improvement when using the GeM instead of the naive mean pooling.\n\nQUESTIONS:\nPlease see the 'Weakness' section for detailed questions.\nIn conclusion, I expect the authors to provide their thoughts about the necessity of the proposed composition-IL setting, as well as some explanations of the small performance gap when equipped with the proposed modules.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel task termed Compositional Incremental Learning (composition-IL), which aims to enable models to recognize a variety of state-object compositions incrementally. The authors propose a new model called CompILer, which employs multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to overcome the challenges in this task. The study utilizes two newly tailored datasets, Split-Clothing and Split-UT-Zappos, and demonstrates state-of-the-art performance through extensive experiments.\n\nSTRENGTHS:\n1. The introduction of composition-IL addresses a critical gap in incremental learning by focusing on the recognition of state-object compositions, which traditional class-IL and blur-IL approaches overlook\n2.  The proposed CompILer model is well-conceived, leveraging multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to enhance the learning of compositions.\n\nWEAKNESSES:\n1. The paper shows that CompILer only slightly outperforms LGCL and CODA-Prompt on the Split-Clothing and Split-UT-Zappos datasets. However, the authors do not detailedly analyze why LGCL and CODA-Prompt perform well in these tasks or why CompILer's improvements are limited. A detailed comparative analysis is needed.\n\n2. Lack of analysis on model efficiency, especially on the size of learnt prompts comparing to current prompt-based methods\n\n3. The paper lacks a systematic analysis of the hyperparameters $\\lambda_1$ and $\\lambda_2$ in Equation 9.\n\nQUESTIONS:\nModel Efficiency: How does the size of the learnt prompts in CompILer compare to current prompt-based methods? Please provide a detailed analysis of model efficiency.\n\nHyperparameter Analysis: Can you provide a systematic analysis of the hyperparameters $\\lambda_1$ and $\\lambda_2$ in Equation 9? How do variations in these values impact the model's performance?\n\nPerformance Comparison: Why do LGCL and CODA-Prompt perform well on the Split-Clothing and Split-UT-Zappos tasks, and what factors limit the performance improvements of CompILer over these methods? A detailed comparative analysis would be beneficial.\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their work, particularly concerning the performance challenges on the Split-UT-Zappos dataset due to long-tail distributions and the difficulty in distinguishing state classes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose compositional Incremental Learning to enable models to recognize state-object compositions incrementally. The paper provides two tailored datasets for composition-IL by modifying two existing datasets in the fashion domain. The paper propose a new prompt-based model comprising of multi-pool prompt learning, object-injected state prompting and generalized-mean prompt fusion. The method achieves competitive performance on the two proposed datasets.\n\nSTRENGTHS:\n1. The paper discusses a new direction which aims to exploit state primitives of objects for incremental classification. The authors propose a new incremental setting where objects or states can reappear in new classes in new tasks.\n2. The paper provides two curated datasets in the fashion domain to study composition-IL.\n3. The paper propose a new prompt-based model for composition-IL. It is nice to report HM accuracy for better evaluation.\n\nWEAKNESSES:\nI have major concerns with the experimental part of the paper.\n1. Hyperparameters: It is very concerning that six hyperparameters are tuned on the testing set. The hyperparameter values even differ for different task splits on the same dataset. It is weird that the authors use 25 epochs for one dataset, while for the other dataset, 10 epochs for 5-task setting while 3 epochs for the 10 task setting. Even different learning rates are used for different settings on same datasets. How are these decided? It looks like everything is optimized for the test sets in all settings. This is not a fair way of doing experiments. It is acceptable if the authors fine-tune the model for one dataset and use the same parameters for all settings across different datasets (I think this is commonly done in continual learning domain). How can the proposed method be useful/practical if it needs to fine-tune so many hyper parameters on every setting using the test set to get good results?\n2. Lack of experiments with random seeds: The experiments are conducted using a single random seed. The proposed method has improvements of 1% or even less in some settings. It is standard practice in CL to report results with multiple random seeds for fair evaluation and to establish the robustness of the model.\n3. Competitive recent baselines like HiDe-prompt [1] are not included in the comparison.\n4. Simple methods like NCM classifier [2] and Mahalanobis-distance based classifier [3] outperforms prompt methods like L2P on several datasets with first-task adaptation and no training in new tasks (using frozen model after the first task and doing continual evaluation). It would be interesting to see how these strong baseline methods work in the proposed settings.\n\n[1] Wang, Liyuan, et al. \"Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[2] Paul Janson,  et al, A simple baseline that questions the use of pretrained models in continual learning. arXiv preprint arXiv:2210.04428, 2022. \n\n[3] Dipam Goswami, et al, Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.\n\nQUESTIONS:\n1. I am curious what is the difference between object-state pairs and a group of concepts describing a class? Like why limit the concepts to just object and state, there can be more concepts attached to a class. So, a class can also be described as a group of concepts and classes can then be learned incrementally with overlapping concepts from old classes.\n\nLIMITATIONS:\nLimitations of this work is not explicitly addressed in paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}