Not Just Object, But State: Compositional Incremental Learning without Forgetting

Authors: yanyi  zhang
binglin  qiu
qi  jia
yu  liu
ran  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:28:46

Abstract: Most incremental learners excessively prioritize object classes while neglecting various kinds of states (e.g. color and material) attached to the objects. As a result, they are limited in the ability to model state-object compositionality accurately. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL), which enables the model to recognize a variety of state-object compositions in an incremental learning fashion. Since the lack of suitable datasets, we re-organize two existing datasets and make them tailored for composition-IL. Then, we propose a prompt-based Composition Incremental Learner (CompILer), to overcome the ambiguous composition boundary. Specifically, we exploit multi-pool prompt learning, and ensure the inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting which injects object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-art performance achieved by CompILer. Code and datasets are available at: https://github.com/Yanyi-Zhang/CompILer.

Main paper: 
 introduction: 
 Class Incremental Learning (class-IL) [37, 22, 16, 10] gathers increasing attention due to its ability to make the models learn new tasks rapidly, without forgetting previously acquired knowledge. Yet, traditional class-IL sets a strict limit on the old classes such that they should not recur in newly incoming tasks. To break such a strict limitation, recent studies develop a new setting mostly called Blurry Incremental Learning (blur-IL) [24, 11] , where the incremental sessions allow the recurrence of previous classes, resulting in a more realistic and flexible scenario. Despite such empirical progresses on incremental learning, they aim to improve object classification only, while overlooking fine-grained states attached to the objects. For instance, analyzing how the clothing styles (akin to states) have changed over time is important for forecasting the future trends that will emerge.To simultaneously model objects and their states, some efforts are dedicated to Compositional Learning whose aim is how to equip the models with compositionality [4, 31, 45] . The core of compositional learning lies in the structure of class labels, which conceptualizes a state-object pair (e.g. "Brown Pants" and "Yellow Dress") as a whole, rather than a lonely object label. In this way, the model can dissect and reassemble learned knowledge, achieving a more fine-grained understanding about the objects. However, existing works are mainly focused on zero-shot generalization from seen Different from them, the classes in composition-IL involve state-object compositions apart from the object classes. Besides, the compositions do not reoccur, but the primitives (states or objects) may randomly reappear across incremental sessions.compositions to unseen ones [26, 27, 5, 15] , whereas none of them consider the challenging fact that the model must deal with a significantly larger number of composition classes than object classes. As a result, it is hardly feasible to learn all compositions by training the model once.To remedy the limitations inherent in incremental learning and compositional learning, we conceive a novel task named Compositional Incremental Learning (composition-IL), enabling the model to continually learn new state-object compositions in an incremental fashion. As compared in Fig. 1 , we can see that composition-IL integrates the characteristics of class-IL and blur-IL. Although the composition classes are disjoint across incremental tasks, the primitive classes (i.e. objects and states) encountered in old tasks are allowed to reappear in new tasks. Unfortunately, existing incremental learning approaches are challenged by such a compositional scenario, because their models excessively prioritize the object primitives while neglecting the state primitives. Consequently, the compositions with the same object but with different states become ambiguous and indistinguishable.To tackle the problem, we propose a rehearsal-free and prompt-based Compositional Incremental Learner (CompILer). Specifically, our model comprises of three primary components: multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion. Firstly, we construct three prompt pools for learning the states, objects and compositions individually. Upon that, we add extra restrictions to regularize the inter-pool prompt discrepancy and intra-pool prompt diversity. This multi-pool prompt learning paradigm strengthens the fine-grained understanding and reasoning towards primitive concepts and their compositions. In addition, as the state classes are more difficult to distinguish than the object ones, we propose object-injected state prompting which incorporates object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean fusion manner, which helps to adaptively eliminate irrelevant information learned in the prompts. Last but not least, we also leverage symmetric cross-entropy loss to alleviate the impact of noisy data during training.In summary, the main contributions in this work are encapsulated as follows: (1) We devise a new task coined compositional incremental learning (composition-IL). It enables learning fine-grained state-object compositions continually while the isolated primitive concepts can randomly recur in incremental tasks. (2) To address the lack of datasets, we re-organize two existing datasets such that they are tailored specifically for composition-IL. For the two new datasets (Split-Clothing and Split-UT-Zappos), we split them into 5 and 10 incremental tasks for evaluating the methods. (3) We propose a novel learning-to-prompt model for composition-IL, namely CompILer. Our stateof-the-art results on Split-Clothing and Split-UT-Zappos validate the effectiveness of CompILer for incrementally learning new compositions without forgetting old ones. 
related work: 
 Incremental Learning. The approaches to addressing catastrophic forgetting for incremental learning can be broadly grouped into four categories: regularization based methods [6, 39] aim to protect influential weights of old experiences from updating; knowledge distillation based methods [16, 34] distill knowledge from the model trained on the previous tasks and adapt it to new tasks; rehearsal based methods [32, 48, 20] require a memory buffer to store some old data, so as to make the network remember previous tasks; parameter isolation methods allocates different model parameters to each task, to prevent any possible interference. Different from the methods, L2P [43] proposes an innovative learning-to-prompt paradigm, which incorporates plasticity and stability through adapting a set of learnable prompt tokens on top of a frozen pre-trained backbone. Inspired by L2P [43] , more recent works [42, 36, 2, 28] take full advantage of various prompt tuning strategies, achieving new state-of-the-art performance for incremental learning. However, such methods take into account object classes solely, while neglecting various kinds of state classes associated with the objects. To this end, our work proposes compositional incremental learning with the purpose to continually identifying the composition classes of state-object pairs. Note that, Liao, et al [18] conduct an initial study toward the compositionality in incremental learning, whereas their attention is on the composition of multiple object classes (e.g. "Car" and "Person") in one image, rather than the state-object compositions in this work Compositional Learning. A major line of compositional learning research focuses on Compositional Zero-Shot Learning (CZSL) [23] , which aims to infer unseen state-object compositions by acquiring knowledge from seen ones. Subsequent approaches building upon the CZSL setting further incorporate graph neural networks to model the dependency between primitives and compositions [25] , and employ cosine classifiers to avoid being overly biased toward seen compositions [21] . Other approaches [13, 14, 12] propose training two classifiers to identify states and objects separately. The latest works [19, 3, 38, 8, 47] model both composition and primitives simultaneously, achieving state-of-the-art results. Albeit the numerous attempts made in compositional learning, they fail to consider an incremental learning paradigm given the increasing number of composition classes in open-world scenarios. Besides, directly applying CZSL methods to composition-IL might lead to a stale and decaying performance on forgetting. By contrast, our proposed CompILer markedly bypasses catastrophic forgetting with the help of multi-pool prompt learning. 
preliminaries: 
 In this section, we firstly define the task of composition-IL, and then introduce two datasets we construct for the task, followed by revealing the ambiguous composition boundary problem.For composition-IL, a model sequentially learnsN tasks T = {T 1 , T 2 , • • • T N } corresponding to a set of composition classes C = {C 1 , C 2 , • • • C N }.We note that the composition classes between incremental tasks are always disjoint, which means C i ∩ C j = ∅ for any i ̸ = j. Different from the composition classes, the primitive classes are allowed to recur in different tasks. That means it allows the tasks to share some primitive concepts of objects and states. Therefore, we can define the set of all state and object classes withS = {s 1 , s 2 , • • • , s n } and O = {o 1 , o 2 , • • • , o m }, respectively.Given each image x, it has a composition label c which is constructed with a state label s and an object label o, i.e. c =< s, o >, where c ∈ C, s ∈ S and o ∈ O. We take the example of "red shirt", where "red" is denoted with s, "shirt" corresponds to o, and "red shirt" is expressed with c. For the compositions with the same object but with different states, our CompILer achieves more distinguishable boundaries than the L2P baseline.As there are no existing datasets suitable for composition-IL, we re-organize the data in Cloth-ing16K [46] and UT-Zappos50K [44] , and construct two new datasets tailored for composition-IL, namely Split-Clothing and Split-UT-Zappos. To be more specific, we firstly sort the composition classes based on the number of their images, and then select the foremost 35 compositions from Cloth-ing16K and the top 80 from UT-Zappos50K, so as to construct Split-Clothing and Split-UT-Zappos, respectively. In this way, Split-Clothing encompasses 9 states and 8 objects while Split-UT-Zappos consists of 15 states and 12 objects in total. For Split-Clothing, we randomly partition the compositions into 5 tasks. Regarding Split-UT-Zappos, the compositions are sorted by count and are evenly divided into 5 and 10 tasks. The image distribution for each task is shown in Fig. 2 . Note that, we elaborate more details on both datasets in the following technical appendix.The main stumbling block in composition-IL is the ambiguous composition boundary. Although the composition label consists of two primitives (i.e. object and state), we note that the model excessively prioritizes the object primitive while neglecting the state primitive. Consequently, the compositions with the same object but with different states become ambiguous and indistinguishable. To prove that, we apply L2P [43] to composition-IL, whereas it is challenged by significant ambiguities in composition classification. As illustrated in Fig. 3 (a), the t-SNE visualization showcases the entanglement among the compositions like "white dress", "black dress" and "blue dress". We conjecture that this ambiguous problem tends to become more severe when more tasks are arriving incrementally. To address it, we propose a new model namely CompILer, which disentangles compositions and primitives via a multi-pool prompt learning. Advantageously, our method promotes the learning on the states and establishes clearer composition boundaries, as shown in Fig. 3 (b ). 
methodology: 
 Overview. We leverage the learning-to-prompt paradigm [43] and develop a novel compositional incremental learner (CompILer) tailored specifically for composition-IL. As depicted in Fig. 4 , CompILer comprises three primary components: multi-pool prompt learning, object-guided state prompting, and generalized-mean prompt fusion. Firstly, we initialize three prompt pools dedicated to learning and storing visual information related to states, objects and their compositions. In order to differentiate the knowledge learned across and within prompt pools, we define inter-pool discrepant loss and intra-pool diversified loss jointly. We then employ object prompts to guide the selection of state prompts, thereby improving the state representation learning. Moreover, we utilize a generalizedmean fusion to integrate the selected prompts in a learnable manner. Ultimately, we optimize the classification objective with symmetric cross-entropy loss, to alleviate the effect of noisy data.The learning-to-prompt paradigm [43, 35] , especially suitable for large pre-trained backbones, has opened up a new path for incremental learning. It has proven to incorporate plasticity and stabilityℒ !"# $ Object Prompt Pool State Prompt Pool Composition Prompt PoolEmbedding Layer 𝑓 ! (#)Transformer Encoder 𝑓 " (#) The multi-pool prompt learning mechanism captures information related to states, objects, and their compositions, each through a dedicated pool. The object-injected state prompting utilizes the object prompt to promote the state representation learning. Moreover, the generalized-mean prompt fusion is used to prioritize the useful prompts and diminish the irrelevant ones.𝜑 𝑐 (#) 𝜑 𝑠 (#) 𝜑 𝑜 (#) ℒ !"# ( ℒ !"# ) ❄ 🔥 ••• Selection 𝑲 # Selection 𝑲 $ Selection 𝑲 % GeM Fusion Object-injected State Prompting Feature Extractor 𝑓($) ❄ ❄ GeM Fusion GeM Fusion ❄ {𝑃 # $ ! } &'( ) {𝑃 $ $ ! } &'( ) {𝑃 % $ ! } &'( ) 𝑷 # 𝑷 $ 𝑷 % Prompt Image Token Prompt-Key Pair 𝜑 𝜔 (#) Classifier Learnable ❄ Frozen Composition State Object Yellow Dressbetter through adapting a set of learnable tokens in a prompt pool to a frozen pre-trained backbone. Nevertheless, existing prompt-based approaches are initially designed for class-IL, thereby building a single prompt pool for object classification solely. when dealing with state-object composition classification, they tend to excessively prioritize the object primitive while neglecting the state primitive. To this end, we propose to construct three discrepant and diversified prompt pools P s , P o and P c , which serve to learn visual information related to states, objects and their compositions, respectively. Besides, each pool is associated with a set of learnable keys K ω for query-key prompt selection. The three prompt pools and their keys are defined as:EQUATIONwhere P i ω ∈ R L×D is a single prompt with token length L and embedding dimension D. K i ω ∈ R D , the key of P i ω , is a learnable token with the same size. M is the number of prompts in each pool. One important concern in such multi-pool prompt learning is how to enrich the prompts with the avoidance of identical pools. To achieve it, we consider integrating inter-pool prompt discrepancy and intra-pool prompt diversity jointly. On the one hand, the inter-pool prompts should be discrepant as the visual information about states, objects, and compositions should be different. One the other hand, within each pool, the intra-pool prompts should be diversified so to capture more comprehensive features from all the classes.In practice, we formulate a unified objective to regularize both inter-pool discrepancy and intra-pool diversity, by leveraging a simple and effective directional decoupled loss used in [17] . The directional decoupled (dd) loss between any two pools (e.g. P i and P j ) is formulated as:EQUATIONEQUATIONwhere θ nm measures the angle between any two prompts, n and m; ϵ is a scalar to avoid division by zero. Note that, L (i,j)dd encourages the angles between each prompt to be at least θ thre degrees. Since (i, j) is unordered Cartesian product of ω, i.e. (i, j) ∈ {(i, j) | i ∈ ω ∧ j ∈ ω}, the inter-pool prompt discrepancy loss for the three pools can be expressed withL inter = L (s,o) dd + L (s,c) dd + L (o,c)dd , and the intra-pool prompt diversity loss becomesL intra = L (s,s) dd + L (o,o) dd + L (c,c)dd . As opposed to L inter , L intra computes the angle between any two prompts within the same pool. Thus, it contains the case when n = m, for which we set θ thre -θ nm = 0.••• ••• ••• ••• ••• Query Feature 𝑞(𝑥) Q K V Fused Object Prompt 𝑷 ! Injected Query Feature 𝑞 " (𝑥) ••• Token 1 Token 2Token NFigure 5 : Architecture of object-injected state prompting. Query feature serves as Q, while fused object prompt serves as both K and V.Akin to the query-key matching mechanism in other work [43, 42, 40] , we utilize a fixed feature extractor f (•) to obtain a query feature q(x) = f (x)[0, :], determining which prompts in the pool to be selected. However, pre-trained backbones are typically trained for object classification, thus under-performing for state representation learning. In addition, it is more difficult to predict the state classes due to their more abstract and fine-grained characteristics.To tackle this problem, we strategically inject object prompts to guide the selection of state prompts. Intuitively, once we have learned knowledge about the object class, it may be easier to predict the correct state class and avoid mistaken results. For instance, given an object is "heels", we can expect that the corresponding state is unlikely to be "canvas" or "plastic". To summarize, we select object and composition prompts in each pool based on the original query feature, which means q o (x) = q c (x) = q(x); but for the selection of state prompts, we propose object-injected state prompting to ameliorate the query feature as shown in Fig. 5 . Specifically, we employ the fused object prompt P o (see Sec. 4.3) to perform cross attention on the query feature q(x), resulting in object-injected query feature q s (x) for the state prompt selection:EQUATIONwhere W Q , W K and W V are learnable projections. To establish alignment between the query and the selected prompts, we optimize a surrogate loss for state, object and composition prompting jointly:EQUATIONwhere COS(•, •) denotes cosine similarity, K ω represents the subset of top-k keys selected from K ω , and{s i } k i=1 is a subset of top-k indices from [1, M ] (prompt number).Despite the simplicity of the object-injected state prompting, it facilitates more judicious prompt selection within the state prompt pool, alleviating the hurdles posed by state learning.After obtaining the selected top-k prompts {P si ω } k i=1 , the next step is fusing these prompts into a single prompt. It is general to utilize a simple mean pooling whereas it overlooks the relative importance of each prompt. Besides, when the prompts contain information that is unrelated or contradictory to current task, it is critical to strengthen useful prompts and eliminate irrelevant ones. To this end, we draw inspiration from generalized-mean pooling [29] and exploit generalized-mean (GeM) prompt fusion which is given by:EQUATIONwhere η is a learnable parameter. When η = 1, GeM becomes mean pooling; as η approaches infinity (η → ∞), it converges to max pooling. By taking over mean and max pooling, GeM learns to achieve an optimal fusion, mitigating the influence of irrelevant information present in the prompts.Classification Objective. We prepend three fused prompts (i.e. P s , P o and P c ) with x e , which is the output from a ViT embedding layer f e (•). The extended token sequence is x p = [P c ; P s ; Then, we feed x p to a transformer encoder layer f r (•) and achieve P r s , P r o and P r c for classifying state, object and composition classes, respectively. We estimate the probability via a classifier φ ω (•):P o ; x e ].p(ω | x) = φ ω (P r ω ).For each image x, we denote its ground-truth distribution over labels with q(ω | x). When ω is consistent with the ground truth, then q(ω | x) = 1; otherwise, q(ω | x) = 0. As a result, the cross entropy (CE) loss used for classification objective is:EQUATIONwhere Ω represents the number of classes. However, the model optimized with a standard CE loss is easily affected by noisy samples during training. Instead, we advocate using a symmetric cross entropy loss (SCE) [41] , which incorporates an additional term called reverse cross entropy (RCE), to mitigate the impact of noisy data. Contrary to CE, the formula for RCE loss is defined as:EQUATIONThen, the SCE loss combines two loss terms by L ω SCE = L ω CE + αL ω RCE , where α is a hyperparameter that controls the weight of the RCE term. As a result, the whole SCE loss becomesL SCE = L c SCE + β(L s SCE + L o SCE ), where β adjusts the weights between primitives and compositions. Total Loss. The total loss for training the whole CompILer model is:EQUATIONwhere λ 1 , λ 2 , λ 3 are hyper-parameters balancing different terms.Inference. During inference, we incorporate the primitive probabilities to aid the composition probability. Hence, the final probability for composition classification is expressed with:EQUATIONwhere µ adjusts the probabilities. 
datasets and metrics: 
 We conduct experiments on two newly split datasets: Split-Clothing and Split-UT-Zappos as elucidated in Section 3.2. We assess the overall performance on compositions using Average Accuracy (Avg Acc) and Forgetting (FTT). A higher Avg Acc signifies stronger recognition abilities, while a lower FTT indicates improved resilience against forgetting. Additionally, we provide individual Average Accuracy scores on states and objects, denoted as State and Object for simplicity. These metrics imply the ability to recognize fine-grained primitives. Furthermore, we calculate the Harmonic Mean (HM) between State and Object, i.e. HM = 2 × (State×Object) (State+Object) . We provide more emphasis to Avg Acc and HM due to their more comprehensive assessment. Avg Acc encompasses the plasticity and stability [33, 2] and HM provides a holistic evaluation on both state and object. For a fair comparison with previous works [43, 42, 2, 33] , we also employ ViT B/16 [1] pretrained on the Ima-geNet 1K dataset as the feature extractor and backbone. For multi-pool prompt learning, the size of each pool is set to 20, and each prompt has 5 tokens. We select top-5 prompts from each pool and generate a fused prompt.During training, we utilize the Adam optimizer [9] with a batch size of 16. The whole CompILer undergoes training for 25 epochs on the Split-Clothing, for 10 epochs on the 5-task Split-UT-Zappos, and for 3 epochs on the 10-task Split-UT-Zappos. For the Split-Clothing and the 10-task Split-UT-Zappos, we set the learning rate to 0.03, while we use a learning rate of 0.02 for the 5-task Split-UT-Zappos. Note that, for all the methods, their results are averaged over three runs with the corresponding standard deviations reported to mitigate the influence of random factors.As there are a few hyper-parameters in the model, we conduct a rigorous tuning on them. For instance, we set θ thre to π 2 for all settings. For Split-Clothing, the loss weights λ 1 and λ 3 are set to 0.1; λ 2 is set to 10 -7 ; α and β for SCE loss are 0.006 and 0.3, and the parameter µ during inference is 0.5. For 5-task Split-UT-Zappos, λ 1 , λ 2 , λ 3 , α, β and µ are set to 1.0, 3 × 10 -6 , 0.7, 0.01, 0.7 and 0.02, respectively. For 10-task Split-UT-Zappos, λ 1 , λ 2 , λ 3 , α, β and µ are set to 0.5, 10 -7 , 0.1, 0.05, 0.4 and 0.03. We elaborate more details on hyper-parameter analysis in the appendix.To demonstrate the effectiveness of the proposed method, we compare CompILer with state-of-the-art incremental learning methods, including prompt-free approaches [10, 16, 32] and prompt-based methods [43, 42, 33, 7] . All the methods are rehearsal-free except iCaRL [32] . Note that, due to LGCL [7] relying on CLIP [30] to achieve language guidance at the task level, it is limited by the length of class names per task. Thereby, LGCL fails to operate on the 5-task Split-UT-Zappos since the total length of class names exceeds the limitation.To streamline our CompILer, we further implement a simplified version named Sim-CompILer and report its results. Sim-CompILer is optimized using cross entropy loss and is comprised solely of multi-pool prompt learning and generalized-mean prompt fusion. In other words, we exclude the object-injected prompting, directional decoupled loss, and reverse cross entropy loss, resulting in a large reduction of hyperparameters to only β, λ 3 , and µ.The compared results on Avg Acc and FTT are reported in Table 1 . Overall, CompILer consistently outperforms all competitors on Avg Acc by a significant margin. For FTT scores, CompILer excels previous methods with 0.32% on the 5-task Split-Clothing and with 0.14% on the 5-task Split-UT-Zappos, while falling behind Dual-Prompt [42] and LGCL [7] for the 10-task Split-UT-Zappos. We notice that, the main reason is these methods sacrifice more plasticity for lower forgetting rates. Besides, the number of model parameters in these methods dynamically increases along with more incremental tasks arriving, whereas our CompILer does not rely on imposing task-specific parameters to reduce the forgetting.We also report the primitives accuracy and their HM in Table 2 and Table 3 . Likewise, our method surpasses other methods considerably in terms of State and HM. Interestingly, the prompt-free methods [16, 10, 32] achieve higher accuracy in state prediction than object prediction for Split-Clothing, which is contrary to other results. This is because the states in Split-Clothing are colorrelated descriptions, which are easier to capture with the help of parameter fine-tuning. The promptbased methods do not exhibit this phenomenon because their pre-trained backbones are initially trained for object classification, and are frozen across incremental sessions. As the performance improvements are mainly attributed to the accuracy of state recognition, it suggests that our model enhances the understanding on fine-grained compositionality. Effect of multi-pool prompt learning. This experiment aims to delineate the contribution of three pools in Com-pILer. We firstly implement a baseline model with composition prompt pool only. Building upon the baseline, we develop two additional models, which incorporate either object or state prompt pool. As reported in Table 4 , the inclusion of primitive prompt pool yields consistent gains over the baseline. Furthermore, the best results are achieved when the model integrates all three pools simultaneously. This experiment signifies the significant necessity of exploiting multiple prompt pools for composition-IL.Effect of object-injected state prompting. To provide insights into object-injected state prompting, we compare three models: None (vanilla model), S→O (state-injected object prompting) and O→S (object-injected state prompting). As shown in Table 5a , compared to the None model, the S→O exhibits a decrease in all metrics, implying that state prompts may interfere with the selection of object prompts. On the contrary, O→S outperforms the None model as we expect. This phenomenon validates our motivation that state recognition is harder than object recognition, and thereby the former cannot help the latter easily. Yet, it is a promising direction for future research.Effect of generalized-mean prompt fusion. This study aims to study the impact of prompt fusion on CompILer. As shown in Table 5b , GeM performs better than both max and mean pooling across various metrics. It validates the benefit of GeM on mitigating irrelevant information in the selected prompts. as it may hamper the model's attention on image tokens.Effect of loss functions. As shown in Table 6 , we investigate the influence of loss functions used in our model, including directional decoupled loss (L inter and L intra ) and symmetric cross entropy loss (L CE and L RCE ). The baseline model (the first row) includes all modules but is trained by cross entropy loss only. By adding the RCE loss, the model is equivalent to training with the SCE loss, which help to improve the robustness to noisy labels. The use of either L inter or L intra improves the performance on both datasets, and synchronously applying them witnesses all-around improvements compared to the baseline. Eventually, we achieve the best results when combing all the loss terms during training.In order to study the repeatability characteristic in composition-IL, we exhibit more results on Split-Clothing in Fig. 6 : in (a), it shows a decreasing trend in composition accuracy along with the introduction of new tasks; however, the green rectangles in (b) and (c) showcase that the accuracy occasionally increases as more tasks are learned. We conjecture the reason is mostly attributed to the re-occurrence of primitive concepts. This forward transfer is critical for incremental learners. We compare the composition predictions between CompILer and L2P [43] in Fig. 6 (d) . CompILer predicts all the images correctly, while L2P makes some mistakes, particularly for state labels. This limitation arises from an excessive focus on the dominant object primitive, while weakening the attention toward state primitive. Fortunately, CompILer relieves the bias toward object classes, and enhances the perception on state classes. 
conclusion: 
 In this paper, we have proposed a novel task coined compositional incremental learning (compostion-IL), which is stumbled by ambiguous composition boundary. To tackle it, we develop a learningto-prompt model, namely CompILer. Our model exploits multi-pool prompt learning to model composition and primitive concepts, object-injected state prompting to improve the selection of state prompts, and generalized-mean prompt fusion to eliminate irrelevant information. Extensive experiments on two tailored datasets show that CompILer achieves state-of-the-art performance. In the future, it is challenging yet potential to consider reasoning multiple state classes per object.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide all the training and test details, including but not limited to: data splits, hyperparameters, how they were chosen, type of optimizer, etc.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper appropriately reports the relevant information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide sufficient information on the computer resources in implementation details section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss potential societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly credited the creators and original owners of assets used our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new assets introduced in the paper are well documented and the documentation is provided alongside the assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In addition to the content in the main paper, this appendix elaborates more details on the datasets, algorithm procedure, empirical analysis, hyper-parameter analysis, and quantitative and qualitative experiments. Meanwhile, please refer to the source code and some data samples included in the supplementary material we submit. (iii) Split-UT-Zappos (10 tasks) We provide more information about two newly tailored datasets (Split-Clothing and Split-UT-Zappos) and extensively discuss the limitations of Split-UT-Zappos. The number of images in each task is shown in Fig. 7 .Split-Clothing is derived from Clothing16K, which is originally used for multi-label classification tasks and consists of 37 state-object compositions and 16,170 images scraped from Google, Bing, and DuckDuckGo. We sorted these compositions by the number of images and selected the top 35 compositions, resulting in a total of 15,915 images in the Split-Clothing dataset. Specifically, the object in Split-Clothing includes various types of garments such as "dresses" and "hoodies", while the state delineates the color of the clothing, such as "silver" and "yellow". We allocate 80% of the dataset for training and the remaining 20% for testing. We split this dataset for 5 incremental tasks.Split-UT-Zappos is a subset of UT-Zappos50K, which is a large shoe dataset consisting of 50,025 catalog images collected from Zappos.com. This dataset is originally created for an online shopping task and we selected a total of 28,497 images to comprise the Split-UT-Zappos. The state classes in Split-UT-Zappos describe different materials (e.g. canvas and leather) and the object classes are related to footwear (e.g. heels and slippers). Consistent with Split-Clothing, we employ 80% of the images for training and 20% for testing. We split this dataset for either 5 or 10 incremental tasks.Limitations of Split-UT-Zappos. It is noteworthy that, even for the upper bound model (i.e. supervised learning with all the data), satisfactory performance remains elusive on Split-UT-Zappos as shown in Table 1 and Table 3 . This deficiency can be attributed to the pronounced long-tail distributions and invisible states inherent in UT-Zappos50K. For example, the composition "Faux Fur Slippers" comprises a mere 25 training images, whereas "Leather Sandals" accounts for 1783. This disproportionate distribution predisposes the severe bias towards the compositions prevalent in the head of the distributions, while inadequately capturing those in the tail. Besides, some states in Split-UT-Zappos lik,e "Leather" vs "Synthetic Leather", are material differences that are not always visible as visual transformations. The deficient state description poses a greater challenge for classification, thereby resulting in suboptimal upper bound results. We contend that tackling the long-tail distribution challenge within Split-UT-Zappos and constructing more balanced datasets conducive to composition-IL represent promising avenues for future research.We summarize the training procedure of the proposed CompILer in Algorithm 1. Perform object-injected state prompting by: q s (x) = CrossAttn(f (x)[0, :], P o ); In Section 4.1, we design multi-pool prompt learning to learn visual representations of state, object, and composition, respectively. By applying inter-pool discrepant loss and intra-pool diversified loss, we ensure discrepancy across pools and diversity within each pool. To validate the effectiveness of the approach, we conduct t-SNE visualization on the prompt pools within the three experiment settings. Figure 8 illustrates that the learned knowledge across the three pools is remarkably discriminative, suggesting that each pool has effectively captured unique features. Meanwhile, the prompts within each pool exhibit a relatively wide dispersion demonstrating that the learned information within each pool is diversified.The detailed operation of Generalized-mean Prompt Fusion is elaborated on Section 4.3. The parameter η in Eq. 6 can be learnable as this operation is differentiable, allowing it to be included in the whole back-propagation process. To prove that, the corresponding derivatives of Eq. 6 are given by:EQUATIONIn this section, we delve into why symmetric cross entropy (SCE) loss can effectively mitigate noisy data in the dataset. We define two distributions, q and p, where q = (k | x) represents the ground truth class distribution for sample x and p = (k | x) represents the predicted class distribution. The relation between the cross entropy H(q, p) and KL-divergence KL(q ∥ p) is expressed as:EQUATIONin which H(q) denotes the entropy of q. From the perspective of KL divergence, the optimization objective is to make the prediction distribution p closely resemble the ground truth distribution q, essentially minimizing their KL divergence. However, in the presence of noise within the dataset, q may not accurately reflect the ground-truth class distribution, whereas p might reveal the real distribution. Therefore, it becomes necessary to consider the reverse direction of KL divergence, i.e. KL(p ∥ q) and construct symmetric KL divergence represented by:EQUATIONBy transferring this idea from KL divergence to cross entropy, the formula for symmetric cross entropy is given by:EQUATIONwhere RCE = H(p, q) is the reverse counterpart of H(q, p), known as reverse cross entropy.We investigate the impact of several hyper-parameters, as depicted in Fig. 9 . Specifically, we explore the influence of weights in the total loss. We present the results of tuning the weight λ 1 , λ 2 , and λ 3 on the Split-UT-Zappos (5 tasks). The average accuracy shown in Fig. 9 (a), (b), and (c) initially exhibits an increasing trend, followed by an overall decreasing trend. CompILer reaches its peak performance when λ 1 = 1.0, λ 2 = 3e -6, and λ 3 = 0.7.Effect of multi-pool prompt learning. In addition to presenting the results of Avg ACC, FTT, and HM on Split-Clothing (5 tasks) for multi-pool prompt learning, we also include the remaining results for State and Object in Table 7 . An intriguing observation is that the model achieves the best results for Object when the object pool is used exclusively, while it reaches the peak performance for State when only the state pool is introduced. This phenomenon suggests that incorporating the primitive prompt pools significantly aids in learning the representations of the corresponding primitives. Furthermore, when the model integrates the state prompt pool alone, not only does the accuracy of state is improved, but the accuracy of object also experiences significant gains, and vice versa. This implies that the learning of one type of primitive representation is crucial for the learning of another. Only when all three pools are integrated does the model achieve state-of-the-art performance in the most important metrics, namely average accuracy and HM.Effect of object-injected state prompting. In this section, we report the remaining results for object-injected state prompting on Split-Clothing (5 tasks) as shown in Table 8 . State-injected object prompting (S→O) introduces interference information for composition learning, resulting in a decline across all metrics expect Object. By contrast, object-injected state prompting (O→S) leads to a significant improvement across all metrics, demonstrating the effectiveness of the proposed method.Effect of generalized-mean prompt fusion. We show the intact results of the ablation study on pooling methods on Split-Clothing (5 tasks) in Fig. 9 . It is obvious that max pooling hits the lowest result, while integrating mean pooling leads to a stable growth. Only deploying Generalized-mean pooling achieves the state-of-the-art results, surging from 84.70 to 89.21 in Avg Acc and plunging to 7.26 in FTT. Consequently, the theory that GeM pooling can diminish the irrelevant information and emphasize the crucial aspects is proved.Effect of loss functions. Eventually, we present supplementary ablation experiments on loss functions in Table 10 and 11. The first row indicates that the model is optimized using cross entropy loss exclusively which achieves the poorest results. Furthermore, integrating either reverse cross entropy loss, inter-pool prompt discrepancy loss, or intra-pool diversity loss contributes to improvements across all metrics. Notably, the effectiveness of reverse cross entropy loss is higher on Split-UT-Zappos (5 tasks) compared to that on Split-Clothing (5 tasks). To conclude, the best performance is achieved when all loss functions are simultaneously employed.More qualitative results are presented to demonstrate the effectiveness of CompILer. We conducted a comparative analysis of the prediction results between CompILer and L2P [43] on Split-Clothing (5 tasks), Split-UT-Zappos (5 tasks) and Split-UT-Zappos (10 tasks). The ground truth is presented in the first row, followed by the prediction results of CompILer and L2P in the second and third rows, respectively. As illustrated in the Fig. 10 , CompILer achieves correct predictions for all the images, while L2P consistently yields wrong classification results due to ambiguous composition boundary. Specifically, L2P exhibits a higher occurrence of misclassification on states compared to objects, which can be attributed to the disproportionate emphasis placed on objects, diminishing the model's focus on states. This experiment proves that CompILer effectively enhances the model's attention towards states, resulting in an improved overall fine-grained perception capability. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: The research conducted in the paper aligns with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: After the rebuttal and the authors-reviewers discussion, all reviewers recommended acceptance of the work. After reading the paper, the reviews, and the discussion, I see no reason to overturn their assessment, as the proposed approach and problem formulation can inspire future research in the intersection between compositionality and continual learning. The authors are encouraged to include and discuss the promised baselines and results (e.g., multiple seed runs, Sim-CompILer, additional baselines from the pdf), and follow x9g2's suggestions on captions and hyperparameters discussion.

Reviews: review: paper_summary: paper_summary: The paper propose compositional Incremental Learning to enable models to recognize state-object compositions incrementally. The paper provides two tailored datasets for composition-IL by modifying two existing datasets in the fashion domain. The paper propose a new prompt-based model comprising of multi-pool prompt learning, object-injected state prompting and generalized-mean prompt fusion. The method achieves competitive performance on the two proposed datasets.
strength_weakness: strength_weakness: I have major concerns with the experimental part of the paper.
1. Hyperparameters: It is very concerning that six hyperparameters are tuned on the testing set. The hyperparameter values even differ for different task splits on the same dataset. It is weird that the authors use 25 epochs for one dataset, while for the other dataset, 10 epochs for 5-task setting while 3 epochs for the 10 task setting. Even different learning rates are used for different settings on same datasets. How are these decided? It looks like everything is optimized for the test sets in all settings. This is not a fair way of doing experiments. It is acceptable if the authors fine-tune the model for one dataset and use the same parameters for all settings across different datasets (I think this is commonly done in continual learning domain). How can the proposed method be useful/practical if it needs to fine-tune so many hyper parameters on every setting using the test set to get good results?
2. Lack of experiments with random seeds: The experiments are conducted using a single random seed. The proposed method has improvements of 1% or even less in some settings. It is standard practice in CL to report results with multiple random seeds for fair evaluation and to establish the robustness of the model.
3. Competitive recent baselines like HiDe-prompt [1] are not included in the comparison.
4. Simple methods like NCM classifier [2] and Mahalanobis-distance based classifier [3] outperforms prompt methods like L2P on several datasets with first-task adaptation and no training in new tasks (using frozen model after the first task and doing continual evaluation). It would be interesting to see how these strong baseline methods work in the proposed settings.

[1] Wang, Liyuan, et al. "Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality." Advances in Neural Information Processing Systems 36 (2023).

[2] Paul Janson,  et al, A simple baseline that questions the use of pretrained models in continual learning. arXiv preprint arXiv:2210.04428, 2022. 

[3] Dipam Goswami, et al, Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
questions: questions: 1. I am curious what is the difference between object-state pairs and a group of concepts describing a class? Like why limit the concepts to just object and state, there can be more concepts attached to a class. So, a class can also be described as a group of concepts and classes can then be learned incrementally with overlapping concepts from old classes.
limitations: limitations: Limitations of this work is not explicitly addressed in paper.
paper_summary: paper_summary: This paper introduces a novel task termed Compositional Incremental Learning (composition-IL), which aims to enable models to recognize a variety of state-object compositions incrementally. The authors propose a new model called CompILer, which employs multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to overcome the challenges in this task. The study utilizes two newly tailored datasets, Split-Clothing and Split-UT-Zappos, and demonstrates state-of-the-art performance through extensive experiments.
strength_weakness: strength_weakness: 1. The paper shows that CompILer only slightly outperforms LGCL and CODA-Prompt on the Split-Clothing and Split-UT-Zappos datasets. However, the authors do not detailedly analyze why LGCL and CODA-Prompt perform well in these tasks or why CompILer's improvements are limited. A detailed comparative analysis is needed.

2. Lack of analysis on model efficiency, especially on the size of learnt prompts comparing to current prompt-based methods

3. The paper lacks a systematic analysis of the hyperparameters $\lambda_1$ and $\lambda_2$ in Equation 9.
questions: questions: Model Efficiency: How does the size of the learnt prompts in CompILer compare to current prompt-based methods? Please provide a detailed analysis of model efficiency.

Hyperparameter Analysis: Can you provide a systematic analysis of the hyperparameters $\lambda_1$ and $\lambda_2$ in Equation 9? How do variations in these values impact the model's performance?

Performance Comparison: Why do LGCL and CODA-Prompt perform well on the Split-Clothing and Split-UT-Zappos tasks, and what factors limit the performance improvements of CompILer over these methods? A detailed comparative analysis would be beneficial.
limitations: limitations: The authors have acknowledged the limitations of their work, particularly concerning the performance challenges on the Split-UT-Zappos dataset due to long-tail distributions and the difficulty in distinguishing state classes.
paper_summary: paper_summary: This work presents a new task called Compositional Incremental Learning (composition-IL).
This new task extends the existing class incremental learning to a more fine-grained scenario for more realistic applications.
This work formulates and designs a new composition-IL benchmark based on Clothing16K and UT-Zappos50K datasets.
Technically, the authors propose several novel strategies based on prompting on pretrained ViT.
Specifically, this work suggests using three prompt pools for states, objects, and their compositions respectively.
To overcome the gap between the pretrained task and the composition task, the object prompts are further injected to guide the selection of state prompts.
The authors also suggest a learnable generalized-mean prompt fusion scheme for prompt reducing.
Extensive experiments are provided to evaluate the models.
strength_weakness: strength_weakness: 1. The main concern for me is the proposed composition-IL itself. From the experimental results (Tables 1 and 2), it seems that the previous prompt-based works can also handle this task with comparable performance. So my double is that if this work overclaims the difficulty of composition-IL? As someone working in the continual learning field, I think new settings and tasks are always welcomed only if the new task is challenging and practical enough.
2. The effectiveness of the proposed components. As shown in Table 3, I think the performance gap between C+S+O and C+S/C+O is small, so the effectiveness of multi-pool prompt learning is not that significant. Similar results are also in Table 4. In Table 4(a), there is only a 0.32% improvement in HM when introducing the object injection. In Table 4(b), there is also only a 0.3% improvement when using the GeM instead of the naive mean pooling.
questions: questions: Please see the 'Weakness' section for detailed questions.
In conclusion, I expect the authors to provide their thoughts about the necessity of the proposed composition-IL setting, as well as some explanations of the small performance gap when equipped with the proposed modules.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper presents a novel setting of incremental learning, named Compositional Incremental Learning (composition-IL). This setting differs from existing ones as it involves recognizing not only new objects (e.g., a shirt), but also their states (e.g., red) and the resulting compositions (e.g., a red shirt). To address this novel setting, the authors present a prompting-based approach that leverages a pre-trained transformer network to learn the tasks incrementally, thus sharing some similarities with existing methods such as L2P, Dual-Prompt, and CODA-Prompt. Additionally, the authors devise a three-way retrieval mechanism, devoting tailored prompt pools to state, object, and composition-level representation learning. In the experimental section, the authors assess several aspects: the comparison with the state of the art, the benefits of the three-way prompting strategy, the impact of injecting object-level information into the state-level prompt, and the contributions of each involved loss function.
strength_weakness: strength_weakness: **Many hyperparameters/tuning complexity (major)**. From a methodological perspective, the model incorporates several loss terms, each requiring a tailored balancing hyperparameter. Additionally, other factors such as the length of each prompt and the size of the prompt pool must be considered. In truth, the authors report the chosen configuration in their experiments in the second paragraph of Sec. 5.2. As can be observed, there are many hyperparameters, which could complicate the model’s application in real-life scenarios. In this respect, the authors should provide a simplified (yet still reliable) version of their approach, pruning some loss terms and configurations that have a negligible impact on results. Indeed, by examining the results of the ablation studies, it appears that there is room for simplifying the training stage.
Statistical significance (major). Concerning Tables 3 and 4, the results achieved by the final configuration of the Compiler are sometimes very close to the simpler, ablated versions. For example, consider Table 4(a) where 'no injection' is close to 'O->S', and Table 3 where 'C+S' is close to 'C+S+O'. In these cases, it is difficult to conclude whether the benefits are due to effective technical advances or should be ascribed to statistical noise. In this respect, I strongly encourage the authors to perform multiple runs for each experiment and average the final results. This approach would render the comparisons more significant.

**More rehearsal baseline (major)**. I would suggest including in the experimental section the results for DER++ [b] and ER-ACE [c], two straightforward baselines based on rehearsal. As the main contribution of this paper concerns the introduction of a new experimental setting, I believe it is important to provide an extensive evaluation of the existing literature. Moreover, the authors should compare their approach with SLCA [d], a recently introduced baseline for the continual fine-tuning of pre-trained models.
Relation with existing works (major). There is a recent research line on multi-label incremental learning; see [a], one of the most cited papers in the field. In this respect, how does the proposed setting deviate from the multi-label scenario? To justify the introduction of a novel setting, the authors should discuss potential overlaps with existing benchmarks more thoroughly. From a more methodological perspective, instead, I would encourage the authors to provide more references while introducing the learning objectives used in their approach. For example, CODA-Prompt devises orthogonality constraints between prompts, sharing strong similarities with the objectives outlined by Eq. 2 and 3. Moreover, aligning the query and the selected prompts is common in prompting-based approaches (see L2P). 

**Lacking ablation studies (minor)**. In Tab. 3, it would be important to see the results of single-pool prompt learning, especially for ‘S’ and ‘O’ (‘C’ has instead been already provided). Indeed, I would guess that a simple strategy based only on object-level prompts would be enough to reach satisfactory performance. 

**Clarification requested (minor)**. In Sec. 4.2, the authors discuss an attention-based mechanism to inject object-level cues into the state-level prompt. In this respect, they use three learnable projections, indicated by W_Q, W_K, and W_V. However, as these transformations are continuously updated while tasks progress, did the authors consider that these projections could suffer from catastrophic forgetting? It seems that there are no explicit countermeasures against forgetting regarding these variables, which could hinder the efficacy of the retrieval stage.

**Formatting issue (minor)**. Table 4 appears before Table 3.

[a] Kim, C. D., Jeong, J., & Kim, G. (2020). Imbalanced continual learning with partitioning reservoir sampling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16 (pp. 411-428). Springer International Publishing. 

[b] Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020). Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33, 15920-15930.

[c] Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., & Belilovsky, E. (2021). New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025.

[d] Zhang, G., Wang, L., Kang, G., Chen, L., & Wei, Y. (2023). Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19148-19158).
questions: questions: I have no questions.
limitations: limitations: The limitations are briefly discussed in the supplementary materials.

Comments: comment: Dear Authors,

Thank you for your clear responses and additional analyses. I appreciate your work and believe it adds valuable contributions to the field. I raise my rating to 6.
I appreciate the authors efforts in improving the paper and clarifying most of my concerns.

It is good that the authors propose a simpler version of the method which still works with lesser hyper-parameters and should be included in the paper. I expect the authors to have the discussion in the paper on the need to reduce the hyper-parameters and use simpler models. 

It is interesting to see that the baselines of NCM and FeCam performs very competitively with the prompt-based methods and is better in predicting the state while worse in object prediction. It is good to have a detailed discussion on comparison with these frozen-model baselines which highlights how the learnable prompt-parameters (also using frozen model) are affecting the performance. It is also good to add them in the model efficiency analysis (like in Table 3 rebuttal) since they do not learn any new parameters.

Overall, I am now quite positive about the contribution of this work which opens up a very interesting direction in Composition-IL with very intuitive and novel setting, method, datasets and very good analysis. The extensive experiments (including the baselines and also more rehearsal-based method comparisons added during rebuttal) makes it a good benchmark for future works. 

One minor remark for the final version would be to add proper informative captions for all figures in the paper which now is quite empty, it makes the paper more readable and easy to understand. I raise my rating to 6.
The authors have answered my questions. I would like to thank them for their efforts, which address most of my initial concerns. As a result, I will raise my score to weak accept.
Thank you very much for further discussion and comments. We appreciate very much for your raising the rating. We are happy to answer your questions below:

**Q1: About hyperparameters.**

**A1:** We have acted on your suggestion to reduce the number of hyperparameters by streamlining the model. Specifically, we implement a simplified alternative model (namely Sim-CompILer) , which removes the directional decouple loss, RCE loss, and Object-injected State Prompting from the original CompILer, thereby eliminating hyperparameters $\lambda_1$, $\lambda_2$, and $\alpha$, Note that, $\lambda_3$ still remains in order to balance CE loss and surrogate loss, and $\beta$ is still needed to adjust the trade-off between compositions and primitives. The results, shown in the table below, demonstrate that, even though CompILer achieves the best performance across both datasets, Sim-CompILer slightly underperforms but still ranks second. We agree with your suggestion and believe this simplified version is more practical and easy-to-follow. We will include the results for Sim-CompILer in the final version. Hopefully, this simpler alternative would provide more flexibility in potential practical applications.

| Name          | Split-Clothing (5 tasks) |   |   |   |   | Split-UT-Zappos (5 tasks) |   |   |   |  |
|---------------|--------------------------|---|---|---|---|---------------------------|---|---|---|---|
|               | Avg Acc                  | FTT(↓) | State | Object | HM   | Avg Acc                  | FTT(↓) | State | Object | HM   |
| Sim-CompILer | 88.22                    | 8.37  | 90.95  | **96.47** | 93.36 | 46.43                    | 19.31  | 56.83  | **79.58** | 66.31 |
| CompILer      | **88.74**                | **6.98** | **91.61** | 96.34 | **93.92** | **47.06**                | **18.84** | **57.52** | 79.53  | **66.75** |

**Q2: Comparison with baselines.**

**A2:** Thank you for pointing out this issue. After checking our experiments, we confirmed that there was an error in the implementation of NCM and FeCAM. Concretely speaking, FeCAM provides two official implementations, one under the Avalanche Library [1] and the other under PyCIL [2]. Since the Avalanche Library includes both NCM and FeCAM, we conducted our experiments within this framework to ensure a fair comparison. Unfortunately, during the process, we overlooked the need to update the class mask, determining which classes have been seen during training the model incrementally. This negligence leads to the unreasonable results you have noticed. Also, we made the same mistake when implementing the NCM method, which also obtained unsatisfactory results. To correct the error, we have fixed the mask and re-conducted the experiments from scratch. The new results, as shown in the following table, verify the fact that FeCAM should achieve higher accuracy than NCM. 

CompILer outperforms NCM and FeCAM by enhancing adaptation through the tuning of learnable prompts across all sessions. In contrast, NCM and FeCAM are limited by their inability to update the model during incremental learning. Additionally, it is noteworthy that CompILer significantly surpasses NCM and FeCAM in Object accuracy. This is because CompILer freezes the backbone across all sessions and leverages the pre-trained parameters, while the other two methods update the backbone in the first session, which weakens their effectiveness in object recognition. We will include these methods as baselines in the next version and provide proper citations. 

| Method   | Prototype | Avg Acc | State  | Object | HM     |
|----------|-----------|---------|--------|--------|--------|
| NCM      | ✓         | 31.09   | 41.91  | 39.71  | 40.78  |
| FeCAM    | ✓         | 33.71   | **46.32** | 40.44  | 43.18  |
| CompILer | ✗         | **34.66** | 45.82  | **77.06** | **57.47** |

[1] Avalanche: an End-to-End Library for Continual Learning (CVPRW 2021)

[2] Pycil: a python toolbox for class-incremental learning (SCIENCE CHINA Information Sciences 2023)
I thank the authors for their rebuttal.
The rebuttal solved part of my concerns, so I will keep my score unchanged.
I appreciate the detailed rebuttal for the authors which answers most of my concerns.
1. Hyperparameters: While the authors argue that several papers use different training hyper-parameters for different settings even using the same dataset, I still believe that this is not a robust solution, it is just optimizing the results on the test set which does not make sense. Just saying that some existing papers do the same overfitting on the test set is not enough. I would urge the authors to reduce the number of hyper-parameters (following the discussion with Reviewer EVGX) to make the method usable and more realistic. Improving the accuracy by small margins is not everything, what's more important is how robust and practical your solution is. 
2. Comparison with baselines: It is appreciable that the authors reproduced the baseline methods. HiDE-prompt was expected to do better, it's good to have a discussion in the paper about the trade-offs. NCM is a baseline which can be considered as a lower bound since it does not update the model. FeCam is naturally expected to do better than NCM since it uses Mahalanobis-distance and has improved over NCM using ViTs [3]. So, it's quite surprising to see that it gets worse. I would ask the authors to look into the implementation or find appropriate justifications for this since FeCam results in Table 2 looks like an error or faulty implementation. It is important to discuss these baselines since they form the lower bounds and shows the scope of improvement using the prompt methods. 

I still maintain the concerns for the hyper parameters and urge the authors to look into this. I raised my rating to 5.
We appreciate the reviewer EVGX’s willingness to engage in further discussion. The followings are our point-point answer:

**Q1: About hyperparameters and complexity.**

**A1:** We appreciate your suggestion very much. We thereby have acted on it by developing a simplified model to reduce the number of hyperparameters. Specifically, we focus on the 'C+S+O' configuration, namely Sim-CompILer, which eliminates the directional decoupled loss, RCE loss, and Object-injected State Prompting, accordingly removing $\lambda_1$, $\lambda_2$, and $\alpha$ in the total loss cost. As a result, Sim-CompILer is constrained by vanilla CE loss and a surrogate loss, represented as $L_{total}=L_{CE}+\lambda_3L_{sur}$, where the remaining hyperparameter  is $\lambda_3$ only, which controls the balance between CE and surrogate loss. Apart that, $\beta$, is still needed to adjust a balance between compositions and primitives. The results, shown in the table below, show that, while CompILer performs best across both datasets, Sim-CompILer slightly lags behind but still achieves second-best results compared to the baselines in Table 1 and Table 2. We opted for three pools rather than two due to poor performance with dual prompt pools in Split-UT-Zappos, as noted in our previous rebuttal. We will include the results for this Sim-CompILer alternative in the final version. Hopefully, this simpler alternative would provide more flexibility for the potential users to conduct their related research. 

| Name          | Split-Clothing (5 tasks) |   |   |   |   | Split-UT-Zappos (5 tasks) |   |   |   |  |
|---------------|--------------------------|---|---|---|---|---------------------------|---|---|---|---|
|               | Avg Acc                  | FTT(↓) | State | Object | HM   | Avg Acc                  | FTT(↓) | State | Object | HM   |
| Sim-CompILer | 88.22                    | 8.37  | 90.95  | **96.47** | 93.36 | 46.43                    | 19.31  | 56.83  | **79.58** | 66.31 |
| CompILer      | **88.74**                | **6.98** | **91.61** | 96.34 | **93.92** | **47.06**                | **18.84** | **57.52** | 79.53  | **66.75** |

**Q2: About forgetting**

**A2:** We apologize our explanation confused you due to the complex biological theory. We acknowledge that we have not implemented explicit methods to circumvent catastrophic forgetting, and this choice was intentional for the following reasons:
 
First of all, following L2P, our model avoids catastrophic forgetting by freezing a ViT backbone across all incremental sessions. Thanks to this simple yet effective strategy, L2P and its improved variants (such as Dual-Prompt, CODA-Prompt, LGCL and our CompILer) achieve much lower forgetting rates than EWC and LwF, as shown in Table 1 of the submitted manuscript. In addition, following the suggestion from Reviewer x9g2, we perform more comparison with NCM and FeCam. While these two methods are not built based on L2P, they still suggest freezing the backbone to overcome the catastrophic forgetting effectively. Inspired by these results, we can note that the core of learning without forgetting relies on a frozen backbone, and a limited number of learnable parameters have less impact on raising the forgetting. This finding is consistent with the fact that L2P did not exploit any explicit countermeasures between the learnable prompts to alleviate the forgetting yet. Akin to the learnable prompts, we advocate learning the parameters in the object-injected state prompting module devoid of extra explicit countermeasures.

On the other hand, we would like to emphasize again that the aim of continual learning is to pursuit an optimal balance between stability and plasticity. Since the stability has been obtained by freezing the backbone, another challenge is how to improve the plasticity when adapting the model on new tasks. Thanks to the learnable parameters in the prompt pool and object-injected state prompting module (even though their number is slighter than that of the backbone), it allows us to improve the adaptability of the model and increase the accuracy further.

Last but not least, the model inclines to forget the compositions rather than objects and states, because the object and state primitives may reappear in new tasks. Hence, we apply object-injected state prompting to the primitive branches, leading to minor forgetting on the compositions.
I sincerely thank the reviewers for the efforts they spent during the rebuttal period. I still have several considerations and questions for the authors.

**Hyperparameters and complexity**. While I acknowledge that tuning several loss coefficients is a standard practice in deep learning, my concerns still persist. Therefore, I urge the authors to reconsider their approach in the final version and, if feasible, reduce the number of terms involved during optimization. Based on the experimental results, there appears to be some flexibility to achieve this reduction. It's important to keep in mind that future users will need to apply this method to new benchmarks, and the abundance of hyperparameters could make this process quite challenging.

**Forgetting** In my original review, I expressed some concerns regarding the learnable projections W_Q, W_K, and W_V, which are likely to suffer from catastrophic forgetting as no explicit countermeasures have been taken to protect these parameters. The authors' response, which relies on biological arguments, is somewhat difficult for me to follow. Could the authors provide a more practical discussion on this point? I believe addressing this issue is important.