Plug-and-Play Compositionality for Boosting Continual Learning with Foundation Models
Authors: Weiduo Liao
Fei Han
Hisao Ishibuchi
Qingfu Zhang
Ying Wei
Venue: ICLR 2026 Oral (2026)
Keywords: Continual learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce CompSLOT, a universal concept learning method to continual learning with foundation models system to establish a concept-level understanding of class prediction for alternative continual learners.

Abstract: Vision learners often struggle with catastrophic forgetting due to their reliance on class recognition by comparison, rather than understanding classes as compositions of representative concepts. 
This limitation is prevalent even in state-of-the-art continual learners with foundation models and worsens when current tasks contain few classes. 
Inspired by the recent success of concept-level understanding in mitigating forgetting, we design a universal framework CompSLOT to guide concept learning across diverse continual learners. 
Leveraging the progress of object-centric learning in parsing semantically meaningful slots from images, we tackle the challenge of learning slot extraction from ImageNet-pretrained vision transformers by analyzing meaningful concept properties. 
We further introduce a primitive selection and aggregation mechanism to harness concept-level image understanding. 
Additionally, we propose a method-agnostic self-supervision approach to distill sample-wise concept-based similarity information into the classifier, reducing reliance on incorrect or partial concepts for classification. 
Experiments show CompSLOT significantly enhances various continual learners and provides a universal concept-level module for the community.

Published as a conference paper at ICLR 2026
PLUG-AND-PLAY COMPOSITIONALITY FOR BOOSTING CONTINUAL LEARNING WITH FOUNDATION MODELS
Weiduo Liao! Fei Han', Hisao Ishibuchi' Qingfu Zhang* Ying Wei "Zhejiang University, 'hhe Hong Kong University of Science and Technology 'Southern University of Science and Technology, "City University of Hong Kong liaowdema 11, sustech. sustthc edu. en, fhanaceconnes ust .hk hi sisosusstech edu. on, qingfu. zhang@cityu edu. hk ying. . weiez juju jet.iu.uuuuuuaa... edu.cn cn
ABSTRACT
Vision learners often struggle with catastrophic forgetting due to their reliance on class recognition by comparison, rather than understanding classes as compositions of representative concepts. This limitation is prevalent even in state-of-the -o-tthe-rtt continual learners with foundation models and worsens when current tasks con- tain few classes. Inspired by the recent success of concept-level understanding in mitigating forgetting, we design universal framework CompSI.O7 SLOTT to guide concept learning across diverse continual learners. Leveraging the progress of object-centric learning in parsing semantically meaningful slots from images, we tackle the challenge of learning slot extraction from praageNet-pretrainne vision transformers by analyzing meaningful concept properties. We further introduce primitive selection and aggregation mechanism to harness concept-level image un- derstanding. Additionally, we propose method-agnostic self-supervision approach to distill sample-wise concept-based similarity information into the classifier, re- ducing reliance on incorrect or partial concepts for classification. Experiments show CompSLOT significantly enhances various continual learners and provides a universal concept-leve module for the community
INTRODUCTION
Artificial intelligence systems mimic the lcarning behavior of human intelligence by collecting infor- mation and managing knowledge pools from continually assigned tasks in the open world. This need to handle non-independent and identically distributed training data has driven research in continual learning (CL) (Zhou et al., 2024c at Biesialska et al.) 2020). which aims to balance the objectives of overcoming catastrophic forgetting (McCloskey & Cohen 1989) of learned tasks and achiev- ing efficient adaptation to future tasks, also known as the stability-laasiiiity dilemma (Grossberg 2012). Leveraging powerful pre-trained backbone to ensure basic understanding of the world, CL methods of foundation models (FMs), including prompt-basee methods (Gao et al., [2023] Smith
let al.
2023
WWan al. 2022c b. 2024 Gao et al. 2024). representation- based methods (Zhou et al.
2025
2024b
McDonnell et al. 2023] Zhang et al. 2023) and model-mixture- based methods (Gao
al.
2023 Wang et al.] 2024 Marouf al.) 2024). have emerged a popular direction in this
field.
However, FMs need to be updated when encountering oul-of-distribution data in the upcoming
tasks
(Yang al. 2025).
The human brain exhibits compositionalily (Hupkes et al. 2020: Liao et al. 2024) when comprehend- ing the world, decomposing seen concrete objects into abstract concepts. For example, a Chihuahua consists of general dog concepts such as body shapes and chihuahua-specific concepts like small size and head shapes. This interpretability is intuitive humans, enabling them to generalize novel dog species by decomposing them into combinations of existing concepts while learning disentangled new concepts to refine the knowledge base, thus, facilitating efficient reuse (Liao et al., 2024). common strategy for existing CL methods for FMs to alleviate forgetting is to inherit parameters learned from
"Corresponding Authors: Ying Wei, Qingfu Zhang, Hisao Ishibuchi
Cooee is available at gggtthh.commiiioecdddoCCCCoomooo

---

Published as a conference paper at ICLR 2026
VIT CL Methods
Logiss
Lee
Primitve sim matrix
Cos sim with sottmax
Logits dim matni
Primitivest
Image Batch
Concept Leaminn
Lg
Siots
Slot
LN
Linear
Primitive Selection
KP
Leve
Enhoncing intra consistincy
Lp
VIT
Attention
Learnable
Figure 1: Proposed CompSLOT framework. Given an image batch, we extract primitives for each image with concept learning procedure. Then, we distill the sample-wist similarity from the primitive representations of the image batch into logits. Takeaway: This conceptual pair-wise similarity enables the model make decisions by additionally considering low-dimensional concept combinations, rather than relying solely on high-dimensional features.
old tasks when initializing new tasks' models, as done in Wang et al. (2024) Gao et al. (0224). These state-of-the-an (SOTA) approaches generally not fully exploit cross-task potential correlations (i.e., common concepts shared across tasks). In contrast, learning low-dimensional concept combinations to understand classes does not require establishing class representations from the high-dimensional feature level, as in traditional methods, thereby mitigating catastrophic forgetting and enabling rapid adaptation to novel classes (Liao et al., 2024 Yu et al.) 2025 Yang et al., 2024 Kundargi et al., (2025) Lai al. 2024). Thus, set of CL methods leverages interpretable tools, e.g., ChatGPT (Brown et al. 2020), prototypes (Rymarczyk et al. 2023 Ahrens et al. 2023). and concept bottleneck models Yuu al. 2025) Yang al. 2024]| Lai al. 2024) to bring attention concepts within images, which achieves great success on boosting CL performance. Another challenge that canonical benchmarks, such as Split-CIIe Split-CIFAR CIFARIO0 100 (Krizhevsky et al.) 2009) and Dplit-ImagNNet-R (Hendrycks et al., [202]) are not specifically designed to evaluate the compositionality of continual models. The CFST evaluation framework (Liao et al. 2024) (including CGQA and COBJ) the only work, to our knowledge, that systematically studies the compositionality of continual learner. CFST introduces two component- relevant phases which the data share a common concept set with different combinations, In the first phase, the dataset is split into several continual tasks, aiming to train continual learner. Subsequently, the second phase used to evaluate the learner's compositional generalization performance on unseen concept combinations.
Motivated by the above analysis, we pose the research question: Can the compesitionality in concept learning truly enhance the CL performances of SOTA continual learners with FMs? We propose a Compositional Slot plug-in (CompSLOT) for continual learning to answer the above question, which is illustrated in Figure The first step involves extracting concepts from raw images, namely, conducting concept learning. Several studies have shown significant progress in concept learning by utilizing explicit concept-level supervision obtained from segmentation masks (Kirillov al. 2023) Ravi et al.) 2025) or natural language annotations (Ramesh et al.) 202T Yu et al. 2025). Nevertheless, crucial compare with SOTA CL methods of FMs, where only labels from the current CL tasks are available as supervision. Consequently, Slot Attention (Locatello et al.) (2020). as an SOTA unsupervised object-centric learning approach (Greff al., 2020), has effectively emerged as a viable self-supervised solution. A Slot Attention module learns to group and encode spatial features into a set of low-dimensional distinct slots, with each slot representing a disentangled region and binding an object (i.e., concept) in the image. To avoid additional learning of the encoder the input to Slot Attention can be specified as semantic patch features provided by a pre-trained vision transformer (VIT), which also serves the learner's backbone. We present preliminary experiment demonstrating that the learned Slot Attention module exhibits almost no forgetting across compositional tasks, shown in Figure
2

---

Published as a conference paper at ICLR 2026
With the above method to extract the hidden concepts in images, the next step is to introduce concept learning into CL methods with FMs. The challenge is that there is no unified forwarding framework to organize all CL methods with FMs that we easily perform concept learning and assist vanilla learning processes of feature extractors. Hence, we propose regularizing the outputs of learners with sample-wise similarity based on concepts. This makes our approach a method-agnostic plugin for any CL method with FM. We first use a learnable aggregation mechanism based on attention to extract class-relevant concepts (i.e.. primitives (Zou et al. 2024)) as the weighted sum of slots based on their similarity to learnable task key. The distance of primitives between two images carries information about the similarity in concept level. For example, a Chihuahua is close other dog species (e.g.. German Shepherd) rather than cat species (e.g., Siumese) because they share considerably more concepts (e.g., dog body). Subsequently, we propose a method-agnostic primitive-logi alignment plugin to distill the learned sample-wist concept-level similarity into the outputs of models based on a contrastive loss. Our experiments demonstrate that the above procedures successfully select meaningful concepts in images primitives and ultimately achieve superior continual learning performance attributed to a better compositional generalization performance.
The contributions of this work are summarized as follows:
We proposed CompSLOT, a method-agnosiie plug-in comprising 1) a concept learning module that leverages Slot Attention and rich vision foundation models to extract primitives. and 2) concept knowledge distillation module that enables learners to intentionally discover shared and distinct concepts among classes, thereby guiding the maiiinnnmmkinng process of classifiers.
We designed 1) a primitive selection mechanism with an additional primitive loss that effectively achieves robust primitive extraction across different examples of the same class, and 2) a primitive- logit alignment loss that contrastively regularizes the sample-wise similarities between continual learners' outputs.
The experimental results demonstrate that CompSLOT successfully leverages concept-wise com-- positionality to significantly enhance a wide range of continual learners.
2 RELATED WORKS
Continual Learning of Foundation Models Benefiting from the rich knowledge in large-scale pre-trained VÝT, CL methods with FMs Zhhou al.] 2024a) greatly mitigate forgetting previously learned classification tasks and achieve fast adaptation to new ones. The community has mainly
developed three families of approaches, according to the way of utilizing the pre- pre-trained knowledge:
1) Prompt-bused methods (Gao et al. 2023 [mith et al. 2023 Wang et al., 2022c by Gao et al. 2024]
I Liang& & Li 2024 Le et al. 20224 efficiently tune prompts for tasks rather than tne the backbone;
2) Representation-bassee
methods Zhou al. 2025 2024b McDonnell et al. 2023 Zhang et al.)
2023) involve leveraging the ad advantages of representations from the pre-traine trained backbonn with a class
prototype-based classifier; 3) Model-mixture based methods (Gao et al.] 2023 Wang et al. 2024
Marouf et al. (224)) utilize hybrid techniques such as model fusion (Wang et al. [224] Marouf et al.
2024) and model ensemble (Gao et al.) (203) to query set of models, thus, making the prediction more robust; Moreover, rehearsing old samples is an effective way to alleviate forgetting old tasks.
Several methods (Wang et al., 2022a Yan et al. 202T Zhou et al., 2023) contribute efficient sample storage mechanisms and auxiliary supervision to address class imbalance, achieving a better subility-plasticci trade-off. However, the above methods ignore hidden conceptual relationships among classes, limiting their significance on handling compositionally relevant tasks.
Compositionality Compositionality has been extensively studied in natural language process- ing (Bicsialska et al. 2020; Kaushik Martin 2020 Lake & Baroni 2018 Keysers al., 2020).
To achieve compositional learner, methods include the introduction of sparse coding (Murphy
et al., 2012), regularization (Sun al. 2016]] Luo et al. 2015). and applying independent component analysis (Musil & Mareček) 2022 Yamagiwa wa et al., 2023). In Hupkes et al. (2000), the authors sum- marize five types of tests for language compositionality, which are further extended to vision in Liao et al. (2024). Meanwhile, researchers in vision utilize compositional information between objects and attributes to boost zero-shot inference through regularization (Nagarajan & Grauman 2018).
separate learning (Ruis et al., 2021), causal reasoning {Atzmon et al. 2020), self-attention (Khan jet al., [2023), and uniting energy-bart based modules (Wu et al. (022). Common strategies to learn
hidden concepts among continual tasks are external interpretabiliiyy tools (Yang al. 2024), learnable
3

---

Published as a conference paper at ICLR 2026
mapping (Li et al. 2024), prototypes Rymarczyk al. 2023 Ahrens et al. 2023 Rymarczyk et al. 2021). CLIP (unddrrgi et al. 2025) (Agrawal et al. (025). ChatGPT (Yu et al. 2025). and assigning different module paths for tasks (Rajasegaran al. Ostappeko al. (021). Our work. instead, leveraging Slot Attention, does not require prior concept-level supervision for training or an extra concept bottleneck model (Yu et al. 2025), making it more adaptable and easier to integrate with
different methods.
Object-centric Learning We adopt object-centric learning autonomously extract concept infor- mation directly from images. The introduction of Slot Attention (Locatello et al., (020) marked the emergence of a new paradigm for disentangling objects (i.e., concepts) within a scene. Subsequent re- search has focused on improving its robustness in complex environments primarily through encoder enhancements like covariance regularization (Stange et al. 2023) and bi-level optimization (Jia et al.) 2023, Chang et al. 2022). Other efforts have explored advanced decoders to refine decomposition. For example, SLATE (Singh et al. 2022) uses an autoregressive transformer decoder, while Wu et al. (2023); Jiang et al. (2023) propose diffusion-based approaches. Kakogeorgiou al. (2024) leverages distillation to refine object segmentation via decoder-guie guided encoder training, and Kori et al. (2023)
povironments-piimarllly
introduces conditional Slot Attention with foundational slot dictionary address specialization limitations. Our method, instead, employs a lightweight MI.P decoder to minimize computational cost while preserving effectiveness. Experiments show that this simple design can still significantly benefit continual learning.
3 PRELIMINARIES
Class-incremental vision continual classification tasks We consider T sequential vision classi- fication tasks with a dataset 'D [DI,.... DD"), where each D' consists of image samples E X with corresponding labels y E y². Here, )² is a subset of the global label set y, and VY² n n = for & +f k, with task identity unknown during inference, i.e., class-incremental learning (CIL) set- ting. A general model- based continual learner includes a Vision Transformer (ViT)- baaeed backbone f-10r) and classification heads h2 (-10,.), where t is the task identity. Each head is trained sepa- rately for the corresponding task, but the outputs from all heads are concatenated for final inference: HGc [ [0], where [0] indicates the [CLS] token (i.e., the first dimension of the output of f), and pred (Zte) = arg min [/1 (Heel0n, ;;..;;;. fr (Hee 10 where [:3-] denotes concatenation.
Slot attention (Locatello al. 2020) As the state-of-the-arr object-centric RRB. slot attention aims to decompose a single image into a set of K disentangled slots $ E RK each one compositional component of the image. D, is the dimension of slot representation. The output f from pre-trained ViT backbone consists of two parts: the uninstructed image feature H f( 0 [0] E Rº with the token [CLS] and the semantic patch features E = f(x [0,][1 :] E ND where N is the patch number. These N patches are further encoded into the slot space and refined into K slots through an iterative attention procedure. The K slots are first initialized with a learnable Gaussian distribution. In each refinement iteration, slots collect soft assignment information from each patch with an attention mask A R& The weighted mean A is then computed along the patch 2014) aggregates the patch information into dimension, and a Gated Recurrent Unit (GRU) (Cho et al.
each encoding
the assigned slots, as follows: A = o ((S) VD,
Ain Σ A., S + GRU(S, Av(E)),
A,
where q(-), k(-), v(-) are learnable query, key, value projections, respectively, and (--) is the softmax function.
METHODS
We present our CompSLOT framework in Figure 1 For each continual task 2), we first perform concept learning (detailed in section 4.1 through a slot decomposition and a primitive selection mechanism, and then distill the pair-wise similarity statistic of the extracted primitives to model outputs (detailed in section 4.2) in a -agnosi- manner.
Unless otherwise stated, the proposed slot attention and primitive selection modules globally shared across tasks, and no parameters except the ViT backbone are frozen. They are initialized at the beginning of the first CL task. In future CL tasks, their architectures (e.g., the number of slots)
4

---

Published as a conference paper at ICLR 2026
remain fixed, while parameters will be fine-tuned throughout all CL tasks. This design prevents parameter explosion and supports long-sequence tasks, as demonstrated in Figure 3b
4.1 CONCEPT LEARNING
Firstly, we define concepts the ground truth slot decomposition of image. Since slot attention exhibits permutation equivalence w.r.t. the order of the slots (and masks) (Locatello et al. 2022)) we regard A) the corresponding set representations of {S, A}. where S A= = {a K ==1 with 8 E P. and a R. being the i-th row of S and A, respectively.
{ and
Definition (Concept & Disentanglement), Let æ be an image, then A) is a disentangled decomposition of æ (a.k.a., concepts and corresponding attention regions), if 1) az aj E A, at aj R a; 1 aj., and 2) S satisfies arg ming ,,,,, |sim( (s 8j)], where 8,, 8j D. and - is the absolute value. 1 is orthogonal symbol, and sim(.,.) is a similarity score function, e.g., cosine similarity.
Remark 1. The examples of concepts are Chihuahua's head w.r.t. Chihuahua objects in section I and buildings w.r.t images in Figure In Figure A corresponds to patch regions and S corresponds to slot representations used to calculate cosine similarity.
To train the slot attention and primitive selection modules, we resort continually reconstructing D and an additional contrastive primitive loss.
Continual image reconstruction For clarity, we denote the forward path of slot attention as A) = s E 0.), S e I RKXD, A E R We augment the position embedding into slots S when reconstructing the image, as the ViT does. S = S @ posn, where pos, E TRD, is the learnable position embedding at patch and ( is the dlement-wise addition with broadcasting. Next, S' E x a collection of N position-augmeente slots, are mapped back individually to the D-dim patch space with an MLP slot decoder d(. d(-10a). Subsequently, we apply weighted-sum with the attention mask A and finally get the reconstructed patch features E. The reconstruction loss Lye the MSE loss between the ground truth patch features E and the reconstructed E, as follows:
E A (()) E RND, Lre E Ell2-
(1)
We present preliminary experiment demonstrating that the learned Slot Attention module exhibits almost no forgetting across compositional tasks, as shown in Figure 2 Attention module on COBJ 3-tasks continual reconstruction task (i.e., trained with reconstruction loss). We then extract slots for images from the first task using the modules after training on the first, second, and third tasks. We observe that each corresponding slot consistently represents numan-interpretabbe concept and remains stable after training on new tasks, maintaining high cosine similarity.
Specifically, we train Slot
Primitives When describing the object Chihuahua, some concepts (like Chiluahua's 's head) are class-relevant, while others (like sky) are class-irrelevant. We name such class-relevant concepts as primitives, containing information to identify the desired classes.
Definition (Primitives).. Let X S be an image set labeled y and the corresponding set of concept sets, respectively, and S E S", then concept subset P C S is primitives of S, if VS' E S, C S'.
Our goal is to identify a unified primitive representation sP, which is regarded as the linear combination of concepts S. The basic idea is that primitives have higher probability appearing in æs from the same class x and are likely to carry important information describing this class. Thus, we have the following two questions: 1) How represent the selected primitives s from S? and 2) How to minimize the distances among sPs extracted from the images in the same class?
Primitive selection To answer the first question, we propose learnable attention-basee primitive selection mechanism to aggregate K slots. We use a linear module with layer norm and a tanh activation layer to map slots into unified similarity space. The similarity to a learnable primitive key KP E R D. measures the slot significance. Then this similarity wp weights the mapped slots and aggregates them into single representation s" (i.e., primitive representation), which is summarized as follows:
$ (ann((ineer((NN(()),
"p (kPPP) = n S,
5

---

Published as a conference paper at ICLR 2026
Task
Task
Tisk
Bulking
rocon
Tree
Task1
0.00
0.00
Backg ound-
0.84
Boat-
Task
0.85
0.80
0.85
0.85
Punt
Takc3
AW
Floor
Figure 2: Examples of learned slots by continual COBJ reconstruction tasks and validation recon- struction losses. Left: slots are extracted for three example images from the first task on learners after training the 1-st, 2-nd, and 3-rd tasks. Slots are masked with different colors. Middle: corresponding slot cosine similarity matrices grouped by correlated regions. Each group contains slots from three tasks and is identified by the Hungarian matching algorithm. Right: validation reconstruction loss matrix. Each row indicates a learner trained after a specific task and evaluated on all seen tasks, respectively. Takeaway: learned Slot Attention module enjoys almost no forgetting across compositional relevant tasks.
where is temperature coefficient controlling the sparsity of slot selection wp, which is set to 100/vD; in practice. A larger Ty indicates a smaller number of slots to be selected to represent this image æ.
Contrastive primitive loss To answer the second question, we rewrite Definition 2 as follows:
Theorem 1 (Intra-class consistency). Consider S2 E S and two corresponding largest primitive sets Pi C S1, P2 C S2 are identical, i.e., Pi = P2 and |Pil| M, where || is the cardinality of set. In other word, consider the pair-wise ordered sets ( P2] match(P,, P,)), where match(...) matching algorithm (without loss of generality, Hungarian algorithm (Kuhn 1955 ). d
then the corresponding matched concepts should be the same: P = {s} 11 P. = + and Vi M).. sim(s), (s},
This form of pair-wise primitive similarities from images within the same class motivates the use of label supervision and contrastive learning (Khosla et al.) 2020; Chen et al.) 2020b). We first collect the normalized similarity d²,, between one-hot label and the softmax similarity di, between s?
Then, we use mini-batch clustering loss that a small KL divergence between d',, and di, means a small distance between sP, 8y in the same class and a large distance between those in different classes. The primitive loss Lip is as follows:
Iim(I,,I,) dy Eneer En si, J2)
))) Enpen exp(T, sp s))
exp(Tysim( s
di
I
Σ EL
di log
di
where II; is the one-hot label for sample I4, and Tp is a temperature coefficient controlling the strength of primitive loss. The learned slot visualizations section G (including CGQA, COBJ, ImageNet-R. CIFAR- demonstrate that meaningful concepts (represented by primitives, third column "Sum") remain stable across tasks for the same images. We attribute this robustness to "concept rehearsal": although class labels change, many visual concepts are shared and recur across tasks, helping stabilize the primitive selection weights. Section|G Glass visualizes the pair-wis wise primitive similarities, showing that concept relationships are preserved across images of the same class and shared concepts remain consistent even when images are from different tasks.
By jointly minimizing Lre: Lp. the leameed slot attention module equips the abilities of extracting concepts, identifying primitives, and achieving intra-class primitive consistency. Specifically, we group these losses as Tsstot Tyee + alep where a is coefficient balance the impact of Iap.

---

Published as a conference paper at ICLR 2026
4.2 BECHOD-AAGOSTTE PRIMITIVE-LOGI KNOWLEDGE DISTILLATION
The learned primitive & equips superior property of aggregating important class-relevant concepts. Such understanding can be self-supervision to regularize the output of the continual learner, i.e., the distribution of logits. Thus, the model gives predictions based on the exact extracted concepts. For example, a chihuahua image should have relatively higher logits on other dog classes than logits on cat classes because they share similar concepts such as dog body shapes. Specifically, we design primitive-logit alignment loss to contrastively distill the learned primitive statistics to logit statistics, i.e., minimizing the KL divergence between softmax logit similarity d' and previously learned primitive similarity d', as follows:
sim+ (
exp(Tasim(4 14)) 4k))
di E sim, (8 ()
d
Lo= FL
E
di log
LBgEn
where 4, he (H,)) is the logits of z; for the current task, sim+ (-.)) cosine similarity with min-max normalization, and is temperature coefficient controlling the loss strength. We employ min-max normalization (instead of softmax) to sharpen slot supervision. Note that I.a method-agnostic as long the CL method has an FM backbone to support extracting semantic features. Finally with the cross-entropy task loss Lee, the training loss is as Lrr Liee + BLaa, where a coefficient to balance the impact of La.
5 EXPERIMENTS
In the experiment part, we highlight the research question we will answer: How and why does our CompSLOT benefit a large range of continual learning with foundation models? To answer this, we compare algorithms with and without CompSLOT and perform ablation studies in section (5.2 We analyze the influences of hyperparameters in section H.2 investigate different backbones in section H4 and compare with other concept learning methods section H.5 and visualize the slot extraction to analyze how CompSLOT enhances CL performance in section
5.1 EXPERIMENTAL SETTINGS
Baselines To verify the universality of the proposed CompSL.OT, we adopt wide range of SOTA
continual learners with foundation models, including: 1) prompt-based methods: CPrompt (Gao
al. 2024); representation-
methods: ADAM+adapter (Zhou et al., 2025). Ran-
PAC (McDonnell et al.] 2023). EASE (Zhou et al. (024b)) 3) Model-mixture biss- meth-
ods: CoFiMA (Marouf et al. 2024). FOSTER* (Wang et al. 2022a), DER Yaan et al. 2021).
MEMO* (Zhou et al. 2023). Methods with "** postfix indicate that they adopt rehearsal process.
Algorithms are implemented using the PIL.TT (Sun et al.) 2025) platform with default hyperpa- rameters. Methods with CompSLOT are denoted with postfix "4" Unless otherwise stated, the backbone ViT-B/16 backbone pretrained on ImageNet-2IK: while we also investigate the effect of different backbone architectures in section H.4 We also compare recent concept bottleneck models for continual learning, including CLG-CBM ( Yu et al. 2025), and another concept knowledge plugin, SACK (Kundargi al. 2025) integrated with CODA- (Smith al.) 2023). In this experiment, we use CLIP VIT--/16 (Radford et al.) 2019) backbone for CompSL.01 to extract slots. For the details of the implementation details, please refer to section E
Benchmarks We conduct experiments on compositional datasets, including CGQA and COBJ al.) 2024). and commonly used datasets, including ImageNet-R (Hendrycks et al., 202 The former classification datasets contain sufficient number of combinations of concepts, allowing for visual analysis and evaluating the compositionality. When comparing with other concept-based methods, we conduct experiments on CUB200 (Welinder et al. (2010) and CIFAR 100 (Krizhevsky ct al., 2009). We choose different continual task settings to evaluate different compositionality levels. Specifically, we denote "F-S tasks" as that the first task contains F classes and the following tasks contain S classes. For example, "50-10 tasks" means splitting 100 classes into six tasks with sequence of class numbers [50, 10, 10, 10, 10, 10). In the main context, we report 10- 10 tasks results for CGQA. For results on other benchmarks, please refer to section H.3 All the experiments are conducted on a single Tesla V100 GPU and we analyze the computational cost in section H.8
Liao
7

---

Published as a conference paper at ICLR 2026
Table Main result on CGQA. Methods with CompSLOT are denoted with a postfix "f" Methods
rehearse old samples are denoted with a postfix "*". We report results over 3 trials with (mean + 95% confidence interval). We replace the backbones of all methods to Imagenet-2 - ViT-B/16.
Continual CA (%) f
CFST
Methods
AA (%) f
FF (%) +
Hn (%) f
R
CPrompt CPrompt
60.179:11.699 46.753+0.570
15.670=0.950
78.063=00.17
0.964
48.537::0.427
61.483++
18.315=1.111
79.091+
0.969
ADAM adapter
41,930+:1.14)
53.983++0:444
13.800
68.649940.25
0.932
ADAM adapter i
49.480 1.201
60.989++06441
12.8% + +0.399
+
0.958
RanPAC
65.810+:08802
75.504
10.515=
78.868 +0,918
1.016
RanPAC
66.753+00867
76.584:10.003
10.219:102281
79.815=00822
1.032
EASE EASE
47.657=11499
59,475+4..774
18.215:1.107
79.713 449
0.996
49.323+1..16
62.603
22.470==24472
82.887
1.001
CoFiMA
65.107:10.508
13.227
05.248
86.711=1.448
1.011
CoFiMA 1
66.170:10.578
74.322+0.463
+.2004+0.880
88.297++0.278
1.017
FOSTER*
60.863++0.271
68.800+0:496
2,441 + 0.122
89.791 00.06
1.087
FOSTER* t
66.290+1.4451
7.6828+2.619
6.470=5.770
89.910=1.7710
1.154
DER
52.003+1.019
11.775=1.699
40. 1..222.9907
90.11911.5100
1.080
DER
54.900+11.993
66.020+11.099
38.941
88.986
1.096
MEMO
56.553+1 58.653+11.49
66.462:10.702
9.339
82.425
1.029
MEMO
68.037+:1.499
8.944
84.003+14511
1.050
Metrics For continual training stage, we report the average accuracy of all tasks after training (xpred((x1))) [Pr), v)], the average cumulative accuracy for each task CA $ Σt T-1+1 Ex w) (predd v)]. and average forgetting for each task FF 1 En E ()) (prred Pt). y) AA, where Di is the testing dataset for task and A ((,-)) is the equal function. After training on all continual tasks, specifically for CGQA and COBJ, we perform CFST on five compositional test suites including sys, pro, sub, non, noe, which contain novel recombinations, more combinations, shifting attributes, seen combinations, novel concepts of testing samples, respectively. We generate 300 few-shot tasks for each test suite. For clarity, calculate the Harmonic mean (i.c., Hn 3/(1/sy) 1/pro + 1/sub),, Hr 2/(1 /non + 1/noc), as suggested in Liao et al. (2024). Then we report Hn and the ratio of Hn and Hr (i.e., to section H.l Larger Hn and R indicate that the extracted features have better compositional generalization performance.
the last task AA ET
Hn/Hr), For detailed results on each compositional test suite, please refer
5.2 RESULTS
Overall results We report the statistical results in Table Across all baselines, CompSLOT con- sistently enhances performance, with the most significant improvement observed ADAM+adapter (absolute gain: +7.550 in AA). Notably, CA and FF demonstrate consistent superiority over other methods (except CPrompt and FOSTER, because the original methods do not perform well on the finished tasks, thus, forget less), indicates that our CompSLOT not only mitigates catastrophic forgetting of old tasks but also preserves strong forward adaptation to novel tasks. This robustness is primarily attributed CompSLOT's improved compositional generalization (manifested by higher Hn and R scores), confirming its ability to learn latent conceptual units and dynamically compose them for robust classification across diverse methodological frameworks.
Learning curve Figure 3a shows the learning curves of all methods the 10-10 tasks from CGQA. We observe that concept learning significantly improves continual learning performance across the entire training process, demonstrating its ability to stabilize learning and mitigate forgetting.
8

---

Published as a conference paper at ICLR 2026
fanAKC
ADAM
XDAM
85
1.00
69t plap?
90
0.96
55
10
<
0.84
n0
0.82
30
0.90
Chrome
22 40
20 00
00
Melicdda
Number Classes
Number of Classea
(a) 10-10 tasks
(b) 5-5 tasks
Figure 3: Learning curves and histograms of methods with and without CompSLOT CGQA a) 10-10 tasks and b) 5-5 tasks. Slot is the case directly using the primitive representation and a cosine similarity classifier for the continual tasks.
Long task sequence Figure 3 presents the comparative performance analysis across a challenging long-task sequence of 5-5 CGQA tasks. The results reveal that CompSLOT consistently enhances both continual learning accuracy and compositional generalization performance, even when the slot attention module globally shared across all continual tasks. This finding underscores the remarkable robustness of employing slot attention mechanisms for boosting concept learning in CL scenarios. Notably, the stable improvement suggests that CompSLOT effectively captures transferable compositional knowledge, enabling better adaptation across sequential tasks without task-specific customization.
Ablation studies To evaluate the contribution of each proposed component, conduct com- prehensive ablation experiments, with results presented in Table 2 First, rule the possibility that performance gains stem solely from increased model capacity, we expand the hidden representation dimensions (denoted as "+param") RanPAC and CPrompt (see sec- tion for details) to match the parameter count of RanPAC i and CPrompt f. respectively. We further perform the following controlled exper- iments: 1) Primitive loss ablation (Lp):: We remove the primitive loss term and replace the primitive selection mechanism with simple slot averaging strategy (avg). Slot-selection function ablation: We substitute the softmax operation Equation with alternative weight- ing methods, including: averaging (avg), sig- moid (sig), sign quantization (sign), and cosine similarity (cos). Across both methods, disabling Lp or altering the slot-selection mechanism leads to significant degradations in AA and R scores, demonstrating the critical importance of each component. 1) The primitive loss Lp ensures intra-class consistency, which is vital for reliable primitive selection and, consequently, improved concept- class understanding. On the other hand, using all slots indiscriminately allows less relevant concepts (e.g., background) to dilute class-relevant ones, leading to confusion. 2) The softmax-based weighting (as formulated in Equation provides selection with convex combination of slots in image to ensure the primitive representations of images are within an appropriate range, which makes the training robust. A more comprehensive ablation study can be found in section H.6
Table 2: Ablation results on CGQA.
Methods
Lp
La
AA (%) 1
R1
X+param
X
65.080
1,010
Xavg
/
58.220
0.969
Vavg
65.870
1.003
RanPAC
/ /sig
65.950
1,020
/ /sign
65.140
1.006
/ cos
63,910
0.989
/ soft
66.753
1.032
X+param
X
46,300
0.969
Xavg
40.230
0.952
Vavg
47.690
0.958
CPrompt
/sig
48.080
0.961
/sign
47.780
0.966
/cos
47.410
0.964
ssoft
48.537
0.969
Concept learning and visualization To evaluate whether the learned concepts align with the ground truth, we establish evaluation experiments and design six metrics including the slot represen- tation MAE and the slot mask mIOU. Due to page limit, we describe the details of the metrics and the experiments in section H.2 After that, visualize the learned concepts and compare primitive similarity with ground truth concept similarity in Figure We observe that CompoSLOT consistently identifies Other shoes, Person, and Chair, which important concepts (primitives), in unsuper-

---

Published as a conference paper at ICLR 2026
Image TO
T1
Image
TO
T1
TO
T1
Logit similarity
Concept Sim
Varila
Person
Perkan
Chair
Primiive
Sim
CampSLOT
Figure 4: Concept learning visualization and fruits of primitive-logi knowledge distillation on COBI. Left: Examples visualization of learned slots that associated with the corresponding concepts after finish TO and TI tasks. Red color indicates high value in slot attention masks. Middle: Primitive (comparing with ground truth concept) cosine similarity matrix for ADAM+adapter i on 30 images in TO (left) and additional 30 images in (right)). Right: Logit cosine similarity for ADAM+adapter with and without CompSLOT on 30 images in TO. Takeaway: The learned primitive successfully mimics concept statisties and CompSLOT successfully distills pair- pair-wise primitive similarity into logits. Red box shows evidences.
vised manner. The frequent existence of concepts between different tasks shows a concept rehearsal phenomenon: although class labels change, many visual concepts are shared and recur across tasks, helping stabilize the primitive selection. The learned primitives mimic concept statistics in terms of cosine similarity and the proposed primitive-logit aligmment loss successfully distills pair-wise primitive similarity into logits, which demonstrate that the models make decisions by additionally considering low-dimensional concept combinations instead of only relying on high-dimensional features. For the details of concept and similarity visualization, please refer to section
Influences of hyperparameters CompSLOT introduces several hyperparameters mainly in the following three mechanisms: concept learning stage, slot attention architecture, and primitive-logit knowledge distillation stage. We conduct comprehensive experiments to investigate the influences of each hyperparameter in section H.2] Here specifically showcase the effect of primitive-logit alignment loss coefficient 3 when continual training CPrompt on the first three tasks of the 10-10 CGQA example. The results shown in Table 6 We observe that AA increases increases but decrease after a threshold (around 2). This indicates that an excessively large B hinders the effectiveness of CPrompt's smooth regularization, leading conflicts. However, within an appropriate range, our CompSLOT works effectively with CPrompt's 's smooth regularization.
CONCLUSION
This work propose CompSLOT,, framework introducing concept learning into the continual learning paradigm for foundation models. The proposed primitive selection mechanism effec- tively extracts class-relevant concepts while maintaining robustness across extended task sequences. Meanwhile, the primitive-logit knowledge distillation mechanism enforces concept-based sample similarity regularization, enabling lightweight adaptation to diverse CL methods with foundation models. Experimental results confirm that the performance improvements stem from enhanced compositional generalization, offering novel concept-level perspective for the continual learning community. A limitation of our current approach is that concept learning must precede providing conceptual self-supervision to the CL task. Future work will explore end-to-end integration of our mechanism into the continual learning pipeline and study the joint effect when combining with regularization methods that also manipulating the logits. We hope this research inspires further advancements in developing resilient and interpretable vision models.
ETHICS STATEMENT
We hereby affirm our strict adherence to the ICLR Code of Ethics. We have carefully considered the ethical implications of our research throughout the entire process of study design, data collection, experimentation, and manuscript preparation, and we confirm that our work does not violate any of the principles outlined in the ICLR Code of Ethies. All datasets used, including ImageNet-R,
10

---

Published as a conference paper at ICLR 2026
CIFARIOO, CUB200, CGQA, COBJ, were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. No personally identifiable information was used, and no experiments were conducted that could raise privacy security concerns. We are committed to maintaining transparency and integrity throughout the research process.
8 REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research presented in this paper. To facilitate the replication of our results and the verification of our findings, have provided comprehensive implementation details in section E.] Additionally, the datasets we used, are publicly available, ensuring consistent and reproducible evaluation results.
ACKNOWLEDGMENTS
We thank all the anonymous reviewers for their constructive suggestions on improving this paper. This work was supported in part by grants from the National Natural Science Foundation of China (No. NSFC62441236), the Research Grants Council of the Hong Kong Special Administrative Region, China (GRF Project No. CityU1 1212524), National Natural Science Foundation of China (Grant No. 62376115), and Guangdong Provincial Key Laboratory (Grant No. 2020B121201001
REFERENCES
Alessandro Achille, Tom Eccles, Loic Matthey, Chris Burgess, Nicholas Watters, Alexander Lerch- ner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. Advances in Neural Information Processing Systems, 31, 2018.
Susmit Agrawal, Deepika Vemuri, Vineeth N Balasubramanian, et al. Walking the web of concept- class relationships in incrementally trained interpretable models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 15320-15329, 2025.
Kyra Ahrens, Hans Hergen Lehmann, Jae Hee Lee, and Stefan Wermter. Read between the lay- ers: Leveraging multi-layer representations for rehearsal-fie free continual learning with pre-trained models. arXiv preprint arXiv:2312. 2023.
Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. causal view compositional zero-shot recognition. Advances in Neural Information Processing Systems, 33:1462-1473, 2020.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey, arXiv preprint arXiv:2012.09822 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Michael Chang, Tom Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. Advances in Neural Information Processing Systems, 35:32694-32708, 2022.
Hung-Jen Chen, An-Chich Cheng. Da-Cheng Juan, Wei Wei, and Min Sun. Mitigating forgetting in online continual learning via instance-awe aastre parameterization. Advances in Neural Information Processing Systems, 33:17466-17477 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Muchine Learning, pp. 1597-1607, 2020b.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decodee for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daclemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, October 2014.
11

---

Published as a conference paper at ICLR 2026
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Viston and Pattern Recognition (CVPR), pp. 9285-928899 9995, 2022.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11483-11 1149,, 2023.
Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
pp.
Alexander L Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. Differentiable programs with neural libraries. In International Conference on Machine Learning, pp. 1213-1222, 1222, 2017.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208. 2020.
Stephen Grossberg. Studies of mind and brain: Neural principles of learning, perception, develop- ment, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of nu-oofdiittibution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8340-8349, 2021.
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Constrained few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6057-9067, 2022.
Heinke Hihn and Daniel A Braun. Hicrarchically structured task-agnostic continual learning. Machine Learning, 1 112(2)655-66) 2023.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In International Conference on Learning Representations, 2023.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self- training with patch-order permutation for object-centric learning with autoregressive transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 227762227778 June 2024.
Karthikeya Ramesh Kaushik and Andrea E Martin. Modelling compositionality and structure dependence in natural language. arXiv preprint arXiv:2012.02038, 2020.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang. Mare van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2020.
Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Alain Pagani, Didier Stricker, and Muhammad Zeshan Afzal. Learning attention propagation for compositional zero- shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3828-3837, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 13:18661-18673 2020.
12

---

Published as a conference paper at ICLR 2026
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 4015 (415-4026,4 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Babbbska,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526 2017.
Avinash Kori, Francesco Locatello, Fabio De Sousa Ribeiro, Francesca Toni, and Ben Glocker. Grounded object-centric learning. In International Conference Learning Representations, 2023,
Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei. Fine-grained recognition without part annotations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5546-5555, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97 1955.
Shivanand Kundargi, Kowshik Thopalli, and Tejas Gokhale. Sequentially acquiring concept knowl- edge to guide continual learning. In Second Workshop on Visual Concepts, CVPR, 2025.
Songning Lai, Mingqian Liao, Zhangyi Hu, Jiayu Yang, Wenshuo Chen, Hongru Xiao, Jianheng Tang, Haicheng Liao, and Yutao Yue. Learning new concepts, remembering the old: Continual learning for multimodal concept bottleneck models. arXiv, 2024.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of seguuence-t--eqqeeee recurrent networks. In International Conference on Machine Learning, pp. 2882-2882,, 2018.
Minh Le, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Ngo, Nhat Ho, et al. Mixture of experts meets prompt-based continual learning, Advances in Neural Information Processing Systems, 37: 119025-119062, 2024.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning. pp. 3925-3934 2019.
Yuxiao Li, Eric J Michaud, David D Back, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27(4)3344, 2025.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1112))223522441 2017.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank udaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23638 23647, 2024.
Weiduo Liao, Ying Wei, Mingchen Jiang, Qingfu Zhang, and Hisao Ishibuchi. Does continual learning meet compositionality? new benchmarks and an evaluation framework. Advances in Neural Information Processing Systems, 36, 2024.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:1 112111555-1111 11538, 2020.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Hongyin Luo, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Online learning of interpretable word
embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1687-1692, 2015.
13

---

Published as a conference paper at ICLR 2026
Arun Mallya and Svetlana Lazebnik. PackNet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In European Conference Computer Vision, pp. 306-324. Springer, 2024.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Mark McDonnell, Dong Gong, Amin Parvanch, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36: 12022-12053, 2023.
Jorge A Mendez and ERIC EATON. Lifelong learning of compositional structures. In International Conference on Learning Representations, 2021.
Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceedings of COLING 2012. pp. 1933-1950, 2012.
Tomáš Musil and David Marešek. Independent components of word embeddings represent semantic features. arXiv preprint arXiv:2212.0099 225.099002 2022.
Tushar Nagarajan and Kristen Grauman. Attributes as operators: Factorizing unseen attribute-object compositions. In Proceedings of the European Conference Computer Vision (ECCV), September 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec. Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-War Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298-30312. 2021.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprinl arXiv:2406.01506, 2024.
Benliu Qiu, Hongliang Li, Haitao Wen, Heqian Qiu, Lanxiao Wang, Fanman Meng, Qingbo Wu, and Lili Pan. CafeBoost: Causal feature boost to eliminate task-induced bias for class incremental learning. In Proceedings of the IEEE/VVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 16016-16025, June 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Ramesh Rahul and Chaudhari Pratik. Model Zoo: A growing brain that learns continually. In International Conference Learning Representations, 2022.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 2021.
14

---

Published as a conference paper at ICLR 2026
Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In International Conference on Learning Reprexentations, 2025.
Mark B Ring. Child: A first step towards continual learning. Machine Leurning, 28(1):77-104, 1997.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Frank Ruis, Gertjan Burghouts, and Doina Bucur. Independent prototype propagation for zero- shot compositionality. In Advances in Neural Information Processing Systems, volume 34, pp.
10641-10653,, 2021.
Paul Ruvolo and Eric Eaton. ELLA: An efficient lifelong learning algorithm. In International Conference on Machine Learning, pp. 507-515, 2013.
Dawid Rymarczyk, łukasz Struski, Jacek Tabor, and Bartosz Zieliñski. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1420-1430, 2021.
Dawid Rymarczyk, Joost Van De Weijer, Bartosz Zieliński, and Bartlomiej Twardowski. Icicle: Interpretable class incremental continual learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1887-1898, 2023.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-e learns to compose. In International Conference on Learning Representations, 2022.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-fre free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909-11919, June 2023.
Andrew Stange, Robert Lo, Abishek Sridhar, and Kousik Rajesh. Exploring the role of the bottleneck in slot-based models through covariance regularization. arXiv preprint arXiv:2336.. 025777 2023.
Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng. Sparse word embeddings using 11 reg- ularized online leaming. In Twetty-Fifth International Joint Conference Artificial Intelligence, 2016.
Hai-Long Sun, Da-Wei Zhou, De-Chuan Zhan, and Han-Jia Ye. Pilot: A pre-trained model-based continual learning toolbox. SCIENCE CHINA Information Sciences, 68(4):14710, 2025.
Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning. Advances in Neural Information Processing Systems, 35:27075-27088 2022.
Zhicheng Sun, Yadong Mu, and Gang Hua. Regularizing second-order influences for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20166-0015,, June 2023.
Fu- Yun Wang, Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 4998414,770 2022a.
Liyun Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang. Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2024.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rchearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022b.
15

---

Published as a conference paper at ICLR 2026
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 149, 2022c.
P. Welinder. S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical Report CNS-TR-2010-000 California Institute of Technology, 2010.
Ross Wightman. Pytorch image models, 2019.
Tailin Wu, Megan Tjandrasuwita, Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosič, and Jure Leskovec. ZeroC: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time. arXiv preprint arXiv:2206. 15049, 2022.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models, Advances in Neural Information Processing Systems, 14.50932-50958 2023.
Hiroaki Yamagiwa, Momose Oyama, and Hidetoshi Shimodaira. Discovering universal geometry in embeddings with ICA. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3014-3023, 2021.
Sin-Han Yang, Tuomas Oikarinen, and Tsui- Wei Weng. Concept-driven continual learning. Transac- tions Machine Learning Research, 2024. ISSN 2835-8856.
Yutao Yang, Jie Zhou, Xuanwen Ding. Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey. ACM Computing Surveys, 57(5))1-38. 2025.
Lu Yu, Haoyu Han, Zhe Tao, Hantao Yao, and Changsheng Xu. Language guided concept bottleneck models for interpretable continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 14976-14986, 2025.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19148-19158, 2023.
Da-Wes Wei Zhou, Qi- Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: To- wards memory-effiieent class-incremental learning. In International Conference on Learning Representations, 2023.
Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (ICCAI), 2024a.
Da-Wei Zhou, Hai-Long Sun, Han-Ji Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23554 23564, 2024b.
Da- Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence,
2024c.
Da-Wei Zhou, Zi-Wen Cai, Hin-Jia Ye. De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 133(3): 1012-1 2025.
Yixiong Zou, Shanghang Zhang, Haichen Zhou, Yuhua Li, and Ruixuan Li. Compositional few-shot class- class-incrementa learning. arXiv preprint arXiv:240S.17022 2024.
16

---

Published as a conference paper at ICLR 2026
A
APPENDIX CONTENTS
CONTENTS
Introduction
2
Related Works
3
Preliminaries
4
Methods
4.1
Concept Learning
Method-agnostic Primitive-Logit Knowledge Distillation
Experiments
5.1
Experimental Settings
5.2
Results
6
Conclusion
10
7
Ethics Statement
10
8
Reproducibility Statement
11
Appendix Contents
17
Discussions
18
Additional Related Works
19
Theorem
19
D. Proof of Theorem
20
D.2 Proof of Thherrmm
20
Hyperparameters and Experimental Settings
20
Pseudo Code
20
Visualization
22
Additional Results
26
Detail CEST Results
26
H.2
Influences of Hyperparameters
26
Results on Other Benchmarks
30
H
Results Other Backbones
31
17

---

Published as a conference paper at ICLR 2026
H.5
Comparing with Other Concept Learning Methods
32
1.6
Additional Ablation Studies
32
Case Studies: the effect of CompSLOT on Finetuning
32
H.8
Algorithm Efficiency Analysis
32
Use of Large Language Models
33
B DISCUSSIONS
Classification Bias from a Concept-Combinatti Perspective The root cause of compromised stability and plasticity often lies in sub-optimal classifier design, particularly when classifiers develop reliance on inaccurate or incomplete feature representations due to concept biases training data. To illustrate, consider scenario where the current vision task T₁ contains human standing by a tree and human inside boat) alongside other classes that lack human-related concepts. In such cases, the learned classifier might develop an over-reliance on distinguishing these two classes based solely on free and boat concepts while neglecting the more critical human attribute. This limited conceptual understanding creates significant generalization problems when encountering unseen concept combinations. For instance, during task 72, a novel image labeled a pig inside boat would likely receive disproportionately high logits for the human inside a boat class due to the classifier's inability to properly disentangle object-class relationships from spatial-contextual cues. Conversely, a human inside a boat image might similarly activate the pig inside a boat class predictions. This conceptual entanglement manifests as catastrophic forgetting in Ti (as evidenced by diminished global accuracies post-training on T2) and severely hampers plasticity for T2 through incorrect plastic responses to novel concept combinations.
specific classes (a
Whether concept sharing a common phenomenon in the real-world? In real-world scenarios, concept sharing is quite common, like, in fine-grained classification cases such as CUB200, and in images with massive objects such as COBJ. This phenomenon also discussed in other works. For example, Welinder al. (2010) claims that fine- fine-grained bird classes share some basic parts, and Krause et (2015) claims that tine-grained categories share similar shapes. In the experimental results, CompSLOT consistently brings significant improvements continual learning algorithms on these real-world cases. In contrast, datasets like CIFAR, which have relatively little concept sharing, uncommon in complicated real-world scenarios.
Fairness issues To demonstrate the effectiveness of our CompSLOT, we list the actions to make the comparison fair as possible:
1. When comparing with and without CompSLOT, e.g., in Table I we used exactly the same backbone for both continual learners and the slot attention, which was the ViT-B/16 backbone pretrained on ImageNet-211 sourced from the Python timm package. When comparing with other concept-based methods in Table TO, we should note that it is impossible to achieve perfect fairness due to the use of external LLMS in CLG-CBM. We used CLIP ViT-B/16 for CompSLOT to extract slots without changing the backbone of the continual learner to show the methoddagnossee nature and the adaptivity of our proposed plugin. When training slot attention, we DID NOT introduce additional supervision, such as concept labels.
Most of the continual learner-related hyperparameters used their default settings, as suggested in the PILOT platform, while for the additional hyperparameters introduced in this work, please refer to section
4. To further ensure fairness and show that performance gains not from the increased model capacity, we also compared with a case extending the number of parameters in an ablation study in section 5
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RELATED WORKS
Continual learning from scratch To mitigate forgetting previously learned vision classification tasks and achieve fast adaptation to new ones, the continual learning community has developed three main families of approaches that do not utilize a pre-trained foundation model:
1. Rehearsal- -based methods (Achille al. 2018: Rolnick al. 2019 Rahul & Pratik 2022 Hersche
et al. 2022; Sun et al.) 2022 Qiu et al., 2023 Sun et al. (023) store memory-efficii samples or
features from past tasks for reviewing knowledge. However, such buffers can cause significant
memory overload as the number of tasks increases.
Regularization-bazt bbasbi methods (Lopez-Paz & Ranzato 2017] Li Hoiem 2017 Kirkpatrick et al.
2017 Achille et al. 2018 Hersche et al. (0022) constrain gradient updates to preserve important
knowledge from old tasks, but this can limit the adaptation capability to new tasks.
3. Architecture-based methods (Mallya & Lazebnik,
Douillard et al.
2022.
Ring 1997: Ruvolo
& Eaton 2013: Gaunt et al. Li et al.
2019 Rajasegaran et al. 2019 Chen et al. 2020a
Mendez & NATONN22021 2000sssaeeea et al. 2021 Rahul & Pratik 2022;Hihh & Braun 2023
aim to create new modules for upcoming tasks, making the determination of module composition
crucial for different tasks.
D THEOREM
Firstly, we define concepts as the ground truth slot decomposition of an image. Since slot attention exhibits permutation equivalence w.r.t. the order of the slots (and masks) (Locatello et al. 2020).
we regard (S,A} as the corresponding set representations of {SS, A}. where S { and with 8; E RDD, and a; R. being the i-th row of S and A, respectively.
A= (a; K E=1.
Definition 3 (Concept& & Disentanglement. equivalent to Def. 1). Let æ be image, then {8, A)
is a disentangled decomposition of x (a.k.a., concepts and corresponding attention regions), if I) a, aj E A, a, RN a; 1 aj., and 2) S satisfies arg mins Ex.,yes (imm(;;, sy)1, wheres; RDD and is the absolute value, . is orthogonal symbol, and sim(-,. is an arbitrary similarity score function, e.g., cosine similarity.
Remark 2 (Requirement 1: Disentanglement) The competitive spatial attention and the limited capability of a slot naturally achieve the orthogonality of A. In practice, S is encouraged to be orthogonal (slots bind to different concepts in æ:) but not ideal since there are some semantically similar concepts. c.g., grass and leaves. Such disentanglement structure is also mentioned in Park al. (2024): Li al. (2025).
Definition 4 (Primitives, equivalent to Def. 2). Let X S" be image set labeled y and the corresponding set of concept sets, and S E S", then a concept subset P C S is primitives of S, if VS' E 8", P S'.
Remark 3. Although i defined at the image level, can also say that it is unambiguously y's primitive set, denoted pu, In general, P + S, because there are always image-specific concepts in the image, e.g., background.
Theorem 2 (Requirement 2: Intra-class consistency, equivalent to Theorem. 1). Consider S1 S and two corresponding largest primitive sets Pi C Si, P2c C Sr are identical, i.e., Pi = P2 and ||Pil = M. where || 11 is the cardinality of set. In other word, consider the pair-wise ordered sets (Po Pa} = match(Pi PI. P2), where match(.,.) is a matching algorithm (without loss of generality, Hungarian algorithm (Kuhn 19551), then the corresponding matched concepts should be the same: M
Pi- = {x1
1r P2 = $ Ji=1 and Vi E (1...., [immsim(s]] |
=1. =1
Theorem 3 (Requirement
Inter-class concept sharing). If there shared primitive subset between y1, 1/. all images in X X2 should contain this subset. If 3P C. p C Pin, |||||| M > 0, then VS, E S E S02, C SS,, S2.
Remark 4 (Requirement 4: Inter-task consistency). After trained on future tasks, the concept sets of the same should be not changed. In 7 CL. tasks, Vn E X" S < T, consider the extracted concept sets {S*]7' JT= after task t E {u...., then Vt,, t2 E {u.... T), St St2
CompSLOT implicitly encourages Requirement 1 via slot attention's 's soft-clustering and supports Requirements 3-4 empirically (Figure 2 and the visualization experiments in section G Require-
19

---

Published as a conference paper at ICLR 2026
ment 2 is enforced through primitive loss, described in Equation 3 ensuring slot stability across class instances.
D.1 PROOF OF THEOREM 1
Proof. By the definition of Pi, P2 as the primitive sets of S1, S2, respectively, and that S1, S2 E SV: without loss of generality, P2 is also primitive set of Sj. Thus, P2 C S1. Assume, for the sake of contradiction, that there exists a concept 8, such that E P2 and 8 E Pı, i... Pi P22. Since Pi is the largest primitive set of S1, must have P2 C PI and vP C Pi,8 initial assumption that E P2.
This contradicts
Therefore, the theorem holds.
Remark 5. The matching algorithm facilitates concept alignment across different sets, thereby abling the computation of our proposed evaluation metrics in section H. as well as supporting the visualizations presented in section G However, this alignment process introduces significant computational overhead that renders it impractical for integration within our distillation framework. To address this limitation, we propose attention-based primitive selection mechanism (detailed in section 4.1) that ensures permutation invariance to concept ordering in the extracted primitives, effectively eliminating the need for explicit concept matching. This design choice maintains compu- tational efficiency while preserving the critical semantic relationships required for reliable evaluation and visualization.
D.2 PROOF OF THEOREM 2
Proof. Assume, for the sake of contradiction, that there exists P',S' and P p C puu,, ||P'|| = M > 0, E Sth (or S such that P' & S'. By the definition of pv as the primitive set for S E Sth, thus P' S'. This contradicts our initial assumption that P' S'.
Therefore, the theorem holds.
E HYPERPARAMETESS AND EXPERIMENTAL SETTINGS
The hyperparameter settings for the concept learning stage are summarized in Table 3 with key values tuned through validation. For the concept knowledge distillation phase, maintain fairness in comparison by adopting the platform-defaull hyperparameters from the PILOT framework for both standard CL baselines and CompSLOT-enhanne
variants, with additional parameters introduced in section 4.2 detailed in Table 4 All configurations employ an 80-20 train-validation split using randomly sampled validation set. To ensure consistent model capacity across methods, all algorithms utilize the ViT-B/16 backbone pretrained on ImageNet-21K as the shared feature extractor unless otherwise stated. When comparing with other concept-based models in Table I0 we use CLIP ViT-B/16 the CompSLOT" 's backbone to extract slot, suggested in Yu et al. (2025). The backbone parameters are sourced from the Python timm [Wightman 2019) package. For the compositional testing in CGQA and COBJ, used randomly generated 300 few-shot tasks for each test suite, as suggested in Liao et al. (2024). For ablation studies specifically examining CompSI.OT LOO's impact, we appropriately scale model capacities through expanded hidden representations: RanPAC: Increased feedforward layer width (ffn_num) from 64 to 256; CPrompt: Extended prompt length (prompt_len from 50 65 tokens. These adjustments ensure fair comparison by matching representational capacity when introducing our architectural modifications, enabling more reliable evaluation of CompSLOT's actual contribution beyond simple capacity increases.
F
PSEUDO CODE
In the main paper, we propose a two-stage procedure, including concept learning (aiming to extract concept-level representation by performing slot representation training and primitive selection) and
20

---

Published as a conference paper at ICLR 2026
Table Detail hyperparameters for concept learning stage in our main experiments.
Hyper-parameter
Value
Optimizer
Adam
LR scheduler
Cosine
LR (1-st task)
le-4
LR (others)
le-5
LR (min)
le-8
Batch size
256
Weight decay
Epoch
50
D
128
K
10
Slot refinemnt iterations Slot decoder hidden embedding dim
N₂
Linear with ReLU (128 1128-4256+2256-22
T:
100
10
10
Table 4: Detail hyperparameters for concept knowledge distillation stage in our main experiments.
Methods
8
To
CPrompt
10
0.05
ADAM adapter
10
0.5
RanPAC
15
0.5
EASE
10
0.1
CoFiMA
1
0.001
FOSTER
2
0.05
DER
7
0.01
MEMO
0.05
0.1
concept knowledge distillation (aiming to distill sample-wise concept-based similarity into logits). We summarize the training framework of CompSLOT in Algorithm [ Specifically, we perform concept learning in Lines 4-9, The slot attention and primitive selection module are initialized at first. For each batch of samples in task we perform Algorithm 2 and use the obtained primitive loss and reconstruction loss to train slot attention and primitive selection modules in Line 6. After E epochs of training, we perform concept knowledge distillation in Lines 11-18. We calculate pair-wise primitive similarity and obtain primitive-logit alignment loss with Equation ii Line 15. We detail the slot representation learning in Algorithm 2, Specifically, we first obtain semantic patch features in Line 3. Then, we slot attention module to decompose it into set of slots in Line 4. Next, reconstruct the patch feature and obtain the reconstruction loss Lines 6-8, After that, we calculate the primitives in Lines 10- 12 and obtain primitive loss with Equation 3 in Lines 14-15.
21

---

Published as a conference paper at ICLR 2026
Algorithm I Continual Learning Framework
1: Input: tasks T, tasks DD1 pi...., D epochs E, candidate CL method CTT(-10,,0))
2: Initialize slot attention and primitive selection module.
3: for from to do
4: /* Concept Learning */
5: for from to E do
6: Sample batch of images (x, y) D'.
7:
Perform Algorithm obtain primitives sP, contrastive primitive loss Lp. and reconstruc- tion loss Lye-
8:
Listot = Leee + ollp.
9: Backward loss and update.
10:
end for
1* Concept Knowledge Distillation */
12:
for from to E do
Sample batch of images (az, y) D'
14:
Perform Algorithm 2 obtain primitives SP without collecting gradients. Perform CL method to obtain logits C'L On) and task loss Lee- Calculate primitive-logit alignment loss La.
16:
17:
Ler = Lee + La-
18:
Backward loss and update.
19:
end for
end for
Algorithm 2 Slot Representation Learning
1: Input: Image batch {a:: CL backbone 8,. # slots K, slot dimension D., # epochs E,,
2: Output: Primitive s", contrastive primitive loss Ly, reconstruction loss Lye-
3: Obtain semantic patch features E = f(x, [0,][ :].
4: Obtain a of K slots and the corresponding attentions {S, A).
5: /* Reconstruction Loss */
6: Add position embedding for each patch: S posn
7: Decode S' and re-construct using A: dd)
8: Lyee = E El12-
9: 1* Primitive Selection #/
10: Obtain Mapped slots $ = (ana
Obtain weights for each slot wp (T,
Obtain primitive sP _. S.
13: 1 Contrastive Primitive Loss #/
Obtain normalized similarity d',, and softmax primitive similarity di,; for image sample 2;, Zy.
15: Obtain primitive loss Lp.
G VISUALIZATION
This section investigates how the CompSLOT framework enhances continual learning performance by first demonstrating through slot attention mask visualizations across various benchmarks that CompSLOT successfully identifies important concepts (primitives) in an unsupervised manner, and then by presenting similarity matrix visualizations of ground truth conceps/prinntive/featuee//////e for specific algorithms to illustrate the regularization effects that improve model compositional generalization and stability during continual learning. We attribute this robustness "concept rehearsal": although class labels change, many visual concepts are shared and recur across tasks, helping stabilize the primitive selection weights. This is also discussed Lai et al. (2024).
Concept learning We evaluate CompSLOT on CGQA. COBJ, ImageNet-R. and CIFAR100 bench- marks by randomly selecting three representative images from each class. The extracted slot masks visualized in Figures and respectively.
22

---

Published as a conference paper at ICLR 2026
Weight
Image
Sum
S
Stot 1
Stot
Stot3
Stot
Stot
Stot
Stot
Stot
Slot
TO
0.00
T1 0:00
Door Plate
1:100
TO
T1
0.10
Door Grass
TO
0:10
Chair Sign
Figure 5: Visualization of learned slots on 30 randomly sampled images (3 images for each class in the first task of the 10-10 tasks) CGQA. Top row: Primitives (weighted-sum of slot masks weighted by wp) for 30 images. Bottom rows: Three examples of images from classes (Door Plate), (Door Grass), and (Chair Sign) after being trained on the first task (TO) and on the second task (TI). From left to right: up, origin image, primitive (weighted-sum of slot masks), and 10 slot masks, respectively. Takeaway: CompSLOT successfully extracts primitives without any concept label.
On CGQA, the weighted slot masks (using weights wp)) effectively localize class-relevant concepts each image. For instance, in the Door Plate class, slot 7 consistently captures the Plate concept while slot 8 focuses the Door, demonstrating precise concept disentanglement. Notably, the learned primitives maintain visual consistency across tasks, that the primitive representation after task TO closely resembles that after T1, confirming the stability of CompSLOT. This phenomenon
was similarly observed in Figure 2
The more challenging COBJ benchmark presents similar results. For an image in the Other-shoes Person class, slot 5 accurately identifies the Other-shoes concept while slot correctly localizes the Person, even in this complex compositional setting.
When examining ImageNet-R and CIFAR100 with K = slots, we observe that the primary concept corresponding each class label is reliably identified, and the representations maintain discriminative power while preserving semantic consistency. However, the concept sharing is visually rare between classes, as demonstrated by the distinct slot activation patterns for different classes.
Primitive-logit alignment We conduct in-depth visualization analysis understand the perfor- mance improvement of CompSL.OT on COBJ, using ADAM adapter as a representative example. We visualize 30 images for TO and 60 images for TI (10 old classes and 10 new classes). Figure 9 presents the cosine similarity matrix visualizations including: (a) Ground truth multi-hot concepts; (b) Extracted primitives; (c) Feature representations; (d) Final logits. The red boxes highlight two pairs of classes with concept sharing: (Other-shoes Person) and (Other-shoes Person Sneaker), as well as (Person Sneaker) and (Other-shoes Person Sneaker). CompSLOT successfully captures these shared concepts in the primitive representations (Figure 9p) and effectively distills them into the final
23

---

Published as a conference paper at ICLR 2026
Weight
Image
Sumn
S
Stot
Slot
Stat
Stot
Stot
Slot
Stot
Stot
Slot
0.2
TO
6.15
T1
0.00
Other-shoes Person
10
T1
0.00
Car Street-light
0.
04
03
Chair Person
Figure 6: Visualization of learned slots on 30 randomly sampled images (3 images for each class in the first task of the 10-10 tasks) on COBJ. Top row: Primitives (weighted-sum of slot masks weighted by wp) for 30 images. Bottom rows: Three examples of images from classes (Door Plate), (Door Grass), and (Chair Sign) after being trained on the first task (TO) and the second task (TI). From left to right: up, origin image, primitive (weighted-sum of slot masks), and 10 slot masks, respectively. Takeaway: CompSLOT successfully extracts primitives without any concept label.
Weight
Image
Sum
Siot
Siot
Siot
Stots
S
Weight
Image
Sum
Sler
Slet
Stat
Stat
Stat 4
Figure 7: Visualization of learned slots on 60 randomly sampled images (3 images for each class in the first task of the 20-20 tasks) ImageNet-R. Top row: Primitives (weighted-sum of slot masks weighted by wp)) for 60 images. Bottom row: Two examples of images after being trained on the first task (TO) and the second task (T1). From left to right: Wp. origin image, primitive (weighted-sum of slot masks), and slot masks, respectively. Takeaway: CompSLOT successfully extracts primitives without any concept label, and the concept sharing is between classes.
logits (Figure (d). Notably, this alignment process also induces regularization at the feature level, as evidenced by the more coherent feature representations shown in Figure 96
24

---

Published as a conference paper at ICLR 2026
Wnight
Image
Sum
Blato
Biat
Biot
Biot
Siot
Weight
Image
Sum
Sloto
Slat
Slat
Slat
Slot
TO
Figure 8: Visualization of learned slots on 30 randomly sampled images (3 images for each class in the first task of the 10-10 tasks) on CIFAR 100. Top row: Primitives (weighted-sum of slot masks weighted by wp) for 30 images. Bottom row: Two examples of images after being trained on the first task (TO) and on the second task (TI). From left to right: Wp. origin image, primitive (weighted-sum of slot masks), and slot masks, respectively. Takeaway: CompSLOT successfully extracts primitives without any concept label, and the concept sharing is rare between classes.
T1
TO
1.0
1.00
0.75
0.8
0.50
0.6
0.25
0.00
0.4
-0.25
0.2
-0.75
a) Concept
b) Primitive sim
Vanilla
With plugin
Vanilla
With plugin
1.0
1.0
0.8
0.8
0.6
0.6
0.4
c) Feature sim
d) Logit sim
Figure 9: Visualization a) concept; b) primitive; c) feature; d) logit cosine similarity matrices on sampled images (three images for each class in the first task TO and second task Tl of the 10-10 tasks) on COBJ. a) Left: Multi-hot concept cosine similarity matrix of 30 images for TO; right: Multi-hot concept cosine similarity 60 images (from the first-2 tasks TO and T1). b) The primitive cosine similarity of the corresponding images. We use the learned pair-wise primitive similarity to mimic the statistics of the pair-wise concept similarity and regularize logits. c) Left: The learned feature cosine similarity matrix of 30 images in TO for ADAM + adapter; right: The learned feature cosine similarity matrix of 30 images in TO for ADAM adapter f. d) The logit cosine similarity of the corresponding images in c). Takeaway: The learned primitive successfully mimics concept statistics without concept supervision, and our L, successfully distills pair-wise primitive similarity into logits and affects the feature representations (as demonstrated with the regions marked with red box), while ADAM + adapter does not capture this concept sharing statistic.
We further validate CompSLOT ImageNet-R, standard CL benchmark without ground truth concept labels. Figure 10 shows the case performing CompSLOT on FOSTER, Our slot attention mechanism identifies shared concepts across six images (highlighted in red boxes), particularly revealing consistent "Fabric" concept (Figure[TOaa) This automatic discovery of hidden relationships demonstrates CompSLOTTss ability generalize concept learning across different benchmarks.
25

---

Published as a conference paper at ICLR 2026
T0
T1
1.0
Class
0.8
0.6
Class 2
0.4
0.2
a) Images
b) Primitive sim
Vanilla
With plugin
Vanilla
With plugin
1.0
1.0
0.8
0.6
0.5
0.4
0.0
0.2
0.0
-0.5
c) Feature sim
d) Logit sim
Figure 10: Visualization of a) images related to red box; b) primitive; c) feature; d) logit cosine similarity matrices on sampled images (three images for each class the first task TO and second task T1 of the 20-20 tasks) on ImageNet-R. a) Six images from two classes in TO which corresponding to the red box. b) The primitive cosine similarity of the corresponding images. c) Left: The learned feature cosine similarity matrix of 60 images in TO for FOSTER; right: The learned feature cosine similarity matrix of 60 images in for FOSTER f. d) The logit cosine similarity of the corresponding images as in c). Takeaway: The learned primitives show that CompSLOT discovers hidden relationships based on concept as demonstrated with the regions marked with red box, while FOSTER does not capture this concept sharing statistic.
The consistent performance improvements reported in Sections and H3 validate that CompSLOT effectively captures meaningful semantic relationships, leading to better generalization and composi- tional learning capabilities.
H ADDITIONAL RESULTS
DETAIL CFST RESULTS
The statistical analysis of each compositional test suite for the 10-10 tasks CGQA benchmark is presented in Table [ All accuracy metrics are reported with their corresponding +95% confidence intervals quantify statistical significance. The key metrics include:
Hn: Harmonic mean of compositional testing metrics (systematicity sys, productivity pro, substitu-
tivity sub);
Hr: Harmonic mean of reference testing metrics (Non-novel non, Not compositional noc);
Ha: Harmonic mean across all test types;
R=Hn/Hr: Ratio measuring compositional generalization improvement.
The results consistently demonstrate superior performance in both R and Hn (except for DER t). confirming CompSLOT's ability to enhance compositional generalization, particularly for system- aticity and productivity properties. This aligns with our hypothesis that the slot plugin improves compositional reasoning capabilities. However, as previously reported in Liao et al. (2024) for ViT-based architectures, we observe no significant improvement in substitutivity, suggesting inherent limitations of ViT feature extractors in dealing with attribute shifting (e.g., color).
H.2 INFLUENCES OF HYPERPARAMETERS
In this section, we investigate the effect of the introduced hyperparameters the slot module w.r.t. the slot extraction performance and in the primitive-logit alignment loss. Without loss of generality,
26

---

Published as a conference paper at ICLR 2026
Table 5: Detail CFST results. We report the average with + 95% confidence interval.
Methods
sys
pro
sub
Hn
CPrompt CPrompt
73.533=1.552 7.833
75.367=1.001
78.063-10.817 85.967= 84.600=
08.13
79.091
ADAM + adapter
63.400=10.244
68.667110.38
74.833=+0.107
68.649-0.225
ADAM + adapter i
68.533=0..62
75.033+L0.53
80.400=0.0092
74.335
RanPAC
74.867=0.912
0..56
83.667=1.536
70.918
RanPAC i
7..633
80.600
83.433=1.783
09.815
EASE EASE i
74.900=00423
80.567
8..23
79.713+0:449
78.267=0.500
0.509
86.200=0.480
82.887+0.320
CoFiMA
83.100=1.135
86.767+00267
90.600=0.606
86.711
CoFiMA
0.324
08.96
91.767=0.141
88.297+0.278
FOSTER
86.900=10.514
91.400-10.489
9.9233
000091
FOSTER 1
0.660
91.733+0.979
90.500=0.733
897110
DER DER f
87.000
91.33
91.033-0.828
90.11910.510
86.567=0.509
90.300+00.66
90.200=0.320
88. 88.986110.129
MEMO
28
82.500
82.425 0.503
MEMO f
1..733
85.133
87.533+
8.450
Methods
Hr
R
CPrompt CPrompt
76.400=00.973
86.033+10:477
80.96
0.964
77.167=0.681
86.533+0.601
8.4580
0.969
ADAM adapter ADAM + adapter f
66.16710.930
82.867:00.15
73.580-0.809
0.932
71.267=0.417
84.967
7..216-0.323
0.958
RanPAC RanPAC |
75.267+1.063
80.033+10.833
0.813
1.016
76.600=0.606
1..13
77.314=0.440
1.032
EASE EASE i
76.400-00666
83.96710.141
80.00411.420
0.996
79.900
85.86710.541
82.775
1.001
CoFiMA
03.96
0..23
85.529-00353
1.011
CoFiMA
85.600
09323
0.544
1.017
FOSTER
89.833-0.141
76.4331
82.592
1.087
FOSTER t
1..700
2.999
77.847+1.999
1.154
DER
89.967=0.373
1.689
0.833
1.080
DER t
88.600=0.370
1.536
01.951
1.096
MEMO
80.533=1.802
79.60010.489
80795
1.029
MEMO I
82.433+22211
7..700+2.080
79.985+1.847
1.050
27

---

Published as a conference paper at ICLR 2026
Table 6: Varing / results on CPrompt 10-10 CGQA (the first three tasks).
B
0.1
0.5
2
10
50
AA (%)
68.33
68.43
69.67
70.17
70.87
70.40
70.13
69.00
f2
f3
f2
f3
F2
0.2 p.a. 3. 8
P. P. 0 8.0
0. 4-
f4
f1
f4
f4
f1
f5
16
f5
16
f5
16
Te: 500.0
T3.
1000.0
50.0
500.0
10.0
200.0
F; 1.0
Tt 1000.0
Tel 1.0
a: 200.0
a: 1.0
To 100.0
Tp: 0.1
t;: 100.0
Tτl 0.1
a: 100.0
a: 0.1
To: 50.0
ty: 10.0
(a) Primitive Loss Coefficient
(b) Primitive Loss Temperature
(c) Slot Selection Temperature
Figure 11: Radars of different hyperparameters in slot representation learning.
we report the model performance after training on the second task of 10-10 tasks CGQA in this section.
Metrics We learn slot representation S, attention mask A, and primitive representation s as intermediate products of the forwarding process. Thus, it is necessary to design quantitative metrics to represent the performance the learned slot as follows:
Primitive-label matching score: fl MMA((d", d²), where d and d are described in Equation
Primitive-concept matching score: f2 MAE(d d'), where de is similar with dM but the one-hot label is replaced with the multi-hot concept label. Note that the concept label is only used analyze the performance of the learned slots and never seen during training.
Task-wise matched attention mask mloU: f3 = Mean, (loU(AA- AP)). where loU(:.
is the intersection over union metric and A Ar are matched attention sets (by Hungarian algorithm) extracted from the same image by the learners trained after task t 1 and t, respectively.
Task-wise weighted attention mask mloU: f4 = Mean, (Io W0 (T).
Task-wise matched slot matching score: SM))
Task-wise primitive matching score: f6 -Mean, (MAH(ss- {.)}
For clarity, the matching scores are normalized to [0, 1] to align with the range of mloU. A large value of any metric above indicates a better performance according to the corresponding assessment.
Slot representation learning First, fixing Tp 100, 7, 100, we vary the coefficient as shown Figure Tia While smaller values (e.g., 0.1) achieve marginally better scores (indicating greater primitive stability across tasks), they significantly degrade other critical metrics, particularly and f2. This trade-off suggests that excessively stable primitives may fail to adequately capture diverse label semantics necessary for effective primitive-logit alignment.
Next, we examine the temperature parameter Tp by fixing a = 10, T. = 100 (Figure Tib). The radar chart demonstrates that TP 100 provides optimal balance across all metries, confirming hypothesis that moderate temperature settings enable better concept generalization while preventing over-regularizatioo.
28

---

Published as a conference paper at ICLR 2026
1.
1.0
1.0
0.0
04
0.4
04
0.
Num layers
(a) Number of Slots
(b) Number of Iters
(c) Slot Dim
(d) Number of layers
Figure 12: Line charts of different hyperparameters in slot attention architecture.
Softmax
Minmax
63.5
82.8
$3.59
93.57
68.83
8244
60
62.5
62.0
81.6
20
81.5
10
10
10'
10
10'
ADAM-adapler Methods
B
Ta
(a) Alignment Coeff.
(b) Alignment Temp.
(c) Slot Supervision Norm.
Figure 13: Line charts of different hyperparameters in primitive-logit knowledge distillation.
Finally, we analyze the task temperature т with fixed no single 7, value dominates across all metries, observe that т 100 achieves the highest fl score. Section G provides up visualizations showing that larger 7. values produce sharper slot selection distributions for primitive construction, which benefits concept representation but may reduce flexibility in extreme cases.
10, тp 100 (Figure 1lc While
Slot attention architecture Figure 12a examines the impact of increasing the number of slots (K). While higher values initially improve slot performance by enabling representation concepts, observe diminishing returns beyond K 10. This saturation occurs due to two factors: (I) the limited number of visually discriminable concepts per image, and (2) the finite capacity of the slot attention mechanism. Redundant slots tend to converge similar representations, creating performance plateau. Our slot mask visualizations G confirm this phenomenon, showing that excessive slots merely replicate existing patterns rather than capturing novel information.
more
Figure 1bb investigates the effect of refinement iterations (N,) in the slot attention module. While increasing N₂ enhances slot discriminability by promoting greater inter-slot differences, find that three iterations (N, 3) achieve optimal performance. Further increases do not mcaningfully improve results, suggesting that three iterations strike an effective balance between refinement and computational efficiency.
Figure 12c explores the relationship between slot dimensionality (capability) and performance. We observe that larger slot dimensions consistently improve fl scores, indicating better concept representation. However, this comes the cost of increased computational overhead, necessitating careful trade-off considerations for practical applications.
Finally, Figure 12d examines the impact decoder architecture by varying MLP layer depth. Contrary to expectations, deeper decoders fail to improve extraction performance, suggesting that the current decoder architecture has sufficient capacity for the task.
Primitive-Logit knowledge distillation We apply our learned slot attention mechanism to compute concept-based sample-wit wise similarities RanPAC, systematically evaluating key hyperparameters in our primitive-logit knowledge distillation framework.
Figure 3a demonstrates that increasing the coefficient 3 for L. consistently improves CL accuracy (AA). This indicates that stronger super-uuno from concept- based similarity effectively enhances the model's ability preserve task-specific knowledge while adapting to tasks.
29

---

Published as a conference paper at ICLR 2026
Table 7: Main result 10-10 tasks COBJ. We report the average accuracy after training the last task (AA), the cumulative average accuracy for each task (CA), and the final forgetting (FF). For CFST, we report the Harmonic mean of compositional testing (Hn) and the ratio of Hn and reference testing (R). Methods with CompSLOT are denoted with a postfix "f". Methods rehearse old samples are denoted with postfix "** We report results over 3 trials with (mean 95% confidence interval).
Continual CA (%)
CFST
Methods
AA (%) f
FF (%) 4
Hn (%) f
R1
CPrompt CPrompt
42.015=t0118
51.172:19.718
58.961 62.775=66.479
0.878
45.520:10.421
52.565-10.931
19.575+1.029
59.880++2.032
0.880
ADAM adapter
45.750+10.346
52.800
12.175+ 1.836
1..793-11.88
0.914
ADAM adapter t
50.150-1.229
57.767115.461
11.050=1.802
61.581+1.399
0.938
RanPAC
59.285+22377
66.203114116
7.650=0.624
60.90913:240
0.882
RanPAC
61.950=0.5277
67.36714.075
7.875=0.104
62.317
0.889
CoFiMA CoFiMA f
57.330 139
5.765
17.075+00035
66.998 22112
0.890
0..43
63.462++0.59
16.650-1.200
66.232++2.497
0.898
FOSTER*
47.800-10:542
43.291
10.575 0.759
62.750-10.337
0.852
FOSTER* t
50.250
59.735-10.556
14.525=0.240
63.69511.312
0.908
DER
55.815+0.714
64.90543.342
23.650+2.425
68.558
0.844
DER* 1
56.813+11.08
66.39343.904
25.800::4.534
68.586::0.441
0.872
Figure 13b highlightt the critical importance of properly tuning the temperature parameter Ta- We observe performance plateau when Ta within optimal range (approximately [0.1. 1.0J). Values beyond this range exhibit clear trade-offs. This is because (1) Large Ta 1.0) causes excessive emphasis sample-wist differences, undermining concept sharing; (2) Small ra(< 0.1) produces overly smooth logit similarities, degrading classification performance.
Regarding normalization strategies primitives (Equation 5 min-max Equation 3 softmax), Figure 13c shows that min-max normalization outperforms softmax normalization. This advantage stems from min-max normalization's 's ability provide sharper supervision through its linear scaling properties, and maintain better sensitivity to subtle concept differences between samples.
H.3 RESULTS ON OTHER BENCHMARKS
COBJ The results in Table clearly demonstrate that incorporating CompSL.T into CL methods with FMs leads to significant performance improvements across various metrics. Specifically, CompSLOT enhances compositional generalization ability, evidenced by higher Hn and improved R (most significant gain of Hn for ADAM + adapter from 57.793 to 61.581), which in turn drives better overall CL performance (AA for ADAM + adapter improves from 45.75 15). CompSI.OT's ability to strengthen compositional generalization appears to be the key factor behind these gains, enabling the model to better handle complex concepts and retain knowledge more effectively across tasks.
ImageNet-R It can seen that CompSL.T can generally improve the performance of CL methods with FMs in Table The improvement is likely due to the observation that the learned slot attention can discover hidden concept sharing between images, as evidenced by the visualization analysis in section G Rehearsal methods (e.g., FOSTER* and MEMO*) achieve better performance in terms of AA and CA, comparing with rehearsal- methods. This is because rehearsal methods access old samples, thus, CompSLOT's primitive-logit alignment loss can provide pair- wise contrastive seff-supervision on concept sharing, which enhances the model's 's compositional generalization performance.
30

---

Published as a conference paper at ICLR 2026
Table Main result on 20-20 tasks ImageNet-R. We report the average accuracy after training the last task (AA), the cumulative average accuracy for each task (CA), and the final forgetting (FF). Methods with CompSLOT are denoted with postfix "T" Methods rehearse old samples are denoted with postfi "*", The data for methods with citations is reported from the original paper. We report results over 3 trials with (mean 95% confidence interval).
Methods
AA (%) 1
CA (%)
FF (%) +
CPrompt (Gao et al. CPrompt
2024
74.790=10.280
81,400=0.930
7.340++0.650
7..225
1.96441..07
6.989:1.126
RanPAC RanPAC
78.375-0.002
82.519#0.839
4.85611..67
0..450
02.900
5.294+000339
CoFiMA CoFiMA f
80.025+1..44
83.927+
7.614+0.142
80.250-10.016
84.118
7.022
FOSTER* *
76.001
80.974+1.083
2.259
FOSTER *
0.2250
82.392+
2.608
MEMO* *
64.200
72.11810.074
4.967
MEMO* *1
65.200
72.995+1.251
5.344+0.256
Table 9: Varying backbone on 10-10 tasks CGQA. We report the average accuracy after training the last task (AA), the cumulative average accuracy for each task (CA), and the final forgetting (FF). The candidate CL algorithm is RanPAC. Methods with CompSLOT are denoted with postfix "1"
Backbone
AA (%) f
CA (%) f
FF (%)
Hn f
VIT-B16
65.81
75.50
10.51
78.86
ViT-B16 t
66.75
76.58
10.21
79,81
ViT-BI6-DING t
66.58
76.62
10.24
80.39
ViT-B16-SAM
67.30
77.76
9.67
81.22
ViT-L16
67.11
77.54
9.85
80.82
Table 10: AA results on 10-10 tasks CUB200 and CIFAR100.
Datasets
SACK
CLG-CBM
CompSLOT
CUB200
71.78
85.40
88.38
CIFAR100
87.26
84.49
89.57
H.4 RESULTS ON OTHER BACKBONES
This section is to answer: Do better vision foundation models contribute to better concept learning and continual learning performance? We investigate the effect of model scaling increasing the size and depth of the ViT architecture (c.g., ViT-L16 vs ViT-B16), and the effect of pretraining strategy via leveraging greater pretraining objectives, such as DINO (Oquab al., 2024) (e.g., ViT-B16-DINO) and SAM (Kirillov et al., 2023) (e.g. ViT-B16-SAMI which have been shown to enhance semantic understanding. especially on segmentation and concept-rich tasks. We conduct experiments along the two key dimensions above and report the results in Table 9 The results show that ViT-L16 with larger model sizes demonstrates stronger representation modeling capabilities compared to ViT-B16, B1,, thus further boosting the significance of our CompSLOT. ViT-B 11--DINN and CUT-B16-SAMM with greater pre- gre-training objectives exhibit better compositionality in decomposing concepts and continual learning performance, reflected by higher Hn values.
31

---

Published as a conference paper at ICLR 2026
Table 11: Additional ablation results on CGQA.
Methods
Lp Lo
AA
(%) f
FF (%)
SimpleCIL
x
36.16
13.9
SimpleCIL x
24.71
22.93
RanPAC
X
x
65.81
10.51
RanPAC x
41.59
11.87
Table 12: Results on Finetuning on 10-10 CGQA. We report the average with + 95% confidence
interval.
Methods
AA (%) f
CA (%)
FF (%) 4
Finetuning
29.91 + 0.84
49.04 + 0.39
58.36 + 1.02
Finetuning
33.48 0.04
52.43 + 0.65
50.01 + 2.13
H.5 COMPARING WITH OTHER CONCEPT LEARNING METHODS
This section compares CompSLOT with other concept learning methods, i.c., CLG-CBM (Yu et al. 2025) and SACK (Kundargi et al. 2025). We conduct experiments on 10-10 tasks CUB200 and CIFAR1O 100 to show the superiority of CompSLOT with RanPAC. To achieve fair comparison, we replace the CompSLOT's backbone to CLIP ViT-B/16. The results are shown in Table 10 with the top performance mentioned in their original papers. CompSLOT shows the best AA on both benchmarks, because of benefiting from slot attention to extract concept information and the plug-and and--lay property that be applied alternative CL backbones. Most importantly, CompSLOT fully utilizes the capability of the CL backbone and does not need extra interpretable tools, like ChatGPT.
H.6 ADDITIONAL ABLATION STUDIES
To clearly substantiate the contribution of slot attention combination with primitive selection, conduct an ablation study where we remove knowledge distillation and instead directly use the learned primitive representations with cosine similarity classifier for continual tasks, in SimpleCIL. (Zhou et al.) 2025). We also integrate this strategy into RanPAC and the results are shown in Table II This naive approach suffers from severe forgetting, confirming that primitive representations are insufficient for long-term retention when learning new tasks. In contrast, our alignment loss distills pair-wise relational information (i.c., compact, low-dimensional encoding of concept combinations) rather than high-diment -igenninnnii raw representations. This enables methods equipped with CompSLOT to maintain stable performance while accumulating higher accuracies over time, demonstrating the efficacy of CompSLOT in mitigating catastrophic forgetting.
H.7 CASE STUDIES: THE EFFECT OF COMPSLOT ON FINETUNING
In this section, we answer that question: Does CompSLOT itself benefit continual learners without associating with other continual learning algorithms? We perform CompSLOT on naive continual learner finetuning, which uses frozen feature extractor backbone (VIT-B/16) and extendable classifier. The results are shown in Table 12, We observe that CompSL.TT successfully achieves higher AA with smaller FF. This observation indicates that CompSLOT itself, as plug-in, benefits the continual learner without the need to combine other mechanisms.
H.8 ALGORITIM EFFICIENCY ANALYSIS
Parameter overhead We evaluate the parameter overhead introduced by slot attention module. As this module requires pretrained ViT as its semantic feature extractor, which is a standard component in all continual learning of foundation models frameworks, the additional trainable
32

---

Published as a conference paper at ICLR 2026
10
Vanilla
# Parameters
With plugin
10"
Crom-Prompt
CPrompt
ADAM-adapter
RenPAC
EASE
LAE
CoFIMA
DER
FOSTER
MEMO
Methods
Figure 14: Visualization of the parameter numbers for methods with and without the slot module. Note that the data are collected according to the default implementation in the PILOT (Sun et al. 2025) platform and after the training of the last 10-way CGQA continual task. Takeaway: CompSL.OT requires a ViT backbone that is already in any model-based continual learner with a foundation model, thus, it is light-weight and free to be applied.
Table 13: Computational overhead (h) on CGQA.
Slot module
FOSTER
FOSTER t
5.5
9.1
10.1
parameters are negligible compared to the total model size, as illustrated in Figure 14 This makes our CompSLOT computationally efficient while delivering significant performance benefits.
Computation overhead In Table 13 study the computational overhead introduced by the slot attention mechanism and primitive extraction. As an example, we choose FOSTER as representative baseline, since it achieves nearly top performance among others. We compare three cases: 1) Continual training of just our slot module plugin, including both slot attention and primitive selection components, without applying it to other continual learning algorithms; 2) Full continual training of FOSTER; 3) Full continual learning of FOSTER with pretrained slot module plugin (FOSTER t). We highlight that the slot module can be learned offline as a reusable component which only associated with the benchmark and is independent of algorithms. Once trained, it serves as pretrained plugin that can be directly loaded for any continual learning algorithm with minimal additional overhead. It only requires adding alignment loss L, for logit regularization and spending an additional 10% of total training time for FOSTER from to 10. 1h. This design is particularly beneficial when running multiple continual learning algorithms on the same data distribution.
Importantly, we conduct an ablation study (Section 5), where we deliberately increase the parameter count of baseline CI. methods to match our CompSL.OT-enhancee models. The results demonstrate that the performance gains stem not from increased capacity, but from CompSLOT's improved com- positional generalization capabilities. This confirms that CompSLOT provides genuine algorithmic advantages rather than simply benefiting from more parameters.
I USE OF LARGE LANGUAGE MODELS
In the process of preparing this paper, we employed LLMs to polish the writing of the paper. The assistance provided by LLMs was mainly focused on improving the clarity, coherence, and overall quality of the language used in the manuscript. We input sections the paper into the LLM and requested it to suggest rephrasings, correct grammar and spelling errors, and enhance the readability of the text. is important to note that LLMs did not play a significant role in the research idcation. The core ideas, research questions, experimental designs, and methodological choices were independently
33

---

Published as a conference paper at ICLR 2026
conceived and developed by the human authors. The LLM was not involved in formulating the hypotheses, determining the research direction, or making decisions regarding the data collection and analysis methods.
34


Reviews:
[{"review_id": "G4qAFGetTV", "reviewer": "Reviewer_gbFN", "summary": "This work addresses the problem of catastrophic forgetting in continual learning (CL) models, particularly those using Foundation Models (FMs), which struggle because they rely on simple class comparisons rather than understanding complex objects as compositions of basic concepts.\n\nThe proposed solution is CompSLOT, a universal, plug-and-play module that injects concept-level compositionality into any CL method with an FM backbone.\n\n1. It uses a self-supervised Slot Attention mechanism to break down images into low-dimensional representations called slots (concepts). \n2. It introduces a primitive selection mechanism to identify and aggregate the most class-relevant concepts from the slots. A primitive loss ensures these primitives are consistent across different examples of the same class.\n3.  The core plug-in component is a primitive-logit alignment loss. This loss distills the concept-level similarities between images directly into the model’s final predictions. This regularization guides the model to make decisions based on meaningful, shared, and distinct concepts, rather than simple feature comparisons.", "questions": "I would like to ask authors for better contextualization of the work and improved clarity.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Slot attention module is highly stable and shows almost no forgetting across sequential tasks.\n\nCompSLOT is method-agnostic and computationally lightweight as it builds upon the existing FM backbone.\n\nCompSLOT significantly boosts the accuracy of diverse CL baselines.\n\nIt enhances compositional generalization abilities.\n\nBenchmarking is broad, and even includes fine-grained classification such as CUB 200. \n\nThe work is clearly written, images are readable. Only Figure 1 is too detailed for teaser image and should be simplified to improve the clarity.\n\nExperiments are convincing. \n\nIdea is novel, straightforward and easy to follow.", "weaknesses": "The work should be better contextualized in terms of concept-based continual learning, including discussion with work of [1] and follow-up works. \n\nFigure 1 can be improved as it is right now too complex and does not convey the message about novelty well. \n\nContribution description is vague and unclear. It looks like in the second bullet the CompSlot designed something, not the authors. Looks like the artifact from LLM text improvement. The language there should be simpler, and maybe following organisation made:\n\n- first dot about introduction of compslot and its key components.\n\n- second dot about novel training components as losses\n\n- last one about extensive experiments. \n\n[1] Rymarczyk, Dawid, et al. \"Icicle: Interpretable class incremental continual learning.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03X0xSDxOG", "reviewer": "Reviewer_hVcV", "summary": "The proposed CompSLOT (Compositional Slot plug-in) is a novel method designed to address the challenges of continual learning on compositional benchmarks. It enhances vision models by extracting disentangled, class-relevant concepts directly from images using Slot Attention mechanism. The core of CompSLOT lies in its robust concept learning phase, which uses the primitive selection and aggregation mechanism to identify essential class-relevant concepts. Experimental results consistently demonstrate that CompSLOT significantly boosts the performance of various state-of-the-art continual learners across challenging compositional benchmarks.", "questions": "see above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- CompSLOT can significantly mitigate catastrophic forgetting in continual learning.\n\n- CompSLOT features a highly flexible and method-agnostic plug-and-play design.\n\n- Extensive experiments on challenging compositional datasets robustly validate CompSLOT's superior performance in continual learning.", "weaknesses": "- It is unclear how a fair comparison was achieved, as the compared methods were not used in the benchmark paper. It is also not understood how these Class-Incremental Learning methods were applied in this setting.\n\n- Essentially, CompSLOT uses an external model as a teacher to reduce catastrophic forgetting.\n\n- It is unclear how this plugin works with baseline methods. For instance, the plugin generates a set of logits. However, the contribution of some methods, such as CPrompt, is to constrain the logits at different stages during incremental learning. Would the proposed method conflict with these baselines?\n\n- The method is too complex to be reproduced easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0xQRNNERIu", "reviewer": "Reviewer_rGQP", "summary": "This paper proposes a new continual learning algorithm for foundation models called CompSLOT. CompSLOT utilizes an existing object-centric plug in for concept extraction, and incorporates concept learning through decomposition and selection mechanism. Experiment results show that CompSLOT boosts several continual learning methods' performance on standard metrics and benchmarks.", "questions": "1. Does Slot Attention suitable for any ViT, or other vision models?\n2. I was wondering if CompSLOT benefits the naive continual learners: finetuning? Or does CompSLOT need to be combined with other continual learning algorithms? It will be an informative baseline.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method bridges object-centric learning, concept learning and continual learning, which is a novel intersection.  \n2. Extensive experiments have been conducted to confirm that CompSLOT improves continual learning performance on standard evaluation metrics and benchmarks.\n3. The proposed method can be easily combined with other continual learning methods.\n4. The paper is well-written and easy to follow.", "weaknesses": "1. **The models for experiments are not specified.** The authors do not mention of a specific architecture in the main paper. Does all experiments in Table 1 conducted with the same model architecture? Do CLG-CBM and CompSLOT use the same pretrained model? Without these details, it is difficult to confirm the effectiveness and the versatility of the proposed method.\n2. **Lack of concept analysis.** The authors claim that CompSLOT learns human-interpretable concepts for continual learning. However, no experiments are presented to analyze these concepts or evaluate interpretability of the learned representations. \n3. **Hyperparameter sensitivity.** CompSLOT requires hyperparameter tuning ($\\alpha, \\beta, \\tau_t, \\tau_p, \\tau_a$), which is data-dependent. The sensitivity to these hyperparameters is not discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)