Dual-Space Semantic Synergy Distillation for Continual Learning of Unlabeled Streams
Authors: Donghao Sun
Xi Wang
Xu Yang
Kun Wei
Cheng Deng
Venue: NeurIPS 2025 poster ()
Keywords: Unsupervised Learning
Class Incremental Learning
Visual-Text Complementary Guidance
Primary area: general_machine_learning
TL;DR: This paper proposes a visual-text complementary guidance method to solve the problem of unsupervised class incremental learning

Abstract: Continual learning from unlabeled data streams while effectively combating catastrophic forgetting poses an intractable challenge. Traditional methods predominantly rely on visual clustering techniques to generate pseudo labels, which are frequently plagued by problems such as noise and suboptimal quality, profoundly affecting the impact on the model evolution. To surmount these obstacles, we introduce an innovative approach that synergistically combines both visual and textual information to generate dual space hybrid pseudo labels for reliable model continual evolution. Specifically, by harnessing the capabilities of large multimodal models, we initially generate generalizable text descriptions for a few representative samples. These descriptions then undergo a `Coarse to Fine' refinement process to capture the subtle nuances between different data points, significantly enhancing the semantic accuracy of the descriptions. Simultaneously, a novel cross-modal hybrid approach seamlessly integrates these fine-grained textual descriptions with visual features, thereby creating a more robust and reliable supervisory signal. Finally, such descriptions are employed to alleviate the catastrophic forgetting issue via a semantic alignment distillation, which capitalizes on the stability inherent in language knowledge to effectively prevent the model from forgetting previously learned information. Comprehensive experiments conducted on a variety of benchmarks demonstrate that our proposed method attains state-of-the-art performance, and ablation studies further substantiate the effectiveness and superiority of the proposed method.

Dual-Space Semantic Synergy Distillation for Continual Learning of Unlabeled Streams
Donghao Sun"
Xi Wang"
Xu Yang!
Kun Wei
Cheng Deng
School Flectronic Engineering, Xidian University, Xi'an 710071, China {donghaosun508, wangxi6317, xuyang xd, weikunsk, chdeng xx)@@gmil..com
Abstract
Continual learning from unlabeled data streams while effectively combating catas- trophic forgetting poses an intractable challenge. Traditional methods predomi- nantly rely on visual clustering techniques generate pseudo labels, which often suffer from semantic inconsistencies and limited discriminative precision, thereby impeding stable model evolution. To surmount these obstacles, we introduce an innovative approach that synergistically combines both visual and textual infor- mation to generate dual space hybrid pseudo labels for reliable model continual evolution. Specifically, by harnessing the capabilities of large multimodal mod- els, we initially generate generalizable text descriptions for few representative samples. These descriptions then undergo 'Coarse to Fine' refinement process capture the subtle nuances between different data points, significantly enhancing the semantic accuracy of the descriptions. Simultancously, a novel cross-modal hybrid approach seamlessly integrates these ine-grained textual descriptions with visual features, thereby creating more robust and reliable supervisory signal. Finally, such descriptions are employed to alleviate the catastrophic forgetting issue via semantic alignment distillation, which capitalizes the stability inherent in language knowledge to effectively prevent the model from forgetting previously learned information. Comprehensive experiments conducted variety of bench- marks demonstrate that our proposed method attains state-of-the-arr performance, and ablation studies further substantiate the effectiveness and superiority of the proposed method.
Introduction
Deep learning models have demonstrated robust and well-established performance when trained on independently and identically distributed data, but real-world data is often nonstationary and arrives sequentially in tasks. In such scenarios, the model must continually learn new tasks while retaining knowledge of previous ones to avoid catastrophic forgetting 2 This learning paradigm is known as continual learning (CL.) among which class-increne class-incrementll learning (CIL.) 4 is particularly challenging and realistic, as it requires the model to perform unified classification while new classes are introduced progressively.
In many real-world scenarios, due to the high cost or even the infeasibility of labeling, data generally arrives in continuous, unstructured, and unlabeled manner (5). This raises a critical challenge: how can we perform structured modeling of continual streaming data in a fully unsupervised setting? Unlike conventional unsupervised learning (6), what makes the problem thornier is that unsupervised
"Tese authors contributed equally. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

continual learning demands that models incrementally extract semantic structures over time, contin- ually adapt to new data, and retain previously acquired knowledge. In this context, Unsupervised classenneremental Learning (UCIL) specifically targets the problem of progressively discovering class structures without any label supervision, and serves as a key step toward large-scale open- world learning.
In unsupervised learning, especially when dealing with large-scale unlabeled data, clustering-basee pseudo- labeling is commonly used to guide model training M However, existing clustering algorithms often perform poorly in complex visual scenarios, especially when deal- ing with visually similar categories (c.g., as illustrated in Fig. where samples from different classes are eas- ily confused. This frequently results in many incorrect pseudo-labels, severely hindering the model's ability to learn accurate feature representations. Such biased labels not only compromise training performance at the current stage but also cause irreversible knowledge corruption [8] in UCIL tasks, making it difficult for the model to correct early-stage misconceptions in later learning phases.
and iddeencarfyy nrundkl diferer drectors.
endhery usalyy extriaaed Rovenart.
Lizand V
Figure Classification results under dif- ferent guidances. a) Visual-only (confu- sion risk) b) Coarse semantic (oracle la- bels) c) Fine-grained (optimized descrip- tions)
With the rapid advancement of large multimodal models (LMMs) researchers have begun lever- aging synthetic texts generated by LMMs as auxiliary supervision signals or contextual information to enhance overall performance [0. Inspired by these developments, we explore the potential of using LMMs to uncover the latent semantic information within samples to further improve model effectiveness. We argue that natural language descriptions inherently contain semantic structures that can compensate for visual ambiguities (as illustrated in Ibg. Moreover, the relative stability of the semantic space provides a useful inductive bias for constraining model training, which turn helps mitigate the problem of catastrophic forgetting 13
Experimental results (Fig.2) demonstrate that directly employ- ing LMMs as supervisory signals fails to enhance model per- formance while increasing computational complexity. Specifi- cally, the standard unsupervised framework, fine-tuning through text clustering of L.MM- image labels, shows limited performance gains. We attribute this to two factors: 1) The coarse- bearse-grained semantic representations from LMMs inade- quately capture nuanced distinctions among visually similar samples [4] 2) Inherent knowledge biases and inter-category semantic ambiguity [15] lead to semantic inconsistencies and inaccuracies in the generated pscudo labels, which critically constrain the efficacy of textual supervision.
improvemnnt
pieddr-bbelingg
9.11
52.7%
Visual Clusaring
clustrnng
MAc/N/CD
Inherent knowledge bias refers to the tendency of large-scale scale pre-trained models (e.g., CLIP or large language models) to rely more heavily on concepts that appear frequently in their pre-training corpus during open-world reasoning. Consequently, these models often yield biased predictions for low-frequency or visually indistinct categories [16). For example, frequent animals such as "cat" and "dog" are consistently recognized due to their salient visual features and stable semantic expressions, whereas rarer species like "weasel" or "lynx" tend to suffer representational drift because of limited occurrence or ambiguous semantics in the training data. This reflects the statistical bias inherent in the 'language model as knowledge base" paradigm, where prior exposure determines the reliability of recognition [17]
Figure 2: Quantitative experiments on ImageNet-R with 5 tasks.
Inter-category semantic ambiguity arises when categories share overlapping attributes in the visual or textual domain, blurring their semantic boundaries. For instance, "seal" and "sea lion" exhibit highly similar visual traits (body shape, texture, and background), making them difficult to separate visually, while "octopus" and "jellyfish" thouuh visually distinct-are often semantically conflated by descriptors such as "marine animal" or "tentacles" [8]. These within- and cross-modality ambiguities increase the likelihood of label confusion when LMM -enerraeet descriptions used directly for supervision.

---

In light of the above constraints, we select small subset of representative samples (approximately 2% of the dataset) based on image clustering, and query LMMs to generate multi- multi-granularity descriptions ranging from coarse to fine levels. This low-query strategy significantly reduces computational cost while enhancing the richness and discriminability of the extracted semantic information (as illustrated in Fig. Meanwhile, considering the complementary strengths of visual and semantic modalities in perceptual granularity and representational capacity, we propose a language-guided collaborative alignment strategy to bridge and integrate supervisory signals from both spaces. By introducing language as an intermediary, we establish a connection between these two imperfect sources of supervisipp enabling the model to learn discriminative features that jointly capture abstract semantics and fine-grained visual details. Finally, as the semantic space excels at capturing abstract concepts and prior knowledge, while the visual space is more effective modeling low-level details such as textures and edges, we introduce a semantic-to-visut distillation mechanism that leverages the stability of the semantic space to regularize model updates. This cross-modal alignment strategy effectively addresses the challenges of supervision scarcity and catastrophic forgetting in UCIL.
Our main contributions can be summarized as follows:
We propose low-resouree UCIL approach that generates hierarchical semantic descriptions from LMMs using a small set of representative samples, effectively reducing computational cost and improving the reliability of pseudo labels, thereby enhancing the quality of semantic supervision.
We design a dual- modaltty pscudo-labeling strategy that jointly leverages visual and semantic cues for robust representation learning, and introduce semantic distillation mechanism to effectively mitigate catastrophic forgetting.
Through comprehensive experiments and ablation studies, our method demonstrates superior performance and robustness on multiple benchmark datasets, highlighting its advantages in UCIL scenarios.
2 Related works
2.1 Unsupervised Class Incremental Learning
Unsupervised Class Incremental Learning (UCIL) focuses learning new classes from unlabeled data while preserving knowledge of previously learned classes [19]. The challenge lies in preventing catastrophic forgetting as new classes are introduced. Self-supervised learning (SSL) is often employed (20). where models learn useful representations without labeled data. Contrastive learning, a popular SSL approach, distinguishes between positive and negative samples, enabling the model to learn discriminative features for new classes.
To address forgetting, techniques like memory replay and knowledge distillation used (1). Mem- ory replay stores previously encountered data and replays it during training to maintain performance on old classes while learning new ones (22. Additionally, combining unsupervised learning with pseudo labels or cross-modal information can help improve learning efficiency and task adaptability UCIL settings (23).
2.2 Fine-Tuning CLIP Models
CLIP models, pretrained on large-scale image-text pairs, have demonstrated strong zero-shot per- formance, but fine-tuning is often necessary to improve task-specific performance [24. A common approach to fine-tuning CLIP involves training a classifier on top of the pre-trained features. One efficient method is adding fully connected linear layer at the output of CLIP's visual encoder. This approach freezes the CLIP model's parameters and only fine-tunes the linear layer, minimizing the risk of overfitting and reducing the computational cost compared to full fine-tuning 255. This method is effective in adapting CLIP to specific tasks while maintaining the robustness of its pre-trained representations.
Recently, methods such as contrastive loss (26 and linear probing 27 have also been applied to CLIP models for task adaptation. However, freezing the core CLIP model while fine-tuning a small number of parameters, like the output layer, has emerged as a popular technique, as it balances computational efficiency with high performance in specific applications.
3

---

3 Method
The objective of this work is to enable the network to learn from continuous, unlabeled data streams that more accurately reflect the real-world environment. The proposed method efficiently utilizes the cross-modal information inherent in the data streams, while minimizing resource consumption. It is anticipated that this approach to cross-modal information alignment will offer a more comprehensive solution for future unsupervised tasks. In this section, we first present the definition of UCIL, followed by a detailed explanation of the method we have introduced.
3.1 Preliminary
3.1.1 Problem definition.
In the context of UCIL, the model needs to be trained on T consecutive tasks. Each task f provides an unlabeled dataset D' { where N represents the number of instances in task i, and these instances belong to C new categories. The categories across tasks are disjoint, meaning there is no overlapping between any two different tasks: CH n C*v ž, + j. The goal of UCIL is to enable the model to incrementally discover semantically meaningful categories from D' and assign instances to these categories without label information. In each task, the model has access only to the current unlabeled dataset D' and cannot access any prior information. The number of new categories C" introduced in task is known in advance, but for the specific category instances of each task, the model can only discover them through the exploration of the data. During the incremental learning process, the model not only needs to learn the new categories in the current task but also must retain the categories learned previously, preventing the forgetting of knowledge from earlier tasks. To formalize this problem, let X be the input data space. The objective is learn mapping function f X + U'= C which can map any test sample x to the set of categories discovered across all tasks without relying on task identifiers.
3.1.2 Base model
CLIP was chosen as the base pre-trained model for work due its ability simultaneously process data from both visual and linguistic spaces. An effective approach to fine-tuning a pre-trained model for downstream tasks involves incorporating a lightweight network an adapter. To facilitate our subsequent explanation, we denote the visual encoder as E,,, the text encoder as E,, and the linear adapter f- Under supervised conditions, when textual labels employed as supervision information for category classification, the probability that the model predicts any input sample x, as category is:
pred, - E, (E)():)).
(I)
In the previous analysis, we performed classification in the visual and textual knowledge spaces using traditional clustering methods, but the results were unsatisfactory. Even though we obtained information from both modalities in the data stream, the high visual similarity between categories and the inherent semantic ambiguity in the textual information derived from LMMs made it difficult to achieve satisfactory classification performance with existing methods. Therefore, in this paper, we propose two approaches to address the confusion within tasks caused by the lack of supervisory signals, as well as the confusion between different tasks induced by incremental data. The architec- tures of these methods are illustrated in the Fig.3. We will now provide detailed description of the proposed methods.
3.2 Semantic Collaborative Facilitative Supervision
To fully exploit cross- moosl information, avoid performing clustering independently in single modality. Instead, we propose semantic collaborative supervision, which aligns visual and semantic knowledge in a unified framework. Specifically, we reduce the reliance LMM queries by selecting only few representative samples near visual cluster centers, rather than querying every instance. To enhance the quality of semantic signals, design a series of progressively refined prompt templates for hierarchical description extraction. The resulting visual and textual cues are then jointly used in a collaborative supervision mechanism guide model training more effectively.
For task I, we process the input samples x; through the CL.IP visual encoder to obtain their feature representations E, (xx) in the visual space. We then apply K-means clustering in this space to partition

---

Unlobeled Images
Visualization of Clustered Features
Confidence weighting
Visual Prototypes
Encoder
Visual
Adapter
Textual Protstypes
CLIP-based -a--aased Similanty Matching
Prodictions of Spaces
Prompe
(Darker colors indicate finer granularity)
00000
Filrrtto
Assign
Frozen
Fine Fine-tuning
LMM
Encoder
Textual
A
Representative
!
Sample Selection
logila (i.)
Representative images
Mixod loss
samples visual prototypes
Figure 3: Framework for single task in unsupervised class-incremental learning. Visual clusters and LMM-generated descriptions provide dual-modality pseudo-labels, which jointly supervise adapter training via seflidnneeweiighee loss.
the samples into C* clusters, denoted as { For the k-th cluster Φs. we treat the samples that are closest to the cluster center as belonging to the k-th class and assign them corresponding visual pseudo-label Po. Since the visual feature space may contain similar but semantically different image representations, the resulting visual pseudo labels often suffer from inter-class confusion. Using such confusing pscudo labels supervision to fine-tune the pre-trained model may lead to an unstable optimization process, hindering transferability to downstream tasks. Therefore, it is necessary to introduce more reliable supervisory signals. Given the superior stability and generalizability of the semantic space, we leverage independently designed textual descriptions to filter the data and reduce the class confusion in the visual pseudo labels.
To obtain text descriptions, the most straightforward method is to directly input these samples into an LMM and prompt it with the query, 'What object is depicted in this image?', which generates coarse-graines category description for each sample x;, Besides the additional time and financial costs associated with utilizing the LMM, the generated text may still exhibit a degree of randomness due to the LMM's 's inherent knowledge biases and the potential ambiguity between different classes. Under such textual supervision, this randomness may exacerbate the confusion between samples. Therefore, based on the previous clustering results, for any given cluster k, we first obtain the prototype corresponding to each cluster k:
n2 Ex(x),, Φk:
where ny represents the number of samples in dk. Then, calculate the similarity between all samples in the cluster and the cluster prototype to construct the similarity matrix Qs = [gilign, for any Φk, q represent the cosine similarity score corresponding to each sample,
q: cos (E. pk).
Based on the similarity matrix, we can obtain the most representative m samples from each clustering result. We consider these m samples to have high-quality representations of each cluster. By providing fine-grained text descriptions for these samples, we avoid semantic ambiguity introduced by outlier or boundary samples and reduce computational resource consumption.
At the same time, reduce the randomness caused by sample selection and obtain more generalizable text descriptions, we designed the following three prompt templates in sequence from coarse-grained fine- fine-grained guide the LMM in generating the corresponding text descriptions.
Prompt 1: Pleass tell me the name of the object in the image without any descriptors." This prompt quickly captures the main object in the image by directly asking for its basic name, avoiding redundant information.
Prompt 2: 'Please describe the most distinctive visual attributes in the photo.' This prompt focuses on the fine-graines visual features of the image, helping to identify subtle details or distinctive visual markers that may not be immediately noticeable.
5

---

Prompt 3: 'Please describe the most common scenes of the object in the photo.' This prompt describes the common scenes of the object, providing coarse bearse-grained background. usage, and context.
Under the guidance of these three prompts, the LMM generates total of { [ textual descriptions for the m samples in each clustering cluster ox. These texts are treated as the textual descriptors for independent classes, Based on these textual descriptors, we can perform fine-grained filtering of the visual features. For any given input sample Fú, the probability of it belonging to the k-th class is given by:
3m
1ogit,(
3m Σ E( Ex)))).
1
(4)
We consider the class with the highest probability as the textual pseudo-label p, for the input sample. By using fine-grained textual descriptions, we optimize the initial purely visual labels and leverage the stability of the semantic space to provide higher-quality supervision signals for unsupervised tasks, For any given sample, we assign its corresponding visual pseudo-label Py and textual pseudo-label Pt. With the assistance of these supervision signals, we can thereby transform the originally unsupervised task into a supervised one and optimize the model using the cross-entropy loss (T) p).
Textual pseudo labels, derived from the semantic space, provide high-level and stable representations but may overlook fine-gra aineggrinedd visual differences. When generated by LMMs, they can also exhibit semantic bias or inaccuracy. In contrast, visual pseudo labels capture detailed features but are more sensitive to inter-class confusion and ambiguity. To leverage their complementary strengths and offset their limitations, we propose a hybrid supervision strategy that combines both modalities. Inspired by the general principle of multimodal fusion that higher-confidene confidence predictions tend to provide more reliable supervisory signals (28). our visual-semantic dynamic weighting mechanism adaptively balances the two modalities according to their relative confidence. Increasing the weight of the more confident modality helps stabilize training and improve effectiveness. Based on this intuition, we design a sample-level adaptive weighting strategy that fuses supervision from the visual and semantic spaces according to their confidence difference. For each sample F;, the training loss is defined as follows:
Ca = foe (2;, Po) + w/ Loe Pt).
(5)
For the input sample x,, if the clustering results in the visual space and textual space are identical, meaning that po = we consider the sample to be in a stable state. In this case, we consider the additional information provided by both the visual and textual spaces be of equal importance.
For cases where the clustering results differ between the two modalities, Ps * Pt. we consider the sample to be in state of confusion. In this case, need to find the balance between the visual and textual spaces. To achieve this, we first calculate the difference between the maximum probability and the second maximum probability at the stage of assigning pseudo labels samples.
In the visual space, for any sample i, we first compute the cosine similarity between its feature and all class prototypes in the current task:
(x( My
j= j=
(6)
The confidence margin for sample in the visual space is then defined as the difference between the top-1 and 2 similarities:
Hi = top, (s)) (sp)
Similarly, in the textual space, the confidence margin computed as:
Hi top, (logit, (x:)) toog(to (a.)), j=
(8)
This difference, after normalization, is regarded as the reliability of the supervisory information provided by the pseudo labels from different spaces:
Hi (..t)
(
(9)
H + H!

---

Therefore, under the supervision of visual-text dual pseudo labels, the model's training loss is given by:
if Po = Pt else. Finally, we perform weighted summation for batch of training samples Lets Es Co which serves as the loss of semantic supervision.
(Ts, Po) ) + (Ti. (Ti
H: Lee (24, + Hi+H1 (TI, Pt)
Hi
(10)
By balancing the reliability of supervisory information from the visual and textual spaces, we effectively mitigate the inconsistency between the two supervisory signals, maximizing the utility of pseudo labels. This, in turn, enhances the success rate of transferring the model to downstream tasks. The effectiveness of this weighting mechanism is further validated by our experiments Table 3
3.3 Semantic Alignment Distillation
Due to the influence of continuous data streams, the constantly changing visual features affect the feature space constructed by previous data, leading catastrophic forgetting in the model. Compared with the instability of visual representations, the semantic space offers stronger structural consistency and scalability. To leverage this advantage, we propose Semantic Alignment Distillation (SAD), which utilizes the stability of the language space to regularize the evolution of the visual space and mitigate catastrophic forgetting.
After each task training, we store the class prototypes μ4 = + Th (xx)) and covariance matrices Ex for all categories based on the textual space prediction results. During subsequent tasks, for each training batch, we randomly sample instances from the Gaussian distribution of old data. For these sampled instances, we compute the similarity (xg between each sample and its corresponding textual descriptor, as well as logit, 1ogit,(a:)4) (a:2d) between the same sample and the textual descriptor of a randomly selected new class. Here, k and y denote the categories of the old and new classes, respectively. Given the invariance of textual representations, we define the semantic aligmment loss for each sample as:
C = 1 - logit, (1gld) + logit, (nratd).
(11)
The first term in Eq. I1 measures the similarity be- tween the sample and its corresponding textual pro- totype, while the second term evaluates its similarity to randomly selected new class prototype. This for- mulation encourages old samples to remain close to their semantic anchors while keeping a margin from new class descriptors, thereby constraining the visual representation space during incremental updates. The semantic alignment loss for batch is expressed as Leat i=1 C.
Taskt-1
He Pookrao
Countrrn Storage
b
Close
Lns
old
Visua Encoder
Task
Teras Decoree
L.
Far Away
Figure 4: Semantic distillation leverages sta- ble textual descriptors to constrain visual training and mitigate catastrophic forgetting.
To further stabilize feature evolution, we add dis- tillation constraint between the previous and current adapters:
Lxd [oou fnew (E, (z))ll2.
(12)
The overall training objective integrates three components:
Les A1 AzCua-
(13)
Different from prototype-based distillation methods that operate solely within the visual space, SAD introduces a cross-modal alignment mechanism that transfers stability from the semantic space to the visual domain. It constructs a language-drive driven semantic scaffold composed of textual prototypes generated via hierarchical prompts (Sec. 3.2), whose embeddings are obtained from frozen text encoder and remain invariant throughout learning. These prototypes act semantic anchors that define a stable manifold for guiding visual feature updates. By enforcing directional alignment between visual features and their fixed textual counterparts, SAD emphasizes semantic consistency rather than visual reconstruction, providing a novel and effective approach to mitigating catastrophic forgetting in unsupervised inssmmnnceeettal learning.
7

---

Experiments
4.1 Experiments Settings
Datasets. We conducted experiments three datasets, including the widely used image classification dataset CIFAR 100, the style-varied classification dataset ImageNet-R, and the fine-graine aineggrained dataset CUB200. The CIFARIOK dataset contains 100 distinct classes, while both the ImageNet-R and CUB200 datasets consist of 200 different classes. We partitioned each of the three datasets into 5, 10, and 20 consecutive task streams, with equal distribution across the tasks.
Metrics. In continual learning, performance commonly evaluated using two metries: the Last Stage Accurucy (IA) and the Overall Averuge Accuracy (AA), following common practice in class- incremental learning 3] The average accuracy is computed as A Ei= AL, where A, denotes the accuracy on all seen classes after learning task f, and T is the total number of tasks. The final accuracy (LA) defined Ar, which measures the performance on all classes after completing the last task, reflecting the model's overall retention capability. In the main paper, we report the final accuracy (LA) of each method for clarity and fairness of comparison. To provide more comprehensive evaluation, the corresponding AA results of our method are included in Appendix A.4.
Comparison methods. We compare method with state-of-the-ar algorithms, including unsuper- vised class incremental learning methods CaSSLe [20], PFR (29]. POCON [30 and MSc-iNCD [3]. Unsupervised representation learning UPS 32 The original term MSc-iNCD refers to novel class discovery, but its setup is identical to the UCIL in this paper. For unsupervised representation learning methods, directly trained the model using the UPS algorithm under two scenarios:
without any constraints and the other with adding knowledge distillation. Meanwhile, we directly use CLIP to cluster the data, and also labeled all the samples in the test using LMM and then clustered the text to enhance the comprehensiveness of our experiments 33
Implementation details. For both datasets, pretrained model is the VIT-L/14 version of CLIP, and we train the model with the Adam optimizer for 30 epochs, with a learning rate x 10 10-3 We CosineSchedule adjust the learning rate. To ensure experimental fairness, all comparison methods were conducted the same environment. In our experiments m = and A = for all datasets and λ = 0.03 for ImageNet-R and CIFAR100 λę 0.15 for CUB200. For all datasets, the LMM that we used is GPT-4-turbo. In order to ensure the fairness of our experiment, we try our best keep the training environment the same, and all the methods obtained by re-running them Python 3.8, Pytorch 2.0.1, and single GPU A6000. All baselines were also re-run using the same ViT-L/14 backbone for fair comparison.
Table 1: Comparison experiments on different benchmarks. Bold indicates the best, while underline represents the second-best Results are averaged over 20 random seeds for robustness verification (see Appendix A.4).
Method
ImageNet-R tasks
CIFARI00 tasks
CUB200
tasks
20 tasks
tasks
tasks
tasks
tasks
tasks
UPS [ICLR 21|
20.9
16.5
14.3
18.4
15.7
12.9
15.9
UPS KD [ICLR'2]]
37.9
35.2
32.6
43.9
37.8
35.6
29.7
25.3
22.1
CassLe [CVPR'S 22]
45.2
40.5
37.8
59.6
52.5
49.6
PFR [CVPR'22 22]
41.6
37.9
31.0
59.8
$4.3
44,8
POCON [WACV'24]
40.3
41.5
41.1
63.1
60.5
56.8
MSc-iNCD [ICPR 24)
52.7
53.4
51.2
64.9
62.7
60.3
32.9
31.9
30.4
CL.I--based clustering
36.9
36.9
36.9
32.9
32.9
31.6
LMM-bused text clustering
51.1
51.1
51.1
41.7
41.7
41.7
34,5
34.5
34.5
Ours
81.7
82.2
79.8
66.1
63.6
59.4
66.7
64.8
64.0
4.2 Experimental Results
We conducted experiments on various datasets under different settings, and the results are shown in Table I On ImageNet-R with 5 tasks, our method achieves 81.7%, yielding 29.0% improvement over MSc-iNCD. This gain is not simply due to the use of CLIP, as the clustering result obtained by CLIP alone only 36.9%. Similarly, clustering with L.MM-genrateed text performs better than visual clustering but still falls short of the previous SOTA. When the number of tasks increases to 10

---

and 20, our method remains robust, achieving accuracies of 82.2% and 79.8%. On the fine- ane-ggained CUB200 dataset, where distinguishing similar image features is particularly challenging, all prior methods show relatively low performance. In contrast, our method reaches 66.7% on the 5-task setting, surpassing the previous SOTA by 33.8%, and maintains 64.8% and 64.0% with 10 and 20 tasks. For CIFARION 100, our method achieves 66. 16.1% 63.6%, and 59.4% with 5, 10, and 20 tasks, respectively. The relatively lower performance can be partly attributed to the 32 x 32 resolution of CIFARION 100 images, which limits the LMM's ability to extract fine- fine-grained semantics and generate accurate textual descriptions, thus affecting pseudo-label quality. In comparison, on higher-resolution datasets, method consistently demonstrates strong and stable performance.
4.3 Ablation Study
In this section, we analyze the effectiveness of each component in our proposed method. The experiments were conducted on ImageNer-R across tasks, with the results presented in Table 2] As shown in the table, although pre- pre-trained models possess powerful representational capabili- ties, relying solely on clustering fails to achieve highly accurate classification results when faced with unsupervised data, clustering method based solely on visual features achieved accuracy of 36.9%. This result does not meet the performance requirements. By treating the clustering results visual pseudo-label and fine-tuning the model under the constraint of distillation loss, the model ultimately achieved an accuracy of 50.4%. Due to the poor clustering performance in the previous step, the visual pseudo labels contain substantial inter-class confusion and label- ing inaccuracies. As a result, relying solely on the supervisory information introduced by the visual space is insufficient to successfully transfer the pre-trained model to downstream tasks. Considering the remarkable stability and knowl-
edge expansion capability of textual informa- tion, we used our generated generalized text descriptors filter the samples. Under this LLM- textual supervision, where diverse and unstructured labels are generated for each image and clustered into category prototypes, the fine-tuned model achieved a performance of 76.8% on downstream tasks. Building upon this, our proposed hierarchical prompting strat- egy queries only a few representative samples per class to construct consistent and discrimina- tive textual prototypes, and further introduces SAD loss to enhance cross- modal consistency. This combined strategy improves the performance to 81.9%. For reference, using ground-truth labels construct text prompts in the form of "a photo of a [label]" yields a supervised upper bound of 84.2%. This comparison demonstrates that our semantically guided pseudo-label design, even without any real labels, achieves performance close to the supervised upper limit.
Table 2: Ablation study on ImageNet-R, 5 tasks.
Components
Accuracy
Pv
Pt
Lkd
Lind
a) b)
36,9
V
50.4
V
76.8
d)
78.1
e)
81.9
To maximize the utilization of supervisory infor- mation from both modalities, we further reduce pseudo-label inaccuracies by adaptively weight- ing each space according to its reliability. On the ImageNet-R benchmark, shown in Ta- 3 this balanced dynamic weighting strategy outperforms the fixed weighting scheme, provid- ing strong evidence for the effectiveness of the proposed method.
Table 3: Dynamic vs. fixed weighting.
Dynamic Weighting (%)
Fixed Weighting (%)
Tasks
5
81.7
80.2
10
82.2
80.3
20
79.8
77.1
4.4 Further Analysis
Annotating images with LMM.
We analyzed the initially proposed approach of directly employing a LMM to annotate the entire dataset and conducted experiments on the ImageNet-R benchmark. The results indicate that LMMs are highly susceptible to interference from complex backgrounds and environmental variations, often leading to inaccurate predictions(as exemplified by the misannotations shown in Fig. 5. This

---

naive annotation strategy not only lacks reliability but also incurs substantial computational costs. These findings underscore the necessity and effectiveness of the proposed Semantic Collaborative Facilitative Supervision framework introduced in this work.
T Soldentah
Bid
S Coge LLW Tay
LLM haining
Spide
Aed ULe and
LLM Batten
LLM: Turte
Cnone
Beople
Hyee
Figure 6: The confusion matrix of pseudo la- bels: the left represents visual pseudo labels, while the right represents textual pseudo la- bels.
Figure 5: Incorrect labels generated by the LMM.
Comparison of pseudo-label prediction results.
In this paper, we propose dual pseudo-labeling strategy for both visual and textual spaces. We randomly selected 10 classes from ImageNet-R and constructed the confusion matrix shown in Figure 6. where the horizontal and vertical axes represent the true labels and pseudo-labels, respectively. Larger values along the diagonal indicate higher accuracy of the pscudo-labels. Experimental results demonstrate that textual descriptions, due to their stronger generalization capability, outperform visual clustering. However, both types of pseudo-labels still contain errors, which led us to design a balanced strategy that integrates predictions from both spaces more effectively leverage supervisory information.
To further validate the robustness and adaptability of the proposed method, we include additional experimental analyses in the supplementary material, including, but not limited to: (1) the impact of the number of representative samples m on the quality of LMM-generated descriptions; (2) the effect of different prompt; (3) sensitivity analysis of hyperparameters (c.g., A1 and 22). We also include additional analyses on the computational and resource overhead introduced by integrating the LMM component, comparing API-based and fully local deployment setups. Detailed results and cost breakdowns provided in Appendix A.5.
5 Conclusion
In this paper, we propose a hierarchical, language-guider approach to generate fine-grained text pseudo labels. By combining these with traditional visual clustering pseudo labels, create mul- timodal weighted text-visual pseudo labels to guide training, addressing the challenge of missing supervision in unsupervised class-incremental learning. We also introduce collaborative align- ment method that uses text eseudo-label, to constrain training and reduce catastrophic forgetting between incremental tasks. Our method is compared with various approaches in unsupervised class- incremental learning, and extensive experiments on benchmark datasets demonstrate its effectiveness. Additionally, we conduct ablation studies to validate the rationale behind our approach.
We sincerely hope that the proposed method offers a novel and impactful perspective for advancing the fields of unsupervised and incremental learning. By strategically harnessing the capabilities of modern multimodal models, our approach aims to unlock the latent structure within vast, unannotated real-world data streams. We believe this work serves step toward more intelligent, scalable, and label-efficient learning systems capable of adapting to the complexity and openness of dynamic environments.
Acknowledgment
This work is supported in part by the National Key Research and Development Program of China (No. 2023YFC3305600) Joint Fund Ministry of Education of China B82214, 8091 (091800072404) National Natural Science Foundation of China (62132016, 62571412, and 62571339 1393), Key Re- search and Development Program of Shaanxi (2024GX-YBXX))) Foundation of China (Grant No. HTKJ2024KL50401),
and National Key Laboratory
10

---

References
[1] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
[2] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrabska-Barwinskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 11<(13):55113352 2017.
[3] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[4] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
[5] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[6] Mare Masana. Xialei Liu, Bartlomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(5):5513- 5533, 2022.
[7] Jiaolong Xu, Liang Xiao, and Antonio M López. Self-supervised domain adaptation for computer vision tasks. IEEE Access. 7:156694-156700 2019.
[8] Chenghao Xu, Jiexi Yan, Muli Yang, and Cheng Deng. Rethinking noise sampling in class- imbalanced diffusion models. IEEE Transactions on Image Processing, 33:6298-6300 2024.
[9] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
[10] Hasan Abed AI Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? arXiv preprini arXiv:2402.01882. 2024.
[11] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36:35544-35575. 2023.
[12] Chenghao Xu, Guangtao Lyu, Jiexi Yan, Muli Yang, and Cheng Deng. Lim knows body language, too: Translating speech voices into human gestures. In ACL, pages 5004-5013, 2024.
[13] Xi Wang, Xu Yang, Jie Yin, Kun Wei, and Cheng Deng. Long-tail class incremental learning via independent sub-prototype construction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 28598 28607, 2024.
[14] Yuwei Zhang, Zihan Wang, and Jingbo Shang. Clusterllm: Large language models as a guide for text clustering. arXiv preprint 20051:2305..4877, 2023.
[15] Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We're afraid language models aren't modeling ambiguity. arXiv preprint arXiv:2304. 14399, 2023.
[16] Chenghao Xu, Jiexi Yan, and Cheng Deng. Keep and extent: Unified knowledge embedding for few-shot image generation. IEEE Transactions on Image Processing, 34:2315-2322 2025.
[17] Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision-language models. In Proceedings of the IFEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12988-12999 June 2024.
11

---

[18] Jing Wang, Zhiqiang Kou, Yuheng Jia, Jianhui Lv. and Xin Geng. Label distribution learning by exploiting fuzzy label correlation. IEEE Transactions on Neural Networks and Learning Systems, 16:8979-8990 2025.
[19] Shivam Khare, Kun Cao, and James Rehg. Unsupervised class-incremental learning through confusion. arXiv preprint arXiv:2104.04450, 2021.
[20] Enrico Fini, Victor G Turrisi Da Costa, Xavier Mameda-Pinedke Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621-9630 2022.
[21] Xi Wang, Xu Yang, Kun Wei, Yanan Gu, and Cheng Deng. Class incremental learning via contrastive complementary augmentation. IEEE Transactions on Image Processing, 34:3663- 3673, 2025.
[22] Quentin Jodelet, Xin Liu, Yin Jun Phua, and Tsuyoshi Murata. Class-incremental learning using diffusion model for distillation and replay. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3425-3433, 2023.
[23] Kun Wei, Xu Yang, Zhe Xu, and Cheng Deng. Class-incrementee unsupervised domain adaptation via pscudo-label distillation. IEEE Transactions on Image Processing, 33:1 13:1188- 1 1198, 2024.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 9748-8763. PMLR, 2021.
[25] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision- Lasoon-annguage models with feature adapters. International Journal of Computer Vision, 132(():581-599) 2024.
[26] An Yang. Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese clip: Contrastive vision-langee laiiugl pretraining in chinese. arXiv preprint arXiv:2211.01333 2022.
[27] Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. Lp++: A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23773-23782, 2024.
[28] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying ami-superviieed learning with consistency and confidence. arXiv preprint arXiv:2001.0766 2020.
[29] Alex Gollz,Villa, Bartlomiej Twardowski, Lu Yu, Andrew D Bagdanov, and Joost Van de Wei- jer. Continually learning self-supervised representations with projected functional regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3867-3877, 2022.
[30] Alex Gomez- Villa, Bartlomiej Twardowski, Kai Wang, and Joost Van de Weijer. Plasticity- optimized complementary networks for unsupervised continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1690-1700, 2024.
[31] Mingxuan Liu, Subhankar Roy, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Large-scale pre- trained models are surprisingly strong in incremental novel class discovery. In International Conference on Pattern Recognition, pages 142. Springer, 2024.
[32] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arxiv preprint Stti:::11010062225 2021.
[33] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1)221-27, 1967.
12

---

A Appendix supplemental material
To further assess the robustness, generalizability, and practical effectiveness of the proposed method, we present series of additional experimental analyses in this supplementary material. These analyses offer deeper insights into the behavior of the model under varying conditions and validate the design choices made in the main paper. Specifically, we investigate the influence of key components and parameters that affect the performance of our approach.
A.1 Impact of the Number of Representative Samples (m1
To assess how the number of representative samples affects the quality of LMM-generated descriptions and overall model performance, we conducted experiments on the ImageNet-R dataset with tasks. The results are summarized in Table
Table 4: Effect of the number of representative samples (m) classification accuracy (ImageNet-R, 5 tasks).
m
m
m
5
81.0
81.4
81.9
82.0
82.2
As shown in Table increasing the number of representative samples generally improves perfor- mance. The accuracy increases steadily from 81.0% (when m 1) to 82.2% (when m = 5). This improvement is attributed to the richer and more diverse descriptions generated by the LMM when provided with more query samples.
In our final setting, we choose = 3 as trade-off between accuracy and computational efficiency. Generating high-quality descriptions via LMM relatively time-consuming, and larger values m incur additional query and processing overhead. The setting m 3 achieves competitive accuracy (81.9%) while keeping the LMM query cost low, making it a practical and scalable choice for continual learning scenarios.
A.2 Effect of Different Prompt Designs
To evaluate the impact of prompt design on the effectiveness LMM-generated textual descriptions, conducted series of ablation experiments using multiple prompt templates. The goal is to determine the sensitivity of our method to prompt formulations and to explore whether certain prompt combinations lead more informative and discriminative descriptions, ultimately improving classification performance.
We abbreviated the prompt templates as where P1, P2, and P3 were all derived from the main text. Additionally, we introduced P4 to increase the diversity of the experiment. P4 consists of the generic prompt "a photo of []". which was completed by directly querying the LMM, serving widely-used baseline template. The results of this experiment on the ImageNet-R dataset with tasks summarized in Table
Table 5: Accuracy (%) ( of different prompt combinations ImageNet-R (5 tasks).
Prompt Combination
Accuracy (%)
Pl + P2+
81.9
Pl
79.4
P2
77.6
P3
76.9
P1 P2
80.8
PI + P3
80.1
P2 P3
78.4
P4
79.0
These results demonstrate several key findings:
13

---

Prompt diversity helps: Combining multiple prompts (PI+P2+P3) yields the best accuracy, suggesting that richer semantic descriptions enhance the model's ability generalize.
Even simple prompts work: Although P4 uses very simple and generic format, it still achieves competitive performance (79.0%), outperforming many prior SOTA baselines, which highlights the robustness of our method prompt variations.
Overall, this experiment confirms that while our method is relatively robust to different prompt formats, careful prompt engineering further boost performance. Additionally, it illustrates that even basic prompts can produce effective representations when combined with our proposed learning framework.
A.3 Sensitivity Analysis of Key Hyper-paramcters (A1, Ag)
In this section, we analyze how variations in the key hyper-parameters A X2 affect the per- formance of our proposed method. These hyper-parameters play critical roles in balancing the contributions visual and semantic modalities during training.
Hybrid Loss Balancing Mechanism
Our method employs hybrid loss function that integrates visual and textual supervision. The balaneing of this hybrid loss is adaptive and depends on the consistency between the predictions from the visual and semantic spaces. Specifically:
If the predicted class labels from both modalities agree, we assign equal weights to both terms.
If the predictions differ, the weights are adjusted dynamically based on the difference in confidence (i.e., probability gap between the top two predicted classes) in each modality.
This mechanism helps to assign higher weight the more confident modality, thereby improving stability and reducing the impact of unreliable pseudo labels.
Sensitiiit of A
Senstivily of As
83.0
$3.0
02.5
82.5
82.0
81.2
Accurssy (%6
11.3
Accuraty
81.0
80.9
81.0
$1.0
80.5
30.5
78.0
78.0
0.8
1.0
1.2
0.01
0.02
0.08
0.04
0.06
A1
Az
Figure 7: Sensitivity analysis of hyper-pane parameters A₁ and X2 with respect to classification accuracy (ImageNet-R tasks). Each bar represents the model's accuracy under a different value of the corresponding hyper-parameter.
Sensitivity Analysis of A
We conducted a sensitivity analysis on the hyper-paramettr \ι, which governs the weight of the visual modality in the hybrid loss. The default setting was X 1.0. To evaluate the robustness of this parameter, we varied its value within a small range, specifically testing values from 0.8 to 1.2 in increments of 0.1.
As shown in Figure 7 model performance remains relatively stable across this range. The accuracy peaks at the default value of X 1.0, and only minor performance degradation is observed when
14

---

moving away from this setting. These results suggest that the method is robust to small perturbations in A1.
Sensitivity Analysis Ag
We further analyzed the sensitivity of X2, which controls the influence of the semantic distillation loss. The default value of X2 was set to 0.03. We explored its effect on classification accuracy by varying it between 0.01 and 0.05 in increments of 0.01.
model achieves the best performance the default value, Although performance decreases slightly when X2 is adjusted by #0.01, the drop remains within 1.3 percentage points, indicating relative stability in this parameter as well. However, Ag appears slightly more sensitive than A1, with more pronounced performance peak at its tuned value.
As depicted in Figure 7
Discussion
Overall, the results illustrated in Figure 7 demonstrate that the proposed method exhibits good robustness moderate changes in both À and Ag. Although the model is not highly sensitive to these seper-parameters, careful tuning can still yield incremental improvements in classification performance.
A.4 Multiple-Seed Evaluation and Accuracy Metrics
To further assess the stability and robustness of our proposed method, we conducted 20 independent runs with different random seeds on all three datasets, following the evaluation protocol described in Section 4.1. We report both the Last Task Accuracy (LA) and the Averuge Accuracy (AA) (-o two standard metrics widely used in class-incremental learning [3] The results are summarized in Table where each score represents the mean accuracy (%) + standard deviation (%) over 20 runs. The small variances demonstrate the strong consistency of our approach under different initialization conditions, while the close gap between LA and AA indicates stable performance across incremental stages.
Table 6: Performance of our method under 20 independent runs, reporting both Last Task Accuracy (LA) and Average Accuracy (AA).
Dataset
Tasks
LA (%)
AA (%)
5
81.7:L0.9
89.8
ImageNet-R
10
82,2+0.7
85.9++0.6
20
79.8
83.4
66.3++1.0
74.8+
CIFAR100
10
63,8::0.8
70.5
20
59.6
65.1
5
67.1
72,4+0.7
CUB200
10
65.3
69.6
20
64.2+00.5
67.8+0.5
These results confirm that our dual-space distillation framework maintains stable performance across multiple independent runs and task configurations. The narrow performance gap between the Last Task Accuracy and Average Accuracy further validates the model's 's robustness and consistent learning behavior throughout the continual process.
A.5 Computational Cost and Feasibility of LMM Integration
We further analyze the computational overhead introduced by integrating the LMM component from two perspectives: (1) the time and memory cost when querying GPT-4-turbo via API, and (2) the fcasibility of fully local deployment using open-source large multimodal model (LMM).
15

---

Time and Memory Cost with GPT-4-turbo (API-based setup)
In our experiments, semantic descriptions were queried from GPT-4-turbo for a small number of representative samples per class (m {0, 2, 4, 4.5)) Here, m denotes the baseline setting without any LMM involvement, where only the visual encoder and adapter are used for forward propagation. We measured training time, memory usage, and the total number tokens queried on the ImageNet-R dataset under 5-task incremental setting.
Table 7: Resource usage when querying GPT-4-turbo during training. Bold values denote model training resources, and italic values denote the LMM querying phase.
m
Train Memory (MB)
Train Time (s)
Tokens Queried
Test Accuracy (%)
0
3144
3912.47
50,4
I
3472
4836.49 + 1809 (236,053)
236K
81.0
2
3474
4931.53 + 4953 (444,700)
445K
81.4
3
3477
4941.75 + 7203 (681,405)
681K
81.9
4 3478
4939.51 + 8553 (931,251)
931K
82.0
5
3481
4998.79 + 9455 (1,158,350)
1.16M
82.2
All GPT-4-turbo queries were performed only during training, not inference. As shown in Table 7 increasing the number of representative samples m slightly extends the total training time, additional cost remains moderate relative to the performance gain. Once trained, the inference pipeline no longer involves any LMM queries and thus maintains the same efficiency as a standard ViT-based model.
Local Deployment with an Open-Source LMM
To evaluate the feasibility complete local deployment, we replaced GPT-4-turbo with the open- source model ----V---2D--Isttuu--PPS-- on four A6000 GPUs using FP8 precision, and the same semantic querying and pseudo-label generation pipeline executed offline.
(via Hugging Face). The model was deployed
Table 8: Resource usage when using locally deployed LMM (Qwen2.5-VL-322) Instruttttt...-P...
Dynamic).
m
Train Memory (MB)
Train Time (s)
LMM Overhead (MB)
Test Accuracy (%)
0 3144
3915.30
50.4
1
3478
4850.23 1483
50.024
79.6
2
3481
4940.17 2721
50,024
79.2
3
3485
4952.31 4140
50.024
80.4
4
3489
4947.95 5946
50,024
80.6
5
3493
5004.80 7213
50,024
80.6
The total memory footprint of the locally deployed Qwen2.5-Vi LL model was approximately 50 GB across four GPUs. Compared with GPT-4-turbo, the accuracy decreased slightly (by about 1-1.5%), but the improvement over the baseline without external textual knowledge (m 0) remained substantial. Local deployment introduces additional memory usage only during training, while the inference phase remains identical to pure visual model. This confirms that full local deployment is feasible without dependence on external APIs or network access.
Summary
Overall, the above results demonstrate that integrating LMM introduces only moderate computa- tional overhead, confined to the training stage. Local deployment is fully feasible on high-memory GPUs (e.g., 4x A6000), and querying small number of representative samples (m < provides an effective balance between accuracy and resource efficiency. The proposed framework thus remains lightweight and practical for both API-based and offline multimodal setups.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline the paper's key contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We acknowledge limitations: (1) Performance degrades on low-resolution datasets (e.g., CIFAR 100) as LMMs struggle with fine details (Sec 4.2); (2) Pseudo-label
quality depends on L.MM outputs, which may introduce noise/bias (Sec 3.2, Fig 5).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspcciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification: We prioritize empirical validation over theoretical proofs, relying on practical algorithms (chusteringddisiillatioo)) and experimental results (Tables 1-2) without formal theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
We present datasets, models, and implementation details in the Experiments
section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
and models (CLIP/GPT-4) but do not provide code/preprocessed data, though future open-soureing is planned for reproducibility.
Justification: We use open datasets
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
oot.eesouubbsssoooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
gewiii/gguee//eeeeeoeooi.iisseee CodeeSumiiiseeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly specify all key experimental details including data splits (5/10/20 tasks), hyperparameters (=1, =0.030.11111 and optimizer (Adam with CosineSchedule).
Guidelines: The paper specifies essential training/test details:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars or statistical significance tests for its experimental results (Tables 1-2). Performance metrics are presented as single- point averages without variance measures.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We do not specify computational resources (e.g.. GPU type, memory, or runtime) for experiments. Only mentions using "a single GPU A6000" (Sec 4.1) without further details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: :ttp:p:://n/uriii httppp/p...pppp.p.tttaaaacaaaaaa
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: violations.
The research complies with the NeurlPS Code of Ethics without any identified
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
20

---

Justification:
The paper focuses on technical contributions (e.g., methodology and bench- marks) and does not explicitly discuss societal impacts, positive or negative.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the datasets and baseline papers in the experiment section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.paperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper explicitly describes using GPT-4 as a core methodological com- ponent (e.g., generating hierarchical text descriptanaions for pseudo-labels in Sec 3.2) and addresses its limitations (noise/bias Sec 4.2).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
23


Reviews:
[{"review_id": "i7TyENKbtH", "reviewer": "Reviewer_jsLU", "summary": "This paper focuses on unsupervised class-incremental learning and proposes a novel framework to address its challenges. The method leverages CLIP and GPT-4-turbo to generate more reliable pseudo labels for training data. It introduces both visual pseudo labels and textual pseudo labels to complement each other. Additionally, knowledge distillation is employed to mitigate catastrophic forgetting. Experimental results demonstrate the effectiveness of the proposed approach.", "questions": "See weakness.\nI will consider to raise my rating if the authors can address all my concerns.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.The motivation behind the proposed method is compelling. The authors aim to address the challenge of inaccurate pseudo labels in existing methods, which is a meaningful problem.\n2. The design of Prompt 3 within the Semantic Collaborative Facilitative Supervision module is both novel and insightful. It captures common visual scenes associated with objects, which is often overlooked in prior works.\n3. The paper is well-written, with high-quality presentation and clear structure.\n\nWeaknesses\n1. Novelty of Semantic Alignment Distillation: The authors utilize stored class prototypes and matrices to alleviate forgetting, which aligns with standard practices in prototype-based continual learning. This technique has been explored in many prior works. The main difference appears to be the use of textual prototypes as positive and negative reference points. The authors should clarify how their approach differs from existing works and emphasize the novelty more clearly.\n2. Lack of Multiple Runs: All reported experiments appear to be conducted only once. To better demonstrate the robustness of the method, the authors should run experiments using multiple random seeds and report the average results along with the standard deviation.\n3. Weight Design Between Visual and Semantic Spaces: The paper lacks theoretical justification or analysis for the weighting mechanism between visual and semantic components. The authors should discuss how the weights are selected and whether any theoretical insight supports this design.\n4. Typographical Error: There is a typo in the description of \"chameleon\" in Figure 1, where “independen-tly” is incorrectly hyphenated.\n5. Ambiguity in Lines 61–71: The explanation in this section is vague. The authors should elaborate on the notions of inherent knowledge bias and inter-category semantic ambiguity. Providing concrete examples would significantly improve clarity.", "was_revised": true, "final_justification": "I appreciate the authors' efforts, and their rebuttal has addressed some of my concerns. However, I find the proposed prototype-based method still to be relatively common. As a result, I have decided to raise my score from 2 to 4.", "rebuttal": "We sincerely thank the Reviewer for the thoughtful feedback and recognition of our work, particularly the novel UCIL framework and the design of the semantic prompt module. Your comments on the SAD mechanism, multiple runs, and weighting strategy are very helpful. We provide detailed responses below and will revise the paper accordingly. The full code and models will be released after publication. We hope this clarification helps the Reviewer reconsider the contributions of our work.\n\n## Novelty of Semantic Alignment Distillation(Weakness 1)\n\nWe would like to clarify that SAD is **not a simple extension of traditional prototype-based methods**, but rather a **cross-modal alignment strategy that leverages the stability of the language space to guide the evolution of the visual space**.\n\nOur proposed **hierarchical prompt template** generates high-quality and consistent class descriptions, forming a set of textual prototypes with strong discriminability and structural stability. Since the text encoder remains frozen during training, the distribution of these features remains unchanged throughout the incremental learning process. We therefore use the relative distances between textual prototypes as a set of **stable anchors**, which are projected into the visual space to constrain its drift and mitigate catastrophic forgetting.\n\nSpecifically, SAD constructs a directional distillation loss (Eq. 10) by comparing each old sample’s distance to its original text prototype and to new class prototypes, thereby enforcing structural alignment between the language and visual spaces.\n\nAlthough SAD may appear superficially similar to certain prototype-based distillation methods in form (e.g., using class prototypes as constraints), **the fundamental design philosophy is distinctly different**. Traditional approaches typically construct prototypes in the visual space and focus on preserving or reconstructing visual feature structures. In contrast, our method **transfers semantic structure from the language space to the visual space via cross-modal alignment**. Finally, our ablation results in Table 2 (rows d → e) empirically demonstrate the significant contribution of SAD to overall performance, validating both its effectiveness and independent value.\n\n## Lack of Multiple Runs(Weakness 2)\n\nWe conducted **20 independent runs with random seeds** on the ImageNet-R dataset under three different task configurations (5 tasks, 10 tasks, and 20 tasks). We report the Top-1 average accuracy along with standard deviation as shown below:\n\n| Tasks | Mean Accuracy (%) | Std. Dev (%) |\n| :---: | :---------------: | :----------: |\n|   5   |       81.7        |     ±0.9     |\n|  10   |       82.2        |     ±0.7     |\n|  20   |       79.8        |     ±0.6     |\n\nWe will include the corresponding results for CIFAR100 and CUB200 in the revised version to further validate the stability and robustness of the proposed method.\n\n##  Theoretical Justification of the Visual-Semantic Weighting Mechanism(Weakness 3)\n\nOur design of the visual-semantic dynamic weighting mechanism is inspired by the general principle in multi-modal fusion that higher-confidence predictions tend to provide more stable and reliable supervision signals (e.g., [1]). Increasing the weight of this part will help improve the model's effectiveness. Building on this intuition, we propose a dynamic weighting strategy based on the confidence difference between modalities, enabling adaptive fusion of supervision signals from the visual and semantic spaces on a per-sample basis.\n\nAs shown in Eq. (6)–(9), we compute the confidence margin between the top-1 and top-2 predicted classes within each modality to estimate its relative reliability, and assign weights accordingly to the visual and semantic pseudo labels. This mechanism allows the model to dynamically adjust the importance of multi-modal supervision based on each sample’s characteristics, thereby enhancing both discriminability and training stability. Following the setup in Section 4.1 of our paper, we conducted experiments on three incremental settings of ImageNet-R, where we replaced the confidence-based dynamic weighting strategy with a fixed weighting scheme (each weight set to 1/2). The experimental results shown in the table below demonstrate the superiority of our proposed weighting mechanism over the fixed-weight strategy.\n\n[1] Sohn, Kihyuk et al. _FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence_, ICCV 2020.\n\n| Tasks | Dynamic weighting Acc(%) | Fixed weighting Acc(%) |\n| :---: | :----------------------: | :--------------------: |\n|   5   |           81.9           |          80.2          |\n|  10   |           82.3           |          80.3          |\n|  20   |           79.4           |          77.1          |\n\n## Typographical Error (Weakness 4)\n\nThank you for your careful review. We have corrected this typographical error in the revised version.\n\n##  Clarification of term(Weakness 5)\n\nWe will further clarify these two concepts and supplement them with concrete examples, as detailed below:\n\n**Inherent Knowledge Bias** refers to the tendency of large-scale pre-trained models (such as CLIP or large language models) to rely more heavily on concepts or feature patterns that appear frequently in their pre-training corpus during open-world reasoning. As a result, the models often exhibit biased predictions for low-frequency or visually indistinct categories. For instance, high-frequency animals like “cat” and “dog” are consistently recognized due to their clear visual appearance and consistent semantic expression, while less common categories such as “weasel” or “lynx” are more prone to representational drift due to their ambiguous semantics or underrepresentation in the training data. This phenomenon is closely tied to the models’ prior semantic knowledge, and partly reflects the statistical bias inherent in the “language model as knowledge base” assumption. For example, Parashar et al. (2024) [1] demonstrate that such differences in prior exposure directly lead to better recognition of frequent concepts and degraded performance on rare categories. In our initially proposed method, directly querying the LMM for each sample led to numerous incorrect labels due to this issue, which significantly degraded the model's performance.\n\n**Inter-category Semantic Ambiguity** refers to cases where different categories share highly overlapping attributes in either the visual or semantic space, making it difficult for the model to draw clear boundaries. For example, “seal” and “sea lion” are semantically distinct species, yet their visual appearances—such as body shape, texture, and typical background—are so similar that vision models often confuse them. Conversely, “octopus” and “jellyfish,” while visually very different, may be semantically conflated due to shared textual descriptors like “marine animal” or “tentacles,” bringing them closer in the text embedding space. Such within-modality ambiguities make unseen-class inference more error-prone.\n\nTo address the two aforementioned issues, our method incorporates several key mechanisms:\n\nFirst, **to mitigate the issue of inherent knowledge bias**, we avoid directly querying the large language model (LMM) for every individual image. Instead, we design a **clustering-based representative sample selection strategy**, where only a small number (e.g.m=3) of representative images per category are selected for LMM description generation. This approach not only reduces the system’s reliance on the variability of LMM outputs, but also enhances the semantic representativeness of the generated descriptions by focusing on the visual centers of each category. Furthermore, we introduce a **hierarchical prompt template** to structurally guide the LMM in generating more stable and accurate descriptions that reflect multi-level semantic attributes. Examples of these prompts and their output consistency are provided in the appendix.\n\nSecond, **to handle inter-category semantic ambiguity**, we propose a **dual-modality co-supervision framework** that leverages both visual and textual spaces for pseudo-label generation and model training. Instead of using predictions from a single modality, our method performs cross-modality alignment between the category structures in the visual and language spaces to improve discriminability. In addition, our proposed SAD mechanism leverages the stable textual prototypes from the language modality as soft constraints in the visual feature space, effectively alleviating category confusion caused by visual feature drift.\n\nIn summary, our hierarchical prompt design, representative sample selection, and dual-modality co-supervision framework work in concert across generation and training phases to systematically alleviate the challenges introduced by inherent knowledge bias and semantic ambiguity.\n\n[1] Parashar, Aditya et al. _The Neglected Tails in Vision-Language Models_, CVPR 2024."}, {"review_id": "5T5oo6l40i", "reviewer": "Reviewer_TKos", "summary": "This paper focuses on class incremental learning of vision-language models (VLMs). While existing approaches have achieved increasing success in supervised class incremental learning, they struggle to deal with complex visual scenarios and fully unsupervised settings that appear in many real applications. This is because the LLMs capture nuanced distinctions among visually similar samples, and inherent knowledge biases and inter-category semantic ambiguity introduce substantial stochastic noise in pseudo labels. To address this limitation, this work designs a hybrid approach that leverages the coarse-to-fine characteristic of language modality to refine the visual samples and also proposes a language-guided collaborative alignment to improve visual feature discrimination. The main contributions of this work are three-fold: (1) proposing a low-resource UCIL that captures the hierarchical semantic description to enhance the semantic supervision quality of unlabeled samples, (2) exploiting dual-modality pseudo-labeling strategy that jointly leverages visual and semantic cues for robust representation learning, and (3) conducting comprehensive experiments that demonstrate the efficiency of the proposed methods.", "questions": "1. For a fair comparison, the compared methods designed for CLIP and methods using LLMs in CIL should be reported in an unsupervised manner. The comparison experiments of CLIP-based class incremental learning methods, such as ZS-CLIP [2], MOEAdapeter [3], and RAPF [4] are not sufficient. Methods with LLMs, such as GMM [5] should also be compared.  \n\n2. It has been proved that the impressive vision-text zero-shot ability of the CLIP in ZS-CLIP [2]. Given the coarse hypothesis that the origin text label is the upper bound performance, the pseudo label quality of pure LLMs and the proposed methods should be compared with the real text labels for precise verification.\n\n[1]Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 2001–2010, 2017.\n\n[2]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021.\n\n[3]Zhou, D.W., Ye, H.J., Zhan, D.C., Liu, Z.: Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need.\n\n[4]Linlan Huang, Xusheng Cao, Haori Lu, and Xialei Liu. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. In ECCV, 2024.\n\n[5]Xusheng Cao, Haori Lu, Linlan Huang, Xialei Liu, Ming-Ming Cheng.  Generative Multi-modal Models are Good Class-Incremental Learners. In CVPR, 2024.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\nThe idea of exploring robust and accurate semantic supervision of unlabeled samples based on the coarse-to-fine descriptions of LLMs and complementary strengths of visual and text information is instructive. The entire framework of constructing the hypergraph and designing the hypergraph convolution is well-explained. The experiments' analysis depicts the effectiveness of the proposed approach. \n\nWeakness \n\nHowever, some aspects of the manuscript and experiment results need further clarification and support. Below, I provide some minor and major comments.\n\n1. The novelty of the proposed method is relatively low in solving the main challenge of unsupervised CIL. The pseudo-labeling technique relies on the LLMs. The main contribution of preventing catastrophic forgetting, corresponding to section 3.3 is also not novel. The metric learning loss in Eq. (10) and distillation loss in Eq. (11) have been broadly used in CIL and unsupervised CIL methods. \n\n2. Page 5, line 194-195, how to determine the most representative m samples from each clustering result? \n\n3. Provide the computation method of the evaluation metric proposed in Section 4.1 . It is recommended to report the results of two metrics, the last stage average accuracy and overall average accuracy following [1], which have been broadly utilized in CIL papers, in the experiment part to validate the performance superiority. \n\n4. Page 6, line 234, the symbol of $i$ should not be subscript.", "was_revised": true, "final_justification": "Thanks for addressing all my concerns. The rebuttal is convincing. I will raise my rating.", "rebuttal": "We thank Reviewer TKos for recognizing the merits of our work, especially your positive comments on our framework that leverages language modality to enhance semantic supervision under the UCIL setting. We acknowledge your suggestions for clarifying the novelty, reporting evaluation metrics, and improving the presentation in certain areas. We respond to each of your comments in detail below, and will include the necessary theoretical discussions and experimental updates in the revised manuscript. We will release the source code to the community and hope that the Reviewer will reconsider the rating of our paper.\n\n\n## Novelty(Weakness1)\n\nWe thank the reviewer for raising this important point. We clarify that the core novelty of our work **lies not in merely calling LLMs**, but in **how we systematically integrate their semantic priors into a closed-loop pseudo-labeling and fusion framework to enable vision-language collaborative supervision**.\n\nTo our knowledge, **this is the first work to incorporate LLMs under the challenging Unsupervised Class-Incremental Learning (UCIL) setting**, where no real labels are available. We avoid querying LLMs for every sample due to inefficiency and noise, as demonstrated in Figures 2 and 5. Instead, we query only a few representative samples per class (e.g., 3), ensuring semantic stability and low overhead. More critically, we introduce a **cross-modal co-supervision mechanism** that combines visual sensitivity with language consistency to enhance learning.\n\nRegarding Section 3.3 and the SAD module, we would like to clarify that SAD is **not a simple extension of traditional prototype-based methods**, but rather a **cross-modal alignment strategy that leverages the stability of the language space to guide the evolution of the visual space**.\n\nOur proposed **hierarchical prompt template** generates high-quality and consistent class descriptions, forming a set of textual prototypes with strong discriminability and structural stability. Since the text encoder remains frozen during training, the distribution of these features remains unchanged throughout the incremental learning process. We therefore use the relative distances between textual prototypes as a set of **stable anchors**, which are projected into the visual space to constrain its drift and mitigate catastrophic forgetting.\n\nSpecifically, SAD constructs a directional distillation loss (Eq. 10) by comparing each old sample’s distance to its original text prototype and to new class prototypes, thereby enforcing structural alignment between the language and visual spaces.\n\nAlthough SAD may appear superficially similar to certain prototype-based distillation methods in form (e.g., using class prototypes as constraints), **the fundamental design philosophy is distinctly different**. Traditional approaches typically construct prototypes in the visual space and focus on preserving or reconstructing visual feature structures. In contrast, our method **transfers semantic structure from the language space to the visual space via cross-modal alignment**. Finally, our ablation results in Table 2 (rows d → e) empirically demonstrate the significant contribution of SAD to overall performance, validating both its effectiveness and independent value.\n\n## Selection of Representative Samples(Weakness2)\n\nOur goal is to achieve ideal recognition performance with a small text supervision cost. We have detailed the representative sample selection process in lines 192–197 on page 5. Specifically, we apply K-means clustering to obtain class centers, then select the top-m samples with the highest cosine similarity to each center. This ensures strong visual coherence and supports the generation of high-quality text descriptions.\n\nAdditionally, we analyze the impact of the parameter _m_ in the supplementary material. As shown in Table 1, increasing _m_ from 1 to 5 improves classification accuracy from 81.0% to 82.2%, suggesting that selecting more representative samples enhances the semantic richness of pseudo-labels.\n\n##  Metrics(Weakness3)\n\nFollowing the reviewer’s suggestion, we have re-evaluated the performance of our method on the ImageNet-R dataset under three different task splits (5-task, 10-task, and 20-task) based on the setting in Section 4.1 of the paper. We now report two more standard evaluation metrics commonly used in class-incremental learning: **Last Task Accuracy** and **Average Accuracy**, as shown in the tables below. Results for the other datasets (e.g., CIFAR-100 and CUB-200) will be updated accordingly in the revised version.\n\n| Tasks | Last Task Accuracy (%) | Average Accuracy (%) |\n| :---: | :--------------------: | :------------------: |\n|   5   |          81.9          |         89.9         |\n|  10   |          82.3          |         86.4         |\n|  20   |          79.4          |         83.2         |\n\n## Format(Weakness4)\n\nThank you for the careful review. We confirm that the subscript formatting of the variable _i_ in line 234 is indeed inconsistent. We will correct this in the final version to ensure consistency and proper notation throughout the paper.\n\n## Inapplicability of Fully-Supervised Baselines under the UCIL Setting(Question1)\n\nWe appreciate the Reviewer’s concern regarding the fairness of comparisons. To begin, we would like to clarify the **fundamental differences between Unsupervised Class-Incremental Learning (UCIL)** and traditional supervised CIL, which are essential for properly contextualizing the scope of our method.\n\nIn the UCIL setting, the model has **no access to any ground-truth class labels throughout training**. At each stage, it receives only unlabeled images from novel classes. In most cases, the number of new classes is known, but their specific identities are not. The model must rely on pseudo-labeling mechanisms and self-supervised structures to gradually construct semantic knowledge—introducing uncertainty and openness far beyond that of traditional supervised setups. Prior work, such as CURL [1] and UCIL-Confusion [2], has demonstrated that methods relying on label supervision are generally unsuitable in this context. Furthermore, the classic clustering survey by Jain et al. [3] emphasizes that in unsupervised clustering, the class structure must be discovered solely based on inter-sample similarity, making the task fundamentally different from supervised learning.\n\nUnder this premise, we have carefully reviewed the methods highlighted by the Reviewer—RAPF, ZS-CLIP, MOEAdapter, and GMM—and based on their design assumptions and workflow, we argue that they cannot be fairly compared under the UCIL setting for the following reasons:\n\n1. **RAPF** relies on ground-truth labels to construct semantic similarity constraints and perform feature distillation and parameter fusion. In UCIL, we can only use pseudo-labels derived from visual clustering or LMM queries. Our preliminary experiments show that these pseudo-labels are of low quality, yielding classification accuracies of only 36.9% (visual space) and 51.1% (language space), which are insufficient to support RAPF's supervision-driven mechanisms.\n2. **ZS-CLIP** assumes access to known class names to construct text prototypes for zero-shot inference. However, class names are unknown in UCIL, making its assumptions incompatible with ours.\n3. **MOEAdapter** requires explicit class labels at each stage to activate expert routing and build supervised prototypes. These components cannot function without supervision and thus are not transferable to UCIL.\n4. **GMM**, though incorporating a language module, relies heavily on class labels for pseudo-label modeling, contrastive learning, and text alignment. When labels are unavailable, their core mechanisms fail to operate and may even collapse due to semantic misalignment.\n\nIn summary, while these methods are representative under their respective **supervised** settings, they are all fundamentally dependent on labeled data for both training and structural design. This makes them incompatible with the **fully unsupervised UCIL** framework we propose. Therefore, including them in direct comparison under the UCIL setting would be theoretically inappropriate and practically infeasible. Instead, our experiments focus on comparisons with established UCIL methods and demonstrate the robustness and effectiveness of our approach on benchmarks such as ImageNet-R.\n\n**Reference:**\n\n[1] Rao, D. Continual Unsupervised Representation Learning. *Advances in Neural Information Processing Systems.\n\n[2] Gautam, R. Unsupervised Class-Incremental Learning Through Confusion. *arXiv preprint arXiv:2110.06674*.\n\n[3] Jain, A. K. Data clustering: A review. *ACM Computing Surveys (CSUR)*, 31(3), 264–323.\n\n## Verify Text Label Quality(Question2)\n\nWe thank the reviewer for the valuable suggestion. To evaluate pseudo-label quality, we conducted three experiments on ImageNet-R with five incremental tasks:\n (1) In an open-world setting, the LLM generates diverse, unstructured labels for each sample. To address this, we applied clustering to the generated texts and utilized the cluster centers as class prototypes, achieving an accuracy of 76.8%.\n (2) Under the same unsupervised assumption (unknown classes), our proposed hierarchical prompting queried only a few representative samples per class to construct consistent and discriminative prototypes, improving performance to 81.9%.\n (3) Using ground-truth labels to construct “A photo of [label]” text prompts served as a supervised upper bound (84.2%).\n\nOnly (3) uses real label names. Our method, without any true labels, achieves performance close to the upper bound, confirming the effectiveness of our semantic-guided pseudo-labeling design. Detailed experiments will be supplemented in the revised version."}, {"review_id": "VMFJcu4c4c", "reviewer": "Reviewer_Mfes", "summary": "This paper uses a multimodal approach to do unsupervised class incremental learning. It uses the visual embeddings initially, then textual descriptors on cluster centers, to help guide cluster annotation. The results look pretty good.", "questions": "All already asked", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I personally felt this is one of the better written papers I've seen in this area; the whole story was very clear and well motivated. The results also look very good. I have some comments that could maybe be considered for improving the paper, and I have to admit I'm not familiar with related work so I can't comment on novelty, but for the most part the paper seems a solid contribution to me.\n\nweaknesses / areas for improvement:\n - Sometimes the language is a bit precise. For example, using \"noisy\" to describe the visual embedding system feels imprecise; after all, it's not the images or the labels that are noisy. I guess it's something like, a lizard and gecko may look alike, but the image embedding doesn't always discriminate at that precise level as, say, the orientation of the scales or distance of the eyes. This seems more like a lack of focused or intentional granualiry, rather than noise.\n\n - in the abstract, what is \"semantic alignment distillation\"? Is this a commonly used thing? \n - Why is language more stable than vision? Is this a feature / fault of the model, or of the data distribution, or something else?\n\n - line 70: biases in labels cannot intorduce *stochastic* noise. In general I think the use of words like \"randomness\", \"noise\", and \"stochastic\" should be more carefully doled out; if something is inherent in the task or dataset, it cannot be random. See also line 187, line 223.\n\n - Minor: 162: \"in the figure\" which figure?\n\n - minor: 235: could be more explicit that C^t isn't an exponent, but you compute this for all j = 1,...,C^t and then pick the top 1,2. Maybe describe (6) in two lines? \n\n - (11),(12) How sensitive is this to the choice of lambda1, lambda2? I guess drift could be a big issue? \n\n - line 287: do you mean after training, or during training? Do you evaluate during the sequence, or all at once after the sequence is doled out?\n\n - imprecise: line 319 \"This performance is attributed to the low resolution of CIFAR images....\" But this is true for all the other baselines, no? The more precise the language, especially in experimental results, the better.", "was_revised": true, "final_justification": "The overall positive review need not change", "rebuttal": "We sincerely appreciate Reviewer Mfes for the encouraging feedback, especially the recognition of the clarity and potential of our methodology and manuscript presentation. Your comments have helped us identify areas where the language, terminology, and experimental explanations can be further refined. We address each of your points below and will revise the manuscript accordingly to improve precision and clarity. We also plan to release our code and models upon publication to ensure reproducibility. Thank you again for your support and valuable suggestions.\n\n## Clarification on the Use of the Term “Noisy” (Improvement 1)\n\nWe sincerely thank the reviewer for the insightful comment. You are absolutely right that the term “noisy” used to describe the visual embedding system may be imprecise. Our original intent was to address the inaccuracy of the generated pseudo labels, which arise when visual features fail to distinguish between fine-grained categories fully. In the literature on unsupervised and noisy-label learning, this phenomenon is often referred to as the label noise problem [1].\n\nWe greatly appreciate your suggestion, and in the revised version, we will adopt more precise terminology, such as “sub-optimally discriminative representations” or “lack of fine-grained semantic discrimination.” We will also clarify in context that our concern lies with the discrepancy between pseudo labels and ground-truth semantics, not with stochastic noise in the feature extraction process itself.\n\n**Reference:**\n\n[1] Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I. W., & Sugiyama, M. (2020). _A Survey of Label-noise Representation Learning: Past, Present and Future_. arXiv preprint arXiv:2011.04406.\n\n\n\n## Clarification on “Semantic Alignment Distillation” (Improvement 2)\n\nThank you for pointing out the ambiguity regarding the term “semantic alignment distillation.”\n\nWe acknowledge that **“semantic alignment distillation” is not a standard term in the literature**, but rather a novel distillation strategy proposed in this paper. The core idea is to **leverage the stability of the semantic space (i.e., the text encoder representations) to align and constrain the visual feature space**, thereby mitigating catastrophic forgetting in the class-incremental learning process.\n\nConcretely, after training on each task, we preserve the semantic prototypes and covariance matrices derived from pseudo textual labels. During subsequent tasks, we use these saved semantic prototypes to guide the model such that visual features of old-class samples remain close to their corresponding semantic descriptions while being **repelled from those of new classes**. This contrasts with traditional distillation approaches (e.g., logit-based or feature matching), and instead **introduces a cross-modal (semantic-to-visual) constraint**.\n\nWe chose the term _semantic alignment distillation_ to emphasize two aspects:\n\n1. **Semantic**: The targets for distillation originate from the **semantically stable text space**, not from the evolving visual encoder.\n\n2. **Alignment distillation**: The goal is to **align visual representations with previously acquired semantic knowledge** across tasks, ensuring continuity in the feature space despite the incremental setup.\n\n\n\n ## The Stability of the Language Space Compared to the Visual Space (Improvement 3)\n\nWe thank the reviewer for raising this important point. Our claim that the **language (text) space is more stable than the visual space** is based on both theoretical reasoning and empirical design:\n\n1. **Frozen representations**: Textual features generated by the LMM (GPT-4-turbo) remain frozen during training, providing consistent semantic anchors across tasks. In contrast, visual features are updated via task-specific adapters, making them more prone to drift.\n2. **Category-level abstraction**: Language naturally encodes semantic concepts at the class level and is less sensitive to variations such as pose or background, offering more stable and generalizable representations.\n3. **Temporal consistency**: While visual distributions evolve with each new task, textual prototypes—built from a few representative samples—remain unchanged, helping to preserve semantic continuity.\n\nThese properties justify our use of the text space as a stable reference for semantic alignment across incremental tasks.\n\n\n\n##  The Use of the Term “Stochastic Noise” vs. “Bias” (Improvement 4)\n\nWe thank the reviewer for pointing out the imprecise use of “stochastic noise.” We agree that **biases inherent to the data should not be described as random**. Our original intent was to express that **semantic ambiguity and bias lead to label inconsistency**, not true randomness.\n\nTo address this, in the revised version:\n\n1. We will replace “stochastic noise” with more accurate terms such as “label uncertainty” or “pseudo-label inconsistency.”\n\n2. We will add clarifications to indicate that “noise” refers to semantic misalignment or unreliability in pseudo-labels, not statistical perturbations.\n\nWe appreciate the reviewer’s careful reading and will revise the manuscript accordingly to improve clarity and rigor.\n\n## Clarification of Figure Reference in Line 162(Improvement 5)\n\nThis refers to our system framework, which is shown in **Figure 3**. Thank you for pointing this out. We will clarify the reference in the final version.\n\n## Clarification of Formula and Notation for C^t in Line 235 (Improvement 6)\n\nWe thank the reviewer for pointing out the ambiguity in our notation involving C^t in line 235.\n\nIn our original writing, C^t was intended to denote the **number of classes in the current task**, but we recognize that the superscript form could be easily misinterpreted as an exponent. Moreover, the compact expression we used may have hindered interpretability.\n\nIn the revised version, we will take the following actions to address this issue:\n\n1. **Explicitly clarify that C^t denotes the number of classes**, not an exponent.\n2. **Rephrase the original expression into clearer steps**, improving readability and interpretability of our confidence-based weighting mechanism.\n\nWe believe these revisions will eliminate confusion and enhance clarity. We appreciate the reviewer’s insightful and constructive feedback.\n\n## The Sensitivity of Hyperparameters $\\lambda_1$ and $\\lambda_2$ (Improvement 7)\n\nWe thank the reviewer for highlighting the importance of hyperparameter sensitivity. We have conducted a sensitivity analysis of the hyperparameters $\\lambda_1$ and $\\lambda_2$, and the corresponding results are included in the supplementary material.\n\n\n\n## Clarification of Evaluation Strategy and Metrics (Improvement 8)\n\nThank you for your question regarding the evaluation protocol. In our main experiments, we adopt the standard **last-stage evaluation strategy**, where the model is evaluated **after completing all tasks** on **the entire set of classes** learned so far. Accordingly, the main metric we report in the paper is **Last Stage Accuracy (LA)** — a commonly used measure in unsupervised class-incremental learning (UCIL) to assess final performance.\n\n\n\n## Clarification on the Impact of Image Resolution on Method Comparison(Improvement 9)\n\nWe agree with your point: **all methods are evaluated at the same image resolution**, so we cannot simply attribute the lower resolution as a unique disadvantage of our method. Our intention was not to emphasize that \"low resolution leads to poorer performance in our method,\" but rather to highlight that, **compared to datasets with higher resolutions (such as ImageNet-R and CUB200)**, generating LMM-based semantic descriptions on CIFAR-100 is more affected by image quality limitations, which indeed impacts the extraction and discrimination of semantic information to a greater extent.\n\nIn other words, the key component of our method—**the fine-grained semantic description capability of the LMM**—is less stable and effective in the CIFAR-100 setting compared to higher-resolution datasets. This has a greater impact on the quality of the pseudo-labels generated by our method. To avoid misunderstanding, we will clarify this statement in the revision and more accurately emphasize that the issue arises from **the higher dependency of our method on the semantic information perceived from the images**, rather than the dataset itself."}, {"review_id": "RqCDwhbNba", "reviewer": "Reviewer_MwSA", "summary": "In this work, the authors propose a method for Unsupervised Continual Learning. They use a dual-space semantic approach where the vision component is processed through a CLIP ViT-L/14 encoder, and GPT-4-turbo processes the text. The approach consists of extracting the visual embeddings of the samples in the current task and creating clusters. The most significative images (i.e., the top-k samples closest to each centroid) are then used to query GPT-4-turbo for preliminary predictions. In addition to the image, the LMM is prompted with 3 different prompts to have different levels of granularity. These two embeddings are merged together to create pseudo-labels based on the confidence of the predictions of the two spaces. After the pseudo-labeling, an adapter for the visual encoder is trained using an objective considering the agreement between the LMM and the visual encoder. Finally, they also add a knowledge distillation component to reduce forgetting. The authors evaluate their approach on ImageNet-R, CIFAR100, and CUB200 with various classes per task. The evaluation comprises an ablation of all the components in the objective function. In the supplementary materials, the authors perform further ablation and analysis.", "questions": "- How do the methods compare in a fair environment with the same backbone?\n\n- Does the LMM introduce a noticeable overhead regarding time and memory requirements?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The manuscript is straightforward and easy to read.\n\n- The pipeline is interesting and smartly introduces LMMs in the whole framework.\n\n- The results are really promising.\n\n- The ablations are exhaustive and confirm the importance of each component.\n\n## Weaknesses\n\n- The results of the competitors reported in Tab. 1 employ different backbones with respect to the pretrained ViT-L/14 used by the authors. Indeed, the numbers are extremely close to those reported by the respective papers. Specifically, MSc-iNCD employs a ViT-B/16,  while POCON, PFR, and CassLE use a ResNet-18. In addition to the fact that the authors also use GPT-4-turbo (while the number of parameters of the turbo version is not public, GPT-4 has around 1.7 trillion parameters), this results in a very unfair comparison. The authors should use the same visual backbone for all methods and better report the competitors' backbone.\n\n- A computational analysis and comparison are missing, which is important given that the employed LMM is not negligible.", "was_revised": true, "final_justification": "My initial concerns were primarily due to unclear reporting of backbone consistency and the lack of analysis on LMM-related computational overhead. The authors have now provided thorough clarifications and additional experiments addressing both points, including detailed evaluations of API-based and local deployments. The method is well-motivated, technically sound, and demonstrates strong empirical results. Based on these improvements, I am updating my rating to Accept.", "rebuttal": "We thank Reviewer MwSA for recognizing the strengths of our work, particularly the effective integration of multimodal information under the UCIL setting and the comprehensive ablation study validating each module. We appreciate your concerns regarding backbone consistency and the computational overhead of LMMs, which we address in detail below. We will further clarify these aspects in the revised manuscript and release the complete code to the community. Thank you for your constructive feedback and we look forward to your updated rating.\n\n## Clarification on Backbone Consistency Across Methods in Table 1(Questions 1)\n\nAll results in Table 1 have already been reproduced using the same visual backbone **ViT-L/14**. We apologize that this was not clearly stated in the main paper, which may have caused confusion. We will revise the manuscript to explicitly specify the backbone used for each baseline and add a note below the table indicating that **all results are reproduced with ViT-L/14** for fair comparison. We are committed to making the code publicly available for community research in the final version.\n\n\n\n## Time and Memory Overhead of Using LMMs(Questions 2)\n\n**Unsupervised Class-Incremental Learning (UCIL)** is a highly practical yet extremely challenging setting, primarily due to the complete absence of ground-truth labels throughout the continual learning process. With the emergence of large language models (LLMs), leveraging their semantic prior knowledge as a form of supervision has become a promising direction. However, in our experiments conducted on the five-stage incremental setting of ImageNet-R (as described in Section 4.2), we observed that this strategy incurs **significant computational overhead** without delivering corresponding performance gains.\n\nSpecifically, we experimented with using LLMs to generate labels for **every single sample**, which were then used for text-based clustering (**Text Clustering**) and downstream model training. Even when combined with visual feature alignment (**Prototype + Language KD**), this approach achieved only **76.8%** accuracy. In contrast, our proposed strategy selects only a **few representative samples per class** (e.g., 3 samples) for LLM queries to construct class-level semantic prototypes. This dramatically reduces the computational cost while yielding better performance. Moreover, when combined with our **proposed Semantic Alignment Distillation (SAD)** mechanism, the method achieves the best results.\n\n| Text Clustering Only | Prototype + Language KD | Visual + Prototype + Language KD | Full Method (Ours) |\n| :------------------: | :---------------------: | :------------------------------: | :----------------: |\n|         51.1         |          76.8           |               78.1               |      **81.9**      |\n\n\n\nThese results clearly demonstrate that **our method not only alleviates the time and memory overhead brought by LLMs, but also achieves superior classification performance**. This strategy—**selective LMM integration combined with cross-modal collaborative supervision**—strikes a favorable balance between efficiency and effectiveness, showing strong practicality and scalability in real-world UCIL scenarios."}]

Author rebuttal:

Meta-review: The paper tackles unsupervised class-incremental learning (UCIL), where models learn from a stream of unlabeled data without forgetting previously acquired knowledge. Traditional methods rely heavily on visual clustering, which often produces noisy pseudo-labels. This work proposes a dual-space approach that fuses visual and textual representations to generate more reliable pseudo-labels.

Strengths:
- The paper offers a compelling motivation by addressing the problem of inaccurate pseudo-labels in continual learning, clearly identifying a research gap and justifying its importance.

- The proposed method is both novel and insightful, intelligently integrating large multimodal models with a coarse-to-fine prompting strategy to deliver robust semantic supervision for unlabeled data streams.

- The method achieves strong experimental results, and ablation studies convincingly validate the importance and effectiveness of each component in the framework.

- The manuscript is exceptionally well-written and well-structured, making the content easy to follow and understand.

Weaknesses:
- The prototype-based component remains somewhat conventional compared to more radical continual learning techniques.

- It is not fully clear whether the claim that language semantics are inherently more stable than visual features is sufficiently substantiated or risks subjectivity.

Final Recommendation:

The initial reviewer ratings were mixed. During the rebuttal phase, the authors provided additional explanations and introduced new experimental results that addressed all major concerns. After rebuttal, reviewers reached a strong and consistent consensus that this paper makes significant contributions and should be accepted.
Decision: Accept (poster)