Naming to Learn: Class Incremental Learning for Vision-Language Model with Unlabeled Data
Authors: Qiwei Li
Xiaochen Yang
Jiahuan Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
incremental learning
vision-language model
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Class Incremental Learning (CIL) enables models to adapt to evolving data distributions by learning new classes over time without revisiting previous data. While recent methods utilizing pre-trained models have shown promising results, they often assume access to fully labeled data for each incremental task, which is often impractical. In this paper, we instead tackle a more realistic scenario in which only unlabeled data and the class-name set are available for each new class. Although one could generate pseudo labels with a vision-language model and apply existing CIL methods, the inevitable noise in these pseudo labels tends to aggravate catastrophic forgetting. To overcome this challenge, we propose a method named N2L employing a regression objective with mean squared error loss, which can be solved in a recursive manner. To refine the pseudo labels, N2L applies feature dimensionality reduction to the extracted image features and iteratively updates the labels using a classifier trained on these reduced features. Furthermore, a bi-level weight adjustment strategy is proposed to downweight low-confidence pseudo labels via intra-class adjustment and compensate for pseudo-label class imbalance through inter-class adjustment. This incremental learning with adjustment can be solved recursively, yielding identical performance to joint training with unlabeled data and thereby mitigating forgetting. Our theoretical analysis supports the effectiveness of the pseudo label refinement process, and experiments on various datasets demonstrate that our proposed method outperforms SOTA methods. Code is available at https://github.com/zhoujiahuan1991/ICLR2026-N2L

Published as a conference paper at ICLR 2026
NAMING TO LEARN: CLASS INCREMENTAL LEARNING FOR VISION-LANGUAGE MODEL WITH UNLABELED DATA
Qiwei Li', Xiaochen Yang² & Jiahuan Zhou
'Wanxxuan Institute of Computer Technology, Peking University
Harbin Institute of Technology (lqu, Heew,jiahannee jiahuanzhou @pku edu. en, 202221 iit
ABSTRACT
Class Incremental Learning (CIL) enables models to adapt evolving data distri- butions by learning new classes over time without revisiting previous data. While recent methods utilizing pre-trained models have shown promising results, they often assume access to fully labeled data for each incremental task, which is often impractical. In this paper, we instead tackle more realistic scenario in which only unlabeled data and the class-name set are available for each new class. Although one could generate pseudo labels with a risoonlanguuage model and apply exist- ing CIL methods, the inevitable noise in these pseudo labels tends to aggravate catastrophic forgetting. To overcome this challenge, we propose a method named N2I. employing regression objective with mean squared error loss, which can be solved a recursive manner. To refine the pseudo labels, N2L applies feature dimensionality reduction to the extracted image features and iteratively updates the labels using a classifier trained on these reduced features. Furthermore, bi-level weight adjustment strategy is proposed downweight low-confidence pseudo labels via intra-class adjustment and compensate for pscudo-label class imbalance through inter-class adjustment. This incremental learning with adjustment be solved recursively, yielding identical performance to joint training with unlabeled data and thereby mitigating forgetting. Our theoretical analysis supports the ef- fectiveness of the pseudo label refinement process, and experiments on various datasets demonstrate that our proposed method outperforms SOTA methods. Code is available at htttp://gthbb..cm///oojjaaauu 1991/ICLR2026-ND
INTRODUCTION
Class Incremental learning (CIL) (Zhou al. 2024c a Wang et al. 2024), has attracted sustained attention in recent years due to its ability to adapt models to continuously evolving data scenarios, enabling lifelong lcarning capabilities. In the CIL setting, new classes are introduced sequentially, and the model must learn them without direct access to previously seen data, while maintaining unified model capable of recognizing all encountered classes. The core challenge lies in mitigating catastrophic forgetting (French, 1999), a phenomenon where newly acquired knowledge overwrites and degrades previously learned information.
Recent advances in CIL utilizing traitraine model-based method (Zhou et al. 2024a), such as Vision Transformers (VIT (Dosovitskiy et al. (02TT) and Contrastive Language- Image Pretraining (CLIP (Radford et al. 2021)), have shown promising performance in CII. Leveraging their rich prior knowledge, these methods achieve competitive performance by fine-tuning only classification heads and subset of parameters. However, they typically assume fully labeled data for all incremental tasks which is often impractical in real-world applications where annotations are scarce or costly. To address this, as shown in Fig. [. we propose a more realistic class incremental learning paradigm: at each incremental task, the model receives only class names and unlabeled data for each category.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
A straightforward solution is to exploit a pre-trained vision-language model: convert each class name into its textual embedding, compute similarities between image and text embeddings, and use the highest-scoring class pseudo label for each image. However, these pseudo labels are inherently noisy, which degrades performance and exacerbates forgetting.
To address these challenges, propose N2L, a method for CIL with frozen vision-language models and unlabeled data. Inspired by ana- lytic CIL (Zhungg et al., 2022). N2L adopts mean squared error regression objective in- stead of the standard cross-entrope loss, which has been shown to more robust to noisy la- bels (Ghosh et al., 2017). Prior work (Zhuang al. 2022) has demonstrated that combining such regression loss with recursive update scheme in an incremental setting, effectively mit- igating forgetting. Building this foundation, we first develop a novel pseudo label refinement mechanism. Unlike previous method (Xu et al., 2024) that increases feature dimensionality. our method applies feature dimensionality reduction and trains label refinement classifier on the reduced features to iteratively update the pseudo labels. We further provide theoretical guarantees for the effectiveness of this procedure. Additionally, N2L incorporates a bi-level weight adjustment strategy: inter-class adjustment address class imbalance introduced by noisy pseudo labels, and intra-class adjustment to downweight low-confidence samples that are more likely to be mislabeled. Finally, we derive a recursive formulation regression with weight adjustment, ensuring performance identical joint training with unlabeled data and alleviating forgetting.
Unlabeled Data
Vision-Languagee
Model
Class
Classes:
Classes: Battery, Hat, Notehook
Bird, Cur.
Banana
Figure 1: Class incremental learning for vision-
language model with unlabeled data.
In summary: (1)
We propose a practical class incremental learning setting in which only unlabeled data and class names are available at each task. (2) We propose feature dimensionality reduction-basee pseudo label refinement method, supported by theoretical analysis.
(3) We design inter-class and intra- class adjustment schemes compensate for pseudo-label class imbalance and incorporate confidence information from noisy pseudo labels. (4) Experiments on various benchmarks demonstrate that N2L outperforms state-of-the-art approaches by a large margin.
RELATED WORK
2.1
CLASS INCREMENTAL LEARNING
Class incremental learning methods can be broadly categorized into three directions: parameter regularization, exemplar replay, and architectural expansion. Regularization-basee methods introduce additional constraints to limit changes in model parameters (Aljundi et al., 2018 Kirkpatrick et al.
2017) intermediate representations (Li & Hoiem 2017: Zhang al. 2020 Kang et al. 2022
Li et al. 2024a), aiming to preserve performance without accessing previous data. Replay- based approaches typically
representative samples features from previous learned classes. Some works focus on selecting more effective samples (Sun et al.) 2023 Liu et al. 2220 Lopez-Paz & Ran- zato 2017). while others investigate how to and utilize the feature representation of each classes (7hu et al. 1 Toldo & Ozay 2022; Li et al., 20224b.. Architecture-base
methods (Douillard et al. Wang al.) 2022a] Hu al. (202)) dynamically expand model capacity by introducing new
2022
parameter modules for new classes while keeping existing parameters fixed.
2.2
PRE-TRAINED MODEL BASED CIL
Recent CIL research leverages pre-trained models due to their rich prior knowledge. These methods typically adopt parameter-efficien efficient fine-tuning (PEFT) to adapt to new tasks while keeping the
backbone frozen. A large portion of these methods build on VIT. Some works (Wang et al. 2022c by Smith et al. 2023 Wang et al. 2023) Li & Zhou 2025) introducing learnable prompt parameters along with corresponding prompt selection weighting strategies. While others employ LoRA (Liang & Li
2

---

Published as a conference paper at ICLR 2026
2024
Wu et al. 2025) Adapter (Tan et al. 2024 Zhou al. 2024b) to achieve efficient adaptation. More recently, there has been growing interest in using CLIP as the backbone, motivated by its strong multimodal capabilities and generalization performance. Several studies (Zheng al. 2023
2024 Gao et al. 2024 Yu et al. 2024) aim to preserve CLIP's generalization while adapting to new tasks via parameter regu regularization task identification mechanisms. Other approaches (Huang et al. 2024 Zhou et al. 2025a) leverage CLIP's image-text alignment capability to maintain knowledge of old classes and enhance discrimination of new classes through textual guidance task adaptation techniques. Additionally, some replay-based methods (Zhou et al., 2025b; Jha et al. (2024) improve inter-class feature consistency under limited exemplar budgets using probabilistic modeling or task- specific module expansion. However, all these methods assume that accurate label is available for each training sample, which can be labor-intensive.
2.3 LEARNING FROM UNLABELED DATA
In many real world scenarios, category labels are scarce, and the majority of available data is unlabeled. creating demand for effective learning from unlabeled data. Pscudo labeling methods (Lee et al.) 2013; |asmus et al.) 2013; |Shnn et al. 2020) have been widely used to address this challenge. In the context of vision-language models, UPL (Huang al. 2022) proposes generating more reliable pseudo labels by selecting multiple examples with the highest confidence for each class. LaFTer (Mirza et al. 2023) generates pseudo labels by training a pure text classifier on a corpus of text data generated by a large language model. CPL (Zhang et al., (024)) introduces a candidate pseudo label generation strategy that select reliable pseudo labels by intra- autr-instance and inter-instance confidence score. However, these methods assume a static learning scenario where all data is provided once. In this paper, we propose a new setting, class incremental learning with unlabeled data, in which the model must continuously acquire new knowledge from unlabeled data while mitigating forgetting of previously learned information.
3 METHOD
3.1 PRELIMINARIES
Problem definition. CIL with vision-language models considers the scenario where a pre-trained model, such as CLIP, composing an image encoder and text encoder V = (fing fwa)), learns from sequence of tasks (Ti, (Ti,T..... T Tr Each task T, introduces a disjoint set of classes Yr, satisfying )% n 0 for i * j. At each task the model is trained with new data D, but has access to previously seen datasets DI...., Dr-1. The goal is to update the model to recognize all classes seen far while mitigating catastrophic forgetting. In this work, we address a more realistic and challenging setting in which n, unlabeled samples, 24, = {I, f- and a corresponding class name set C, = fCylu E } are available. Hence, the dataset at task f is defined as D, {14, C.
3.2 OVERVIEW
As shown in Fig. the overall pipeline of N2L during task / has four steps: (1) Pseudo Label Generation. Using the zero-shot capability of the pre-trained vision-language model, pseudo labels, Y, are assigned to unlabeled images based on the similarity between visual and textual features. (2) Progressive Label Refinement. We apply feature dimensionality reduction to the extracted image features X, of task f, obtaining a reduced representation X₁ The pseudo label is iteratively updated by label refinement classifier, W which is learned using the reduced representation and pseudo label with the objective of regression. (3) Bi-level Weight Adjustment. To compensate for pseudo- label class imbalance and utilize the confidence information of samples, we propose a bi-level weight adjustment strategy. A recursive solution is derived for learning with weight adjustment. (4) Finally, an incremental classifier, W, is learned using full features X, updated label Y%
3.3 ANALYTIC CIL WITH UNLABELED DATA
Pseudo Label Generation. The absence of labels necessitates reliable pseudo-labeling to associate each image with class. Leveraging the zero-shot capabilities of pre-trained vision-language model
3

---

Published as a conference paper at ICLR 2026
Progressive Label Refinement
Text Classifier
A photo of vict -bikke --bird "pplane
Text Encoder
Logi
Argmax Class
Logit
Class
Bi-level
Weight
Adjustment
Image Encoder
Feature Reduction
Updated Logit
Classifier
Bi-level Weight Adjustment
Task
Bi-level Weight
Intra-class Adjustment
Adjustment
Entropy
E Sort
Sumpled Weight
Logit
At-i M,Xe Gy-1 M.Y,
Inter-class Adjustment
+21
Argmax
Class
Task
Apug
Figure 2: Overview of N2L during task First, each unlabeled image is assigned a pseudo label using CLIP. To refine these noisy labels, feature dimensionality reduction is applied to the extracted features, and the pseudo labels iteratively updated using label refinement classifier W learned by a regression objective. Meanwhile, intra-class and inter-class adjustment strategies are introduced to leverage sample confidence and address class imbalance. Finally, the incremental classifier W, is learned using the updated pseudo labels and the weight adjustment strategy.
(e.g., CLIP), we generate a pseudo label Jt for each unlabeled image T; E Ur based on the similarity between visual and textual features:
3 arg max (fimm fret
(I)
Pe is the prompt for class (e.g.. photo of [CLASS]").
While this approach enables CIL without annotations, the generated pseudo labels are often noisy, Although applying cross-entropy loss is the common approach for classification tasks, existing method (Ghosh et al., 2017) has shown that mean squared eπто (MSE) loss is more robust to noise. Recent studies in CIL (Zhuang al. 2022; Xu al.) 2024 Zhuang et al., [2024] Fang et al. (024) further demonstrate that using MSE loss along with recursive update of the classifier achieves
performance comparable to joint training, effectively mitigating forgetting.
Analytic CIL. Given extracted features of unlabeled data of task 1 to T, X1.7, (each row consists of a image feature, fimg(2), , for the unlabeled data .) and one-hot label tensor Y 1.T,, the training objective of joint training all tasks is formed ridge regression (Zhuang et al. 2022):
L C(Wr) I. Wr YI. A WT
(2)
The closed-form form solution of WT is:
WT arg min ((Wr) (Ar + AI) AII- WT
where matrix Ar and Cr can be calculated in an recursive form, with Ag and Co are initialized as zero matrices:
A: = = xX, X., C: C
Then, by obtaining and updating the matrix A and C, during the training of each task t, the final classifier WT converges to the same solution as that obtained through joint training. The derivation provided in Appendix. F
4

---

Published as a conference paper at ICLR 2026
3.4 PROGRESSIVE LABEL REFINEMENT
In the task of CIL with unlabeled data, the pseudo label generated by zero-shot CLIP can be noisy, and tackle this problem, we propose a label refinment method based on the dimensionalityy reduced features. For the task I, we first apply singular value decomposition (SVD) to the extracted feature matrix with feature dimension d. X UDVT, where X, E RR U E RR.xx,
V [V1,- Val E Rddd The i-th diagonal elements of D is d, We select the top-k singular vectors V = [V1, val whose singular values are above the threshold and project the features as: X,, X,V, R We then perform regression using the reduced features Xx, and pseudo label Y, obtain a refining classifier W. The refined pseudo labels are computed as:
arg max X W
This process is iterated several times progressively refine the pseudo labels.
We provide a theoretical guarantee for the effectiveness of our pseudo label refinement method:
Theorem 1. Consider the regression model with noisy labels: y Xw" + where features X E R w E I is the true regression coefficient vector, and & is the noise with zero mean and variance o². Let the SVD of X UDV - (V1, Vall 0g is the i-th coordinate of vector a V "w"
If 2 2 (2X + d3) (az)² projecting X onto the eigenvectors without j-th components (denoted
X = UDV V') yields a smaller expected mean squared error:
E x Xw" 12: E X - Xw" 112.,
(6)
where @ and w are the regression solutions using X and X', V [V....., VJ-1, V2+11 val. À is the regularization hyperparameter in ridge regression, typically set to 0.1.
Proof. Provided in Appendix. E.1
In this theorem, a3 represents how much of the w* lies in the j-th singular direction of X, and dj is the singular value corresponding to that direction. This theorem implies that when a is small (little true signal) or d, is small (poor information of X contained in the j-th direction), the model is more likely to overfit noise in that direction. Then, including this direction harms rather than helps.
In practice (Appendix, E.2 we empirically observe that の² v 2X(a;) 2 for most a; This leads to threshold of 0
2A. In our experiments, we use = 10, and k in Eq corresponds to the largest index such that the singular value dz 2 f. Analysis for different values of provided in Fig 5
3.5 BI-LEVEL WEIGHT ADJUSTMENT
Although the pseudo labels generated by the pre-trained vision-language model are refined using the proposed method, the learning process still faces two challenges: (1) The number of samples per class predicted by pseudo label varies, which hinders the learning of the minor classes. (2) The pseudo labels are hard ()-1 labels obtained via the argmax operation, which discards useful confidence information embedded in the soft predictions. To address these issues, we propose a weight adjustment strategy, which consists of two components: inter-class adjustment and intra-class adjustment, allowing dynamic weight assignment to individual samples.
Inter-class Adjustment. For task f, given samples with features X, and their corresponding pseudo labels Y let N,, denotes the number of samples belong to class c,. To balance the samples from different classes, we define an adjustment factor Minter E I as:
n4
Minter,
N
C.
This ensures that the total weight assigned to each class is normalized to [द. effectively balancing different classes during training.
5

---

Published as a conference paper at ICLR 2026
Table I: Comparison of different methods. Best scores in bold. The second-best scores are in
underline.
Aircraft
Cars
CIFARI00
Method
B0Inc10
B50Inc10
B0Inc10
B50Inc10
B0Ine 10
B50Ine 10
A
As
A
Ar
A
An
A
Aa
A
A
As
7S-CLIP
26.61
17.16
21.66
17.16
82.90
76.73
78.74
76.73
81.81
71.38
76.49
71.38
Label
66.38
56.31
59.18
56.31
93.57
89.15
90.92
89.14
88.52
81.92
85.23
81.92
CODA Prompt
26.56
17.79
18.11
12.28
79.69
69.03
67.81
56.59
79.36
67.74
66.94
54.64
MoE-Adapter
26.73
17.92
22.19
17.52
83.16
77.02
79.21
77.72
81.86
71.53
76.57
71.49
RAPF
29.07
19.52
20.83
18.46
83.77
73.26
75.73
71.56
84.47
76.08
77.30
74.33
RAIL
36.23
33.59
23.75
25.99
88.64
84.68
82.72
82.13
87.34
80.37
81.44
78.89
ENGINE
34.77
25.41
26.94
23.56
86.90
78.76
82.67
79.93
85.15
77.11
79.89
76.15
N2L
43.73
40.21
29.69
32.42
92.38
87.50
86.42
85.45
87.80
81.13
82.92
80.30
CUB
ObjectNet
UCF
Method
B0Ine20
B100Inc20
B0Inc20
B B100Inc20
BÖIne 10
B500ne10
A
Ab
A
Ag
AA
A
A
Aa
Ag
A
AB
ZS-CLIP
75.47
63.72
69.06
63.72
38.43
26.43
31.12
26.43
75.88
67.79
71.68
67.79
Label
86.43
79.05
82.29
79.06
53.18
45.27
49.71
45.26
98.74
97.75
98.37
97.75
CODA-Prome Prommpt
67.40
51.91
53.24
37.64
38.25
26.23
26.22
19.22
81.61
75.18
67.06
58.87
MoE-Adapter
73.69
63.07
68.29
63.09
40.14
28.70
32.45
28.53
75.35
67.36
72.28
69.07
RAPF
72.08
58.14
62.27
56.02
39.16
25.88
28.93
24.70
85.99
81.10
75.70
74.31
RAIL
81.64
73.93
73.08
70.91
39.80
35.13
31.21
30.19
90.18
89.90
81.05
84.27
ENGINE
77.06
65.07
70.85
64.92
44.57
31.24
34.62
29.96
87.85
84.46
82.30
81.04
N2I.
83.41
76.48
75.16
73.40
49.31
41.59
41.42
38.65
95.00
93.29
86.41
87.87
Intra-class Adjustment. To incorporate prediction confidence, we propose an intra-class adjustment strategy based on entropy of the logit. For task 6, given the logits X., W (as described Eq. 5). we compute the entropy for each sample, yielding E (Ei,... ... E,,). ). A lower entropy indicates higher confidence, suggesting that such samples should weighted more heavily. While a naive choice might be e. this risks numerical instability when entropy values variations in entropy across different datasets make it challenging for single function to generalize effectively. Instead, sample weights from gaussian distribution A (1,,0²) and sort them in ascending order.
close zero. Additionally,
m/. my....... ,.,,,, ~ ~((,0²²,
(8)
where m/ S ml S
m/, Then we rearrange the weight according the order of the entropy:
Mintra, m/ (rank(E)))
where, rank(Fs) denotes the position of E, among E when the values are sorted descending order.
Further analysis is provided in Table in experiment section.
Finally, the weight for the i-th sample is computed as:
m; Mintra. Minter.
(10)

---

Published as a conference paper at ICLR 2026
Analytic CIL with Weight Adjustment. With the computed weights, the optimization objective (Eq. 2 is reformed as:
(X WT YIT)) MM( Yi.r) W
(11)
where M = diag(M,, 1M ... Mr). M, is the diagonal weight matrix for task t, whose diagonal elements are my. The closed-form solution for WT can be recursively calculated:
Wr arg min ((Wr) (Ar + XII-CCr,
(12)
WT
with
A. - A MX,,
C. = C-=C.- X, M,Y,
(13)
The derivation is provided in Appendix.
F
Finally, task t, the incremental image classifier, W, is learned recursively for different classes using the feature X,, updated label Y,, the stored matrix Ar-1, Ct-1 and weigh adjustment strategy.
3.6 INFERENCE STEP
During inference, following the strategy of RAIL (Xu et al. 2024), compute the final prediction by taking a weighted sum of the zero-shot prediction logits and the learned classification head logits.
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Datasets. Following prior works Zhou et al. 2025a b), we evaluate our method on six widely used benchmark datasets: FGVCAireralt (Maji al., 2013). StanfordCars (Krause et al. 2013).
CIFAR100 (Krzhevsky et al. 2009), CUB200 /Wah et al. (011). ObjectNet (Barbu et al. 2019), and
UCF (Soomro et al.] 2012). he classes in each dataset are split according to two class incremental
learning protocols: (1) No base classes: all classes are evenly and disjointly split into 10 tasks. (2)
With base classes: the first task includes half of the total classes as base classes, and the remaining classes are divided into 5 incremental tasks.
Evaluation Metries. Following prior works (Zhou et al. 2025a), we report two metrics to evaluate the performance: Ag and A. Aв represents the accuracy at the final task. A represents average accuracy across all tasks, computed A 1
Comparison Methods. We compare our method with existing CIL methods based on vision-language models, including non-exemplar-basee methods: MoE-Adapters (Yu et al.) (2024). RAPF (Huang et al. 2024). ENGINE (Zhou et al., 2025a).. RAIL (Xu et al.) (0244) and unimodal CIL baseline CODA-Prompt (Smith al. 2023). ENGINE proposed re-ranking method to boost performance. For fair comparison, we report the results before applying re-ranking. The zero-shot prediction of CLIP is utilized to generate the pseudo labels for comparison methods. Additionally, we report results for zero-shot CLIIP (denoted as ZS-CLIPP) and N21. trained with ground truth labels for each sample, serving as the upper bound in this setting (denoted as Label). All methods use the same LAION-400M pre-trained CLIP ViT-B/16 backbone and identical task splits. Additional experiments with OpenAI pre-trained CLIP ViT-B/16 and comparison with exemplar- methods, PROOF (Zhou et al. 2025b) and CLAP4CLIP (Jha et al. 2024). provided in Appendix. C
For hyperparameters, the threshold for feature dimensionality reduction is set to 10, and pseudo-labels are updated over 3 iterations. The standard deviation to sample intra-class weights is set to 0.5. All results are averaged over three runs. Experiments are conducted using PyTorch NVIDIA RTX 4090 GPU.
Implementation Details.
7

---

Published as a conference paper at ICLR 2026
100
$4.80
95.15
RAIL
ZS-CLIP
93.29
Accurscy (%)
80
26.75
NZL Label
Accuraty (5))
40
60
30
50
ZS-ULIP
40
35.13
Bosc
26.43
Hese- lnter
20
Bose Liter tota Redine
CUB200
ObjectNet Datasets
UCF
10
20
30
30
60
70
90
100
Number of Classes
Comparison of ZS- ZS-C RAIL, N2L and, N2L with ground truth labels.
Figure 3:
Figure 4: Ablation study of different com- ponents Aircraft-B0Inc10.
4.2 MAIN RESULTS
Table. summarizes the performance comparison across six datasets under various class-incrementie settings. It can be observed that, the CIL methods designed for unimodel VIT achieve inferior performance, and even worse than the ZS-CLIP on some datasets such as Cars, CIFAR 100 and CUB. This is because they only use the visual part of the CLIP, without utilizing the ability of aligning image and text in CLIP. The noisy pscudo label also hinders the learning of these methods and exacerbates forgetting. Our method consistently outperforms existing approaches all scenarios. Notably, on datasets with large distribution shifts from CLIP's pre- vre-training data (e.g.. Aircraft and ObjectNet), our method surpasses the second-bes best method by a significant margin of 2.75%-8. 8..6%. For the remaining datasets, we also achieve consistent improvements of at least 2.82%, 0.46%, 1.77%, and 3.39% on Cars, CIFARION 100, CUB, and UCF, respectively.
To further analyze the gap between learning without labels and learning with perfect labels, Fig. and Table. presents the results of 7S-CLIP, the second-best method RAIL,, our method N21., and an upper bound represented by N2L trained with ground truth labels. In addition to significantly outperforming both ZS-CLIP and RAIL, N2L narrows the gap to the upper bound by nearly 50% on Cars, ObjectNet and UCF, highlighting its effectiveness in incremental learning with unlabeled data.
In Table. 2 we compare our method with the unlabeled data learning approach CPL (Zhang et al.) 2024). Specifically, integrate CPL with various methods by replacing the pseudo labels with those produced by CPL. The results show that even when enhanced with CPI., existing methods still underperform compared N2L. Furthermore, N2L itself achieves better performance when combined with CPL.
These improvements stem from our feature dimensionality reduction-based pseudo label refinement strategy, which further refines noisy labels. Additionally, the proposed inter-class and intra-class weight adjustment schemes help balance class frequency and leverage prediction confidence. More- over, our regression- begreesion-based learning objective with weight adjustment is inherently more robust to label noise than the traditional cross-entropy loss and can be solved in recursive manner, thereby mitigating forgetting.
4.3 FURTHER ANALYSIS.
Effectiveness of Different Components. Fig. presents ablation study different components of our approach. The baseline method is RAIL [Xu et al. 2024). which employs the ridge regression tasks, benefiting from its
in Eq to optimize the classifier. RAIL outperforms ZS-CLIP noise-robust regression objective and the recursively calculated classifier, which can be solved in recursive manner.
Results show that incorporating inter-class adjustment improves performance, which can be attributed its effectiveness in alleviating class frequency imbalance caused by imperfect pseudo labels. Moreover, the proposed intra-class adjustment strategy further enhances performance by assigning greater weight high-confidence samples and reducing the influence of low-confidence ones, thereby minimizing the impact of erroneous pseudo labels. Finally, the proposed pseudo label refinement further leads to dramatic performance gain on all tasks.
8

---

Published as a conference paper at ICLR 2026
Mean Squared Error different threshold
Performance of difterent threshold
Last
Mean Squared Error
Accuracy (%)
40.0
37.5
0.14
35.0
0.12
32.5
30.0
14
16
18
8
10
12
15
Threshold
Threshold
Figure 5: Left: MSE between the true labels and the predictions of classifiers learned on reduced features of different tasks across different thresholds. Different colors denote different tasks. Right: the classification accuracy across different thresholds.
Table Comparison of different methods com- bined with unlabeled learning method CPL.
Table 3: Ablation of different intra-class adjust- ment methods Aircraft-BOln 10 setting.
Aircraft-BO
CUB-BO
Aircraft-
Method
Method
A
Ap
Ap
A
Ap
RAIL
36.23
33.59
81.64
73.93
w/o. Intra-class Adjustment
43.39
39.69
RAIL CPL
43.61
35.80
81.93
74.89
w/. Intra-class
43.25
38.45
ENGINE
34.77
25.41
77.06
65.07
w/. Intra-class (0.5,, 1.5)
43.71
40.08
ENGINE + CPL
37.07
25.47
78.80
68.53
w/. Intra-class (0.25 1.75)
43.55
40.12
N2L
43.71
40.21
83.41
76.48
w/. Intra-class ((1, +
43.61
40.23
N2L +
47.48
42.99
83,50
77.69
w/. Intra-class N(1,
43.73
40.21
Progressive Label Refinement. The core of this component lies in the fea- ture dimensionality reduction strategy, supported by the theoretical guarantee provided in Theorem
w/o, Inter-class Adjustment
w/. Inter-elass Adjustment
10
10
1.2 Norn
To evaluate whether removing certain singular di- rections of the feature matrix X leads to lower mean squared егror (MSE), we show the MSE between the true la- bels without noise and the predictions of classifiers learned on reduced features across different threshold values 0. Re- sults for various tasks under the Aircraft- BOIne 10 setting are shown in the left of Fig. The error curves exhibit a clear U-shape across tasks: as 0 increases, more singular directions satisfying the inequality o² 2 (2X + df) (a;)² 2 are removed, which leads to a drop in error. However, when 0 becomes too large, even singular directions that do not meet this condition are eliminated, resulting in increased error. The corresponding accuracy curves for different 8 values, shown on the right, follow similar trend. We empirically A 10.
10 20
30 40 50
100
40 50 60 70 80 90 100
Class Number
Class Number
Figure 6: Norm of the classifier for different classes.
Bi-level Weight Adjustment. To evaluate inter-class adjustment, we visualize the norm of the classifier for different classes across tasks the demonstrate that the proposed inter-class adjustment strategy significantly suppresses large norms, preventing classes with more samples from dominating the classifier and thereby improving class balance. Table. presents the performance of various intra-class adjustment strategies. The first row, which omits intra-class adjustment, yields the lowest performance, highlighting the necessity of incorporating such techniques. When intra-class adjustment applied, using the reciprocal of entropy as the weighting factor results in lower performance compared to not applying weight adjustment. This is because assigns excessively large weights low entropy (high confidence) samples, which
setting in Fig- 6 The results

---

Published as a conference paper at ICLR 2026
may dominate the learning and bias the model. Weight generation strategies that sample from predefined distributions. such as uniform and Gaussian, demonstrate improved performance. Among them, the Gaussian distribution (V1, +) achieves better overall results both A and Ap. Therefore, we adopt adopt.V(11 2) for intra-class adjustment in our approach.
CONCLUSION
In this work, we introduce more realistic paradigm for class incremental learning with vision- language model in which, at each task, only class names and unlabeled data for new categories are provided. To tackle this problem, we proposed N2L, a method that combines the regression learning objective with two key mechanisms. First, a pseudo label refinement method is proposed to iteratively refine initial pseudo labels by learning label refinement classifier on features after dimensionality reduction, with theoretical guarantees on its denoising effectiveness. Second, we adopt a learning strategy comprising inter-class adjustment to address class imbalance introduced by pseudo labels, and intra-class adjustment to downweight low-confidence samples, thereby reducing the impact of noisy supervision. This regression with weight adjustment can be solved a recursive form, achieving identical performance to joint training with unlabeled data and mitigating forgetting. Our work paves the way for more annotation-efficien incremental learning.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation China (6237601 1) and the National Key R&D Program of China 2024YFA1410000)
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 1old-100- discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461 Springer, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. ICLR, 2021.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9285-9299 ,, 2022.
Di Fang, Yinan Zhu, Runze Fang, Cen Chen, Ziqian Zeng, and Huiping Zhuang. Air: Analytic imbalance rectifier for continual learning. arXiv preprint arXiv:2408. 10349, 2024.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 1999.
Zijian Gao, Xingxing Zhang, Kele Xu, Xinjun Mao, and Huaimin Wang. Stabilizing zero-shot prediction: A novel antidote to forgetting in continual vision-language tasks. Advances in Neural Information Processing Systems, 37:1 17:1122421184888 2024.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of dut-of-distiibution generalization. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 8340,8349, 2021.
Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11858-11111 118671 2023.
Linlan Huang, Xusheng Cao, Haori Lu, and Xialei Liu. Class-incremennal learning with clip: Adaptive representation adjustment and parameter fusion. In European Conference on Computer Vision, pp. 214-231. Springer, 2024.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint NxXiv:2204003444 2022.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vasion-language models. arXiv preprint arXiv:2403. /9/37, 2024.
Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 16071-16088 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
11

---

Published as a conference paper at ICLR 2026
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 25545661,0133 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896. Atlanta, 2013.
Qiwei Li and Jiahuan Zhou. Caprompt: Cyclic prompt aggregation for pre-trained model based class incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 18421-18429, 2025.
Qiwei Li. Yuxin Peng, and Jiahuan Zhou. Fes: Feature calibration and separation for non-exemplar class incremental learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18495-285004 2024a.
Qiwei Li, Yuxin Peng, and Jiahuan Zhou. Progressive prototype evolving for dual-forgetting mitiga- tion in non-exemplar online continual learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 2477-2486, 2024b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-294) 2017.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638-23647, 2024.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 12245-12255 2020.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. Advances in Neural Information Processing Systems, 36:5765-5777 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. Advances neural information processing systems, 28, 2015.
James Seale Smith. Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bomilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal free continual learning. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pp. 1 11909-11919, 2023.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi suni-supereieed learning with consistency and confidence. Advances in neural information processing systems, 13:596-608, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2 2012.
12

---

Published as a conference paper at ICLR 2026
Zhicheng Sun, Yadong Mu, and Gang Hua. Regularizing second-order influences for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20166-20175, 2023.
Yuwen Tan, Qinhao Zhou, Xiang Xiang, Ke Wang, Yuchuan Wu, and Yongbin Li. Semantically- shifted incremental adapter-tuning is continual vitransformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 23252-23262, 2024.
Marco Toldo and Mete Ozay. Bring evanescent representations to life in lifelong class incremental learning. In Proceedings the the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16732-16741, 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. 2011.
Fu-Yun Yun Wang, Da- Zhou, Liu Liu, Han-Jia Ye, Yatao Bian, De-Chuan Zhan, and Peilin Zhao. Beef: Bi-compatible daas-inccrmmnnal learning via energy based expansion and fusion. In The eleventh international conference on learning representations, 2022a.
Liyun Wang. Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt- based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36:69054 -6006, 2023.
Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European conference on computer vision, pp. 631-648. Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022c
Yichen WU, Hongming Piao, Long-Kat Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. In The Thirteenth International Conference on Learning Representations, 2025.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Advancing cross-domain discriminability in continual learning of vision-language models. arXiv preprint Arxiv:2466. 1006./8866 18868, 2024.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-expert adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230, 2024.
Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pscudolabel learning: Enhancing vision- language models by prompt tuning with unlabeled data. arXiv preprint arXiv:2406. 10502, 2024.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremente learning via deep model consolidation. In Proceedings of the IEFF/CVF winter conference on applications of computer vision, pp. 1 1131-1144 2020.
Mengyu Zheng, Yehui Tang, Zhiwei Hao, Kai Han, Yunhe Wang, and Chang Xu. Adapt without forgetting: Distill proximity from dual teachers in vision- languaee models. In European Conference Computer Vision pp. 109- Springer, 2024.
13

---

Published as a conference paper at ICLR 2026
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of linionelanguage models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 19125-19136, 2023.
Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: a survey. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. pp. 1363-8371, 2024a.
Da- Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23554-23564, 2024b.
Da- Wei Zhou, Qi- Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024c.
Da- Wei Zhou, Kai-Wen Li, Jingyi Ning, Han-Jia Ye, Lijun Zhang, and De-Chuan Zhan. External knowledge injection for clip-based class-incremental learning. arXiv preprint arXiv:2503.085 10, 2025a.
Da- Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025b.
Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5871-5880, 2021.
Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35:11602-11614, 2022.
Huiping Zhuang, Yizhu Chen, Di Fang, Run He, Kai Tong, Hongxin Wei, Ziqian Zeng, and Cen Chen. Gacl: Exemplar-free generalized analytic continual learning. Advances in Neural Information Processing Systems, 04883224483447, 2024.
14

---

Published as a conference paper at ICLR 2026
A LIMITATION
In this paper, we propose setting where only unlabeled data and class names are provided for each incremental task. We assume that the unlabeled data strictly corresponds to the given class names. However, in real-world scenarios, this assumption may not hold, the collected data could include samples from previously seen classes as well as from unseen future classes not included in either the current or past tasks. As part of future work, we plan explore more challenging setting where the unlabeled data may contain mix of past, present, and future classes.
B CODE
We provide the core implementation of the training process in code py. The incrementa..trang function serves as the main entry point for performing incremental training, while the (rain function carries out the training for each incremental task.
C MORE RESULTS
C.I DIFFERENT EPOCHS OF PROGRESSIVE LABEL REFINEMENT
42
40
28
36
34
Epochs of Pscudo Label Refinemeni
Figure Results on Aircraft B0Inc10 with different epochs progressive label refinement.
In Fig. 7 we present an ablation study on the number of epochs used for label refinement. Setting the number of epochs to meaning the pseudo label refinement strategy is not applied, results in the lowest performance. When the refinement strategy is incorporated, there is significant improvement performance. Moreover, the results remain stable when the number of epochs is set between 2 and 5. Based on these observations, choose 3 epochs the default setting in our experiments.
C.2 COMPARISON WITH EXEMPLAR B-SSED METHODS
In Table we compare our method with existing exemplar-based based approaches, PROOF (Zhou et 20256) and CLAP4CLIP (Jha et al. 2024). Following their setting, the exemplar buffer uses fixed memory strategy that stores samples per class. contrast, our method does not use any exemplars, i.e. the exemplar buffer size is zero. Results show that even without storing exemplars, N2L outperforms existing methods on all datasets, except for the setting with base classes on CIFARIO0 CUB, and ObjectNet, where the performance gap is only 0.27%, 0.58%, and 0.43% respectively. These results highlight N2L's strong ability to mitigate forgetting, primarily due to the recursively computed closed-for form solution for the classification head, along with intra-clat class and inter-class adjustment strategies. Additionally, the proposed pseudo- label refinement method contributes to more accurate label refinement.
15

---

Published as a conference paper at ICLR 2026
Table 4: Comparison with exemplar- based methods. Best scores are in bold. The second- best scores are in underline.
Aircraft
Cars
CIFARIO0
Method
B0Inc10
B50Inc10
B0Inc10
B50Inc10
B0Ine 10
B50Inc 10
A
As
A
An
A
Aa
A
As
A
A
A
ZS-CLIP
26.61
17.16
21.66
17.16
82.90
76.73
78.74
76.73
81.81
71.38
76.49
71.38
CLAP4CLIP
41.15
35.81
26.88
28.43
90.02
86.46
79.68
75.88
86.34
79.11
82.90
80.57
PROOF
38.10
33.62
28.10
28.15
89.23
85.09
83.83
83.03
84.85
77.47
79.17
76.36
N2L
43.73
40.21
29.69
32.42
92.38
87.50
86.42
85.45
87.80
81.13
82.92
80.30
CUB
ObjectNet
UCF
Method
B0Ine20
B100Ine20
B0Inc20
B100Inc20
B0lne10
B50Ine10
A
Ab
A
AB
A
Aa
A
Ab
A
Ab
A
Ah
ZS-CLIP
75.47
63.72
69.06
63.72
38.43
26.43
31.12
26.43
75.88
67.79
71.68
67.79
CLAP4CLIP
82.45
74.85
75.74
72.71
49.00
38.55
41.85
37.77
87.10
83.82
81.38
83,50
PROOF
80.50
73.98
73.74
71.83
45.46
34.93
35.36
32.16
89.41
86.00
81.23
81.76
N2L
83.41
76.48
75.16
73.40
49.31
41.59
41.42
38.65
95.00
93.29
86.41
87.87
Table Comparison of different methods with OpenAl pre-trained weight. Best scores are in bold.
The second-best scores are in underline.
Aireraft
Cars
CIFARI00
Method
B0Inc10
10
B50Inc10
B0Inc10
B50Inc10
BOInc
B50Inc 10
A
A
A
Ag
A
Ap
A
Ap
A
An
A
A
ZS CLIC
32.96
21.96
27.40
21.96
71.99
56.36
64.51
56.36
78.66
68.22
72.97
68.22
Label
64.11
53.42
57.04
53.45
87.79
79.22
83.60
79.23
87.23
79.94
82.40
79.98
CODA-Promee
32.59
22.89
22.30
15.75
68.91
53.34
54.89
43.56
77.94
64.98
64.21
52.00
MoE- Adaater
33.37
22.87
27.75
22.55
70.64
56.39
63.43
55.88
80.17
69.69
74.44
69.86
RAPF
36.10
25.68
26.57
23.63
73.36
57.02
61.55
55.10
83.43
74.84
76.08
73.26
RAIL.
42.44
37.72
27.86
28.77
80.33
69.55
66.96
63.88
85.59
77.78
77.81
75.82
ENGINE
38.40
27.97
30.30
25.45
76.72
62.14
69.38
62.42
82.94
73.29
77.03
72.85
N2L
46.75
42.11
30.46
32.88
83.65
75.47
74.32
71.94
86.35
79.00
79.38
78.03
CUB
ObjectNet
UCF
Method
B0Inc20
B 1000Inc20
B0Inc20
B
1100Inc20
B0Ine 10
B50Ine 10
A
A
A
A
A
A
Ab
A
A
A
ZS-CLIP
68.48
54.04
59.95
54.04
45.61
33.01
38.16
33.01
79.92
70.75
76.22
70.75
Label
84.55
76.45
79.92
76.46
57.81
47.93
52.99
47.90
98.73
97.32
98.26
97.32
CODA- Prooprp
64.40
46.89
47.75
35.09
42.45
30.11
31.24
23.43
82.23
74.84
68.27
58.96
MoE-Adapter
67.93
54.44
59.37
54.52
44.83
33.46
38.24
33.79
80.09
71.66
74.96
70.62
RAPF
69.33
51.58
56.96
48.94
48.91
34.92
38.83
33.63
89.10
83.11
79.37
76.51
RAIL
78.57
70.01
66.58
65.01
48.37
41.22
39.08
36.59
91.86
90.70
82.93
84.96
ENGINE
72.80
58.29
64.34
58.32
50.00
35.58
40.49
34.53
90.02
85.73
84.76
82.85
N2L
81.18
72.58
69.99
68.01
54,99
45.21
45.26
42.09
95.58
93.57
89.16
89.91
C.3 DIFFERENT PRE-TRAINED WEIGHTS
In the main paper, we compare various methods using the CLIP model pre-trained on LAION400M[] In Table 5 we present the results using the CLIP model tre-trained by OpenA The results demonstrate that N2L consistently outperforms other methods across both settings.
http:///ghhhaccmm//fffaaaaoooooo clip bhtps://gthuhh.o///gnniCCCCCC..
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of different methods Food, ImageNet-R, and SUN datasets.
Food
ImageNet-R
SUN
Method
B0Ine10
B50Ine10
B0Inc20
B100Inc20
B0Ine30
B150Ine30
A
A
AA
An
A
As
A
An
A
As
A
A
ZS-CLIP
87.86
81.99
84.78
81.99
83.12
76.62
79.23
76.62
79.45
72.14
74.99
72.14
ENGINE
88.30
81.78
84.72
81.55
84.14
76.93
79.26
76.56
83.04
76.17
78.06
75.10
ENGINE + Label
88.52
81.85
85.08
81.59
84.61
77.12
79.96
76.78
83.51
76.21
78.83
75.20
N2L.
90.58
85.50
87.18
85.01
86.00
81.06
81.89
80.29
84.38
78.85
79.15
77.40
N2L + Label
90.95
85.84
88.38
85.84
86.61
81.52
83.88
81,51
87.42
81.50
84,31
81.50
C.4 MORE DATASETS
Following the setting of ENGINE (Zhou et al. 2025a), in Table 6 three datasets: Food BBossard al. 2014). ImageNet-R (Hendrycks et al. (202)) and SUN (Xiao et al. 2010). ENGINE + Label and N2L + Label represent providing the methods with annotated data instead of unlabeled data in ENGINE and N2L. In these datasets, results show that the gap between learning with unlabeled data and annotated data is small. This is because these datasets have a high zero-shot performance, making the generated pseudo labels more reliable, which aids the learning with unlabeled data. On these datasets, N2L with unlabeled data even achieves comparable performance to the ENGINE + Label.
we also report the results of
D INTRODUCTION ABOUT COMPARISON METHODS
ZS-CLIP. This method enables direct inference without requiring training specific downstream tasks. It leverages the pretrained image-text alignment ability CLIP to recognize images.
CODA-Prompt (Smith et al. 2023). This method introduces attention iitention-based end-to-end key- query framework. learns a set of prompt components and combines them using inpu--oonditione attention weights to generate input-aware prompts, thereby enhancing the model's 's adaptability to new tasks. This method relies solely on the visual branch of CLIP.
MoE-Adapters (Yu et al., 2024). This method adapts continuously arriving tasks by dynamically inserting Mixture-of-Experr adapters into a pre-trained CLIP model. It enables progressive model expansion without altering the original backbone, effectively mitigating forgetting in long-term continual learning.
RAPF (Huang et al. 2024). This method is built upon frozen pre-trained CLIP. During the training of new tasks, it measures the influence of new classes on old ones using textual features and adaptively adjusts the representations of old classes. the adapter fine-tuning phase, a decomposed parameter fusion strategy is introduced to further mitigate forgetting.
RAIL (Xu et al., 2024). This method employs recursive ridge regression- based adapter to learn tasks across multiple domains and decouples inter-domain correlations by projecting features into higher-dimensional space. It also uses a training- fusion module preserve zero-shot capability without the need for reference data.
ENGINE (Zhou et al.) 2025a). This method employs dual-branch tuning framework: the visual branch enriches image features through data augmentation, the textual branch leverages GPT-4 to generate more discriminative class descriptions, and during inference, a re-ranking strategy is applied to further improve prediction results.
The following are two exemplar-based methods. For a fair comparison, we adopt a storage strategy of 20 samples per class for both methods.
CLAP4CLIB (Jha et al. 2024). This method performs probabilistic modeling over the visual-guided textual features for each task, enabling
reliable continual learning fine-tuning strategy. Unlike traditional forgetiig--miiiation methods that rely on large amounts of data, it leverages the rich prior knowledge of CL.P for parameter initialization and distribution regularization.
17

---

Published as a conference paper at ICLR 2026
PROOF (Zhou et al. 2025b). This method trains task-specific projection modules for each task, expands new projections while keeping the old ones fixed, and introduces a fusion module to jointly adjust visual and textual features.
E DETAILS OF THEOREM
E. PROOF THEOREM
Theorem 1. Consider the regression model with noisy labels: y Xw" E where features X E Rn w" E TR is the true regression coefficient vector, and E is the noise with zero mean and variance 0². Let the SVD of UDV [Vi..'., Oç is the i-th coordinate of vector a V.w.
o2 (2X + d3) (a7), projecting X onto the eigenvectors without j-th components (denoted
X = UDV V') yields a smaller expected mean squared error:
[[IX'W' W -Xw" -I S R [[X Xw
(14)
where ve and W are the regression solutions using X and X', V = [V Yz-1, Vi-1, val.
Proof.
The objective function Ridge Regression is given by:
min ly - Xw| 7- + Al w|
(15)
w
whose analytical solution is:
w = X x + XI) x y.
(16)
Substituting the SVD into the ridge solution, we get:
w xx -D
=VDDVV + +XI) -VDU'
(17)
+VD2++ DU y.
According to the definition a V w" (aj as.... T we have Xw* UDa*.
Let á = (D² + AII-IDD' y. then according to Eq.
17 Xw
UDå.
The mean squared error of the prediction is:
MSE EE Xw Xw |1
EE [ a }|
E [|[(a - a² a²'
Σ EE E - a3)
(18)
1=1
Σ d (B(ag) - at)² t Var(a.)
Bias?
Var,
The & can also he represented as:
á = (D "DUTy
(D² AI) "DUT (Xw" +
(19)
(D² - (UDa" +
(D² + -DD'' + UTe),
18

---

Published as a conference paper at ICLR 2026
which can be decomposed into signal and noise components:
df
d
à df + a
of
+
di + (u. E), i= 1=1,... i=.....,d d.
(20)
Bias: The expectation of the á;:
Bias; = E[as] -
df
de
+
E(u
E) - oz
di +
di +
df
(21)
1)
05
d' + X
04.
df + X
And the squared bias is:
)
d
2
Bias
dl + x
(a;)²
(22)
Variance: The variance of di
dc
dj
Var(ā = Var
04
+
u,
ds +
de
+
di
Var
(23)
ds + X
d?
(df + 2)2
Then the mean squared error is:
MSE
Σ
df
[E(a:] - -a;)² + Var(a:))
A
2
&
Σ dc
de +
(a;)²
(d. + 2aK
(24)
1m1
(Adi)
d
(at)²
+ Σ
(d) + x12
Cdf x1o
1=1
1=1
Consider removing the j-th singular direction:
XV'
UDV
[dju,, ... 1:14; ,,d,,,)
(25)
''D'
where U' is removing j-th column of U, and D' is removing j-th column and row of D.
The objective function of Ridge Regression is: min w
Ily X'wl + A/ w|
(26)
whose analytical solution is:
W/ = (X'T x' + AI) 'X y.
(27)
(D + XI) D'U'TTy y.
19

---

Published as a conference paper at ICLR 2026
Similar to the definition of à, also define aj (D + -I) '''U"T y then have:
à'= = (D + A1) ' y
(28)
(D + )1) D'(D'' + +U* di
where
df di + X
of +
(u, E) di;
i= 1,....j- -1
(29)
di +
dg
de
di+1
+ X 0i+1 + L + x (u]+18) = ditl: i= i=j...,,-
(30)
Then the prediction error of using X is:
MSE' = E X'w' - 1|
= E U'D'' á' UDa ||:]
d [(d' a Σ
d R [(a] -a [(a]]]
Σ 1=1
of E [(a: a;} + Σ
di [(. & ((
(31)
1=/+1
MSE E [āg - a3)² + dz [[a))²]
MSE d [B[a,] a3)² + Var(ā ) + d [[a}]]]
MSE (Ad,)²
d
(d3 + A)2 (a3)²
(d) + 1)2 o² + (o3))
With the assumption that o² 2 (2) + d)) (a
(2d,)²
X²
(a))
(a;)² d2(1 (d) + xxx
(di + 2)2
(a,)) +d (df + 2X) (az)² (dz + +22
(Adz)²
(32)
(d' 1)²
(Adz)²
d
(dz + 2)2 (a,)²
(d} + 2)2 0
Then
MSE' S MSE,
(33)
finish the proof.
E.2
THE SCALE OF AND a*
In this section, we show the scale of a and a to show that the assumption of o² 2 (2X+ de) (az)² in Theorem can be achieved with appropriate dy. A is the regularization parameter of ridge regression which is typically 0.1. During the learning of task f, for specific class k and the sample x,, the ground truth label is one-hot value Yi,k. the pseudo label is also one-hot value Úi Then the standard variance a of class & can be estimated by yi - Wi.k with different samples. For ax V w*, V is the part of the SVD decomposition of X and w can be estimated by solving ridge regression with the feature X and ground truth label Y. Then, we show the standard variance a of different classes and the value of different dimensions of a in Fig. g The horizontal coordinate of the gray line represents the estimated noisy standard variance. And the absolute value of a; is presented by the bar chart. The experiment is conducted on the first task of Aircraft- BOlnc 10 setting. Results show that 18.1 of the at satisfy the condition of 2² 2 (az)² and 99.9% of the a satisfy the condition of o² 2 2X(a²)) )2 with = 0.1. Therefore, when d, takes an appropriate value, o² 2 (2X + (ag)² can be satisfied.
20

---

Published as a conference paper at ICLR 2026
160
140
120
Frequency
100
60-
40
0.1
0.4
Value
Figure 8: The estimated noisy variance (gray line) and the chart the absolute value of az on the first task of Aircraft-BOInc 10 setting.
F ANALYTIC CIL WITH WEIGHT ADJUSTMENT.
In this section, we derive how get the closed-form solution of ridge regression with re-weight in the incremental learning setting. The derivation of the standard ridge regression can be seen as setting the weight matrix the identity matrix. The optimization objective of ridge regression with weight can be reformed as:
((Wr) = (X WT - YIT) T M(X WT Y
(34)
+W WT,
where M diag(m1."" ... mn) is the diagonal weight matrix.
The gradient w.r.t. WT is:
V W 2X MX WT 2X MY 1:T 2AWr
(35)
Setting gradient to zero:
[ T. MX
IT +
AI)
X T
MY 1:T
M
Xi
Mi
0
Y
X
+
X
x
Mr
Xr
M7
Yr
MMMX+
Σ x X
(36)
where M, is the diagonal weight matrix X,. Then the closed-form solution for Wt can be updated recursively as:
-Ar+
(37)
with
A, = A + X, M,X,, C. -C-
(38)
G DETAILED PERFORMANCE ON EACH TASKS
To present the results in detail, we present the accuracy of different methods with e LAION-400M pre-trained CLIP V after different tasks in Fig. 9 (without base classes) and Fig. T0 (with
21

---

Published as a conference paper at ICLR 2026
base classes). Unlike the typical declining trend of accuracy across tasks, observe increasing accuracy trajectory on certain datasets. This phenomenon stems from varying difficulties in generating pseudo labels for unlabeled data at different stages, when higher quality pseudo labels are produced, the performance improves accordingly. Notably, our method achieves the best performance on nearly all tasks. This is attributed our proposed pseudo label refinement strategy, which effectively refines noisy labels, as well as the intra-class and inter-class adjustment methods that mitigate class frequency imbalance and leverage confidence information from pseudo labels.
50
100
100
40
90
90
Ansincy (%)
30
80
80
Z5-CLIP
ZS-CLIP
25-CLIP
MoE-Adapes
ENGINE
MoE. Adaptes
ENGINE
MoE-Adytes
ENGINE
RAE.
Ouzs
RAIL
Cuzs
RAIL
Curs
10
60
20
40
60
80
100
20
100
60
100
Number of Classes
Number of Classes
Number of Classes
(I) Aircraft
Inc10
(3) CIFAR100 Inc10
100
70
100
90
60-
Accurazy (%)
80
50-
40
50
20
ENGINE
McE-Adapess
ENGINE
MoE-Adopees
ENGINE
40
10
60
100
150
100
Number of Classes
Number of Classes
Number of Classes
(4) CUB BaseO Inc20
(5) ObjectNet BaseO Inc20
(6) UCF Based Inc10
Figure 9; Results of each task without base classes.
H
PSEUDO CODE
We provide the pseudo code in Algorithm For task first obtain the image features X, and the pseudo labels Y,, As deseribed in the first paragraph of Sec. 3.4 the projected features X,, together with Y, are used perform regression step that yields a refined classifier W'. The refined pseudo labels Y are then computed according to Eq. 4. We replace the original pseudo labels Y, with Y to enable progressive label refinement. After the refinement, the classification head for continual learning is trained using X,, the latest refined labels Y/, and the bi-level weight adjustment strategy.
MORE RESULTS
I.1 STATISTICAL RESULTS
Besides the reported results reported in Table T also report standard deviation across three independent runs Table These statistical results show that our method maintains stable performance across different runs, demonstrating its robustness.
22

---

Published as a conference paper at ICLR 2026
30
Accuracy (%)
80
80
20
70
10
ZSCUIP
RAPY ENGINE
ZS-LIP
RAPY ENGINE
23-CLIP
RAPP
MaE-Adlpie
MoE-Addeiess
MuE-Auhpiits
ENGINE
RAIL
Ouza
RAIL
Cuza
RAIL
Ours
50
60
70
80
90
100
50
60
70
80
90
100
50
70
80
90
100
Number Classes
Number Classes
Number Classes
Aiteraf Base50 Inc10
Base50 Inc
(3) CIFAR 100 Base50 Inc
80
50
90
70-
40
Accuracy (%)
60
30
MrE-Adgiie
ENGINE
MoE-Adeiee
ENGINE
MoE-Adkpies
ENGINE
Cua
40
10
60
120
180
120
140
180
50
70
100
Number of Classes
Number Classes
Number of Classes
(4)
(5) ObjectNer Base
(6)
Figure 10: Results of each task with base classes.
Table 7: The mean and standard deviation results of N2L.
Aircraft
Cars
CIFARI00
Method
B0Inc10
B50Inc10
B0Ine10
B50Ine10
B0Ine10
B50Ine10
A
A
Ab
A
Ar
A
AB
Ar
N2L
43.73
40.21
29.69
32.42
92.38
87.50
86.42
85.45
87.80
81.13
82.92
80.30
+0.24
+0.20
+0.24
+0.26
#0.14
+0.15
+0.02
+0.09
10.10
+0.03
+0.03
+0.04
CUB
ObjectNet
UCF
Method
B0Inc20
B 1000nc220
B0Ine20
B100Ine20
B0Ine10
B50Ine10
A
Ag
A
Ag
A
Ag
A
Ag
A
Ag
A
Ap
N2L.
83.41
76.48
75.16
73.40
49.31
41.59
41.42
38.65
95.00
93.29
86.41
87.87
+0.07
#0.05
+0.08
+0.10
+0.35
+0.20
+0.24
#0.26
10.10
#0.21
10.08
+0.03
1.2
MORE ABLATION RESULTS
In the main paper, we present subset of the results in Fig. to maintain readability. Additional results for each individual component are provided in Table. When incorporating only a single component, the pseudo-label refinement module achieves the most significant improvement. The inter-class and intra-class adjustment modules further enhance performance by balancing the weights of different samples. Finally, combining all components yields the best overall results.
I.3 NOISY TASK BOUNDARY
In this section, evaluate approach under noisy task boundary scenario, where 20% 50% of the training samples are drawn from C1a-1- The results in Table. 9 show that our N2L still outperforms existing methods under such noisy conditions, demonstrating its effectiveness.
23

---

Published as a conference paper at ICLR 2026
Algorithm N2L Training Procedure
Input: Pre-trained VLM V =(iing, fixt); data stream D {Di]e=1 where D. = C):
regularization A.
Output: Incremental classifier Wr.
1: Initialize Ao 0, Co O.
2: for = to T' do
3: Step 1: Pseudo Label Generation
4: for each z; E U4 do
(Eq.
5: Compute
arg maxcec, (fin (Fe), fisa(p))
6: end for
7:
Form pseudo-label matrix Y..
8:
9:
Step 2: Progressive Label Refinement
10:
Extract visual features X
Compute SVD: X, UDVT
12:
Select top-k singular vectors above threshold #: V.. Project features: X,, V.
14:
for iter = 1 to refinement steps do
15:
Learn refinement classifier W using (Xe, Y,) and weight adjustment. Update labels Y, arg max W)).
(Eq. 5
17:
end for
19:
Step 3: Bi-level Weight Adjustment
20:
Compute class-frequency weights Minter
(Eq.
21:
Compute entropy E from logits X W
(Eq.
22:
Sample intra-class weights from (N1, sort and reorder by entropy.
23:
Compute weights my Minter, Minim,
(Eq. 10)
24:
Form diagonal weight matrix M,.
25:
26:
Step 4: Recursive Analytic CIL Update
27:
Update sufficient statistics:
(Eq. 13
A, A X + Compute incremental classifier:
28:
(Bq. 12
W. = (A,+
end for
Table More ablation results N2L.
Kireraft-BOIncll
A
Ar
Base
36.23
33.59
Base+Intra
37.29
34.35
Base+Inter
37.31
35,31
Base+Refine
39.50
35.64
Basc+Inter+Intra
38.14
36.45
Base+Intra+Refine
41.17
36.17
Base+Inter+
42.34
39.66
Base+Inter+Iuiee Refine
43.73
40.21
J INFERENCE STEP
During inference, we follow the prediction fusion strategy RAIL (Xu et al. 2024), which computes the final output by taking weighted of the zero-shot CLIP logits and the logits produced by the
24

---

Published as a conference paper at ICLR 2026
Table Results of different methods under noisy task boundary scenario, where 0%, 20% or 50% of the training samples at task are drawn from previous tasks.
Aircraft
Cars
CIFAR100
Method
B0Ine10
B50Inc10
B0Inc10
B50Ine10
B0Ine
B50Ine10
A
A
A
A
A
A
Ab
A
Aa
Ä
A
RAIL(0%)
36.23
33.59
23.75
25.99
88.64
84.68
82.72
82.13
87.34
80.37
81.44
78.89
ENGINE(O%)
34.77
25.41
26.94
23,56
86.90
78.76
82.67
79.93
85.15
77.11
79.89
76.15
N2L(0%)
43.73
40.21
29.69
32.42
92.38
87.50
86.42
85.45
87.80
81.13
82.92
80.30
RAIL(20%)
35.19
31.71
23.06
24.60
87.95
82.81
82.37
81.02
87.01
79.95
81.10
78.38
ENGINE(20%)
33,95
24.72
26.51
22.56
86.45
78.32
82.16
79.18
84.72
76.61
79.52
75.86
N2L(20%)
43.11
38.70
29.13
30.51
91.06
85.26
86.00
84.23
82.78
80.07
RAIL(50%)
33.46
28.83
21.97
21.81
85.56
78.74
81.52
78.99
86.55
79.20
80.67
77.69
ENG
33.36
23.25
25.63
21.39
85.44
76.56
80.83
75.90
83.72
75.68
78.64
74.50
N2L(50%)
40.49
36.00
27.45
27.75
89.32
81.61
85.10
81.58
87.02
80.27
82.39
79.23
CUB
ObjectNet
UCF
Method
B0Inc20
B 100Inc220
BOIne20
B 1000Inc00
B0Inc10
B50Inc10
10
A
Ap
A
A
A
Ap
A
A
A
Ag
A
Ap
RAIL(0%)
81.64
73.93
73.08
70.91
39.80
35.13
31.21
30.19
90.18
89.90
81.05
84.27
ENGINE(0%)
77.06
65.07
70.85
64.92
44.57
31.24
34.62
29.96
87.85
84.46
82.30
81.04
N2L(0%)
83.41
76.48
75.16
73.40
49.31
41.59
41.42
38.65
95.00
93.29
86.41
87.87
RAIL(20%%
80.25
71.61
72.72
70.02
38.61
33.66
30.29
28.74
88.54
88.33
80.45
83.21
ENGINE(20%)
75.68
63.60
70.20
64.19
44.01
30.34
34.28
29.39
86.94
84.54
82.04
80.64
N2T,20%)
82.10
74.28
74.46
71.76
48.10
40.84
40.96
37.73
93.15
91.21
85.85
86.43
RAIL(50%)
78.38
68.45
71.20
66.33
34.48
29.53
28.09
24.98
86.31
85.75
78.44
79.58
ENGINE(50%)
74.39
61.79
69.22
62.82
42.72
29.75
33.53
28.41
84.28
82.83
80.75
78.63
N2L(25%)
79.86
70.52
72.80
68.54
46.48
37.09
39.05
34.94
90.80
88.86
84.75
84.99
learned classification head. Specifically, the final logit for sample i, denoted as final is computed as:
,inall = - 3) Y, YY
(39)
where Y; is the logit predicted by N2L, Y, is the zero-shot CLIP logit, / is weighting parameter set 0.2 following RAIL.
25


Reviews:
[{"review_id": "Y9V2dBLWq5", "reviewer": "Reviewer_o5Et", "summary": "This paper proposes N2L (Naming-to-Learn), a class-incremental learning framework for vision-language models when only unlabeled data and class-name sets are available at each task. The method combines pseudo-label generation via zero-shot CLIP, progressive label refinement, bi-level weight adjustment, and analytic recursive learning. The approach is theoretically supported and achieves strong results on multiple benchmarks.", "questions": "Please see the weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper proposes N2L (Naming-to-Learn), a class-incremental learning framework for vision-language models when only unlabeled data and class-name sets are available at each task. The method combines pseudo-label generation via zero-shot CLIP, progressive label refinement, bi-level weight adjustment, and analytic recursive learning. The approach is theoretically supported and achieves strong results on multiple benchmarks.", "weaknesses": "(1) Terminology ambiguity\nThe phrase “unlabeled data and the corresponding class names” is conceptually imprecise. At each task, the model receives a class-name set rather than per-sample labels; this wording conflates label sets with labels and may mislead readers about the supervision form.\n(2) Definition of pseudo-label noise\nThe paper mentions “noise in pseudo labels” many times without clear definition. Does it refer to misclassification errors produced by the pretrained VLM when assigning zero-shot pseudo labels.\n(3) Baseline adaptation fairness\nReplacing ground-truth labels with pseudo labels for fully supervised baselines may conflict with their design and amplify CLIP prediction errors. Reporting their supervised results would help isolate the effect of pseudo-label noise from methodological differences.\n(4) Overly restrictive pseudo-label assumption\nPseudo labels in each task t are generated only from new-class candidates CtC_tCt. This closed-set assumption simplifies the problem. To align with standard CIL inference, the authors should test or discuss a more realistic case where pseudo labels are drawn from the union of all seen classes (C₁:ₜ), reflecting data sampled from previously observed distributions.\n(5) Why not conduct experiments on MTIL or X-TAIL benchmark?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bmtkzIzPN", "reviewer": "Reviewer_k1FB", "summary": "This paper addresses a new class-incremental learning paradigm where only unlabeled images and the corresponding set of class names are provided at each training stage. To this end, the authors introduce N2L consisting of four steps: (1) pseudo label generation; (2) progressive label refinement; (3) bi-level weight adjustment; (4) learning a classifier. Specifically, they introduce a feature dimension reduction strategy at step 2, and present intra- and inter-class adjustment schemes at step 3. Experimental results on various datasets demonstrate the effectiveness of N2L for image classification.", "questions": "Please refer to the Weaknesses section. Minor questions are as follows:\n- It would be interesting to explore the scalability of the proposed method to dense prediction tasks, given that several methods have already adapted CLIP for detection and/or segmentation. How do the authors expect their method to perform in such scenarios?\n- While the proposed method consistently outperforms other baselines, Figure 9(5) shows that it performs worse than ENGINE during the first three incremental stages. Could the authors provide a plausible reason?\n- Line 138-139: I recommend explaining the notation 'n_t' here.\n- Line 188: it seems that the opening double-quotation mark is incorrect.\n- Line 217-218: it would be helpful if there is an explicit explanation of the notation 'd'.\n- Line 219: ‘above’ might be appropriate (not ‘below’).\n- Line 232: the hyperparameter '\\lambda' should be explained here rather than Line 243-244.\n- It would be helpful if there was an explanation in Figure 5 (left) like 'different tasks are marked in different colors'.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed problem setting, i.e., class-incremental learning with unlabeled data, seems practical.\n- The proposed method, N2L, achieves state-of-the-art performance on standard benchmarks.", "weaknesses": "My major concerns are Originality and Clarity. The detailed questions are listed below.\n- Although there have been several works on semi-supervised [A,B] and unsupervised continual learning [C], the manuscript is missing any discussion on them. Thus, I would recommend adding a discussion on the similarities/differences between the proposed setting and these prior works. In particular, it would be helpful if the authors could also provide a quantitative comparison with them.\n- Although the authors provide Table 3, it remains unclear why the Gaussian distribution is more beneficial than the uniform distribution. It would be better if the authors provided a more detailed explanation.\n- Figure 7 shows that increasing the number of epochs required for label refinement beyond 3 is sub-optimal. While performance saturation is expected, it is unclear why the performance decreases. Could the authors provide a plausible explanation?\n- The authors frequently use the term 'identical' (e.g., Lines 22, 28, 65, 75, 414 etc). However, considering that the proposed method still performs much worse than 'Label' in Table 1, clarification would be helpful.\n\n[A] ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning, CVPR 2021\n\n[B] Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learning, ICCV 2025\n\n[C] Representational Continuity for Unsupervised Continual Learning, ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0vqmqdwkl", "reviewer": "Reviewer_S9dK", "summary": "This paper proposes the N2L, a method for continual learning based on CLIP with not fully labeled data. This paper specifies the problem of continual learning with tasks containing image and corresponding texts and the noisy generated pseudo-label. A progessive label refinement process is proposed to address the noisy label problem and a bi-level weight adjustment to reduce the imbalance within existing analytic continual learning method. Several experiments are conducted to validate the effectiveness of N2L and its components.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper concentrate on the continual learning of vision-language models with unlabeled data, which rare in the realm of continual learning. \n2. Overall, this paper is well wirtten with clear motivation.\n3. Theoretical analysis in this paper is sound.", "weaknesses": "1. The setting of unlabeled data in this paper seems not fully obeying the unlabeled constrain. In the task t, the text label of the learning samples are provided. Since the CLIP has good zero-shot performance, annotating the samples within a limited set of text labels can be much more easier. For a fully unlabeled constrain, the samples within a task should not be accompanied with the text labels. Searching labels in the vocabulary of CLIP might be a better setting when claiming \"continual learning with unlabled data\". \n2. The presentation of algorithm pipeline and details is somehow not clear. For example, what is the pipeline of utilizing the progressive label refinement? In the Figure 2, is the refine classier the same as the classification head for the continual learning? This paper should illustrate the details clearer. Also, a psedo-code of the proposed method is needed to better understand this algorithm. \n3. Lack of statistical results of the experiments. The comparative study has only the average results, and results like standard deviations should be provided to validate the robustness of the methods.\n4. The ablation study is not sufficient. Although this paper provides the results with/without the three components of the proposed N2L, the direct effect of progress label refinement and inter-class adjustment on the performance are not provided. What is the performance degradation when progessive label refinement and inter-class adjustment is not appied? Since they are the major contribution upon the RAIL, the contribution of this paper is narrowed if not demonstrating their superiority. \n5. Lack of comparative study with different imbalance adjustment techniques. In this paper, the inter-class imbalance is adjusted via applying weighting coefficients in the training of classier, which is simple and similar to that in AIR [1]. \n \n[1] Di Fang, et al. AIR: Analytic Imbalance Rectifier for Continual Learning. arXiv:2408.10349.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMFOyhw4sf", "reviewer": "Reviewer_YoKW", "summary": "This paper addresses a realistic scenario: only unlabeled data and class names for new classes. Pseudo labels from vision-language models have noise that worsens catastrophic forgetting. Thus, the authors propose N2L: it uses MSE regression (matching joint-training results), refines pseudo labels via feature dimensionality reduction and iterative updates, and adopts bi-level weight adjustment (downweight low-confidence labels, balance class imbalance). The method mitigates forgetting, matches joint-training performance, outperforms SOTA in experiments (theoretically supported)", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper works on a realistic CIL problem.\n- Adopting MSE loss works better than the CE loss.\n- Very good to have theoretical support.\n- Strong empirical support", "weaknesses": "- The line of work in analytic CIL often offers a very large advance in training speed. Is the proposed method also share this merit?\n- In Eq 5, not very clear how to obtain new labels. Y prime obtain through LS? and replace Y prime with Y tilda to do iteration?\n- Eq 11-13 should be of stronger role in this paper, but it seems not very highlighted.\n- Need to dicuss the difference from the existing analytic CIL, including ACIL and RAIL, etc.\n- 3.6 too short. Need to be self-contained with details.\n- In implementation details, how to determine these hyperparameters?\n- Lack of legends in figure 5.\n- In main context, where are the claimed 6 datasets? They should be taken into the main text, not in the appendix, also the different split scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rvyKoNexjm", "reviewer": "Reviewer_gGDh", "summary": "This paper proposes a new scenario in which only unlabeled data and the corresponding class names are available for each new class. The authors propose N2L to address the noise in the pseudo labels generated by vision-language model.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe illustration is clear.\n2.\tThe method and the theoretical analysis seem solid.", "weaknesses": "1.\tThe proposed scenario seems to be unrealistic. At each incremental task, the unlabeled samples are from several specific classes, but if the samples are unlabeled, there is no guarantee that the samples are constrained in these classes. Please give an example in real world where such scenario happens.\n2.\tThe experiments are mainly on standard datasets, the task boundary and definition are perfectly designed, lacking sufficient performance guarantee on real world scenarios where this paper is focusing on.\n3.\tThere are no experiments to evaluate the robustness under different quality of the pseudo labels. How does the performance be affected with more noisy pseudo labels? There should be an experiment under different accuracy level of pseudo labels.\n4.\tThe comparing methods are for CIL. No baseline methods for the newly proposed scenario is compared with, making it hard to support the superiority of N2L on tackling the noise of the pseudo labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)