SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
Authors: Lorenzo Caselli
Marco Mistretta
Simone Magistri
Andrew D. Bagdanov
Venue: ICLR 2026 Poster (2026)
Keywords: Generalized Category Discovery
Spectral Filtering
Semi-Supervised Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: SpectralGCD uses CLIP similarity scores as a unified cross-modal representation, expressing images as a mixture of concepts for GCD. It exploits a teacher to select relevant concepts, and knowledge distillation to preserve semantic quality.

Abstract: Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.

Published as a conference paper at ICLR 2026
SPECTRALGCD: SPECTRAL CONCEPT SELECTION AND CROSS-MODAL REPRESENTATION LEARNING FOR GENERALIZED CATEGORY DISCOVERY
Lorenzo Caselli, Marco Mistretta, Simone Magistri, Andrew D. Bagdanov University of Florence, Media Integration and Communication Center (MICC), Italy {name. surname s @@nifi..
ABSTRACT
Generalized Category Discovery (GCD) aims to identify novel categories in un- labeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multi- modal approach to GCD that uses CLIP gross-modal image-concepl similarities unified tross-modal representation. Each image is expressed mixture over semantic concepts from large task-agnostic dictionary, which anchors learning to explicit semanties and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we intro- duce Spectral Filtering which exploits cross moosl covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge dis- tillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six bench- marks, SpectralGCD delivers accuracy comparable to or significantly superior state-of-the-as methods at a fraction of the computational cost. The code is pub- licly available at: sttps::/githhin com/mi ccuni Fi/Spect ral GCD)
INTRODUCTION
Pretrained Vision Transformers achieve remarkable performance, but fine-tuning them requires large labeled datasets. Collecting annotations is expensive, whereas unlabeled data is abundant but lacks category information. This reliance extensive labeled datasets can limit the ability of models to adapt to unfamiliar categories the real world. This poses challenges when encountering novel concepts, performance is tied to the quality and breadth of the training data. Generalized Category Discovery (GCD) addresses this by identifying novel categories in unlabeled data while exploiting small labeled subset of known elasses to cluster unlabeled data into known (Old) and unknown (New) categories (Vaze et al. 2022 Pu et al. 20233 Rastegar et al., (023). Unlike Novel Category Discovery, which assumes that unlabeled data contains only new categories (Han et 2020 Zhao
& Han 2021; Fini et al. 202T). GCD considers the general case in which unlabeled data contains both Old and New classes.
A key challenge in GCD is balancing performance on Old and New categories, as overfitting to scarce labeled data often leads models to misclassify new samples as Old (Liu & Han (025) (see (igure I] (Left)). SimGCD addresses this via a parametric classifier on top of the image en- coder (Wen al. 2023). The classifier is trained with supervised and unsupervised cross-entropy losses plus contrastive losses on the image features. This parametric framework has been widely adopted in recent GCD approaches (Wang et al., 2024 Zheng et al., 2025 Wang et al. 2025a).
Recently, TexIGCD (Zhnng et al. 2025) showed that incorporating multimodal textual information via CLIP (Radford et al., 2021 significantly improves performance over unimodal approaches. To bypass the need for unknown class names, TextGCD generates LLM-based descriptions for each entry of large lexicon, uses a frozen teacher for image-caption assignment, and then trains both image and text encoders with parametric, modality-specific classifiers.

---

Published as a conference paper at ICLR 2026
Existing
Existing Multimodal Approaches
SpectralGCD (our approach)
Unimodal Approaches
Agnostic Dictionary
house
ntaig
Spectral Filtering
Solect lask-related concepts
CLM descriation
Textual Unversson
wings lgather
crow besk
Image Encoder
Image Encoder
Text Encoder
Image Encoder
Text Encoder
Indepandent
Crose-MRodal represontations
represent images muxurcs over concapts
Image Classifier
Image Classifier
Text Classifier
Cross-Modal Classifier
X XOverfit to Old visual cues
Good generalization on New X Efficiency
/ Best performance on Old and New Efficiency
Efficiency
Figure Motivation and overview. (Left) Unimodal methods are efficient, but tend to over- fit to spurious visual cues. (Center) Introducing textual supervision improves generalization, but increases computational overhead. (Right) SpectralGCD leverages an agnostic dictionary by se- lecting task-relevant concepts, improving generalization while remaining efficient unimodal techniques. While existing multimodal approaches treat modalities independently, our method uses unified, cross-modal representation train the classifier.
More recently, GET (Wang et al. 2025a trains textual inversion network for pseudo-caption signment, training the CLIP image encoder and a parametric classifier, treating each modality inde- pendently.
Although effective, these methods significantly increase training cost compared to unimodal ap- proaches. Efficiency crucial in GCD, as realistic deployments require periodically re-running dis- covery when new unlabeled data arrive (Zhao & Mac Aodha 2003 Rypesé et al. (2225). Moreover, existing multimodal approaches treat visual and textual modalities as independent inputs to separate or shared classifier(s) (see Figure (Center)). Such approaches fail to exploit the rich cross-modal relationships that vision-language models like CLIP inherently capture (Mistretta et al., 2025a).
We propose SpectralGCI), multimodal GCD method inspired by probabilistic topic models (Blei al. 2003) in which documents are represented as mixtures over latent topics. Analogously, Spec- tralGCD represents images mixtures over semantic concepts by using CLIP's cross-modal image- text similarity scores as a unified representation. For each image, we compute cosine similarities with large dictionary of concepts, yielding representation that captures how strongly each con- cept relates to the image (e.g.. a sparrow scoring high on "bird" and "wings," but low on "car" or "building"; see Figure (Right)). Training classifier directly these cross-modal representations anchors learning to explicit semantics and reduces overfitting to spurious visual cues (Peng et al.) 2025). To retain only task-relevant concepts and eliminate the need for manual annotation noisy LLM descriptions, we introduce Spectral Filtering which exploits an cigendecomposition of a cross- modal covariance matrix, derived from the softmax strong teacher cross- cross-modal representations. To ensure semantic quality of the representations learned by student we employ knowledge distil- lation from the same frozen teacher. SpectralGCD offers an efficient approach leveraging explicit semantic concepts to guide representation learning, achieving state-of-the-a results on six bench- marks while requiring less computation than existing multimodal methods and comparable to that required by unimodal ones.
RELATED WORK
Generalized Category Discovery (GCD). GCD clusters unlabeled datasets containing both known
and unknown categories using knowledge from a labeled set of known classes. Vaze et al. (2022)
introduces the task using supervised and unsupervised contrastive learning with semi-supervised k-
means on pretrained DINO representations. SimGCD (Wen et al. (023) adds a parametric classifier and optimizes it with cross-entropy and self-distillation losses. PromptCAL (Zhang al. 2023) uses learnalle visual prompts, while SelEx (Rastegar et al.) 2025) employs hierarchical semi semi-supervised k-means for fine-grained datasets. Recent approaches enhance visual feature representations in GCD)
2

---

Published as a conference paper at ICLR 2026
by increasing intra-class visual diversity, either by regularizing visual features, in MTMC Tang et al. 2025a). or by decomposing images into visual primitives, as in ConGCD Tang et al. 2025b
Multimodal Generalized Category Discovery (Multimodal GCD), Most existing GCD methods focus on visual information with DINO backbones. CLIP-GCD (Ouldnoughi et al. (023) first uses CLIP (Radford et al. 2021 and textual information by concatenating visual features with text de- scriptions from a text corpora to improve clustering performance. More recently, GET (Wang et al.) 2025a) and TextGCD (Zeeng et al., 2025) better leverage CLIP and textual features. GET trains an inversion network to convert the image features to textual tokens that can be used to extract text features, and then trains a classifier, treating each modality independently. TextGCD builds a visual lexicon with HLM-generater descriptions, uses frozen teacher for caption assignment, and trains modality-specifee classifiers with soft-voting for final predictions.
In contrast, we train a parametric classifier directly on CLIP cross-modal representations (image- text cosine similarities over a concept dictionary), representing images mixtures over semantic concepts. Unlike prior methods, we neither treat visual and textual features independent inputs nor require additional components such as inversion networks or noisy LLM-ggnnrated text corpora.
3 GCD: PRELIMINARIES AND PARAMETRIC LOSSES
Generalized Category Discovery. GCD operates on a training dataset D comprising both a labeled and an unlabeled subset. The labeled subset D. = { m contains N₁ image-label pairs where each image o/} E X' is associated with known class label yf E V4. The unlabeled subset
Du {x²" J N. contains N₁ images with hidden labels V'. where }/2 C. Vu. Following standard terminology (Vaze et al. 2022, Wen al. 2023), we denote known classes Yi as Old and novel classes Vu 1i as New. The goal of GCD is to cluster all unlabeled images D., into K = 13) U Val categories using supervision from D, and knowledge of Old class names only.
Contrastive and Parametric Classification Losses. Many unimodal and multimodal GCD meth- ods (Wang al. 2024| Zheng et al. 2025; Vaze et al. 2023 Lin et al. 2024, Wang al. 2025a Liu & Han (0255), ), ours included, adopt the training framework introduced in SimGCD (Wen et al.
2023). combining contrastive learning with parametric classification losses. Given a feature extrac- tor fe and a projector M, typically a 3-layers MLP, this approach involves multiple objectives.
The supervised contrastive loss Cinn is applied to labeled samples from the same class to encourage similar representations, while the unsupervised contrastive loss C applies to all samples, treating augmented views as positives:
(w,) =
B
Σ FEB
PP()
Σ log PEP(()
2 (w:)
=
B
log Σ jz
Σ jjti
EEB
where Ee denotes labeled samples in the minibatch, P(i) represents positive samples sharing the same label as sample i, 7 is the temperature parameter, w/ ((fo(z)) is the projected representa- tion for the sample 2; obtained by the image encoder with parameters 6, and w/ is the representation of an augmented view of sample v4.
A parametric classifier La predicts class probabilities p; L (f) The supervised classifi- cation loss L3 uses standard cross-entropy on labeled data, while a self-distillation objective Ly encourages consistency between augmented views while maximizing prediction diversity:
(Pi, vi)
((ss,Pe), SEB
(a)) =
(p..p)) cHH())
BJ
B
EEB
where is the cross-entropy loss, ys is the ground-truth, and pl is the predicted probability of an augmented view x! with a lower temperature. H()) is mean-entropy maximization term used to regularize the unsupervised self-distillation, to use the full classifier and preventing from considering only Old categories. It is defined as: H(p) = Σn F² log p² where ş य Eie((; + y/)) and p; and pf are the probabilities of x; and 2/e both with the same temperature. e is a hyperparameter that weights the impact of the mean-entropy regularizer.
3

---

Published as a conference paper at ICLR 2026
All training Images
Teacher
Image Encoder
Swiect lask-elattd concepts
Filtered Dictionary
Recreserrattrr
Cross-Modal Covariance
Agnostic Dictionary
engine time
Rower rock book
Teacher
of I
engino
Encoder
Wheal
(a) Phase Spectral Filtering
(c) Why Cross-Modal Covariance?
Student
Cross-Modt Modal Represerrations
Image Encoder
encie
Ciu Ci + Ledd
C
Or
Filtered
Dictionary
RCC
Student
L%
engine 1ro
Text
Dovar aince captures semantic studieee ampliasiónng repersmntaionn
(b) Phase II: SpectralGCD training
mcaningfuu comclations
The SpectralGCD two-phase approach. (a) Spectral Filtering uses cross-modal covariance computed from teacher cross-modal representations to retain only its most informative components and isolate semantically relevant concepts. (b) During training, we jointly optimize the image encoder fo. linear projection W, classifier Tag, and MLP M using both parametric and contrastive objectives, while refining the semanties of our unified representation via distillation of teacher cross-modal representations computed on the filtered dictionary. (c) The cross-modal covari- ance makes explicit which concept co-activations carry meaningful signal and should be preserved.
Figure 2:
4 CROSS-MODAL REPRESENTATION LEARNING VIA SPECTRALGCD
In this section we present SpectralGCD, a multimodal GCD approach that trains a parametric clas- sifier directly on CLIP's image-text similarities rather than separate visual and textual streams. We motivate this with the notion of a sufficient representation that preserves all class-relevant informa- tion in image and mitigates overfitting to Old classes. CLIP's similarities can be regarded as approximate sufficient representation in concept space, which we call the cross-modal representa- tion (Section 4.1p
Building on this, our approach proceeds in two phases: (i) Spectral Filtering automatically selects task-relevant concepts from a large agnostic dictionary to improve the representation (Section 4.2) and (ii) SpectralGCD training, where forward and reverse distillation from a frozen teacher pre- serve the semantic meaning of the cross-modal representation, while efficiently refining it during Figure 2 gives an overview of the full pipeline.
contrastive and parametric training (Section 4.3).
4.1 TRAINING WITH CROSS- MODAL SUFFICIENT REPRESENTATIONS IN GCD
Let be of class y E 3) U U.... Suppose we have an ideal infinite text corpus of concepts C and an ideal mapping
2(:CC -
that assigns to each image x mixture over semantic concepts 2(x;C) [aclesc. where &e measures the relevance of concept to x. Inspired by probabilistic topic models (Blei et al. 2003), which represent documents as probability distributions over topics, we represent images as mixture of semantic concepts. Just document about "machine learning" would have high topic weights for "model" and "likelihood", an image of "sparrow" would have high 2, values for concepts like "bird" and "wings", and low values for unrelated ones like "car".
class identity y depends only on such concepts, i.e. p(y[z) = (x; C)), then <(2; C) is a sufficient representation (Achille & Soatto, 2018), meaning it captures all the information in that relevant for classification. In this case, the Bayes-optimal classifier (Tong & Koller 2000) can be defined entirely on 2(x;; C), without needing the raw image.
4

---

Published as a conference paper at ICLR 2026
Sufficient representations are crucial in GCD whose training objective combines supervised and un- supervised components (Eqs. and (2) prone to overfitting. In unimodal GCD, scarce labeled data often drives the supervised branch to exploit irrelevant visual clues, such as backgrounds, boosting 2025 Liu & Han 2025). while the unsupervised one suffers from noisy pseudo-labels and spurious correl porrelation (Cao et al. 2024 /Zhang et al. 2025), de-
performance on Old classes (Peng et al.
grading performance on New classes. Although raw image features may form approximate sufficient representations with a strong encoder, in practice they are more prone to overfitting. By representing images through semantic concepts, we aim to approximate the sufficient representation in [Eq. (3) retaining relesatelevant information more robustly in practice and helping to mitigate both issues.
Parametric Classifier Training on CLIP representations. In practice, we empirically approxi- mate =a;; C) using a large finite concept dictionary C = / and CLIP model with image encoder fo and text encoder Iø- For each image z; and concept cj. we compute their cosine simi- larity:
fo(z)) go(c)) (lo()))
9 EC ERM
(4)
where 7 is the CLIP's logit temperature. This yields our cross-modal representation zo, 20.0(1;) (2z; C), feature vector in which each entry reflects how well concept cj describes image F;, similar to how Concept Bottleneck Models (Koh et al. (2020) project imputs onto interpretable concept activations. is cross-modal as it combines visual and text modalities, and a representation since we train a parametric classifier it, unlike standard CLIP which uses cosine similarities only for zero-shot prediction.
This representation is then projected through linear layer W, which combines concept similarities into a compact embedding u, = W 2e, (2,; C). A parametric classifier Ly then maps u, to class probabilities Pt (u:). Training uses the classification and contrastive losses from Section
Less AP v) + (1 (p...(
C. = (C) + (1 (w;),
(5)
where the classification branch acts on pt, while the contrastive branch operates on wy M(u).
All losses operate on the cross-modal representation 20, (I;; it projected to u, mapped to class proba- bilities Pt. and to W/ for contrastive learning. We train W and sh together with the contrastive MLP M, while fine-tuning only the last transformer block of So (i.e. CLIP ViT-B/16) and keeping the text encoder 9s fixed. Training on the approx- imate representation 20,0 anchors features to semantic concepts, reducing overfitting and spurious cues from Old classes compared to unimodal training (see Figure 3). This in turn, enables the classifier generalize better New classes (for additional results see Appendix B.2)
CUB
tima
SimGCD ICross Modal)
SpectraIGCD (Image)
SpectralGCD (Cross-Moda
Accuracy
Figure 3: Comparison of SimGCD (CLIP backbone) and Spectral- GCD when using image features or cross-modal representations to train the classifier. Image features slightly better Old, while cross-modal ones improve on New.
However, the quality of the representation 2g. (3,, C) depends on three factors affecting the performance. First, the dictionary Č is finite and may miss relevant concepts needed describe the images; building an agnostic dictionary from large text corpus improves coverage but also introduces many irrelevant concepts, adding noise to the representations. Second, even with a good dictionary, weak encoders fa and do fail to assign meaningful weights concepts, limiting representation quality. Finally, the image encoder is jointly trained with the parametric classifier, the learned representation 28,6 can drift away from semantically interpretable concepts.
To address these limitations we leverage a stronger frozen teacher CLIP, with encoders fo. and Iø.. to enhance (0, We introduce two complementary strategies: Spectral filtering, which uses the teacher to retain relevant concepts and align the dictionary more closely with concepts de- scribing the images; and Forward and Reverse Knowledge distillation, which transfers structural information from the teacher preserve and refine the semanties of the cross-modal representation.
5

---

Published as a conference paper at ICLR 2026
4.2 CONCEPT SELECTION VIA SPECTRAL FILTERING
Our Spectral Filtering phase automatically selects relevant concepts from large agnostic dictionar- ies C = { { Figure 2 (a)). This approach filters out noisy concepts in the dictionary based on the visual information contained in the entire dataset.
Extracting Cross-Modal Covariance. Given the teacher model with image and encoders fe. and Po. respectively, we compute the cross-modal representation 20, o0..o. (x: C), on the images x and the large. task-agnostic dictionary C, as defined in Eq. (4)
Then, after normalization with softmax qa = a cross-modal covariance matrix:
E RM we compute the sample
2 9: G=N-ICG
N ( 14)
G= N
-n) RMM
where N = Nı + Na is the number of samples in the dataset consisting of N, labeled samples and N₂ unlabeled samples and μ is the empirical mean of q: across the dataset.
The eigendecomposition of G yields eigenvalues A {
M and eigenvectors V {wa M 1
ordered as Å₁ 2 2 Aa-- Since G encodes concept similarities, the eigenvalues quantify the strength of co-variation among concept activations in the cross-modal representation: large ones capture informative correlations, while small ones reflect noise. The corresponding eigenvectors distribute this variability across concepts, highlighting the most relevant ones (see Figurr 2[c) for an example). We then select a subset of concepts from the task-agnostic dictionary two stages:
Noise Filtering. To retain only the most informative combinations of concepts, we compute the cumulative explained variance ratio from the eigenvalues:
Tk
k
- M
11=1
Given a threshold Be (0, 1), determine k- M as the minimum index l such that rkk > S.. where k* corresponds to the number of retained principal components. The resulting reduced eigenspace determined by A. (A1;...,AA=) and V. [u,,...; v.-)
Concept Importance Selection. From this eigenspace we compute what we call the concept im- portance vector
A o EM
(8)
where 8² denotes the element- wise square of eigenvector Us and each component 8j quantifies the importance of concept cj C. We the components of the vector in decreasing order, 81 2 ... 8M, obtaining vector E R associated with the concepts, and the dictionary accordingly. Finally, we retain only the concepts with cumulative importance above a threshold 3e (0, to obtain the filtered dictionary:
c= cc Eciss" === min {
Be
Σ 8
representing the relevant concepts from the large, aankssgi dictionary
Discriminativeness of Selected Concepts. Spectral filtering inherently favors concepts that are discriminative for the classes in the dataset. This is because the covariance matrix is computed after a softmax normalization on teacher mooss representation (see Fiq. (6). which ampli- fies high similarity (foreground) concepts while suppressing common or weakly informative back- ground ones. Together, with CLIP's intrinsic object bias (Schrodi et al., Gurung et al. 2025 Du- fumier et a this causes the dominant eigenvectors of the covariance matrix G to concentrate on task-relevant object semantics (see Appendix for an empirical analysis). The effect of softmax is qualitatively analogous to term- wernhwiigting in Latent Semantic Analysis (LSA) (Deerwester al., 1990). a classical technique in topic modeling: both mechanisms increase the influence of salient components by redistributing the variance toward them while downplaying non-informative ones.

---

Published as a conference paper at ICLR 2026
4.3 SPECTRALGCD TRAINING WITH FORWARD AND REVERSE DISTILLATION
Spectral Filtering selects relevant concepts with the frozen leacher, which are then used to build the cross-modal representation 20,0 (x,, C), arising from the trainable image encoder fa and frozen text encoder 9o (the student model). The quality of this representation depends on how effectively the student represents the teacher-selected concepts C. As training proceeds, however, the cross-modal student representation loses semantic meaning due to the joint training of the image encoder with the classifier (see EEq. (5) To counter this, we apply both forward and reverse knowledge distillation, leveraging teacher guidance for cross- moosll representation learning (Figure 2(b)).
We define å = ( C) and غ = =0..o. (I:,, C), as the cross-modal representations of the student and the teacher, respectively, on image x; using the filtered dictionary Ć. We then apply forward and reverse distillation, yielding the overall loss:
1
Cka Cid, where Lia
(oga Las B
Ed(2) log (()),, (10) B
and B is minibatch containing both labeled and unlabeled data. The forward term Li encourages the student match the teacher distribution, while the reverse term La sharpens student predictions by penalizing probability mass on concepts the teacher deems unlikely (Wang et al. 2025b Mistretta cet al. 2025b). The two combined terms yield a tighter alignment between student and teacher. Aligning student cross-modal representation with that of the teacher leads to improved performance (see Table 2}. Moreover, these losses are computationally efficient: since the teacher is frozen, its representations can be precomputed once, avoiding redundant forward passes during training.
In conclusion, the combination of the distillation losses with the parametric training losses intro- duced in Section resssusss in the SpectralGCD training objective:
Lets + Le + Lid-
(11)
By leveraging CLIP's s cross- modal representations, our method yields a tractable representation that can be effectively and efficiently employed within parametric training losses for GCD)
5 EXPERIMENTAL RESULTS
In this section we report a range of experiments comparing SpectralGCD with the state-of of-the-art.
5.1 EXPERIMENTAL SETUP
Datasets. Consistent with carlier approaches (Vaze et al.
2022: Wen et al., 2023 Wang et al.
2025a Zheng et al.) 2025), we evaluate coarse-grained grained
classification datasets CIFAR 10/100 2009), as well as fine-grained datasets from the Semantic Shift Benchmark (Vaze al. 2021). CUB (aah et al. 2011). Stanford Cars Krause et al. 2013), and FGVC-Aircraft (Maji et al. 2013)) (see Appendix A for additional details).
KKrizhvskyy et al.) 2009). and ImageNet-100 (Deng et al.
Implementation Details. Following previous Multimodal GCD settings, train only the last vi- sion transformer block of a CLIP ViT-B/16. For spectral filtering and knowledge distillation, we use the CLIP ViT-H/14 model as the teachef] which is consistent with TexIGCD ensure fair compar- ison. When not mentioned otherwise, we use the Tags concept dictionary from TextGCD as default textual information source C. Note that TextGCD uses both the Tags and an LLM-based Attributes dictionary, rather than relying solely on Tags. See Table 3 for an analysis on the robustness of our approach when selecting different dictionaries. We use clustering accuracy to evaluate each method on Old categories, New categories, and their union (Alf), only on the unlabeled data following the protocol Vaze et al. (2022) and Wen et al. (2023). All reported results are averaged over three seeds (see Appendix A for additional details).
on 832H-b79K

---

Published as a conference paper at ICLR 2026
Table 1: Comparison on fine- and gaaase-raiiedd datasets with state-of-the-art Unimodal and Multi- modal approaches, Results are measured in accuracy (%) for All, Old, and New classes. Best results are in bold and second best are underlined. We also report as references the zero-shot (ZS) accura- cies of the student and teacher CLIP models supposing access to true class names.
Method
Textual Source
CUB
Stanford Cars
Aireraft
CIFAR-10
CIFAR-100
ImageNet-100
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
ZS CLIP B/16
Class Names
56.
56.1
36.3
61.3
67.6
58.2
28.2
20.8
31.9
89.3
88.0
90.0
68.3
67.3
70.3
84.0
84,9
83.5
SimGCD
60.3
65.6
51.7
53.8
71.9
45.0
54.2
59.1
51.8
97.1
95.1
98.1
80.1
81.2
77.8
83.0
93.1
77.9
Unimodal
PrompiClL SPTNet
62,9
64.4
62.1
50.2
70.1
40.6
52.2
52.2
52.3
97.9
96.6
98.5
81.2
84.2
75.3
83.1
92.7
78.3
65.8
59.0
79.2
49.3
97.3
75.6
81.4
SelEx
73.6
75.3
72.8
58.5
75.6
50.3
57.1
64.7
53.3
95.9
98.1
94.8
82.3
85.3
76.3
83.1
93.6
77.8
DabGCD
66.3
71.8
63.5
63.3
81.6
57.4
61.7
63.9
60.6
97.2
94.8
98.4
83.0
84.6
79.9
85.9
94.3
81.6
ClipGCD
CC-12M
62.8
77.1
55.7
70.6
88.2
62.2
50.0
56.6
46.5
96.6
97.2
96.4
85.2
85.0
85.6
84.0
95.5
78.2
Moltmodsl
GET
InversionNet
77.0
78.1
78.5
86.8
74.5
58.5
97.2
94.6
75.5
91.7
89.7
TexIGD
Tags+ Auributes
76.6
80.6
74.7
86.9
87.4
86.7
50.8
44.9
53.8
98.2
96.0
98.6
55.7
86.3
84.6
88.0
92.4
85.2
TextGDD*
Tags
69.9
74.9
67.3
86.2
91.2
83.8
48.1
46.3
49.0
98.3
97.8
98.5
83.7
84.7
81.6
86.6
92.6
83.6
Spectmi(CC))
Tags
92.6
61.4
98.5
96.7
87.2
83.9
Class Names
80.6
93.9
89.8
45.2
96.7
96.1
96.9
83.5
82.7
85.0
86.4
5.2 COMPARATIVE PERFORMANCE EVALUATION
Comparison with the State-of-the-art. Table gives comparative evaluation. We compare with a range of unimodal approaches (SimGCD Wen et al. 2023). PromptCAL (Zhang et al. 2023). SPTNet (Wang et al. 2024), SelEx (Rastegar et al. 2025)) and DebGCD (Liu & Han [2025], all pre- trained with DINO, using the same image encoder as CLIP for faimesss. For multimodal approaches, compare ClipGCD (Ouldnoughi al. 2023) (which uses CC- C2M (Changpinyo al. 2021 as textual source). GET (Wang et al. 2025a)) TexIGDD (Zhngg et al., (025). as well TextGDD using only the Tags dictionary as textual input, reported in the original work and denoted as TextGDD* For TextGDD* we additionally reproduced results on datasets not covered in the original paper.
As seen Table SpectralGGD consistently advances the state-of offtthe-ar art on both fine- and coarse- grained benchmarks. It improves All accuracy over TextGCD on CUB and Stanford Cars (+2.6 on CUB and +.22 on Stanford Cars in absolute performance gain), sets a new best on FGVC-Aireraft (+1.3 compared to DebGCD), and surpasses GET on ImageNet- 100 (+1.7), while achieving com- petitive results on CIFAR10/100.
By comparing multimodal and unimodal methods, we show that our approach - leveraging CLIP cross-modal representations- not only improves overall performance, but, like GET and TextGCD, also overcomes the overfit- ting of unimodal methods to Old classes. This is evident from the larger gap between Old and New classes in uni- modal methods compared to multimodal ones (see Ap- pendix B for additional evidence).
All Acturacy
60
80
40
20
104
750
2000
4000
6000
8000
Training Time
In Table we also include the zero-shot performance (ZS)
Figure 4: Accuracy vs. training time (s) for all methods on CUB. GET, TextGDD, and our method, require a preparation phase, whereas the uni- modal approach SimGCD does not.
of the teacher (ViT H/14) and the student (ViT B/16)
CLIP, by considering access to all class names. Con- trary to expectations, SpectralGCD outperforms its zero- shot teacher on multiple benchmarks (e.g. +6.6 points on ImageNet-100), demonstrating that our method yields small student model that generalizes better than the teacher.
Training Efficiency. In Figure 4 we report the total training times for SimGCD, GET and TexiGDD on the CUB dataset, each trained for the same fixed number of epochs as in their original implemen- tations. All methods are trained on single RTX 4090 GPU using the original source code. State- of-the-ar art multimodal approaches, included SpectralGCD, require a preparation phase: GET trains an inversion network (3121 seconds). TextGDD performs image-to teteettt--ttt-eeeseeeee descriptiins assignment (102 seconds), and SpectralGCD employs Spectral Filtering to select task relevant concepts from an large agnostic dictionary (194 seconds).
8

---

Published as a conference paper at ICLR 2026
Stanford Cars
Stanford Cars
CIFAR100
CIFARIOD
90-
90
selected concepts
4,000
All Accuracy
All Accuracy
68
400
68
3,000
86
Selected
500
2,000
84
1.000
0.90
0.95
0.99
0.90
0.99
0.90
0.95
0.90
0.95
0.99
9:
A.
P.
A.
8.-0.90
0-50995
A.-0.99
No filtering
SpectraIGCD (Our)
Figure 5: Ablation on thresholds 3e and 8.. For each threshold we report All accuracy and the resulting number selected concepts. The black dashed line denotes performance without Spectral Filtering; the hatched bar marks the chosen configuration used for our main results.
Only the unimodal SimGCD does not require preparation phase. Our method achieves the highest accuracy while training faster than both GET and TextGCD, with training time comparable the unimodal SimGCD. This highlights the efficiency of our approach, desirable characteristic in real- istic settings where discovery must renun as new unlabeled data arrives. Additional timing results on other datasets are reported in Appendixx F
5.3 ABLATIONS AND ANALYSES
Forward/Reverss distillation ablations. Table 2 reports the mean Spearman correlation between student and teacher cross-modal representations trained for different distillation variants (FD+RD: forward+reverse, FD: forward only, RD: reverse only, No KD: baseline). The Spearman correlation evaluates whether the student preserves the teacher's relative similarity structure, providing rank-based perspective that is more robust than direct distance comparisons. Combining forward and reverse distillation yields the strongest alignment, reflected in higher correlations and All accuracy. All variants outperform the baseline, showing that KD effectively shapes the representation space, and improves downstream performance (see Appendix Cj.
Table 2: Spearman correlation and All accuracy between student and teacher representations on Stan-
ford Cars varying distillation type.
Distillation
Spearman p
All
FD RD
0.665 = 0.09
89.1
FD
0.639 = 0.11
86.0
0.611 = 0.11
87.5
No KD
0,487 0.15
77.4
Performance with Different Thresholding Values. In Figure we show the performance on Stanford Cars and CIFAR-100 when filtering the dictionary with different values of Be (threshold on eigenvalues) and Be (for the concept importance selection). Our default values are 3. = 0.95 and B. 0.99. We also consider no Spectral Filtering, using all concepts from the large dictionary. Spectral Filtering consistently improves results, with the effect being most pronounced on Stan- ford Cars. This highlights that carefully selecting concepts is especially beneficial for fine-grained datasets in which the number of selected concepts is of the same order of magnitude as the number of categories (200-450 450 concepts for 196 classes). On the coarse-graines CIFAR- -000, the impact of filtering is more modest, with more concepts (1000-4000 concepts for 100 classes) being selected.
Dictionary Choice Analysis. Table compares
concept dictionaries: the Tags used by TextGDD, a list of ~22 concepts arising from several benchmark datasets and used for reporting performance in Table I] and the large-scale OpenImages-v7 (Krasin et al.) (017)) label set, which covers 21K visual categories. We report TexIGCD results using only these dictionaries, without the auxiliary LLM--generteed Attributes dictionary, so that both methods operate under the same textual constraints.
These results show that Tags provide more relevant cues for category recog- nition on New, yielding consistently higher overall and novel-class accuracy, while OpenImages-v7 occasionally per- forms better on known classes. Our method improves over TextGCD both settings, demonstrating more robustness
Table 3: Comparison of different dictionaries.
Stanford Cars
CIFARIO0
Method
Dictionary
All
Old
New
All
Old
New
TextCDD*
OpenImages V7
78.1
83.3
75.6
82.6
84.6
78.7
TextGCD*
Tags
86.2
84.3
Ours
OpenlmagesV7
85.8
93.8
82.0
84.9
87.6
79.4
Ours
Tags
89.1
92.6
87.4
86.1
87.2
83.9
to dictionary choice and consistently improving performance (see Appendix D for additional results and Table for additional comparative results using the WordNet dictionary).

---

Published as a conference paper at ICLR 2026
Choice of Teacher. In Table 4
Table 4: Comparison of different teachers.
show the effects of using different teacher models for both Spectral Filter- ing and distillation: the ViT-B/16 trained by OpenAI, the ViT-H/14 trained LAION-2B (our and TextGDD's teacher for results in Table D. and the ViT-
CUB
CIFARI00
Teacher Model
All
Old
New
All
Old
New
VIT-B/16
72.7 74.2
71.9
78.5
82.5
70.4
11/4 (LAION-28 28)
79.2
80.4
78.5
86.1
87.2
83,9
H/14-QuickGelu (DEN-5B)
80.6
81.8
80.0
88.8
90.9
84.6
H/14- UV14-QuicGEELU pretrained on DFN-
5B dataset (Fang al., 2024), which has comparable number of parameters but bigger pretraining dataset. Through the use of Spectral Filtering and forward-reverse distillation losses, a stronger teacher can transfer higher-quality representations to the student, leading to higher performance (additional details are provided in Appendix Ep.
Student Backbone Capacity, To quantify the im-
Table 5: Comparison of different students.
pact of the student architectures, in Table 5 we port results on CUB with the same student backbone the teacher (i.e. ViT-H/14, LAION-2B). For a fair comparison, we report the original SimGCD and Tex1GDD results with the same architecture from TextGDD paper, marked by i. We reproduced meth- ods marked by * We observe that increasing student capacity leads to a small but consistent improvement in overall accuracy, indicating that our method can benefit from stronger student backbone. The gains, however, are modest and the performance on New classes is essentially unchanged. Importantly, when using the same high-capacity architecture the teacher (ViT-H/14), our approach still surpasses SimGCD and TexIGDD which also uses H/14 and achieves comparable performance to the teacher model with full knowledge of all class names. This suggests that the improvements primarily driven by the contributions of SpectralGCD rather than by model scaling.
CUR
Method (Backhone)
All
Old
New
SimGCD (VIT-B/16)*
71.1
75.6
68.8
TextGDD (VIT-B/16)
76.6
80.6
74.7
Ours (ViT-B/16)
79.2
80.4
78.5
SimGCD (VIT-H/14))
69.1
76.3
65.4
TexIGDD (VIT-H/14))
78.6
81.5
77.1
Ours (ViT-H/14)
80.0
82.9
78.6
Spectral Filtering on Different Splits Table 6|re- ports results from additional experiments in which Spectral Filtering is computed only from Old sam- ples (either all Old or only labeled Old). We note that performance remains close to the full version, indicating that SpectralGCD is robust to moderate distribution changes, since it works with a dictio- nary filtered using only partial knowledge of the full distribution of the dataset. Even so, if the distribution changes significantly and recomputing the full filtering step becomes the best alternative, the overall pipeline remains significantly cheaper than other multimodal GCD methods (see FFguure 4 and Appendix F).
Table 6: Spectral Filtering performance on CUB: Full dataset vs. Old (labeled and unla- beled) subsets.
CUB
Spectral Filtering Data Split
All
Old
New
Full Dataset
79.2
80.4
78.5
Only Old (Labeled Unlabeled)
77.8
80.8
76.4
Only Old (Labeled)
78.0
80.4
76.8
CONCLUSIONS
Generalized Category Discovery requires carefully balancing performance on Old labeled classes and generalization to New unlabeled classes, without sacrificing computational efficiency. We pro- pose SpectralGGDD an efficient multimodal method that achieves state-of-the art performance across six benchmarks, improving results on both Old and New classes while significantly reducing com- putational overhead. By leveraging CLIP image-concept similarities as a unified cross-modal rep- resentation, SpectralGCD grounds parametric classifier learning in explicit semantics and reduces reliance on spurious visual correlations that cause overfitting on Old classes. Through Spectral Fil- tering, retain the most relevant concepts from a large agnostic dictionary to build the cross-modal representation, while forward and reverse distillation further refine it during training.
Limitations and Future Work. SpectralGCD depends on the choice of teacher and dictionary Ta- ble 3 and 4). It requires a teacher with sufficient domain knowledge and a dictionary with sufficient concept breadth for the downstream task. This limits robustness when domain coverage is incom- plete. As future work, we aim to develop image-specific cross-modal representations to reduce reliance on the teacher and dictionary, further improving generalization.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken steps to ensure the reproducibility of our work. The full source code, along with scripts to reproduce all results in the paper, is publicly available. All experiments were performed on publicly available datasets, and details model architectures, and main training hyper-parameters are given in the main paper with additional details included in the supplementary material (see Ap- pendix A). To ensure the reproducibility of stochastic processes, such as weight initialization and dataset shuffling, fix random seeds across all experiments reporting standard deviations in the appendix (see Table 7).
ACKNOWLEDGMENTS
This work was partially supported by the A14Debunk project (HORIZON-CL4-2223 HUUMAN---- CNECT grant 1.1 13177577..
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. J. Mach. Learn. Res., 19(1):1947-199) January 2018. ISSN 1532-4435.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3 March 2003. ISSN 1532-4435.
Xinzi Cao, Xiawu Zheng, Guanhong Wang, Weijiang Yu, Yunhang Shen, Ke Li, Yutong Lu, and Yonghong Tian. Solving the catastrophic forgetting problem generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16880-16889, June 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-407, 1990.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1
Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean-Philipre Thiran. What to align in multimodal contrastive learning? In The Thirteenth International Conference on Learning Representations.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. The Twelfth International Conference on Learning Repre- sentations, 2024.
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Con- fference Computer Vision (ICCV), pp. 9284-9292, October 2021.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-55 595, 2024.
Gurung, David T Hoffmann, and Thomas Brox. Common data properties limit object-attribute binding clip. In German Conference on Pattern Recognition (GCPR), 2025.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt. Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. arXiv preprint 20Xx::2002.057/4 2020.
11

---

Published as a conference paper at ICLR 2026
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 13-18 Jul 2020.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont- Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fineggrained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, June 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Haonan Lin, Wenbin An, Jiahao Wang, Yan Chen, Feng Tian, Mengmeng Wang, Guang Dai, Qiany- ing Wang, and Jingdong Wang. Flipped classroom: Aligning teacher attention with student in generalized category discovery. In A. Globerson, L. Mackey, Belgrave, A. Fan, U. Paquet, J. Tomcza and C. Zhang Advances in Neural Information Processing Systems, volume 37, pp. 60897-60935. Curran Associates, Inc., 2024.
Yuanpei Liu and Kai Han. DebGCD: Debiased learning with distribution guidance for generalized category discovery. In The Thirteenth International Conference on Learning Representations, 2025.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.51: 2013.
Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D. Bagdanov. Cross the gap: Exposing the intra-modal misalignment in CLIP via modality inversion. In The Thirteenth International Conference on Learning Representations, 2025a.
Marco Mistretta, Alberto Baldrati, Marco Bertini, and Andrew D. Bagdanov. Improving zero-shot generalization of learned prompts via unsupervised knowledge distillation. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.). Computer Vision ECCV 2024, pp. 459-477, Cham, 2025b. Springer Nature Switzerland. ISBN 978-3- 031-72907-2.
Rabah Ouldnoughi, Chia- Kuo, and Zsolt Kira. Clip-ged: Simple language guided generalized category discovery. arXiv preprint arXiv:2305.10422 2023.
Zhengyuan Peng, Jinpeng Ma, Zhimin Sun, Ran Yi, Haichuan Song. Xin Tan, and Lizhuang Ma. Mos: Modeling object-scene associations in generalized category discovery. In Proceedings of the IEEF/CVF Conference on Computer Vision and Paterrn Recognition (CVPR), pp. 151 15118-15128, June 2025.
Nan Pu. Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7579-7588, June 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021.
12

---

Published as a conference paper at ICLR 2026
Sarah Rastegar, Hazel Doughty, and Cees Snoek. Learn to categorize categorize to learn? self- coding for generalized category discovery. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 72794-72818. Curran Associates, Inc., 2023.
Sarah Rastegar, Mohammadreza Salehi, Yuki M. Asano. Hazel Doughty, and Cees G. M. Snoek. Selex: Self-expertise in fine-grained generalized category discovery. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.). Computer Vision ECCV 2024, pp. 440-458, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72897-
Grzegorz Rypešc, Daniel Marczak, Scbastian Cygert, Tomasz Trzciński, and Bartlomiej Twar- dowski. Category adaptation meets projected distillation in generalized continual category discov- ery. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.). Computer Vision ECCV 2024, pp. 320-337, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-73247-8
Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- language models. The Thirteenth International Conference on Learning Representations.
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang. Jianmin Wang, and Mingsheng Long- Andreas Krause, Emma Brunskill,
CL.IPodd: Generalizing CLIP to out-of-distributions. Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 31716-3173 PMLR, 23-29 Jul 2023.
Luyao Tang, Kunze Huang, Chaoqi Chen, and Cheng Chen. Generalized category discovery via token manifold capacity learning. arXiv preprint 2rXiv:2505.14044 2025a.
Luyao Tang, Kunze Huang, Chaoqi Chen, Yuxuan Yuan, Chenxin Li, Xiaotong Tu, Xinghao Ding, and Yue Huang. Dissecting generalized category discovery: Multiplex consensus under self- deconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 297-307, October 2025b.
Simon Tong and Daphne Koller. Restricted bayes optimal classifiers. In Proceedings of the Sev- enteenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, pp. 658-664 AAAI Press, 2000. ISBN 0262511126.
S Vaze, K Han, A Vedaldi, and A Zisserman. Open-set recognition: a good closed-set classifier is all you need? In International Conference Learning Representations. OpenReview, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7492-7501, June 2022.
Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 19962-19989. Curran As- sociates, Inc., 2023.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Enguang Wang, Zhimao Peng, Zhengyuan Xie, Fei Yang, Xialei Liu, and Ming- Ming-Ming Cheng. Get: Unlocking the mulli-modaa potential of clip for generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20296- 20306, June 2025a.
Guanghui Wang, Zhiyong Yang. Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. ABKD: Pursuing a proper allocation of the probability mass in knowledge distillation via a- 3-divergence. In Forty-second International Conference Machine Learning, 2025b.
13

---

Published as a conference paper at ICLR 2026
Hongjun Wang. Sagar Vaze, and Kai Han. Sptnet: An efficient altemative framework for gener- alized category discovery with spatial prompt tuning. In International Conference on Learning Representations (ICLR), 2024.
Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category dis- covery: A baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16590-16600, October 2023.
Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Shah- baz Khan. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern June 2023.
Recognition (CVPR), pp. 3479-3488,
Wei Zhang, Baopeng Zhang, Zhu Teng, Wenxin Luo, Jumnan Zou, and Jianping Fan. Less attention is more: Prompt transformer for generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 30322-30331, June 2025.
Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp. 22982-22994. Curran Associates, Inc., 2021.
Bingchen Zhao and Oisin Mac Aodha. Incremental generalized category discovery. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19137-19147, October 2023.
Haiyang Zheng, Nan Pu, Wenjing Li, Nicu Sebe, and Zhun Zhong. Textual knowledge matters: Cross-modality co-teaching for generalized visual class discovery. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 41-58, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-77243-00
14

---

Published as a conference paper at ICLR 2026
APPENDIX A IMPLEMENTATION DETAILS
We use the CLIP with image encoder ViT-B/16 trained by OpenAl (Radford et al. 2021 to ensure fairmess with the rest of the multimodal methods in GCD. We train only the last transformer block of the visual encoder. The text encoder is left frozen and used only once to extract the student textual embeddings. The linear layer W which maps the cross-modal representation has 768 as output dimension, to match the input size of both the classifier and MLP. The balance coefficient is set to 0.35 as in Wen et al. (2023). while the thresholdings 3e and 3. are set to 0.95 and 0.99 respectively (see Figure 5) We set the same CLIP's logit temperatures for both the teacher and the student (7/ T2 0.0T), while the r temperature used for classification and contrastive objectives is fixed to 0.1, following previous works. We train the model for 200 epochs. We used two different learning rates with cosine annealing for the image encoder and the remaining Icarnable parameters. The visual backbone is fine-tuned with learning rate of 5e 3, while for the classifier, MLP and linear layer learning we set it to 0.1. Training batch size is fixed at 128.
In the evaluation protocol, the clustering accuracy is computed by matching model predictions ÿ to the ground labels y, as:
Dul Σ 1=1
ACC-
max PEP(O.)
1l p())),
(12)
[Dul
where D, is the unlabeled set and P())) is the set of all label permutations for the unlabeled classes Vu. Accuracy is then reported for three subsets: All: all instances in Pu: Old: instances in Du whose labels belong to the labeled set 3/; and New: instances in D., whose labels belong to the novel classes Vu. \ V/. All experiments are conducted on a single NVIDIA RTX 4090. Reported performance is averaged on three different seeds (see Table 7 for standard deviations). In Table we include additional dataset details, regarding the samples count and splits used in our evaluation.
APPENDIX B BALANCE BETWEEN OLD AND NEW PERFORMANCE IN GCD
In this section, we discuss how unimodal approaches suffer from the Old/New tradeoff more severely than multimodal approaches. We then discuss how replacing image features with modal fea- tures helps mitigating the gap between Old and New classes. Finally, we demonstrate that our cross- modal representation produces more compact clusters compared relying only on visual features.
B.1 TRADEOFFS UNIMODAL AND MULTIMODAL APPROACHES
Figure 6 compares the relative accuracy (i.e. the ratio between the New and Old accuracy) of different unimodal and multimodal methods across Stanford Cars (left) and ImageNet-100 (right). Unimodal methods suffer from more pronounced New/Old tradeoff, with substantially lower rel- ative accuracy. In contrast, multimodal methods, particularly TexIGDD and SpectralGCD, achieve higher relative accuracy, narrowing this gap.
B.2 CROSS-MODAL REPRESENTATION VERSUS IMAGE FEATURES
To better understand the contribution of our cross-modal representation, we compare our method trained using either the cross-modal representation image features in input to a parametric clas- sifier, denoted as Cross-Modal and Image features in Table 9 respectivelly The ablation results on
Table 7: Mean and standard deviation of Spectral- GCD across the six evaluated benchmarks.
Table benchmark datasets.
Statistics of all the evaluated
Dataset
All
Old
New
Labelled
Unlabelled
Dutaset
CUB
79.2
8..1=0.4
78.5 =
Images
Classes
Images
Classes
Stanford Cars
89.1 0.9
92.6
87.4 1.6
CUB
1.5K
100
1.5K
2001
Airr-air
63.0 1.6
66.1 1.5
2.2
Stanford Cars
2.0K
6.1K
196
CIFARIO
98.5 - 0.0
9.17
99.4
FGVC-Aircraft
1.7K
50
5.0K
100
CIFARIO
12.5K
37.5K
10
CIFARI00 100
86.1 0.8
83.9 0.3
CIFAR 100
20.0K
80
30.0K
100
ImageNet-100
93.4 0.7
9.0
92.0=
ImageNer-100
31.9K
95.3K
100
15

---

Published as a conference paper at ICLR 2026
Reartve Recurscy (0ew
1.0
Unimoda Nullimedal
Stanford Cars
ImageNNt-1DD 100
Siooe (lmvge)
SimOCD
CUB
Stanford Cars
0.8
0.7
0.6
70
Senticc
Debraoo
Ge7
ToraticD
Spectraloc
Sioooo
Suet
Detricc
GET
Spectrasico
Performance comparison between SimGCD trained with CLIP backbone and SpectralGCD when training the classifier ei- ther image features (Image) or cross-modal rep- resentations (Cross-Modal).
Figure 7:
Figure Relative accuracy (New/Old) cate- gories for Stanford Cars and ImageNet-1000 Multimodal methods consistently reduce the New/Old gap compared to unimodal baselines.
Image
Cross-Modal
Silousste Score: 0.38
Silhouette Score: 0.53
Figure 8: t-SNE comparison across all the 20 CIFARION 100 New classes, between SpectralGCD image features extracted when training the classifier only on image features (left), and cross cross-modal features when training SpectralGCD classifier cross-modal features, following the proposed approach (right). Silhouette scores show the efficacy of our cross-modal based approach.
CUB, Stanford Cars, and CIFARI00 show that training a classifier using cross-modal representa- tions consistently achieves the best overall performance, offering stronger balance by generalizing well to New categories while maintaining competitive accuracy on Old ones. In contrast, the image- only approach tend to overfit to Old categories. This trend aligns with the observations depicted in Figure 7 which shows that a similar pattern is observed when training SimGCD with image or cross-modal features.
To further illustrate the differences between image-only features and cross-modal representations, in Figure 8 we present a t-SNE comparison of the learned embeddings across all the 20 CIFAR 100 New classes available in the evaluation set. The visualization contrasts SpectralGCD image features when the classifier trained only on visual inputs (left), with cross-modal features obtained when the SpectralGCD classifier is trained under our proposed cross-modal supervision (right).
The cross-modal representation produces more compact clusters compared the image-only fea- tures, reflecting the stronger discriminative power of the cross-modal trained model to learn more structured embedding space. To quantitatively validate this observation, we report silhouette scores, which measure intra-class compactness and inter-class separability. Consistently, the cross-modal representations achieve higher silhouette scores, confirming their stronger discriminative power compared to image-only features.
APPENDIX C EFFECTS OF DIFFERENT DISTILLATIONS
We analyze the impact of forward distillation (FD), reverse distillation (RD), and their combination on refining the student's 's cross-man cross-modal representation (aalle 10). RD proves especially effective at retaining previously acquired knowledge, yielding the highest accuracy Old classes. In contrast,
16

---

Published as a conference paper at ICLR 2026
Table 9: Ablation comparing cross-modal representation to image representation parametric clas- sifier inputs. Cross-modal representation achieve better generalization to New categories, while retaining good performance on Old.
CUB
Stanford Cars
CIFAR100
Configuration
Old
New
All
Old
New
All
New
Cross-modal
79.2
80.4
78.5
89.1
92.6
87.4
86.1
87.2
83.9
Image Features
76.3
81.6
73.7
84.9
93.4
80.8
85.0
88.7
77.8
Table 10: Ablation on knowledge distillation strategies. We compare the usage of only forward dis- tillation (FD), only reverse (RD), their combination (FD+RD), and no distillation (No KD). Employ- ing both forward and reverse distillation yields the best performance, or performance comparable to the best, across the datasets evaluated.
Stanford Cars
CIFARI00
Aircraft
Distillation
All
Old
New
All
Old
New
All
Old
New
All
Old
New
FD+RD
79.2
80.4
78.5
89.1
92.6
87.4
86.1
87.2
83.9
63.0
66.1
61.4
RD
79,3
81.0
78.4
87.5
93.3
84.6
85.5
87.4
81.7
61.4
63.8
60.2
FD
77.0
79.3
75.9
86.0
92.3
82.9
84.5
86.7
80.1
59.7
60.1
59.5
No KD
71.6
75.7
69.5
77.4
86.2
73.1
80.0
83.8
72.5
52.0
56.9
49.6
FD better facilitates adaptation to New categories. Combining FD and RD strikes the most balanced trade-off, achieving strong generalization New categories while maintaining competitive results on Old oees.eadiigg to the overall best performance across datasets. We also evaluate training without any distillation, i.e., without leveraging the teacher's guidance. This leads marked drop in performance, confirming the importance of distillation for enriching the target model.
We also note that in some cases (like for CUB), using only the reverse term actually helps in achiev- ing better performance compared to using both directions. This is because while forward distillation forces the student to match high probability concepts of the teacher, the reverse actually does the opposite: it tells where not to focus and reduce the probability of the highly unlikely concepts.
C.1 STUDENT-TEACHE ALIGNMENT
In Table 11 we report the Spearman correlation and the mean P-value on CUB, Stanford Cars, and CIFAR100. We use the Spearman correlation to assess the aligmment between teacher and student modsl representations. Unlike raw distance measures, it evaluates whether the student preserves the relative ordering of similarities produced by the teacher, a rank-based perspective that more robust and better reflects the semantic structure of the representation space.
We compute the Spearman correlation for models trained under different distillation schemes: using both forward and reverse (FD+RD), using only forward distillation (FD), only reverse distillation (RD) and employing not distillation at all (No KD). Using both forward and reverse distillation consistently yields the highest Spearman correlation between student and teacher features, indicating stronger student-teached alignment. Importantly, this stronger alignment directly translates into better performance (as shown in Table 10), confirming that maintaining high alignment with the teacher is beneficial for effective knowledge transfer.
APPENDIX D ANALYSIS OF DICTIONARY CHOICE
Table 12 compares two dictionary sources: Tags, curated from benchmark datasets and used in the main results, and the large-scale OpenImages-v7 [Krasin et al., 2017) which spans thousands of cat- egories. Both contain a similar number entries. Here, we report TexIGDD results using only these dictionaries, excluding the auxiliary LLM-generated Attributes dictionary, ensure that both meth- ods operate under the same textual constraints. Results on CUB are consistent with those observed on the other datasets: Tags provide more relevant cues for novel category recognition, yielding
17

---

Published as a conference paper at ICLR 2026
Table 11: Spearman correlation between student and teacher cross-modal representations for varying distillation strategiy on CUB, Stanford Cars, and CIFAR100. We compare the usage of only forward distillation (FD), only reverse (RD), their combination (FD+RD), and no distillation (No KD). We report the mean P and the p-value.
CUB
Stanford Cars
CIFARI00
Distillation Loss
Spearman p
p-value
Spearman p
p-value
Spearman p
p-value
FD + RD FD
0.7113 + 0.09
7.0c 10
0.6654 - 0.09
I.le
0.5809 + 0.07
2.2e-25
0.6819 = 0.09
2.4e-6
0.6391 = 0.11
1.3e-5
0.5648 = 0.07
2.9e-25
RD
0.6762 + 0.09
L.Ge
0.6111 = 0.11
4.2e 5
0.5542 0.08
1.le-12 12
No
0.5314 0.12
3.7e
0.4873 = 0.15
8.9e
0.3040 0.15
1.6e
Table 12: Performance comparison using different dictionaries (OpenImagesVZ, Tags) for both method and TextGCD, Results on fine-grained benchmarks show that the Tags dictionary benefits both methods, underscoring the importance of dictionary selection for generalization.
CUB
Stanford Cars
CIFARI00
Method
Dictionary
All
Old
New
All
Old
New
All
Old
New
TexiGDD
OpenImagesV7
64.2
64.2
64.2
78.1
83,3
75.6
82.6
84.6
78.7
Tex1GD
Tags
73.8
75.9
72.7
86.2
91.2
83.8
84.3
84.8
83.3
OpenlmagesV7
77.3
85.8
93.8
87.6
79.4
Ours
Tags
79.2
80.4
78.5
89.1
92.6
87.4
86.1
87.2
83,9
higher overall and novel-class accuracy, whereas Openlmages-v7 occasionally performs better on known classes. Overall, our method demonstrates greater robustness dictionary variations.
To further evaluate the robustness of our method even when using non vision-centric dictionar- ies, in Table 13 report results from additional experiments for SpectralGCI) and TextGDD ing WordNet, a general linguistic ontology that includes many concepts without visual grounding, Specifically, we extracted the first lemma of each WordNet synset create dictionary covering all lexicalized concepts present in WordNet. We further compare TexIGDD and SpectralGCD with GET, which however does not require any additional dictionary to work.
SpectralGCD remains robust even with a generic dictionary that is not vision vision-centric, outperforming TexIGDD under the same conditions and still outperforming GET. This shows that our method can extract useful semantic structure even when the dictionary is not curated from visual benchmark concepts. As expected, Tags (a domain-aligned dictionary of visual concepts) yields the best per- formance. These results show that SpectralGCD is not limited to vision-centric dictionaries and can operate effectively with generic concept sets such as WordNet.
We further examine scalability. While the method involves computing an M x M covariance matrix, practice this operation is performed only once per dictionary and can be efficiently managed using low-rank approximations. To assess scalability beyond the 20K-concept dictionaries used in main experiments, we conducted additional tests with a substantially larger dictionary of roughly 60K concepts (based WordNet, shown in Table 13). For this setting, used low-rank approximation (i.e., computing only the top-K eigenvectors needed by our method rather than the full spectrum). This reduces both compute time and memory. The memory requirement in particular goes from O(M²) to approximately ((M K), making spectral filtering feasible even at this scale on single GPU (an NVIDIA RTX 4090 in our case). Thus, low-rank approximations keep the method scalable, and we verify feasibility up to 60K concepts.
APPENDIX E
IMPACT OF TEACHER SELECTION
In Table 14 we report the impact of using different teachers for both Spectral Filtering and dis- tillation. We consider three candidates: the ViT-B/16 released by OpenAl (also used stu- dent), the ViT-H/14 trained on LAION-2B (the main teacher in our reported results), and the VTT-H//4-QQuccGEEELL11 trained the DFN-5B dataset (Fang al. 2024), where Data Filtering
httpss:/hugginggffcccco////
P-
18

---

Published as a conference paper at ICLR 2026
Table 13: Performance comparison between using the Tags dictionary versus using WordNet dictio- nary. We report results for both SpectralGCD and TextGDD.
CUB
Stanford Cars
Method
Dictionary
All
Old
New
All
Old
New
GET
N/A
77.0
78.1
76.4
78.5
86.8
74.5
TextGDD
WordNNtt
76.2
63.6
84.7
53.4
TexIGD
Tags
69.9
74.9
67.3
86.2
91.2
83.8
Ours
WordNet
77.1
79.9
75.7
83.0
93.5
78.0
Ours
Tags
79.2
80.4
78.5
89.1
92.6
87.4
Table 14: Comparison of teacher variants (ViT-B/16 architecture released by OpenAI, ViT-H/14 (LAION-2B) and ViT-H/14-QuickGell (DFN-5B)), We evaluate the influence of different teacher models when applied both for Spectral Filtering and during the distillation of cross-modal represen- tations to the student.
CUB
Stanford Cars
CIFARTO0
Teacher Model
All
Old
New
All
Old
New
All
Old
New
VIT-B/16
72.7
74.2
71.9
75.1
84.7
70.4
78.5
82.5
70.4
H/14 (LAION-288 28)
74.2
80.4
78.5
89.1
92.6
87.4
86.1
87.2
83.9
H/14-QuicKGELU (DFN-SB)
80.6
81.8
80.0
89.5
94.4
87.2
88.8
90.9
84.6
Networks were employed to curate 5B image-text pairs from pool of 43B. Notably, the two ViT- H/14 teachers share the same backbone size but differ in their pretraining data.
demonstrate that stronger teachers consistently provide richer semantic cross- modal signals. These signals not only enhance the quality of the distilled student representations but also improve the reliability and effectiveness of Spectral Filtering. This reinforces our main findings Table 4p: leveraging teachers with higher capacity and broader pretraining yields more discrim- inative and better- alignee representations. Such representations, in turn, facilitate more faithful transfer of semantic knowledge across modalities and ultimately translate into superior Spectral- GCD performance.
Results in Table 14
APPENDIX F TIMING ANALYSIS
In Figure 9 we report total training times across four datasets, including the preparation phase time for multimodal methods. All methods are trained with the same fixed number of epochs, as in their original implementations and following standard GCD protocol. The trends observed on CUB in Figure 4 are consistent for Stanford Cars, CIFAR 100, and ImageNet-100: our method trains substantially faster than the multimodal competitors GET and TexIGDD, while remaining compa- rable to the unimodal baseline SimGCD, which does not require preparation phase multimodal methods. This efficiency arises from training only the student's visual encoder, while freezing the text encoder. The latter is used once the beginning of training to extract dictionary embed- dings, after which no further forward passes are required. The efficiency particularly beneficial in realistic scenarios where category discovery must be performed repeatedly new data streams in. This makes the method especially well-suited for dynamic environments with incremental category growth, where fast adaptation essential.
APPENDIX G FINE-TUNING THE TEXT ENCODER
In Table 15 we show the effects of fine-tuning solely the visual encoder versus fine-tuning both visual and text encoders. In SpectralGCD, we intentionally keep the text encoder frozen because the text embeddings define stable semantic basis for measuring image-concept similarities. Fine- tuning both encoders causes them to drift together, breaking this anchor. While OLD categories see marginal gains CUB, this comes at the expense of a degradation in discovering NEW categories. This behavior aligns with recent findings on assion-laguuage model adaptation (Shu et al, 2023 Gao al. 2024), which demonstrate that keeping the text encoder frozen preserves the pretrained seman-
19

---

Published as a conference paper at ICLR 2026
CUB
Cars
12000
Total Training Time (S)
10000
lunoC3
Shn0cp
6000
4000
2000
OurS
GET
TexGCD
SimGCD
Ours
GET
Tex6CD
SimáCo
CIFAR-100
ImageNet-100
Total Tatal Training Time 15)
150000
125000
75000
50000
25000
Tea
Figure 9: Total training time on CUB, Stanford Cars, CIFARI00, and ImageNet- 100, including the preparation phase for multimodal methods (Ours, GET, and Tex1GDD). SimGCD is the only method that do not require preparation phase.
Table 15: Proposed SpectralGCD Vision only fine-tuning vs Vision + Text fine tuning.
CUB
Stanford Cars
Method
All
Old
New
All
Old
New
Vision only
79.2
80.4
78.5
89.1
92.6
87.4
Vision + Text
77.6
81.0
75.9
87.7
91.3
86.0
tic structure essential for generalization. In our setting, the frozen text space provides the semantic foundation for spectral filtering, enabling discovery of novel categories through their relationship to known concepts. Fine-tuning causes the text encoder to overfit to labeled OLD categories, losing the general semantic structure needed in GCD.
APPENDIX H ABLATION ON DIFFERENT OLD/NEW IMBALANCE.
To assess sensitivity to class im-
Table 16: Sensitivity to Old/New imbalance on the CUB
balance the unlabeled set, Ta-
dataset.
ble 16 show additional experiments on CUB by varying the proportion of New classes between 20% and 80%. With 20% New, SpectralGCD main- tains strong performance and con- tinues to outperform unimodal and multimodal approaches. The 80% New setting is much more challeng- ing. Still SpectralGCD improves upon SimGCD and GET by a large margin. TexIGCD performs best in this imbalanced scenario since it directly leverages the Teacher's 's text assignments, giving it strong initialization when the vast majority of unlabeled samples come from unseen categories. This advantage disappears once the distribution becomes less extreme, as represented by the performance on the 20% New settings and the standard performance in the 50/50 split from Table 1
CUB (20% New Classes)
CUB (80% New Classes)
Method
All
Old
New
All
Old
New
SimGCD
77.4
81.1
54.5
41.1
56,2
GET
80.7
82.8
76.5
61.3
49.3
62.8
TextGD
78.0
79.0
75.9
71.6
65.5
72.4
Ours
82.5
85.8
75.8
68.1
59.7
69.1
20

---

Published as a conference paper at ICLR 2026
Table 17: Top-ranked discriminative concepts and common background concepts (with relative
ranks) obtained using Spectral Filtering on the CUB dataset with WordNet dictionary.
Top Discriminative Concepts
Common Background Concepts
Rank
Concept
Rank
Concept
Tern
354
Beak
Warbler
9,950
Wing
3
Waodpecker
9,700
Branch
4
Vesper Sparrow
21,623
Leg
5
Indigo Bunting
23,420
Grass
Vermilion Flycatcher
25,964
Tree
Red-headed Woodpecker
29,177
Bush
Rusty Blackbird
36,945
Leaf
9
Evening Grosbeak
43,612
Sky
10
Whip-poor-will
44,501
Cloud
Table 18: Comparison of linear probing performance (Left) and image-to-image retrieval (Right)
between Image Features and our Cross-modee modal representations on CUB and Stanford Cars datasets.
Accuracy (%)
Cars
Feature Type
CUB
Cars
Feature Type
mAP
R@)
mAP
R@
Image Features
88,5
95.1
Image Features
56.3
80.4
61.8
88.9
Cross-mode representations
87.6
94.9
Cross-modal representations
62.1
81.7
75.4
91.4
APPENDIX
DISCRIMINATIVENESS OF THE CROSS-MODAL
REPRESENTATION
To qualitatively show the effectiveness spectral filtering, in Table 17 we report the top-10 selected concepts on the CUB dataset dataset containg 200 fine-grained bird species) obtained when filter- ing the WordNet dictionary ( (60K concepts) using the concept importance score defined in Eg. (8)
Finally, provide more articulated, Table we show empirical evidence of the discriminative power of the selected concepts with two additional experiments, comparing the image features and our filtered cross-modal representations computed using the frozen Teacher ViT-H/14 on CUB and Stanford Cars. In the first experiment (Talle 18 (Left)), we follow the standard linear probing evaluation protocol. We keep the encoder frozen and train linear classifier on the full training split, supposing access to all labels. In the second experiment (Table (Right)) we run a standard image-to-image retrieval evaluation considering images of the same class as positive examples.
Results show that the cross-modal representation (even in smaller representation space of 263 concepts for CUB and 377 for cars) are as linearly separable as the original image features (of embedding dimension 1024), with accuracy of 87.6% compared to 88.5%. More interestingly, our filtered cross- modsl features are better calibrated for the harder, unsupervised task of image-to- image retrieval, improving mAP by nearly 6% CUB and 14% Cars over the image features. This showcases the efficacy of our approach in selecting task-relevant concepts.
21


Reviews:
[{"review_id": "eiCNMoSjFy", "reviewer": "Reviewer_ei3h", "summary": "The authors propose a new GCD method inspired by probablistic models.\n\nThey model images as mistures over semantic concepts in CLIP embedding space.\n\nThey use spectral filtering to filter out irrelevant concepts from a large task agnostic dictionary.\n\nThey achieve better accuracy on standard benchmarks than existing unimodal and multimodal approaches to GCD.", "questions": "Minor:\n- It may be helpful to clarify how \"New\" accuracy is defined for those not familiar with the literature.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of spectral filtering on a dictionary of concepts is cute.\n- The paper is quite complex. There are multiple stages to the proposed method. Yet the proposed method is still efficient.", "weaknesses": "- The authors claim that the dictionary of concepts is task agnostic. But this isn't really true right? There must be some overlap between the dictionary and the concepts of the target dataset. Otherwise it would not work.\n- GCD is useful for discovering concepts in a dataset that do not fit neatly into the existing label set. However, I would argue that the new assumptions that the authors are introducing render the task of GCD meaningless.\n  - In particular, the authors use a Teacher model CLIP H/14 that has been pretrained on all the concepts (both new and old) across all the benchmarks. So the \"novel classes\" can't really be considered novel. \n  - It would be more impressive if the authors test their method on datasets that have a smaller conceptual overlap with LAION. e.g. bacteria species classification based on cell cultures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHbycbnzOI", "reviewer": "Reviewer_YwSE", "summary": "This paper tackles the problem of Generalized Category Discovery (GCD), where models often overfit to known classes. The authors propose SpectralGCD, a multimodal approach that represents images not by their visual features directly, but as a mixture over semantic concepts from a large dictionary. This \"cross-modal representation\" is derived from CLIP-based image-concept similarities. The core technical contribution is \"Spectral Filtering,\" a method that automatically prunes the concept dictionary by performing an eigendecomposition on a cross-modal covariance matrix derived from a strong teacher model, retaining only concepts deemed most informative. The learning process for a smaller student model is then guided by a combination of standard GCD losses and a forward-and-reverse knowledge distillation objective to align its representations with the teacher's.", "questions": "1. The core assumption of Spectral Filtering is that concepts contributing most to the variance of the cross-modal covariance matrix are the most \"relevant\". Could you provide a more rigorous justification for this? How does this method distinguish between concepts that are genuinely discriminative and those that are simply common or part of a shared background, which could also lead to high variance?\n\n2. The paper frames the problem as representing an image as a \"mixture over semantic concepts.\" This is an appealing analogy to topic modeling. However, the current implementation simply uses a linear projection on the similarity vector. Did you explore enforcing a probabilistic constraint (e.g., ensuring the representation is a valid probability distribution over concepts) to more closely follow this analogy?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses the Generalized Category Discovery  task, focusing on the common problem where models overfit to the labeled \"Old\" classes and perform poorly on unlabeled \"New\" classes.\n\n2. Proposed Core Idea: It introduces a novel \"cross-modal representation\" for each image. Instead of using raw image features, it represents an image as a vector of similarity scores against a large, task-agnostic dictionary of semantic concepts, computed using a pre-trained CLIP model.\n\n3. To refine this representation and reduce noise from irrelevant concepts, the paper proposes \"Spectral Filtering.\" This technique uses a strong teacher model to compute a cross-modal covariance matrix across the entire dataset. Through eigendecomposition (PCA), it identifies and retains concepts that contribute most to the principal components (i.e., high-variance directions) of the concept-similarity space.", "weaknesses": "1. The primary weakness lies in the justification for \"Spectral Filtering\". The motivation is to select \"task-relevant\" concepts. However, the mechanism (performing PCA on the global cross-modal covariance matrix) selects concepts that explain the most variance across the dataset. High variance does not necessarily equate to high discriminative power or task relevance. For example, a common background (e.g., 'sky', 'grass') present across many different classes could easily form a principal component with high variance. The method might then prioritize these non-discriminative concepts. \n\n2. The paper makes a conceptual leap by equating \"high contribution to dataset variance\" with \"semantic relevance for classification\" without providing a strong theoretical or empirical argument to support this crucial link.\n\n3. The paper feels more like a report on a successful engineering recipe than a deep scientific inquiry. This lack of insight limits the paper's contribution. An outstanding paper should not only present a method that works but also provide the understanding that allows the community to build upon its core ideas. SpectralGCD in its current form feels more like a well-tuned heuristic than a principled approach, making it less inspiring for future exploration.\n\n4. In the task of discovering general categories, there has already been similar work [1] that decomposes objects into combinations of various attributes (textual or visual). I believe there needs to be more comparative discussion with the current work.\n\n5. In addition, there has been progress in the discussion on the information represented by the covariance matrix of features in general category discovery. A comparison with those works [2,3] should be made.\n\n\n[1] Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstruction. In ICCV, 2025.\n\n[2] Generalized Category Discovery via Token Manifold Capacity Learning. In Arxiv, 2025.\n\n[3] Continual Generalized Category Discovery: Learning and Forgetting from a Bayesian Perspective. In ICML, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "viO7k3VTTJ", "reviewer": "Reviewer_GDAc", "summary": "The paper addresses Generalized Category Discovery, aiming to find novel categories using limited labeled data. It introduces SpectralGCD, which builds a unified cross-modal representation by expressing images as mixtures over CLIP-derived semantic concepts. A teacher-guided Spectral Filtering selects relevant concepts via a cross-modal covariance matrix, and bidirectional knowledge distillation keeps the student’s representations aligned and semantically sufficient. On six benchmarks, SpectralGCD matches or surpasses SOTA.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The idea of using the cross-modal representations is interesting.\n\n(2) The paper is clearly written and easy to follow.\n\n(3) The performance is promising.", "weaknesses": "(1) Using VLMs (e.g., CLIP) for GCD risks data leakage, as these models may have been exposed to images or names of the “unknown” classes. Prior work (e.g., GET) evaluates on splits unseen by CLIP to mitigate this. Please discuss this issue and, if possible, include experiments on CLIP-unseen splits or provide a robustness analysis addressing the leakage problem.\n\n(2) What is the performance when using ViT-B/16 as the teacher or using ViT-H/14 as student? To what extent do the gains stem from distillation from a larger teacher rather than the proposed components? An ablation varying teacher and student capacity (e.g., ViT-B vs ViT-H) would help isolate the contribution.\n\n(3) Please report or elaborate on the zero-shot performance of the CLIP models in Table 1, to contextualize the improvements over zero-shot.\n\n(4) The KD component seems fairly standard and lacks technical novelty. Please clarify the insight beyond common KD practices.\n\n(5) What are the total inference costs compared with multimodal methods (GET, TextGCD) and unimodal baselines (SimGCD)? Latency would clarify efficiency trade-offs, as the proposed approach appears quite complex.\n\n(6) Have you evaluated fine-tuning the text encoder? Reporting this result would be informative.\n\n(7) The abstract states: “Training a parametric classifier solely on image features often leads to overfitting to old classes.” Is this primarily due to the absence of labeled images for the novel classes during training, which biases the classifier toward seen (old) categories? Please clarify more about this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3MGbBrOQa", "reviewer": "Reviewer_hK7S", "summary": "The paper proposes SpectralGCD, a multimodal approach for Generalized Category Discovery (GCD) that leverages CLIP’s cross-modal image–concept similarities as a unified representation. Instead of treating visual and textual modalities independently, SpectralGCD represents each image as a mixture over a large task-agnostic concept dictionary, which is then filtered via a novel Spectral Filtering mechanism based on eigendecomposition of a cross-modal covariance matrix derived from a frozen teacher model. The method further employs forward and reverse knowledge distillation to preserve semantic fidelity during student training. Evaluated across six benchmarks, SpectralGCD achieves state-of-the-art or competitive performance with significantly lower computational cost than existing multimodal GCD methods, while maintaining efficiency comparable to unimodal baselines.", "questions": "Please see Weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The core idea of using CLIP’s cross-modal similarities as a sufficient representation for GCD is both conceptually elegant and practically effective, grounding classification in explicit semantics and reducing overfitting to spurious visual cues.  \n(2) Spectral Filtering provides an automated, unsupervised way to prune irrelevant concepts from a large dictionary without relying on LLM-generated descriptions or manual curation, improving both representation quality and computational efficiency.  \n(3) The combination of forward and reverse knowledge distillation ensures strong alignment between student and teacher cross-modal representations, which is empirically validated through Spearman correlation and ablation studies.  \n(4) The method achieves state-of-the-art results across diverse benchmarks (fine- and coarse-grained) while being significantly faster to train than other multimodal approaches like GET and TextGCD, making it suitable for real-world deployment scenarios requiring repeated discovery.", "weaknesses": "(1) It is hard to figure out the novelty as there is many works that constructs hierarchal fine-grained knowledge when performing tasks. Also, the paper assumes access to a “Tags” dictionary derived from benchmark datasets, but it is unclear how generalizable this dictionary is to truly out-of-domain tasks (e.g., medical or satellite imagery). While OpenImages-v7 is tested, both dictionaries are still vision-centric and curated from existing classification datasets. Could the authors clarify whether SpectralGCD would still perform well with a generic, non-vision-specific concept set (e.g., WordNet or Wikipedia titles), and what minimum coverage or semantic alignment is required between the dictionary and the target domain?\n(2) Spectral Filtering relies on computing the full cross-modal covariance matrix G ∈ ℝ^{M×M}, where M is the dictionary size (~20K). For very large dictionaries (e.g., 100K+ concepts), this becomes memory-prohibitive (O(M²) storage). The paper mentions efficiency but does not discuss scalability limits of Spectral Filtering. Did the authors explore approximations (e.g., randomized SVD, Nyström) for larger M, and what is the practical upper bound on dictionary size given current GPU memory constraints?\n(3) The distillation loss uses softmax-normalized similarities σ(zˆi) and σ(zˆi∗), but CLIP’s original logit scaling already includes a temperature τ. The paper sets τ = 0.01 for both teacher and student (Appendix A), yet the distillation loss applies another softmax. This may over-smooth or distort the relative concept rankings. Could the authors justify this design choice and provide ablation results comparing raw cosine similarities vs. softmax-normalized logits in the distillation objective?\n(4) The student only fine-tunes the last transformer block of ViT-B/16, while the teacher is ViT-H/14. This architectural mismatch raises questions about the fairness of distillation: the student has far fewer parameters and less capacity. Would the performance gap between SpectralGCD and TextGCD shrink if both used the same backbone size? Also, why not use a ViT-B/16 teacher for a more direct comparison of the representation learning strategy alone?\n(5) The evaluation protocol follows standard GCD practice, but all benchmarks assume that the unlabeled set contains a known split of Old and New classes (e.g., 50/50 in CIFAR100). How sensitive is SpectralGCD to imbalanced Old/New ratios in the unlabeled data? For instance, if New classes dominate (>80%), does the method still avoid collapsing New clusters into Old prototypes, and how does entropy regularization interact with such shifts?\n(6) The paper claims that cross-modal representations reduce overfitting to Old classes, but Figure 3 and Table 7 show that on Stanford Cars, the “Image Features” variant actually achieves higher Old accuracy (93.4 vs. 92.6) than the cross-modal version. This contradicts the stated benefit. Could the authors explain this anomaly and clarify under what conditions cross-modal representations might sacrifice Old performance for New gains—or vice versa?\n(7) The preparation phase for SpectralGCD (194s on CUB) includes precomputing teacher representations and performing Spectral Filtering. However, if new unlabeled data arrive incrementally (as mentioned in the introduction), does the entire filtering step need to be recomputed? If so, this could undermine the claimed efficiency in dynamic settings. Please clarify whether the filtered dictionary Cˆ is fixed after initial filtering or must be updated with new data.\n(8) The reverse distillation term L_rd = −σ(zˆi) log σ(zˆi∗) penalizes the student for assigning high probability to concepts the teacher deems unlikely. However, if the teacher itself is biased or misaligned with the true semantics of a novel class (e.g., mislabeling a “sparrow” as “eagle”), wouldn’t reverse distillation reinforce this error? How robust is the method to teacher mistakes, especially on fine-grained novel categories where even strong CLIP models struggle?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)