Delving into Spectral Clustering with Vision-Language Representations
Authors: Bo Peng
Yuanwei Hu
Bo Liu
Ling Chen
Jie Lu
Zhen Fang
Venue: ICLR 2026 Poster (2026)
Keywords: Spectral Clustering
Vision-Language Models
Neural Tangent Kernel
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Spectral clustering is known as a powerful technique in unsupervised data analysis. 
The vast majority of approaches to spectral clustering are driven by a single modality, leaving the rich information in multi-modal representations untapped. 
Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of spectral clustering from a single-modal to a multi-modal regime. 
Particularly, we propose Neural Tangent Kernel Spectral Clustering that leverages cross-modal alignment in pre-trained vision-language models. 
By anchoring the neural tangent kernel with positive nouns, i.e., those semantically close to the images of interest, we arrive at formulating the affinity between images as a coupling of their visual proximity and semantic overlap. 
We show that this formulation amplifies within-cluster connections while suppressing spurious ones across clusters, hence encouraging block-diagonal structures. 
In addition, we present a regularized affinity diffusion mechanism that adaptively ensembles affinity matrices induced by different prompts.
Extensive experiments on \textbf{16} benchmarks---including classical, large-scale, fine-grained and domain-shifted datasets---manifest that our method consistently outperforms the state-of-the-art by a large margin.

Published as a conference paper at ICLR 2026
DELVING INTO SPECTRAL CLUSTERING WITH VISION- LANGUAGE REPRESENTATIONS
Bo Peng. Yuanwei Hu; Bo Liu, Ling Chen, Jie Lu, Zhen Fang
Australian Artificial Intelligence Institute, University of Technology Sydney, Australia (bo. noo.peng-7, yuarwei. hu)eesudennt. uts. edu. au [bo.liu, ling. chen, jie.lu, zhen. fang) @et.))duts...u edu.au
ABSTRACT
Spectral clustering is known as a powerful technique in unsupervised data analy- sis. The vast majority of approaches to spectral clustering are driven by single modality, leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of spectral clustering from single-modal to multi-modal regime. Particularly, we propose Neural Tangent Kernel Spectral Clustering that leverages cross-modal alignment in pre- pre-trained Asion-language models. By anchoring the neural tangent kernel with positive nouns, i.e., those semantically close to the im- ages of interest, we arrive formulating the affinity between images coupling of their visual proximity and semantic overlap. We show that this formulation am- plifies within-cluster connections while suppressing spurious ones across clusters, hence encouraging block-diagona structures. In addition, we present a regular- ized affinity diffusion mechanism that adaptively ensembles affinity matrices in- duced by different prompts. Extensive experiments on 16 benchmarks-inch classical, large-scale, fine fineggrained and Nomain-shifted datasets manifest that our method consistently outperforms the state-of-the-art by large margin. Code:
INTRODUCTION
Clustering aims to partition a set of unlabeled samples into groups such that samples within the same group semantically similar. Among various clustering techniques, spectral clustering has demonstrated superior effectiveness thanks to its ability to capture non- non-linear pairwise affinities. By reformulating clustering as wraph-partitioning problem, spectral clustering represents samples as nodes and pairwise affinities as edge weights. Leveraging the spectrum of the graph Laplacian, low-dimensional embeddings are then learned to reveal cluster structures. Despite these theoretical advantages, most existing approaches remain confined to visual-only representations. As result, they often suffer from inherent limitations when semantically distinct images are visually similar, thereby yielding sub-optimal affinity graphs and degraded clustering quality.
This paper explores new landscape for spectral clustering by moving beyond the classical single- modality paradigm toward multi-modal regime. While the motivation is appealing, core chal- lenge arises: how to effectively utilize joint vision-language features for spectral clustering? In the visual domain, existing methods typically require discriminative feature representations (Shaham et al., 2018; Yang et al., 2019; Duan et al., 2019) and a distance metric (Zhang et al., 2021b; Guo et al., 2025; Li et al., 2022a), under which within-cluster images are relatively far from between- cluster images. However, such approaches do not directly translate into the multi-moda regime, where semantic alignment between modalities plays decisive role.
On the representation learning side, the emergence of large-scale vision- languaee pre- vre-training such CLIP (Radford al., 2021), provides powerful alternative to purely visual encoders. By map- ping textual and visual inputs into a unified hyperspherical embedding space, CLIP captures cross- modal correspondences to enrich the semantic structure of image representations. Building on this
"qqual Contribution Corresponndecce to Zhen Fang (zhen.fang @ @wwuucccee.. ad)

---

Published as a conference paper at ICLR 2026
capacity, recent works (Li et al., 2024; Cai et al., 2023) select positive nouns' from large-scale lexi- cal databases in the wild to serve as semantic anchors in the absence of nlass-name priors. Despite promising potential, spectral clustering based such aligned multi-modal features remains under- explored due to the challenge of designing a principled framework to integrate textual semantics with visual similarities in constructing an affinity matrix.
This paper addresses this research gap from a novel perspective of Neural Tangent Kernel (NTK) (Ja- cot al., 2018). Our method capitalizes on the compatibility between visual and textual features. By anchoring a proxy network with features of the filtered positive nouns and computing its NTK on pairs of image features, we formulate the affinity between two images as a multiplicative cou- pling of (i) their visual proximity in the CLIP feature space and (ii) their semantic overlap measured by how strongly and consistently each image aligns to the positive nouns. Our theoretical analysis reveals that this coupling enhances within-cluster affinities (high visual proximity and shared seman- ties) and suppresses cross-cluster links (visual proximity alone is insufficient), thus sharpening the block-diagonal structure acknowledged by spectral clustering. Moreover, we present Regularized Affinity Diffusion (RAD) mechanism to adaptively ensemble various affinity matrices induced by different prompts. In particular, RAD allows for robust affinity matrix to be constructed through joint optimization of ensemble weights and the equivalent objective of the diffusion process.
Extensive experiments on 16 benchmarks empirically demonstrate the effectiveness of our proposed method. For example, our method achieves 98.3% ACC and 84.9% ACC on STL-10 and ImageNet- Dogs, respectively, outperforming the latest TAC (Li et al., 2024) by 3.8% and 9.8%. In addition, on three more challenging datasets (DTD, UCF-101, and ImageNet- K), our method surpasses TAC (Li al., 2024) by an average of 7.7%, 2.5%, and 6.3% w.r.t. ACC, NMI, and ARI, respectively. We also validate our method in fine-grained and domain-shifted settings, and ours significantly outperforms TAC (Li et al., 2024) by 5.1 % on Pets and 5.3% on ImageNet-sketee w.r.t. ACC.
PRELIMINARY
Notation. We denote matrices and vectors bold-faced uppercase and lowercase characters respec- tively. In the remaining of this paper, we write A [i, as the ij-th element of the matrix A, (5 as the inner product, and vee(-) the vectorization operator. Let e(i] be the i-th element of the vector e E RK and [K] = (I....,K] K}. we then define softmax; (e) = exp (e(k))/ ΣiE(K] exp (eli)).
Zero-shot Classification. Let X' and T be the visual and textual input space respectively, CLIP- based models adopt dual-stream architecture with one text encoder fr and one image en- coder fx to map inputs of two modalities into an uni-modal hyper-spherical feature space Z = z E R | = }]. Considering an image classification task with known classes {cI,..., cx) CLIP-based models make prediction for any input x E X' by computing
exp [T (x) T fr (A(c))
arg
max
Lj=1 cxp ( [x) (x fr(());))]
(1)
where 7 > 0 a temperature, AC.;) T with (()) as the prompt template for the input class name.
Leveraging Unlabeled Textual Data in the Wild. Despite remarkable effectiveness (Radford et al., 2021) and provable guarantees (Chen et al., 2023), the zero-shot paradigm in Eq. (1) relies on the prior knowledge of true class names, therefore inapplicable to the unsupervised settings. In response, advanced methods (Li et al., 2024; Cai et al., 2023) propose to select N-sized set of positive nouns (čI...., CN } from unlabeled "in-the-wild" textual datasets, such WordNet (Miller, 1995).
Spectral Clustering. For a given image dataset Dx = {xI.... XM3. one aims to group images into K distinct clusters. Let A E R.MMMM denote an affinity matrix where the clement Ali. j] represents the similarity between x; and Xj. the classical method SC-Neut (Shi & Malik, 2000) converts the clustering task as graph eut problem given by:
Y*
arg min tr (Y YERMxX
(YTLY), LY), s.t. Y -Y==KG II.
(2)
"By definition, positive nouns are those semantically velevant/similar to any ID label. .Generally, "in-the-wild" data are those that can he collected almost for free upon deploying machine leamn ing models in the open world.
2

---

Published as a conference paper at ICLR 2026
Phase Feature Extraction
Phase 2: NTK Affinity Construction
Phase Output
Generate B Affinity Matrices
Parameter Anchoring
6g vec(W))
Causter Assigument
CLIP Text Encoder
Text Space
Parameterzation
Positive Nouns Prompts
*
Proxy Network g(%)
Compute Empincal NTK
Spectral Clustering
Compule graph smoothness on each simiarity graph Eq410)
CLIP Image Encoder +
Imaga Space
iterate until convergence
Unlabeled Images
Optimize weights with regularization In Eq. (14)
Difluse affniltes all graphs and update by Eq. (15)
Converged
Refined Affinity Matrix Á
Phase Regularized Affinity Diffusion (RAD)
Figure 1: Overview of the proposed NTK-hased spectral clustering pipeline.
where Is is K K identity matrix, IM AD 1/2 a normalized Laplacian matrix, D is diagonal matrix of D(i, E, A ji. and tr(-) denotes the trace of a matrix. The optimal spectral embedding matrix Y consists of the top-K minimum eigenvectors of L.
Neural Tangent Kernel, Let define a proxy network as function go(-) w.r.t. parameters 0 E IR" (stretched into single vector). In a small neighborhood region around initialization Og. the function go can be linearly approximated by first-order Taylor expansion:
E differentiable
[z(z) = (z; 0o) = go. (z) + ( 0o:
0go,
00.
Eq. (3) implies that the obtained approximation (z: 0n) can be regarded as linearized network that maps the weights functions residing reproducible kernel Hilbert space (RKHS) deter- mined by the empirical neural tangent kernel (Jacot et al., 2018) at Og i.e.,
(0geo(z.) Oge
Ko, (2i, 2z)
(4)
00,
a0.
Intuitively, (:..) can be interpreted as condensed representation of gradient magnitude and gradient correlations. More concretely, the gradient magnitude governs the scale of update that each input induces on the model during optimization, while the gradient correlations dictate the alignment stochasticity of update directions across inputs (Xu et al., 2021). Since Ke, (..-) directly quantifies how two inputs interact through the dynamics of function learning, this naturally yields a higher-
order notion of affinity in function space rather than their geometric closeness in the input space of
Go. In this rest of this paper, we abbreviate empirical Neural Tangent Kernel as NTK for simplicity.
3 METHODOLOGG
Motivation. Eq. (2) implies that the effectiveness of spectral clustering (SC) fundamentally depends on the quality of the affinity matrix A. To enhance the expressiveness of the affinity matrix A, prior works (Zhong et al., 2021; Lu et al., 2024; Huang et al., 2019b) compute the affinity matrix by applying kernel tricks on the latent representation space. Taking the widely used RBF kernel as example, with pre- pre-traine trained CLIP-based models, the affinity matrix can be formulated as follows:
fx.)) E NG(fx(xx)) 4) fx(x,) E (f((f(()))))
AREF[i.5]
otherwise.
where (zz,, 2,) z, zj 112 and (z, 2) represents top-g nearest neighbors of z with regard to the metric function ( z). While Table shows that CLIP(SC), which uses the precomputed AREF in Eq. (5) for SC, outperforms most traditional training- based competitors CLIP(SC) relies
Experimental results using other kernels are provided in appendix (c.f. Table

---

Published as a conference paper at ICLR 2026
on single-modal visual features and thus leaves the semantic knowledge embedded in text largely unexploited. In response to this, naive strategy is to perform the RBF-based affinity measure on the tereneetxt-- feature space from TAC (Li et al., 2024) for SC. Unfortunately, our experimental results (c.f. Table and Table 2) reveal that TAC(SC) yields performance comparable to TAC(KMeans). This observation means that merely incorporating text the feature level does not adequately guide affinity construction, prompting the research question underlying this work:
How can effectively integrate textual semantics with visual similarities in constructing the affinity matrix A.?
3.1 NEURAL TANGENT KERNEL SPECTRAL CLUSTERING
To address this challenge, we present a novel spectral clustering framework build upon the NTK in Eq. (4). To implement our method, we begin with extracting features from CLIP-based models for the observed positive nouns {ci,... êx J to have W = [WI...., WN] E Rdx where w; = fT (A(e)) for each i E [N]. Thanks to the cross-modal alignment of pre-trained CLIP-based models, we subsequently take 6u = vec( (W) to anchor the NTK in a semantically meaningful parameterization. This enables the gradient age, (z) to capture how the input z functionally interacts with positive nouns, hence injecting the learned semantic structure of texts from CLIP- based models into the NTK-based affinity matrix ANTK that is given follows:
Kap (xx(xx).fxxxx))
if (x(x E (N,((x(x))) Ko,) A fx(x)) (xx
ANTx(1,]]
otherwise.
(6)
While the formulation of Ka, (:) generic the choice of ga. (+), when applying it to spectral clustering, this paper proposes to design goo (-) in log-sum-exp form, i.e.
go. (fx(x:)) = loo wh fx(xx)/T_
k=1
In the following, we provide theoretical justification of Eq. (7) by showing that it can promote block structure in the constructed affinity matrix.
In particular, combining Eq. (7) with Eq. (4), the closed-fom formulation of Ko,, (:))) is given by
N
Ko. (x(x)),,xxxx)) = 72 [x((x) [x((x))
Σ
sy [A]s
(8)
Dis
where s.[k] = softmax, (WT fx (x;))/T) Since is usually set to a relatively small value (e.g., T 0.04 in this paper), each softmax probability s; can be highly skewed practice.
find that Ke, (fx(xx), fx (x,)) is multiplicative coupling of visual proximity Ui and semantic overlap Vij. For images within the same cluster, both terms tend to be simultane- ously large- their embeddings are close in the CLIP space and their softmax probabilities concen- trate on positive nouns-so their affinities are strongly amplified, filling the diagonal blocks with high values. For images across different clusters, even if Usj moderate due visual similarity, their softmax probabilities would concentrate on a different subset of positive nouns, yielding a suf- ficiently small Vi, and therefore suppressing cross-cluster affinities. This multiplicative mechanism enforces high intra-cluster and low inter-cluster similarity, thereby sharpening the block dock-diiagonal pattern of the NTKKiiduced affinity matrix ANTK-
From Eq. (8),
3.2 AFFINITY ENSEMBLING VIA REGULARIZED AFFINITY DIFFUSION
In this section, we extend our method to the same multi multi-prompt prompt scenario as TAC (Li et al., 2024)*. Formally, given a B-sized pool of prompt templates (A (1) ,.... A (B)), ), we can construct prompt-
specific affinity matrix A NTK for each A (8) with b [B]. In view of this, our goal is to learn
(b)
'thhough the TAC paper states that single prompt template is used, its official implementation in fact employs multiple prompt templates indeed.
4

---

Published as a conference paper at ICLR 2026
new affinity matrix À which 1) captures the geometry of the underlying manifold encoded in each affinity matrix, and 2) leverages the complementarity among multiple prompt templates.
To this end, the most straightforward strategy is to simply average the B affinity matrices A (I) A A NTK. i.e., À = = Eb= ANT. Despite simplicity, this naive practice ignores the correlations among different affinities as it assigns uniform weight to each affinity matrix. For- tunately, Bai et al. (2016) show that weight learning can be exerted on affinity matrices to assist neighborhood structure mining. This motivates to formulate the weight learning and the affinity learning as a unified optimization problem given by
(B)
min B| (A,, NTR + u||À E + 11BBI s.t. 0 [|6] I and Σ B16 B.A
where is positive semi-definie matrix to avoid À from being extremely smoothed 3ib]] de- notes the contribution of ATK the whole affinity diffusion process, and
M
Alk i] Va
{(A, A
(b) NIK
Σ
(b) Gj
Oel
(10)
2
Va
JJATE1
the objective value the affinity diffusion process (Bai et al., 2018; 2017) with a. = ANTR [. M
(b)
and d = j=
A (b)
NTR [..
Note that the optimization problem in Eq. (9) simultaneously depends on ,3 and Á, therefore making it inherently complex and impractical to find a direct solution. To resolve this, we propose numer- ical method that decomposes the optimization problem into two sub-problems: 1) Optimize Å with Fixed B and 2) Optimize 3 with Fixed A. This allows for systematic iterative approximation of the optimal result through the fixed-point scheme.
Optimize with Fixed B. In this scenario, as we show in Appendix B, the optimization problem in Eq. (9) can be transformed follows:
min
B vec(A) (IM= - ((e)) + allvec(()
vec E
(11)
where ((b) (b)
Sib) MM is the Kronecker product of (b)) and itself with ( as the As we show in Appendix C, the closed-form of the solution to Eq. (11) can
row-normalized A be given follows:
(b)
NIK-
A== 1
vec-1
IM/
Σ b=1
B
3
8(6)
vec (E)
(12)
1+1
where te-) is the inverse operator of vea(.).
Since is computationally infeasible to directly solve the inverse M2 x M matrix, we alter- natively resort to efficient iderationbbased solver, i.c..
B SO AS b= 1+1
+
E.
(13)
1 ++1
Appendix D proves the convergence of the iterative process in Eq. (13) to the solution in Eq. (12).
Optimize 33 with Fixed Å. In this case, the optimization problem Eq. (9) can be simplified
B
B
min
A + 1 s.t. 0.5 [[6] < 1 and Σ
Bb = 1.
(14)
b=1
'In practice, we find that simply setting E = L helps. II this case, all vectors in A nearly identical.
5

---

Published as a conference paper at ICLR 2026
Table 1: Clustering performance (%) on widely-used datasets. The best results are shown in bold.
Dataset
STL-10
CIFAR-10
CIFAR-20
ImageNet-10
ImageNet-Dogs
Metries
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
CLIP (zero-shot)
93.9
97.1
93.7
80.7
90.0
79.3
55.3
58.3
39.8
95.8
97.6
94.9
73.5
72.8
58.2
DAC
36.6
47.0
25.7
39.6
52.2
30.6
18.5
23.8
8.8
39.4
$2.7
30.2
21.9
27.5
11.1
DCCM
37.6
48.2
26.2
49.6
62.3
40.8
28.5
32.7
17.3
60.8
71.0
55.5
32.1
38.3
18.2
IIC
49.6
59.6
39.7
51.3
61.7
41.1
22.5
25.7
11.7
PICA
61.1
71.3
59.1
69.6
31.0
33.7
17.1
80.2
87.0
76.
35.2
35.3
20.1
CC
76.4
85.0
72.6
70.5
79.0
63.7
43.1
42.9
26.6
85.9
89.3
82.2
44.5
42.9
27.4
IDED
64.3
75.6
57.5
71.1
81.5
66.3
42.6
42.5
26.4
89.8
95.4
90.1
54.6
59.1
41.3
SCAN
69.8
80.9
64.6
79.7
88.3
77.2
48.6
50.7
33.3
61.2
59.3
45.7
MICE
63.5
75.2
$7.5
73.7
83.5
69.8
43.6
44.0
28.0
42.3
43.9
28.6
GCC
68.4
78.8
63.1
76.4
85.6
72.8
47.2
47.2
30.5
84.2
90.1
82.2
49.0
52.6
36.2
NNM
66.3
76.8
59.6
73.7
83.7
69.4
48.0
45.9
30.2
60.4
58.6
44.9
TCC
73.2
79.0
90.6
49.1
31.2
84.8
89.7
82.5
59.5
41.7
SPICE
81.7
90.8
81.2
73.4
83.8
70.5
44.8
46.8
29.4
82.8
92.1
83.6
57.2
64.6
47.9
SeCu
70.7
81.4
65.7
79.9
88.5
78.2
51.6
51.6
36.0
DivClust
71.0
81.5
67.5
44.0
43.7
28.3
85.0
90.0
81.9
51.6
52.9
37.6
RPSC
83.8
92.0
83.4
75,4
85.7
73.1
47.6
51.8
34.1
83.0
92.7
85.8
55.2
64.0
46.5
TCL
79.9
86.8
75.7
81.9
88.7
52.9
53.1
35.7
89.5
83.7
62.3
64.4
51.6
ProPos
75.8
86.7
73.7
85.1
91.6
83.5
58.2
57.8
42.3
89.6
95.6
90.6
73.7
77.5
67.5
CLIP (KMeans)
90.1
93.8
92.4
70.3
74.2
61.6
49.9
45.5
28.3
96.9
98.2
96.1
39.8
38.1
20.1
CLIP (SC)
88.1
87.8
82.4
65.4
69.2
57.1
45,2
42.7
30.1
96.4
96.8
95.6
72.8
70.6
56.9
SIC
95.3
98..
95.9
84.7
92.6
84.4
59.3
58.3
43.9
97.0
98.2
96.1
69.0
69.7
55.8
TAC (KMeuns)
92.3
94.5
89.5
80.8
90.1
60.7
55.8
42.7
97.5
98.6
97.0
75.1
75.1
63.6
TAC (SC)
92.6
94.3
94.2
81.2
90.3
80.1
56.9
54.5
30.1
97.0
98.3
96.8
75.3
75.8
64.4
Gradnorm
95.6
98.3
96.2
82.6
91.1
81.5
61.3
60.6
43.6
98.7
99.4
98.7
81.0
81.2
70.9
Ours
95.8
98.3
96.3
83,3
92.0
83.0
63.3
59.6
43.5
97.8
99.2
98.4
82.4
84,9
71.4
Since the optimization objective in Eq. (14) takes the form of a Lasso optimization problem, the solution to Eq. (14) can be efficiently obtained with the coordinate descent method (Wu Lange, 2008: Wright, 2015). To keep the main text concise, we provide detailed illustration in Appendix E.
The optimization procedure is guaranteed converge since we obtain the optimal solution each subproblem. By solving two subproblems alternatively, the objective value of Eq. (9) keeps decreas- ing monotonically, which is consistent with the visualization in Fig. 3. For clarity, the optimization procedure is summarized in appendix (c.f. Algorithm 1). Finally, we compute clustering assignment by performing spectral clustering the ensembled affinity matrix Å produced by Algorithm 1.
4 EXPERIMENTS
Evaluation Metric. We evaluate clustering performance with three widely used metrics, including Accuracy (ACC), Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI).
Baselines. We compare our method with DAC (Chang et al., 2017), DCCM (Wu et al., 2019), IIC (Ji et al., 2019). PICA (Huang et al., 2020), CC (Li et al., 2021b). IDFD (Tao et al., 2020), SCAN (Van et al., 2020), MICE (Tsai et al., 2020), GCC (Zhong et al., 2021), NNM (Dang et al., 2021), TCC (Shen et al., 2021), SPICE (Niu et al., 2022), SeCu (Qian, 2023), DivClust (Metaxas et al., 2023), RPSC (Liu et al., 2024), TCL (Li et al., 2022b), ProPos (Huang al., 2022), SIC (Cai al., 2023), TAC (Li et al., 2024), GradNorm (Peng et al., 2025c).
Implementation. Following prior works (Li et al., 2024; Peng al., 2025b), we adopt the pre- trained CLIP model with ViT-B/32 (Dosovitskiy et al., 2020) and Transformer (Vaswani et al., 2017) default image and text backbones, respectively. We use the positive nouns filtered by TAC (Li et al., 2024) from WordNet (Miller, 1995). Following Li et al. (2024), we filter positive nouns based on the train split of each dataset, followed by evaluating the clustering performance on the test split of each dataset. We fix 0.04, q 30, μ = 0.1 and X 10 for all datasets.
Prompt Templates. In consistent with TAC (Li et al., 2024), we use the following B templates to construct prompts for the filtered positive nouns: itap of
a {), bad photo of the (), origami O, a photo of the large (. a O in a video game, art of the (}. a photo of the small ()-

---

Published as a conference paper at ICLR 2026
Table 2: Clustering performance (%) on challenging datasets. The best results are shown in bold.
Datasel
DTD
UCF-101
ImageNet-IK
Average
Metrics
NMI
ACC
ART
NMI
ACC.
ARI
NMI
ACC
ARI
NMI
ACC
ARI
CLIP (zcro-shol)
56.5
43.1
26.9
79.9
63.4
50.2
81.0
63.6
45.4
72.5
56.7
40.8
CLIP (KMcans)
57.3
42.6
27.4
79.5
58.2
47.6
72.3
38.9
27.1
69.7
45.6
34.0
CLIP (SC)
57.9
46.7
31.8
81.7
63.3
73.9
40.0
29.8
71.2
38.9
SCAN
59.4
46.4
31.7
79.7
61.1
74.7
44.7
32.4
71.3
39.1
SIC
59.6
45.9
30.5
81.0
61.9
53.6
77.2
47.0
34.3
72.6
51.6
39.5
TAC (KMeans)
60.1
45.9
29.0
81.6
61.3
52.4
77.8
48.9
36.4
73.2
52.0
39.3
TAC (SC)
58.6
44.0
27.1
79.6
60.0
50.1
78.0
49.1
36.2
72.1
51.0
37.8
Gradnorm
63.1
50.9
34.2
82.5
62.7
53.2
79.2
52.6
39.1
74.9
55.4
41.7
Ours
61.7
52.0
33.6
83.0
67.9
99.4
56.3
39.4
44.1
4.1
MAIN RESULTS
Datasets. We evaluate effectiveness of our method by conducting experiments over 1) five widely-used datasets: STL-10 (Coates et al., 2011), CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR-20 (Krizhevsky & Hinton, 2009), ImageNet-10 (Chang et al., 2017) and ImageNet-Dogg (Chang et al., 2017); 2) three more complex and challenging datasets: DTD (Cimpoi et al., 2014), UCF101 (Soomro et al., 2012) and ImageNet-1K (Deng et al., 2009).
Evaluation Classical Datasets. Different with early baselines adopting ResNet-34 ResNet-18 as the backbone, this paper mainly focuses on comparisons with zero-shot CLIP and CLIP-based methods. As shown in Table our methods consistently outperforms TAC (Li et al., 2024) and zero-shot CLIP (Radford et al., 2021) on five classical datasets. Notably, our method achieves 7.8% and 9.8% improvement in ARI and ACC on ImageNet-Dogs. respectively. On CIFAR-10 dataset, SIC (Cai et al., 2023) is marginally superior to our method, as it entails more trainable parameters and more sophisticated training strategy.
Since the rapid development of pre- pre-trained models has made clustering on relatively simple datasets such STL-10 and CIFAR- no longer challenging, we evaluate our proposed method three more challenging datasets: DTD (Cimpoi et al., 2014), UCF101 (Soomro al., 2012) and ImageNet-1k (Deng et al., 2009). Our proposed method notably achieves the trat-offthe-art perfomance which is provided in Table 2. To be specific, the proposed method outperforms TAC over 7.0% in ARI and 6.9% in ACC on UCF101. The results prove the effectiveness of proposed strategy in studying spectral clustering from the perspective of NTK.
Evaluation on Challenging Datasets.
(a) CLIP (NM1=72.8%)
(b) TAC (NMI=75.3%)
(c) Ours (NM1=82.4%)
Figure 2: Visualization of affinity matrices on ImageNet-Dogs..
4.2 VISUALIZATION
Affinity Matrix. We examine how different choices affinity measure impact the resulting affinity matrix in Figure 2, where and columns of each affinity matrix permuted according to the ground-truth labels of images. In contrast to affinity matrices that are computed by applying the RBF kernel the pre-trained CLIP features (Fig. 2(a)) and TAC's image-eex--oomcenraaeed

---

Published as a conference paper at ICLR 2026
0.05
0.85
The objective value
0.10
0.65
ebjeetive
0.77
NMM
0.00
0.02
0.07
10
20
30
5
10 15
20
25
30
0.70
0.00
10
20
30
10
15 Iteration
20
25
30
0.53
Iteration
Iteration
Iteration
(a) CIFAR-10
(b) DTD
Figure 3: The objective value of Eq. (9) and the clustering performance (measured by NMT) each optimization iteration on CIFAR-I0 10 and DTD, respectively.
features (Fig. 2(b)) respectively, the affinity matrix Fig. 2(c), which computed by applying proposed NTK on the pre-trained CLIP features, exhibits the sharpest block-diagons structure: within-block entries are dense and homogeneous while off-block values are suppressed toward zero.
Visualizations on other datasets can be found in appendix (c.f. Fig. 8 and Fig. 9).
Convergence Speed. In Fig. 3, we plot the objective value of Eq. (9) and the clustering performance
at each iteration of the diffusion process. As can be clearly seen from Fig. 3, when affinities propagated iteratively, the objective value keeps decreasing and the clustering performance keeps increasing until reaching the equilibrium. Moreover, Fig. 3 shows that the objective value of Eq. (9) converges within a small number of iterations, which implies the efficiency of our proposed method.
Performance (%)
0.020.030.04
0.06
0.08
0.1
0.020.030.04
0.06
0.08
0.1
0.020.030.04
0.05
0.08
0.1
Figure 4: Ablation analysis of clustering performance by varying the value of CIFAR-10 (left), DTD (middle) and UCF101 (right), respectively.
100
Purformanca (%6)
ACC
NMI
AfI
ACC
NMI
ARI
ACC
NHI
ARI
Perform
12
16
20
30
40
8
12
16
20
30
40
12
16
20
30
40
Figure 5: Ablation analysis of clustering performance by varying the value of CIFAR-10 (left), DTD (middle) and UCF101 (right), respectively.
4.3 ABLATION STUDY
Ablation on Hyper-parameters. We evaluate the hyper-parameters most essential to our algorithm design, including the temperature 7 in Eq. (6), the number of nearest neighbors in Eq. (8), and the weighting parameters μ and in Eq. (9). Fig. 4 and Fig. 5 show that having large small value of and g does not necessarily improve the performance. As can be found in Fig. 6 and Fig. 7, our method is stable across wide range of the weighting parameters μ and X.
Ablation on Visual Encoder. We evaluate it with two different visual encoders, ViT-B/16 and VIT- L/14, and report the clustering results in Table 3. It can be seen that the clustering performance be enhanced by more powerful visual encoders. While our proposed method consistently outper- forms TAC across both backbones, indicating better generalization.
8

---

Published as a conference paper at ICLR 2026
Parformance (96
ACC
ACC
NMI
ARI
ACC
NMI
0.1 0.2
0.3
0.5
0.7
0.9
0.1
0.2
0.3
0.5
0.7
0.9
0.1 0.2 0.3
0.3
0.7
0.9
Figure 6: Ablation analysis of clustering performance by varying the value of μ CIFAR-10 (left), DTD (middle) and UCF101 (right), respectively.
100
70
30
Performance (%)
ACC
NMI
ARI
ACC
1011
ARI
+ ACC
NMI
ARI
90
60
80
50
70
70
40
60
60
30
15
50
Figure 7: Ablation analysis of clustering performance by varying the value of À on CIFAR-10 (left),
DTD (middle) and UCF101 (right), respectively.
4.4 DOMAIN-GENTRALLIZABL IMAGE CLUSTERING.
To assess the transferability of our method, we perform clustering on several versions ImageNet- IK variants, including ImageNet-C (Hendrycks & Dietterich, 2019), ImageNet-V2 (Korbblihh al., 2019) and ImageNet-S (Wang et al., 2019). As observed in Table 4, both TAC and our method suffer performance drops under these shifts, underscoring the challenge of clustering in such settings. Even so, our method consistently surpasses TAC across the in-distribution variants, highlighting its stronger robustness domain shifts.
4.5 FINE-GRAINED IMAGE CLUSTERING.
We validate our method under the fine-grained scenario by conducting experiments on five fine- grained datasets, including Aircraft (Maji et al., 2013), Food (Bossard al., 2014), Flowers (Nils- back & Zisserman, 2008), Pets (Parkhi et al., 2012) and Cars (Krause et al., 2013). As shown in Table 5, our proposed method consistently outperforms the state-of-the-an which highlights the superiority of text-informed affinities for image clustering.
RELATED WORK
Deep Spectral Clustering. Deep spectral clustering integrates traditional spectral clustering with deep neural networks for effective clustering. Tian et al. (2014) observe the similarity between the optimization objectives of autoencoders and spectral clustering. They thus propose replacing spectral embedding with deep autoencoder that is trained to reconstruct the pre-defined affinity matrix. However, obtaining such affinity matrix can be challenging for complex data, and the matrix size can become prohibitively large for large datasets. SpectralNet (Shaham et al., 2018) proposes to directly embed raw data into the eigenspace of given affinity matrix, which has inspired numerous extensions. Yang et al. (2019) enhance the robustness of SpectralNet's embeddings using a dual autoencoder. Huang et (2019b;a), respectively, extend SpectralNet handle multi-view data. Another line of works (Duan et al., 2019; Affeldt et al., 2020; Golikov et al., 2022) highlight the benefits of joint optimization and propose joint spectral embedding learning and clustering.
Image Clustering with Vision-Language Representations. The seminar work called SIC (Cai et al., 2023) uses textual semantics to enhance image pseudo-labeling, followed by performing im- age clustering with consistency learning in both image space and semantic space. Note that, SIC essentially pulls image embeddings closer to embeddings semantic space, while ignoring the im-

---

Published as a conference paper at ICLR 2026
Table Clustering performance (%) on five widely-used datasets. The best results are in bold.
Backbone
Dotaset
STL.10
CIPAR-10
ImageNer- Dogs
Metries
NMI
ACC
ART
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
TAC (KMeans)
95.1
96.1
93.6
82.9
91.2
80.5
62.6
60.4
45.7
82.3
81.9
72.9
62.6
50.4
33.6
WT-B/16
63.0
82.1
73.0
Ours
97.2
99.0
97.4
86.0
93.1
85.4
65.7
64.4
49.1
84,9
86.7
75.0
66.3
55.8
37.2
(KMeans)
95.4
89.1
64.8
47.6
75.4
64.7
WT-1/14
TAC (SC)
93.8
96.7
93.8
89.4
94.1
88.0
65.0
63.0
47.5
84.0
84.0
75.0
65.1
53.5
35.9
Ours
97.7
99.5
98.1
92.1
96.6
90.7
66.9
66.1
50.8
86.2
88.3
79.0
68.1
58.0
38.9
Table 4: Clustering performance (%) robustness to domain shift. The best results are in bold.
Dataset
ImageNet-C
ImageNet-N
ImageNet-S
Average
Metrics
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NM1
ACC
ARI
TAC (KMeans)
71.4
39.2
25.6
71.7
38.5
23.0
70.7
34.8
22.1
71.3
37.5
23.6
TAC (SC)
70.9
39.0
25.5
72.0
39.0
23.6
70.1
33.7
21.6
71.0
37.2
23.6
Ours
75.0
44.0
28.1
74.5
42.7
27.2
72.7
40.1
25.3
74.1
42.3
26.9
Table 5: Clustering performance (%) on five fine- fine-grained datasets. The best results are in bold.
Dataset
Aircraft
Food
Flowers
Pets
Cars
ACC
ARI
ACC
ACC
ARI
ACC
ACC
TAC (KMeans)
47.7
20.1
10.4
69.4
59.2
43.9
86.0
66.9
58.5
80.4
66.9
59.2
64.7
33.2
21.7
TAC (SC)
47.1
20.3
10.0
68.9
59.0
43.4
67.0
59.5
79.9
66.0
58.5
65.0
33.9
22.0
GradNorm
50,3
24.0
13.1
75.0
67.8
52.4
86.7
70,8
64.2
81.5
72.0
62.8
68,3
38.0
26.6
Ours
51.9
24.2
14.2
72.1
61.8
47.2
88.3
61.0
84.9
72.0
64.0
67.7
39.0
26.4
provement of text semantic embeddings. Differently, Li et al. (2024) and Peng al. (2025b;c) focus on leveraging textual semantics to enhance the feature discriminability by either simply concentrat- ing textual and visual features or its proposed cross-modal mutual distillation strategy
Due to space limitation, more related work to this paper is discussed in Appendix A.
6 CONCLUSION
In this paper, we advance spectral clustering into the multi-modal era by presenting Neural Tan- gent Kernel Spectral Clustering, which anchors the NTK with positive textual semantics couple visual similarity and semantic overlap in defining affinities. This design sharpens block-diagonal structures by amplifying intra-cluster connections and suppressing cross-cluster noise, while our proposed regularized affinity diffusion further enhances robustness through adaptive ensembling of pompp--ppeific affinities. Empirically, our method achieves strong performance compared com- petitive baselines on 16 datasets, which echoes our theoretical insights. Besides, extensive ablations provide further understandings of our proposed method. We hope this work could serve as cata- lyst, motivating future studies spectral clustering with vision- language representations, which is believable to be a promising direction for methodology improvement and real-world application.
ACKNOWLEDGMENT
This work is supported by the Australian Research Council Discovery Early Career Researcher Award (DE250100363), Australian Future Fellowship (FT230100121), Australian Laureate Fellow- ship (FL.19010114)) and Discovery Project (DP250100463).
ETHICS STATEMENT
Our study relies solely on publicly available datasets and models. No private or personally iden- tifiable information used. The work aims advance the scientific understanding of spectral clustering while upholding principles of transparency, faimess, and responsible research.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All the pre-trained CLIP-based models used in this paper are publicly accessible. We provide de- tailed proofs in the appendix and source codes in the supplementary materials.
REFERENCES
Séverine Affeldt, Lazhar Labiod, and Mohamed Nadif. Spectral clustering via ensemble deep au- toencoder learning (sc-edae). Pattern Recognition, 108: 108:107522, 2020.
Song Bai, Shaoyan Sun, Xiang Bai, Zhaoxiang Zhang, and Qi Tian. Smooth neighborhood structure mining on multiple affinity graphs with applications to context-sensitive similarity. European conference on computer vision, pp. 592-608. Springer, 2016.
Song Bai, Xiang Bai, Qi Tian, and Longin Jan Latecki. Regularized diffusion process for visual retrieval. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Song Bai, Xiang Bai, Qi Tian, and Longin Jan Latecki. Regularized diffusion process on bidirec- tional context for object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 41(5): 1226, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Shaotian Cai, Liping Qiu, Xiaojun Chen, Qin Zhang, and Longteng Chen. Semantic-enhancer image clustering. In AAAI Conference on Artificial Intelligence, volume 37, pp. 6869-6878, 2023.
Jianlong Chang, Lingfeng Wang. Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In International Conference on Computer Vision, pp. 3879-5887, 2017.
Chunchun Chen, Wenjie Zhu, Bo Peng, and Huijuan Lu. Towards robust community detection via extreme adversarial attacks. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2231-2237.. IEEE, 2022.
Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable representation learning and zero-shot transfer in clip. arXiv preprint arXiv:2310.009) 2023.
Mireea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. JEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics, pp. 215- 223, 2011.
Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13693-13702, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929, 2020.
Liang Duan, Charu Aggarwal, Shuai Ma, and Saket Sathe. Improving spectral clustering with deep embedding and cluster estimation. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 170-179. IEEE, 2019.
11

---

Published as a conference paper at ICLR 2026
Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence, 35(11):275- 2013.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision, pp. 5736-5745, 2017.
Eugene Golikov, Eduard Pokonechnyy. and Vladimir Korviakov. Neural tangent kernel: A survey. arXiv preprint ar0.v:2208.1300000 13614, 2022.
Wengang Guo, Wei Ye, Chunchun Chen, Xin Sun, Christian Böhm, Claudia Plant, and Su- santo Rahardja. Bootstrap deep spectral clustering with optimal transport. arXiv preprint arXiv:2508. 04200, 2025.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 12000..2221,,, 2019.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning, pp. 1558-1567. PMLR, 2017.
Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In /EEE/CVF Conference Computer Vision and Pattern Recognition, June 2020.
Shuning Huang, Kaoru Ota, Mianxiong Dong, and Fanzhang Li. Multispectraanet: Spectral cluster- ing using deep neural network for multi-view data. IEEE Transactions on Computational Social Systems, 6(4):749-760, 2019a.
Zhenyu Huang, Joey Tianyi Zhou, Xi Peng, Changqing Zhang, Hongyuan Zhu, and Jiancheng Lv. Multi-view spectral clustering network. In IJCAI, volume 2. pp. 4, 2019b.
Zhizhong Huang. Jie Chen, Junping Zhang, and Hongming Shan. Learning representation for clus- tering via prototype scattering and positive sampling. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(6):7509-7524, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In International Conference on Computer Vision, pp. 9865-9874, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang. and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148. 2016.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languuer transformer without convo- lution or region supervision. In International conference on machine learning, pp. 5583-5594.
PMLR, 2021.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEF/VV conference on computer vision and pattern recognition, pp. 2661- 2671, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ane-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, June 2013.
12

---

Published as a conference paper at ICLR 2026
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:/988.035577 2019.
Xuelong Li, Tengfei Wei, and Yang Zhao. Deep spectral clustering with constrained laplacian rank. IEEE Transactions Neural Networks and Learning Systems, 35(5):7102-7113 2022a.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language- -magg pre- training paradigm. arXiv preprint arXiv:2110.05208, 2021a.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In AAAI Conference on Artificial Intelligence, volume 35, pp. 8547-8555, 2021b.
Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. Twin contrastive learning for online clustering. International Journal of Computer Vision, 130(9):2205-222 2022b.
Yunfan Li, Peng Hu, Dezhong Peng, Jiancheng Lv, Jianping Fan, and Xi Peng. Image clustering with external guidance. In Forty-first International Conference on Machine Learning, 2024.
Junfeng Liao, Qizhou Wang, Shanshan Ye, Xin Yu, Ling Chen, and Zhen Fang. Explainable Ilm unlearning through reasoning. In The Fourteenth International Conference on Learning Repre- sentations, 2026.
Guangean Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE transactions on pattern analysis and machine intelligence, 35(1):171-184, 2012.
Sihang Liu, Wenming Cao, Ruigang Fu, Kaixiang Yang, and Zhiwen Yu. Rpsc: Robust pseudo- labeling for semantic clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14008-14016, 2024.
Yiding Lu, Yijie Lin, Mouxing Yang. Dezhong Peng, Peng Hu, and Xi Peng. Decoupled contrastive multi-view clustering with high-order random walks. In Proceedings of the AAAI conference artificial intelligence, volume 38, pp. 14193-14201, 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.3151,, 2013.
Ioannis Maniadis Metaxas, Georgios Tzimiropoulos, and Ioannis Patras. Divelust: Controlling di- versity in deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3418-3428, 2023.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 41, 1995.
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4610-46111 2019.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, doi: 10. 10.10/C/GGPPP
pp. 722-729, 2008,
Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pscudo-labeling for image clustering. IEEE Transactions Image Processing, 31:7264-7211 -278, 2022.
13

---

Published as a conference paper at ICLR 2026
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012.
Bo Peng and Wenjie Zhu. Deep structural contrastive subspace clustering. In Asian Conference on Machine Learning, pp. 1145-1160. PMLR, 2021.
Bo Peng, Wenjie Zhu, and Xiuhui Wang. Deep residual matrix factorization for gait recognition. In Proceedings of the 2020 12th International Conference Machine Learning and Computing, pp. 330-334, 2020.
Bo Peng, Zhen Fang, Guangquan Zhang, and Jie Lu. Knowledge distillation with auxiliary variable. In Forty-first International Conference Machine Learning, 2024a.
Bo Peng. Yadan Luo, Yonggang Zhang, Yixuan Li, and Zhen Fang. Conjnorm: Tractable density estimation for out-of-distribution detection. arXiv preprint arXiv:2402. 17888, 2024b.
Bo Peng. Jie Lu, Guangquan Zhang. and Zhen Fang. An conommmtioonthhoretiti framework for un- derstanding dustr--dist detection with pretrained vision-language models. The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025a.
Bo Peng, Jie Lu, Guangquan Zhang, and Zhen Fang. On the provable importance of gradients for autonomous language-assisee assisted image clustering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19805-19815 2025b.
Bo Peng, Jie Lu, Guangquan Zhang, and Zhen Fang. On the provable importance of gradients for language-assisted image clustering. arXiv preprint arXiv:2510. 16335, 2025c.
Bo Peng. Jie Lu, Yonggang Zhang, Guangquan Zhang. and Zhen Fang. Distributional prototype learning for out-of-distribution detection. In Proceedings of the 31st ACM SIGKDD Conference Knowledge Discovery and Data Mining V. I, pp. 1104-1114, 2025d.
Xi Peng, Shijie Xiao, Jiashi Feng, Wei Yau, and Zhang Yi. Deep subspace clustering with sparsity prior. In Ijeai, pp. 1925-1931, 2016.
Qi Qian. Stable cluster discrimination for deep clustering, In IEEE/CVF International Conference on Computer Vision, ICCV 2023, 2023.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprini AXXiv:15/1.06434 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectrainet: Spectral clustering using deep neural networks. arXiv preprint arXiv:1807.00 001.88007.01877,.. 2018.
Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip Torr, and Ling Shao. You never cluster alone. Advances in Neural Information Processing Systems, 34:27734-27746, 2021.
Jianbo Shi and Jitendra Malik. Normalized and image segmentation. IEEE Transactions pattern analysis and machine intelligence, 22(8):888-99)) 905, 2000.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 121::0222 2012.
Yaling Tao, Kentaro Takagi, and Kouta Nakata. Clustering- representation learning via instance discrimination and feature decorrelation. In International Conference on Learning Rep- resentations, 2020.
Fei Tian, Bin Gao, Qing Cui, Enhong Chen, and Tie- Yan Liu. Learning deep representations for graph clustering. In Proceedings of the AAAI conference artificial intelligence, volume 28, 2014.
14

---

Published as a conference paper at ICLR 2026
Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In International Conference on Learning Representations, 2020.
Gansbeke Wouter Van, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning classify images without labels. In European Conference on Com- puter Vision, pp. 268-285. Springer, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
Zhen Wang. Zhaoqing Li, Rong Wang. Feiping Nie, and Xuelong Li. Large graph clustering with simultancous spectral embedding and discretization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12):4426 4440, 2020.
Stephen Wright. Coordinate descent algorithms. Mathematical programming, 151(1):3-4,, 2015.
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In International Conference Computer Vision, pp. 8150-8159, 2019.
Tong Tong Wu and Kenneth Lange. Coordinate descent algorithms for lasso penalized regression. 2008.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference machine learning, pp. 478-487. PMLR, 2016
Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: green neural architecture search. In International Conference on Machine Learning, pp. 11613-11625. PMLR, 2021.
Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1066-4075. 2019.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arX1v:2111.07783, 2021.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111 0130300, 2021a.
Shangzhi Zhang, Chong You, René Vidal, and Chun Bhun-Guang Li. Learning self- eeff-expressive network for subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12393-12403, 2021b.
Yonggang Zhang, Jie Lu, Bo Peng. Zhen Fang, and Yiu-ming Cheung. Learning shape in- distribution feature space for out distriibutt detection. Advances in Neural Information Pro- cessing Systems, 37:49384 49402, 2024.
Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Xian-Sheng Hua. Graph contrastive clustering. In International Conference on Computer Vision, pp. 9224 9233, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16816 16825,, 2022.
15

---

Published as a conference paper at ICLR 2026
Wenjie Zhu and Bo Peng. Sparse and low-rank regularized deep subspace clustering. Knowledge- Based Systems, 106119,, 2020.
Wenjie Zhu, Bo Peng, and Chunchun Chen. Self-supervised embedding for subspace clustering. In Proceedings of the 30th ACM International Conference on Information & Knowledge Manage- meni, pp. 3687-3691, 2021.
Wenjie Zhu, Bo Peng, and Wei Qi Yan. Deep inductive and scalable subspace clustering via nonlocal contrastive self-distillation. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
16

---

Published as a conference paper at ICLR 2026
A MORE RELATED WORK
Deep Clustering. Due to its ability to reveal the inherent semantic structure underlying the data without requiring laborious and trivial data labeling work, clustering has been shown to benefit downstream tasks (Zhang et al., 2024; Peng et al., 2020; 2024a; Chen et al., 2022; Peng et al., 2025d; 2024b; 2025a) in computer vision. The popularity of deep image clustering can be attributed to the fact that distributional assumptions in classic clustering methods, e.g., compactness (Ester et al., 1996), connectivity (Wang et al., 2020; Zhu et al.. 2025), sparsity (Elhamifar & Vidal, 2013; Zhu et al., 2021) and low rankness (Liu et al., 2012: Zhu & Peng, 2020), can not be necessarily conformed by high-dimensional structural RGB images. To exploit the powerful representative ability of deep neural networks in unsupervised manner, the earliest attempts seek self-supervision signals by considering image reconstruction (Ghasedi Dizaji et al., 2017: Peng et al., 2016; Xie et al., 2016; Jiang al., 2016; Mukherjee et al., 2019; Radford et al., 2015; Peng & Zhu, 2021) and mutual information maximization (Hu et al., 2017; et al., 2019) as proxy tasks. Despite remarkable progresses. the learned representations may not be discriminative enough to capture the semantic similarity between images. More recently, the advance in self-supervised representation learning have led to major breakthroughs in deep image clustering. On the hand, IDFD (Tao al., 2020) proposes to perform both instance discrimination and feature de-correlation while MICE (Tsai et al., 2020) proposes a unified latent mixture model based on contrastive learning to tackle the clustering task. On the other hand, CC (Li et al., 2021b) and its followers TCC (Shen et al., 2021) perform contrastive learning both instance and cluster levels. Differently, ProPos (Huang al., 2022) performs non-contrastive learning the instance level and contrastive learning on the cluster level, which results enjoying the strengths of both worlds.
Vision-language Models (VLMs). Pretraining large-scale image-text pairs has made VLMs standard backbone for multi-modal transfer. Regarding the type of architectures, existing VLMs can be divided into two categories: (i) single-stream models that process concatenated visual and textual features into one transformer, such as VisualBERT (Li et al., 2019), ViLT (Kim et al., 2021), models that keep visual and text encoders seperate while learning modal alignment through contrastive pairing, e.g.. CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), SigLIP (Yao et al., 2021) and FILIP (Yao et al., 2021). More importantly, CLIP based models are widely adopted and has motivated a series of follow-ups aimed to improve data efficiency and better adaptation on downstream tasks (Li et al., 2021a; Zhang et al., 2021a; Zhou et al., 2022; Liao al., 2026). This paper uses CLIP as the pre-trained model, but our method can be generally applicable to contrastive models that promote vision-language alignment.
and (ii)
Neural Tangent Kernel. Neural tangent kernel (NTK) (Golikov et al., 2022) is kernel that reveals the connections between infinitely wide neural networks trained by gradient descent and kernel methods. NTK enables the study of neural networks using theoretical tools from the perspective of kernel methods. There have been several studies that have explored the properties of NTK: Jacot et al. (2018) propose the concept of NTK and showed that it could be used to explain the generalization of neural networks. Lee et al. (2019) expand on this work and demonstrated that the dynamics of training wide but finite-width NNs with gradient descent can be approximated by linear model obtained from the first-order Taylor expansion of that network around its initialization. This paper, rather than exploring the interpretability of infinite width neural networks, explores empirical (i.e., finite width) NTK to construct text-informed affinity matrix for spectral clustering.
B DERIVATION OF EQ. (11)
We consider the following optimization problem
B
B
min
A NIK) - A E
ク s.t. 0 < (b) < 1 and
Σ b=1
3| = 1, (15)
B.A
0=1
With fixed B, we can simplify Eq.
(15) as follows:
B
min Â
3b]
A A NTK) + A -EI
(16)
17

---

Published as a conference paper at ICLR 2026
Table 6: Spectural clustering performance (%) using different kernels for affinity measurement based on either CLIP image features or TAC image-text-come meage-text-conceettated features. The best results are high- lighted bold. indicates our reproduced results.
CHRARIT
CIARRTT TI
Teagrlee
Teegrhe Trig
DITI
DXPHR
Teegrhee
Marris
NMI
ACC
ARE
NMI
MCC
ARI
ACC
ARI
NMI
ACE
LRI
NMI
ACC
ARI
NWI
6OC
ART
NME
ACC
ARI
XMB
ACC
ART
NAB
ACC
ARI
Eddag CLiPieag
wi1
343
25.1
Leoz
661
85.1
133
84.8
66.
5338
44.
612
200
12.6
18.0
15.6
013
06.1
53.
51.6
404
12.3
915
928
54.
985
11:
28-3
18.8
823
30.5
851
87.
834
83.4
881
571
48.
433
301
12.4
18.3
364
124
706
95.9
11.0
467
218
91.
85.3
23-1
133
480
205
12.3
045
360
657
87.4
183
87.
88.6
58.5
4344
432
367
12.8
18.6
960
TEA
0022
13.9
26-7
4611
34.3
91.2
828
243
997
36-
213
71.8
63.
$6.1
12.3
15.
915
511
211
512
7:3
11.5
60.1
459
11A
31.1
22.
115
4.3
364
113
01.
88.1
792
12.3
TL.
710
456
01.
10.2
06.0
145
07.6
96.5
13.1
900
170
56.
133
011
98.
1611
01.0
12.6
14.4
460
706
13.
91.5
04.
11.5
513
246
05.
62.1
12.
360
13.
036
720
13.1
460
030
565
11.0
133
168
where
((A, A (b)
NTK)
1
Bi Oil
(b) (b)
Akk (Vath de
AlI
(17)
V Vajaa!
1,,,,1=1
(b) with 03
= A [ and a!
=
= j= A ANIK J1
Let us define
w((h
NIK ANTK E MMM
(b)
(b)
( D (8)
@ D (b)
NTK ER MM
NIK
with D NTK diagonal matrix of D (b)
NTK i
Ej= M A (b)
NTK jl.
and
(
(b) NTK
SKiK E RMMPXMM
with ((t) as the row-normalized (b)
NTK. i.e., S = (D
-1/2
A NTT (D(x),
(b)
-1/2
One can easily check that |
E = ||vee Á) vec(F) (E)
By introducing two identical coordinate transformations: E M(i - 1) le and 5 = M(j +l we have the following:
2
(8) {
Σ
(b)
alE]
NIK
w
6.6
(b)
2
2
w
(b)
á(c)2
M
alc]
W
5,5
(b)
a
0,6
(8)
(b)
1,8=1
6,5=1
1ee 66
(18)
M
a[E] à T (T)))
-1 1/2 (7)(1) (T(b) 11/2
á
Iyaa (()) -1/2 -1/2 wi())((0)) 1/2) à à
à 1/2 8(b)) à
where á vec( A), ( w 8) and 166
T(b) [66 8].
The following three facts applied for the derivation of Eq. (18):
1. ()(() is symmetric since (()) is symmetric.
18

---

Published as a conference paper at ICLR 2026
[6, Eei w/b [6,5] since
T (6, 6]
D
NTK
(M)
2]D NTK [k.
M
M
Σ
A 1T
A ATTRI Ik:
(19)
M M
(6) NIK
Σ
w [6
3. g(0) = T 1/2 W (T)) 11/2
since
g [6, s) S [A.
-D
NTK [i. (8)
NTK [i
0.3
NTK
(
0.5
A
NTW
(0)
[k.
1D
(0)
NTK 14 0..
(20)
( D NTK K kj
NTK R.
NTK
ID
NTK NTK 11,1 11
(6)
= =T 0000
(6,
[e,e] [e,]]
0.5
In summary, the objective function in Eq. (15) can be rewritten as follows:
B
Σ Bib] à
IIaa - -S()) à m1/á vee (
(21)
C DERIVATION OF EQ. (12)
By taking the partial derivative in Eq. (21) with regard to å, we have the following:
Σ
pb
a
{) 11 Oa
{lá vec(E))l
0à
0å
6=1
(22)
Σ
Bbb]
[IN= S((h)
al + 2p (a - vee
b=1
By setting 0J/0a-
0, we have the following:
vea A) = á
1
IMa
B Σ
3
((b)
-1
vec (E)
(23)
+
p
+1
b=1
Applying vec- (.)) to both sides of Eq. (23) results in the following:
1 vec (1
3
Σ
g(h)
vec (E)
(24)
11
6=1
D CONVERGENCE OF EQ. (13) TO EQ. (12)
Lemma 1. Let A E RRxx, the spectral radius of A is denoted p(A) max o(A)), where a (AA) is the spectrum of A that represents the set of all the eigenvalues. Let || || be matrix norm on guxn, given a square matrix A guxnn X an arbitrary eigenvalue of A, then we have
[|] < A{A) < A|.
Lemma 2. Let E m B Baxn, denote ()i, x. 1m and {uu y as the eigen-pairs of A and B respectively. The set of mn eigen-pairs of A 8 B is given by:
(hpg æ; & yi Wjji=l..... j=l.....
Lemma 3. Let A P XX X RaXu and B R2x4 respectively, then
Raxp
rec AXB
[ A)vec(X).
19

---

Published as a conference paper at ICLR 2026
Lemma 4. Let A R=*", then limg A 0 if and only if (A) <. Lemma 5, Given matrix A e TRaxn and p(A) 1, the Neumann series I, A A? +... converges to (In- A)
To get started, first consider the matrix (D
(b)
A
NTK. whose induced -norm is equal to 1,
(b)
i.e.,
(D
NTK
A ANTIles
co 1, since the i-th diagonal element in matrix D D equal to the summa-
tion of the corresponding i-th row in matrix A
(b) NTK.
Lemma
gives that (DDXi
(b)
A
ATK
(b)
S 1.
As for the matrix SKYK
D
(b)
NTK
1/2
A
(b) NTK
D
NTK
112
we concerned about, since
we can rewrite it as (b))
(D
NTK
(b)
1/2
D
(b) NTK
A
(DNTK]
11/2 thus it is similar to NTK
D (b)
A
NTK This implies that the two matrices share the same eigenvalues, such that
NTK
NIK
S 1. By applying Lemma 2, we can conclude that both the spectral radius of the Kro-
necker product g(b) = ((h)
{ is no larger than 1, i.., (8()) < 1.
@
By applying Lemma 3, Eq. (13) be vectorized as the following:
á (
Σ
((())
( a(r)
1 vea E)
1
1 +
B
(())
((()
vec E)
#+
+
(( (PP)
+ P
vec
(25)
#+1
#+1
3 (00
(P)
vec(E)
11 + vec(E)
++1
1
#
B
B
(())
(())
11
Σ
B
6 6
((b)
vec (E)
# +
1 1+
where the second step is derived based on the fact that ((h) is symmetric.
Since we have already proved that o (S(b) < 1, we have the spectral radius of Ebn upper-bounded by Σ
Blb] g(b)
to be
u+1
B
3(6] #+1
Moreover, since μ > O, have
B
b
3
EB
p
Σ
(())
<1
(26)
p+1
p+1
6=1
b=
By taking advantage of Lemma 4 and 5, we can easily demonstrate that the following two expres- sions hold true:
B
lim
Σ
8(b)
0,
(27)
Σ
(b)
I
m7
3 b
((
(28)
11
Therefore, the iterative sequence of á (p+1) asymptotically approaches a stable solution, converging to:
lim á (p+1) 40000
P
Inp
Σ
3b
8(6)
E).
(29)
20

---

Published as a conference paper at ICLR 2026
By performing the inverse operator vec (-) vec
the right side of Eq. (29), we have the following
(TTRe Σ
-1
A 1
vec-1
Bb
g(b)
vCc (E)
(30)
1a 1+1
+
E OPTIMIZE B WITH FIXED Á
Again, we consider the following optimization problem
X 111311 2
B
min Σ B|6 -(. NTR) + 4I| E B.A
s.t. 0 [[b] S 1 and
Σ b=1
3|b 1, (31)
When fixed, the objective value of É(A, A NTK) for each adjaceney matrix NTK Eq. (31) can be directly computed. As result, the optimization of /3 reduces to solving the following problem:
(b)
(b)
B
B
min
B|6 ((A, ((A ATK) + 2 111111 s.t. = 3 S 1 and Σ
-b =1.
(32)
Specifically, the objective function in Eq. (32) takes the form of a Lasso optimization problem, which can be solved by utilizing the coordinate descent method.
In each iteration of the coordinate descent, two elements 3[1] and زز are selected to be updated,
while the others fixed. Taking into account the Lagrange function for the constraint Σ Bbb
1, we have the following updating scheme:
X( + B(]) (HU) - H())
B
(33)
2A
B
B B B
(34)
where (b) (A, A NTR) To avoid the obtained [r] and [3] to violate the constraint 0 - [[b], we set 3 [1] = if A(3 + Bli)) + (HG) - ()))) < O, and 3 [j] O otherwise.
However, this strategy requires multiple iterations since only pair of elements of can be updated together: To address this issue, we propose a more efficient solution that allows updating all elements of [3 simultaneously, explicitly eliminating the need for iteration.
By taking advantage of the coordinate descent method, we can filter out the valid elements that are not governed by the boundary constraints, formally denoting the valid index set as B. Consequently, the inequality constraints of < 3 < 1 are slack to the weight set (B(b) and the optimization problem can be directly solved.
In particular, by introducing a Lagrangian multiplier 17, the Lagrangian function L(B, 17) can be formally defined as:
2(3,1) 1)
5100>>ap0 bcB
Σ (b) (+) + > 1111 n1 Σbbb)
(35)
The corresponding Karush-Kuhn Kkuhn-Tucker (KKT) conditions can then be formulated as:
C(B,)) aC(B, n)
(b)) {/ 17 = 0, be B,
03 aL(3,1) 11)
(36)
V C(B.Bm)) = Om
1 Eben B(b] = 0.
Note that have already taken the equation constraint EoEs 3(b] 1 into consideration when deriving the representation of Vono v)). The optimal result be obtained by solving the
[ + 1 equations. By summing up all the V Ven along b within B, the Lagrangian multiplier 7) can be obtained as:
E (b)
(37)
17
B
21

---

Published as a conference paper at ICLR 2026
Algorithm 1: Affinity Ensembling via Regularized Affinity Diffusion
Input: Affinity matrices (ANTK 18 mariter, weighting parameter μ and À
the reference matrix E, max number of iterations
Output: The ensembled affinity matrix À
1. initialize = 0, A E, and G = 1/B for each = 1,..., .....,
repeat
3. update Å with fixed (3 following Eq. (13)
4. compute H e(A, NTK) for each b = 1, 1...., B..... B
5. set [[b] = 0 for each b = 1,,...... 1,...,
6. filter the valid index set B following Eq. (39)
update [|b] with fixed À following Eq. (40) for each b
t ++t+1
until convergence or mariter
Therefore, by taking 7 back into the KKT conditions, we can obtain the optimal solution 3 [b], following:
Even 11 - -BIH VEI.
B [b]
(38)
A|B
Since all the element 3 [b) should satisfy the inequality constraint S (g) < 1, the above rela- tionships provide an effective strategy to determine the valid index set B, i.e., the corresponding H in B should satisfy H¹ (Ever 11 A)B1B). Therefore, can develop formalize definition
of the valid index set, as follows:
B = (u|11"
Σn + + 1,2...,,.................... ..
(39)
VEB
In practical implementation, we first sort all H in descending order and then sequentially remove the indices that fail to satisfy the constraint of Eq. (39), leading to the valid The optimal result can be obtained in single round of iteration, with the resulting weight vector 3 given by:
EveB H B +
U B,
A|B
(40)
0, O DE
F ABLATION ON REGULARIZED AFFINITY DIFFUSION
In this section, we conduct ablation study on our proposed Regularized Affinity Diffusion (RAD) mechanism in Section 3.2. To examine the effectiveness of RAD, consider the following two alternatives to RAD.
The first alternative is naively average the obtained B affinity matrices A (I) A A
NTK- i.e.,
(B)
(6) ATK-
A
(41)
B
hml
We refer this alternative as Ours (naive).
The second alternative, motivated by Prompt Ensembling (PE), uses Eq. (6) and Eq. (7) to construct
the ensembled affinity matrix À except that 0, vec (W) vec B b= w where (b))
the CLIP features of positive nouns induced by b-th prompt template A (b) (-). Formally, Å computed
22

---

Published as a conference paper at ICLR 2026
Table Clustering performance (%) on eight image clustering datasets between prompt ensemble and our methods. The best results are in bold.
16.3
512
683
363
915
163
153
185
211
900
901
661
16.3
702
991
meare
11.2
31.
129
111
213
300
18.
523
12.
603
331
91.
968
722
114
764
86.3
by the second alternative, called Ours (PE), can be given as follows:
Ali,j] = Kw (xx(x)))
if fx(x)) E (N)(fx(xs)) fx(x)) E (x), (42)
otherwise.
where
N w. (x(x)//7
sw (Xx(xx))) = log
(43)
with Wa D E
Table shows that Ours (RAD) achieves the best clustering performance, which validates the ef- fectiveness. Besides, one can also find that Ours (PE) significantly outperforms TAC (SC) and TAC (KMeans), which implies the effectiveness of our proposed NTK-induced affinity measure.
G BROADER IMPACT
This work proposes new deep clustering paradigm by leveraging external knowledge. As funda- mental problem in machine learning, clustering has a wide range applications, such as anomaly detection, person re-identification, community detection, etc. The proposed method is evaluated on public image datasets that are not at risk. However, just like any learning method, the perfor- mance of method depends data bias and cannot be guaranteed in complex real-worlde applications. In this sense, might bring some disturbances in decision-making and thus should be carefully used, especially in areas such health care, autonomous vehicles, etc.
H USAGE CLAIM OF LLMs.
We ChatGPT for grammar and spelling checks only, with prompt "Proofread the sentences".
I
LIMITATION
The proposed method requires manually setting the target cluster number. In world applications, one may resort to other cluster number estimation methods in the lack of a cluster number prior.
J ABLATION STUDY ON PROMPT TEMPLATES
To investigate how much has been prompt ensembling contributing to the performance compared to using only one prompt, report the clustering performance under piggle-prompttt setting in Table 8.
Even with a single prompt template, ours is still outperforms TAC with multiple prompt templates and achieves comparable performance the performance gains do NOT come from prompt engineering. Compared with TAC, the clustering performance of ours is less sensitive the choice single prompt template. Ours consistently and significantly outperforms TAC (where the kernel way is not used), which implies the advantage of our proposed NTK-based graph construction still holds with a single prompt template.
with multiple prompt templates, which implies that
23

---

Published as a conference paper at ICLR 2026
(a) CLIP (NM1=)7.9%)
(b) TAC (NMI=58.6%)
(c) Ours (NMI=61.7%)
Figure 8: Visualization of affinity matrices DTD.
(a) CLIP (NMI=88.1
(b) TAC (NMI-92.6%)
(c) Ours (NMI=95.8%)
Figure Visualization of affinity matrices on STL-100
Table 8: Ablation Study on Prompt Templates
11.3
03.5
11.3
115
102
96.
515
103
A15
V1.
23.4
30.5
11.5
46.4
11.
31.5
10.1
530
11.7
61.4
46.3
10
16.5
19.4
$2.5
11
91.2
16.
350
16.1
301
18.0
2911
65.2
06.1
616
67.9
69.1
463
411
S6.
60.5
57.6
67.5
55%
16.1
05.5
66.5
653
575
105
31.2
17.2
555
917
9.9
907
36.4
163
0.01
381
1611
11.5
16.3
33.
557
13.5
35.
355
60.2
60.
455
46.
65.0
20.0
616
30.0
67.4
96.1
01.9
06.6
48.1
60.0
61.
70.6
61.8
11.4
56.
153
90.5
76.1
6-1
91.5
63.9
66.
201
11.9
15.
1611
055
9699
91.
45.1
15.1
713
06.9
15.3
06.1
913
825
301
919
60.5
53.4
61.6
942
56.
16.3
13.5
66.1
TAC IK mmann
91.3
$33
9111
583
75.1
961
33.3
06.5
6633
01.0
50.3
26.1
44.
25.6
569
61.
153
41.
135
764
06.3
Okn
455
46.5
66.
21.0
015
32.3
$30
10.9
610
493
46.
69.1
41.2
760
60.1
51.4
53.4
85.5
07.5
06.1
163
551
16.6
76.6
6-1
IK eeen
27.6
26.
190
56.
709
911
142
06.6
221
51.4
14.
613
16.
007
133
14.3
16.1
7..
06.1
40.8
05.9
00.6
66.
69.1
11.8
69%
464
412
$0.
66.
51.
533
55.6
669
163
75.
60.6
013
16.3
09.
21.1
000
501
001
762
71.
3366
45.
954
06.
K VALIDATION OF IMAGE DATA LEAKAGE
It should be noted that many evaluation datasets used in this work have appeared in the training of the vision-language foundation models, which can raise a data-leakage problem since the rep- resentations under interest has been well obtained. To maximally prevent data-leakage problem in the OpenAl's pretrained CLIP models, we additionally conduct experiments using the pretrained CLIP models in OpenCLIP on CIFAR-10, CIFAR-100, ImageNet-A and ImageNet-IK, where the overlap percentage with the pre-trained dataset is 0.02%, 0.03%, 0.04% and 1. 1.02%, respectively.
24

---

Published as a conference paper at ICLR 2026
Table 9: Clustering performance with OpenCLIP's pretrained models. The best results are in bold.
Dansor
STL-10
CHRAR-10
CHEAR-10
ImageNa-11
ImngerirriDngs ACC
DCPIO
ImageNat-KK ACC
Awerggg
ACC
ARM MCC
MOC
ISM
ACC
MCC
ACC
ACC
TAC:
958
91.4
16.7
9/3
814
674
61.
90.1
91.8
910
96.8
91A
718
00.1
50.3
16
81
M1
168
67.5
80.0
72.9
61.5
TAC (SC)
93.6
95.2
92.8
86.1
93.3
85.9
62.
576
45.3
913
980
9699
14,
74.5
61.2
65.2
534
36.7
81.
648
562
76.7
473
55.1
79.7
73.2
62.5
Oum
96.1
9%1
99.3
88.9
984
98.9
665
642
513
973
99A
963
81.5
R32
16.7
66.
36.
016
R2A
601
580
768
550
W01
R24
775
664
Table 10: Validation of label-name leakage.
Dataset
CIFAR-10
CIFAR-20
ImageNet-
Metrics
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
WordNet w/o ground ground-truth class name Full WordNet
83.3
91.8
83.1
63.4
59.8
43.7
79.3
56.2
39.6
83.3
92.0
83.0
63.3
59.6
43.5
78.8
56.3
39.4
be found from Table 9 that using OpenCLIP's checkpoints contribut better compara- ble performance than OpenAI's checkpoints, which means that performance benefit of CLIP-based clustering does not result from the data-lcakage problem. Besides, we note that, using both kinds of checkpoints, our method consistently outperforms TAC.
L VALIDATION OF LABEL-NAME LEAKAGE
To further rule out potential label-name leakage, we additionally conducted experiment where we remove all words that exactly match any ground-truth class name from WordNet and re-run our method. As shown in Table 10, the clustering performance (averaged over 5 runs) of our method remains essentially the same (ACC NMI ARI differences are small), indicating that our emprical advantages do not rely on the presence of true label names but on the richer semantic structure provided by generic "in-the-wild" nouns and their interactions via our proposed NTK- affinity.
M STATISTIC SIGNIFICANCE
We report the statistical significance of our method via statistical comparisons over 8 benchmarks used in Table I and Table 2. To this end, very common practice yields the paired t-test. So before we list the results, let's s define the following hypothesis to test.
Po The compared two models may not have significant statistical differences regarding their clustering performance.
Pi The compared two models may have significant statistical differences regarding their clustering performance.
Table 1l reports the p-value of Wilcoxon Signed-Ranks Test where the clustering performance is measured by NMI. According to the p-values of our method against all compared CLIP-based baselines, conclude method consistently rejects the null hypothesis with p < 0.05, which means our clustering performance is statistically significant enough to be distinguished from the others.
N TIME AND SPACE COMPLEXITY
The time and space complexity of the proposed method can be broken down into two main compo- nents: (a) NTK-induced graph construction and (b) Regularized Affinity Diffusion (RAD).
Table 11: Paired hypothesis p-values.
TAC
SIC
CLIP (Kmeans)
Ours
0.0078
0,0039
0.0078
25

---

Published as a conference paper at ICLR 2026
Table 12: Computation cost of our method and spectral clustering on ImageNet-
NTK-induced graph construction
RAD
Spectral clustering
1.1 mins
8.2 mins
3.6 mins
For NTK-induced graph construction, one can easily check from Eq. (8) that time and space com- plexity is O(BM²d + N)) and O((M + BN)d) where the number of prompt template, the CLIP feature dimension, N and M is the number of positve nouns and images.
For RAD, it seems that we need space complexity O(BM²) space to store the B prompt-specific affinity matrices. However, since the affinity matrices share sparse mutual q-nn pattern (each row has O(q) nonzeros), the space complexity accordingly drops to O(BMq). which is linearly dominated by M since q, B < M. In practice, by storing the affinity matrices Torch sparse tensor, the total memory usage is around 30 MB.
Regarding to the time complexity of RAD, it has two parts. The first one is updating Ávia Eq- (13). Since each (b) (the row-normalized A NTK) has (g)) nonzeros, each multiplication is around O(Mq²), giving (tt (M2²)) for computing Eq. (13) for t steps. The second part is updating B Considering the overall iteration
in (t B²) with step coordinate descent shown Appendix number T in alternating optimization, the final space complexity of RAD is ((T (2 Since 1, l2, T, B², B², ,,² < M, we can conclude that the over time complexity is dominated by O(M).
Table 12 reports computation cost of our method and spectral clustering ImageNet-IK on single Nvidia A100, where one can find that RAD takes 8 mins and spectral clustering takes 3.6 mins. Therefore, argue that our method and spectral clustering can be scaled to large datasets.
26


Reviews:
[{"review_id": "547HhSa6jB", "reviewer": "Reviewer_KbNX", "summary": "This paper aims to take advantage of multi-modal (i.e., vision-language) representations of data for spectral clustering. Concretely, given a pre-trained vision-language model, data without any supervision can be anchored using neural tangent kernel with positive nouns, so that data representations can be further refined by considering the language modality of interest. This method can also take benefit from the ensemble of multiple language prompts. Thereafter, the refined affinity matrices can be used to do spectral clustering and better performance can be observed on various image datasets.", "questions": "Questions can be found in the above weakness section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The problem of considering multiple modalities of data in spectral clustering is interesting and on time given the current advances in multi-modal foundation models.\n\n2) Using neural tangent kernel to anchor the data using multi-modal information is interesting and sound.\n\n3) Better clustering performance can be observed on various image data.", "weaknesses": "1) To consider both vision and language modalities in spectral clustering of images, this work extends existing techniques/strategies of \"positive nouns\", \"neural tangent kernel\", etc, which is making this work's technical contribution moderate.\n\n2) It should be noted that many evaluation datasets used in this work have appeared in the training of the vision-language foundation models, which can raise a data-leakage problem since the representations under interest has been well obtained. Therefore, it is necessary to include a discussion on if the performance benefit is directly from the foundation model's representation or from the proposed method. Another potential way is using data not included in the training of the foundation model for clustering evaluation.\n\n3) An ablation study on the contributions of each component in the proposed method would be interesting. For example, how much has been the ensemble contributing to the performance compared to using only one prompt? How much does a single prompt contribute using the proposed kernel way vs without using the kernel way.\n\n4) Some typos can be fixed, e.g., \"aim to grouping\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VtdkxJBfD", "reviewer": "Reviewer_nmSD", "summary": "The paper proposes Neural Tangent Kernel Spectral Clustering (NTK-SC) for image clustering with vision–language models. \nThe core idea is to anchor a proxy network at text features of positive nouns, compute an empirical NTK over CLIP image embeddings, and use it as an affinity for spectral clustering. \n\nConcretely, with CLIP's unit-normalized visual features \\\\(z=f_X(x)\\in\\mathbb{R}^d\\\\) and noun features \\\\(W=[w_1,\\dots,w_N]\\\\), the proxy is chosen as a log-sum-exp \\\\(g_{\\theta_0}(z)=\\log\\sum_{k=1}^N e^{w_k^\\top z/\\tau}\\\\) (Eq. (7)). The empirical NTK between two images has the closed-form (Eq. (8)).\n\nAffinities are sparsified via mutual \\\\(q\\\\)-NN and fed to SC (Eq. (2)). RAD alternates a diffusion update\n\n$$\n\\hat A \\leftarrow \\sum_{b=1}^B \\frac{\\beta[b]}{\\mu+1}\\,S^{(b)}\\hat A S^{(b)\\top}+\\frac{\\mu}{\\mu+1}\\,E\n$$\n\n(Eqs. (11)-(13)) and a Lasso-like weight update for \\\\(\\beta\\\\) (Eq. (14)), with \\\\(E=I\\\\). Experiments on 16 datasets claim consistent gains over TAC/SIC/CLIP baselines.", "questions": "In addition to the issues raised above, I have the following questions:\n\n1. Do you ever rectify \\\\(K_{\\theta_0}\\\\) or clip \\\\(A\\\\) to be nonnegative before computing \\\\(L=I-D^{-1/2}AD^{-1/2}\\\\)? Do you remove the diagonal prior to eigen-decomposition? Please report results with \\\\(A\\leftarrow\\max(0,K_{\\theta_0})\\\\), \\\\(A\\leftarrow (K_{\\theta_0}+|K_{\\theta_0}|)/2\\\\), and with/without diagonal. (Eqs. (2), (6), (8))\n\n2. What exactly is the algorithm for selecting “positive nouns,” especially given the definition “similar to any ID label”? What signals from the train split are used to filter nouns, and how do you prevent label-name leakage?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow.\n- Eq. (8) decomposes the affinity into visual proximity and a text-induced overlap. It is easy to compute once noun logits are available. \n- The RAD update and its fixed-point interpretation are derived and come with a convergence argument for the linearized step.   \n- Results cover classic, fine-grained, and domain-shift benchmarks. Ablations on \\\\(\\tau,q,\\mu,\\lambda\\\\) are provided.", "weaknesses": "> 1. I wonder if negative weights violate standard SC assumptions. \n\nSince CLIP features lie on the unit sphere, \\\\(\\langle z_i,z_j\\rangle\\in[-1,1]\\\\) (Eq. (1)). With Eq. (8), the NTK inherits negative values whenever the cosine is negative:\n\n$$\nK_{\\theta_0}(z_i,z_j)=\\frac{1}{\\tau^2}\\langle z_i,z_j\\rangle \\sum_k s_i[k]s_j[k].\n$$\n\nYet the method sets \\\\(A_{ij}=K_{\\theta_0}\\\\) on mutual \\\\(q\\\\)-NN (Eq. (6)) without rectification or signed-graph treatment. Normalized-cut SC (Eq. (2)) typically presumes nonnegative affinities, negative edges break stochastic interpretations and many guarantees. No analysis or ablation is given for rectifying \\\\(A\\ge 0\\\\) (e.g., \\\\(A\\leftarrow\\max(0,K)\\\\) or shifting).\n\n> 2. Graph quality and connectivity under fixed q are unexamined.\n\nThe paper fixes \\\\(\\tau=0.04,\\,q=30,\\,\\mu=0.1,\\,\\lambda=10\\\\) for all datasets (sizes and densities vary widely). There is no report of connected components or sensitivity on large graphs (e.g., ImageNet-1K). Disconnected graphs produce multiple zero eigenvalues and can distort clustering. \n\n> 3.Scaling and runtime are not discussed.\n\nThe RAD fixed-point (Eq. (12)) is acknowledged infeasible to invert (\\\\(M^2\\times M^2\\\\)), hence an iterative update (Eq. (13)). However, per-iteration costs with \\\\(B\\\\) prompt graphs and sparse matrices (expected \\\\(O(B\\cdot M q^2)\\\\)) and memory footprints are not reported\n\n> 4.Comparisons omit crucial baselines and uncertainty.\n\nAbsent is a fair product-kernel baseline (cosine \\\\(\\times\\\\) text overlap).\n\n> 5. Evidence issues\n\nThe paper claims consistent SOTA improvements and sharper block diagonals, but (i) the evidence rests on fixed hyperparameters across datasets of very different statistics, (ii) TAC multi-prompt usage is argued via an implementation detail (footnote) rather than a principled comparison, and (iii) no statistical significance is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cbh7NcxgGt", "reviewer": "Reviewer_rFTU", "summary": "This paper is about a new clustering approach named Neural Tangent Kernel Spectral Clustering. This approach that extends traditional spectral clustering to leverage multi modal repreentations from pre trained vision language models. The core contribution is formulating image affinity as a coupling of visual proximity and semantic overlap through anchoring neural tangent kernels with semantically relevant positive nouns. The paper includes a regularized affinity diffusion mechanism to ensemble affinity matrices from different prompts and demonstrate substantial improvements across 16 datasets.", "questions": "- How does performance degrade with noisy or out of distribution images?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The integration of neural tangent kernel theory with vision language representations for spectral clustering is novel. The idea of anchoring NTK with positive nouns to create semantically aware affinity matrices represents a way to incorporate linguistic priors into clustering.\n+ The paper includes a comprehensive experimental validation, where testing is performed on 16 benchmarks. The consistent improvements suggest that the method is generalizable.\n+ The presentation is excellent, the paper is easy to read and well written.", "weaknesses": "- The method uses some positive nouns that are semantically close to images of interest. This could limit practical applicability and introduce some bias. The paper should provide some analysis about it.\n- A minor weakness, is that the main paper is condensed and the related works in that are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "STXT4f9RBV", "reviewer": "Reviewer_kiq4", "summary": "This paper proposes a spectral clustering method for features extracted from vision language models. The main contributions lie in integrating the affinities from the vision and text modalities, as well as multiple prompts. Experimental results on 13 datasets demonstrate the effectiveness of the proposed method.", "questions": "I expect the authors to address my concerns on the weakness section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is mathematically grounded, with detailed derivations provided in the appendix.\n2. The paper is clearly organized and written in general.\n3. Experiments show that the proposed method outperforms existing baselines on both classic and more challenging, fine-grained datasets.\n4. Parameter analyses are conducted to investigate the robustness of the method.", "weaknesses": "1. To improve the readability of the paper, I encourage the authors to provide a figure to illustrate the proposed method, instead of purely presenting mathematical notations and equations.\n2. What is the clustering performance of the NTK-induced affinity matrix $A_{NTK}$?\n3. The mentioned baseline TAC has a variation that trains a clustering head to further improve the performance. Currently, it seems that the authors only compare its training-free version. A more thorough comparison is expected.\n4. Ablation studies on each component of the method could be attached.\n5. What is the time and space complexity of the proposed method? Considering the computational efficiency of spectral clustering methods, can the proposed method scale to large datasets? How much time does it take to perform clustering on ImageNet-1K?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)