Clustering via Self-Supervised Diffusion
Authors: Roy Uziel
Irit Chelly
Oren Freifeld
Ari Pakman
Venue: ICML 2025 poster (2025)
Keywords: Self-supervised
Clustering
Diffusion
Primary area: general_machine_learning->clustering
TL;DR: Our method, CLUDI, leverages diffusion models to learn continuous latent embeddings for clustering, achieving SOTA unsupervised classification on diverse datasets.

Abstract: Diffusion models, widely recognized for their success in generative tasks, have not yet been applied to clustering. We introduce Clustering via Diffusion (CLUDI), a self-supervised framework that combines the generative power of diffusion models with pre-trained Vision Transformer features to achieve robust and accurate clustering. CLUDI is trained via a teacher–student paradigm: the teacher uses stochastic diffusion-based sampling to produce diverse cluster assignments, which the student refines into stable predictions. This stochasticity acts as a novel data augmentation strategy, enabling CLUDI to uncover intricate structures in high-dimensional data. Extensive evaluations on challenging datasets demonstrate that CLUDI achieves state-of-the-art performance in unsupervised classification, setting new benchmarks in clustering robustness and adaptability to complex data distributions.

Clustering via Self-Supervised Diffusion
Roy Uziel Irit Chelly 12 Oren Freifeld 123
Ari Pakman 432
Abstract
Random Noise
Diffusion models, widely recognized for their suc- in generative tasks, have not yet been applied to clustering. We introduce Clustering via Diffu- sion (CLUDI), self- supervisdd framework that combines the generative power of diffusion mod- els with pre-trained Vision Transformer features to achieve robust and accurate clustering. CLUDI is trained via a stuceer-ttudet paradigm: the teacher uses stochastic diffusion-based sampling to produce diverse cluster assignments, which the student refines into stable predictions. This stochasticity acts as a novel data augmentation strategy, enabling CLUDI to uncover intricate structures in high-dimensional data. Extensive ovaluations on challenging datasets demonstrate that CLUDI achieves state-of-the--e performance in unsupervised classification, setting new bench- marks in clustering robustness and adaptability to complex data distributions.
z z
N F3)
CLUDI'S Diffusion model Sampling
Pretrained Vision Transformer
Multiple Cluster Assignments
Averaged Cluster Assignments
Assignment
Assignment
Assignment
Assignment
Figure 1: Overview of the CLUDI Framework at Infer- ence. Images pass through a pre-trained Vision Transformer to obtain their feature representations. Multiple random vectors are sampled from Gaussian distribution. A diffu- sion model, conditioned on the features, refines the random vectors into class assignment embeddings. Each refined embedding corresponds to candidate cluster probability vector. By averaging multiple such assignments, the frame- work produces robust and accurate clustering predictions.
1. Introduction
Clustering is a fundamental task in unsupervised learning, essential for uncovering meaningful groupings within data.
These groupings play vital role in diverse downstream ap- plications, such image segmentation (Mittal et al., 2022: Friebel et al., 2022), anomaly detection (Song et al., 2021a), and bioinformatics (Karim et al., 2021). Despite significant advancements, traditional methods face significant chal- lenges, particularly in datasets with intricate structures and varying intra-class similarity, where such approaches often struggle to capture underlying patterns (Ben-David, 2018). To address these limitations, deep learning-bass based clustering
approaches have gained substantial attention for their ability to tackle complex and diverse data landscapes (Zhou et al., 2024; Ren et al., 2024; Wei et al., 2024).
However, deep learning-based clustering methods still face persistent challenges that limit their practical utility. A primary issue is model collapse (Amrani et al., 2022). where learned representations degenerate into trivial so- lutions. Another obstacle arises from underutilizing pre- trained features, resulting in suboptimal performance. No- tably, Adaloglou et al. (2023) demonstrate that using pre- trained Vision Transformers (Caron et al., 2021) can sur- pass methods that attempt to learn both representations and cluster assignments simultaneously, thereby highlighting
Department of Computer Science, Ben-Gurioa University of the Negev, Beer Sheva, Israel Data Science Research Center, Ben- Gurion University of the Negev, Beer Sheva, Israel 'The School of Brain Sciences and Cognition. Ben-Gurion University of the Negev, Beer Sheva, Israel "Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Beer Sheva, Isracl. Correspondence to: Roy Uziel <uzielr@ post. bgu.a.uu....l...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Clustering via Self-Supervised Diffusion
the effectiveness of high-quality feature initialization. Self- supervised representations have consistently outperformed supervised ones in transfer tasks (Ericsson et al., 2021), highlighting the benefits of leveraging powerful pre-trained models. Meanwhile, many clustering frameworks rely on complex augmentation pipelines, such as neighbor min- ing (Van Gansbeke et al., 2020; Adaloglou et al., 2023), or employ multiple clustering heads trained in parallel, select- ing the best- pest-performing one evaluation time (Adaloglou et al., 2023; Amrani et al., 2022). Although these strategies can improve accuracy, they introduce additional complexity and computational overhead during training.
gue, however, that their ability to model and sample from complex, high-dimensional distributions makes them partie- ularly suited for clustering high-dimensional data, such as images. By iteratively refining noisy representations, diffu- sion models can uncover underlying structure and variability in the data, providing a natural mechanism for capturing meaningful cluster assignments. Their stochastic nature further enables robust and diverse clustering predictions.
Our key contributions are as follows:
We introduce Clustering via Diffusion (CLUDI,, novel framework that is the first to leverage diffusion models for the clustering task.
To address these limitations, we introduce Clustering via Diffusion (CLUDD), a self-supervised framework that lever- ages the generative strengths of diffusion models to pro- duce robust cluster probabilities. As Figure 1 illustrates, CLUDI takes as input a pre-trained Vision Transformer fea- ture vector x E IR" and refines an initial random assignment embedding vector through an iterative diffusion process. Conditioned on x, this process evolves over multiple steps and culminates in a final assignment embedding 2 E IRd, which encodes the likelihood of each data point belonging to each of the K clusters.
We propose self-supervised training paradigm based on a teacher-student architecture, where a diffu- sion model generates stochastic and informative cluster assignments to guide the student in learning meaning- ful cluster probabilities.
We demonstrate the effectiveness of CLUDI through extensive evaluations on benchmark datasets, achiev- ing state-of-the-ar accuracy and robustness in unsuper- vised classification tasks.
During inference, CLUDI generates multiple stochastic sam- ples of Zg and aggregates the corresponding predictions. By averaging across diverse representations, this strategy mitigates uncertainty, uncovers subtle structures in high- dimensional feature spaces, and yields more stable and ac- curate cluster assignments, even in complex and diverse data scenarios.
2. Related Work
Deep learning-basee models that learn cluster are usually referred to as performing deep clustering or unsupervised classification. Cluster categories are learned during the training phase and remain fixed during inference. These methods have received much interest in the machine learn- ing community. Comprehensive reviews on this vast field can be found in (Zhou et al., 2024; Ren et al., 2024; Wei et al., 2024).
Our model is trained using self-supervised Siamese ar- chitecture comprising teacher and student branches. The teacher, implemented as a diffusion model, generates assign- ment embeddings 2o E Ed and cluster probabilities p(k x) for i E {1. 2, ........ K}. through stochastic sampling. These outputs, which capture diverse and complementary views of the data, serve as targets for the student.
End-to-end training. Most previous works learn simul- taneously feature representations and cluster categories. An early work in this area is Deep Embedded Clustering (DEC) (Xie et al., 2016), which jointly optimizes feature learning through an autoencoder and assigns clusters using Kullback-Leibler (KI.) divergence- baaeee loss. Although DEC can be effective in learning clusters, its performance is sensitive to initialization and prone to model collapse.
The student adapts its predictions to match the teacher's out- puts, enabling it to uncover meaningful and distinct clusters in the data. The training process optimizes two complemen- tary objectives: an asymmetric non-contrustive loss (Grill et al., 2020; Chen & He, 2021; Caron al., 2021) for the as- signment embedding 2o and a non-collapsing cross-entropy loss for the cluster probabilities p(k)))) To facilitate ef- fective clustering and prevent trivial solutions, the cross- entropy loss incorporates a uniform prior over the data mini- batch (Amrani et al., 2022), encouraging well-separated and diverse cluster assignments.
DeepCluster (Caron et al., 2018) takes a different approach by alternating between pseudo-label generation and feature representation refinement. The method iteratively updates cluster centroids and uses the centroids to assign pseudo- labels, which are then used to optimize the feature space. However, the computational overhead associated with gen- erating and updating pseudo-labels can be significant, espe- cially for large-scale datasets.
Why use diffusion for clustering?
Despite their success in generative tasks, diffusion models have not been explored for clustering until now, We ar-
Invariant Information Clustering (IIC) (Ji et al., 2019) of- fers a complementary method by maximizing mutual in-
2

---

Clustering via Self-Supervised Diffusion
formation between different augmentations of the same data. Among probabilistic approaches, Variational Deep Embedding (VaDE) (Jiang et al., 2016) integrates varia- tional autoencoders with Gaussian mixture models to learn probabilistic cluster assignments. Recent works leverage popular self-supervised approaches. Self-Classifier (Amrani et al., 2022) uses a Siamese network to simultaneously learn representation and cluster labels. To avoid degenerate solu- tions, it uses variant of the cross-entropy loss which we adopt in our model.
DDPMs, an initial data sample denoted by Z E Rd, is trans- formed into pure Gaussian noise, 2r ~ NO, Ia), 2r RR. through a sequence of incremental additions of Gaussian noise. This forward process is Markovian and defined by:
(2.1/1 =
-N (245 V BIze-1, BLLa;
(1)
where we introduced discrete time steps / 1... T, we assume T 1000, and 33 is predefined noise schedule. Note that the forward processes in Equation allows closed- form sampling at any timestep t. Using the notation == 3e and o == II 0tg, we have:
Training on pre-trained features. The idea of decoupling feature learning from cluster learning has been advocated by SCAN (Van Gansbeke et al., 2020) and TwoStageUC (Han et al., 2020). These models, however, lack efficiency, as they learn features from scratch for every dataset.
g(2, = -N(zs ((zsvaazo va,zo, (1 -1-60 a
(2)
The use of pre-trained features, particularly Vision Trans- formers (ViTs) (Dosovitskiy, 2020), was proposed by TSP (Zhou & Zhang, 2022) and TEMI (Ren et al., 2024). These approaches allow the model to focus on refining clus- ter assignments without relearning low-level features, sig- nificantly reducing the computational cost for large-scale datasets. Our CLUDI approach relies on similar feature ex- traction backbone, based on the DINO model (Caron et al., 2021), but proposes a more refined clustering model and training setup that leads to superior results.
DDPMs are a set of deep network models and sampling techniques which reverse this process: starting from a sam- ple ZT NO, Iz), one generates samples earlier times until sample zo from the data distribution is obtained.
In the next section we present the particular model and sampling technique we use in CLUDI. For more details on diffusion models see recent overviews (Luo, 2022; Turner et al., 2024; Chan, 2024; Nakkiran et al., 2024). We use discrete time steps I, but continuous time formulations also exist (Song et al., 2021c). Note that our use of diffusions in continuous space to generate discrete data resembles their use to generate discrete language tokens (Dieleman et al., 2022; Gao et al., 2024; Gong et al., 2022; Li et al., 2022).
DeepDPM (Ronen et al., 2022), which can be used in either end-to-end fashion or with pretrained features, is a deep clustering method that infers K and is inspired by Dirichlet process mixture sampler (Chang & Fisher III, 2013; Dinari et al., 2019). However, unlike CLUDI, it makes a stringent assumption about the distribution of the features within each cluster.
Sele-Supervision Self-supervised learning (SSL) has emerged a powerful paradigm for learning from unla- beled data. The type of SSL that we employ uses Siamese architecture (Chicco, 2021), where two views of the input produce different representations. The model is trained to ensure that these representations are informative and mu- tually predictable. A major challenge is avoiding model collapse, where the representations become mutually pre- dictive by minimizing the information about the inputs.
Amortized clustering. In another approach, called amor- tized clustering, the model does not learn fixed cluster cat- egories. Instead, it learns to organize full datasets into clusters discovered at test time. This approach allows for real-time adaptation of clusters as new data is introduced. It is thus a form of meta-leamiig (Hospedales al., 2021), and research on this task is just beginning to unfold (Pakman et al., 2020; Jurewicz et al., 2023; Wang et al., 2024; Chelly et al., 2025).
Several mechanisms have been proposed in the SSL context to avoid collapse. such as contrastive losses (Chen et al., 2020; Jaiswal et al., 2020) or clustering constraints (Caron et al., 2018; 2021). In this work we adopt the teacher- student framework (Grill et al., 2020; Chen & He, 2021; Caron et al., 2021), in which the teacher model generates labels for the data which the student learns to predict. As we will detail in Section 5, we avoid collapse via stop-gradients, a predictor layer, and strong prior assumptions. We refer the reader to recent SSL surveys (Balestriero et al., 2023; Özbulak et al., 2023; Shwartz Ziv & LeCun, 2024; Gui et al., 2024) for thorough overviews.
3. Background
Diffusion Models. Denoising diffusion probabilistic mod- els (DDPMs) are recent pivotal shift in the landscape of generative modeling (Sohl-Dickstein et al., 2015; Ho et al., 2020), with considerable success across a range of appli- cations, including image synthesis, audio generation, and molecular design (Song et al., 2021b; Dhariwal & Nichol, 2021; Nichol & Dhariwal, 2021; Rombach et al., 2022). In
3

---

Clustering via Self-Supervised Diffusion
4. Clustering via Diffusion
74
Clustering via Diffusion (CLUDI) a latent variable model for classification of the form
72
Accuracy
70
(k x) = f f dzo p( (kilzo))po
B
Σ
(3)
68
B
66
Clean Features Augmented Features
where the last term is Monte Carlo approximation obtained from B samples of po (zo x data-conditioned diffusion model that generates assignment embeddings. The clas- sification head is simple logit projection followed by a tempered softmax,
64
12
16
32
Number of Samples
Classification accuracy for clean and aug- mented inputs. The model classification accuracy on aug- mented data (feature dropout plus Gaussian noise) becomes similar to that of clean data as the number of data samples grows. Results from ImageNet 100 validation data. Stan- dard deviation based on 10 repetitions.
Figure 2:
exp( [ليج Σド= exp( (xol2
p(klzo)
K
Uge
(5)
where L E Rxd Figure illustrates the averaging over B = 4 samples of Zo. while Figure 2 shows classifica- tion accuracy as a function of B for both clean and noise- augmented data.
2g have followed (Salimans & Ho, 2022), but there exist other possibilities, such as modeling (1) the added noise in Equation
c which represents
Starting from an initial Gaussian sample
We treat the noise scale F² in (6s.(6) as hyperpa- rameter (Gao et al., 2024). Note that the above backward sampling requires choosing the values for the time steps in Equation 7, a freedom we exploit below in our training scheme.
2T NO, 1²1a),
(6)
the diffuuson model outputs Zo Rd For reverse sampling we adopt a stochastic version of the Denoising Diffusion Implicit Model (DDIM) (Song et al., 2021b), which allows sampling backwards at arbitrarily earlier times s < t by running the backward dynamics
Noise schedule. As in Li et al. (2022) we adopt the sqrt noise schedule with on cio and
=
0.0001, t 1.
(12)
po(z,|2e) NZzzi 1o (o x, s, F2t), L3))
As shown in Figure 3, this schedule accounts for the reduced sensitivity of the discrete labels to noise added near = O. leading to most of the noise in Equation 2 being introduced at lower t values. As t grows, the noise addition slows down, casing the learning process of the denoising network.
for < t, where
Z: VI
aq
Eg
(t)
Ze. x)
Mo(2;,X,s,t) x,,, = =aa
var
+ 1 ag
0² Eg
(1)
(2
Os
1 at
Noise
Cit
and we defined
Normallzed Timesteps
(1) cg (z,,))
Cz 2:
Vāa 20 (Z,,
(10)
Figure 3: sqrt noise scheduling. The noise grows faster near t 0, reflecting reduced sensitivity to noise at early timesteps, and grows gradually slower at later times.
Here (0 x) Rdd Rd is network that predicts Zo. and is trained by minimizing
5. Learning via Self-Distillation
Exx (1) [20 x,1) zol
(11)
CLUDI follows a self-supervised learning framework based on self-distillation, similar to BYOL (Grill et al., 2020), Sim- SIAM (Chen & He, 2021) and DINO (Caron et al., 2021).
Here wt) are fixed weights. This loss is a weighted varia- tional lower bound on the data log- liglliiel In modeling

---

Clustering via Self-Supervised Diffusion
Assignment Embeddings (8 x N N
(2 F9 Random Noise (8 x
Clustering Probabilitiss (B x X
Teacher Model
Lagits Projection Softmax d)
DDIM Denoising Steps {x} xo 10
(x Feature Vectors (1 XR)
Assignment Embeddings
(8
Noising Step fo T((
Narm + Scale
E
[|tt, vaj
Embeddings Projection (d I))
Feature Augmentations
Eat
Augmented Features
(B x x
L Logits Projection Softmax (K x d)
Student Model
Denoising Step
[4] - 2316-1
Assignment Embeddings (8 X d)
Clustering Probabilities (8 x N X]
Figure 4: Overview of CLUDI's training phase. Given a set of image features x, the teacher model generates denoised assignment embeddings žo which are used to create two targets for the student: (i) clustering probabilities and (ii) assignment embeddings zo, obtained from via predictor layer. The student network aims to predict both targets based on version of x corrupted by feature dropout plus Gaussian noise. Components whose parameters are updated via gradient-based optimization are marked with the V symbol.
In this setup, two versions of the network process different representations of the same data x, and one network predicts the output of the other. We adopt the SimSIAM approach, where the teacher and student networks share weights, like BYOL and DINO, where the teacher is updated via exponential moving average of the student.
for the teacher, the student, and the loss function.
5.1. Teacher model
The teacher generates cluster probabilities u and assign- ment embeddings, Zu which serve as training targets for the student. receives N input feature vectors x² E IR", E [], and runs the denoising algorithm B times for each of them, yielding B N denoised embeddings bbs E Rd. The time schedule of the teacher denoising is chosen to contain 25 equally-spaced timesteps from = T = 1000 to f = 0, offering balance between ef- ficient denoising and capturing essential cluster characteris- tics. Acting on 2 with and softmax (see Equation 4- Equation 5), yields N probability targets u R for the student.
key distinction of CLUDI that, for each data point x, the student has two different learning targets: the denoised assignment embedding Zg and the classification probabili- ties uk (Equation 5). Each target requires different strategies avoid trivial solutions. For the embeddings 2o, gradients applied only to the student while keeping the teacher fixed, and projection and normalization layer is added to structure the teacher's output into a useful learning tar- get (Grill et al., 2020; Chen & He, 2021; Caron et al., 2021); These modifications prevent representational collapse and provide informative embeddings, though their precise effect remains an active of research (Tian et al., 2021; Wang et al., 2021; Liu al., 2022; Tao et al., 2022; Richemond et al., 2023; Halvagal et al., 2023).
Empirically, however, the learning is less effective when the student's 's targets directly the denoised embeddings abi Instead, we map the probabilities u back to the embedding space by means of an embedding matrix E E R and then normalize and scale the target embeddings as
To avoid collapse when learning the softmax cluster prob- abilities uy. we enforce a uniform prior over mini-batches that regularizes the cross-entropy loss (Amrani et al., 2022).
b,
b, = Vd Z0
(13)
E
Figure 4 summarizes the sequence of teacher and student operations from the input data into the loss function. In the following section, we provide more details on our choices
The role of E is akin to the predictor network in BYOL or SimSIAM, which assists in generating effective repre- sentations during training. Once the model is fully trained, however, no longer required. Given the above embed-
5

---

Clustering via Self-Supervised Diffusion
dings, the teacher class probabilities are
as defined Equation 14 and Equation 17,
La
p(klz0 h.i)
u
bi
pk(1220)) = b.
(19)
exp(
b,s u
(14) Naively using a cross entropy loss,
Σラ= K exp( [
((20 bi bid)
Σ p(klzz))) + log log
(20)
5.2. Student model
quickly degenerates to a solution that puts all the proba- bilityy single category. We regularize this loss using idea from Amrani et al. (2022), which amounts to treat- ing the indices (b, i) as random variables with a uniform prior p(zl)") 1/(N B) and distribution conditioned on k given by a column softmax
The student network aims to predict both targets in Equa- tion 13 and Equation 14. Since the input x consists of pre- trained features, we use an abstract augmentation strategy. For each data feature vector x' E RD, E NJ. we create B augmented versions x b E [1, B), each obtained by first zeroing the components of x' with probability 0.2, and then adding zero-mean Gaussian noise, with variance o² sampled uniformly in [0.1, 0.3). We present the student with noisy versions of the teacher's outputs in Equation 13,
Lzg
exp(
(21)
B.N 2
LZg
exp(
Tool
with its own temperature Tool. Assuming also a uniform class prior pkk) = 1/K, Bayes rule and the law of total probability imply
b. 2
N (a b. (1 āc,)) I
(15)
where to is sampled uniformly from [0, T.. We denote the student predictions for the denoised assignment embedding and class probabilities as
p(klz)))) = p(zn bi 1))((())))))))))))))))))))))11
p(zp ((Z))(kk)
(22)
p(zz") (it)
E P p(z)
p(k12b01) p(k))(k(((""))
(NB/ K (((())))))))
20 = 2o (zi," xbi tt) ;tb)
(16)
p(k)
B.N P (3
(23)
b exp( [La
(17) Inserting these expressions in Equation 20 leads to a loss which was shown in (Amrani et al., 2022) not to admit collapsed distributions as optimal solutions. In practice, we use a symmetric version of Equation 20,
K Σラ= exp(
[T "b
5.3. Augmented views
(b, = (e(zh 20
) + l bit , 2b()).
Having presented the teacher and student models, we note that their different views of the data and assignment embed- dings originate from (i) the stochastic nature of the teacher denoising, which starts with pure noise in Equation 6 and performs 25 sampling steps, (ii) the augmentation of the student feature vector (feature dropout + Gaussian noise) and (iii) the noise added in Equation 15 to the embedding that the student is required to denoise. We remark that un- like common augmentation strategies acting on raw images (cropping, color alterations, etc), our augmentations act di- rectly on the data features or their assignment embeddings. In partieular, the views generated by (i) and (iii) exploit the intrinsic randomness of the diffusion model.
(24)
Weighting the minibatch elements. Addressing the vary- ing difficulty of the predictions of
z n based on the sampled timestep ts, we incorporate the Min-SNR-y (Hang et al., 2023) loss weighting strategy, which treats each timestep's denoising task as distinct and assigns weights based on their difficulty:
att City
SNR,
(25)
max maaR 7)
W/b
(26)
SNR,, +1
where 7 is predefined threshold set to 5, enhancing the stability and ensuring that no single noise level dominates during training. The overall loss function, which incorpo- rates these weights, is structured as follows:
5.4. Loss functions
Embeddings. We employ an MSE loss between the teacher denoised embeddings zo bi from Equation 13 and the student- student-predicted predicted embeddings zb from Equation 16:
B.N
L =
Σ wbb (Carsle bi1=1
to 2001) + (bcc(). (27)
BN
b. 20 = 1120 b.s 20 11.. lais (Zo
(18)
This formulation enables a nuanced control over the learning process, adapting to the challenges posed by different levels of noise.
Class probabilities. We use here a more explicit notation for the teacher and student class probabilities, respectively,
6

---

Clustering via Self-Supervised Diffusion
Vending Machine
Dipper
Vish Setter
Cosboy Hat
Neck Brace
Box Turtie
Schtoperic
Bletric Guitar
Obalisk
Threshing Machine
Dlamondback Snake
Collic
File Cablret
Oboz
Tobazco Shop
Scorpion
Coyote
Heir
Mourtain Lion
Hetchet
Wocooworking
Vending
Water
Soda
Washing Machine
Alaatross
Guanon
Nanhole Cover
Purse
Wreck
Cain lemeer
Probescis Monkey
Maze
Shoe Shop
molcamole
Giant Schnauzee uzer
"resh din
Nicrophone
Shower Curtain
Hashed Potato
Flyt-Coated Retriever
Broom
Steel Bridge
Box Turtle
Maze
Banana
Figure 5: Left: t-SNE visualization of the assignment embedding space of ImageNet 50 demonstrating the model's ability to organize data points into well-separated clusters. Right: Examples of correctly classified images
6. Experiments
from DINO (Caron et al., 2021) based on Vision Trans- formers (VIT-S/16 and ViT-B/16) pre-trained on ImageNet. We present comparisons with four leading self-supervised clustering models: SCAN (Van Gansbeke et al., 2020), Pro- pos (Huang et al., 2022), TSP (Zhou & Zhang, 2022) and TEMI (Adaloglou et al., 2023). We implement variant of Self-Classifier (Amrani et al., 2022), denoted as Self- Classifier*, in which the feature extractor is frozen and only the classification heads are optimized. This setup ensures that CLUDI and Seff-Classifie** share identical feature rep- resentations, enabling
Datasets. We evaluate CLUDI on comprehensive suite of benchmark datasets to rigorously assess its scalability, adaptability, and clustering performance. The datasets in- clude subsets of ImageNet (Deng et al., 2009), Oxford-IIIT Pets (Parkhi al., 2012) (with 32), Oxford 102 Flower (Nilsback & Zisserman, 2008), Caltech 101 (Fei-Fei al., 2004), CIFAR-10 (Krizhevsky et al., 2009), and STL-10 (Coates et al., 20.1). Each dataset introduces unique chal- lenges: the ImageNet subsets cover broad and diverse cate- gories, providing robust test of scalability (see Table 2); Oxford-IIIT Pets and Oxford 102 Flower focus on fine- grained distinctions that assess CLUDI's precision; Caltech 101 evaluates generalization across diverse object types; and CIFAR-10 and STL-10 offer additional complex data that further validate CLUDI's ability to handle intricate cluster- ing tasks (Table 1).
direct and fair comparison.
All CLUDI results were obtained by running the embed- ding denoising for 100 equally spaced time steps. Figure 6 presents the ablation the classification-loss weight X, while Figure 7 illustrates the results scanning different values the embedding dimension in Equation 27. For additional implementation details and ablations, see our Appendix.
Evaluation Metrics. We use three popular metrics (Fahad al., 2014): (i) Normalized Mutual Information (NMI) quantifies shared information between predicted and ground- truth clusters; (ii) Clustering Accuracy (ACC) measures the alignment of predictions with true labels; (iii) Adjusted Rand Index (ARI) adjusts for chance, providing a robust measure of similarity between predicted and true clusters.
Results. CLUDI achieves state-of-the-art performance across all tested datasets, consistently outperforming previ- ous approaches on clustering tasks of varying complexity. As detailed in Table and Table 2, CLUDI significantly surpasses established baselines in NMI, ACC, and ARI, es- pecially on ImageNet subsets, showcasing its robustness in both general and fine- ime-inrieed clustering scenarios.
Experimental Setup. For all experiments, we use features

---

Clustering via Self-Supervised Diffusion
Table I: Clustering performances on smaller datasets. The results for SCAN, Propos, TSP, and TEMI on CIFAR-10 10 and STL-10 are from the original papers, except for TEMI (ViT-S/16), which we trained using the official code. For the Oxford and Caltech datasets, trained all models. The best result is shown in bold, the second best is underlined.
Table 2: Clustering performances on ImageNet subsets.
The results for SCAN, Propos and TEMI are from the origi- nal papers. 'he best result is shown bold, the second best is underlined.
Methods
NMI (%)
ACC (%)
ARI (%)
ImageNet 50
Methods
NMI (6)
ACC (%)
ARI (%)
SCAN (Resnet50)
82.2
76.8
66.1
CIFAR
Propos (Resnet50)
82.8
69.1
SCAN (Resnet50)
79.7
88.3
77.2
TEMI (ViT-S/16)
84.2
77.8
68.4
Propos (Resnet18)
88.6
94.3
88.4
TEMI (VIT-B/16)
86.
80.1
71.0
TSP (VIT-S/16)
84.7
92.1
83.8
Self SaffCClassifie* (ViT-S/16)
87.1
76.3
71.1
TSP (VIT-B/16)
88.0
94.0
Self-Classifier* (ViT-B/16)
77.1
73.2
TEMI (VIT-S/16)
85.4
92.7
84.8
Ours (VIT-S/16)
90.1
81.3
75.8
TEMI (VIT-B/16)
88.6
94.5
88.5
Ours (ViT-B/16)
91.2
82.1
76.2
Self-Classifier (VIT-S/16)
83.8
91.2
82.1
ImageNet 100
Self-Classsfie** (VIT-B/16)
84.2
89.0
SCAN (Resnet50)
80.8
68.9
57.6
Ours (VIT-S/16)
88.0
94.2
87.7
Propos (Resnet50)
83.5
63.5
Ours (ViT-B/16)
89.6
95,3
89.8
TEMI (ViT-S/16)
72.5
62.3
TEMI (VIT-B/16)
85.6
75.0
65,44
SCAN (Resnet50)
69.8
80.9
64.6
Self- Solf-Classifier* (VIT-S/16)
83.9
68.9
61.8
Propos (Resnet18)
75.8
86.7
73.7
Self-Classifier (VIT-B/16)
86.2
71.4
64.3
TSP (ViT-S/16)
94.1
97.0
93.8
Ours (ViT-S/16)
74.3
67.5
TSP (VIT-B/16)
95.8
97.9
95.6
Ours (VIT-B/16)
87.1
76.6
67.8
TEMI (VIT-S/16)
85.0
88.8
80.1
ImageNet 200
TEMI (VIT-B/16)
96.5
98.5
96.8
SCAN (Resnet50)
77.2
58.1
47.0
Self-Classifier* (VIT-S/16)
90.5
83.1
82.4
Propos (Resnet50)
80.6
53.8
Self-Classifier* (ViT-B/16)
91.5
87.7
85.7
TEMI (VIT-S/16)
71.9
59.8
Ours (VIT-S/16)
95.7
98.2
96.1
TEMI (VIT-B/16)
85.2
73.1
62.1
Ours (ViT-R/16)
96.8
98.7
97.1
Self-Classifier" (VIT-S/16)
80.5
54.5
46.7
Oxford-I1IT Pets
Self-Classiferr (ViT-B/16)
78.3
53.1
44.1
TEMI (VIT-S/16)
69.7
49.3
41.0
Ours (ViT-S/16)
73.2
60.9
TEMI (VIT-B/16)
71.1
47.0
41.7
Ours (VIT-B/16)
86.1
73.7
63.2
Classsassiffer* (VIT-S/16)
82.7
67.5
59.2
Self-Classifier* (VIT-B/16)
83.5
68.2
63.0
Ours (VIT-S/16)
87.3
74.1
71.6
Ours (VIT-B/16)
86.7
73.8
71.1
The effectiveness of the CLUDI model in- fluenced by the choice of the diffusion parameter F² and the embedding dimensionality d, both of which play critical
Limitations.
Oxford 102 Flower
TEMI (VIT-S/16)
50.1
26.0
14.2
TEMI (VIT-B/16)
50.2
25.9
16.9
Luf-Classifier* (VIT-S/16)
51.5
Self-Classifier* (VIT-B/16)
72.5
57.8
42.9
Ours (ViT-S/16)
76.1
62.2
52.6
0.740
Ours (ViT-B/16)
81.5
69.7
61.8
0.735
Caltech 101
0.730
TEMI (VIT-S/16)
78.9
50.2
35.6
Max ACC
TEMI (VIT-B/16)
80.4
51.4
0.725
Soff-Classifie** (VIT-S/16)
82.5
56.1
59.4
Self-Classifier* (VIT-B/16)
83.5
58.2
61.2
0.720
Ours (VIT-S/16)
86.5
66.7
65.7
0.715
Ours (VIT-B/16)
87.9
68.1
66.3
0.710
0.705
Qualitative Analysis. A t-SNE plot (Figure 5) of CLUDI's embeddings on ImageNet-50 demonstrates its ability to form well-separated clusters, reinforcing its quantitative metrics and illustrating its effectiveness organizing complex data structures into distinct clusters. This visualization highlights CLUDD's capacity to capture subtle inter-class variations.
20
40
60
80
100
Lets weight À
Figure 6: Ablation Study on the Lets weight X. Each point in the curves shows the maximum validation accuracy on ImageNet- 100 achieved during training.
8

---

Clustering via Self-Supervised Diffusion
Acknowledgments
0.74
This work was supported in part by the Lynn and William Frankel Center BGU CS, by Israel Science Foundation Personal Grant #360/21, and by the Isracli Council for Higher Education (CHE) via the Data Science Research Center at BGU. A.P. was supported by the Israel Science Foundation (grant No. 1138/23). I.C. was also funded in part by the Kreitman School of Advanced Graduate Suuiies, by BGU's Hi-Tech Scholarship, and by the Israel's 's Ministry of Technology and Science Aloni Scholarship.
0.72
Max ACC
0.70
0.68
0.66
0.64
20
40
60
80
100
120
Embedding dimension d
Impact Statement
Figure 7: Embedding dimension selection d. Each point in the curves shows the maximum validation accuracy ImageNet- 100 achieved during training.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
roles in determining clustering quality. While these pa- rameters were tuned to optimize performance on the tested datasets, further refinement may be necessary for new data distributions or specific clustering tasks. Additionally, al- though CLUDI demonstrates strong capability in generating be impacted
References
Adaloglou, N., Michels, F,, Kalisch, H., and Kollmann, M. Exploring the limits of deep image clustering us- ing pretrained models. In 34th British Machine Vision Conference, 2023.
well-separated clusters, its performance when scaling to large number of clusters, maintain- ing high-quality, distinct embeddings becomes increasingly complex with higher cluster counts.
Amrani, E.,, Karlinsky, L., and Bronstein, A. Self-supervised classification network. In European Conference Com- puter Vision, pp. 116-132. Springer, 2022.
7. Conclusion
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A. G., Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., and Goldblum, M. A cookbook self-supervised learning. 2023.
In this work introduced novel use of diffusion mod- els generate clustering embeddings of pretrained data features. Our experimental results underscore CLUDDIs ad- vantages over both traditional and contemporary clustering techniques, validating its robustness, flexibility, and supe- rior self-supervised clustering performance across diverse datasets and visual challenges. Future studies could build upon this work by investigating adaptive data-driven hyperparameter selection techniques, well advanced clustering frameworks, such hierarchical or multi-scale methods, potentially more scalable to large K settings.
Ben-Davie S. Clustering whaat both theoreticians and prac- titioners are doing wrong. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning visual features. Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 2021.
Chan, S. H. Tutorial on diffusion models for imaging and vision. arXiv preprint arXiv:2403. 18103, 2024.
Chang, and Fisher III, W. Parallel sampling of dp mixture models using sub-cluster splits. Advances in Neural Information Processing Systems, 26, 2013.

---

Clustering via Self-Supervised Diffusion
Chelly, I., Uziel, R., Freifeld, O., and Pakman, A. Consistent amortized clustering via generative flow networks. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178-178. IEEE, 2004.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Friebel, A., Johann, T., Drasdo, D., and Hoehme, S. Guided interactive image segmentation using machine learning and color-based image set clustering. Bioinformatics, 38 (19):462-44688, 2022.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 15750015758, 2021.
Gao, Z., Guo, J., Tan, X., Zhu, Y., Zhang, F., Bian, J., and Xu, L. Empowering diffusion models on the embedding space for text generation. In Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 4664-4683. 2024.
Chicco, D. Siamese neural networks: An overview. Artifi- cial neural networks, pp. 73-94, 2021.
Coates, A., Ng. A., and Lee, H. An analysis of single- layer networks in unsupervised feature learning. In Pro- ceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223, JMLR Workshop and Conference Proceedings, 2011.
Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. Diffuseq: Sequence to sequence text generation with diffusion mod- cls. arXiv preprint arXiv:2210.088)
2022.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo. Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A scage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., and Tao, D. A survey self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., et al. Continuous diffusion for categorical data, arXiv preprint arXiv:2211.1500 2022.
Halvagal, M. S., Laborieux, A., and Zenke, Implicit variance regularization in non-contrastive ssl. Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, pp. 33409-63436, 2023.
Dinari, O., Yu, A., Freifeld, O., and Fisher, J. Distributed mome inference in dirichlet process mixture models using julia. In 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). pp- 518 525. IEEE, 2019.
Han, S., Park, S., Park, S., Kim, S., and Cha, M. Mitigating embedding and class assignment mismatch in unsuper- vised image classification. In European Conference on Computer Vision, pp.
Springer, 2020.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.I 1929, 2020.
Hang. T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., Geng, X., and Guo, B. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 7441-7451, October 2023.
Ericsson, I.., Gouk, H., and Hospedales, T. M. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5414-5423, 2021.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6857 2020.
Fahad, A., Alshatri, N., Tari, Z., Alamri, A., Khalil, I., Zomaya, A. Y., Foufou, S., and Bouras, A. A survey of clustering algorithms for big data: Taxonomy and empirical analysis. IEEE transactions on emerging topics in computing, 2(3):267-279) 2014.
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5169, 2021.
10

---

Clustering via Self-Supervised Diffusion
Huang, Z., Chen, J., Zhang, J., and Shan, H. Learning representation for clustering via prototype scattering and positive sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Jaiswal, A., Babu, A. R., Zadeh, M. Z.., Banerjee, D., and Makedon, F. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Özbulak, U., Lee, J., Boga, B., Anzaku, E. T., Park, H., Van Messem, A., De Neve, W., and Vankerschaver, J. Know your self-supervised learning: survey image- based generative and discriminative training. TRANSAC- TIONS ON MACHINE LEARNING RESEARCH, 2023.
Ji, X., Henriques, J. F., and Vedaldi, A. Invariant informa- tion clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 9865-9874, 2019.
Pakman, A., Wang. Y., Mitelut, C., Lee, J., and Paninski, L. Neural clustering processes. In International Conference on Machine Learning, pp. 7455-7465. PMLR, 2020.
Jiang. Z., Zheng, Y., Tan, H., Tang, B., and Zhou, H. Varia- tional deep embedding: An unsupervised and generative approach to clustering, arXiv preprint arXiv:1611.05144 2016.
Parkhi, M., Vedaldi, A., Zisserman, A., and Jawahar, Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. -505-. IEEE, 2012.
Jurewicz, M. M., Taylor, G. W., and Derczynski, L. The catalog problem: clustering and ordering variable-sized sets. In International Conference on Machine Learning, pp. 15528 15545. PMLR, 2023.
Ren, Y., Pu, J., Yang, Z., Xu, J., Li, G., Pu, X., Philip, S. Y., and He, L. Deep clustering: A comprehensive survey. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Karim, M. R., Beyan, O., Zappa, A., Costa, I. G., Rebholz- Schuhmann, D., Cochez, M., and Decker, S. Deep learning-basee based clustering approaches for bioinformatics. Briefings in bioinformatics, 22(1):393-4155 2021.
Richemond, P. H., Tam, A., Tang, Y., Strub, F., Piot, B., and Hill, F. The edge of orthogonality: simple view of what makes byol tick. In International Conference on Machine Learning, pp. 29063-29081. PMLR, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CV con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Li, X.. Thickstun, J., Gulrajami, I., Liang P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 4343, 2022.
Ronen, M., Finder, S. E., and Freifeld, O. DeepDPM: Deep clustering with an unknown number of clusters. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 9861-9870, 2022.
Liu, K.-J., Suganuma, M., and Okatani, T. Bridging the gap from asymmetry tricks to decorrelation principles in non- contrastive self-supervised learning. Advances in Neural Information Processing Systems, 35:19824-11 19835, 2022.
Salimans, T. and Ho, J. Progressive distillation for fast sam- pling of diffusion models. In International Conference Learning Representations, 2022.
Luo, C. Understanding diffusion models: A unified perspec- tive. arXiv preprint arXiv:2208.1 1970, 2022.
Mittal, H., Pandey, A. C., Saraswat, M., Kumar, S., Pal, R., and Modwel, G. A comprehensive survey of image segmentation: clustering methods, performance param- eters, and benchmark datasets. Multimedia Tools and Applications, pp. 1-26, 2022.
Shwartz Ziv, R. and LeCun, Y. To compress or not to compress sumpress--sef--suppeeee self-supervised learning and information the- ory: review. Entropy. 26(3):252, 2024.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Nakkiran, P., Bradley, A., Zhou, H., and Advani, M. Step- by-step diffusion: An elementary tutorial. arXiv preprint (:Xiv:2406.08929, 2024.
Song, H., Li, P.,, and Liu, H. Deep clustering based fair out- lier detection. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 1481-1489, 2021a.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 1162-8171. PMLR, 2021.

---

Clustering via Self-Supervised Diffusion
Song, J., Meng, C.,, and Ermon, S. Denoising diffusion implicit models. In International Conference Learning Representations, 2021b.
Song, Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021c.
Tao, C., Wang, H., Zhu, X., Dong, J., Song, S., Huang, G., and Dai, J. Exploring the equivalence of siamese self- supervised learning via a unified gradient framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14431-14440, 2022.
Tian, Y., Chen, X., and Ganguli, S. Understanding self- supervised learning dynamics without contrastive pairs. In International Conference Machine Learning, pp. 10268-10278. PMLR, 2021.
Turner, R. E., Diacomu, C.-D., Markou, S., Shysheya, A., Foong, A. Y., and Mlodozeniec, Denoising diffusion probabilistic models in six simple steps. arXiv preprint arXiv:2402.04388 2024.
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proes- mans, M., and Van Gool, L. SCAN: Learning classify images without labels. In European conference puler vision, pp. 268-285 Springer, 2020.
Wang, X., Chen, X., Du, S. S., and Tian, Towards de- mystifying representation learning with non con-contrastive self-supervision. arXiv preprint arXiv:21 1000221/00000 10.04947, 2021.
Wang, Y., Lee, Y., Basu, P., Lee, J., Teh, Y. W., Paninski, L., and Pakman, A. Amortized probabilistic detection of communities in graphs. Structured Probabilistic Infer- ence & Generative Modeling workshop ICML,, 2024.
Wei, X., Zhang, Z., Huang, H., and Zhou, Y. An overview on deep clustering. Neurocomputing, pp. 127761, 2024.
Xie, J., Girshick, R., and Farhadi, A. Unsupervised deep embedding for clustering analysis. In International con- ference on machine learning, pp. 478 487. PMLR, 2016.
Zhou, S., Xu, H., Zheng, Z., Chen, J., Li, Z., Bu, J., Wu, J., Wang, X., Zhu, W., and Ester, M. A comprehensive survey on deep clustering: Taxonomy, challenges, and future directions. ACM Computing Surveys, 2024.
Zhou, X. and Zhang, N. L. Deep clustering with fea- tures from self-supervised pretraining. arXiv preprint 13X1: 2022.
12

---

Clustering via Self-Supervised Diffusion
In this Appendix we present the results of several hyper- parameter ablations. All the curves shown correspond to performance metrics evaluated on the validation set of Im- ageNet 100, We also show some clustering examples on ImageNet 50.
C. Ablation Study on Rescaling Factor F2 and Embedding Dimension
In Figure S2 and Figure S3 we explore the effect on cluster- in performance of varying the rescaling factor F and the embedding dimension d.
A. Hyperparameters
The noise rescaling factor F² plays crucial role in our diffusion model by modulating the noise variance during the forward process. This parameter directly influences the model's ability balance stability and exploration, which are essential for effective clustering. Insufficient noise (low F²) results in trivial solutions where embeddings fail to explore the latent space adequately. Conversely, excessive noise (high F²) destabilizes the learning process, disrupting inter-cluster separability.
The model requires three hyperparameters: the embedding dimension d, the noise rescaling factor F2 in Equation 6 and the coefficient Ả on the loss in Equation 27. A systematic scan on the validation set of ImageNet 100 yielded the optimal values d = 64, F2 = 25 and = 50, which we adopted for all the datasets with K 2 100. For datasets with fewer clusters (ImageNer 50, Oxford-IIIT Pets, STL 10, CIFAR 10) used smaller embedding 32.
Our results, consistent with those in (Gao al., 2024) for text generation, confirm the need find an optimal value (for us F 25.0) that strikes balance between enabling the model to avoid degeneracy and preserving cluster co- herence. In practical terms, our findings highlight that 1²2 serves as key hyperparameter for fine-tuning clustering models, with significant impacts on metrics such as NMI, ACC, and ARI.
B. Ablation Study on Lets weight À
In Figure SI we show the impact of varying the weight À associated with the classification loss Lete on the overall clustering performance.
0.872
D. Clustering Visualization on ImageNet-50
0.870
Max NMI
In Figure S4 we provide a detailed visualization of the clus- tering results on the ImageNet-50 dataset. For ten different categories, we show images with highest and lowest con- fidence of belonging to each category. This illustrates the strength of model and allows to visually explore the extent to which misclassifications are understandable.
0.868
0.866
0.864
20
40
60
80
100
Ceb weight À
0.675
0.670
Max ARI
0.665
0.660
0.655
0.650
20
40
60
80
100
Lets weight
À
Figure S1: Clustering performance metrics (NMI, ACC, ARI) across values for embedding dimension = 64.
Each image corresponds to specific metric: (a) NMI, (b) ACC, and (c) ARI.
13

---

Clustering via Self-Supervised Diffusion
0.60
0.66
0.835
Max NAI
Max ACC
0.64
Max ARI
0.825
0.56
0.62
0.54
0.815
0.60
0.805
0.58
0.50
40
60
80
100
120
20
40
60
80
100
120
Embedding dimension
Embedding dimension
Embedding dimension
NMI for F 4.0
ACC for F² = 4.0
ARI for F = 4.0
0.66
0.86
0.72
0.64
0.70
Max NM1
0.85
Max ACC
Max ARI
0.62
0.68
0.60
0.84
0.66
0.58
0.84
0.56
0.62
20
40
60
60
100
120
20
40
60
80
100
120
20
40
60
60
100
120
Embedding dimension
Embedding dimension
Embedding dimension
NMI for F = 9.0
ACC for F = 9.0
ARI for F
9.0
0.64
0.860
0.63
0.62
NMI
0.850
0.61
0.845
0.60
0.840
0.58
40
60
80
20 40
60
60
100
120
20
40
60
80
100
120
Embedding dimension
Embedding dimension d
Embedding dimension d
NMI for F2 16.0
ACC for E
16.0
ARI for F 16.0
Figure S2: Clustering performance metrics for rescaling factors F2 = 4.0, 9.0. 16.0 (top. middle, bottom, respectively) as function of the embedding dimension d. Each column shows curves for different clustering metric (NMI, ACC, and ARI).
14

---

Clustering via Self-Supervised Diffusion
0.870
0.66
0.72
Max NMI
0.860
Max ACC
0.70
Max ARI
0.855
0.62
0.60
0.66
0.84
20
40
60
80
100
120
20
40
60
80
100
120
Embedding dimension
Embedding dimension
Embedding dimension
NMI for F² 25.0
ACC for F2 = 25.0
ARI for F = 25.0
0.870
0.74
0.66
0.865
0.72
0.64
Max NMI
Max ACC
0.70
Max ARI
0.855
0.62
0.850
0.68
0.60
0.845
0.66
0.840
0.64
20
40
60
100
120
20
40
60
80
100
120
20
40
60
60
100
120
Embedding dimension
Embedding dimension
Embedding dimension
NMI for 1 = 36.0
ACC for F2
36.0
ARI for F
36.0
0.870
0.66
0.72
0.865
0.860
0.70
NMI
0.855
0.68
0.62
0.850
0.66
0.60
0.845
0.58
0.840
0.835
0.56
20
40
60
80
100
120
20
40
80
100
120
20
40
60
80
100
120
Embedding dimension
Embedding dimension d
Embedding dimension
NMI for 12 = 49.0
ACC for F2
49.0
ARI for F2 49.0
Figure S3: Clustering performance metrics for rescaling factors F² 25.0, 49.0 (top, middle, bottom, respectively) function of the embedding dimension d. Each column shows curves for different clustering metric (NMI, ACC, and ARI).
15

---

Clustering via Self-Supervised Diffusion
Banana
Broom
Dipper
File Cabinet
Hair Dryer
Manhole Cover
Schipperke
Shoe Shop
Unicycle
Soda Bottle
Dile
Figure S4: Visualization supervised clustering results on the ImageNet dataset. Each row corresponds to a single class, with the class name displayed on the left. For each class, the first four images represent the correctly classified samples with the highest confidence (outlined in green), while the last two images represent the incorrectly classified samples with the lowest confidence (outlined in red).
16


Reviews:
[{"review_id": "rscHMeVull", "reviewer": "Reviewer_o743", "summary": "- This paper presents CLUDI, a framework that combines pre-trained Vision Transformer (ViT) features with diffusion models for clustering tasks. \n   - While leveraging ViT for feature extraction and using diffusion models to enhance performance might offer some improvements, the significance of this approach could be questioned, \n   - as the core feature extraction is already handled by ViT, \n   - and the role of diffusion models in further boosting clustering performance remains to be fully justified.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions.\n\n**Non-incremental nature of the results:** The experimental results show that ViT features cannot by themselves explain the success of CLUDI, our model. CLUDI's advantage is evident in its superior test metrics across all models and datasets (including ImageNet subsets) we explored. Note in particular the absolute increase in accuracy of around 10% (Caltech 101 and Flower datasets) or 5.9% (Pets dataset) compared to the best of several alternative models trained on the same features, which can hardly be described as incremental.\n\n**On fairness of comparisons:** Since the core of CLUDI is a novel diffusion-based classification head (given pre-learned features), it seems that adopting the same pre-learned features across different learned classification heads is only fair for comparisons.\n\n**A novel use for diffusion models:** Although diffusion models were originally developed to generate realistic data for different modalities (images, audio, video, etc), the innovation of CLUDI resides precisely in showing that diffusion models excel in the novel task of learning to generate cluster embeddings.\n\n**Figures:** We have made special efforts to make all the figures clear and informative. Of course, we will gladly incorporate any suggestion.\n\n**The contribution of the diffusion model:** Unlike all previous self-supervised models for classification, CLUDI relies on a probabilistic formulation based on the marginalization of latent diffusion variables. This probabilistic component is the main contribution of the diffusion model and arguably the reason for CLUDI's robustness (see Figure 2) and state-of-the-art performance (see Tables 1 and 2).", "claims_and_evidence": "- Most of the claims are supported clearly.", "methods_and_evaluation_criteria": "- All of the experiments were well done.", "theoretical_claims": "- Most of the techniques are existing, so there is no need to further check their theoretical claims.", "experimental_designs_or_analyses": "- All of the experiments were well done.", "relation_to_broader_scientific_literature": "- DDPM (Denoising Diffusion Probabilistic Models) / VPSDE\n- Contrastive learning without negative pairs\n- Consistency regularization", "essential_references_not_discussed": "NAN", "other_strengths_and_weaknesses": "- Strengths \n   - Utilizes diffusion models for deep clustering.\n- Weaknesses\n  - The experimental results outperform the self-classifier, which I believe makes the comparison unfair. Alternatively, if the performance of the self-classifier is largely due to the ViT features, then this paper only presents some incremental experiments.\n   - Figures 1 and 4 do not effectively convey the motivation; they need to be improved to better showcase the core idea.", "other_comments_or_suggestions": "- Figures 1 and 4 do not effectively convey the motivation; they need to be improved to better showcase the core idea.", "questions_for_authors": "- Diffusion models are excellent generative algorithms, typically used for generative tasks. While the creative combination of A+B in discriminator tasks is interesting, shouldn't generative models primarily focus on generation tasks?\n- The core contribution to the clustering performance seems to stem from the ViT feature rather than the diffusion model. Can you tell me the   contribution of diffusion model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5JV4wYEgs", "reviewer": "Reviewer_Qpdh", "summary": "This paper proposes Clustering via Diffusion (CLUDI), a method using diffusion models to cluster unlabeled image data. The authors take pre-trained ViT features as input, then learn a diffusion-based generative process that refines random noise into “assignment embeddings.” A classification head maps these embeddings to cluster-probability vectors. During inference, the model samples multiple such embeddings per image and averages their predicted cluster probabilities, which is shown to yield robust performance. The empirical results on CIFAR-10, STL-10, several ImageNet subsets (50, 100, 200 classes), etc., show that CLUDI outperforms a range of SOTA clustering baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the comments and questions.\n\n\n**Number of diffusion steps at inference:**\nOur experiments show that increasing the number of DDIM diffusion steps at inference reduces the variance of the accuracy across independent runs of the diffusion model. Moreover, for small latent samples $B$, the mean accuracy increases with more diffusion steps. To illustrate this, we present below mean accuracy and standard deviation (based on 10 independent runs) for the ImageNet 100 validation set, for different values of samples $B$ and DDIM diffusion steps.\n\n\n| Number of samples B | DDIM steps |  Mean accuracy    |   Std   |\n|---------|------------|----------|---------|\n| 1       | 5          | 0.7390   | 0.0031  | 0.7348  | 0.7420  |\n| 1       | 25         | 0.7403   | 0.0024  | 0.7385  | 0.7443  |\n| 1       | 50         | 0.7407   | 0.0018  | 0.7398  | 0.7451  |\n| 1       | 100        | 0.7409   | 0.0015  | 0.7400  | 0.7452  |\n|---------------------------------------\n| 2       | 5          | 0.7394   | 0.0028  | 0.7357  | 0.7425  |\n| 2       | 25         | 0.7406   | 0.0019  | 0.7386  | 0.7446  |\n| 2       | 50         | 0.7410   | 0.0015  | 0.7402  | 0.7453  |\n| 2       | 100        | 0.7411   | 0.0014  | 0.7403  | 0.7454  |\n|---------------------------------------\n| 4       | 5          | 0.7396   | 0.0026  | 0.7355  | 0.7427  |\n| 4       | 25         | 0.7407   | 0.0018  | 0.7388  | 0.7448  |\n| 4       | 50         | 0.7410   | 0.0015  | 0.7399  | 0.7453  |\n| 4       | 100        | 0.7412   | 0.0011  | 0.7404  | 0.7455  |\n|--------------------------------------\n| 8       | 5          | 0.7447   | 0.0014  | 0.7437  | 0.7452  |\n| 8       | 25         | 0.7452   | 0.0007  | 0.7450  | 0.7454  |\n| 8       | 50         | 0.7455   | 0.0002  | 0.7452  | 0.7458  |\n| 8       | 100        | 0.7456   | 0.0001  | 0.7454  | 0.7458  |\n\n\n\\\nFor our test results, we chose $B=32$ and 100 DDIM steps, as we did not observe further improvements in the validation results for higher values.  \n\n**Supplementary material:** Kindly note that there is an Appendix with four pages of supplementary material in the main submssion, immediatly after the bibliography (not as a separate file).\n\n**Sensitivity to noise scale $F$ and classification-loss weight $\\lambda$:**\nIndeed the model's performance is sensitive those parameters and also to the embedding dimension, as illustrated in the Appendix. However, we used the same hyperparameters across all of our experiments, indicating that while some tuning is helpful, it need not be dataset-specific.\n\n\n\n**Scalability to larger numbers of clusters:** Our method has no inherent algorithmic limitation on the number of clusters; the reason we restricted our experiments to up to $K = 200$ clusters was due to memory demands. Since our classification loss assumes a balanced assignment over categories, the minibatch size should grow with the number $K$ of categories for good performance. Such large big minibatch size, along with the $B$ augmented views for the diffusion student, results in high memory demand for large $K$. Nonetheless, we tested ViT-S/16 on full ImageNet ($K= 1000$) and achieved 55.95% accuracy, which is close to TEMI's accuracy [1].\n\n\n**Computation time:** The computation time increases linearly (not exponentially) with the number $B$ of diffusion samples.\nThe latter can be done in parallel, unlike DDIM steps which are sequential. Thus, adding more samples $B$ does not pose a severe computational burden in practice.\n\n[1] Adaloglou et. al., Exploring the limits of deep image clustering using pretrained models, BMVC 2023.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method is well-aligned with clustering in high-dimensional feature spaces by using pretrained ViT features from DINO. This paper employs standard unsupervised classification metrics for evaluation: NMI, Accuracy (ACC) (after best label alignment), ARI. The datasets used are well-established in image clustering, including CIFAR-10, STL-10, subsets of ImageNet, fine-grained sets like Oxford Pets/Flowers, etc., and the authors give ablation studies in different embedding dimensions, weight the classification loss, and rescaling factor.", "theoretical_claims": "There are no theoretical claims in this paper", "experimental_designs_or_analyses": "The experimental design in this paper is well-structured.\nThis paper uses various image sets, including small CIFAR-10, moderate STL-10, medium/large ImageNet subsets and fine-grained sets. Then uses public pretrained ViT weights and only learns the diffusion-based assignment and classification heads. Also, the standard metrics, NMI, ACC, ARI, are computed and compared with previous approaches, including SCAN, TEMI, TSP, etc. Further, the authors also examine key hyperparameters, i.e., embedding dimension, noise scale, and weighting λ($\\lambda$) of the classification loss.", "relation_to_broader_scientific_literature": "This paper extends the well-known class of diffusion models (DDPMs, DDIMs) to a new, previously underexplored domain: unsupervised clustering. The proposed method employs a teacher–student setup in the style of BYOL, SimSIAM, and DINO, but rather than relying on a fully deterministic teacher network, it leverages a diffusion sampler to generate cluster assignments. Further, it aligns with methods such as DeepCluster, SCAN, and Self-Classifier, but crucially replaces fixed pseudo-labels & re-clustering and purely end-to-end latent clustering with diffusion-based assignment embeddings. Like TSP and TEMI, the proposed approach makes use of pretrained ViT features to avoid retraining low-level image representations.", "essential_references_not_discussed": "the authors have a good coverage of references.", "other_strengths_and_weaknesses": "Strengths:\nThis paper is well organized, and the proposed method is described clearly. As far as I know, this paper is the first application of diffusion-based generative modeling to produce embeddings for clustering.\n\nTheir experiments achieve top performance on multiple established benchmarks, CIFAR-10, STL-10, ImageNet subsets, etc.\nWeakness: \nThe authors mention choosing 100 time steps in the reverse diffusion at inference. But it could be clearer on how the tradeoff changes with less step counts, like 25 steps, 50 steps.\nThe proposed method was found to be sensitive to hyperparameters.  With different size of datasets, careful tuning of the noise scale, embedding dimension, and classification loss weight λ($\\lambda$) is required. This may complicate deployment on new datasets without a validation set.", "other_comments_or_suggestions": "See weakness above.", "questions_for_authors": "How does the approach scale beyond 200 clusters or to larger image sets like full ImageNet (1000 classes)? Is there an exponential blow-up in computation from multiple diffusion samples?\nHow sensitive is the model to choosing noise scale or weight λ\\lambda when no labeled validation set is available?\nDo you have any results for fewer inference steps?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L45RZft8Y9", "reviewer": "Reviewer_6A3N", "summary": "This paper introduces a novel self-supervised image clustering framework incorporating the ideas of diffusion models to achieve accurate and robust clustering. The framework is designed in a teacher-student paradigm to train a teacher model to produce diverse cluster assignments and a student model for stable predictions. The effectiveness of the proposed framework has been validated on extensive benchmark datasets.\n\n## update after rebuttal\n\nThis paper's ideas are interesting and novel to me. My questions have also been answered properly by the authors. I'd still recommend an acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the comments and questions.\n\n**Robustness:**\nThe experimental evidence for the robustness of our approach is presented in Figure 2, which shows that when we corrupt the ViT inputs (via feature dropout + Gaussian noise), the degradation in classification accuracy is completely overcome by using big enough $B$ latent samples from the diffusion model.\n\n**Importance of the Min-SNR-$\\\\gamma$ weighting:** The table below shows an ablation study on the accuracy of ImageNet 100 (validation), with and without  Min-SNR-$\\gamma$ weighting, as a function of the training epoch.  \n\n\n| Training epoch | ACC (with Min-SNR-$\\\\gamma$) | ACC (without Min-SNR-$\\\\gamma$) |\n|---------------|--------------------------|-----------------------------|\n| 0             | 0.0100                   | 0.0100                      |\n| 1             | 0.6066                   | 0.5842                      |\n| 2             | 0.6860                   | 0.6648                      |\n| 3             | 0.7164                   | 0.6820                      |\n| 4             | 0.7132                   | 0.7036                      |\n| 5             | 0.7260                   | 0.6916                      |\n| 6             | 0.7196                   | 0.7002                      |\n| 7             | 0.7344                   | 0.7128                      |\n| 8             | 0.7320                   | 0.7110                      |\n| 9             | 0.7356                   | 0.7136                      |\n| 10            | 0.7330                   | 0.7146                      |\n| 11            | 0.7440                   | 0.7172                      |\n\n\nAs is clear from these results, Min-SNR-$\\\\gamma$ speeds up convergence and yields higher final accuracy. These results are consistent with those in the work that proposed this weighting strategy [1], and will be included in the Supplementary Material.\n\n\n\n**Training features:** We fine-tuned Self-Classifier [2] on ImageNet 100 with a DINO backbone (so that augmentations are now on raw images rather than features), and obtained less than 1% improvement in accuracy relative to the frozen backbone. This is consistent with the observations in TEMI [3] that fine-tuning the ViT often yields marginal gains on large datasets. On the other hand, modifying CLUDI to fine-tune the entire backbone would require additional memory (since we maintain multiple augmented views and diffusion states), so for large-scale data it can become quite demanding.\n\n\n**Pets dataset:** This dataset is special because of its small size.\nIt has about 200 images per class and small intraclass variance (compare with more than 1000 images per class for ImageNet and Flowers, with higher intraclass variance). Thus ViT-B is arguably overfitting the data here, as evidenced by ViT-S superior performance.    \n\n\n\n\n[1] Hang et.al., Efficient Diffusion Training via Min-SNR Weighting Strategy, ICCV 2023.\n\n[2] Amrani et. al., Self-supervised classification network, ECCV 2022.\n\n[3] Adaloglou et. al., Exploring the limits of deep image clustering\nusing pretrained models, BMVC 2023.", "claims_and_evidence": "One of the key advantages claimed for clustering with diffusion is its robustness. However, I failed to find any experimental evidence supporting this claim.", "methods_and_evaluation_criteria": "Yes, the proposed method is interesting and technically solid to me. The evaluation criteria are standard in the field.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I went through both the main experiments and ablation studies provided in the supplementary. Most of them are reasonable to me, with just a few concerns/questions:\n1. How important is the Min-SNR-$\\gamma$? I failed to find the ablation study of this.\n2. Does the Self-Classifier perform better with trainable features? If so, how does CLUDI compare to Self-Classifier if both are with trainable features?\n3. It is interesting to see that the ViT-S variant of CLUDI performs even better than the ViT-B variant on Oxford-IIIT Pets. Is there anything special about this dataset?", "relation_to_broader_scientific_literature": "The key contribution of this paper is to leverage the ideas of diffusion models for image clustering for stronger robustness and performance. This is novel to me and holds the potential to benefit also unsupervised representation learning that serves as the foundation for a lot of CV tasks by providing high-quality visual features.", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "The ideas of using diffusion models for image clustering is interesting and novel to me, the paper is generally well-structured and easy to follow. My main concerns and questions remain in the experiment parts as detailed above.", "other_comments_or_suggestions": "See comments above.", "questions_for_authors": "See question in “Experimental Designs or Analysis”", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vBWeOV5hXF", "reviewer": "Reviewer_Azc9", "summary": "This paper introduces Clustering via Diffusion (CLUDI), a self-supervised clustering framework that uses diffusion models on top of pre-trained Vision Transformer (ViT) features. The core idea is a teacher–student setup: a diffusion model (teacher) generates stochastic cluster assignment embeddings, while the student refines them into stable predictions. The paper claims this approach mitigates model collapse and underutilization of pre-trained features, yielding state-of-the-art clustering performance (NMI, ACC, ARI) across multiple datasets (ImageNet subsets, CIFAR-10, STL-10, Oxford Pets/Flowers, Caltech 101). The authors argue that diffusion’s ability to model high-dimensional distributions significantly enhances clustering robustness and accuracy.\n\n## update after rebuttal\nI have read the rebuttal to my review and found that the authors addressed the points within the expected scope. The response does not change my overall assessment, and I maintain my score of 4 (Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions.\n\n\n**Embedding-matrix ablations:** Our explorations on the learned projection matrix $\\bf{E}$ yielded the following results for ImageNet 100:\n- Using the teacher output $\\tilde{\\bf{z}}_0$ directly as a target gives a drop in accuracy of around 4.7%.\n- Setting $\\bf{E}=\\bf{L}^T$ (removing the separate matrix) gives a drop in accuracy of around 3.2%.\n- Randomly-initializing and fixing $\\bf{E}$ gives a 0.8% drop in accuracy.\n\nThese results are consistent with findings in BYOL [1] and SimSiam [2]\n\n**Scalability to larger numbers of clusters:** We kindly refer the reviewer to our reply above to Reviewer 2.\n\n**Domain focus:** We focused on images because this is the domain where most comparable baselines exist for self-supervised classification. But our framework should similarly work on other modalities (e.g., language, audio), provided high-quality pretrained features are available.\n\n\n\n\n**Computational costs:** Bigger computational times, compared to  single forward pass models, are a known challenge of diffusion models. This is due to the sequential DDIM steps, which cannot be parallelized.\nMoreover, for *training* the memory demand is relatively high due to  large batch sizes (to maintain balanced assignments) and $B$ augmented views. These factors jointly increase both memory usage, and can be traded for gradient accumulation, increasing further the training time.\nFor *inference*, we again do sequential DDIM denoising for each image, and sampling many embeddings scales linearly with the number $B$ of draws. This can be parallelized, but total cost still exceeds a single forward pass.\n\n\n[1] Grill et.al., Bootstrap your own latent - a new approach to self-supervised learning, NeurIPS 2020.\n\n[2] Xinlei Chen and Kaiming He, Exploring Simple Siamese Representation Learning, CVPR 2021.", "claims_and_evidence": "- The authors claim that using a diffusion model as a teacher leads to better cluster assignment embeddings. They provide clear comparative results on multiple benchmarks (ImageNet-50/100/200, CIFAR-10, STL-10, Oxford datasets) showing consistent improvements in NMI, ACC, and ARI over existing methods (e.g., SCAN, TSP, TEMI, Self-Classifier*).\n- They argue that combining a stochastic diffusion process with a uniform-prior cross-entropy loss avoids trivial solutions (collapse). Ablations and references to prior work (Amrani et al., 2022) support this claim.\n- The authors state that using an embedding matrix E for learning, similar to BYOL and SimSiam, leads to better empirical performance compared to directly learning from the denoised embedding z₀ produced by the teacher model. However, the paper does not provide strong theoretical justification or detailed ablation experiments specifically analyzing the impact of E. It would have been beneficial to include such analyses to further support this design choice.", "methods_and_evaluation_criteria": "- The authors employ standard clustering metrics: NMI, ACC, ARI. These are widely accepted for measuring how well predicted clusters align with ground truth labels.\n- Benchmark Datasets are relevant and cover a spectrum of complexity: ImageNet subsets, small-scale (CIFAR-10, STL-10), and fine-grained tasks (Oxford Pets/Flowers).\n- Freezing a pre-trained ViT and comparing classification heads ensures fairness in highlighting the contribution of the diffusion-based assignment component.\n\nThese methods and criteria make sense for evaluating clustering performance and appear appropriate for the task.", "theoretical_claims": "- The paper relies mainly on established results in diffusion modeling (DDPM/DDIM) and non-collapsing self-supervised frameworks (BYOL, SimSiam).\n- There is no novel theorem or proof that requires deep scrutiny. The authors do present equations for their backward sampling and training objectives, but these follow known formulations in diffusion literature.", "experimental_designs_or_analyses": "- Experiments are well-designed:\n    - They compare CLUDI to multiple recent baselines under consistent settings (same backbone, same evaluation metrics).\n    - They present ablations on the embedding dimension, diffusion noise scale (F²), and classification loss weight (λ).\n    - They show t-SNE visualizations to illustrate cluster quality.\n- The analyses appear valid, with no obvious methodological flaws or major confounding factors left unaddressed.", "relation_to_broader_scientific_literature": "This paper extends deep clustering literature (e.g., DEC, DeepCluster, VaDE, Self-Classifier) by incorporating a diffusion process for assignment embeddings. It also aligns with self-supervised learning frameworks (BYOL, SimSiam, DINO), adapting the teacher–student setup in a novel way.", "essential_references_not_discussed": "I do not see major omissions critical for contextualizing their key contributions. The authors cite a comprehensive set of core works in diffusion-based generative modeling and deep clustering.", "other_strengths_and_weaknesses": "**Strengths**\n\n- **Novelty**: Diffusion-based cluster assignments represent an innovative contribution.\n- **Empirical Rigor**: SOTA results on diverse benchmarks, with thorough ablations.\n\n**Weaknesses**\n\n- **Limited Large-K Validation**: Results up to 200 clusters are strong, but the method’s performance on 500, 1000, or higher K remains underexplored.\n- **Single Domain Focus**: Only tested on vision tasks, leaving question marks about text, audio, or other modalities.\n- **Training Time & Computational Overhead**: The paper lacks a detailed analysis of how the diffusion-based approach compares to simpler clustering methods in terms of runtime and resource demands, especially at larger scales.", "other_comments_or_suggestions": "More details on training time or computational overhead vs. simpler clustering methods would be helpful.", "questions_for_authors": "No questions, as the paper's limitations are well-discussed, and there are no aspects that would change my overall evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes CLUDI, a novel self-supervised image clustering framework that integrates diffusion models with Vision Transformer (ViT) features in a teacher-student setup. The method aims to generate diverse and robust cluster assignments, and is validated on several standard benchmarks, achieving competitive performance.

Reviewers appreciated the novelty of applying diffusion models to clustering tasks, noting that this could inspire future directions in unsupervised representation learning. The paper is clearly written, well-organized, and presents strong empirical results on CIFAR-10, STL-10, and subsets of ImageNet.

However, concerns were raised about the extent to which the diffusion model contributes beyond the already powerful ViT features. In particular, the method's sensitivity to hyperparameters and the unclear impact of diffusion step count at inference time may limit its practicality and generalizability. The dependence on careful tuning across datasets could make the method harder to deploy in real-world unsupervised settings.

Overall, the paper offers an interesting and novel perspective, but further analysis is needed to better isolate and justify the benefits of the diffusion component and to improve robustness across datasets.
Decision: Accept (poster)