Learning from Sample Stability for Deep Clustering
Authors: Zhixin Li
Yuheng Jia
Hui LIU
Junhui Hou
Venue: ICML 2025 poster (2025)
Keywords: Deep clustering
sample stability
Primary area: general_machine_learning->clustering
TL;DR: This paper uses the sample stability to improve deep clustering performance.

Abstract: Deep clustering, an unsupervised technique independent of labels, necessitates tailored supervision for model training. Prior methods explore supervision like similarity and pseudo labels, yet overlook individual sample training analysis. Our study correlates sample stability during unsupervised training with clustering accuracy and network memorization on a per-sample basis. Unstable representations across epochs often lead to mispredictions, indicating difficulty in memorization and atypicality. Leveraging these findings, we introduce supervision signals for the first time based on sample stability at the representation level. Our proposed strategy serves as a versatile tool to enhance various deep clustering techniques. Experiments across benchmark datasets showcase that incorporating sample stability into training can improve the performance of deep clustering. The code is available at https://github.com/LZX-001/LFSS.

Learning from Sample Stability for Deep Clustering
Zhixin Li Yuheng Jia
Hui Liu
Junhui Hou
Abstract
Prior studies have utilized various supervision signals such autoencoder reconstruction (Xie et al., 2016; Li et al., 2018; Yang et al., 2017), sample similarity and neighbor relationship (Chang et al., 2017; Ji et al., 2019; Tao et al., 2021; Li et al., 2024; Huang et al., 2019; 2023), and pseudo labels (Van Gansbeke et al., 2020; Niu et al., 2022; Tian et al., 2017; Cai et al., 2023). Recent advancements in self- supervised learning (Grill et al., 2020; He al., 2020: Chen et al., 2020) have prompted many deep clustering methods to explore contrastive learning for generating clustering- compatible representations (Huang et al., 2023; Li et al., 2022; 2021b; Yu et al., 2023). Despite the effectiveness of these supervision signals enhancing clustering per- formance, current research often overlooks the learning progress of individual samples during network training.
Deep clustering, an unsupervised technique inde- pendent of labels, necessitates tailored supervi- sion for model training. Prior methods explore supervision like similarity and pseudo labels, yet overlook individual sample training analysis. Our study correlates sample stability during unsuper- vised training with clustering accuracy and net- work memorization on a per-sample basis. Un- stable representations across epochs often lead to mispredictions, indicating difficulty in memo- rization and atypicality. Leveraging these find- ings, introduce supervision signals for the first time based on sample stability at the rep- resentation level. Our proposed strategy serves as a versatile tool to enhance various deep clus- tering techniques. Experiments across bench- mark datasets showease that incorporating sam- ple stability into training can improve the perfor- mance of deep clustering. The code is available petts://ggthhee com/LZX 0011 SS.
Throughout training, the representations of the same input sample may vary across epochs due to ongoing weight up- dates. We propose new metric called sample stability, which is defined the cosine similarity of representations for the same input across two consecutive epochs in the model, to represent the learning progress of individual sam- ples. Through extensive experiments, we consistently ob- serve that samples with the most unstable representations are prone to incorrect clustering (see Observation 1). In addition, we find that unstable samples exhibit non-random long-term instability during training (see Observation 2). Further examination reveals that consistently unstable sam- ples are typically atypical and rare compared to stable coun- terparts, indicating their difficulty in model memorization (see Observation 3). These observations hold true across multiple self superviee learning and deep clustering meth- ods, as well as across various datasets and epochs, detailed in Section 2. The above insights underscore the utility of sample stability in evaluating individual sample training progress. Thus, we pioneer the use of sample stability to construct supervision signals and introduce novel deep clustering method LFSS (Learning From Sample Stability). Our approach incorporates predecessor network to track previous epoch weights and assess sample stability. More- over, based on the contrastive learning paradigm (Chen et al., 2020), we employ sample stability from both the instance level and the cluster level.
1. Introduction
Deep clustering trains deep neural network in unsu- pervised manner to assign appropriate labels to samples. Unlike supervised learning, where the ground truth labels are used to guide model optimization, deep clustering lacks such labels to aid network training. Consequently, suit- able supervision signals must be derived from the data it- self to enable deep clustering models to extract meaningful features, yield discriminative high-dimensional represen- tations, and achieve accurate clustering (Lu et al., 2024).
Schooo of Computer Science and Engineering, Southeast Uni- versity, Nanjing 210096, China "Key Laboratory of New Gener- ation Artificial Intelligence Technology and Interdisciplinary Applications (Southeast University). Ministry of Education, China 'School of Computing Information Sciences, Saint Francis Univer- sity, Hong Kong, China 'Department of Computer Science, City University Hong Kong, Hong Kong, China. Correspondence to: Yuheng Jia <yhjia@seuuu se..edu.n...
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In summary, this work makes the following key contribu- tions:

---

Learning from Sample Stability for Deep Clustering
1.0
1.0
1.0
Prodiztion Accursc)
0.3
cton Accuracy
Prediction Accuracy
Prediction accuracy
0.9
0.8
0.8
0.0
0.7
0.5
0.6
0.4
0.4
0.4
0.5
2
5
4
8 9
10
4
5
6
8
10
2
5
4
8
10
4
6
8
9
10
Sample Stability Bins
Sample Stabilty Bins
Sample Stabilty Bins
Sample Stabilty Bins
(a) CIFAR-10, 200-th epoch
(b) CIFAR-10, 400-th epoch
(c) CIFAR-10, 600-th epoch
(d) CIFAR-10, 800-th epoch
Predition Accurazy
Sm3iR
SndL
Prediction Accuracy
0.6
Predictio Accuracy
0.5
Prediction Accuracy
0.6
IDFD
0.5-
0.5
0.4
0.4
0.4
0.3
0.2
0.2
9 10
7 Sample Stability Bins
5
3
9 10
8
Sample Stability Bins
Sample Stability Bins
Sample Stability Bins
(e) CIFAR-20, 200-th epoch
(I) CIFAR-20. 400-th epoch
(g) CIFAR-20, 600-th epoch
(h) CIFAR-20, 800-th epoch
Figure I. Connection between sample stability and clustering prediction accuracy on CIFAR-10 and CIFAR-20. Four methods under various epochs: SimCLR (blue), BOL (black), ProPos (yellow), IDFD (green) shown. It is clear that unstable samples are likely to be predicted incorrectly. All subfigures share the same legend.
For the first time, we identify the strong correlation between sample stability, clustering prediction, and network memorization in the context of unsupervised model training. Notably, unstable samples are prone to erroneous predictions, and we establish clear link between sample stability and network memorization. Our extensive experiments across various methods and datasets validate these findings and reinforce our per- spectives.
the observations from substantial experiments, we elucidate the relationships of sample stability with both clustering prediction and network memorization for the first time. In unsupervised learning, labels are not accessible, and accordingly, we cannot use the accuracy of label prediction to evaluate the training status of a single sample. To this end, we exploit the high-dimensional representations of samples in the embedding space, and use the variations at the representation level the criterion for assessing the training status of sample. Specifically, of a sample the cosine similarity of the representations of the same sample across two consecutive epochs. Assume f(-) is the deep neural network, and æ is an input sample, then sample stability in the t-th epoch can be represented as follow:
define sample stability
We pioneer the utilization of sample stability at the representation level as guiding signal in deep cluster- ing. While recent approaches have introduced diverse supervision signals to enhance deep clustering, they often overlook the evolution of individual samples dur- ing training. Recognizing that sample stability can effectively reflect training quality in an unsupervised setting, hamness guiding signal, introducing novel deep clustering methodology at both the instance and cluster levels.
sff (f (z) |F
(I)
where fe(-) denotes the network at the 1-th epoch, and || denotes & norm. In other words, lower sample stability indicates that the representation of sample is unstable during specific epochs in the training process. Based on the above definition, we have the following three observations.
We assess the clustering performance of LFSS across multiple widely utilized datasets and benchmark it against state-of-the-an techniques. Our experimental results unequivocally showcase the efficacy of our ap- proach. Furthermore, we illustrate that our method can be seamlessly integrated to enhance the performance of existing approaches, serving as valuable plugin.
Observation Unstable samples are more likely to be predicted incorrectly. We observe that sample stability is closely associated with clustering prediction. To validate train several representative unsupervised learning mod- els: SimCLR (Chen et al., 2020) and B BYO (Grill et al., 2020) are two prevalent contrastive learning paradigms and the cornerstones of recent deep clustering methods; ProPos (Huang et al., 2023) and IDFD (Tao et al., 2021) are rep- resentative and ofate-of-the-ar deep clustering methods in
2. Our Insights
In this section, we delve into the common phenomena across different unsupervised deep clustering methods. Based on
2

---

Learning from Sample Stability for Deep Clustering
recent years. We adopt ResNets (He et al., 2016) the backbone. We adopt K-means (Hartigan & Wong. 1979) to cluster as it is the most commonly employed clustering algo-
Orange
rithm in deep clustering (Huang et al., 2023; Tao et al., 2021; Li et al., 2023; Niu et al., 2022). We train the models and record the sample stability at different epochs. We divide all samples into 10 bins based on their stability, from smallest to largest. For statistical fairness, the number of samples in each bin should be similar avoid outliers. So each bin has 10% of the samples. We present the results on the CIFAR-10 and CIFAR- 20 (Krizhevsky, 2009) in Figure 1. Experiments on ImageNet- 10 (Chang et al., 2017), Tiny-ImageNet (Le & Yang, 2015) and ImageNet-1K IK (Deng et al., 2009) are presented in Appendix A. We can observe that under all conditions (across different datasets, methods, and epochs), unstable samples are more likely to be predicted incorrectly than stable samples. This suggests that unstable samples are often poorly learned and difficult to be grouped into specific class. More observations and discussions can be found in Appendix A.
Trailer Truck
King Penguin
Unstable Samples
Stable Samples
Figure 2. Visualization results of SimCLR on stable and unstable samples from ImageNet-10. 10. We can observe that unstable samples tend to be visually more complex or unusual. Such samples can be considered atypical or rare, making them hard memorize, while the stable samples are often typical and conventional.
training. Such samples can be considered difficult for the network to memorize. We further investigate the character- istics of those unstable samples by training SimCLR (Chen et al., 2020) on the ImageNet- 10 dataset. Specifically, we track the top 10% stable and the top 10% unstable samples at the 200-th, 400-th, 600-th, and 800-th training epochs. We select samples that consistently appear in three or four of these training epochs and consider them to be the repre- sentative examples of long-term stable or unstable instances. As shown in Figure 2, we can observe that long-term stable samples are often typical and conventional, with complete presentations and distinctive features, while long-term un- stable samples appear to be rare and atypical, such as, oddly decorated oranges, trailer truck in rearview mirror, penguin in unusual lighting. Compared to stable samples that are easily recognizable by human eyes, unstable sam- ples are visually more complex, with unusual perspectives, incomplete or abnormal compositions, or they may not be the main focus of the image. Such rare or atypical samples are difficult for the network to memorize due to their lack of representativeness.
Table 1. Recurrence counts of unstable samples on CIFAR-10. Some samples exhibit long-term instability.
Count
SimCIR
BYOL.
ProPos
IDFD
Random
Count
11169
10325
8593
10159
17496
Count =
3834
3771
3791
3981
2916
Count =3
1377
1499
1774
1517
216
Count 4
258
409
626
332
6
Observation 2: Some samples exhibit long-term insta- bility in the training process. To verify whether the diffi- culty in accurately predicting unstable samples is due to the samples themselves rather than the networks, we conduct additional experiments by tracking unstable samples over multiple epochs. Specifically, we count the recurrence of the top 10% unstable samples at the 200-th, 400-th, 600-th, and 800-th epochs on CIFAR-10. As a reference, we also randomly assign 10% samples as the most unstable samples at each epoch and count their recurrences. As shown in Table 1, we observe that some unstable samples reappear repeatedly at different epochs. Moreover, the numbers of samples experiencing instability only once for deep cluster- ing methods are observed to be lower than what would be expected under random conditions. Conversely, the num- bers of samples manifesting multiple instabilities signifi- cantly exceed the expectations based on random chance. The above phenomena suggest that some unstable samples are persistently unstable (long-tern instability) rather than being sporadically. Results on other datasets are shown in Appendix B.
3. Related Work
3.1. Contrastive Learning
Contrastive learning is an effective self-supervised learning approach (Gui et al., 2024; Wu et al., 2018). Recently, many well-known contrastive learning approaches were proposed (Grill et al., 2020: He al., 2020; Chen et al., 2020; Caron et al., 2020; Chen & He, 2021: Zbontar et al., 2021: Bardes et al., 2022). Among them, the methods based on negative sample pairs have gained widespread influence. Specifically, each sample undergoes data augmentation to produce dif- ferent views. The views from the same sample are regarded as positive samples, while others are negative samples. By minimizing the distance between positive samples and max-
Observation 3: Sample stability is closely associated with network memorization. As shown in Table 1, some samples fail to learn stable representations during model
3

---

Learning from Sample Stability for Deep Clustering
imizing the distance between negative samples in the latent space, effective representations are learned. Considering NT-Xent loss (Normalized Temperature-scaled Cross En- tropy Loss) in SimCLR (Chen et al., 2020), in a dataset of size N, each sample is augmented twice and this results in 2N views. Let 2; be the representation of the i-th view. The NT-Xent loss can be formulated as follows:
3.3. Network Memorization
A deep neural network (DNN) is a universal approximator that can represent any function when its capacity is suffi- ciently large (Hornik al., 1989; Cybenko, 1989). Some researchers study the learning mechanisms in networks, par- ticularly how networks memorize samples, especially atyp- ical ones, in the training progress (Maini et al., 2023). In supervised learning, previous researches (Maennel et al., 2020; Arpit al., 2017: Zhang et al., 2017; Maini et al., 2023) have revealed that DNNs can memorize samples with random labels. By comparing training with real labels and random labels, they explain how networks memorize. Also, previous works (Feldman, 2020; Feldman & Zhang, 2020) demonstrated that memorizing rare and atypical samples is crucial for achieving close-to-optimal generalization error in supervised classification tasks. In unsupervised learning, how identify and memorize rare and atypical samples is worth studying. However, there is only limited body of related works in this field (Hooker et al., 2019; Jiang et al., 2), further explorations are needed.
2N
exp(sim(2, Re. 2,
LNT-Xent
log
2N
where sim(u, 0) u² //|||||||||| the cosine similarity of representation u and v, T is a temperature parameter and [[z1 returns 1 when l & i clse 0. 2;" is the positive sample of 2,. Eq. (2) serves to pull positive samples closer while pushing negative samples further apart, thereby learning appropriate representations.
Apart from contrastive learning methods that require nega- tive samples, some self-distillation methods can effectively train models using only positive samples (Grill et al., 2020; Chen & He, 2021). These methods often contrast represen- tations produced by asymmetric networks. Considering ft is a target network, fo is an online network and h is a predic- tion head, the loss in a self-distillation method is typically constructed as:
4. Proposed Method
Based on the observations discussed in Section 2, we illus- trate that the stability of samples can serve as indicator of the training advancement of each sample. Samples that exhibit instability are prone to erroneous clustering and are challenging for the network to memorize. Consequently, we can utilize sample stability as a guiding signal to steer the training of models in deep clustering. In this paper, we propose a novel deep clustering methodology called LFSS (Learning From Sample Stability), which capitalizes on sam- ple stability signals from both the instance and cluster levels, in conjunction with self-distillation.
1
Lud
Σ 2- 2-2-- (i((((ss))) N
where 24 and x. are augmented versions of the i-th sample.
3.2. Deep Clustering
Deep clustering aims to train deep neural networks in the absence of labels to perform clustering tasks. Compared to supervised learning, it can significantly save time and labor in label annotation. Although labels are not required, super- vision signals are still needed to provide feedback for model training. Therefore, various supervision signals have been proposed to guide model training (Xie et al., 2016; Li et al., 2018; Yang et al., 2017; Chang et al., 2017; Ji et al., 2019; Tao et al., 2021; Jia et al., 2021; Peng et al., 2022; Li et al., 2024; Liu et al.. 2024; Jia et al., 2025). Specifically, some pseudo-labeling approaches leverage prediction consistency across different epochs or models to filter unreliable training samples (Mahon & Lukasiewicz, 2023; 2021). However, the stability of samples at the representation level remains underexplored. In recent years, due to the advancements in contrastive learning (He et al., 2020; Grill et al., 2020; Chen et al., 2020), contrastive learning has been extensively adopted in deep clustering, leading to excellent clustering performance (Huang et al., 2023; Li et al., 2022; b021b; Yu et al., 2023; Qi et al., 2024; Li & Jia, 2025).
As shown in Figure 3, we adopt an asymmetric network structure as in (Grill et al., 2020). fi and fo denote the target network and online network, respectively. fp is the proposed predecessor network for measuring sample stability. is MIP predictor head. The stop-gradient operation is applied to both f. and fp- fe is momentum updated from fo with 0₂ m0, + (1 - m))., where m E [0, 1] is target decay rate, 0, denotes parameters in the network f. and the asterisk (+) is a wildcard character. fp is directly copied from Se in the last epoch. Given a dataset of size [a. each sample x; is augmented twice to generate two views: of and æq, intended for input into fa and fo respectively. The overall loss of LFSS comprises three components: instance- level loss for LFSS L,, cluster-level loss for LFSS Lc. and self-distillation with noise loss Ls.
Instance-level loss for LFSS. We construct an instance- level supervision signal with sample stability. As introduced in Observation 1, sample with unstable representations are
4

---

Learning from Sample Stability for Deep Clustering
sample partitioning. To this end, apply contrastive learn- ing the cluster level obtain more discriminative cluster centers. We assume that the cluster number K is known as priori, and adopt K-means (Hartigan & Wong, 1979) to obtain the clustering partitions. The center for the i-th cluster can be obtained as follow:
Ls
h
sample stability
Ejec, (x]] c = ]EE {x(3))l
LC
(5)
fp
ft
fo
where of is the center of i-th cluster, obtained from the representations output by network f., and C2 is the set of sample indices belonging to the i-th cluster. Note that we perform clustering on the original samples instead of augmented ones to form {C:
xi
Xi
xi
As unstable samples are more likely to be grouped in the incorrect clusters, excluding them can lead to more accu- rate cluster centers. Thus, we exclude the most unstable instances in the dataset with an unstable ratio å E [o, 1].
Xi
copy from the latest epoch
momentum update
N₂ N * (1 - 6)] indicates the number of remaining stable samples and the set S contains indices of these re- maining stable samples:
The network architecture of LFSS. Each input is aug- mented into two views (xf and zz),, which then fed into the networks (Sp. ft and f.) as inputs. Only the online network fo and the predictor head are involved in gradient backpropagation. The target network ft is momentum updated from fo. and the predecessor network Jp is copied from ft in the latest epoch and is used to measure sample stability. Based on the outputs from dif- ferent networks, we measure the three components in objective function, including instance-level loss L,, cluster-level loss Lc, and self-distillation with noise loss Lg.
Figure 3.
S arggor(si((((()))ff)))))))
:].
(6)
where argsort(-) is a function that returns an array represent- ing the indices of the input elements sorted in ascending order. [-N, returns the last N, elements in the array. In this way, we record the stable samples in the set S. Ex- cluding the most unstable samples, we can obtain more representative cluster centers in the below protocol:
Ejec,ss fa(az) EEJEcsS [x]]]]]
likely to be clustered incorrectly. However, forcing the rep- resentations from two consecutive epochs to be as close as possible could potentially impede the training progress. To balance stability and model training, our approach aims to penalize excessive variations of representations while ensur- ing that these penalties do not harm the overall evolution and improvement of the model. Therefore, we extend the NT--Xent loss and formulate the instance-level LFSS loss as:
Inspired by (Huang et al., 2023), we integrate cluster centers into contrastive learning. Specifically, we treat centers from the same clusters as positive pairs and those from different clusters as negative pairs:
exp(sim(c) (cf, c))/T)
Lc
Σ log (-1 Σx [ (sim +f/T)
(8)
N
K
1
log N
(4)
lny exp(ssm((f((())))))))))))) fx(z)))
In each epoch, we first conduct K K-means (Hartigan & Wong, 1979) globally for cluster assignments and then select stable samples in Eq. (6). The cluster centers in Eq. (8) are computed batch-wise manner.
In Eq. (4), the output representations of the same sample in two consecutive epochs: fo();) and (x(),, form posi- tive sample pairs, while those from different samples form negative sample pairs. The positive samples output by the predecessor network result in larger losses for unstable sam- ples, whereas negative samples prevent the network from staying the same and benefit model training.
dett-listillation with noise. In addition to the aforemen- tioned methods that leverage sample stability, inspired by (Kingma & Welling, 2014; Huang et al., 2023), we adopt self-distillation approach to learn robust representations by exploiting the noise. Specifically, we add random noise to the output of the online network for which serves as the input to the predictor head h. The method implemented
Cluster-level loss for LFSS. In clustering, cluster centers can describe the main characteristics of their corresponding clusters. Discriminative cluster centers help generate better
5

---

Learning from Sample Stability for Deep Clustering
Algorithm 1 Proposed LFSS
five commonly used datasets, and two more challenging large-scale datasets.
Input: Dataset X of size N, epoch number T, batch size b, cluster num K, hyper-parameter X, ')- a and 6, online network for target network ft, predictor head h, predecessor network fy- Output: Clusters {C for t to T do sample batch {-.}i- from X augment 2; to generate sa and =f
5.1. Datasets
We conduct experiments on multiple commonly used datasets, including CIFAR-10 (Krizhevsky, 2009), CIFAR- 20 (Krizhevsky, 2009), STL-10 (Coates et al., 2011), ImageNet- 10 (Chang et al., 2017), ImageNet-Dogs (Chang et al., 2017), Tiny-ImageNet (Le & Yang, 2015) and ImageNet-IKS (Deng et al., 2009). CIFAR-20 is CIFAR-100 using 20 super-classes for evaluation. We train models on STL-10 with extra unlabeled data. ImageNet- 10, ImageNet- Dogs and Tiny-ImageNet are subsets of ImageNet- K, con- taining 10, 15, 200 classes respectively. A summary of the datasets used for evaluation is shown in Table 2.
compute self-distillation with noise Ls using Eq. (9)
compute instance-leve loss for LFSS L, using Eq. (4)
if i > 1/ then
compute sample stability and select stable samples using Eq. (6)
compute cluster centers using Eq. (7)
compute cluster-level loss for LFSS Le using Eq. (8)
end if
Table 2. A summary of benchmark datasets used for evaluation.
update fo and h by minimizing Eq. (10)
Dataset
#Samples
#Clusters
Image Size
update fp by directly copying ft update Jr using momentum update end for
CIFAR-10
60,000
10
32x32x3
CIFAR-20
60,000
20
32x32x3
STL-10
13,000
10
96x96x3
Apply K-means to obtain Clusters {Ci)
ImageNet-10
13,000
10
96x96x3
ImageNet-Dogs
19,500
15
96x96x3
100,000
200
64x64x3
ImageNet-IK.
1,281,167
1,000
224x224x3
as below:
Σ 2 I.s N 1=1
2 sim(f((e)) + +))),, (9)
5.2. Implementation Details
where a > 0 is a hyper-parameter indicating the intensity of
We adopt ResNer-18 18 as the backbone unless specifically specified. We train the models for 1,000 epochs with a batch size of 256, unless noted otherwise. We report the results of the last epoch, using three commonly used metries: cluster- ing accuracy (ACC), normalized mutual information (NMI), adjusted rand index (ARI). We adopt the stochastic gradi- ent descent (SGD) optimizer and the cosine decay learning rate schedule to effectively train our model. Besides, we adopt data augmentation methods following (Chen et al., 2020). We empirically set the trade-off hyper-parameter in Eq. (10) to 1.1 for all experiments unless otherwise spec- ified. We set the unstable ratio ő to 0.1 for all experiments to exclude the most unstable samples in cluster-level loss for LFSS, as indicated by the results in Observation 1. We set the warmup epoch number 17 to 200 for CIFAR- 10 and ImageNet- 10, 500 for CIFAR-20 20, STL-10 and ImageNet- Dogs. The noise intensity in Eq. (9) a set to 0.01 for STL- 10 and ImageNet- 10, 0.001 for CIFAR-10. CIFAR -2 and ImageNet-Dogs. The analysis of these hyper-parameters is presented in Appendix E. We adopt K-means (Hartigan & WWng, 1979) to obtain clustering results and report the average results of 10 trials to avoid the impact of random initialization in K-means. All experiments are conducted based on Py Torch and all models are trained on an NVIDIA RTX 4090 GPU. Other training specifics can be found in
noise and & is a noise vector whose elements are drawn in- dependently from the standard normal distribution N N(00, 1). By minimizing Eq. (9), we can learn more robust represen- tations, which is beneficial for clustering.
Model Training and Clustering Prediction. The overall objective function for LFSS is
L Ls + X(LI + Lc),
(10)
where Ls, L1 and Le are losses in self-distin aif-disiillation with noise, instance-level LFSS and cluster-level LFSS, respectively. we set À as trade-off hyper-parameter to balance two coun- terparts. To obtain suitable cluster centers, we will first train the neural network for 7) epochs using only I.s and I. as a warm-up stage, and then use the complete loss in Eq. (10) for the remaining epochs. After the model training is completed, we apply K-means (Hartigan & Wong, 1979) to obtain the clustering results. The overall procedure is summarized in Algorithm 1.
5. Experiments
In this section, we evaluate the proposed LFSS by compar- ing it with 15 -tate-of-the-art deep clustering approaches
6

---

Learning from Sample Stability for Deep Clustering
Table 3. Comparisons with various methods clustering performance (in percent %) across five henchmarks datasets.
CIFAR-10
CIFAR-20
STL-10
ImageNet-10
ImageNet-Doo Dogs
Methods
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
IIC
51.3
61.7
41.1
25.7
43.1
49.9
29.5
DCCM
49.6
62.3
40.8
28.5
32.7
17.3
37.6
48.2
26.2
60.8
71.0
55.5
32.1
38.3
18.2
PICA
56.1
64.5
46.7
29.6
32.2
15.9
78.2
85.0
73.3
33.6
32.4
17.9
SCAN
79.7
88.3
77.2
48.6
50.7
33.3
69.8
80.9
64.6
NMM
74.8
84.3
70.9
48.4
47.7
31.6
69.4
80.8
65.0
CC
70.5
79.0
63.7
43.1
42.9
26.6
76.4
85.0
72.6
85.9
89.3
82.2
44.5
42.9
27.4
MICE
73.7
83.5
69.8
43.6
44.0
28.0
63.5
75.2
57.5
42.3
43.9
28.6
GCC
76.4
85.6
72.8
47.2
47.2
30.5
68.4
78.8
63.1
84.2
90.1
82.2
49.0
52.6
36.2
PCL
80.2
87.4
76.6
52.8
52.6
36.3
71.8
81.0
67.0
84.1
90.7
82.2
44.0
41.2
29.9
TCC
79.0
90.6
73.3
47.9
49.1
31.2
73.2
81.4
68.9
84.8
89.7
82.5
55.4
59.5
41.7
IDHD
71.4
81.5
66.6
41.8
41.1
26.5
62.3
74.1
55.0
75.5
86.2
69.0
50.2
55.3
36.9
ProPos
85.6
92.0
84.1
54.9
51.8
37.8
72.1
83.2
70.4
81.2
89.0
78.7
54.8
63.9
45.2
CONR
78.1
85.3
71.1
53.5
49.9
35.8
70.2
81.2
69.2
79.2
87.7
75.0
52.7
42.1
DMICC
72.2
83.1
67.5
40.0
41.8
24.3
64.1
75.5
60.5
84.3
92.5
85.2
55.1
54.2
39.8
BYOL LFSS (Ours)
72.3
83.3
68.0
51,9
49.3
34.5
68.0
80.7
63.5
73.2
82.1
67.4
51.8
59.7
41.3
87.2
93.4
86.6
59.9
58.7
43.5
77.1
86.1
74.0
85.6
93.2
85.7
61.7
69.1
53.3
Appendix D.
best results on all datasets, which demonstrates the effec- tiveness of the proposed LFSS. Specifically, on the fine- grained dataset ImageNet-Dogg our method has achieved the best performance improvements, with increases of 5.2% in ACC, 6.3% in NMI, and 8.1 % in ARI. We also conduct significance tests the methods being implemented in Ap- pendix F, demonstrating that our approach is statistically significantly better.
5.3. Main Results
In this section, we evaluate the clustering performance of the proposed LFSS on five commonly used benchmark datasets and compare with both baseline and state-of-the-art meth- ods, as shown in Table 3. We compare LFSS with 14 repre- sentative deep clustering methods, including IIC (Ji al., 2019), DCCM (Wu et al., 2019), PICA (Huang et al., 2020), SCAN (Van Gansbeke et al., 2020), NMM (Dang et al., 2021), CC(Li al., 20) MiCE (Tsai et al., 2021), GCC (Zhong et al., 2021), PCL (Li et al., 2021a), TCC (Shen et al., 2021), IDFD (Tao et al., 2021), ), ProPos (Huang et al., 2023), CoNR (Yu et al., 2023), DMICC (Li et al., 2023) and baseline BYOL (Grill et al., 2020). The results from the IIC to PCL methods are directly quoted from (Huang al., 2023). The remaining four methods, including IDFD, ProPos, CoNR, DMICC, are representative approaches re- cently. To ensure fair comparison, we reproduced these methods using the same backbone, batch size, and number of epochs as our method. To ensure the validity of the re- sults, consistent with our approach, we report the average outcomes of 10 K-means runs. Regarding other experimen- tal setups of each method, we referred their respective papers. We reproduced the baseline BYOL using exactly the same experimental setup as our method.
Additionally, we conduct experiments on large-scale datasets, such as Tiny-ImageNet and ImageNet- LFSS achieves superior performance, demonstrating effective- ness on large-scale datasets. These experimental results can be found in Appendix C.
Given the improvements of this method over other state- of-the-art methods and the baseline, we can attribute the performance improvements to the effective use of instance- level loss LFSS and cluster-level loss for LFSS. We demonstrate that using sample stability as supervision sig- nal for deep clustering models can yield more discriminative representations and achieve better clustering performance.
5.4. A Ablation Study
To evaluate the effectiveness of each component in LFSS, have conducted extensive ablation studies CIFAR-10 in Table 4. Note that all ablation experiments are conducted under the same experimental setup. We have demonstrated that our method achieves significant performance improve- ments compared to B BYO Section 5.3. We further exam- ine the effect of the self-distillation with noise method by training for 1,000 epochs using only the Ls loss. Compared to BYOL, it achieves performance improvements, demon- strating that adding noise the input of the predictor head
As listed in Table 3, it can be observed that the pro- posed LFSS outperforms the state-of-the tf-the-rr methods on all datasets. Compared to baseline YOI,, LFSS has achieved significant improvements across all datasets. Specifically, attains significant improvement with substantial gins ranging from 5.4% to 18.6% in all metrics. Compared to state-of-thed methods, our approach also achieves the

---

Learning from Sample Stability for Deep Clustering
is effective. By separately removing L, and Le from the complete method, we find that compared with LFSS, both LFSS w/o L, and LFSS w/o Le face performance degrada- tion, but there is still a performance improvement compared to the baseline. These results prove that I, and Le are beneficial for deep clustering. Additionally, we conduct experiments by removing the predecessor network. Specifi- cally, we retain all loss functions of LFSS but no longer use the predecessor network (for generating sample stability) supervision signal. L, is computed by the output repre- sentations from Jt and fo. using different augmented inputs. Le is computed when unstable samples are not excluded. The results suggest that the modifications made using sam- ple stability have good effects. The above ablation studies demonstrate that the effective utilization of sample stability leads to better clustering performance.
1.0
Prodctton Accurccy
0.0
0.9
0.8
0.5
0.7
Prediction
0.4
SmyLR BIGL
0.3
LPS5
8 Sample Stabilty sins
3
10
2
3
4
5
0
U
9 10
Sample Stabiliyy Bins
(a) CIFAR-10
(b) CIFAR-20
Figure 4. LFSS benefits the training of unstable samples, improv- ing their clustering accuracy. The subfigures share the same legend.
Table S. The proportion (in percent %) of samples transitioning from unstable to stable starting from the 200-th epoch.
400-th epoch
600-th epoch
800-th epoch
final epoch
CIFAR-10 10
60.8
83.3
97.8
99.3
CIFAR-20 20
72.1
80.6
97.2
99.8
Table Ablation study (in percent %) on CIFAR-10.
NMI
ACC
ART
5.6. Applying Sample Stability into Other Methods
BYOL
72.3
83.3
68.0
The proposed LFSS is based on the BYOL framework (Grill et al., 2020). However, utilizing sample stability as a su- pervision signal not limited to this framework. It can be extended to other methods. To better illustrate the gener- ality of using sample stability have extended LFSS to Moco (He et al., 2020) and SimCLR (Chen et al., 2020) by introducing L, and Le into their frameworks. For LFSS (MoCo) and LFSS (SimCLR), we adopt their respective network architectures and replace the Ls in LFSS with their original loss functions. We train all models for 1,000 epoch with ResNet-18. 18, using a batch size of 256. To achieve fair comparison, implement both the original methods and their corresponding LFSS versions using exactly the same experimental setup. For other experimental settings and ppprmm-rrrettr choices unique to LFSS, we keep them consistent with Section 5.2. The clustering performance shown Table We can ob- serve that LFSS exhibits performance improvements across various frameworks. In some experimental scenarios, such as Moco and B on CIFAR- 10, LFSS can bring about up
Only L. LFSS w/o L1 LFSS w/o Lc
79.5
86.7
75.1
83.6
89.6
80.3
81.6
89.5
79.5
LFSS w/o predecessor network
83.5
90.7
81.5
supervision signal, we
LFSS
87.2
93.4
86.6
5.5. LFSS Benefits Unstable Samples
To further investigate the impact of LFSS, we evaluate the clustering accuracy of samples with different degrees of stability in Figure 4. We replicate the experimental setup in Observation 1. bimning samples by stability and measuring the respective clustering accuracy. At the 1000-th epoch on CIFAR- 10 and CIFAR-20, where LFSS has completed its training stage, it can be seen that LFSS (in red) improves the clustering accuracy of unstable samples compared to other methods. On CIFAR-10, the performance improve- ment of LFSS mainly comes from more accurate prediction of unstable samples. On CIFAR-20, LFSS enhances clus- tering performance for both unstable and stable samples. As higher clustering accuracy signifies more discriminative representations, these experimental results demonstrate that LFSS facilitates the learning of unstable samples.
Table 6. Clustering performance (in percent %) of applying sample stability into various methods.
Also, we focus on the top 10% most unstable samples at the 200-th epoch on CIFAR-I0 and CIFAR-20, using the stability of the sample at exactly the 10-th percentile the threshold for determining instability. In subsequent epochs of training, significant proportion of these samples surpass this threshold and become stable. This indicates that our method can effectively reduce the number of unstable samples. The results are shown at Table 5.
CIFAR-10
CIFAR-20
Method
NMT
ACC
ART
NMI
ACC
ART
BYOL LFSS (BYOL)
72.3
83.3
68.0
51.9
49.3
34.5
87.2
93.4
86.6
59.9
58.7
43,5
Moco LFSS (Moco)
50.5
57.7
40.2
34,8
36,9
19.3
56.8
67.6
49.0
39.3
40.9
22.0
SimCLR R
73.9
83.1
69.0
47.2
46.8
31.2
LFSS (SimCLR)
79.2
88.4
74.4
48.2
47.5
31.4
8

---

Learning from Sample Stability for Deep Clustering
approximately a 10% performance improvement. These results substantiate the scalability of using sample stability as supervision signal.
Cai, S., Qiu, L.., Chen, X., Zhang, Q. and Chen, L. Semantic-enhanced image clustering. In Proceedings of the AAAI conference on Artificial Intelligence, volume 37, pp. 6869-68666668 6878, 2023.
6. Conclusion
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912- 9924. Curran Associates, Inc., 2020.
In this paper, we have presented a novel deep clustering method that harnesses insights from sample stability to en- hance clustering outcomes. Through extensive experimen- tation, established a strong correlation between sample stability, clustering accuracy, and network memorization. Notably, we revealed that unstable samples are more prone to misclustering, highlighting the utility of sample stabil- ity as a guiding signal in unsupervised clustering scenarios. Furthermore, we found that samples exhibiting prolonged instability pose challenges for network memorization, often characterized by complexity and uniqueness in their original features. Building on these findings, we proposed novel LFSS method, which leverages sample stability at both the instance and cluster levels to advance deep clustering perfor- mance. Our method outperforms stae-of-the-an approaches across various commonly employed datasets, demonstrat- ing its efficacy. Through additional experiments, delve into the factors underpinning our method's effectiveness and showcase its adaptability across diverse methods.
Chang, J., Wang, L., Meng, G., Xiang, S., and Pan, C. Deep adaptive image clustering. In Proceedings of the IEEE International Conference on Computer Vision, Oct 2017.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. simple framework for contrastive learning of visual rep- resentations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEEE/CVE Con- ference on Computer Vision and Pattern Recognition, pp. 15750-15758, June 2021.
Coates, A., Ng. A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceed- ings of the Fourteenth International Conference on Artifi- cial Intelligence and Statistics, volume 15, pp. 215-223. PMLR, 11-13 Apr 2011.
Impact Statement
This paper presents work whose goal is to advance the field of deep clustering. There are many potential societal conse- quences of our work, none which we feel must be specifi- cally highlighted here.
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of Control, Signals and Systems, 2(4):303-311 314,,,4,,1 1989.
Dang, Z., Deng, C., Yang, X., Wei, K., and Huang, H. Near- est neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 -3792-13702, June 2021.
Acknowledgments
This work was in part supported by the National Natural Science Foundation of China under Grant U24A20322 and 62422118, and in part supported by the Hong Kong UGC under grant UGC/FDS1 1/E02/22 and UGC/FDS1 1/E03/24. This research work is supported by the Big Data Computing Center of Southeast University.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 248-255, 2009.
References
Feldman, V. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp- 954 959, 2020.
Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233 233-242. PMLR, 2017.
Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence esti- mation. Advances Neural Information Processing Systems, volume 33, pp. 2881-2891. Curran Associates, Inc., 2020.
Bardes, A., Ponce, J., and LeCun, Y. VICReg: Variance- regularization for self-supervised learning. In International Conference on Learning Rep- resentations, 2022.
Grill, J.B.. Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z.,

---

Learning from Sample Stability for Deep Clustering
Gheshlaghi Azar, M., Piot, B., kavukeuoglu, k., Munos, R., and Valko, M. Bootstrap your own latent- a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284. Curran Associates, Inc., 2020.
Jia, Y., Liu, H., Hou, J., Kwong, S., and Zhang, Q. Multi- view spectral clustering tailored tensor low-rank repre- sentation. IEEE Transactions Circuits and Systems for Video Technology, 31(12):4784-477 2021.
Jia, Y., Cheng, J., LIU, H.,, and Hou, J. Towards calibrated deep clustering network. In The Thirteenth International Conference on Learning Representations, 2025.
Gui, J., Chen, T., Zhang, J., Cao, Q.. Sun, Z., Luo, H., and Tao, D. A survey on self--upervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9052- 2024.
Jiang. Z., Chen, T., Mortazavi, B. J., and Wang, Z. Self- damaging contrastive learning. In International Con- ference on Machine Learning, pp. 4927-4939 PMLR, 2021.
9071, 2024.
Hartigan, J. A. and Wong, M. A. Algorithm as 136: A k-means clustering algorithm. Journal of the Royal Statis- tical Society, Series C (Applied Statistics), 28(1):100-108, 1979. ISSN 00359254, 14679876.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference on Learning Repre- sentations, 2014.
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009,
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2016.
Le, Y. and Yang. X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Li, F. Qiao, H., and Zhang, B. Discriminatively boosted image clustering with fully convolutional auto-encoders. Pattern Recognition, 83: 161-133,, 2018.
He, K., Fan, H., Wu, Y.. Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729- 9738, 2020.
Li, H., Zhang, L., and Su, K. Dual mutual information constraints for discriminative clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 37, pp. 8571-8579, 2023.
Hooker, S., Courville, A., Clark, G., Dauphin, Y., and Frome, A. What do compressed deep neural networks forget? arXiv preprint (9ti::/991.00544::: 2019.
Li, J., Zhou, P., Xiong, C., and Hoi. S. Prototypical con- trastive learning of unsupervised representations. In Inter- national Conference on Learning Representations, 2021a.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neu- ral Nenworks, 2(5):359-33) 366, 1989.
Li, Y., Hu, P., Liu, Z., Peng, D., Zhou, J. T., and Peng, X. Contrastive clustering. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 8547- 8555, 2021b.
Huang. J., Dong, Q. Gong, S., and Zhu, X. Unsupervised deep learning by neighbourhood discovery. In Interna- tional Conference on Machine Learning, pp. 2849 2858. PMLR, 2019.
Li, Y., Yang, M., Peng, D., Li, T., Huang, J., and Peng, X. Twin contrastive learning for online clustering. Interna- tional Journal of Computer Vision, 13009))205--2211, 2022.
Huang, J., Gong, S., and Zhu, X. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IFFF/CVE Conference on Computer Vision and Pattern Recognition, June 2020.
Li, Y., Hu, P., Peng, D., Lv, J., Fan, J., and Peng. X. Image clustering with external guidance. In Forty-first Interna- tional Conference Machine Learning, 2024.
Huang, Z., Chen, J., Zhang, J., and Shan, H. Learning representation for clustering via prototype scattering and positive sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7509-752 2023.
Li, Z. and Jia, Y. Conmix: Contrastive mixup at repre- sentation level for long-tailed deep clustering. In The Thirteenth International Conference on Learning Repre- sentations, 2025.
Ji, X.. Henriques, J. F., and Vedaldi, A. Invariant informa- tion clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, October 2019.
Liu, H., Hu, P., Zhang, C., Li, Y., and Peng, X. Interactive deep clustering via value mining. In Advances in Neural Information Processing Systems, volume 37, pp. 42369 42387. Curran Associates, Inc., 2024.
10

---

Learning from Sample Stability for Deep Clustering
Lu, Y., Li, H., Li, Y., Lin, Y., and Peng. X. A survey deep clustering: from the prior perspective. Vicinagearth, 1(1):4, 2024.
Tsai, T. W., Li, C., and Zhu, J. Mice: Mixture of con- trastive experts for unsupervised image clustering. In International Conference Learning Representations, 2021.
Maennel, H., Alabdulmohsin, I. M., Tolstikhin, I. O., Bal- dock, R., Bousquet, O., Gelly, S., and Keysers, D. What do neural networks learn when trained with random la- bels? In Advances in Neural Information Processing Sys- tems, volume 33, pp. 19693-19704. Curran Associates, Inc., 2020.
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proes- mans, M., and Van Gool, I.. Scan: Learning classify images without labels. In European Conference Com- puter Vision, pp. 268 285. Springer, 2020.
Wu, J., Long, K., Wang, F., Qian, C., Li, C., Lin, Z., and Zha, H. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2019.
Mahon, L. and Lukasiewicz, T. Selective pseudo-label clus- tering. In KI 2021: Advances in Artificial Intelligence: 44th German Conference AI, Virtual Event, Septem- ber 27-October 1, 2021, Proceedings 44, pp. 158 178.. Springer, 2021.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised fea- ture learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, June 2018
Mahon, L. and Lukasiewicz, T. Efficient deep clustering of human activities and how to improve evaluation. In Asian Conference on Machine Learning, pp. 722-737. PMLR, 2023.
Xie, J., Girshick, R., and Farhadi, A. Unsupervised deep embedding for clustering analysis. In International Con- ference Machine Learning, pp. 478 487. PMLR, 2016.
Maini, P., Mozer, M. C.,, Sedghi, H., Lipton, Z. C., Kolter, Z.,, and Zhang, C. Can neural network memorization be localized? In International Conference on Machine Learning, pp.
Yang, B., Fu, X., Sidiropoulos, N. D., and Hong, M. To- wards k-means- <-means-friendly spaces: Simultaneous deep learn- ing and elustering. In International Conference on Ma- chine Learning, pp.
PMLR, 2023.
PMLR, 2017.
Niu, C., Shan, H., and Wang, G. Spice: Semantic pseudo- labeling for image clustering. IEEE Transactions on Image Processing. 31:7264 72788 2022.
Yu, C., Shi, Y., and Wang, Contextually affinitive neigh- borhood refinery for deep clustering. Advances in Neural Information Processing Systems, 36, 2023.
Peng, 7., Jia Y., Liu, H., Hou, J., and Zhang, Q Maximum entropy subspace clustering network. IEEE Transactions Circuits and Systems for Video Technology, 32(4): 2199-2210, 2022.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Bar- low twins: Self-supervised learning via redundancy reduc- tion. International Conference on Machine Learning, pp. 123 12110-1220. PMLR, 2021.
Qi, J., Jia, Y., Liu, H., and Hou, J. Superpixel graph con- trastive clustering with semantic-invariant augmentations for hyperspectral images. IEEE Transactions on Circuits and Systems for Video Technology, 34(11): 11:1:):11:33 1360-11372, 2024.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking gen- eralization. International Conference on Learning Representations, 2017.
Zhong, H., Wu, J., Chen, C., Huang, J., Deng, M., Nic, L., Lin, Z., and Hua, X.-S. Graph contrastive clustering. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9224-9233, October 2021.
Shen, Y., Shen, Z.,, Wang, M., Qin, J., Torr, P., and Shao, L. You never cluster alone. In Advances in Neural Informa- fion Processing Systems, volume 34, pp. 27734-27776. Curran Associates, Inc., 2021.
Tao, Y., Takagi, K., and Nakata, K. Clustering- representation learning via instance discrimination and feature decorrelation. In International Conference on Learning Representations, 2021.
Tian, K., Zhou, S., and Guan, J. Deepeluster: A general clustering framework based on deep learning. In Machine Learning and Knowledge Discovery in Databases, pp. 205. Springer International Publishing, 2017.

---

Learning from Sample Stability for Deep Clustering
Appendix
Prediction accurscy
0.8
Prediction accuracy
0.8
Prediction Accurocy
Prediction accuracy
0.8
0.6
0.6
0.4
0.4
0.4
0.2
5
10
2
5
0.2
8
Sample Stabilty Bins
Sample
Sample
Bins
(a) ImageNet-10. 200-th epoch
(b) ImageNet-100
10, 400-th epoch
(c) ImageNet-10. 600-th epoch
(d) ImageNet-10 10, 800-th epoch
Pradctinn Accuracy
0.125
Prediction Accuracy
0.20
Prodiction Accuracy
Prediction Accuracy
2.3
0.20
0.075
0.15
0.2
0.10
0.10
0.1
0.025
0.05
0.05
2
10
2
3
4
5
6
8
9
5
6
10
Sample Stabilty Bins
Sample Stability Bins
Sample Stabiity Bins
Sample Stabilty Bins
(e) Tiny-ImageNer 40-th epoch
(I) Ima--ImaeNNet, 80-th epoch (g) Tiny- ImageNet, 120-th epoch (h) Bun-IImgeNet, 160-th epoch
Prediction Accuracy
0.15
Prediction Accuracy
0.15
Prediction Accuracy
0.20
Prediction Accuracy
0.20
0.10
0.15
0.10
0.10
0.05
0.05
7
10
7 8 9
10
Sample Stability Bins
Sample Stability Bins
Semple Stability Bins
Sample Stabiltty Bins
(i) ImageNet-IK. 10-th epoch
(i) ImageNet-IK, 20-th epoch
(k) ImageNet- 30-th epoch
(I) ImageNet- IK, 40-th epoch
Figure Connection between sample stability and clustering prediction on ImageNct-10, Tiny-ImageNet and ImageNet-IK. The experiments indicate that across datasets of varying scales and at different training epochs, unstable samples are more likely to be predicted incorrectly deep clustering. Four methods under various epochs: SimCLR (blue), BYOL (black), ProPos (yellow), IDFD (green). subfigures share the same legend.
A. More Experiments on Observation
Here, we provide further evidence for Observation 1. In addition to the CIFAR-I0 and CIFAR-20 datasets mentioned in the main text, in Figure 5 we have also conducted experiments on ImageNet-10, Iiny-ImageNet, and ImageNet- IK. ImageNet-Il 10 and Tiny- Iny-ImmgeNet are subsets of ImageNet- 1K. The specifics can be seen Table 2. These three datasets differ significantly in scale. Additionally, for different datasets, present the training progress various stages. In experiments, present the outcomes at the 200-th, 400-th, 600-th, and 800-th epochs ImageNet-10) 10, at the 40-th, 80-th, 120-th, and 160-th epochs on Tiny-ImageNy the results indicate that unstable samples are more likely to be clustered into wrong classes at various epochs across different methods and datasets. These experiments further corroborate Observation
and at the 10-th, 20-th, 30-th, and 40-th epochs on ImageNet- 1K. In Figure 5,
In all our experiments, we set the number of bins to 10, dividing all samples into 10 equally-sized groups based on their stability. We also present results with and 20 bins on CIFAR-I0 in Figure confirm that our findings are robust across different granularities of sample stability partitioning. These results strongly indicate significant association between sample stability and clustering prediction.
We provide the training details of the observational experiments here to ensure reproducibility. We have trained models using four unsupervised methods, which are very influential in deep clustering, including SimCLR (Chen et al., 2020), Moco (He et al., 2020), ProPos (Huang et al., 2023) and IDFD (Tao et al., 2021). For all methods, we use ResNet- 18 as the backbone on CIFAR-10. 10, CIFAR-20 and ImageNet- 10, ResNet-50 on Image and ImageNet- The same experimental
12

---

Learning from Sample Stability for Deep Clustering
1.0
1.0
Prddztion Accarsc)
Prediction accurac)
0.9
Prodiction Accuracy
Prdiction accurscy
0.9
0.7
0.8
0.7
0.6
0.7
0.6
0.5
0.5
0.6
2
3
4
2
4
Sample Stability Bins
Sample Stabilty Bins
Sample Stability Bins
Sample Stabilty Bins
(a) 5 bins, 200-th epoch
(b) 5 bins, 400-th epoch
(c) 5 bins, 600-th epoch
(d) 5 bins, 800-th epoch
10-
04-
0.9
0.7
0.0
0.0
066
0.6
07
0.3
04
BYOL
Sinc.u
06
0.5-
04:
1
1.3.44
3.3.4
350151sesssaamin Semple Stasiity Bins
1.3344
Sample Stasilty Bins
Sample Staciity Bins
Sample Stacilty Bins
(e) 20 bins, 200-th epoch
(1) 20 bins, 400-th epoch
(g) 20 bins, 600-th epoch
(h) 20 bins, 800-th epoch
Figure Observations on the stability of CIFAR-10 samples under partitioning granularity of different levels. can be seen that across different partitions, unstable samples are more likely to be consistently predicted incorrectly. Four methods under various epochs: SimCLR (blue). BYOL (black), ProPos (yellow). IDFD (green). All subfigures share the same legend.
phenomena are observed across ResNets of different depths, further validating the generality of our observations. We adopt a batch size of 256 and train all models on an NVIDIA RTX 4090 GPU. We adopt the stochastic gradient descent (SGD) optimizer and the learning rate is set to 0.05 on CIFAR-10, CIFAR-20, Tiny-ImageNet and ImageNet-IK. We set learning rate 0.00 0.005 on ImageNet- 10 due to smaller sample amount. Regarding the selection of method-specific hyper-parameters, we refer to their respective papers.
B. More Experiments on Observation 2
We present the persistent instability of samples across various datasets in Table 7. As described in the main text, we identify a significant number of samples that exhibit long-term instability during the training process, far exceeding what would be expected by random selection. We selected four epochs on each dataset for statistical analysis, and these epochs are consistent with those used in the experiments described in Observation 1: we count the most unstable 10% of samples the 200-th, 400-th, 600-th, 800-th epochs for CIFAR-20 and ImageNet-101 10, at the 40-th, 80-th, 120-th, 160-th epochs for I'iny-Imagelo at the 10-th, 20-th, 30-th, 40-th for ImageNet- K. Additionally, the probability of each sample being
Table Recurrence counts of unstable samples on various datasets, all datasets, there are certain samples that remain unstable over the long term.
Dataset
Count
SimCLR
BYOL
ProPos
IDFD
Random
Count 1
11897
10734
10175
10915
17496
CIFAR-20
Count =
3870
4037
4109
4049
2916
Count =3
1165
1364
1469
1273
216
Count 4
217
275
300
292
Count I
2187
2753
1296
2868
3790
ImageNet- 10
Count =2
793
658
576
892
Count 3
317
289
448
168
47
Count
Count
17855
Tiny-InageNot
Count =
6038
6182
6451
7026
4860
Count
1586
Count 4
107
389
222
467
10
Count =1
280545
269498
247905
233076
373587
ImageNet-1K 1K
Count 2
81502
81466
83144
88438
62258
Count
19289
21828
26250
26368
4612
Count 4
2762
3638
488
5852
128
13

---

Learning from Sample Stability for Deep Clustering
Table 8. Comparisons with various methods on clustering performance (in percent %) Tiny- Imageee.
DCCM
PICA
CC
GCC
MoCo
PCI.
SimSiam
BYOL
PropPos
Ours
NMI ACC
22.4
27.7
34,0
34.7
34,2
35.0
35.1
36.5
40.5
43.5
10.8
9.8
14.0
13.8
16.0
15.9
20.3
19.9
25.6
26.8
ARI
3.8
4.0
7.1
7.5
8.0
8.7
9.4
10.0
14.3
16.2
Table 9. Comparisons with various methods on clustering performance (in percent %) on ImageNNet-IK,
SimCLR
BYOL
IDFD
PropPos
Ours
NMI
40.6
40.6
24.6
43.2
45.4
ACC
15.2
11.5
3,45
14.8
15.8
ARI
7.81
55.9
1.06
7.19
8.58
randomly selected 0.1, following a binomial distribution. Note that the results of the random selection are rounded the nearest integers.
C. Clustering on Large-scale Datasets
To demonstrate the effectiveness of LFSS on the large-scale dataset, we train the model on the Tiny- ImageNett and ImageNet- 1K to evaluate its performance. Tiny-ImageNet contains 200 classes and 100,000 samples. Imagenet- K contains 1,000 classes and about 1.2 million samples. They are generally challenging for deep clustering due to its relatively large number of classes. It difficult for deep clustering methods effectively learn discriminative representations that can distinguish between the various classes on such datasets.
On Tiny-ImageNet, following (Huang et al., 2023), we train the model with ResNet-18 for 1,000 epochs with a batch size of 256. For hyper-parameters, we set to 200, À to 0.1, to 0.001 and 5 to 0.1. As shown in Table 8, the proposed LFSS achieves superior performance compared to state-of-the-arr algorithms. Specifically, we can observe that our method has significantly improved the NMI, ACC, and ARI by 3%, 1.2%, and 1.9% respectively. On ImageNet-Ie IK, we implement four different methods (Chen al., 2020; Grill et al., 2020; Tao al., 2021; Huang et al., 2023) and report their performance in Table In detail, we train all the models with ResNet-50 for 50 epochs, using a batch size of 256. For syper-parameters in LFSS, we 1) to 50, 0.1, a to 0.001 and 0 to 0.1. As shown in Table 9, LFSS achieves the best clustering performance. It improves ACC by 1%, NMI by 2.2% and ARI by 1.39%. The experimental results demonstrate the effectiveness of LFSS on the large-scale dataset.
D. Training Specifics
Here, we provide the experimental details omitted from the main text facilitate the reproduction of the proposed LFSS. We adopt the data augmentation strategies (Chen et al., 2020). We set the dimension of output representations to 256. The temperature is set to 0.5 for all experiments. We adopt the stochastic gradient descent (SGD) optimizer, whose learning rate is 0.05, momentum is 0.9 and weight decay is 0.0005. We adopt a cosine annealing learning rate strategy to adjust the learning rate at each iteration. Following (Huang et al., 2023), we set the learning rate of the predictor head h to be 10 times that of the rest of the network, to benefit model training. We set the minimum learning rate to 0 for all datasets, while specially setting to 0.025 for ImageNet-Dogs Following (Chen et al., 2020), we modify ResNet-18 18 when training on CIFAR-10 and CIFAR-20,, due to their small image sizes. We remove the first max-pooling layer and use 3x3 kernel the first convolutional layer.
E. Hyper-parameter Analysis
In this section, we analyze how multiple hyper-parameters influence model training. We evaluate models under different choices of syper-parametert to observe their impact on the model. We employ the single-variable control method in parae-ear analysis and conduct experiments on CIFAR-10, CIFAR-20 and ImageNet-10. 10. That is, when studying the effect of particular hyper-paramen paramerr, we keep other hyper-parameten constant and consistent with Section 5.2.
14

---

Learning from Sample Stability for Deep Clustering
Performance
0.90
0.90
Performance
0.90
0.85
0.85
0.85
ACC
NMI
0.80
0.80
0.1
0.3
0.5
0.7
0.9
10
10-3
10-2
10-1
10
10
10-2
10-1
6
(a) 6, CIFAR-10
(b) CIFAR-10
(c) A, CIFAR-10
0.60
0.60
0.60
Perforancc
0.55
Performance
0.55
ferformance
0.55
0.50
0.50
0.50
0.45
0.45
0.45
0.40
ACC
0.40
NMI
0.40
0.35
XRI
0.35
0.1
0.3
0.5
0.7
0.9
10-4
10
10
10
10-1
10-3
10-2
10-
(d) 6, CIFAR-20
(c) a. CIFAR-20
(f) A, CIFAR-20
0.925
0.925
Performance
0.900
Rerformanc
0.900
Perforaanc
0.875
0.875
0.8
0.850
0.850
0.825
0.825
0.7
0.800
ARI
0.1
0.3
0.5
0.7
0.9
10-4
10-3
10-2
10-1
10-4
10-3
10-2
10-1
A
(g) 8, ImageNet-10
(h) 0, ImageNet-10
(i) X, ImageNet-10 10
Figure 7. Hyper--arameter parameter analysis unstable ratio noise intensity a and balance parameter À on CIFAR- 10, CIFAR 20 and ImageNet- 10. All subfigures share the same legend.
Unstable ratio In cluster-level loss for LFSS, we employ cluster centers on contrastive learning and exclude a proportion of unstable samples from generating centers. We set the unstable ratio to 0.1, 0.3, 0.5, 0.7, 0.9 for investigating its effect. From Figures 7a, 7d and 7g. we can observe that the unstable ratio ò increases from 0.1, all metrics show a declining trend. As the unstable ratio 8 increases, the number of representations used generate cluster centers decreases. When ā becomes too large, many accurate representations are also excluded, which can result in cluster centers that are not necessarily representative. In LFSS, we fix 8 at 0.1.
Noise intensity a. Noise generated from a normal distribution introduced to the Eq. (9) for improving the robustness of model training. The intensity of this normally-disteo distributted noise may have a notable impact on the experimental performance. To examine the impact of noise intensity a on clustering performance, we evaluate the models with o = 0.0001, 0.001, 0.01, and O.I. As shown in Figures 7b, 7c and 7h, we observed that as a varies, the clustering performance remains relatively stable. On different datasets, varying the noise level results in performance differences of about 5%, indicating that both excessively high and low noise intensity do not harm model training. In our experiments, we set to 0.01 or 0.001.
Balance parameter A. The parameter À balances the different terms in Eq. (10), determining the contribution of instance- level and cluster-level losses to model training. Self-distillation with noise Ls differs from instance-leve loss for LFSS L, and cluster-level loss for LFSS Le in mathematical forms, which results in different output values even when given the same input. We have investigated the impact of varying X, including 0.1, 0.01, 0.001, and 0.0001, on the clustering
15

---

Learning from Sample Stability for Deep Clustering
Table 10. Hyper-parameter analysis 7) on CIFAR-IO, CIFAR-20 and ImageNet- (in percent %).
Datascts
Metrics
7=
== 50
1 200
q= 500
1 = 800
NMI
78.1
79.2
85.7
86.4
CIFAR-10
ACC
82.0
83.5
93.4
92.6
92.7
ARI
71.1
71.8
86.6
84.9
85.5
NMI
33.5
52.9
55.9
59.9
$7.3
CIFAR-20
ACC
50.9
48.1
56.2
ARI
35.9
33.5
39.5
43.5
41.5
NMI
64.3
71.1
85.6
83.1
83.7
ImageNet-10
ACC
71.6
76.5
93.2
91.1
92.0
Table 11. Significance Test of Various Methods
CHARR-O
CHHAR-20
ImageNe:
ImageNet-Dorg
Metbod
XB4I
ACC
ARI
NMI
ACCC
ARI
NMI
ACC
.RI
XMI
ACC
ARI
NMI
ADC
ARI
IDFM
71 71-40%8:
81-5(2-1>
66.:1.11 11"
41.500.7ye
41.12002""
26 15
$2.900.11"
24.100.9
$5.000.7"
78.511 aj
86.223.7y
69.02 11
90.31-71
55.322711
96-911000
PraPis
$5.00.11
92.0K0.0"
$6.100.0
$4.K1.29
$1.8:16"
37.541.13
72.111.31
83.212.11
70.41.6"
$1.21000*
89.000/01"
78.770.01
54.8/2.1
63.912 11"
45.212.)2
78-101.21"
95.133.1
$3.50039
49.910---
33.500.0"
30.00021"
81-300.3
79.311
87:703-19*
$2.00.39
62:3/255"
42.11111
72.200.21
N0.30*
40000.19
41.8002)
24:300.21
1..00
75.50000
60.500.0"
$4.310.00
02.5/0/0p
85.200.00
39.800.19 54.200.3"
51.91.57"
$7.2/0.0
58.70800)
77.10.3)
74.00.6)
$5.70.00
53.3/0.0
performance. In Figures 7c, 7f and 7i, we can see that overly small À can lead to performance degradation due to the insufficient contribution these losses. In LFSS, we fix at 0.1.
Warmup epoch number 7). We set 1 to O, 50, 200, 500, and 800 to evaluate the impact introducing cluster-level loss different training stages. As shown in Table 10, early introduction (17 50) reduces clustering performance due generating cluster centers of low quality, while later introduction, when representative cluster centers are well established, 1) 200 500.
achieves better final performance. In experiments,
F. Significance Test
The clustering performance we report on the reproduced methods are based on the average of 10 runs of K-means. In Table 11, we report the standard deviation for each method the parentheses. We perform the t-test for significance at the 5% level to compare the results of the proposed method with those reproduced methods. denotes rejection of the original hypothesis and the two results significantly different. We can observe that the labels assigned by our method under K-means very stable and our clustering performance is significantly better than those of other methods in all cases according to the t-test.
16


Reviews:
[{"review_id": "MtSSjGbSpn", "reviewer": "Reviewer_2B7a", "summary": "This work introduces a novel sample stability, which is strongly tied to misprediction and memorization difficulty. By leveraging stability as a supervision signal, the proposed LFSS method outperforms state-of-the-art approaches on multiple benchmarks.\n\n## update after rebuttal\n\nI read the rebuttal, which addressed my questions. I keep the score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive comments. Below is our response:\n\n> **Changes in sample stability during training (Weakness 1)**\n\nThank you for your valuable advice. We conducted experiments to investigate the changes in sample stability as training progresses. The experiments were performed on SimCLR, BYOL, ProPos, and IDFD frameworks, consistent with the main text. We also tested the proposed LFSS under the same experimental setup. Following the Observation 1 in the main text, we divided the samples into ten bins based on their stability, ranging from lowest to highest. On the CIFAR-10 dataset, we calculated the mean sample stability of each bin at 200, 400, 600, and 800 epochs for each methods. The results are visualized in https://anonymous.4open.science/r/ICML25-0E61/1.1.png. **Generally, as training progresses, sample stability gradually increases. This phenomenon becomes more pronounced once the model training has stabilized after 400 epochs**. \n\nAlso, the sample stability values differ among various methods due to their differing learning strategies. Methods like BYOL, ProPos, and LFSS, which use Exponential Moving Average (EMA) to update the target network, tend to have smoother network changes and thus exhibit overall higher sample stability compared to IDFD and SimCLR, which do not utilize EMA. However, as Observation 1 demonstrates, the relatively unstable samples in these methods still exhibit lower accuracy compared to the relatively more stable ones, confirming the applicability and robustness of our findings across methods that differ significantly in their characteristics.\n\n> **Experiments on larger intervals (Weakness 2)**\n\nThank you for your deep thoughts on this work. Following on your suggestion, we conducted experiments on CIFAR-10 and ImageNet-10 to verify whether a larger interval would lead to performance improvement. We chose intervals of the smaller values 5, 10, and the larger value 100 to ensure the thoroughness of the experiments. The results are below：\n\nCIFAR-10:\n\n| Interval  | NMI  | ACC  | ARI  |\n|-----------|------|------|------|\n| 1 epoch   | **87.2** | **93.4** | **86.6** |\n| 5 epochs  | 85.8 | 92.2 | 84.4 |\n| 10 epochs | 84.4 | 91.3 | 82.7 |\n| 100 epochs| 80.4 | 88.5 | 77.7 |\n\nImageNet-10:\n\n| Interval   | NMI  | ACC  | ARI  |\n|------------|------|------|------|\n| 1 epoch    | **85.6** | **93.2** | 85.7 |\n| 5 epochs   | 84.1 | 91.8 | 84.8 |\n| 10 epochs  | 85.5 | 92.5 | **86.1** |\n| 100 epochs | 82.0 | 89.1 | 81.2 |\nWhen the interval is set to 5 or 10 epochs, the clustering performance remains strong, albeit with a slight decline. However, when the interval increases to 100 epochs, the drop in model performance becomes more pronounced. We believe that when the interval is 1 epoch or a few epochs, sample stability can effectively reflect the training quality of the samples; that is, hard samples tend to be unstable. In contrast, with larger intervals, the gradual optimization of sample features during training also becomes a contributing factor influencing sample stability.\n\n> **Advice on hard sample strategy (Weakness 3)**\n\nSince unstable samples can be considered hard samples that are difficult to correctly identify, **LFSS can also be viewed as a method for improving performance in hard sample mining**. Particularly, in our cluster-level loss (Eq. (8)), we exclude unstable samples (hard samples) to ensure more accurate cluster centers for contrastive learning. Without this loss, performance on various metrics for CIFAR-10 would decrease by 4-7%, as described in ablation study. We look forward to further addressing unsupervised hard sample mining from the perspective of sample stability. We are grateful for your kind advice. \n> **Visualization on unstable and stable samples (Weakness 4)**\n\nWe sincerely appreciate your suggestions for improving this work. We have visualized the distribution of embeddings for both stable and unstable samples in https://anonymous.4open.science/r/ICML25-0E61/1.2.png. We perform t-SNE on final embeddings on CIFAR-10. We mark top 10% stable samples in yellow,  top 10% unstable samples in purple and other samples in green. Even though the number of stable and unstable samples selected is similar, there appear to be more purple dots (unstable ones) on the figure intuitively. This indicates that **the distribution of unstable samples is more dispersed, while the distribution of stable samples is more concentrated, even resulting in many overlaps**. This suggests that stable samples tend to be representative samples with similar properties and characteristics. Meanwhile, we also observe that **isolated samples in the gaps between clusters often belong to the unstable ones, which highlights their atypical features**.\n\nWe are very grateful for your sincere suggestions in improving our work as well as for your recognition of our effort. Thank you.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "NA", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "### Strengths\n* The paper is clearly structured, and its motivation is explained in a way that is easy to understand.\n* The experiments are extensive.\n* The method achieves excellent performance on various benchmarks.\n\n### Weaknesses\n* By definition, Sample Stability is likely to be lower at the beginning of training (when representations are still evolving) and higher toward the end of training (when the model stabilizes). The paper observes that high stability corresponds to higher accuracy. A more detailed visualization (e.g., a histogram) showing how Sample Stability changes over the course of training would strengthen the empirical insights.\n\n* The choice to update the final representation with the latest epoch only seems somewhat ad hoc. It would be informative to investigate updating at larger intervals (e.g., every few epochs) for both the representation update and the calculation of Sample Stability, to test whether this yields more reliable estimates or improved performance.\n\n* Figure 2 suggests that samples with lower Sample Stability tend to be harder examples. One question is whether applying a hard example mining strategy could further boost performance—e.g., by giving these challenging samples additional training updates or specialized handling.\n\n* It would be helpful to visualize how different types of samples (e.g., high vs. low stability, easy vs. hard) are distributed in a latent space via scatter plots or other methods. Such a visualization might reveal meaningful structure and give deeper insights into how and why certain samples remain unstable.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JxiSq57JOh", "reviewer": "Reviewer_9ozz", "summary": "This article introduces LFSS, a novel deep clustering method that leverages sample stability, which is measured as the cosine similarity between representations across consecutive training epochs as a supervisory signal. The authors motivate the approach by showing that samples with unstable representations tend to be misclustered and are harder for networks to memorize. Based on extensive empirical observations across various datasets and multiple baselines, the paper proposes two key contributions: an instance-level loss that directly penalizes representation instability and a cluster-level loss that improves the quality of cluster centers by excluding the most unstable samples. The method is integrated into a self-distillation framework with noise, and experiments show significant improvements over baseline unsupervised methods and state-of-the-art deep clustering techniques. Necessary experiments were conducted to validate the claim of the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your important questions. Below is our response:\n> **Hyperparameter Sensitivity (Weakness 1)**\n\nActually, we provided an analysis of the sensitivity of four hyperparameters on CIFAR-10 in Appendix E. Here, we further provide a related analysis on three datasets: CIFAR-10, CIFAR-20, and ImageNet-10, along with a guideline to assist readers in applying the method to other datasets. The results are at https://anonymous.4open.science/r/ICML25-0E61/2.1.png for unstable ratio δ, noise intensity σ, balance parameter λ and https://anonymous.4open.science/r/ICML25-0E61/2.2.png for warmup epoch number η. \n\n**Unstable ratio δ**. Typically, a smaller δ is more effective, while a larger δ may exclude many meaningful representations from contributing to the clustering centers, thereby reducing model performance. **In LFSS, we fix δ at 0.1**.\n\n**Noise intensity σ**. Adding noise improves the model's robustness. On different datasets, varying the noise level results in performance differences of about 5%, indicating that both excessively high and low noise intensity do not harm model training. In our experiments, we set σ to 0.01 or 0.001.\n\n**Balance parameter λ**. The parameter λ balances the different terms in Eq. (10), determining the contribution of instance-level and cluster-level losses to model training. An overly small λ can lead to performance degradation due to the insufficient contribution of these losses. **In LFSS, we fix λ at 0.1**.\n\n**Warmup epoch number η**. We set η to 0, 50, 200, 500, and 800 to evaluate the impact of introducing cluster-level loss at different training stages. Early introduction (η = 0 or 50) reduces clustering performance due to generating cluster centers of low quality, while later introduction, when representative cluster centers are well established, achieves better final performance. In our experiments, we set η to 200 or 500.\n\n> **LFSS on imbalance datasets (Question 1)**\n\nTo evaluate the robustness of LFSS on imbalanced datasets, we conducted experiments on CIFAR-10, CIFAR-20, and STL-10, using an imbalance ratio of 10. We adoped ResNet-18 with a batch size of 256 and trained for 1000 epoch. Besides the three metrics in the paper, we also used **CAA (class-averaged accuracy)** for evaluating performance on imbalanced datasets. The results are at https://anonymous.4open.science/r/ICML25-0E61/2.3.png. **Compared with state-of-the-art methods, e.g., IDFD, CoNR, ProPos, and DMICC, our method achieves the best performance on all three datasets on all four metrics with the maximum improvement over the second-best method is 7% on ACC**. Although none of the methods specifically address the handling of class imbalance, our approach demonstrates greater robustness on imbalanced datasets compared to other methods. \n\n> **LFSS on non-image data (Question 1)**\n\nConsistent with our comparison methods, we use image datasets to validate the effectiveness of our method. But our method is not limited to image datasets. As suggested, we further evaluated our method on two text datasets GoogleNews-T and GoogleNews-S. We replaced the ResNet-18 in LFSS with an MLP. We choose distilbert as the backbone to extract features from original texts.  We compared LFSS with text clustering methods such as BoW, TF-IDF and HAC-SD. We use the same experimental setup as ours to reproduce IDFD and ProPos. The results are at https://anonymous.4open.science/r/ICML25-0E61/2.4.png. LFSS shows **improvements of about 2-3% on all metrics compared with the second best**. Despite not specifically considering the data characteristics and design schemes for text clustering, our method still managed to deliver highly competitive experimental performance. This underscores the superiority of our approach and its robustness across different types of data.\n\n> **Computional cost (Weakness 2, Question 2)**\n\nIn fact, adding a predecessor network into BYOL framework does **not introduce significant computational overhead, as this network is updated by directly copying the weights from the previous epoch, without requiring gradient computation or backpropagation**. We compared the runtime (**minute**) and memory usage (**MB**) for three methods based on the BYOL framework: BYOL, ProPos, and LFSS, as follows:\n|      | BYOL|Propos| LFSS  |\n|------|-----|------|-------|\n|Time  |327.8|416.8 | 456.9 |\n|Memory|4013 | 4137 | 4149  |\n\nIt can be seen that LFSS's computational resource usage is slightly higher than other methods but remains within an acceptable range, all within the same order of magnitude.\n\n> **Computional complexity of LFSS (Question 3)**\n\nThe computional complexity of LFSS loss is $O(N^2*d)$, where N is the batch size and d is the embedding dimension. We do not consider the computional complexity of the backbone or cluster assignment, as they are independent of our method. \n\nWe hope the above response is satisfactory, and we thank you for the time and effort you devote to this review.", "claims_and_evidence": "The paper is clear about its motivation and claims with a good presentation, sufficient significance, quality, and originality.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the claim of the paper.", "theoretical_claims": "The paper is clear about its theoretical claims.", "experimental_designs_or_analyses": "Necessary experiments were conducted to validate the claim of the paper.", "relation_to_broader_scientific_literature": "The article tries to formulate a novel deep clustering algorithm which is significant in the deep clusering field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n- The paper introduces the interesting concept of “sample stability” as a proxy for training progress and memorization difficulty in unsupervised learning. This perspective is both intuitive and well motivated.\n- The experimental evaluation is thorough. The authors provide extensive results across a range of datasets and compare against multiple competitive baselines. Ablation studies further isolate the contributions of each component (instance-level loss, cluster-level loss, and noise-based self-distillation).\n\n**Weaknesses:**\n- While the method shows improved performance, it introduces several additional hyperparameters (e.g., the unstable ratio δ, warm-up epoch number η, noise intensity σ). A more detailed discussion of the sensitivity to these hyperparameters, or guidelines for tuning them across different datasets, is needed.\n- The use of multiple network components (online, target, and predecessor networks) might introduce additional computational cost. The paper should provide a clearer discussion on the computational efficiency or training time compared to baseline methods.", "other_comments_or_suggestions": "1. Consider adding a more detailed analysis or visualization of the hyperparameter sensitivity.\n2. Discuss the computational cost more explicitly, including any trade-offs in terms of training time or memory requirements.", "questions_for_authors": "1. How robust is the LFSS framework when applied to datasets with significantly different characteristics (e.g., highly imbalanced data or non-image data)?\n2. How does the additional computational overhead of maintaining a predecessor network compare with the performance benefits, especially in large-scale scenarios?\n3. What is the computational complexity of LFSS?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UnUVOcvwvI", "reviewer": "Reviewer_ZpFb", "summary": "This paper proposes a deep clustering method by identifying hard samples based on their stability during the training. By taking the sample stability into consideration, the proposed method improves instance-level representation learning and cluster-level grouping, leading to superior clustering results on five image datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of this work. We highly value the insightful comments you have provided, and below we offer our responses. \n\n>　**Essential references not discussed**\n\nThank you for providing the two recent related articles. After careful reading, we believe that these two papers are highly relevant to our work and should be cited in the introduction and related work sections in our work. The first article discusses the approach of solving deep clustering from a prior perspective, which is relevant to our discussion on constructing different supervisory signals based on prior knowledge. The second article innovatively proposes an unsupervised hard sample mining method and employs external user interaction to enhance clustering performance. We enhance clustering performance by utilizing sample stability, which can also be regarded as an indirect approach to hard sample mining. We will cite the two relevant paper in the final version. \n\n>　**The application of K-means is limited when it comes to large datasets (Weakness)**\n\nIndeed, applying K-means to obtain clustering assignments at every epoch can be very time-consuming, especially on large datasets. **However, the cluster-level loss for LFSS is only enabled after η epochs warmup, which means that the network can produce meaningful embeddings and does not necessarily require cluster assignments to be updated in every epoch**. We can reuse the clustering assignments obtained from a single K-means run across multiple epochs. We evaluate model performance under different cluster assignment update frequencies on CIFAR-10 in https://anonymous.4open.science/r/ICML25-0E61/3.1.png. Although the strategy of updating at every epoch achieves the best performance, the performance drop is marginal when the epoch interval is set to 10, 50, or 100. This allows us to significantly reduce training time. Therefore, we can achieve near-optimal performance while saving time by reducing the execution frequency of K-means.\n\n> **Hyperlink to observations (Suggestion)**\n\nWe will add hyperlinks in Introduction section that point to specific observations to enhance readability in the final version.\n\n> **Comparison with confidence-based sample selection (Question 1)**\n\nThe distinction between these two approaches lies in their application scenarios. The confidence-based sample selection method typically requires pre-training a representation model first, then training the clustering head with high-confidence pseudo labels to improve performance. This approach's success hinges on the model's ability to generate good representations and pseudo labels. In contrast, our method is a training-from-scratch approach that can be used for representation model training without the need for pre-training. \n\nWe apply the self-labeling method in SCAN [1] on the final model of ours on CIFAR-10, using a threshold of 0.99 to filter out low-confidence pseudo-labels. Some of the low-confidence samples excluded by this method overlap with the unstable samples we selected from the final model. We count the number of these samples and provided their accuracy under K-means clustering as below:\n\n| Sample Type    |Accuracy|Quantity|\n|----------------|--------|--------|\n|Unstable Samples| 0.791  | 5998   |\n|Low Confidence  | 0.934  | 12008  |\n|Intersection    | 0.809  | 1273   |\n\nThe accuracy of the selected unstable samples is lower, while the accuracy of the low-confidence samples is close to the overall accuracy of the global dataset. **This demonstrates that the sample stability-based method can identify misclustered samples more effectively than confidence-based approaches**.\n\n> **Question on self-distillation with noise strategy (Question 2)**\n\nself-distillation with noise strategy is not a novel contribution in this work and it is a commmonly used trick. This reparameterization trick is oriented in VAE [2] and we should cite it in our paper.\n \n> **Minor in Fig.3**\n\n We sincerely thank you for pointing out this typo. The framework of LFSS is built upon BYOL, where the predictor head is connected after the online network. We will revise this in the final version.\n\nHeartfelt thanks for your efforts in this reivew.\n\n[1] Scan: Learning to classify images without labels, ECCV, 2020.\n\n[2] Auto-encoding variational bayes, ICLR, 2014.", "claims_and_evidence": "The proposed method is grounded on the observations of unstable samples. These observations are proven on different datasets with various methods, making this work technically sound. The utilization strategy for these unstable samples is also reasonable, with ablation studies demonstrating its effectiveness.", "methods_and_evaluation_criteria": "The proposed method is technically sound, and the evaluations on five datasets are convincing enough to demonstrate the effectiveness of the method.", "theoretical_claims": "This work has no theoretical claims.", "experimental_designs_or_analyses": "The proposed method is evaluated on five classic and two large-scale image clustering datasets. The performance comparisons are fair to demonstrate the superiority of the method. Ablation studies and parameter analysis are also conducted to further interpret the effectiveness and robustness of the proposed method.", "relation_to_broader_scientific_literature": "This work might inspire researchers interested in developing clustering methods for other forms of scientific data.", "essential_references_not_discussed": "The authors are encouraged to include a recent deep clustering survey (A survey on deep clustering: from the prior perspective, Vicinagearth 2024), and a recent deep clustering method that also focuses on mining hard and valuable samples (Interactive Deep Clustering via Value Mining, NeurIPS 2024) in the related work section.", "other_strengths_and_weaknesses": "This paper reveals the strong correlation between samples' stability and their clustering accuracy. This finding could inspire future work in handling hard samples in deep clustering. \n\nThe proposed enhancement strategy using sample stability generalizes to different representation learning methods.\n\nApplying k-means to compute the cluster centers could limit the scalability of the proposed method on large datasets.", "other_comments_or_suggestions": "When referring to the three observations in the introduction section, the authors could add hyperlinks to help locate the details to improve readability.", "questions_for_authors": "I expect the authors to respond to my previous concerns. In addition, while the proposed stability measure could help identify hard samples, how is this criterion different from commonly used confidence-based sample selection methods? For example, the authors may compare the intersection between hard samples selected according to different criteria and strategies.\n\nIs the self-distillation with noise strategy a novel contribution of this work? It seems that augmenting with Gaussian noise is a commonly used trick. If this part is not novel, please correctly cite the corresponding works.\n\nMinor: Is the place of the predictor head correct in Fig. 3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SJBGW3cbgB", "reviewer": "Reviewer_vooh", "summary": "A deep clustering method based on the idea that unstable points, whose representations change a lot each epoch, are more likely to be inaccurately clustered. The main proposals are a loss function to encourage representation stability, and the exclusion of unstable points from training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **On main results (Claims And Evidence)** \n\nWe compared four state-of-the-art (SOTA), including IDFD, ProPos, CoNR, DMICC, the baseline BYOL by reproducing them, with the same experimental setup as ours, including backbone, batch size, number of epochs, etc. These experimental settings can significantly affect the model's performance and lead to unfair comparison if non-unified. For other representative methods, we directly adopted the results in ProPos. Below, we provide further clarification.\n\nAbout the compared SOTA: some existing methods surpass the performance of the SOTA we compared, as shown on the website you provided. However, to ensure a fair comparison, the selected SOTA are all methods based on representation learning for deep clustering. These methods focus on learning discriminative representations and then apply K-means to get the final clustering assignments, consistent with the approach used in LFSS. **Many of the higher-performing methods available on that website rely on leveraging the substantial knowledge embedded in pre-trained large foundation models**. Given that these methods do not learn representation from scratch and benefit significantly from their pre-trained foundations, comparing against them would be inherently unfair. Thus, we chose not to include these methods in our comparison, ensuring our comparisons remain equitable.\n\nAbout the reported results of ProPos: Due to the varying experimental settings of the SOTA methods, **we adopted a unified setup for fair comparison**. We used ResNet-18 instead of ProPos's original ResNet-34 and trained on a single GPU, resulting in slightly lower performance than their original results.\n\nWhy not display all from results from ProPos：In fact, ProPos directly cited results from various papers. While many studies adopt this practice, it can lead to unfair experimental comparisons. For instance, a method trained for 3000 epochs may claim superior performance over one trained for only 300 epochs. Given limited computing resources, **we believe it is essential to at least reproduce the most relevant and competitive SOTA methods to validate performance fairly**.\n\nAll the reproduced results we report are credible. We clearly indicate in the paper which methods are cited and which are reproduced, along with the experimental settings used for reproduction. We sincerely hope that these explanations will help you recognize the validity of our approach.\n\n> **Reducing unstable points (Experimental Analyses)**\n\nWe focused on the top 10% most unstable samples at the 200th epoch on CIFAR-10 and CIFAR-20, using the stability of the sample at exactly the 10th percentile as the threshold for determining instability. In subsequent epochs of training, a significant proportion of these samples surpassed this threshold and became stable. This indicates that **our method can effectively reduce the number of unstable samples**. The results are at https://anonymous.4open.science/r/ICML25-0E61/4.1.png.\n> **Essential References Not Discussed**\n\nBoth papers exclude unreliable pseudo-labels during training, due to inconsistent predictions across consecutive epochs and across multiple models respectively. In contrast, LFSS leverages sample stability at the representation level. It allows for training representations from scratch and can be embedded into multiple frameworks. We clarify that our contribution lies not only in the utilization of sample stability at the represention level or good experimental performance, **but more importantly, in uncovering the relationship among sample stability, clustering prediction and network memorization**. Meanwhile, the two papers you provided are relevant to our work, and we will cite and discuss them in the final version.\n> **About hyperparameters （Weakness）**\n\nDespite multiple parameters, LFSS is stable in parameter choice. Please refer to the first answer to Reviewer 9ozz for details. Thank you.\n> **On excluded points (Q1)**\n\nOnly cluster loss, for more accurate centers. For other losses, we do not exclude them, allowing more samples to contribute to training.\n> **On the order of executing K-means (Q2)**\n\nThe approach of excluding unstable samples first and then applying K-means is actually a similar process to performing K-means first and then excluding unstable samples. We trained LFSS with two methods on CIFAR-10 and recorded the clustering accuracy of the non-excluded samples at different epochs, to reflect the precision of the cluster centers obtained by two methods. The results are at https://anonymous.4open.science/r/ICML25-0E61/4.2.png. Two results are roughly similar, suggesting the two methods can achieve comparable outcomes.\n> **On Fig.4 (Q3)**\n\nSorry for the confusion. To clarify, we presented results at the 600th epochs as an example mid-training. Similar effects are seen at final models in https://anonymous.4open.science/r/ICML25-0E61/4.3.png.\n\nThank you for pointing out the typo and we will revise it.", "claims_and_evidence": "The ablation experiments are convincing, especially the demonstration that LFSS can be used as an add-on in existing methods. I have some questions about the main results. \n\nThe comparison methods in Table 3 are referred to as state-of-the-art, but I am not sure this is the case. Looking [here](https://paperswithcode.com/task/image-clustering) it appears that there are several methods that perform better than the ones you compare against, and also better than your method.\n\nThe results for ProPros reported in their paper are higher than you have in Table 3, and in fact higher than your method. Can you explain this difference?\n\nIf you are displaying results as reported as baselines in Huang et al. (2023), why not display all baseline results, or at least the best-performing ones?", "methods_and_evaluation_criteria": "The method makes intuitive sense and appears to be implemented in a reasonable way.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The results are mostly good. I have some questions about the scores for comparison methods, see above. I would also be interested to see whether LFSS actually reduces the number of unstable points.", "relation_to_broader_scientific_literature": "Relation to scientific literature is adequate.", "essential_references_not_discussed": "A similar method is proposed in [1]. That method excludes samples that don't receive the same aligned cluster label across epochs. It differs from your method in that it takes the stability of cluster labels, rather than embeddings, but is also based on the idea that unstable points are less likely to be correctly clustered. Another paper with a similar idea is [2], which trains multiple models in parallel, and only trains on \"confident\" points that are clustered the same way in every model. I believe the first paper is relevant in the context of the claim that few existing methods have used instance-level stability for deep clustering. For the second, I leave it up to the authors as to whether it is sufficiently relevant.\n\n[1] Mahon & Lukasiewicz, 2023, Efficient deep clustering of human activities and how to improve evaluation.\n[2] Mahon & Lukasiewicz, 2021. Selective Pseudo-label Clustering.", "other_strengths_and_weaknesses": "The presence of three different hyperparameters is a drawback, as they it be slightly cumbersome to choose appropriate values in a given application.", "other_comments_or_suggestions": "line 357, LHS: \"are representative approaches recently\" -> \"are recent representative approaches\"?", "questions_for_authors": "are excluded points excluded from all three loss functions or just the cluster loss?\n\nlines 253-255, RHS: why do you conduct k-means before excluding unstable points?\n\nwhy is Figure 4. based on Epoch 600 instead of the final trained model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Regarding this paper, all four reviews are positive. According to the reviews, the strengths of the work are as follows:
* The motivation is clear, and the idea of "sample stability" for clustering is interesting.
* The experimental evaluation is thorough, and the proposed algorithm outperformed the competitors in most cases.

I suggest the authors include the results and discussion produced during the rebuttal and discussion periods in the revised paper.
Decision: Accept (poster)