Breaking the Reclustering Barrier in Centroid-based Deep Clustering
Authors: Lukas Miklautz
Timo Klein
Kevin Sidak
Collin Leiber
Thomas Lang
Andrii Shkabrii
Sebastian Tschiatschek
Claudia Plant
Venue: ICLR 2025 Poster (2025)
Keywords: deep clustering
unsupervised learning
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Centroid-based deep clustering suffers from performance plateaus, and we introduce a method to break through these plateaus.

Abstract: This work investigates an important phenomenon in centroid-based deep clustering (DC) algorithms: Performance quickly saturates after a period of rapid early gains. Practitioners commonly address early saturation with periodic reclustering, which we demonstrate to be insufficient to address performance plateaus. We call this phenomenon the “reclustering barrier” and empirically show when the reclustering barrier occurs, what its underlying mechanisms are, and how it is possible to Break the Reclustering Barrier with our algorithm BRB. BRB avoids early over-commitment to initial clusterings and enables continuous adaptation to reinitialized clustering targets while remaining conceptually simple. Applying our algorithm to widely-used centroid-based DC algorithms, we show that (1) BRB consistently improves performance across a wide range of clustering benchmarks, (2) BRB enables training from scratch, and (3) BRB performs competitively against state-of-the-art DC algorithms when combined with a contrastive loss. We release our code and pre-trained models at https://github.com/Probabilistic-and-Interactive-ML/breaking-the-reclustering-barrier .

Published as a conference paper at ICLR 2025
BREAKING THE RECLUSTERING BARRIER CENTROID-BASED DEEP CLUSTERING
IN
Lukas Miklautz Timo Klein Kevin Sidak Collin Leiber
Thomas Lang Andrii Shkabrii' Sebastian Tschiatschek 1.15 Claudia Plant 1:1,
1 Faculty of Computer Science, University of Vienna, Vienna, Austria
2 UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria
3 Department of Computer Science, Aalto University, Espoo, Finland
4 Department of Computer Science, University of Helsinki, Helsinki, Finland
5 ds:Uniiie, Vienna, Austria
Joint first authors, & Joint last authors
Firstname last
collin, lea.iennaibbrraaaltooo
ABSTRACT
This work investigates an important phenomenon in centroid-based deep clus- tering (DC) algorithms: Performance quickly saturates after period of rapid early gains. Practitioners commonly address early saturation with periodic reclustering, which we demonstrate be insufficient address performance plateaus. We call this phenomenon the "reclustering barrier" and empirically show when the reclustering barrier occurs, what its underlying mechanisms are, and how it is possible to Break the Reclustering Barrier with our algorithm BRB. BRB avoids early over-commitment initial clusterings and enables con- tinuous adaptation to reinitialized clustering targets while remaining conceptu- ally simple. Applying our algorithm widely-used centroid-based DC algo- rithms, we show that (1) BRB consistently improves performance across a wide range of clustering benchmarks, (2) BRB enables training from scratch, and (3) BRB performs competitively against state-of-the-art DC algorithms when com- bined with a contrastive loss. We release our code and pre-trained models at o
barr
INTRODUCTION
110%
Accuracy
"To live is change; to be perfect change often" (Nwmman 1845): Though not originally about ML, this proverb underscores the importance of adaptabil- ity. This adaptability is critical for the success of machine learning algorithms, especially during train- ing. It is particularly crucial in clustering - a family of unsupervised learning algorithms that partition samples into multiple groups based on their similar- only assigning
IDEC Recluster IDEC+ BRB (ours)
200
400
Epochs
Figure 1: Breaking the reclustering bar- rier with BRB. Uniting the deep cluster- ing method IDEC with BRB (IDEC+BRB) breaks through the performance barrier (dashed line) encountered when just using IDEC or combining with (IDEC+Recluster) Performance is measured by clustering accu- racy on USPS.
ity. Deep clustering (DC) involves samples to particular group but also jointly learning a representation the data using deep learning. As this representation improves, the algorithm enhances its ability to identify clusters in the data, requiring it to adjust its assignments frequently.
In this paper, we analyze whether centroid- based DC algorithms can sufficiently adapt during training to facilitate further improvements. The central finding of our analysis is that they cannot and that commonly used techniques, such as periodically reclustering all samples in the embedded space with k-Means, are insufficient to address this issue. Specifically, we find that reclustering by itself

---

Published as a conference paper at ICLR 2025
is not enough to enable late- aate-training improvements to the clustering because it fails to change the structure of the underlying embedded space. A bad initial representation or clustering exacerbates the effect and potentially decreases final performance by more than 10%, our experiments demonstrate. When a clustering algorithm cannot improve late in training despite frequent reclustering, we refer to this as the reclustering barrier.
But can we overcome the reclustering barrier? Yes, as we show in Figure for the centroid-based DC method IDEC with reclustering, without reclustering, and with our proposed approach. Our novel method, BRB, can Break the Reclustering Barrier (BRB). After 100 epochs, BRB breaks through the plateau that cannot be surpassed by the competitors even after 400 additional epochs of training. By analyzing the evolution of the latent space during clustering, we identify a lack of change in the embedding as the primary cause of reclustering's 's limited effectiveness. BRB combines reclustering with a carefully designed weight reset, leveraging synergistic effect between these algorithmic components. This synergy creates a virtuous cycle between generating new clustering targets and adapting to them, enabling our method escape suboptimal performance plateaus, At the same time, BRB is easy to implement and compatible with a wide range of centroid- based DC algorithms.
Our experiments show that despite its simplicity, BRB improves the performance of the widely-used centroid-basee DC algorithms DEC (Xie et al., 2016), IDEC (Guo et al., 2017). and DCN Yang ct al. 2017) across diverse set of benchmarks. When combined with contrastive learning and self-labeling (Gansbeke et al., 2020), BRB even pushes DEC, IDEC, and DCN to state-of-the-an performance. With BRB, these algorithms find good clustering solutions even without the usual pre-training, effectively escaping the reclustering barrier. To summarize our contributions:
We propose BRB, a novel algorithm that can break through existing performance plateaus in centroid-based deep clustering.
We empirically identify the underlying causes of the reclustering barrier and explain the success of BRB: It preserves the variation within clusters in early training and increases exploration of diverse clustering solutions.
We apply BRB on top of several deep clustering algorithms, yielding robust performance improvements across a wide range of datasets and setups. With contrastive learning and self- laheling, BRB achieves competitive performance compared to state-of-the-art approaches.
Research focus Centroid-based deep clustering is highly active research area intersecting with various fields. For instance, the current state-of-the oft-rr--rttttttto image clustering technique SeCu (Qian 2023) is centroid-basee
Self-supervised learning uses prototypes, like centroids, for representation 2018 2020 2021 ). This study concentrates on established centroid- based DC algorithms, specifically DEC. IDEC, and DCN, which are widely used in diverse domains such as biology (Li et al. 2220 Hu et al. 2021 Luo et al. 2021). medicine (Wachowiak) (et al. 2019 Kalweit et al. 2023) or finance (Choi & Renelle 2019). These algorithms also form the foundation for several recent DC methods (Qian 2023 Li et al. 2019). Enhancing DEC, IDEC, and DCN could significantly impact multiple research fields and accelerate progress on downstream applications. With this context, we proceed to discuss some background and related work.
learning (Asano et al. 2019 Caron et al.
2 BACKGROUND AND RELATED WORK
Deep clustering (DC) is the combination of clustering and deep learning. Popular deep learning approaches for DC include self-supervised networks such autoencoders (AEs) or SimCLR (Chen et al. 2020) as they can be trained without given labels. The idea of centroid-based DC is to obtain an initial clustering, typically by running the clustering algorithm k-Means in the latent space of a pre-trained network. The algorithm proceeds to update this result by optimizing neural network parameters and centroids M through a loss of the form L(O, M) = X L. (0) + (2 M). where Le refers to the clustering loss and LSSL refers to the self-supervised loss (e.g., contrastive or reconstruction loss, for SimCLR and AE respectively). Here, the embedding and the clustering can be updated simultaneously utilizing simultaneous optimization is DEC (Xie et al., (016)) It uses a kernel based on the Student's s t-distribution to minimize the Kullback-Leibler divergence between the data distribution and an auxiliary target distribution, which can be trained end-to-end. As Lsst. is not used during the clustering optimization of DEC, distorted embedding could occur (Guo al. 2017). This issue is
iteratively (Zhou et al. 2022). A well-known representative
2

---

Published as a conference paper at ICLR 2025
tackled by IDEC (Guo al. 2017). which concurrently optimizes Le and LSSL. DCN YYng et al. 2017) pursues an iterative optimization, where the embedding is frozen when the clustering is updated, and vice versa. Since the cluster centers not learned via the neural network but explicitly updated, is feasible to use k-Means to obtain non-differentibble hard cluster labels. The well-established algorithms DEC, IDEC, and DCN are the building blocks for number of follow-up works and have diverse applications, making them a natural choice for this study. We provide more details on these algorithms in Appendix [.1
Contemporary DC methods broadly fall into two groups. The first group is focused on image clustering (Gansbeke al. 2020; Dang al., [021] Li et al. 2021 Zhong al. 2021 Qian 2023) and leverages domain knowledge, such as augmentations. The second group consists of data-agnosti
DC methods, like DEC, IDEC, and DCN or (Miklautz et al.) 2021 Leiber et al. 2021 Mahon & Lukasiewicz, 2024) that do not rely on augmentations. For more detailed overview of DC
algorithms, see (Ren et al. 2022 Zhou al. 2022 Lu et al. 2024.
3 PROBLEM SETUP
Given unlabeled dataset X' = {x:]__1 containing n instances RD, our objective is to partition X' into & distinct clusters. Each cluster is represented by centroid μ4j Rd, with j {I...., k). This partitioning requires learning assignments st,j E {0, 1} of instances x; to centroids μ4 subject to Σ 8i, = 1 for all i. Additionally, deep clustering methods aspire to learn ""luster-friendly"" (Yang et al. 2017) latent feature space H S IR4, where d < utilizing non-linear encoder Jo: X + H. For example, k-Means friendly latent space consists of compressed spherical clusters (low intra-cluster distance) that are separated (high inter-cluster distance).
Many methods utilize supersud auxiliary objective prevent trivial solutions in DC, c.g.. by including a reconstruction or contrastive loss in addition to the clustering loss. This auxiliary objective is applied in task-dependent output space Z and is often learned with separate task head : H + Z. In the case of reconstruction, the function g serves as a decoder network, where Z C R corresponds the space of reconstructed data points. For contrastive learning, I acts projector network (e.g., a shallow MLP), where 2 is the projector's output space.
A deep centroid-based clustering algorithm outlined above can be formalized as the itera- tive application of function C: (0,, S,, M,)
65-
S0441, 1 Here, 0, E é represents the network parameters step f, M, R
55-
the centroid matrix and S, [0, 1 ijnx matrix encoding the cluster assignments for each sam- ple. The function C optimizes clustering objec- tive (O., S,, M,): θ x [o, 1jmx* x kx -> RR utilizing gradient-descent optimizer. Its out- put is improved parameters d.+1 as well as new assignments S and centroids M,41- Typ- ically, is iterated until a local minimum of LOO,S, S, M) is reached.
cp
Inter cD iataa
8
& 8 g
CL change
4 THE RECLUSTERING BARRIER
200
DCN+Recluster
(ours)
Before introducing BRB in Section we ex- amine why reassigning all samples in the em- bedded space via, e.g., k-Means by itself fails to enhance performance beyond a certain threshold during DC training. We refer to the aforemen- tioned procedure as "reclustering" and assess its effect on the embedded space using intra-class distance (intra-CD) CD) to measure variation within ground truth classes and inter-class distance (inter-CD) to assess separation between ground truth
Figure Why does reclustering not work? DCN with reclustering (orange line) shows mini- mal changes compared to unmodified DCN (blue line) in the embedded space (intra/inter-CDD or ex- plored clusterings (CL Change) for GTSRB, This effect generalizes to other datasets (Figure
3

---

Published as a conference paper at ICLR 2025
i
Pre-BRB
2
BRB: weight reclustering
Post-BRB
Centroids Voranal Colls
NMN
870
53
56 95 15
57 35
34
Clustering Epochs
Figure 3: The effect of BRB during training of DEC. (1) Pre-BRB: Before BRB is applied, DEC strongly compresses (low variation within clusters) the five shown clusters and mixes different ground truth classes of USPS (indicated by color), leading to performance plateau (cf. epochs 46 to 50 in the NMI plot). (2) BRB applies weight reset to incrasse the variation in the clusters with subsequent reclustering, which leads, after a small performance drop in NMI (epoch 51). to steep increase (epochs 52 to 59). (3) Post-BRB: After applying BRB, the clusters are compressed again by DEC until BRB is applied another time.
classes; this follows the embedding space analysis in (Lehner et al. 2024). We quantify the impact of
reclustering on the clustering targets by cluster label change (CL Change) between epochs in terms of the normalized mutual information (NMI) (Kaasseth 1987) as (1 NMI(S,, S,-1)) 100. value of means no change between two consecutive clusterings, while a value of 100 indicates complete change. Figure 2 tracks these three metrics and the clustering accuracy (Yang al. 2010)) with BRB application every T 20 epochs on the GTSRB dataset
Although reclustering marginally enhances clustering accuracy (orange line), does not substantially alter the embedded space: the intra-CD and inter-CD are almost the same at the end of the training, irrespective of whether reclustering is applied. Correspondingly, the CL Change is largely unaffected by the reclustering, indicating only marginal differences in the obtained clustering solutions. These findings suggest that if want further improve performance, we require larger and better- structured changes in the embedded space to systematically impact the clustering. In contrast to only reclustering, BRB induces such changes and is able improve the clustering performance (black line in Figure Figure 3 schematically illustrates the changes caused by BRB in the embedded space. we describe how our approach, BRB, effectively introduces such changes.
In Section
Reclustering Barrier
Strongly compressed and well-separated clusters in the embedded space prevent reclustering from discovering new solutions late training. Figure P provides an intuition.
5 OUR APPROACH BRB
Our goal with this work is to overcome the reclustering barrier in a centroid based deep elustering algorithm C. To this end, we establish three desiderata for our approach:
1. Explovation:
The proposed modification should increase the exploration of clustering solutions. Knowledge preservation: The knowledge encoded network parameters 0 should be preserved while exploring wider range of solutions.
Generality: The proposed modification should be applicable to existing DC approaches.
Section 6.1
describes all datascts used in this work in detail.
4

---

Published as a conference paper at ICLR 2025
We now define modified algorithm
CBRB: (0,, M2) + C( (0z), (S,, M₁),,
consisting of two modifications lor and as well as optional momentum resets, each impacting different components of the deep clustering algorithms.
Weight resets Section motivates the need for structured changes in the embedded space to enable effective reclustering. BRB achieves such changes by applying a soft weight reset (Ash & Adams 2020 D'Oro et al. 2023) to the the non-linear encoder fo,- In particular, we obtain modified
parameters 0; through a convex combination of the network's 's prior parameters and freshly sampled weights (D'Oro al. 2023). denoted bu (0)):
+ (1- a)Ø',
(1)
where 0j are layer i's weights at step t, ø' is sampled from the initial weight distribution and (0, 1) is a hyperparameter. This strategy balances the need for inducing changes in the embedding function fo, escape local minima while preserving previously learned information.
While Equation [ outlines a particular reset strategy, our approach is capable of accommodating different reset techniques, such as adding noise (cf. Section 663) or layer-wise resets (Alabdulmohsin We discuss our choices in the paragraph "Implementation details" later in this section.
et al. 2021
Reclustering Applying a weight-reset Lu (0,) to the encoding function fe, induces a shift in embeddings: So (X) H. + H. fa. (X). Consequently, the centroids M, at clustering step t do not accurately characterize the cluster centers for H,, We address this issue by proposing to recalculate the centroids based on the updated embedding matrix H,:
S. M, (S,, M,, 0,) = recluster (Jg, (X))
Here, M, denotes the centroid matrix obtained from reclustering with the perturbed embeddings fg. (X) = H. and S, E [0, 1]m*k represents the corresponding assignments. Any centroid-based method, such k-Means (Lloyd 1982) or k-Medoids (Rdusseeun & Kaufman, 1987). is applicable for reclustering. However, we choose k-Means in BRB for two key reasons. First and foremost, it aligns with the spherical cluster model and initialization used by DEC, IDEC, and DCN, e.g., DCN already uses k-Means in its assignment step. Second, our ablation study (Appendix H. 1. shows that k-Means leads to better exploration clustering solutions and improved performance compared to the assignment procedures of DEC and IDEC.
Momentum resets DEC and IDEC parameterize and iteratively refine cluster centers through gradient descent. However, integrating comenntum-aased optimization with weight resets and reclustering may introduce misalignment between momentum terms and re-calculated centroids, for which we provide ablations Appendix H.2 To mitigate this, we re-initialize the momentum terms for the cluster centers to zero after reclustering. This aligns gradient steps and recalculated centroids, preventing the moment estimates from diverging and harming task performance (Lyle et al. 2023). For DCN, which does not parameterize centroids, momentum resets are unnecessary.
Implementation details The pseudo-code in Algorithm I shows how BRB is added to the training loop of an exemplary DC algorithm, highlighting its straightforward implementation and minimal overhead. We use four key implementation details for BRB that we highlight in the following.
Resetting the encoder's final embedding layer can adversely affect performance due to excessive perturbations in the latent space (see Appendix H. We observe a similar phenomenon in deep CNN architectures like ResNet (He al. (016), where resetting all layers also induces excessive perturbations (see Appendix HH.3). Our solution is to only reset the last ResNet Block and the MLP encoder. We (re- Jinitialize all weights using Kaiming uniform initialization (He et al. 2015), which is Pytorch's 's default (Paszke et al. 2019).
(2) We introduce a reset interval hyperparameter T and reset only every T-th epoch. For the reclustering step, use the k-Means algorithm (Looyd 1982) due to its simplicity,
5

---

Published as a conference paper at ICLR 2025
performance, and speed. Appendix Table 10 shows runtime comparison of different reclustering algorithms, while Appendix Table 12 shows a performance comparison.
(3) We sub-sample the dataset when reclustering to enhance computational efficiency. We use a subsample size of 10,000 for all datasets, which leads to a negligible runtime overhead of approximately 1.1% % when using BRB. In Tables 4
and 9 in Appendix G we analyze the
impact of the subsample size on clustering accuracy and runtime performance in detail.
(4) Using image augmentation a common enhancement for DC algorithms, and we find complements BRB as well. We augmentations for all baselines and BRB by augment- ing each sample and enforcing consistent cluster assignments between a sample and its augmented counterparts (see Appendix for details). We split datasets into grayscale and color (cf. Section 6. applying a random affine transformation for grayscale data and the SimCLR CIFAR augmentations (Chen al. 2020) for color data. Exact augmentation parameters are described in Tables 5 and respectively.
EXPERIMENTS
We integrate BRB into DEC, IDEC, and DCN, presenting experiments that demonstrate the per- formance improvements resulting from this integration for wide range of datasets. The first part of our analysis describes the datasets and setups used in our experiments. We then proceed to benchmark BRB against unmodified versions of DEC, IDEC, and DCN in three scenarios: With and without pre-training and with a contrastive auxiliary task. Then, we dig deeper into the underlying mechanisms of BRB and examine how it is able to break the reclustering barrier. Overall, we find that BRB improves performance in 88.10 % of all runs while incurring a minimal runtime overhead of approximately 1.1 %
6.1 EXPERIMENT SETUP AND DATASETS
We evaluate BRB on eight common deep clustering datasets, divided into two groups, each with
distinct setup. Dataset and preprocessing details are provided in Appendix C The first group (MNIST (LeCun et al. 1998), KMNIST (Clanuwat et al. 2018), FMNIST (Xiao et al. 2017). USPS
(Hull 1994), OPTDIGITS (Alpaydin & Kaynak| 1998), , and GTSRB (Stalkkmmp et al. 2012 forward autoencoder with reconstruction as an auxiliary task, a learning rate of 0.001, and fixed 0.8 and T = 20 across all algorithms and setups. The second group (CIFAR10 and CIFAR 1 00-20 (Krizhevsky et al.) 2009)) consisting of more challenging color image
BRB hyperparameters
datasets, uses SimCLR (Chen et 2020) as an auxiliary task, ResNet18 (He et al. 2016) encoder,
and self-labeling (Gnnseeke et al.) 2020) after clustering as in (Qian) 2023). experimental details.
See Appendix t for full
Given the datasets outlined above, we evaluate DEC, IDEC, and DCN with and without BRB in three scenarios: (1) starting from representation pre-trained with reconstruction as an auxiliary task, (2) training from scratch with reconstruction as auxiliary task, and (3) starting from a representation pre-trained with contrastive learning. Scenario (1) examines whether BRB can break through performance plateaus late in training after pre-training took place. Scenario (2) tests whether BRB is able to improve the exploration expect the baselines to perform poorly due to premature convergence to suboptimal solutions. BRB, however, should leverage resets and reclustering to continuously escape local minima and achieve better performance. In scenario (3), we use contrastive pre-training to scale approach more challenging datasets.
clustering solutions. When training from scratch, we
We measure algorithm performance using three standard clustering metrics (Lu et al. 2024 Huang et al. 2024 Zhou et al. 2025) caccuracy, Adjusted Rand Index (ARI) (Hubert & Arabie 1985). and Normalized Mutual Information (NMI) (Kvalseth) 1987). All metrics scaled between [0, 100), where higher values indicate better performance. Results are reported as the average over ten runs, with standard errors unless otherwise noted. Similar to Lehner al. (2024) we track the separation of ground truth clusters in the embedded space with the inter/intra class distances (inter- Siieer-DDintra-eee -D), which measure the variation between and within classes, respectively. We measure the change in
Appendix G provides detailed theoretical and empirical runtime analysis.

---

Published as a conference paper at ICLR 2025
cluster labels (CL Change) during training using the NMI of cluster labels in subsequent epochs. See Section for details on inter/intra- and cluster label change, and Appendix D a full description of all metrics.
6.2 BENCHMARK RESULTS
Scenario 1: with pre-training Figure 4 compares the performance of BRB against baselines several DC benchmark datasets. As BRB consists of weight reset and subsequent reclustering, provide ablation studies for each component individually. Our results suggest that neither reclustering nor resetting can consistently improve the baseline on their own. For example, on GTSRB, DCN+Reset performs much worse than the baseline (-10%), while DCN+Recluster surpassed by the baseline on USPS.
Relative Change
In contrast, BRB leverages the synergy between weight resets and reclustering and improves per- formance across all datasets. The ablation exper- iments for IDEC and DEC confirm the average improvement BRB over the unmodified base- line and are shown in Appendix Table 13
10%
0%
-10%
MNIST
USPS
OPTDIGITS
FMNIST
KMNIST
GTSRB
DCN+BRB (ours)
DCN+ Recluster
RCN+Reset
Scenario 2: without pre-training DEC,
Figure 4: Improved performance BRB. Rela- tive change in average accuracy against the un- modified DCN algorithm for BRB and impor- tant ablations with pre-training: DCN+Reser refers to performing only weight resets (Eq. []: DCN+ Recluster, refers to only reclustering. Our DCN+BRB with both weight resets and recluster- ing consistently improves performance for DCN.
IDEC, and DCN heavily rely on pre-training to get sufficiently good initial clusters. To our sur- prise, we find that BRB reaches strong levels of performance even without pre-training. Figure shows the relative improvement clustering ac- curacy for DEC and IDEC when compared to the unmodified algorithms, with absolute values shown in Table 13 Without BRB, the perfor- mance of DEC and IDEC plateaus early without pre-training, as highlighted already in Figure for IDEC. BRB significantly improves the performance for both algorithms, sometimes by up to 30%, e.g., when using DEC on OPTDIGITS or GTSRB and for IDEC on USPS. Table 13 underscores that these large improvements hold for DCN as well. Interestingly, BRB improves DEC's accuracy from 61 to 77 on OPTDIGITS without image augmentation, showing it can build strong representations from random initialization. However, we still recommend using image augmentations for generally better results.
DEC+BRB - DEC
* IDEC+BRB IDEC
Scenario 3: with contrastive learning All of the above ex- periments use AE with re- construction loss as an auxiliary task. Table [ shows that BRB's performance improvements also transfer to other auxiliary tasks. Here, we compare DEC, IDEC, and DCN with and without BRB using SimCLR and self-labeling
FMNIST
USPS
FMNIST
USPS
OPTDIGITS
OPTDIGITS
MNIST
KMNIST
GTSRB
KMNIST
GTSRB
Figure 5: BRB enables training from seratch. Relative improve- ments of clustering accuracy for DEC and IDEC when using BRB without pre-training.
(Gansbeke al., 2020); see Section E for implementation details and Table 7 for specific hyper- parameter settings. We follow the experiment setup of Qian (023) by using a ResNet18 for all contrastive experiments and reporting the best performance after self-labeling on the respective test sets of CIFAR10 10 and CIFARIO0 for IDEC and DCN by about 2-3%, while DEC benefits from BRB for CIFARIO by more than 2%. Surprisingly, we find that the established methods IDEC and DCN outperform more recent deep clustering algorithms when combined with a contrastive task and BRB. To the best of our knowledge, this is the first time that such high performance has been reported for DEC, IDEC, and DCN on CIFAR1O and CIFAR 100-20 using a ResNet18. The findings highlight the continued relevance of these "older" methods up to today. For CIFAR100-24 DCN+BRB and IDEC+BRD current state-of-the-art deep clustering method SeCu (Qian 2023). We provide additional results in Appendix Table and results without self-labeling in Appendix Table 17
In Table we see that BRB consistently improves performance
beat the
7

---

Published as a conference paper at ICLR 2025
Table Clustering performance with contrastive task and self-labeling, BRB improves overall performance for most baselines and DCN+BRB outperforms the state-of-the-arr method SeCu on CIFAR100-20. The per-method best results are in bold, and the overall best results are underlined.
CIFAR10
CIFAR100-20
Methods
ACC
NMI
ARI
ACC
NMI
ARI
Pretraining + k-Means
68.97
63.98
40.13
37.22
42.25
14.86
DEC
88.29
80.60
77.23
50.16
51.66
35.37
DEC + BRB
90.57
82.57
81.18
50.46
51.72
35.05
IDEC
88.30
79.50
77.27
52.73
52.79
36.79
IDEC + BRB
90.72
83.26
81.81
55.43
54.81
38.81
DCN
88.55
81.02
78.17
53.27
52.13
37.30
DCN BRB
91.23
83.66
82.42
56.92
56.76
41.15
SCAN Gansbeke et al. 2020)
88.3
79.7
77.2
50.7
48.6
33.3
GCC Zhong al. 2021
90.1
52.3
SeCu
QQian 2023
93.0
86.1
85.7
55.2
55.1
39.7
Additional experimental results BRB introduces two key hyperparameters: reset strength a and reset interval 7'. These parameters require trade-off between sufficient perturbation of the embedded space and algorithm recovery. Our hyperparameter sensitivity analysis in Appendix E.2 demonstrates
BRB's stability for moderate reset strengths (0.8 I a S 0.9) and intervals (T (10, 20, 40}). We use these ranges for all experiments, with both autoencoders and contrastive learning. In Appendix G
we validate that BRB is lightweight and adds only minor computational overhead by performing
detailed runtime analysis. Lastly, we also study the impact of different reclustering algorithms in Table
10
OPTDIGITS
MNIST
6.3 ANALYSIS OF BRB
Accuracy
85.0
Our central hypothesis is that BRB's soft resets and subsequent reclustering struc- turally alter embedded space and enable finding new clustering solutions. We vali-
90.0
82.5
85.0
date this hypothesis by analyzing changes in the embedded space via the inter-class distance (inter-CD), with higher values in- dicating enhanced separation, and measur- ing variation within ground truth classes with the intra-class distance (intra-CDD CD). BRB's impact on clustering assignments is measured using the NMI-based cluster la- bel change (CL Change) metric, detailed in Section 4 BRB's impact differs slightly depending on the dataset, we choose to analyze datasets where it has a substan- tial, moderate, and minor impact mance. Therefore, we focus our analysis of BRB the DCN algorithm and GT- SRB (Figure 2). OPTDIGITS, and MNIST datasets (Figure|6).
20.0
25.0
10.0
20.0
60.ol
Inter
40.0
20.0
40.0
Change
15.0
10.0
5.0
100
200
300
400
20
100
200
300
400
Epochs
Epochs
perfor-
DCN
DCN+ Recluster
DCN BRB (ours)
Figure 6: Effects of BRB. We track average clustering accuracy, inter/intra-class distance (inter/intra-CDD and cluster label change (CL Change).
Early over-commitment For all three considered datasets, BRB increases intra-CD after the first few resets, counteracting premature cluster compression and potential early misassignments. For GTSRB and OPTDIGITS, the intra-CD increases over the whole training, whereas for MNIST, the intra-CD decreases around epoch 60. This leads lower intra-CD at the end of the training compared to the baselines. We find that avoiding cluster compression is more important in early training than later on, judging from BRB outperforming the baselines on MNIST despite lower Intra-CD at the end of training. We analyze the specific behavior of the increasing intra-CD and inter-CD for DCN and Rec+serl on MNIST in more detail Appendix H.10
8

---

Published as a conference paper at ICLR 2025
One possible hypothesis as to why BRB is effec-
800
* Pre-Reset Post-Reiat
tive at preventing early coemimmmmtt is that its resets move samples closer to other centroids,
700
600
allowing for more frequent reassignments dur-
s00
ing training. To verify this hypothesis, we look
400
at the ratio p(x,) = d1 (x.)/d2(x,) (x.) between the distance the closest centroid di (x,) and the second closest centroid d2 (x;) for sample x;. If this ratio is close to I, then x; lies directly be- tween two centroids and has high likelihood of potentially being reassigned. If p(x)) = O, then the sample's s position in the embedded space is directly on top of the centroid that it is assigned to. Figure 7 shows how p is distributed before and after applying BRB during DCN training. We choose DCN for this experiment because its clustering notion relies on the Euclidean dis- tance, which is easy interpret compared the Student-t kernel of DEC or IDEC. Looking at the distribution of distance ratios p in Figure we see that after applying BRB, there are substantially more samples with a ratio p between 0.9 and 1. This means that samples are more evenly distributed between centroids, which in turn facilitates potential reassignments future update steps, effectively preventing early mrrmcommitment.
300
200
100
0.5
0.6
0.7
0.8
0.9
Distance Ratio
Figure 7: Effects of BRB on centroid dis- tances. Distribution of distance ratios p(xx) di (xx)/d((x,) for the distance to the closest di (x;) and second-closest centroid dx;).. BRB prevents early over-commitment by distributing samples more evenly between centroids, allowing for easier reassignment.
Cluster separation Because BRB increases variation within clusters, needs to separate the classes by a higher margin in order to achieve the same better clustering accuracy with less compressed clusters (higher intra-CD). Consequently, Figures [ between ground truth clusters as indicated by a higher inter-CD in the third row of both figures. For GTSRB and OPTDIGITS, the high intra-CD leads to a correspondingly higher inter-CD. For MNIST, the training ends with similar inter-CD values the baselines, whereas the intra-CD at the end of training is lower than the corresponding baseline values.
that BRB leads to higher distances
Exploration of solutions For all three datasets, we observe a significant spike in CL Change after applying BRB, As noted in Section 4 the Recluster step minimally changes the clustering compared to the unmodified baseline. We attribute this to reclustering only reinforcing existing solutions rather than generating substantially different ones, thus only slightly increasing CL Change and clustering accuracy. In contrast to the reclustering baseline. BRB manages to explore more diverse clusterings (higher CL Change) during training while retaining the knowledge of previous clustering solutions.
6.4 ABLATION STUDIES
We conduct four ablation studies to investigate the impact of different components on our method. First, we isolate the role of new cluster labels in shaping the cluster structure while keeping the encoder fixed (Appendix F). Second, we replace soft resets with non- conppersittent noise injections (Appendix FF) Third, we ablate the individual components of Equation (T) in Appendix H.13 Finally, we examine the effect of reclustering with k-Means for DEC and IDEC Appendix 14). Figure shows that both disentangling label changes from weight resets and replacing soft resets with noise leads to performance degradation below 80% accuracy on OPTDIGITS. BRB, on the other hand, maintains its ability to adapt new targets, achieving over 85% accuracy. We find that disentangled labels suffer from insufficient label changes, while the non-persistent noise introduces excessive changes late in training. Neither modification affects intra-cluster or inter-cluster distances. Figures 19a and T9b]show the impact of removing the contraction component, i.c., the left side of Equation and replacing the weight perturbation with sealed Gaussian noise. Removing contraction consistently hinders training, while Gaussian noise degrades performance on USPS. This highlights the positive effect of soft resets on gradient updates, as observed in supervised learning (Ash & Adams, 2020). Furthermore, Appendix [4.14] investigates the implications of reclustering with k-Means for DEC and IDEC. Unlike DCN, which naturally generates assignments with k-Means, DEC and IDEC initially generate soft assignments that are subsequently hardened. Tables 19 and [8 show that BRB with M-Means instead of their native soft assignment consistently improves performance. We attribute this to the new h- 4-Means labels providing an improved exploration of clustering solutions.

---

Published as a conference paper at ICLR 2025
BRB Mechanisms
BRB prevents early over-commitment by increasing intra-class variance while preserving cluster separation.
BRB explores clustering solutions by inducing cluster label changes without destroying the cluster structure, allowing the reassignment of samples late in training.
BRB's resets are necessary for performance improvements. Cluster label changes without corresponding structural changes in the embedding fail to alter the optimization trajectory.
7 DISCUSSION
Broader insights for deep clustering research Over-commitment to early clustering targets is of great interest to the deep clustering community, given the use of powerful deep neural networks to fit changing clustering targets. Our study identifies sharp drops in the intra-class distance early in training as the main culprit of this problem. Once objects with different ground truth classes are compressed in a cluster, reclustering is insufficient to separate them again the "reclustering barrier". Section 6..4 shows that the network fails to adapt to the more diverse targets even if the variation in clustering labels is increased. While we use these findings to develop BRB, they may help to guide the design of future deep clustering methods as well.
phenomenon we term
Connection to early overfitting in other fields The primacy bias is an example of early overfitting (Nikishin et al. 2022) in deep reinforcement learning. In supervised learning. /aidi et al. (2022) explore the role of weight resets regarding generalization performance. They find that soft resets enhance test accuracy by preventing overfitting to noise during early training (Zaidi et al. 2022) and hypothesize that resets may force the network prioritize learning the "easy" data points first. The resets potentially prevent the network from "confidently fitting to [the] noise" (Zaidi et al. (022). In DC algorithms, fitting to noise corresponds Subsequent optimization steps, particularly the compression loss Lc, inhibit these points from being re-assigned later. BRB mitigates this issue to some extent, as shown by resets increasing the intra-CD,
premature assignment of points specific clusters.
Limitations and future work BRB is partially inspired by theoretical research on perturbation- robust clustering (Bilu et al. 2013; Awasthi et al. 2012). We hypothesize that BRB's 's resets and clustering instance, where the soft resets mimic the o-perturbation, altering the Fuclidean distance in the embedding space by a factor of at most a. A full theoretical analysis of BRB in deep clustering is complex due the nonlinearity of neural networks and minibatch SGD and would be significant contribution on its own. Therefore, we focus on empirical results and ablation studies and want to explore the theoretical aspect in future work. Another limitation is that we investigate the reclustering barrier for centroid-based DC algorithms only. Whether a similar phenomenon exists for non non-centroid- based DC algorithms, such as density-based based or agglomerative clustering methods, is an exciting direction for future work.
reclustering create analog an o-perturbation-reslll
8 CONCLUSION
This paper identifies a common problem in centroid-hased DC algorithms: Reclustering during training fails explore new clustering solutions due early over-commitment to a sub-optimal clustering. We call this phenomenon the reclustering barrier. Integrating our proposed algorithm BRB into widely-established DC methods, like DCN, DEC, and IDEC, demonstrates consistent performance gains under multiple conditions, including range of different datasets, different auxiliary tasks (contrastive learning and autoencoding), and settings with and without pre-training Under all of these conditions, BRB is able to break through existing performance plateaus of DEC, IDEC, and DCN. Notably, BRB with contrastive learning yields competitive performance to state-of- the-art algorithms CIFARIO and CIFARI00-20.
Through empirical analysis, we identify three key mechanisms contributing to BRB's ' success. These mechanisms consist of (i) prevention of premature commitment to clustering solutions by DC methods (ii) exploration of a wider range of clusterings and (iii) enabling adaptation to improved clustering targets. BRB can be easily integrated in centroid-basee DC methods. Given that DCN, DEC, and IDEC underpin numerous DC algorithms, we anticipate that BRB will impact the development of centroid-based DC methods going forward.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
Our code
and
models
are
publicly
available on github at
Additionlllllllllllllllllll Additionally, we provide pseudocode of
our algorithm in Section and all experiment hyperparameters together with implementation details in Appendix Section
ETHICS STATEMENT
Our work contains foundational research in the field of deep clustering and, as such, has the potential to accelerate scientific advances in many other areas. At the same time, it is difficult to rule out malicious use due the broad applicability of our work different use cases. This is especially true as we plan to release pretrained cenerrl-purpose models for image feature extraction and clustering. Nevertheless, we believe that the ability of these models enable future research outweighs the drawbacks of potential malicious use.
Focusing on the medical domain, BRB can improve the grouping of patients into different categories, enabling targeted treatments based on the patient's characteristics. While BRB may improve the ca- pabilities of DC algorithms in this respect, is imperfect and will still produce errors. In applications such medicine, where errors are prone to high scrutiny, algorithmic failures may lead to public backlash and slow down machine learning adoption as a whole. Similar concerns arise regarding use cases such as fraud outlier detection finance. We therefore recommend human oversight when deploying any machine learning algorithm in the real world. terms of machine learning research, believe that findings also interesting other fields where algorithms struggle deal with early overfitting, such as semi-supervised learning.
ACKNOWLEDGMENTS
We acknowledge EuroHPC Joint Undertaking for awarding us access to Meluxina at LuxProvide, Luxembourg. Also, we gratefully acknowledge financial support from the Vienna Science and Technology Fund (WWTF-ICTI9-041), the Austrian Science Foundation FWF (project the Federal Ministry Education, Science and Research (BMBWF) of Austria within the interdisci- plinary project "Digitize! Computational Social Science in the Digital and Social Transformation" We thank the open-source community for providing the tools that enabled this work: Py Torch (Paszke cet al. 2019), NumPy (Harris et al. 2020), Matplotlib (Hunter 2007), and ClustPy (Leiber al. 2023). Lastly, we want to thank the anonymous reviewers who helped improve our work through their feedback.
15113) 15113), and
REFERENCES
Ibrahim M. Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitial- ization generalization in convolutional neural networks. CoRR, abs/2109.00267, 2021. doi110044550/rxx22110..000
E. Alpaydin and C. Kaynak. Optical Recognition of Handwritten Digits, UCI Machine Learning Repository, 1998.
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Proceed- ings of the Eighteenth Annual ACM- -CMM-IMM Symposium on Discrete Algorithums, SODA '07, pp. 1027 1027-1035,00 USA, 2007. Society for Industrial and Applied Mathematics. ISBN 9780898716245.. URL htt.thtp:///d/e acm. www.../wwo.oo..ocooc.ooooooooo.t 110. 1281328833388 2884994
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi.
Self-labelling via simul- taneous clustering and representation learning. arXiv preprint arXiv: 19t.:/991.0071, 2019.
10.4555555/// 1911
Jordan T. Ash and Ryan P. Adams. On warm- stamtitarting neural network training. Proceedings of the
34th International Conference Neural Information Processing Systems, NIPS '20, Red Hook,

---

Published as a conference paper at ICLR 2025
NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. URL wttph:htpp/ll.c. org/
3496051
Pranjal Awasthi, Avrim Blum, and Or Sheffet. Center-based clustering under perturbation stability. Information Processing Letters, 1 11((1-)):4--54, 2012. doi 10. 10.1016/j.p1..21111.00
Tudor Berariu, Wojciech Czarnecki, Soham De, Jörg Bornschein, Samuel L. Smith, Razvan Pascanu, and Claudia Clopath. A study the plasticity of neural networks. CoRR, abs/2106.00042, 2021.
URL /httts:///axiiitt org/ abs/ 21 16. 00042
Yonatan Bilu, Amit Danicly, Nati Linial, and Michael Saks. On the practically interesting instances of MAXCUT. In Natacha Portier and Thomas Wilke (eds.), 30th International Symposium on Theoretical Aspects of Computer Science (STACS 2013), volume 20 of Leibniz International Proceedings in Informatics (LIPlcs), pp. 526-537, Dagstuhl, Germany, 2013. Schloss Dagstuhl Leibniz-Zentrun für Informatik. ISBN 11188-3-99977---777 1o:40...33ML Sww.es.SSAASS..... 2013.526
Léon Bottou and Yoshua Bengio. Convergence properties of the k-means algorithms. In Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS 94, pp. 585 592, Cambridge, MA, USA, 1994. MIT Press. | http:: //dp:s://d..do acm. org doi/10) 5555/299868 2998760
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. Vittorio Ferrari. Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 139-156, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01264-99 doi: 10. --007//977---00002666616-
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
URL httpt://dd/dd.. acm ott.moi.rooooooooo/ 100
htt.:/11.5555/5 5555555555554 34 342234499924.4. 3496555
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In ICCV, pp. 9630- 9640. IEEE, 2021. doi
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020. URL https::///// /dl. acm. wwm. 1 55555302555555555 3525087
abs
Stephen Choi and Tyler Renelle. Deep learning price momentum in us equities. In
2019 International Joint Conference on Neural Networks (ICNN), pp. 1-8. IEEE, 2019. 119/II/NN.20011888....
Tarin Clanuwat, Mikel Brier- Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David
Ha. Deep learning for classical japanese literature. arXiv preprint arXiv: 18/1v218/2.00718. 2018.
doi 10.4455////. 1812001718
Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 13693-13702. Computer Vision Foundation IEEE, 2021.
doi T0. 1099CVPR44437 222.2021..0111 1,01488
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society: series B (methodological), 39(1): 1-22, 1977. 10.11..1111//225 111.2//766611..7776000666
Shibhansh Dohare, A. Rupam Mahmood, and Richard S. Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. CoRR, abs/2108.06325, 2021. 00.104445500/rxX22110.....
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron C. Courville. Sample-efficient reinforcement learning by breaking the replay ratio bar- rier. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL ht http/////p/////e///////teeeeeeet net/pdf? /
dd=0pp- 9aBBVJe
12

---

Published as a conference paper at ICLR 2025
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. SCAN: learning to classify images without labels. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vtsion ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X, volume 12355 of Lecture Notes in Computer Science, pp. 268 285.285.22 Springer, 2020.
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In IJCAI, pp. 1753-1759. ijcai.org, 2017. URL https /aa//////00
a.dl.acm.orge http://ooaabb//////
..172131
Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sehastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7825):357-3666 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers Surpassing human-level performance imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034, IEEE Computer Society, 2015. doi 10. 10.1100//C..22015.100
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. pp. 770-778. IEEE Computer Society, 2016. doi 10. 1O. 109.109...2220..0000
Jian Hu, Xiangjie Li, Kyle Coleman, Amelia Schroeder, Nan Ma, David J Irwin, Edward B Lee, Russell T Shinohara, and Mingyao Li. SpaGCN: Integrating gene expression, spatial location and histology to identify spatial domains and spatially variable genes by graph convolutional network. Nature methods, 18(1 15:322313))133 2021. TO. 150-000-12522-00-0222555
Huajuan Huang, Chen Wang, Xiuxi Wei, and Yongquan Zhou Deep image clustering: A survey, Neurocomputing, 599:128101, 2024. TO. NEU.OO//EE
URL https
//doi.orq// 10111101..... http://u....../e..uuu...m.....
www.oe.cco....m 2024. 2024 128101
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218.
1985. do. 1007/BFFO 19080755
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994. doi 10. 109/34.2994440
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3): 90-95, 2007. doi 10. 109/MCSE.2007.55
Maria Kalweit, Andrea M Burden, Joschka Boedecker, Thomas Hügle, and Theresa Burkard. Patient groups in rheumatoid arthritis identified by deep learning respond differently to bi- ologic targeted synthetic dmards. PLOS Computational Biology, 196):e1010733, 2023.
doi: 1.. 137. 1073
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. doi 14.104448555//XX1.111.669900
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009. URL https:// cs. www.ttorootoooooooo www.onnt...ddd/eeeeeeeee wwwpdddw/////eee http://eeeoeeeeeeeeeeeeennnnefea features---e--eee 2009- TR
pdf
Harold W Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):33-97, 1955,
Tarald Kvalseth. Entropy and correlation: Some comments. IEEE Transactions on Systems, Man, and Cybernetics, 17(3):517-51, 1987. 10.101..097 9.3999999999
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1) 1):2278-2324, 1998. doi 10.1109/5.726791
13

---

Published as a conference paper at ICLR 2025
Johannes Lehner, Benedikt Alkin, Andreas Fürst, Elisabeth Rumetshofer, Lukas Miklautz, and Sepp Hochreiter. Contrastive tuning: A little help to make masked autoencoders forget. In AAAI, pp. 2965-29733 AAAI Press, 2024. 1o. 1009/aaii..38.4.2880000 v3814.28078
Collin Leiber, Lena G. M. Bauer, Benjamin Schelling, Christian Böhm, and Claudia Plant. Dip-based deep embedded clustering with k-estimation. In KDD 21: The 27th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pp. 903-913, ACM, 2021. doi 10. 0000144//444544834773311
Collin Leiber, Lukas Miklautz, Claudia Plant, and Christian Böhm. Benchmarking deep clustering algorithms with clustpy. In 2023 IEEE International Conference Data Mining Workshops (ICDMW). pp. 625-632 632. IEEE, 2023. doi 110.1110//CDDWWG00000 202..0000000000
Xiangjie Li, Kui Wang. Yafei Lyu, Huize Pan, Jingxiao Zhang, Dwight Stambolian, Katalin Susztak, Muredach P Reilly, Gang Hu, and Mingyao Li. Deep learning enables accurate clustering with batch effect removal in single-cell RNA-seq analysis. Nature communications, 11(1):2338, 2020.
10 1038//48 0200-0000-1--11--
Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering- In Thirty-Fifih AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 8547-85555 AAAI Press, 2021. doi 1..009..1//AA 1WL.160099AAALL55110...000. V35110.17037)
Zhaoyang Li, Qianqian Wang, Zhiqiang Tao, Quanxue Gao, and Zhaohua Yang. Deep adversarial multi-view clustering network. In IJCAI, pp. 2952-2958. ijcai.org, 2019. doi 10.21006///cccii20000 2019///999
Stuart P. Lloyd. Least squares quantization in PCM. IEEE Trans. Inf. Theory, 28(2):129-136, 1982.
doi 10. T09/T1T. 1982. 1056489
Yiding Lu, Haobin Li, Yunfan Li, Yijie Lin, and Xi Peng. A survey on deep clustering: From the prior perspective. CoRR, abs/2406. 19602, 2024. 401:0004550/ARXX.24006 19602 URL https://dol. org/10. 48550/ 2t155//arxiv..mm 2406 19602
Zixiang Luo, Chenyu Xu, Zhen Zhang, and Wenfei Jin. A appology-preeevinng dimensionality reduction method for single-cell RNA-seq data using graph autoencoder. Scientific reports, 11(1): 20028, 2021. doi 10. 1038/s411110 118441598--80 02---021--00 021-9900--
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Conference on Machine Learning. ICMI. 2023, 23-29 July 2023, Honolulu, Hawaï, USA, volume 202 of Proceedings of Machine Learning Research, pp. 23190-23211 PMLR, 2023. URL https //proceedings ings. ml press / w2.//12222./. 1233b. html
Louis Mahon and Thomas Lukasiewicz. Hard regularization to prevent deep online clustering collapse without data augmentation. In AAAI, pp. 14281-14288. AAAI Press, 2024.
Lukas Miklautz, Lena G. M. Bauer, Dominik Mautz, Sebastian Tschiatschek, Christian Böhm, and Claudia Plant. Details (don't) matter: Isolating cluster information in deep embedded spaces, In IJCAL, pp. 2826-2832, ijcai.org, 2021. 4o.1 2021//333
Kevin P Murphy. Probabilistic machine learning: introduction. 2022.
John Henry Newman. Churchill quotes others without credit. May 1845. URL https:// winstouchurchii1.h1iieee htt.::/a.....aaaaaaceceeeeeeeeee
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022. 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 168477 PMLR, 2022. URL https //proceeding ngs. ml prw.s / 62 sh n22a html.
14

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Z. Yang, Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024 8035, 2019. URL 2ebba2288fee17122223332222270000 Abstaattt..t...tttttttt....a.... html
wwtppccppppppppppppeeeeeeepppppp 9/ahh//
Qi Qian. Stable cluster discrimination for deep clustering. In /EEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. (6599-166055 IEEE, 2023.
202.2003.15
LKPJ Rdusseeun and P Kaufman. Clustering by means of medoids. In Proceedings of the statistical data analysis based on the L. norm conference, neuchatel, switzerland, volume 31, 1987.
Yazhou Ren, Jingyu Pu, Zhimeng Yang. Jie Xu, Guofeng Li, Xiaorong Pu, Philip S. Yu, and Lifang He. Deep clustering: A comprehensive survey. CoRR, abs/2210.04142, 2022. 11010..110//NNLSS22220.000
Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987. 1011600777
(427(87)9112--7
Max Schwarzer, Johan Samir Obando-Ceron, Aaron C. Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 3006553303333 PMLR, 2023. URL htttps://proces #202/schwarzer23..r html
ngs. ml press/
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, (0):-, 2012, ISSN 0893-6080. 1016// www.eettu1...2.....
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CORR, abs/ 887.03744 2018. 0o/110048550/arxi..117...00 1007///0XXX.1180007.55.
Mark P Wachowiak, Jason J Moggridge, and Renata Wachowiak-Smolne Smoliková.. Clustering continuous wavelet transform characteristics of heart rate variability through unsupervised learning. In 2019 41 Annual International Conference of the IEEE Engineering Medicine and Biology Society (EMBC), pp. 4584 4587. IEEE, 2019. 100011000/MBBB.211.8888333
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 1929-9939 PMLR, 2020. wang20k.paF
mll. www.srrsssvvvle. /ilesss/vlll/ wttps//wwl//wwwen..pppeees 9/w/ngg2/////////
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset
for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
10.5555555// 170.00777
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pp. 478 487. JMgR. 2016. https:::::///
acm. wttpa//.w.......0...555533333350
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k- means-ffrendlly spaces: Simultaneous deep learning and clustering. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 3871-3870. PMLR, 2017. URL https: //d1. acm. org do1/00
5555/3305890 3306080
15

---

Published as a conference paper at ICLR 2025
Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. Image clustering using local discriminant models and global integration. JEEE Transactions on Image Processing, 19(10): 2761-2773, 2010. 1o. 10111001199
Sheheryar Zaidi, Tudor Berariu, Hyunjik Kim, Jörg Bornschein, Claudia Clopath, Yee Whye Teh, and Razvan Pascanu. When does re-initialization work? CoRR, abs/2206. 10011, 2022. doi 1000004855//XXX122..0001111
Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Xian-Sheng Hua. Graph contrastive clustering. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pp- 2204-9213. IEEE, 2021. 10/11119/2/
Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao Li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, and Martin Ester. A comprehensive survey on deep clustering: Taxonomy, challenges, and future directions. CoRR, abs/2206.07579, 2022. URL https: //dp: acm. org/doil
abs/10..115/368900
Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao Li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, and Martin Ester. comprehensive survey on deep clustering: Taxonomy, challenges, and future directions. ACM Comput. Surv., 57(3):69:1-69:33 2025, do. URL https::/doii 1tt.:/o000/1111118888811101
16

---

Published as a conference paper at ICLR 2025
Appendix
Table summarizes the contents our Appendix:
Appendix Section
Content
Appendix A
Pytorch-style pseudocode for BRB
Detailed information about the clustering algorithms used in this study
Appendix B
SimCLR (Chen al. 2020) overview
Overview of resetting methods in other fields
Appendix
Datascts our experiments, data properties, and preprocessing
Appendix D
Evaluation metrics
Implementation details
Appendix E
Hyperparameters & Hyperparameter sensitivity analysis
Appendix
Detailed setup and results the noise disentanglement ablations
Appendix G
Runtime analysis
Additional experimental results and figures Full result tables
Appendix H
Table 2: Structure of our appendix.
A PSEUDOCODE
Algorithm 1 PyTorch-style pseudo-code of BRB
model:
neural network
deep clustering method
# alpha: BRB reset factor (0 <= alpha <= 1)
recluster algorithm: algorithm for reclustering
interval
# optimizer: optimizer to be used, default : Adam
subsample size: size of sample used for reclustering
deep clustering training loop for epoch in epochs:
epoch perform and apoch 0:
reeset_model (model, alpha) ambed data with reset model
embed (model, loader, subsamplo_sit size) reinitialize cluater centroids da. centroids
cellutte_algorittee
de. centroids.... momentum()
# reset momentum of learnable centroids
mwmemmum (optimizer, dc.centroids)
load minibatch x
for in loader:
perform deep clustering update steps da. dc. update (x)
B BACKGROUND
In this section, we provide additional background for some of the methods employed in our main paper. It is structured as follows: First, provide more details deep clustering objective functions Section B.I Then, we briefly review SimCLR (Chen et al.) 2020) as the contrastive learning method used in this work. Lastly, we review reset methods in other fields.
17

---

Published as a conference paper at ICLR 2025
B.1 DEEP CLUSTERING OPTIMIZATION
In general, the procedures mentioned in this work optimize L(O, M) = Asss + (0, M).
To highlight the differences in detail, explain below how Le is chosen. Applying BRB to DEC. IDEC, or DCN does not change the loss functions defined in Equation (5) and Equation 66 or how backpropagation is performed.
DEC and IDEC: As mentioned in Sec. 2 the clustering loss Le of DEC (Xie et al. 2016) and IDEC (Guo et al. 2017) is based on the Kullback-eeblerr divergence, Here, the data distribution Q is compared with an auxiliary target distribution P. The data distribution Q is quantified by kernel based on the Student's t-distribution:
(1 I|., 14 12)
Pijj
Ej. (1+ zz Pey
where z, is an encoded sample, i.e., 2i fx(x)), and pej is the center of cluster j. The target distribution P defined
L.gh P.j EE
where f3 = E; R. are the soft cluster frequencies. Here, L. is used to strengthen predictions on data points that are assigned with high confidence. Simultaneously, a division by fj should prevent large clusters from dominating the loss.
Putting everything together, we receive:
Lc(0, M)) KL(PII) = -LPPI)==
ΣΣ
Pad log Pi.i 9:
(5)
In the case of DEC, A1 0 and A = 1 during the clustering optimization. For IDEC, Å = 1 and A2 = 0.1 to prevent distorted embedding.
DCN: The optimization strategy of DCN (Yang et al. 2017 is based on the classical k-Means (Loyyd
1982 algorithm. First, the network parameters 8 are updated using [(0,, M), where
Le (0, = Σ 1 PM(,.)
and llzz)) returns the cluster z, is assigned to. Afterward, each sample z, is assigned to its closest cluster center. Last, the cluster centers are updated by calculating:
updated
(pn(z.) 2:)-
(((.)
Ch((s:)
Here, Gj counts the total number of samples already assigned to cluster j. This update strategy is inspired by the SGD-k-Means algorithm (Bottou Bengio, 1994). Finally, the process repeats by optimizing the embedding using Eq. 6 with the newly obtained cluster parameters.
B.2 SIMCLR
SimCLR (Chen et al. 2020 is contrastive learning method designed to discriminate among pairs of augmented samples. In its pretraining phase, the encoder fe processes batch of N samples, each augmented twice, resulting in batch size of 2N. This yields embeddings h, = fo (x;). Instead of explicitly mining negatives, SimCLR treats all augmented samples not belonging to the anchor as negative samples. Before the contrastive loss is applied, the encoded samples are mapped into another latent space 2, g(h.) by a projector network. SimCLR then applies the InfoNCE loss (van den Oord et al. 2018):
exp (sim (z,, 2;)/7 /T)
Lij log
(8)
[ exp (sim (z;, Z%) /T)
A==
where sim (z,, 2,) denotes the cosine similarity between two projected embeddings, and if is if l # i. Essentially, contrastive learning fosters latent space wherein the features of augmented versions of the same image are close while being distanced from others, thereby promoting a cluster- friendly latent space (Wang & Isola 2020
18

---

Published as a conference paper at ICLR 2025
Table 3: Information regarding the benchmark datasets.
Dataset
Type
# Samples
# Features
# Classes
Optdigits
Grayscale
5,620
GTSRB
Color
7,860
32 x 32 x 3
5
USPS
Grayscale
9,298
16 16
CIFARIO
Color
60,000
32 x 32 x
10
CIFAR 100-20
Color
60,000
32 x 32 x
20
MNIST
Grayscale
70,000
28 28
10
FMNIST
Grayscale
70,000
28 28
10
KMNIST
Grayscale
70,000
28 28
10
B.3 RESET METHODS IN OTHER FIELDS
Reset methods have been used in other fields to great success. For example, has been shown that soft resets named Shrink and Perturb- may be able to alleviate poor generalization performance in the pretrain fine-tune setting supervised learning (Ash & Adams 2020). This phenomenon has been dubbed the generalization gap by Berariu et al. (021))
striking application of resetting has developed in the field of deep Reinforcement Learning (RL): Here, resets have been used to improve the data efficiency of algorithms by mitigating plasticity loss
(Nikishin al. 2022 D' D'Oro et al.) 2023; Schwarzer et al., 2023). Loss of plasticity refers to the loss of network's learning ability over the course of training (Berariu et al. [021] Dohare et al. (2021).
is particularly prevalent RL due to the non-stationarity induced by bootstrapped targets in deep value-based algorithms.
As our disentanglement ablation study in Figure T0 shows, deep clustering algorithms do not suffer from loss of plasticity. Instead, the resets help avoid premature convergence sub-optimal local minima by facilitating the exploration of multiple clustering solutions. When BRB commits to a clustering solution, the resets ensure that it is robust with respect to the structured noise induced by the resets.
The different reset strategy method employs highlights this: Our method never applies Shrink and Perturb to the embedding layer, as this could lead to a collapse of deep clustering performance. On the other hand, deep RI. algorithms often fully reset the last layers completely. as plasticity loss is commonly believed to be concentrated there (Berariu et al. 2021 D'Oro et al. (023)).
C DATASETS
We evaluate our approach by analyzing the results regarding eight image datasets commonly used in machine learning. The information concerning these datasets is summarized Tab,
The grayscale image datasets Optdigits (Alpaydin & Kaynak 1998), USPS (Hull 1994). and MNIST (LeCun et al. 1998) contain images of handwritten digits (0-9), with each dataset having a different resolution. FMNIST (Xiao et al., (017) contains grayscale images of ten articles from the Zalando online store, and KMNIST (Clanuwat al. 2018) is composed of grayscale images showing ten Japanese Kanji characters.
The color dataset GTSRB (Stalikamp et al. 2012) contains images of 43 German traffic signs. We 50', 'Priority road', 'No entry', 'General caution', and 'Ahead only' (as proposed in (Leiber et al. (022I) 'Priority road', 'No entry', and 'General caution' were given 'Right of way', 'No passing" and 'Attention' here) and unify the sizes of the images. CIFAR 10 (Krizhevsky et al. 2009) is composed of color images showing ten different objects with various backgrounds. The CIFAR100-20 dataset uses course course-grained grouping of the 100 classes of the CIFAR100 dataset to summarize the fine- me-grained classes within related super-classes.
use subset of the data containing the signs "Speed limit
19

---

Published as a conference paper at ICLR 2025
All datasets preprocessed by performing zero and a unit variance for each color channel. addition, histogram normalization is conducted for GTSRB due the low contrast.
resulting in of
D METRICS
In the following, we define the clustering evaluation metrics that are used throughout our work. More detailed descriptions and examples can be found in Murphy (2022).
D.1 CLUSTERING ACCURACY
Clustering accuracy (ACC) YYang et al. 2010) measures the fraction of correctly assigned cluster labels over the dataset compared the ground truth labels:
ACC maxmap E 8(vi, (a)(
Here yi the ground-truth label data point i, dJ the predicted label of sample i, and denotes the Kronecker-Delta. As the labels for the found clustering might deviate from the ground truth labels, we require the mapping function map that tests each possible mapping of labels. We use the Hungarian method (Kuhn 1955) to identify the best mapping.
D.2 ART
Adjusted Rand Index (ARI) (Hubert Arabie 1985) is statistical measure that generalizes the notion of accuracy for clustering and additionally adjusts for chance. It provides score ARI E [-1, 1]. An ARI score signifies agreement, -1 indicates disagreement, and score of O suggests agreement by chance. ARI is defined
R R-Ea B- HGeom(kk,n) [R]
ARI =
(10)
1 Ey HHHoom (k.
where R denotes the Rand index, which corresponds to ACC in case the ground truth class labels are used as reference clustering. ARI adjusts for randomness by using the expected value of the Rand index Ey HGeoomk(,,..)) [R] under generalized Hypergeometric distribution HGeom(k, n) with (ki...,,k,) kn) to distribute the samples across the ls classes with uniform random probability.
D.3 NMI
The Normalized Mutual Information (NMI) Kvalseth 1987 calculates the Mutual Information between sets of labels and scales the results between 0 and
NMI(C, 21(C,0) + H(C)
(11)
where (CI....,CC, is the ground truth partition and Ó = { (C,,..., Crr are the predicted clusters. 7 denotes mutual information between sets, whereas H (X) corresponds to the entropy of probability distribution X.
D.4 INTER- & INTRA- CLASS DISTANCE
Lehner et al. 2024 uses the silhouette score Rousseeuw (1987) of the 12 normalized embeddings to measure the separation of ground truth clusters the embedded space of neural networks. The silhouette score is the relation between the intra- and inter-class distances that in this work.
20

---

Published as a conference paper at ICLR 2025
The intra-class distance (intra-CD) of sample X, Cj is defined as the mean distance of the sample to other samples x, in the same cluster Cy:
dinuaa (x1)
1
Σ
d(x;,x))
(12)
Cil
x,ECt
The inter-class distance (inter-CD) of sample x; E Cj is the mean distance of the sample to instances in the next closest cluster Cmztt:
dinner (x²) = min
Σ
d(xx,)1))
(13)
[Cll X, X,ECm
E IMPLEMENTATION DETAILS AND HYPERPARAMETERS
Our codebase builds top of the open-source ClustPy-codebase (Leiber et al. 2023), which provides Pytorch implementations of the centroid based deep clustering (DC) algorithms used in our work. We the Adam optimizer (Kingma & Ba 2013) due to its widespread use and strong performance on wide range of deep learning problems. For DEC and IDEC, the centroids are optimizable parameters. Therefore, we also reset their momentum after each weight reset prevent divergent gradients. Due to the high learning rates used in our contrastive learning experiments (cf. Table ( we rarely encounter single gradient steps with very high gradient magnitudes. Therefore, we clip the le gradient norms value of 500. For the AE, the latent dimension is set to the dataset's 's number of classes, common practice in DC. Additionally, we reset the projector network for contrastive learning with each weight reset of BRB.
We have to slightly adapt the objective functions of DEC, IDEC, and DCN to handle augmentations in meaningful way. An essential constraint is that the cluster assignments of sample should not change due to the applied augmentations. Therefore, the augmented sample x aug(x;) must always receive the same assignments $t as the original sample x4. We denote z fa (x(*) as the encoded version x general, we use the average of the loss with respect to the original sample x; and the loss with respect to the augmented sample x
First, we adjust LSSL (here shown for the AE):
SSL
(1,sL((, (0,
(14)
Next, update the clustering loss Le regarding DEC and IDEC:
LA
1 ΣΣ
Pi.j log( Piuj ) + pij log( Piuj Wi-j
(15)
2
Ti
where
(1 1 1 M4
14, 11
(16)
Ej,(1
+
Note that Pij remains the same compared to the DEC/IDEC version without augmentation. Last, we adjust Le for DCN:
LA
4 Σ (1|z
MM(2.)
12 1|2 pl(2)))
(17)
For contrastive learning, which learns a representation that invariant to the augmentations, we only augmented samples during pretraining and deep clustering.
21

---

Published as a conference paper at ICLR 2025
E.1 SELF-LABELING IMPLEMENTATION DETAILS
Many deep clustering methods have two stages; pretraining and joint deep clustering. Self-labeling (Gansbeke al. 2020) can be used after joint deep clustering to further improve clustering perfor- mance as an additional third stage. Self-labeling uses ideas from self-training in semi-supervised learning only train on samples that have already been assigned with high certainty to a cluster. Self-labeling does not work out of the box with centroid centroid-based deep clustering methods as it assumes a classification head to discriminate between the pseudo-labels learned from the clustering. To solve this problem, we follow the training strategy of (Qian (023). which successfully applied self-labeling to their centroid-based deep clustering method SeCu by introducing a short warmup phase. This warmup phase trains classification head based on the clustering labels obtained from the centroid- based clustering objective and then followed by the self-labeling phase
E.2
HYPERPARAMETER SENSITIVITY ANALYSIS
we provide detailed sensitivity analysis of BRB over the learning rate (LR), the reset interval T, and the reset factor a. We re- port averages over 15 runs (five seeds per DEC, IDEC, and DCN) for each setting on MNIST, resulting in 960 runs in total. For the AE pre- training, we use an LR 0.001 and, therefore, select LRs that are higher (0.005), equal (0.001), and lower (0.0005, 0.0001) than the pretraining LR for DC. Note that usually, only LRs that lower than the pretraining LR are considered in DC. In Figure 8 the lower and higher end of LRs (0.0001, 0.005) suffer more from frequent
In Figure g
100
0.6
BEBBB SE
100
01 d -
100
75
(T {5, 10}) and strong (a E {0.6, 0.7}) re- sets. Intuitively this makes sense: The more often we reset with high reset factor, the more likely it is to destroy useful cluster informa- tion. The lower and higher LRs exacerbate this by either recovering too slowly or too quickly, whereas the moderate LRs (0.0005, 0.001) are still robust. An exception occurs for with T = 5, where the setting with LR 0.0005 see that
100
0.9
5=5 T=
10
T= 20
T= 40
Learning Rates 0.0005
0.6
0.0001
aes 0.001
0.005
loses some performance. In general, the moderate LRs (0.0005 and 0.001) remain stable over all other settings. As rule of thumb, choosing moderate reset value between 0.8 and 0.9 with less frequent resets (T' E {20, 403) and an LR that is equal to slightly lower than the pretraining LR performs well in many settings. This is why for all main experiments, we choose an LR of 0.001 and set a = 0.8 with = 20.
Figure 8: Hyperparrmmterr sensitivity anal- ysis. Clustering accuracy (averaged over DEC/DDC//DN and five seeds per setting on MNIST) for BRB's reset factor a, reset inter- val T and different deep clustering learning rates. Large (a E {0.6, 0.73) and frequent (T' E (5, 10)) resets destroy cluster structure (lower accu- racy), whereas moderate values (T E (20,40)
(0.8,0.9) perform consistently well.
We used the public repository of SCAN at Jnsupervised-Class in
22

---

Published as a conference paper at ICLR 2025
E.3 ADDITIONAL HYPERPARAMETER SENSITIVITY EXPERIMENTS
Figures Da and 9b show hyperparameter sensitivity matrix plots for the datasets USPS and OPT- DIGITS. Compared MNIST, both USPS and OPTDIGITS suffer more from frequent or strong resets. For example, setting a = 0.6 and = 5 with learning rate of 0.001 yields performance well below 50% clustering accuracy on USPS and completely fails learn on OPTDIGITS. contrast, the same settings on MNIST (cf. Figure 8 result in similar clustering accuracy values as the ones obtained by the settings we use in the main paper, namely a 0.8, T 20, and learning rate 0.001. Interestingly, both USPS and OPTDIGITS seem to benefit from using higher learning rate of 0.05 (red bars), whereas increasing the learning rate to this level is harmful on MNIST. conclude that (i) the best-performing ranges of hyperparameters differ depending on the dataset, and (ii) there may be trade-offs between hyperparameters for different datasets. The hyperparameters we used for all main experiments (a 0.8, T 20, learning rate 0.001) within range of settings that perform well on a variety of datasets.
100
100
0.6
0:
75
75
50
50
25
25
100
100
011
='O -0=
75
75
50
50
25
25
100
100
a 1504
75
75
50
50
25
25
100
100
1-10 0.9
6'0=2 D
75
75
50
50
25
25
5=5
=
10
7
20
7=
40
7=5
1= T0
70
Learning Rates 0.0005
Learning Rates 0.0005 HRUW 0.001
0.0001
BoOw 0.001
0.005
0.0001
0.005
(a) Hyperparameter sensitivity analysis on USPS. Clustering accuracy (averaged over DEC/IDE//DNN and five seeds per selting) for BRB's s reset factor a, reset interval T, and different deep clustering learn- ing rates.
(b) Hyperparameter sensitivity analysis OPT- DIGITS. Clustering accuracy (averaged over DEC/DEC/DDN and five seeds per setting) for BRB's reset factor a, reset interval T, and differ- ent deep clustering learning rates.
Figure 9: Hyperparameter sensitivity analysis for USPS and OPTDIGITS.
23

---

Published as a conference paper at ICLR 2025
E.4 COMPUTATIONAL RESOURCES
We primarily utilized internal servers to generate our results with grayscale datasets. For this paper, had access
2x Intel Xeon Silver 4214R CPU @ 2.40GHz server with 48 cores and 2 Nvidia A100 GPUs with 40GB of VRAM each
2x Intel Xeon Gold 6326 CPU @ 2.90GHz server with 64 cores and NVIDIA A100 GPU with 80GB of VRAM
These machines were also used to run preliminary experiments and to identify broad working ranges for the most important hyperparameters.
For the experiments with color datasets and contrastive learning, we had access to 1000h compute hours per month a SLURM supercomputing cluster. A node in this supercomputer EPYC 7452 processor with 128 cores and NVIDIA A A100 GPU with 80GB of VRAM. We also used these resources
AMD
hyperparameters for SimCLR (Chen et al. 2020) pretraining.
The average wall clock time for one run summarized in Table We identify two main factors contributing to the runtime of our method. First, the number of samples in the dataset (see Table 3). where, c.g., MNIST variants take substantially longer to run than USPS or OPTDIGITS. Second, the number of CPU cores we could utilize to parallelize data augmentations.
Table 4: Average runtime of our experiments.
Dataset
Approximate Average runtime
MNIST, FMNIST, KMNIST
4h
GTSRB
20m
USPS
15m
OPTDIGITS
8m
CIFAR 10
5h
CIFAR 100-20
5h
24

---

Published as a conference paper at ICLR 2025
E.5 HYPERPARAMETERS FOR GRAYSCALE DATASETS
Table 5 lists the hyperparameters used for our experiments on grayscale datasets. These consist of OPTDIGITS (Alpaydin & Kaynak 1998), USPS (Huli) 1994), MNIST (LeCun et al. 1998), FMNIST (Xiao et al.] 2017). and KMNIST (Clanuaa et al. 2018). We tune the parameters for E {0.6, 0.7, 0.8,088,0000 J.9} and T {10, 20, 40} once on MNIST and re-use them for all grayscale datasets.
Parameter
Value
GENERAL TRAINING
Batch size
256
Learning rate
1e - 3
PRETRAINING
Epochs
250 False
Augmentation
CLUSTERING
Epochs
400
Augmentation
True
DCN cluster loss
0.025
IDEC cluster loss
0.1
AUGMENTATION
Resize
(28,
RandomAffinn
deg = 116, transl = 0.1,
shear
ARCHITECTURES
Feed Forward
D-1024-512-256-1
BatchNorm
False
Embedding size
# Clusters Adam
OPTIMIZER
Momentum 81
0.9
Momentum $2
0.999
Weight decay
0
BRB
Subsample size
10000
a (Reset interpolation factor)
0.8
T (Reset interval)
20
Reset embedding
False
Reset center momentum
True
Table 5: BRB hyperparameters for grayscale datasets.
25

---

Published as a conference paper at ICLR 2025
E.6 HYPERPARAMETERS FOR COLOR DATASETS WITH RECONSTRUCTION PRETRAINING
Our hyperparameters used for the dataset GTSRB (Stallkamp et al., 2012) are in 6 We use the same hyperparameters. for the grayscale datasets except for the data augmentations.
Parameter
Value
GENERAL. TRAINING
Batch size
256 1e 3
Learning rate
PRETRAINING
Epochs
250 False
Augmentation
CLUSTERING
Epochs
400
Augmentation
True
DCN cluster loss
0.025
IDEC cluster loss
0.1
AUGMENTATIONS
RandomResizedCrog ColorJitter
(32,32)
bright = 0.4, contrast = 0.4,
sat = 0.2, hue 0.1,p = 0.8
RandomGrayscale
p 0.2
Randonllorizotta..Pii
p= 0.5
RandomSolarize
thresh 0.5, p 0.2
ARCHITECTURE
Feed Forward
D-1024-512-256-
Embedding size
Clusters
BatchNorm
False
OPTIMIZER
Adam
Momentum 81
0.9
Momentum 82
0.999
Weight decay
False
BRB
Subsample size
10000
(Reset interpolation factor)
0.8
T (Reset interval)
20
Reset embedding
False
Reset center momentum
True
Table 6: BRB hyperparameters for GTSRB (Stallkamp et al.
2012
26

---

Published as a conference paper at ICLR 2025
E.7 HYPERPARAMETERS FOR COLOR DATASETS WITH CONTRASTIVE LEARNING
We conduct experiments with contrastive learning as auxiliary tasks on CIFARIO Krizhevsky et al. 2009) and CIFAR100-20. Some parameters depend the dataset and algorithm in this setup, therefore report them separately in Table For self-labeling (Gansbeke et al. 2020 we use the same configs as specified their public repository" except for the differing hyperparameters specified at the bottom of Table 7
Table 7: BRB hyperparameters when using contrastive learning as auxiliary task on CIFARIO Krizhevsky et al. 2009) and CIFARI00-200
Parameter
CIFAR1O
CIVAR100-20
PRETRAINING
Fpuchs
Learning rate
3e 3
-
Winght decty
le-4 True
le-4 Trre
Augmentation
CLUSTIRING ING
Learning
1e-3 (DEC), 3e-3 3IDC, 1e-2 (DCK)
1e.3 (DDC), 3e- (IDEC). 2e (DCN)
Epochs
Augmentation
claster weight
IDCC cluster loss weight
1.0
1.0
AUGMINEATIONN
RandomBe zedCrop
(32, 32)
[32, 32)
ColorJitter
bright 0.4, contrast - 0.4, Xit 0.2,, huc 0.1,y 0.8
bright 0.4. cuntrast - 01.4.
Aat 0.2, buc 0.1.p 0.8
RandonGzaysce
0.2 0.5.
0.2 0.5.
IPI
SandonSclarla
ARCITTICTRR
CNN
ResNet-18 D-512-256-d
ResNet-18 D-512-256-d
Foed Forward
.. (FinieeMiing sine)
126
128
Projector depth
Projector witth
2018
2068
Sofunax Temperature BatchiNonst
OPTIMIZER
Momestum
Momestum 9,
DECIDEC Weight devay
Fubse Fubs
Palse
DCN Weight decay
False
BKB
Sulhomple size
1000
10000
(Roset intes polattoiiio
Block
(Reset interval)
Reset embedding
Fube
False
Reset center momcatum
Thue
True
SELF LABELING
We use the same parameters Learning
Ganshhike
2020
exoept for the following:
0.001
0.001 (DEC IDEC)
0.00025 (DCN)
Warmpp Epochs
Momentum 8:
0.9
0.9
Momestum 3,
0.959
0,999
Weight deeny
False
False
'The config for CIFAR100-20 can be found https://github..oo//wwnggg Unsupervi Unsupervised-Classs Ced- cac on/bl blob olboob/aaa htt.///astee//eeemmm /ttaaster//oooooeee con f gaggglaallll.eeoo (ttaaallllbbbbee (alaabe/seelfaate
ci cifar20. yml and the for CIFAR10 in the folder denoted selflabel c0for10.yml.
27

---

Published as a conference paper at ICLR 2025
F ABLATION FOR NEW CLUSTERING TARGETS
In this section, we describe the setups of disentanglement and noise experiments from the main paper. We then provide a more detailed analysis of the results.
F.1 EXPERIMENT SETUP FOR THE DISENTANGLEMENT ABLATION
With the disentanglement ablation, our objective is isolate the impact of altered cluster labels resulting from resets from the resets themselves. For this, we modify the pseudocode of Algorithm| [ as follows: Initially, we reset the weights of a duplicate the DC algorithm's model in the first and second steps, utilizzig it to produce new embeddings. These embeddings are then used to derive new cluster labels while keeping the existing centroids unchanged. Subsequently. new centroids are computed using embeddings from the unaltered model but with the newly generated labels. The modifications are outlined in Algorithm
Algorithm 2 PyTorch-style pseudo-code of Disentangled BRB
model:
neural network
de: deep clustering method
alpha: BRB reset factor (0 alpha <e 1)
# recluster algorithm: algorithm for reclustering
BRB reset interval
optimizer: optimizer to used, default : Adam subsample_size size of sample used for reclustering
training loop
for epoch in epochs:
if epoch & =* and epoch 0:
perform
on copy
reset_coph = reset model weights (copy (model), alpha)
ambed data with reset copy of model optional: subsampling can be used
noise_emb embad_data (reset_copy, loader, subsample,size)
emb embed data (modal, loader, subsample size) new labels
labels predict.labels (dc_method. centroids, dec.mettod.centtridde, model, noise_emb)
embedding unperturbed model
dc. centroids recluster.algoritte (emb, labels)
if de. centroids. has m))mmttum() momentum of learnable centroids
resat momentum (optimizer, de. ce.certroids)
minibatch
for x in loader:
perform deep clustering update steps de. update (x)
F.2 EXPERIMENT SETUP FOR THE NOISE ABLATION
BRB employs soft resets introduce structured noise into the optimization process of the DC algorithm. While this approach offers the advantage impacting both the deep learning components and clustering, one might hypothesize that simply adding noise to generate new clustering could suffice to escape sub-optimal local minima. We explore this hypothesis through an ablation study where add noise the embeddings instead of resetting the network's weights. To ensure bespoke distortions the latent space, we first normalize the noise vector to match the norm the embeddings. We then add fraction of that vector, as described in Equation [8]
Vi (1....,n) n) h, h, _c, with ||||| [b:]
(18)
Here, denotes independent and identically distributed noise across dimensions and samples. E R+ represents positive scaling factor. Empirically, we find that Gaussian noise with scaling factor 3 = 0.3 achieves balance between inducing label changes and minimizing performance degradation in later training stages. Following noise addition, we recluster using the perturbed embeddings while continuing training the unmodified network.
28

---

Published as a conference paper at ICLR 2025
Accuracy
Intra CD
20
%
Intor CD
Change
Epechs
Disentangled
Noise
DCN+BRB (ours)
Figure 10: Importance of resets. Clustering accuracy on OPTDIGITS drops considerably if only targets are changed without resetting the embedding (Disentangled). Similarly, adding Gaussian noise the embedding is insufficient change cluster structure (Noise).
F.3 DETAILED ANALYSIS OF THE DISENTANGLEMENT AND NOISE ABLATION
To disentangle the effect of weight resetting from the influence of new clustering targets obtained through reclustering on the post-reset embedding, we perform two ablation studies: First, the Disentangled variant compares BRB with modified algorithm utilizing labels generated from perturbed embedding, achieved by resetting a duplicate encoder. This is followed by reclustering with k-Means on the perturbed encoder's embedded space without changing the original network. For the second experiment, we add noise to the embedded space before reclustering. The exact experiment setups are described
previous sections.
The results in Figure 10 show that for both modifications of BRB, the performance degrades, whereas BRB maintains the network's ability to fit new targets due the reset and yields strictly better results. The intra-CD and inter-CD of both the Disentangled and Noise versions remain virtually unchanged compared to the baseline, indicating that a persistent change in the embedded space is required to enable enhanced adaptation. In terms of CL change, the modified algorithms either fail induce the required changes to explore new clustering solutions (Disentangled) or generate too large label changes late in training (Noise). Both negatively affect clustering performance. In summary, this lets us conclude that structured, persisteni perturbations induced by our soft resets are a necessary condition for subsequent improvements through the reclustering step of BRB.
G RUNTIME ANALYSIS
G. RUNTIME COMPLEXITY
BRB consists of three components: parameter perturbations, reclustering, and momentum resets. We will first give the runtime complexity of these components and then verify these findings empirically. In summary: Our analysis reveals that the reclustering step of BRB adds the most overhead, but compared the total training time, the impact of BRB is modest, for instance, resulting in 5.38% increase in total training time for our experiments for IDEC on MNIST.
Parameter perturbations. The complexity is -D), where L is the number layers that are to be reset and D are the most neurons across all layers that to be reset. Note that L is usually small as we do not reset the full network for larger models and that the bound does not depend on the size of the dataset.
29

---

Published as a conference paper at ICLR 2025
Reclustering. Here, must analyze two stages. The first stage consists of forward passes to embed the subsampled data points. We assume that our model is an M-layer feedforward network to obtain
quantify runtime complexity. The second stage is the reclustering algorithm. Combined, a complexity of O(N M D + T k N), where N is the number of subsampled data points for reclustering, M is the total number of layers of the network, k the number of clusters and 7 the maximum number of clustering iterations (we use the sklearn default of / = 300). We stress that while the first term seems prohibitively expensive, it usually heavily parallelized through batched GPU implementations of deep learning frameworks such as PyTorch.
Momentum resets. We just set the centroid momentum tensors to zero, which requires negligible compute.
G.2 EMPIRICAL RESULTS
All results discussed in this section generated using Intel Xeon Silver 4214R CPU @ 2.40GHz server with 12 physical and 48 virtual cores and 2 Nvidia A100 GPUs with 40GB of VRAM each.
For our empirical investigations, we measure the time for each BRB component individually in seconds (10 seed averages) and compare them with the time of one epoch of deep clustering training. Table Table88shone shows time estimates for IDEC+BRB on the MNIST dataset, including the constant (as function of the number of samples) and negligible overhead for the perturbations. We separately report the time for embedding and clustering, which together comprise the reclustering phase.
Table 8: Runtime in seconds for IDEC+BRB on MNIST with different subsample sizes.
Subsample size
Reset time
Embedding time
Cluster time
Momentum reset time
BRB time
Total runtime
BRB of total runtime
700 (1%) (1%
0.007
0.639
0.063
0.001
0.710
5.025
14.131
7000 (10%)
0.006
0.630
0.337
0.001
0.975
5,417
17.991
17500 (25%)
0.007
0.710
0.866
0.001
1.583
5.872
26.964
35000 (50%)
0.007
0.803
1.862
0.001
2.672
7.063
37.830
70000 (100%)
0.007
1.068
3.647
0.001
4.724
9.113
51.835
For our experiments, we apply BRB every 20th epoch, and thus, the runtime cost of BRB is distributed over 20 epochs. In this case, reclustering the full dataset leads an increase in the total runtime of the clustering phase of 5.38%, which is reduced to 1.1% when only 10% of the samples are used. 20 (
We calculate these numbers using the formula time(BRB) (NBBBNBBBB)
(Total runtime BRR BBRB In [T<(aa
time) our opinion, the overhead added by BRB is minor compared its benefits and could, for instance, be further reduced by caching embeddings. Note that while the time increases with more frequent application of BRB, subsampling can further reduce the runtime at minimal cost in clustering performance, as shown in Table below:
Table 9: MNIST ARI for different clustering algorithms and subsample sizes.
Subsample size
IDEC
DEC
DCN
700 (1%)
91.84 + 1.99
90.94 + 2.37
89.19 = 1.59
7000 (10%)
93.04 + 1.90
90.04 + 2.23
91.99 + 0.23
17500 (25%)
93.06 + 1.90
89.99
2.43
91.97 + 0.22
35000 (50%)
93.05 + 1.90
90.49 + 1.95
92.08 = 0.24
70000 (100%)
93.02 + 1.90
91.60 + 1.93
92.00 = 0.21
In Section 5 we state that "any centroid-based method" can be used for the reclustering stage of BRB. In Table 10 we analyze four different reclustering algorithms in terms of runtime: k-Means (Lloyd) 1982). M-Means++ initialization (Arthur & Vassilvitskii 2007), M-Medoidds (Rdusseeun & Kaufman 1987) and the expectation expectation-maximizzaii maximization (EM) algorithm (Dempster et al.) 1977). To calculate the
percentage-based timings, we the same formula as above. Unsurprising ly, HM takes the longest
30

---

Published as a conference paper at ICLR 2025
to run it has calculate full covariance matrices for all components of the fitted mixture model. t-Means++-init is the fastest, hinting at a possible strategy to speed up BRB. Our choice of k-Means provides simplicity while having intermediate speed.
Table 10: Runtime in seconds for IDEC+BRB MNIST with different reclustering algorithms.
Clustering method
Reset time
Embedding time
Cluster time
Momentum reset time
BRB time
Total runtime
BRB % of total runtime
EM
0.006
0.465
8.944
0.001
9,416
10.239
91.968
k-Means
0.006
0.429
0.500
0.001
0.936
01.725
54.277
k- -i-Mens++-init
0.007
0.440
0.019
0.001
0.467
01.295
36.088
k-Medoids
0.007
0.446
2.768
0.001
3.222
04.007
80.410
H ADDITIONAL RESULTS
In the following pages, we provide additional results and figures augmenting the findings of the main paper. We begin with ablation studies concerning embedding and momentum resets in Sections H.] and H.2 respectively. An illustrative example of embedded space changes induced by BRB comes next. Then, we report ARI and NMI scores (see Section D) for our main experiments with and without pretraining. This is followed by ablation study how important the contraction part of soft resets are in Section H.13
H.1 EMBEDDING RESET ABLATION
Figure shows an ablation for BRB with and without Embeilding-Rese,, for T 20 and a 0.8 (same settings as in Experiment section of the main paper). We find small but consistent drops in average clustering accuracy for GTSRB and OPTDIGITS. Results for USPS and MNIST are mostly unaffected by embedding resets. Further, we found in initial experiments (not shown here) that embedding resets affect performance more severely for higher resets (a 0.7). Thus, we do not use embedding resets in BRB.
90
80
MNNIST
GTSRB
USPS
OPTDIGTS
Baseline BRBB (ours) Baseline+ BRB (Embedding-Resett)
Figure 11: Embedding reset ablation. Clustering accuracy averaged over ten seeds for baseline methods DCN/DEC/IDEC. Using weight resets on the embedding (Embedding-Reed -eset) leads to performance drops for GTSRB and OPTDIGITS, whereas performance on USPS and MNIST is unaffected by it.
31

---

Published as a conference paper at ICLR 2025
H.2 MOMENTUM RESET ABLATION
DEC and IDEC use learnable centroids that we optimize using the momentum-based Adam (Kingma & Ba 2013) optimizer. In Figure 12 compare BRB with and without momentum resets for 20 and a = 0.8 (same settings as in the Experiment section of the main paper). We find that clustering accuracy drops considerably for IDEC on GTSRB and DEC on OPTDIGITS, while performance remains mostly unaffected for USPS and MNIST. To avoid large performance drops, we decided always reset momentum in BRB for DEC and IDEC.
Accuracy
MNIST
USPS
OPTDIGITS
DEC+BR8 BRB (ours)
2909
IDEC BRB (ours)
DEC BRB (No Momentum Reset)
2000
IDEC 3R8 (No Momentum Reset)
Momentum reset ablation for DEC and IDEC. Clustering accuracy averaged over 10 seeds for DEC and IDEC each. While performance for MNIST and USPS is almost unaffected if No Momentum Resets are used, it drops considerably for IDEC on GTSRB and DEC on OPTDIGITS.
Figure 12:
H.3 RESNET BLOCK RESET ABLATION
Table IT shows the impact of resetting the different blocks of the ResNet18 on cluster accuracy. Note that the ResNer18 consists of four ResNet blocks, where blocks with a higher number are closer to the MLP encoder used in our architecture. This means that ResNet block is closest to the embedding, and ResNet block 1 is closest to the input. We that resetting block 1 does not induce any cluster improvement over the setting without any block reset (first line). We believe this is due to the lack of sufficient change induced in the embedding. The other extreme is resetting all ResNet blocks (last line), which leads to a large drop in performance. Resetting block 4 outperforms other block reset combinations in this experiment.
Reset ablation for IDEC+BRB of different ResNet18 blocks CIFAR10 with and without reset of MLP encoder and fixed a = 0.8 and T = 10 for contrastive auxiliary task, The ResNet18 consists of four ResNet blocks, where blocks with higher number are closer to the MLP encoder, so ResNet block 4 is closest to the embedding, and ResNet block 1 is closest to the input. We report mean cluster accuracy with standard deviations over runs.
Table 11:
Block reset
With MLP-Resee
Without MLP-Reset
No Block reset
82.39 1.29
82.15 1.55
82.36 + 1.12
82.22 + 1.41
84.56 = 1.69
84.29 + 2.90
1,2
82.17 = 0.93
82.03 + 0.95
4,3
81.29 = 1.82
81.73 + 1.90
4,3,2,1
68.84 5.46
70.56 6.08
H.4 RECLUSTERING ALGORITHM ABLATION
To isolate the effect of different reclustering algorithms BRB, conduct ablation study, evaluating k-Means (Lloyd) 1982), k-Means++ initialization (Arthur & Vassilvitskil (007). k- Medoids (Rdusseeun & Kaufman 1987). and expectation pnpecttaiiommaxiizzztioon (EM) (Dempster al. 1977). Due to the computational cost of some algorithms, we use the smaller USPS dataset and run 10 trials for each method.
Table 12 presents the ARI for DEC, IDEC, and DCN using these reclustering methods. EM outper- forms the others, achieving an average ART of 79.46, while k-Means achieves 76.98. However, EM's
32

---

Published as a conference paper at ICLR 2025
runtime is significantly longer, which discuss further in Section G.2 This effect is most notable for larger datasets. Additionally, using BRB with EM is substantially more unstable, as can be seen by the increased standard emror.
Given BRB's emphasis on scalability and simplicity, we choose k-Means as the reclustering algorithm. Although EM may offer improved performance, its higher computational cost makes it less suitable for our application.
Table 12: ARI for DEC, IDEC, and DCN on the USPS dataset averaged over 10 seeds.
Reclustering algorithm
DEC
IDEC
DCN
Average
EM
79.93 + 2.95
82.24 + 2.28
76.22 + 1.02
79,46
M-Means k- int
75.98 + 1.27
79.88 + 0.33
75.09 1.54
76.98
73.70 = 1.60
83.22 + 2.89
67.87 + 1.77
74.93
k-Medoids
70.27 + 2.51
80.22 + 2.40
75.65 1.89
75.38
H.5
VORONOI CELLS
To generate 2D visualizations of the embedded space, select subset of 4 classes on the USPS
(H 1994) dataset and embed these into two dimensions in Figure [3] We use DEC as clustering algorithm.
(a) Pre BRB
(b) Post BRB
Figure 13: Visualization of the Voronoi cells before and after BRB's s combination of weight reset and reclustering. The plot shows how the decision boundary is moved post-BRB. Additionally, inter-class distance is increased, leading more intermixing of clusters. Points close the Voronoi cells are marked as "x".
33

---

Published as a conference paper at ICLR 2025
H.6
ADDITIONAL METRICS WITH PRETRAINING
In Figures 14a and
14b
we report additional NMI and ARI results for experiments with pretraining.
KMN
(ours) Baseline + Recluster
(ours) Baseline + Recluster
Baseline
Baselina
(a) Performance experiments with pretraining measured Normalized Mutual Information (NMI. NMI (averaged over DEC/DDE/DCN and ten sceds) for BRB and important comparison meth- ods: Baseline Reset refers performing only weight resets (Eq. D BusselineRReclusee refers only reclustering and Baseline are DEC/IDEC/DCN without modifications.
(b) Performance experiments with pretraining measured Adjusted Rand Index (ART), ARI (averaged over DEC/IDECDDNN and ten seeds) for BRB and important comparison methods as in Figure 14a
Figure 14: Additional results for experiments with pretraining.
H.7
ADDITIONAL METRICS WITHOUT PRETRAINING
Figures
15a and 15b
we report additional NMI and ARI results experiments without pretraining.
OPTDIT
FMNIST
OPTDICTT
FMNNSSS
Baselina-EBh (ours)
Bassline Daseline
Easeline Reset BB Baseline + +Rcclstter
Daseline
(a) Performance experiments without pretrain- ing measured Normalized Mutual Information (NMI). NMI (averaged over DEC/IDEC/DCN and ton seeds) for BRB and important comparison meth- ods in Figure 14a
(b) Performance experiments without pretraining measured Adjusted Rand Index (ARI). ARI (averaged over DEC/IDC//DNN and ten sceds) for BRB and important comparison methods as in Figure 14a
Figure 15: Additional results for experiments without pretraining.
34

---

Published as a conference paper at ICLR 2025
H.8
DETAILED ABLATIONS PER METRIC TABULAR FORM
Tables 13
14 and 13
show detailed results of our ablation experiments with and without pretraining.
Table 13: Clustering accuracy for various datasets.
MNIST
USPS
OPTDIGITS
FMNIST
KMNIST
GTSRB
AVERAGE
DEC
87.9723.66
60.29 13
62.0143.33
58.30 38.30E1.50
48.29%1.94
47.0002.20
60.64
DEC-BRB
93.55=1.95
74.20=1.66
81.52+2.20
59.83:0.42
55.73=1.09
57.71#2.33
70.42
DEC-Recluster
83.41=2.37
48.79+4.26
74.97+3.16
57.50+2.02
48.6942.00
53.02+2.26
61.06
DHC+Reset
96.12+1.46
53.4745.78
78.6442.18
59.98+0.65
57.73+1.08
60.97+2.93
67.82
TDBC
75.77=4.09
66.67#3.33
69 6842,47
54.23=1.91
54.27+1.22
$2.45=1.62
62.18
IDEC+BRB
85.38+2.47
84.50+2.68
2..60:2.20
60.49:1..0
64.10&1.25
IeeC+Recluster
7..71=1.78
74.49:2.60
18.1221..9
13.93+1.39
12.92+1.57
1..54=1.16
66.62
IDEC+Reset
63.01=4.15
72. 16.16611
73.34
58. 38.0422.
59.02+1.43
15.51=1.65
63.51
90.65=2.01
75.492
87.
54.64H100
58,462
61.721.33
71.3
DCN-BRB
95,90=0.12
79.02+1.44
08.54
58.55=0.40
64.21
64.58=141
74.63
DCN-Recluster
92.99x1.27
78.82+0.56
88.4840.41
55.5110077
63.50&11.05
1.59
72.87
DCN-Reset
86.94s2.06
75.58+1.29
84.52+1.72
54.29+1.077
63.79+1.60
10.40
69.25
DEC*Pretrain
91.56=1.98
78.94+ 09
88.21#1.08
60.00+00.58
65.28+1.04
62.1662.64
74.36
DEC-BRB-Pretrin
91.05+2.12
78.63+0.60
92.32+1.70
59 6440.8877
65.97+1.14
61,73+2.24
74.89
DECCRecluser++Pretriin
90.68&1.84
78.6540.63
10.52
60.3220.50
62.15+2.63
DEC-Reset-Pretrale
93.69:2.15
79.40=:074
89.0241.37
60.74:0.52
64.37=0.87
62.40=2.42
74.94
IDEC*Pretrain
93.1622.20 20
81.5910.28
86. 8310.21
53.5911.01
65.0911 .03
56.6322.1
72.81
IDEC-BRB-Preteinn
94.19=2.20
81.87=0.14
07.35+0.36
1..96
65. 65.61#104
62.84=1.89
74.64
IDECCAReclustrrrPeetnnn
94,24+2.13
81 $1.08+0.
0420.00
53.61=0.93
13+11.01
56.94 34
72.97
IDEC=Reset-Pretrain
89.69a2.37
81.3810.08
85.9 85.9910.53
$3.6611..25
66.05.0.97
60.01
72.8
DICN-Pretrain
89.23+2.04
78.2940.97
86. 64.6444
49.91#4)661
1640.92
54.68+2.33
70.32
DCN-BRBB-rettsin
96.4020.12
80.10:1.44
87.9240.13
52.3420.71
65.19:0.92
65.8921.23
74.64
DCN- MeNN-Reeluster-Pretrall
93.8620.78
77.06
86.06+0.55
52.72::1.33
63.76:0088.
56.42+2.93
71.65
DCN+ CCN--Reett+Pretrail
90.98+2.21
78.64+0.79
37.66+0.21
49.22+0.62
64,25+0.96
48.48*33.38
69.87
Table 14: NMI performance comparison various datasets.
USPS
OPTDIGHTS
KMNIST
AVERAGE
DEC
86.49= 1.96
57.9742.17
70.25+2.80
59.32+1.05
43.1542.19
27.01=2.94
57.36
DEC-BRB
91.361 1.18
76.90=0.76
87.79x1.07
61.6810.50
52.1121.21
41.50:1.96
68.56
DEC-Recluster
83.00=1.85
45.0244.79
79.21=2.30
59,3740.89
42.93+1.71
37.5922.88
57.85
DEC+Reset
93.2620.88
49 19.9847.15
1..75+1.32
62.26:+0.66
13.11
46.03#2.11
64.56
IDEC
80.23+3.99
70.95+2.84
76. 16.8911.88
59, 19.76:11.20
54.46+1.24
33,19=1.58
62.58
IDBC+BRB
91.25+0.88
89.39+0.93
85.85+1.04
65.17+0.73
58.76+1.31
46.39=1.06
72.8
IDEC#Recloster
81.52+2.67
1.05
60.4010.76
54.8011.42
1.899x1.83
IDEC+Reset
81.17+2.16
78.42+1.66
11.43+1.75
63.33+0.85
59.74=1.22
37.35+1.62
66.91
DCN
87.48=0.79
78. 78.6973.73
87.08:0.35
60. 60.1620.57
$7.7240.65
46.42=1.36
69.59
DCN-BRB
91.13=0.19
79. 79.78+0.66
06.36600.66
61.62+0.25
61.81:0.40
47.15=0.97
71.31
DCN-Recluster
88.26=0).77
80.72+0.35
87.24
60.91+0.37
59.1740.41
45.03+1.57
70.22
DCN-Reset
88.8320.68
77.72+0.55
84.6110.74
60.5410.83
61.3110.75
38.28
68.05
Prei-reeruin
91.85=0.11
84.32+0.79 79
89.58+0.67
62.75=0.38
62.89H0
48.37=2.29
73.29
DEC-BRB-Pretrain
91.71a0.98
84.00+0.46
91 91.8011.00
62:16:00
64.6440.37
47.80&2.11
73.68
DEC-Reclaster-Pretrrin
90.9720.87
84.0410.68
90.6420.81
62.7420.42
62.9940.31
48.34=2.27 27
73.29
PeCc-esct-
93.0920.98
85.38+0.65
90.07=0.80
02.7200.48
63.51
48.39=2.21
73.86
IDEC*Pretral
94.1320.78
87.14
39..0.1114
02.3120.82
63.7720.36
17.3
72.21
IDEC+BBB-
94,88+0.67
88.11+0.36
89.29+0.2
63.83+0.86
65, 12+0..25
44.01=1.33
74.21
PDEC-Reclaateer+/hetraan
94.3620.80
87.0840.43
88.40&0.12
62.3010.75
3.93
93.55a0.77
88 00+0.35
2940.27
62. 22.25e1.00
65.6340..4
41.45+2.14
73.2
DCN-Pretrain
87.1910.90
79.2910.67
85 2242000660
58.6 58.6240.64
$7.7540.2
$7.82+2.25
67.65
DCN-BBBB-Pettrai
91.81+0.19
80.3540.72
86.2240.24
57. 97.9770040
61.01:0.31
47.55&1.86
70.82
RCN--Rcclsterr-eeerain
88.8240.47
79.23+0.60
84.43+0.76
1..9521.01
18.1110.21
38.40=2.72
67.99
DCN- resett+retraii
90.20=0.74
80.22+0.54
85. 87.0.30
55.34+0.84
60.44+0.34
30.38+4.09
67.08
35

---

Published as a conference paper at ICLR 2025
Table 15: ARI performance comparison on various datasets.
MNIST
USPY
OPTDIGHTS
FMNIST
KMNIST
GTSRB
AVERAGE
DEC
82.772365
47.02=2.67
55.76t3,34
43.8911.
18.112000
DEC-BRB
89.7742.24
67.5741.31
78.7241.92
47.1140.75
38.61&1.32
31.77+1.97
58.92
DEC-Recluster
76.6842.88 88
35.9424.74
69.4343.16
45.1821.30
30,5621446
28.3742.49
47.69
DEC-Reset
93.07=1.73
42.49+6.58
72.90=2.05
47.68=0.74
39.59=1.14
36.76:2.78
55.41
IDEC
70.3925.16
59.1824.15
62.1723.08
42.9511.59
38.8921.39
25.762
49.89
IDBC+BRB
84.72+2.09
83.00=2.67
1,90
1..25=1126
41.89+1..0
36,55+1_39
62.07
IDCC=Recloster
72.33a2.62
72.341.2.56
73.40&1.88
42.84x:1.06
38.71a1.62
34.65&1.52
55.71
IDEC+Reset
60.9724.80 80
68.1142.11
69.0743.14
1.37
43.23:40.2
1.78
DCN
85,41=1.69
69.9771.03
81.3611.22
43.2110.79
42.8911.62
$5.20=1.00
59.67
DCN-BRB
91.21=0.25
71.89:1.40
81 0177670777
43.66:0.32
46.66#0.75
16.44=1.66
61.9
DCN-Recluster
87.33=1.03
73.1840.71
02.08:0.54
43.57=0.62
44.79:0.62
32.63=1.70
60.6
DCN-Reset
84.38=1.73
68.79=0.83
1..6
43.32=0.97
46.69=1.10
23.67=1.09
57.42
DEC=Pretrain
88.8621.99
1..
83.4121.28
47.0550.62
47.4010.55
40.09=2.70
63.73
DEC-BRB-Pretrain
88.28+2.17
74.34+0.91
88. 1.91
0..9000.22
50.05:0.59
38.91=2.44
64.43
BBC-RRcluster-PPren
87,50=1.92
74.63+1.20 1.00
85.92+
47 47.
47.36+0.33 38
40.07+2.69 69
63.82
DEC=Reset-Pretrain
91.12+2.24
77.04+0.88
84.47&1.60
47 47.6340.73
48.6340.34
39.93a2.59
64.8
PDeC*Pretrain
91.73+2.03
79.26+40.39
88
30.42+2.05
IDEC+BRB+Pretriin
93,05=1.89
79.89=0.38
82.6620.36
45.40:0.99
48.9110.91
37.69=1.93
64.6
PBCC-Reclster--Pretrinn
92.60=2.03
79.0S=0.43
81.8700.18 18
42.88=0.81
48.07 79
30.69=2.08
62.53
IDEC~Reset-Pretrrai
88.65+2.12
79.72+0.35
14:0.66
43.33+1.18
34.29=2.32
DCN-Pretrain
84.26=1991
71.4811 .09
79.47=0.66
40.03=0.60
44.24+0.55
27.51=2.41
57.83
MCN--BRB-Pretaal
92.20=0.26
73.19=1.78
80.99=0.23
40,83+0.45
46.74
39.37=1.68
62,22
CCN-Reclustar-Pretriin
88.1840.91
70.71
78.533.0.8 888
1..40::..25
44,4510.44
29.60s2
58.81
DCN- Prrttttnnn
87.67=1888
71.521.11
80.31
37 37.24+0.63
46.01#0.75
22.04=3.27
57.47
H.9 DETAILED RESULTS FOR CONTRASTIVE AUXILIARY TASK AND SELF-LABELING
Table 17 shows the mean cluster accuracy over ten runs for DEC, IDEC and DCN with and without BRB. We that BRB always improves performance and leads to more than 2% increase for DCN on CIFARIO. Using self-labeling as proposed in SCAN (Gansbeke et al. 2020) we achieve even higher cluster accuracies stated in Table Io While DEC does not benefit much from BRB, find that IDEC+BRB and DCN+BRB clearly outperform SCAN by up to 6% on CIFAR 100-20.
Table 16: Ablation results with contrastive auxiliary tasks and self-labeling. Reporting mean cluster accuracy with standard deviations over 10 runs. Results from SCAN with self-labeling for reference (SeCu did not provide average performance results).
Methods
CIFAR10
CIFAR100-20
DEC
85.77 2.28
48.37 + 1.39
DEC BRB
86.98 2.28
47.18 1 2.63
IDEC
85.89 + 2.04
48.85 + 1.70
IDEC + BRB
86.50 3.24
50.00 2.12
DCN
85.25 - 3.11
51.13 + 1.65
DCN BRB
89.29 0.72
52.32 + 1.95
SCAN (Gansbeke et al. 2020)
87.60 0.40
45.90 + 2.70
Table 17: Ablation performance with contrastive task only. Reporting mean cluster accuracy with standard deviations over 10 runs. Per method, the best result is set in bold, and the overall best result underlined. BRB enables consistent improvements for DCN and IDEC on all datasets.
Methods
CIFAR10
CIFAR100-20
DEC
83.00 + 1.65
46.23 = 1.17
DEC + BRB
83.86 1.72
46.63
1.32
IDEC
82.12 + 1.54
46.79 + 1.84
IDEC + BRB
84.35
1.06
47.92 + 0.87
DCN
79.64 + 1.98
44.03 + 1.13
DCN BRB
82.16 = 0.57
44.89 + 1.49
36

---

Published as a conference paper at ICLR 2025
H. DETAILED ANALYSIS OF INTER-CD AND INTRA- WUTRA-CD BEHAVIOR FIGURE 6.3
The behavior of the continually increasing intra- and inter-class distances (CD) of DCN(+Reclustering) for MNIST in Figure 6.3 can be analyzed through the silhouette score Rousseeuw (1987), which is defined as the ratio between the intra- and inter-CD. The baseline models exhibit proportional increases in intra- and inter-CD, resulting in stable silhouette score (Figure I6p. This indicates that the optimizer expands the embedding space without improving cluster separation, likely due to the absence of weight decay. contrast, BRB's s soft resets help maintain small parameter and gradient norms during training (Ash & Adams 2020). This allows BRB to effectively reduce intra-CD while increasing inter-CD, leading to enhanced cluster separation and higher silhouette score.
Accuracy
90
85
Intra CD
25
20
Inter CD
50-
CL Change Silhouette
40
10
20
100
200
300
400
Epochs
DCN DCN+ Recluster
DCN+BRE (ours)
Figure 16: Behavior of intra- and inter-CD Figure 6.3 of our main paper augmented by showing the silhouette score addition to average clustering accuracy, inter/intra-class distance (inter/intra-CD) CD), and cluster lahel change (CL. Change) for DCN MNIST. The silhouette score is defined as the ratio between the intra- and inter-CD. The plot shows that without BRB, the proportional rise in intra- and inter-CD leads a flat silhouette score, indicating rising embedding magnitudes without improving separation. In contrast, BRB's 's soft resets counteract increases in the embedding norms, allowing the cluster loss to better separate ground truth classes over time (lower Intra-CD, slowly increasing silhouette score).
H.11 BEHAVIOR OF SILHOUETTE SCORE ON PREDICTED LABELS
Figures 17a and 17b show the silhouette scores for DCN on USPS and OPTDIGITS using either the ground truth or the predicted class labels. For USPS, we can observe that BRB starts improving over the baseline in terms of clustering accuracy as early as the first reset. Using the ground truth labels, this improvement also is reflected in higher silhouette score. When using the predicted labels to calculate the silhouette score, BRB only has substantially higher silhouette score after episode 250 despite significantly improved clustering accuracy. On GTSRB (Figure 17b). the difference between using ground truth and predicted labels is even more stark. First, note that BRB significantly outperforms the baseline in clustering accuracy. When using the silhhuutte score with predicted labels, the high values would suggest that both algorithms achieve strong separation, with the baseline outperforming BRB. However, the silhouette score is noticeably higher for BRB compared to the baseline when using the ground truth labels, with both the baseline and BRB achieving lower scores
37

---

Published as a conference paper at ICLR 2025
65.0
60.0
72
55.0
50.0
12
15.0
12.5
Sihoeetee score (predicon labeel Sthouette Igeoue
Sihouctte sore Igrunng
10.0
5.0
0.0
60
40
20
Epochs
Epochz
BRB
Basding
BRB Baseine
(a) Silhouette score with predicted and ground truth labels for DCN on USPS. The figure shows that with ground truth labels, BRB has substantially higher silhouette score than baseline. Using the predicted labels to calculate the silhouette score dis- torts the results. Results averaged over ten seeds.
(b) Silhouette score with predicted and ground truth labels for DCN on GTSRB. Despite signif- icantly higher clustering accuracy for BRB, the sil- houette score with predicted labels indicates superior performance for the baseline. Using the ground truth labels, BRB has higher silhouette score than the baseline. Results averaged over ten seeds.
Figure 17: Silhouette scores with predicted and ground truth labels for DCN.
compared using predicted labels. These experiments highlight the perils of using unsupervised metrics and predicted labels when assessing the performance of deep clustering algorithms.
38

---

Published as a conference paper at ICLR 2025
0.28
B16
Loss
2.04
0.23
0.54
180
150
250
250
30m
350
105
9.58
2.50
ACC
2.84
2.52
0.50
100
150
200
250
300
350
2.84
3.80
Epochs
Figure 18: Behavior reconstruction loss DCN on OPTDIGITS (10 seed average). Whenever soft parameter reset is performed, the reconstruction loss spikes before dropping down again. Overall, the loss value after a reset is on similar level to the reconstruction loss without resets.
H.12 ANALYSIS RECONSTRUCTION ERROR WITH BRB
we analyze the behavior of the reconstruction loss during training of DCN on OPT- DIGITS. With each soft parameter reset, the reconstruction loss first spikes before coming down to similar level compared to when reconstruction loss is used. In terms of clustering accuracy, BRB's beneficial effects on the structure of the latent space outweigh the slightly higher reconstruction loss.
In Figure 18
39

---

Published as a conference paper at ICLR 2025
H.13 ABLATION STUDY ON THE COMPONENTS OF SOFT RESETS
In Section 5
we define soft reset according to Equation ( which we restate below:
w (0:) ab, (1-a)
(19)
where oj are the network's parameter at step and o' are new parameters sampled from the initial- ization distribution. is hyperparameter specifying trade-off between contracting the current parameters and the strength of the added noise. In the following, we show two ablation studies aiming answer the following questions:
(1) How important is the contraction component of the soft reset? (2) How important adding noise as specified by Equation 19
We answer the first question by running experiments without the contraction factor for the original weights 0j in Equation (19) (Perturbation only). To answer the second question, we add Gaussian noise with mean and standard deviation according (Scaled Gaussian noise). Figures 19a and 19b] show our results for DCN trained on the datasets USPS and OPTDIGITS. We
the current weights instead of Equation 19
that removing the contraction part from Equation 19 yields substantially worse performance compared to using full soft resets, which we attribute to the ability of soft resets balance gradients (Ash & Adams, 2020). Analyzing the gradient norms of the Perlurbation only ablation, we indeed observe that they are substantially higher, which in turn promotes a collapse in representation and centroid norms. In contrast, the Scaled Gaussian noise variant our algorithm performs similarly to soft resets on OPTDIGITS and little worse on USPS. Thus, the results of both ablations support performing resets according Equation
85
88
86-
80
Accurary
82
25
70
78
Scaled Gaussian
Scalled Gaussian
65
Perturbation only
Perturbation only
Full soft resets
Full soft resets
100
300
100
300
Epochs
Epochs
(a) Ablation of soft reset components on USPS. Results show the clustering accuracy for DCN, aver- aged over ten seeds.
(b) Ablation of soft reset components on OPTDIG-
ITS. Results show the clustering accuracy for DCN, averaged over ten seeds.
Figure 19: Results of the ablation study regarding soft resets.
H.14 ABLATION STUDY: k-MEANS RECLUSTERING VS. RECLUSTERING WITH DEC/IDEC ASSIGNMENTS
In addition to our experiments ablating different clustering algorithms for the reclustering step of BRB in Appendix H.4 investigate whether the clustering steps of DEC and IDEC perform better than k-Means. DEC and IDEC produce soft assignments that are hardened by assigning sample to the cluster with the highest assignment probability. Thus, the assignment is different from k-Means. Because the reclustering step of BRB is only done after the completion of epoch and only every 20 epochs, does not interfere with gradient updates during an epoch. As DCN already uses the k-Means assignment mechanism get cluster labels, we omit it from this ablation study. Table 18 shows the performance the vanilla DC models and their reclustering variations when training from scratch, and Table 19 shows the results with pretraining. For USPS without pretraining, the performance gap between BRB with k-Means and the baselines more than 10%. For OPTDIGITS with pre-training, the BRB with k-Means reclustering outperforms the baselines by more than 4% for DEC. Overall, BRB with k-Means reclustering is either performing similar or better than the baselines. Given these results, find that l-- G-Means is indeed sensible choice for the reclustering step of BRB.
40

---

Published as a conference paper at ICLR 2025
Table 18: Reclustering ablation without pretraining for k-Means original DEC/IDEC
reclustering. Average clustering accuracy is computed over 10 runs using the same settings as in the main paper.
Without Pretraining
OPTDIGITS
GTSRB
USPS
DEC
60.29
47.00
62.01
DEC+BRB DEC reclustering
67.17
57.98
71.45
DEC+BRB w. k-Means reclustering (ours)
74.20
57.71
81.52
IEC
69.68
52.45
66.67
IDEC+BRB w. IDEC reclustering
72.64
60.10
72.27
IDEC+BRB w. k-Means reclustering (ours)
80.60
64.10
84.50
Table 19: Reclustering ablation with pretraining for k-Means original DEC/DDC recluster- ing. Average clustering accuracy is computed over 10 runs using the same settings as in the main paper.
With Pretraining
OPTDIGITS
GTSRB
USPS
DEC
88.21
62.16
78.94
DEC+BRB DEC reclustering
85.72
62.98
76.96
DEC+BRB w. k-MMans reclustering (ours)
92.32
61.73
78.63
IDEC
86.83
56.63
81.59
IDEC+BRB w. IDEC reclustering
86.58
59.36
80.27
IDEC+BRB k-Means reclustering (ours)
87.35
62.84
81.87
41


Reviews:
[{"review_id": "llji4hNZpx", "reviewer": "Reviewer_xoWL", "summary": "The authors explore a (known) problem when training a clustering model from scratch as follows. The architecture has an encoder followed by a clustering head. With end-to-end optimization, the encoder is updated by the gradients which flow back to it through the clustering head. This means that if and when the clustering head is saturated at a local minima, the encoder will stop learning. This can happen before the encoder has learnt a good encoding, resulting in a model which ceases to learn further despite performing poorly.\n\nThe authors demonstrate the problem, their method (a periodic weight perturbation strategy with cluster centroid reinitialization), show experiments comparing against relevant baselines, and analyze the behavior of their method.", "questions": "Why do the authors only consider the CIFAR100 supersets (CIFAR100-20) instead of clustering 100 classes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The problem is well presented and salient to the field. As far as I am aware, the proposed method is novel. The analysis is thorough, with ablations as well.", "weaknesses": "It would be better if the work studied some harder datasets/tasks, such as ImageNet-1k or some taxonomic image datasets (iNaturalist, BIOSCAN, etc.). A lot of the datasets considered are simpler digit classification datasets. The work could be further improved by evaluating on other modalities.\n\n**Minor points**\n\nSome references are badly formatted and need some attention to correct them.\n- Incorrect casing (\"em algorithm\", \"hungarian method\", \"Spagcn\", \"rna\")\n- DOIs shouldn't include the domain name and should actually be links (the latter is fixed by importing the latex `doi` package in main.tex) \"DOI: https://doi.org/10.24432/C50P49\" -> \"doi:[10.24432/C50P49](https://doi.org/10.24432/C50P49)\". When the DOI is a link, you don't need to also include the URL field.\n- It's good for readers if you can include links in the references (ideally as DOIs) as much as possible, so the reader can easily navigate to the article being cited (and know they are looking at the work you meant to cite)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v3Roc2wl6Z", "reviewer": "Reviewer_XSEo", "summary": "The paper discusses 'reclustering barrier', a common and important phenomenon that occurs in centroid-based deep clustering methods, where performance quickly saturates in the very early stage of the training phase. The authors carefully discuss when and why reclustering barrier happens and propose BRB which prevents this early commitment to initial clustering and gain performance over the training with the reinitializing clusters after every few epochs. They apply BRB on top of several existing deep clustering methods to verify its advantage. The paper contains simulation results to support its main claims.", "questions": "1. You are computing inter-CD and intra-CD to measure variation within ground truth classes (for example, in figure 2). Why is this so important? What about this metrics for the predicted clustering by BRB? I suggest to include a comparison of these metrics for both ground truth classes and predicted clusters, which would provide a more complete picture of BRB's performance.\n\n2. Again for figure 2, it seems even after 400 epochs the clustering labels are changing, but the accuracy is not. What is happening here? Are the clustering labels flipping only?\n\n3. Is BRB optimizing based on the clustering metric (say accuracy)? Or is it updating for a specific epochs (say 500) and after that you are computing the relevant metrics? In other words, is BRB using trues labels while training? At what points the evaluation metrics are computed?\n\n4. Accuracy, NMI or ARI, all of these metrics are supervised and need true labels. However, in practice, we don't have any true labels to compute these metrics. Can you include some unsupervised metric (say Silhouette Coefficient) in your evaluations and comparisons, in addition to the supervised metrics? This would provide insight into the method's performance in more realistic unsupervised scenarios.\n\n5. As BRB modifying the encoding space after each reclustering step, the encoding space may degrade and may not represent the original data point anymore, did you verify it with the original data point with reconstruction loss? I suggest to report reconstruction loss for one or two datasets comparing to baselines, which would help to understand the underlying encoding structure.\n\n6. Momentum resets section is not clear enough, can you elaborate more like how are you exactly re-initializing the momentum for the cluster centers after reclustering?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### SIGNIFICANCE:\nThis work is important because it seeks to overcome an important phenomenon commonly seen in centroid based deep clustering methods and this can be studied further over other recent deep clustering methods (also not centroid based as they may also face the similar issue).\n\n### ORIGINALITY:\nThis work is novel because it demonstrates the application of BRB over some existing methods that improves the performance further.\n\n### CLARITY:\nThis work is well-written and well-organized. It clearly explains the key components of the method and their functions. For example, the discussion about the hyperparameter $T$, $\\alpha$ and learning rate is complete and important for the reproducibility and understanding of this work. In addition, I also have some questions regarding this method (see the next section).\n\n### QUALITY:\nThe quality of this work is good. The work includes carefully designed simulations that are sufficient to support the claim of the paper.", "weaknesses": "1. The biggest weakness is the baselines (on top of which BRB has been applied) are too old to compare, the latest one is from 2017. It would be good to see applying BRB on top of one or two recent DC method (like SCAN, GCC or SeCu mentioned in the paper) and report those results. This would provide a clearer comparison to the state-of-the-art.\n\n2. Hyperparameter sensitivity. It would be good to report similar hyperparameters (T, α and learning rate) sensitivity (figure 8) to other one or two datasets besides MNIST.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3TTP2GIvR", "reviewer": "Reviewer_VgYg", "summary": "This paper addresses the issue of rapid early performance gains followed by quick saturation in deep clustering (DC). It introduces the concept of the \"reclustering barrier\" and explores when and why this phenomenon occurs. To overcome it, the authors propose the Break the Reclustering Barrier (BRB) algorithm, which avoids early over-commitment to initial clusters and allows continuous adaptation to reinitialized targets. The paper demonstrates that BRB improves performance across several clustering benchmarks.", "questions": "In addition to the questions raised in the Weaknesses section:\n\n1. alpha is a critical hyper parameter. From Figure 8, it appears that α = 0.9 yields better performance across various settings. It is unclear why the authors recommend α = 0.8 instead. Additionally, should the choice of α = 0.9 or 0.8 remain consistent across all DC algorithms, or is there room for algorithm-specific tuning? Clarifying whether this parameter can or should be personalized for different algorithms would enhance understanding.\n\n2. How does the approach extend to deep clustering methods that are not centroid-based?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses an important issue in deep clustering, performance saturation.\n- The writing is smooth and easy to follow, with the primary contributions well-communicated.\n- The empirical evaluation is thorough, and the BRB algorithm is shown to enhance performance across benchmarks.", "weaknesses": "1. While the concept of reclustering is mentioned early in the paper, the term can carry specific meanings in the context of deep clustering. Typically, reclustering refers to periodically or iteratively reassigning data points to clusters. However, since DC algorithms inherently involve iterative steps of representation learning and clustering, the authors should better contextualize their use of the term. \n\n2. Building on the previous point, many existing DC methods already incorporate some form of iterative clustering. As such, the distinction between \"DC\" and \"DC+Recluster\" groups in the paper’s figures could be unclear. The authors should explicitly explain what differentiates these two groups and how their use of reclustering is unique. This will help readers better interpret these figures.  \n\n3. The core innovation of BRB lies in its use of weight resets, which the paper notes is inspired by a technique from reinforcement learning. Weight resetting is also known as a popular regularization technique in deep learning, where a portion of layer weights is periodically reinitialized during training. While applying this idea to deep clustering is somewhat novel, this contribution could be perceived as incremental. \n\n4. Some choices in the implementation seem empirical. For example, the authors mention resetting only the last ResNet block and the MLP encoder, and they use hyperparameters such as LR = 0.001, alpha = 0.8, and T = 20. While these values are derived from extensive experimentation, it would strengthen the work to provide more intuitive or theoretical justification for these decisions. This would also make it easier to generalize the choice of hyperparameters (especially alpha) to other settings and DC algorithms.  \n\n5. The algorithms tested in the experiments are relatively older. The authors are encouraged to apply BRB to more recent deep clustering methods to better demonstrate its effectiveness and relevance. This would enhance the impact of the work and align it with the latest advancements in the field.  \n\n6.  The paper uses widely-known data augmentation techniques such as contrastive learning and self-labeling, which are not novel,  in the main results. Therefore, combining these techniques and comparing BRB with SOTA algorithms like SCAN, GCC, and SeCu may not represent a fully fair comparison. The authors should clarify whether the improvements stem from BRB itself or the use of these auxiliary techniques, ensuring a more balanced evaluation of the proposed method.\n\nMinor:\nSome grammar error:  (line 128) Its output are => its output is", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h8Q3RBc5Dx", "reviewer": "Reviewer_JvBA", "summary": "The paper addresses the problem in deep clustering of early over-commitment to sub-optimal partitions. They propose a method to combat this by periodically modifying the network weights and reclustering.", "questions": "What is the initial weight distribution? As this is a part of the reset mechanism, it should be specified in the main paper.\n\nFor the main results, how frequently do you reset?\n\nWhich image augmentations do you how and how exactly are they put to use?\n\nHow does your method compare to SOTA on MNIST, KMNIST, USPS, FMNIST, OPTDIGITS and GTSRB?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The considered problem is an important one, and is reasonably well introduced the paper.\n\nThe method is intuitive.\n\nThe experimental results on CIFAR looking convincing.\n\nThere is some helpful surrounding discussion of how the method works to improve accuracy, e.g. Figures 6 and 7.", "weaknesses": "The method itself is intuitive but maybe a bit simple--just slightly modifying the weights and then reclustering. The weight reset comprises two parts: contracting the current values and then adding noise. It would be good to compare the reset as per Eq (1) to just adding noise to the weights, from the same or similar distribution as the weight distribution (at least the same mean and variance). This would determine whether the contraction part is doing anything.\n\nThe comparison to SOTA methods on Cifar is reasonably convincing, but there's no such comparison on MNIST, KMNIST, USPS, FMNIST, OPTDIGITS and GTSRB. Does that mean your results are substantially worse than SOTA on those datasets?\n\nSome information from the appendices would be better placed in the main paper. The paragraph 'implementation details' is mostly just pointers to various sections in the lengthy appendices, when it could just give the details directly. Specifically, it would be good to specify (1) the initial weight distribution, (2) the frequency across epochs that you performing resetting, and (3) which image augmentations and how exactly they are used. Presumably you just create multiple augmentations for each image and enforce them all to be in the same cluster, but I wasn't sure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)