Enhancing Contrastive Clustering with Negative Pair-guided Regularization
Authors: Abhishek Kumar
Anish Chakrabarty
Sankha Subhra Mullick
Swagatam Das
Venue: Accepted by TMLR ()

Abstract: Contrastive Learning (CL) aims to create effective embeddings for input data by minimizing the distance between positive pairs, i.e., different augmentations or views of the same sample. To avoid degeneracy, CL also employs auxiliary loss to maximize the discrepancy between negative pairs formed with views of distinct samples. As a self-supervised learning strategy, CL inherently attempts to cluster input data into natural groups. However, the often improper trade-off between the attractive and repulsive forces, respectively induced by positive and negative pairs, can lead to deformed clustering, particularly when the number of clusters $k$ is unknown. To address this, we propose NRCC, a CL-based deep clustering framework that generates cluster-friendly embeddings. NRCC repurposes Stochastic Gradient Hamiltonian Monte Carlo sampling as an approximately invariant data augmentation, to curate hard negative pairs that judiciously enhance and balance the two adversarial forces through a regularizer. By preserving the cluster structure in the CL embedding, NRCC retains local density landscapes in lower dimensions through neighborhood-conserving projections. This enables the application of mode-seeking clustering algorithms, typically hindered by high-dimensional CL feature spaces, to achieve exceptional accuracy without needing a predetermined $k$. NRCC's superiority is demonstrated across various datasets with different scales and cluster structures, outperforming 20 state-of-the-art methods.

Published in Transactions on Machine Learning Research (11/2024)
Enhancing Contrastive Clustering with Negative Pair-guided Regularization
Abhishek Kumar
abhishek. kww.ek...mmaeeeeee.e..e wtw.:reeeee111111. www.mmm.......iit...oo.o....
ENET Centre, Centre for Energy and Environmental Technologies VSB- Technicol University of Ostrava, Ostrava, Czech Republic.
Anish Chakrabarty
chakmbbart.ammish wwtpttt..aaiisss@aaaaaa
Statistics and Mathematics Unit Indian Statistical Institute, Kolkata, India
Sankha Subhra Mullick
mullicksonkavvvkkne@@maaal.ccc
Dolby Laboratories, India
S Swagatam Das Electronics and Communication Sciences Unit Indion Statistical Institute, Kolkata, India
Reviewed OpenReview: https: https:///penreeviau
openreview. net/ sermf id= VYsg0MMe
Abstract
Contrastive Learning (CL) aims to create effective embedding for input data by minimizing the distance bet ween positive pairs, iee, different augmentations or views of the same sample. To avoid degeneracy, CL also employs auxiliary loss to maximize the discrepancy between negative pairs formed with views of distinct samples. As self-supervised learning strategy, CL inherently attempts cluster input data into natural groups. However, the often improper trade-off between the attractive and repulsive forces, respectively induced by positive and negative pairs, can lead to deformed clustering, particularly when the number of clusters is unknown. To address this, we propose NRCC, CL-based deep clustering framework that generates cluster- cluster-frendlly embeddings. NROC repurposes Stochastic Gradient Hamiltonian Monte Carlo sampling as an approximately invariant data augmentation, to curate hard negative pairs that judiciously euhance and balance the two adversarial forces through a regularizer. By preserving the cluster structure the CL embedding, NRCC retains local density landscapes in lower dimensions through neighborrhood-cousevving projections. This enables the application of mode-secking clustering algorithms, typically hindered by high-dimensional CT. feature spaces, to achieve exceptional accuracy without needing predetermined k. NRCC's superiority is demonstrated across various datasets with different scales and cluster structures, outperforming 20 state-of-the-art methods.
1 Introduction
Contrastive learning is self-supervised learning technique that trains model by encouraging it to distinguish between similar and dissimilar pairs of data points, enhancing its ability to learn meaningful representations (Chen et al. 2020b). A representation can be considered accurate if two semantically similar samples lic close to each other in the feature space while the dissimilar ones reside al a distance. In other words, the learned embedding space should geometry-aware, preserving the local neighborhood as well as distinguishing the natural groups present the data. Hence, CL methods attempt learn feature mapping by minimizing the distance between a positive pair. Such pair is formed by couple of augmented variants of the same sample and thus resides in close proximity, crowding the local neighborhood. Further, to maintain considerable

---

Published in Transactions on Machine Learning Research (11/2024)
separation between dissimilar imputs, CL methods may actively encourage maximizing the distance between a negative pair consisting of augmented views of two distinct samples. This auxiliary CL. objective also safeguards the learned representation from collapsing into a degenerate solution. Specifically, the positive pairs induce an attraction while the negative pairs contribute a repulsion, resulting in a balanced interplay that sculpts an ideal embedding. However, such a crucial balance is challenging to achieve due to the unregulated repulsion induced by the negative pairs. Often, traditional CL methods show excessive hunger for negative pairs, severely affecting the computational cost. As a remedy, more recently, CL methods use only the positive pairs in a dlamese-inspired architecture along with regulated gradient flow (Grill et al.) 2020). Specifically, one branch of the siamese network called online trains on the positive pair using typical gradient descent. The other branch, known as target, is updated through the moving average of the online branch parameters and, at the end of the training, acts as the feature encoder.
Deep Clustering (DC) methods address the classical unsupervised task of assigning imput data to distinguishable natural groups or clusters using deep neural networks. Both DC and CL methods share a common interest in sculpting an embedding that conserves natural groups. Hence, CL often comes to the aid of DC methods (Li et al. 2020a). However, an embedding learned by CL method may contain various irregularities (Chen & He 2021 Huang et al. 2022). For example, while generating a negative pair, if the two considered samples belong to the same cluster or, worse, lie in a local neighborhood, then that may culminate in deformed cluster structures in the CT feature space (Chuang et al., 2020j. Tf such an embedding is directly used for clustering, the performance is highly likely to be compromised even if the number of clusters & is known. The remedies usually involve simultancous optimization of CL and DC objectives, often with tailored clustering methods (Li et al. ) and always requiring the k in advance (Huang et al. 2022). Moreover, the CL. embedding resides in high-cnough dimensional space where applying state-of-the-art mode-sccking clustering algorithms (Jang & Jiang 2021) that do not require prior knowledge of k is infeasible (Huang et al., 2022). Given that the cluster structure has a risk of not being adequately reflected in the representation, mode-seeking clustering
may also turn futile.
To address these issues regarding the applicability of CL in DC, we propose a Negative pair-based Regularizer for Contrastive Clustering (NRCC). The motivation behind NRCC is to judiciously introduce a higher repulsive force through tailored negative pairs that best preserve cluster structure in the CL embedding. There are two obstacles to this approach. First, if the negative pair-indueed repulsion supersedes the positive pair-driven attraction, that may lead to undesirable fragmented clusters. On the other hand, in the absence or lack of adequate repulsion, clusters may collapse on each other. Second, if the negative pairs are generated using conventional and unregulated data augmentation techniques (Chen et al. 2020b): they suffer significant distortion if made to reside at greater distances. As such, bringing to bear a heightened repulsion comes at. the price of the data density getting deformed. These factors motivate us to explore two remedial directions. First, we search for a data augmentation strategy that generates an approximately invariant (Chen al.) 2020ah view with enhanced departure, still limiting the dissimilarity from the original data distribution. Specifically, the augmentation produces a density-aware crowding of views in the vicinity of the parent sample. The resultant hard-negative pairs exert only enough repulsion to curve out finer cluster boundaries. Sccond, we construct a regularizer that seamlessly integrates the negative pairs in the existing CL frameworks like traditional InfoNCE (Chen et al. 2020b) or positive-pair-oii
maamese-inspires BYOL (Grill et al. 2020).
We consider ro- ro-purposing the Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) sampling method to address the first hurdle as an augmentation strategy (Chen et al. 2014). As per our knowledge, this is the first work where the applicability of SGHMC (or in general sampling technique) as potential augmentation method in CL has been investigated. Originally designed as a scalable improvement over the Hamiltonian Monte Carlo (HMC) technique, SGHMC replaces the computationally expensive exact gradient calculation in HMC with an economic estimate. Further, as an iterative sampling method, SGHMC follows a Markov process to stochastically transform a seed into a sample from an intractable distribution. In our case, such a technique can instead be applied as an augmentation method by propagating toward the distribution of negative pairs with increased dissimilarity in the embedding space. Throughout the rest of this article, we establish SGHMC as suitable augmentation strategy following approximate invariance in theory. Morcover, in practice, SGHMC allows tuning the number of iterations to control the shift of the generated view from the seed sample. These, in turn, ensure that through SGHMC, the distances between a negative pair increase
2

---

Published in Transactions on Machine Learning Research (11/2024)
InfoNCE
InfONCE-NRCC (Ours)
BYOL
BYOL+NRCC (Ours)
Figure 1: UMAP projections of ImageNet-10 embeddings for InfoNCE, BYOL, InfoNCE+NRCC, and BYOL+NRCC, The cluster collapse and fragmentation respectively shown using solid red and dotted black borders. The proposed NRCC brings visible improvement in retaining cluster structure in the UMAP space compared to the corresponding non-regularized baseline CL methods. Given that UMAP preserves local neighborhoods, we conclude that NRCC also obtains a cluster-friendly embedding space.
average while the views reside in the local neighborhood. The second challenge is solved using additional attraction and repulsion through the existing positive pairs, and SGHMC-curated negative pairs that are traded off against each other. Such a stralegy can be added directly with InfoNCE as the paradigm already allows negative pairs. However, minimal architectural and algorithmic modifications are required for positive pair-only BOYL to cater to the effective usage of negative pairs.
The novelty and usefulness lie in the proficiency of NRCC al aiding CL in finding cluster-friendly embeddings. Further, such embeddings can be safely projected lower dimensions by neighborhood preserving techniques (Van der Maaten Hinton 2008 Mclnnes et. al. 2018) to allow mode-seeking clustering algorithms without requiring k. As a motivating example in Figure of embeddings learned by InfoNCE, BYOL, and their NRCC regularized counterparts on the ImageNet-10 Russakovsky et al. 2015) dataset. From Figure it is evident that InfoNCE suffers from both cluster collapse and fragmentation. BYOL, though improved, is still susceptible to collapse. On the other hand, NRCC, both the cases, exhibits considerable improvement in preserving cluster structures. Highlights from our contributions are as follows:
present the UMAP (Melnnes et al. 2018) projections
We propose a new regularization framework, NRCC, that uses hard negatives and can be seamlessly coupled with the cxisting CL paradigms, such as InfoNCE and BYOL, to learn a clustering-frennde embedding space.
We provide a theoretical characterization of negative pair construction that facilitates cluster- friindry representation learning. Further, we show that SGHMC sampling can be repurposed as an augmentation strategy, and the generated negative pairs satisfy approximate invariance (Chen al. 2020a] i.e., they are hard pairs offering intensified repulsion with improved local crowding.
NRCC relieves the user from knowing the number of clusters beforehand. Given that NRCC-enabled CL embedding preserves cluster structure, it allows for non- nou-parametric mode-seeking clustering algorithms to be applied effectively, following local mighborhhod-prrerrvin dimensionality reduction.
In Section 2 we review the existing works on DC and CT. Section 3 explains the NRCC regularizer, justifies both theoretically and empirically the choice of repurposing SGHMC sampling as the preferred augmentation method in NRCC, and demonstrates the promise NRCC offers when coupled with InfoNCE and BYOL. Empirical evaluation of the efficacy of NRCC with GridShift (Kumar el al. 2022 against the state-of-the-arts in Section 4 shows a 4.7% improvement in clustering accuracy by BYOL+NROC+UMM Crisht on an average over eight datasets. Finally, we make concluding remarks in 5
2 Related Works
2.1
Contrastive Learning
CL methods ( Wu al. 2018 maintain their primary focus on finding a generalized representation that can be tailored for multitude of downstream learning tasks such classification, clustering, etc. with minimal
3

---

Published in Transactions on Machine Learning Research (11/2024)
additional effort. The milestone set by Chen et al. (2020b) opened major paradigm of CL methods utilizing both positive and negative pairs. However, it was soon observed that in the case of CL. improved performance and reduced computational cost are both closely associated with finding good negative pairs (Kalantidis et al.) 2020j. Hence, several research directions, such as using a modified objective to prohibit similar samples from Torming negative pairs (Chuang et al. 2020), theoretical and empirical characterization of generated views to find their usefulness (Tian et a 2020). , employing adversarial feedback (Hu et al. 2021), etc. were explored over the years to alleviate the problem. On another front, the work by [H al. (2020) investigated the applicability of a two-network strategy where the primary feature extractor is updated by the moving average of the parameters of an auxiliary model that goes through a traditional gradient descent. This direction later evolved into a new CL paradigm (Grill et al. 2020 Chen & He 2021] that extended the idea of two networks through a siamese model, refined the update of the feature mapping branch with auxiliary branch parameters, and wholly discarded the need for negative pairs through judiciously controlling gradient propagation. In a less explored avenuc, energy-base baaedy learning also found its usage in CL that brought back the negative pairs but altemtted to limit the batch size without affecting performance (Kim & Ye 2022). Recently, slight. deviation from generalized representation to designing a more task and data specific tailored class of CL methods gathered interest such as for computer vision problems like dehazing (Wang et al. 2024], clustering
Deng et al. 2023 Yin et al.) 2023), classification (Zeng & Xie 2021). enzyme function prediction (Yu et al. 2023 time series [iu & Chen 2024), ,,, graph learning (You et al. 2020 Zhang et al.) 2023 Bo et al. 2024))
etc. In a similar direction, we progress towards a clustering-specife CL method that effectively utilizes hard negatives to find an embedding that conserves the natural clusters present in the data.
2.2 Deep Clustering
Clustering performance is highly dependent on the representation of the data, i.c., how well the natural clusters are retained and expressed in the feature space. The rich embedding learned by deep neural networks found its natural usage even in the very early days (Yang et al., 2016 2017). A primary direction for DC methods employs simultaneous optimization of multiple objectives for finding an embedding space that can
be easily clustered by a specific clustering method (Guo et al. 2017 Caron et al. 2018: Asano et al. 2019
Li et. al.) 2020b 2022). For example, Yang et al. (2016) proposed a deep representation learning suitable for hierarchical clustering while Caron et al. (2018) employed classification to iteratively refine weak cluster assignments. With the advent of CL, the DC rescarch community was quick to identify the similarities between the two objectives and came up with several approaches that f'use the two for commendable clustering performances. For example, CL was used by Van Gansbeke et al. (2020p to generate reliable pseudo labels while many others (Li et al. 2021 Shen et al. 2021 Li et al. 2022 Sadeghi et al.
2022) proposed end-to-end frameworks where the CL objective was repurposed for clustering. In an allied direction, (Zhong et al. 2021
Zheng et al. 2021 graphs were employed to identify neighboring pseudo-positive samples. However, these efforts were plagued by the limitations of CL, i.e., irregularities induced by improper negative pairs, class collision (Huang et al. 2022). etc. A more recent work by Huang et al.] (2022) managed to overcome the class collision problem but instead imposed a reliance on the knowledge of the number of clusters to be known in advance, limiting its practical applicability. Instead, with NRCC, we return to the initial DC strategy of employing CL to find a cluster-friendly representation that can be conserved after neighborhoood-presevee dimensionality reduction. Consequently, NRCC offers flexibility for the choice of clustering methods, alleviates the impact of the curse of dimensionality, and imposes no compulsion on the prior knowledge of k.
2.3 Mode Seeking Clustering Methods
Center-based clustering methods, like k-means, require the knowledge of the number of clusters (k) in advance. Moreover, such methods demand structural and distributional constraints on the clusters. These limitations can be avoided by density-baser based modc-socking clustering strategics such as McanShift and its variants (Cheng) 1995 Jang & Jiang 2021 Kumar et al. 2022). The MeanShifi family of methods is non-parametric that does not demand prior knowledge of k. Depending solely on density estimates, these methods can automatically reveal natural clusters by identifying the modes of arbitrary distributions. However, McanShift (Cheng 1995) is not scalable due to its quadratic computational complexity with the number of samples (O(N")). This crucial shortcoming of MeanShift was addressed by MeanShift+++ (Jang & Jiang] 2021 which employs
4

---

Published in Transactions on Machine Learning Research (11/2024)
grid structure to reduce the complexity to O(N). Recently, GridShift (Kumar et al. 2022 approximated cluster centers to further achieve 400 speed up over MeanShift + Moreover, GridShift. demonstrated (both theoretically and empirically) that it maintains a better (or at least at per) clustering performance with McanShift++. Based on these observations, we champion GridShift for clustcring in our work. However, MeanShift+++ and GridShift both fall prey to the curse of dimensionality. In both cases, the computational complexity incrcases exponentially with the dimension of the feature space. To fully exploit the benefits of GridShift, we ensure two properties of the CL embedding space. First, the learned embedding space correctly preserves the cluster structure. Second, the clusters in the CL embedding space can be projected on lower-dimensional space (preferably with minimal loss of information) through neighborhood-preesere
effectively used
methods (aan der Maaten & Hinton 2008 Mclnnes et al. 2018
3 Proposed Method
3.1 Preliminaries
We begin with a set of N E N4 samples {x, hiz that reside in a high-dimensional space X C Rd, where [Z] = N. The high ambient dimension of X prevents distance-based clustering algorithuns like k-means from performing at their true potential. Moreover, the curse of dimensionality makes the application of kernel density estimators infeasible, thereby restricting the usage of non-parametric mode-seeking clustering algorithms on such data. Hence, it becomes reasonable to project the data onto low dimensional embedding space H C Idd where the natural clusters are appropriately conserved or enhanced even further, d < d. As previously elaborated in Section 2 in the case of DC, such cluster-friendly representation space is found through a deep neural network. For example, an encoder f : X + H can be used to map a sample x; from its native space to the corresponding latent representation h, = f(x: E H. Here, Q represents the set of tunable parameters that characterize f.
With this setup, we move on to the CL-specific case. We start with a set of possible augmentations T from which we randomly select two data transformations, namely T and T To facilitate mini-batch learning, from (xi we randomly sample set A containing data points without replacement. Now, given this mini-batch N,, a set of n positive pairs is created as P_ = {(f ((xx) x/) where x; = (x:) and =xx) are two different views of the same data instance. As such, they share a strong semantic similarity. This aids the network in identifying the natural groups present in the data through the induced attraction while enhancing the similarity between a positive pair in the embodding space. However, naively optimizing only the positive pair-driven objective leads to degenerate distribution supported on single point in the embedding space. Traditionally, this is handled using an auxiliary contrastive loss that offers repulsive force by maximizing the dissimilarity between negative pairs. Formally we denote the set of negative pairs for a batch as P_ = P u ph, where P = {(xx, x))
In other words, for the mini-batch N, pa denotes the collection of both the views xf and x² for each of the samples x, in the set N \ ] = (xI.... X,,1, X++1, ... xn) paired with xf, for all i 1,2, .,,.,,............... ,n while ph is constructed analogously. Consequently, the cardinality of P turns out to be 4n(n 1).
and ph = {(x,, x5) -igg)))
In the case of InfoNCE (Chen et al. 2020b), the CL objective is not directly optimized in the latent space learned by f but in fo g, where g H. + Z. Specifically, is realized through a dense neural network that yields z, = g(h.). Assuming that 2, E 2 is normalized, and 7 is a temperature hyperparameter regulating the degrec of affinity between samples, the InfoNCE loss I' can be defined as follows:
1' = Σ
(ia + (11b)) where
(1)
(2,,2))
n
(2g
2tt)
1/
+ log
Σ
Σ exp
and
(2)
T
T
kE(a,b) j=l.jri
(2 z!
n
(
+ log
exp
(3)
T
ka(bbb) j=1.jii
5

---

Published in Transactions on Machine Learning Research (11/2024)
The InfoNCE loss has two major disadvantages. First, there is no provision for considering the similarity between two distinct samples and x, in a batch N during the construction of P.. Tf the samples x, and Xj are from the same cluster, then treating them as negative pair may become detrimental for convergence towards a clustering- chisering-friennlly friendly space. However, in a self-supervised setting, in the absence of labels, there is no way to be confident about the similarity between x; and x, in advance. Second, optimizing the loss demands high negative pair to positive pair ratio, 2(n - 1) to be exact (Chen et al.) 2020b Kim & Yo 2022). However, with a high number of negative pairs comes the issue of class collision (Arora et al. 2019 that compromises the adaptability of the CL-learned feature space to clustering. A potential remedy for both of these issues comes in the form of a negative pair generation strategy that maintains a considerable distance among the negative pairs while closely adhering to the original data distribution.
An alternative way to address the issues of InfoNCE can be found through BYOL (Grill et al. 2020p. Such a CL paradigm discards the negative pairs and the associated auxiliary loss altogether while escaping degeneracy. Specifically, BYOL uses a siamcso-inspired architecture where the two parallel networks are called online and larget, respectively. However, compared to traditional siamese architect ure, the networks in BYOL have three major differences. First, target and online branches do not share common set of weights. Second, the target branch has an additional head q to map 2, E 2 to prediction vector. Third, only the online branch is updated using gradient descent on the loss function, while the weights of the target branch are a moving average of the online parameters. These three key modifications harmoniously ensure that both branches are not affected by the same gradient. Thus, the scenario resembles an alternating optimization that helps to avoid the degenerate collapsing. Specifically, the BYOL objective function 1 can be defined as follows:
Pa + +1 1,61 where lha ||((2]) 21112 and 1. [|a(2))) 2 11.
i=1
Note that the outputs and mappings of the target network are distinguished from their online counterparts using a bar (7). While BYOL directly improves over InfoNCE in general, its clustering polential may not. have been fully exploited, as in the absence of counteracting repulsive forces, the embedding space may still coalesce multiple clusters (Chen & He 2021).
3.2 Regularizing CL Loss for Clustering
Following the discussion in Section 3.1 it is evident that we need to focus on three primary fronts to achieve good clustering. First, maintain trade-off between repulsive and attractive forces such that the clusters neither collapse nor fragment but remain uniformly distributed and well separated in the embedding space. Second, if similar samples form negative pair, the repulsion force should be adjusted adaptively to protect the learning from being misguided. Third, negative pairs are the direct source of repulsion in a CL framework. Hence, ignoring them completely, as in BYOL, may not be helpful for clustering as that may lead to undesirable cluster collapso. We now proceed to detail the proposed NRCC regularizer that can be effortlessly coupled with InfoNCE or BYOL.
Σ i=1
T + +1, 1,b:
(5)
Σド= exp (z 2}))
where i,a = 14,0 + )
log
and T7, is defined analogously,
Lラ= exp (2,, 25))
18
iB
1B
(6)
2,a
n E/=1 exp( x}1) Σド=1 exp( (zz 25)) and 76 is defined similarly.
such that 1B 1,0
= 1. + > log
In the above formulation, T' and T are, respectively, the regularized InfoNCE and BYOL losses. This notational convention also extends to the sample-specifie objectives, i.e., H is the regularized form of 1/
6

---

Published in Transactions on Machine Learning Research (11/2024)
along with the other similar terms. We introduce a new Lagrangian hyper-parannn paper-parameter À that controls the relative weight of the regularizer. Moreover, 2 or in case of BYOL. ø" corresponds to an arbitrary augmented view of x E W. Additionally, for InfoNCE following the canonical objective, the temperature 7 can be included to scale the inner products for finer control of the attractive and repulsive forces.
To better understand how the proposed NRCC regularizer alters the attraction and repulsion in an effort achieve better clustering, we take a closer look at these forces under the light of the gradients. For simplicity without loss of generality, only focus on the loss gradients with respect to zi as the same for other cases follow similar derivation. In the case of the traditional InfoNCE loss,
au
1
01
3,0
01d66
Σ
On j.ke
dzg
2n
dzy
Ozp
1
2z
Σ
texp( (z 23)1 /T/22 + fexp((zf. 12))
(7)
2n
T
Eke(a.b) "in expl (z) 2/)//)/)/))))) (ke(a.)) Di=1 exp (zf 23)/T) 191
Let us now rephrase the expression in terms of the two forces. If we denote and at the negative gradient or attraction and positive gradient or repulsion then,
as respectively u.;
3x
0y' oy_
04 dz."
such that,
dzy dz
01d z!
(8)
Zz 11T
014
Σ Σ
(x),
dz
2n &E
where Ú3
Σ
Σ expl 21/7)
Σ
exp( 22)/7)
(a.b)
1=1,
(a, 1=1,
al dz
or
If we now denote the modified forces for the NRCC regularized variant of InfoNCE
and
then
Jz
07! 0
(z) 2T//T)
(10)
dzg
dz
1=1 expl (z,, 2///T)
zf exp( (2) 2g)) TT/
V2222 exp (xp((ppp 2/)/T)/T
(11)
dzy
Oz
> cxp( (z 2f)/T)
j=1,iıi
where w/= =
Σ exp((2) 211/T)
Exxp(22)22)117 21)/7)
exp((z))
There are three key takeaways from the above discussion:
Comparing with 10), it becomes evident that the proposed NRCC further strengthens the attraction from its non-regularized counterpart.
2. If we consider and 11 then notice that there are two terms of opposite signs that are newly introduced by the regularizer. The first term on the left acts as a countermeasure against negative pairs made semantically similar samples. In such cases, this term is likely to be larger, thereby diminishing the effective repulsion to limit the impact of an improper negative pair on the learning.
3. The second term in the right introduced by the regularizer directly enhances the repulsive force.
7

---

Published in Transactions on Machine Learning Research (11/2024)
We can now move on to BYOL and undertake similar study. However, unlkke InfoNCE, in the case of BYOL, there are additional implementation-speeiii heuristics like the predictor block and stop gradient. These heuristics are important factors to consider when finding an effective position for plugging in the proposed NRCC regularizer in the BYOL architecture. For the moment, we consider a theorctically simpler setting where the additional predictor block and stop-gradient are not used while NRCC regularization is applicd on 2 similar to InfoNCE. This does not sacrifice generality as such an assumption essentially reduces z's to simply z's. Thus, an additional multiplier oz gets removed from the derivation of the canonical case making it easier to interpret. We revisit the case of coupling BYOL with NRCC, without imposing any additional simplification, in a later Section 3.6.1 Meanwhile, borrowing the notational style from InfoNCE, the analogous gradients for the canonical BYOL (although simplified) take the following forms:
de
a z! and = O.
0 dzy
(12)
θzş
Hence, the negative gradient उ.. optional However, in the absence of negative pairs, the repulsion force contributed by the positive gradient becomns zero. Now, if we apply NRCC, then the forces respectively modify to, o2
induced attraction for BYOL is similar to that of InfoNCE except for the
ar
01
z
1
exp( 22,//)
X
(13)
dz
n
2
1= exp( (z (29,2/)/7) /
0
n
2 exp (z) 22)/TT T))/
E exp (z) 2/)/T) - (x)
(14)
dz
2n
j=1
Upon closer inspection, we can see that (10) resembles the same for 13 in the sense that both enhance attraction similarly. However, the lesser additive constant, i..., 2 in BYOL, may slightly reduce the attractive force compared to InfoNCE. This is desirable given that BYOL, by design, driven by a strong negative gradient; thus, to gain the benefit, the NRCC-induced attraction should be added in regulated manner. If we move our focus to the positive gradient-driven repulsion, there also BYOL shares similarity with InfoNCE, as evident from [14] and (II.. Thus, our previous discussion in the context of regularized InfoNCE still holds in the case of BYOL, In other words, the NRCC not only introduces repulsion in BYOL but also adaptively restricts the ill effects of improper negative pairs.
Till this point, we have observed the impact of NRCC in improving the clustering potential of CL methods. However, NRCC can only work at its true potential if z can produce hard negative pair that further improves the repulsion. To this end, we take a deeper look at the standard augmentation techniques and discuss the applicability of repurposing SGHMC in the context.
3.3 Characterizing Data Augmentation in CL
The data samples can be deemed as instances of i.i.d. observations X1, X2, .x P. on the sample space X. Some augmentation methods like magnification and rotation are simply affine transformations. Whereas others, such as cropping or color jitter, may not be so. Either way, semantic features from the original images mostly stay intact post-transformation. We can represent such augmentations as a group of transforms T, acting on the sample space. As such, there exists a map & : T X + X, such that it associative @(TT" x)= (TT, x)) = '(T, for T,T E T and given the identity e T, &(c,x) = cx = x [Chen al. 2020a) ). In our study, we do not assume that exact invariance holds, i.e. given X P, it may happen that TX X. Under such setup, the positive pair (xf xf) obtained by applying T and Td independently on X; may not be identically distributed. This stems from the fact that the augmentation techniques vary vastly between themselves. We emphasize that the goal of the augmentation is not achieving distribut identity. Rather, we want to simulate perturbed offshoots of the given observation that share semantic information, ie., to attain approximate invariance (Chen et al. 2020a).
Let us now focus on the crucial task of learning embed data from X' onto a lwver-dimeniionll space H. Specifically, we employ ResNett-ype architecture (He et al. 2016 that tends to preserve the local geometry
8

---

Published in Transactions on Machine Learning Research (11/2024)
of the imput space X' due to their capability of approximating Lipschitz maps with high precision (Oono & Suauki, 2019), also extendable to the Bi-L.ipschitz criterion. We denote a convolution operation, involving weights to and bias b, by the notation Conv RdxC - Rdxx'' where a is the activation function, applied componentwise. The input and resultant channel sizes are given as C,C' N+. Similarly, the notation used for dense layer is F-) == oWWve-(-) + b), function mapping IRdxc' to Rd Also, let P: I + Raxc be a padding operation, responsible for aligning channels. Thus, in an encoder network with M residual blocks and each block having a depth L, the desired embedding becomes:
Un Rd + Rdd, where v = [FC o (Convers war.bsr + +ld ................................ (Conv w1.b1 Id) o P].
(15)
M.L io Also, = (fw: (fwe) (b] J.j== W,b) denotes the parameter space and Id are the additive identities in subsequent spaces. With this setup, let us now consider those networks that satisfy maxi, ||l] 1l.e V 11 I B and | V ||lb I B', given constants B Bf > 0. These are our preferred candidates that induce the map f. By definition, h (2((*), k E {a, b}. Observe that, h also hail from non-identical distributions. We first show that given a desirable angmentation technique, the realized discrepancy between the embedded observation and its augmented counterpart remains bounded in expectation. This points towards a statistical characterization of augmentations that regulate the fluctuations in estimates. It also testifies that simulated positive pairs indeed crowd the local neighborhood.
where Conv d, Conv .6 .... Conv .o.
fij==1
Firstly, let us assume that the augmentation T follows a probability distribution Q such that the output of the embedding (X.T) + (TX) belongs to (I x Q), i.c. it has finite second moment. Let P, be the plug-in estimator based on the realizations of X2, ... , .N given by PN == t Eiez ōx, Also let Yı, Y2, ... YN be a sample drawn from the Q-augmented distribution, for example, i..dd copies of x". Based on the same, construct Pr.N, the plug-in estimator. Given two non-negative real sequences fan IneN and {bxlneN, we denote the suppression of the universal constant D 0, such that lim sup by S D, by On &bn-
an
Proposition 3.1. Given that is supported on a compuct domain in IR,, d 2, there erists a determénistic error y, obtainable using linear program, such that for >> 0
|xx (PPT,N) (2))l, S V
where M the number of residual blocks present in the embedding network and the activations a is taken as ReLU. The transformation (P) should be understood as the push-forward of IP by V.
Proof. Please find the proof in Appendix B.1
Proposition 3.1 indicates that, on average, observations obtained through a suitable augmentation technique would not deviate significantly from the original inputs in the embedding space. The worst-case discrepancy is bounded by the solution to a linear program (Sriperunbudur et al., 2012), making it finite, easily determined, and controllable. As result, the likelihood of simulated negative pairs lying drastically farther away and causing irregular repulsion, leading to cluster disintegration, is reduced. We now focus on SGHMC, intending to repurpose to fit the desirability standards for CL-based clustering.
3.4 Desirability of SGHMC as an Augmentation Technique in CL-based Clustering
In an effort to generate xf, a clustering-rreenlly negative pairs tailor-made for NRCC, we start by listing down two desirable criteria. First, compared to x and x,, the new augmented sample x should lic significantly further from x; in the feature space. Second, x should resembble x; in terms of their generating law supported on the embedding space. We initialize with a probability distribution over pairwise distances between images. Our methodology for defining the same focuses on producing replicate that exhibits embedding located within the local neighborhood of embedded x,. It is given as follows:
view of a sample E NN, that is more useful for constructing
(x(X) = [1 + sim (X((()))
(16)
9

---

Published in Transactions on Machine Learning Research (11/2024)
where sim(z, 2 signifies a measure of similarity between and z', e.g. the inner product. Morcover, Ω can be deemed an instance of random variable characterizing the distribution Pn. In our discussion (see Section 3.3), S2 emerges as the set of parameters representing the encoder f. Now, the usage of a scalable iterative sampling method like SGHMC becomes necessary since sampling directly from Pn becomes impracticable. We start with randomly selected initial seed 8o = x from {x (x Given the number of iterations
as Ç, we also sample {r,}Ş_o äid (O, This in turn serves the initialization Po ro. Over the iterations 0, Si alternatively update p and following the formulae:
Update p: Ps+1
51)p, 62VPP(()))x, + dzrj+1,
(17)
Update s S4+1 8j 82P5.
(18)
where 81, 62, d3 are control parameters of SGHMC. At completion of iterations, we obtain s* Sç and return the same as the augmented view xf. Note that our expression for the SGHMC algorithm is roparametrized discrelization of the underdamped Langevin dynamics. The SGHMC method is further detailed as the following Algorithm for ease of implementation.
Algorithm 1: Augmented view generation with SGHMC.
Input: sample x; E (x:] set of 2(n 2) augmented samples {x+) U [x]] control parameters 81, d2, and 63, number of maximum iterations Ç. Output: The generated sample returned as the augmented view x².
Take a random seed So = x where x {xf] Mi=1 {x]]==
Further, sample a set (15}5-0 where ry sid NO0,, Ia) for all j E (0,1, ,S} and set Po ro-
for j 10, ; do
Update p: Pj+1 = (1 - 51)P, 82 (X) 83rz41
Update S: Sj+1 = Sj - 62p,
end
Return s* = sc as the augmented sample xf.
The second criterion can be satisfied by showing that SGHMC acts as an approximately invariant augmenta technique and hence is desirable for CL-based clustering methods. Let us start by defining 12 = +. Ei= (x,), based on observations (x1, X2, .x,} in batch N.. To establish statistical characterization, first impose mild regularity conditions, standard in related literature (Raginsky et al. 2017). the underlying distribution.
Assumption 3.2 (Regularity of Pa). Fn is bounded and contimuously differentiable, having bounded gradients such that for any given 22 there exists A satisfying
VPa(x) - vPrly)]l < A||x - yll. Vx,y X.
Assumption 3.3 (Dissipativity of gradients). Given x E X, there exists m > and 2 such that
(x, VPn(x)) 2 ml |||||/-
Assumption 3.4 (Variance control of gradient estimates). Given realizations of replicates i Sn the function G : G(x Un) Σ Viaa, (x) (also bounded and Lipschitz continuous in due to Assumption
3.2 satisfics
( Un) v Σ Pa. (x)ll $1
where Un (Rt,,". I1,)) given that (J,,.., In is random permutation of (1,... nn.. Assumption 3.5. The law corresponding to the initial state (So, Po), say μ4o satisfies
f Cle p) < 0o,
where L(.,)) is the Lyapunov function.
10

---

Published in Transactions on Machine Learning Research (11/2024)
For detailed commentary the significance of the assumptions, refer to Section of Raginsky et al. (2017). Note that, under assumptions 3.2 3.5 the Markov process (s,, Pj)520 turns out to be ergodic following a unique stationary joint distribution (ds, dp) (Gao et al. 2022). This is what imposes the near-perfect invariance to augmented outputs xf using SGHMC. To avoid complications due to notations, we denote the s-marginal of Ug as (o((s). Below, we provide an exact statement that mathematically formalizes this intuition.
Theorem 3.6 (Raginsky et al. 2017)). Given that Assumptions 3.2 3.5 are satisfied, there erists positive conslants {ci)== such that for 8 > O we have the 2- Wasserslein distance (D;ss); mn) S E, whenever c162 + 0 S E and j 2 O ( log( Also, the associaled eacess risk turns out to be S log (Ah + +1) + 0 ) where 8' do2 Here, D(-) denotes the distribution of an underlying random variable. 4
Proof. Please find the proof in Appendix B.2
From Theorem 3.6 we can conclude that after a certain point in time, the augmented views itcratively obtained using SGHMC possess distribution that lies arbitrarily close to the stationary benchmark, dependent on the target. The approximate global minimizer thus obtained, i.e. s", also holds the same property. Since the I- Wasserstein distance Wi is upper bounded by W2, the result obtained in Theorem 3.6 can also be extended to the W1 metric. We have previously observed the efficacy of general augmentation procedures under W1 (see Appendix B This result gives us immediate guarantee that SGHMC achieves approximate invariance in the augmented samples.
3.5 Revisiting Negative Pairs in NRCC
Based on the discussion in Section 3.2 is evident that the amount of repulsion induced by NRCC can be further amplified directly by placing the negative pairs farther away in the embedding space. However, during implementation, z's are always normalized as z/ z/|zz|2 in turn following projected (or offset) distribution supported on hyper-sphere (Mardia & Jupp (2009), Chapter 9). As such, the points being restricted on the surface, pushing two clusters wide apart brings one or both of them closer to some other cluster. This situation can only be avoided if the augmented views are aware the density landscape and crowd the local neighborhood rather than spreading out uniformly. As per our findings Sections 3.3 3.4 we can argue that SGHMC satisfies this owing to its approximate invariance. To further empirically validate this claim, we undertake comparative study covering both the qualitative and the quantitative aspects. The qualitative study in Figure 2 demonstrates that the outputs generated by a single step of SGHMC are indeed visually distinct from their original counterparts, especially in contrast to the four popular data augmentations. This is quantitatively supported by Table D where, given 1000 ImageNet-10 samples, the mean Euclidean distance between a negative pair in the proposed BYOL+NRCC embedding space is higher for a single update of SGHMC, Thus, we can conclude that SGHMC, when used in NRCC, can produce negative pairs that simultancously generate higher repulsion and preserve the cluster structure in the embedding space.
Table 1: Mean and Standard Deviation of Euclidean distance in the proposed NYOC+NRCC embedding space between 1000 ImageNet-10 samples and their corresponding transformed views obtained by five augmentations namely, Random color jitters, Random gray-scale, Random horizontal rotation, Random crop with resizing, and one step of SGHMC (ours).
Data Transfommaiinn
Distance between negative pair in
embedding space (tandarS Deviation)
Random color jitters
0. 0.0038=000000 0.0038=1..00
Random gray-scale
Random horizontal rotation
0. 0.0038=0.00 0019
Random crop with resizing
0.00111000000
SGHMC (Ours)
0.0112 = 0.0060
11

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 2: Augmentation oulcomes corresponding to image inputs (from left): Original, Random color jitters, Random gray-scale, Random horizontal rotation, Random crop with resizing, and single step of SGHMC (ours). Compared to the other four augmentation techniques, SGHMC produces visibly distinct outputs that translate to a larger distance in the embedding space for negative pair.
z
x
g
g
g
h
h
b
h
h
h
f
f
f
f
x
x
SGHMO(-)
T-(-)
(²)))
x2
x(
T*()) (-)
SCHMC(()
(+)-
SCHMC(.))
x1
T')-)
T-)())
x2
x
go
go f
Figure 3: The schematic for the coupling of the proposed NRCC with the existing contrastive learning paradigms. Specifically, in this graphical workflow we focus InfoNCE. However, the workflow can also be extended to BYOL with some modifications current batch namely Xx,,X,, ... xx,, h let us consider three transformations (')), and the SGHMC(.). The network is composed as go f. where f is the encoder and the additional projection. We pass each of the three transformations of x,, respectively x;, x,, and x;, for = 1,2, n, through f and g to obtain the corresponding h. and 2,)s. With this setup, considering exemplary case of x1, the positive pair driven component in traditional InfoNCE loss is shown by the Blue line between z and z/. The negative-pairs in the canonical InfoNCE are defined by Red lines between zî and (z),)} (2]1)]. Finally, the additional NRCC rogularizer component takes care of everything clse as shown by the Green line. Specifically, it considers the pairs of of and {x J (2
highlighted in Figure 5 Given set of samples in the
3.6
Contrastive Clustering with NRCC Framework
3.6.1
Coupling NRCC with InfoNCE and BYOL
In the case of InfoNCE+NRCC, the implementation fairly straightforward, given that the CL paradigm already employs negative pairs. We only pass an additional negative pair formed by SGHMC to find the
regularized loss in 5 and then follow Chen et al. (2020b) for optimizing InfoNCE+ We demonstrate
12

---

Published in Transactions on Machine Learning Research (11/2024)
Accuracy of ENN Classifier
Global Standard Deviation
Clustering Accuracy
0.75
0.0
0.23
1:53
0.2
0.00
0:00
0.
1900
7500
10000
0.58
1.00
0.55
0.25
6.02
6:00
1920
7906
(003)
79000
1900
1963)
25000
50000
1500
TOOND
125030
0.55
(c)
0.50
3.31
0.25
3.02
0.25
0.30
3.00
0.00
2000
12000 (0000 -2403
Figure 4: Plots of k-Nearest Neighbor (ANN) classification accuracy, Global Standard Deviation, and Clustering Accuracy against training itcrations on ImageNct-10,, for BYOL variants terms of different couplings of negative pairs Specifically, the SGHMC augmented view xÿ is passed through (a) an online network with prediction, (b) target network with prediction, and (c) typical target network without prediction. The third case (c) provides the best and most stable performance among the three alternatives.
this schematically in the following Figure However, BYOL+NRCC brings additional challenges, given that negative pair is involved in the original non-regularized formulation. If we consider the case of calculating i, then given Xj * x; to pass its SGHMC augmented view xf one needs to consider two possible scenarios. First, xf can be passed through either the online or the target network path. Second, if x5 follows the target branch, then it can either employ or discard an additional predictor at the end. Thus, to find a suitable network structure, we employ an ablation study following Chen He (202T). We mainly consider three configurations. First, pass x to the online network. Second, feed x to the target network while using an additional prediction block. Third, pass xÿ to the target network without prediction block. For each of the three cases, over the training iteration on the ImageNet-10 dataset, track the k-Nearest Neighbor (&NN) accuracy, global standard deviation, and clustering accuracy using GridShift (Kumar et al. 2022 following a UMAP dimensionality reduction, setting the number of neighbors k as 5. We illustrate the findings in Figure from which we can make three observations. First,, xÿ passes through the online network, then the training is unstable as the momentum- based rumning average in the target network BYOL offers the desired stability. Second, x is passed through the target with a prediction block, then the global standard deviation slowly drops, indicating a collapse. Ideally, for 256-variate projected Normal distribution, the global standard deviation should plateau at 1/v256 s 0.06 (Chen & He 2021). This may be due to the fact that additional prediction in the target brings discrepancy in the learning, such that it is forcing cluster separability in the space spanned by {g(z): E Z} but not in the intended embedding space H. Third, if xf passes through a typical target network armed with stop gradient without additional prediction block, then all three indicators maintain stability and demonstrate improvement over training.
Now that we have the necessary clarity of effectively coupling BYOL and NRCC, let us follow up on the discussion in Section 3.2 If we consider the stop-gradient and predictor block in BYOL, then the actual gradients modify from their simplified versions in
12 to the following:
01
2
Qzg
0L
and
0.
(19)
2n
Ozg
Oz.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 2:
The proposed InRONCE+NRCC.
Input: Inpuu dataset (x1, 1x1,X2,"'" " xN); batch size n, learning rate 8, weight. of NRCC regularizer in the total loss X, temperature parameter т; number of maximum iterations Ç, and pool of transformations T.
Output: The learned encoder f.
Randomly initialize the parameters for and g. while Marimam iteration is not reached do Sample a random batch N of n instances from the dataset.
for all i {1.2..... n do
Sample transformations T= E T and create x = (x,) and x = (xx.). end
Create positive pairs as (xfxx") for all i (1, ,n].
Create negative pairs as (xf, xf), for all i,j E (1,2, , nn,, x E {xj,x() and + + Find the SGHMC transformations xf for all i (1,2, ,n] using Algorithm Create NRCC negative pairs (x1,x) x3) where i,j E (1,2, .n} and + j.
Calculate ! using (5).
11
Update f and g using the gradients of 1t.
12
13 end
14 Discard and return f as the feature encoder.
NRCC achieves its true potential when it is applied on the Z-space of the target network in the BYOL architecture. In other words, while regularizing, we only need to account for the stop gradient and ignore the prediction block. This only impacts the negative gradient-induce repulsion remains the same
attraction while the positive gradient-basee [1]. After revising (13). the regularized negative gradient takes the form:
z! 1 17%
(2,, 221/// (xp((2e (Zp(((z)))) (2222)//)))
+ X
(20)
dzp
n 2 dz
The simplified formulations 12 and 13). when compared with their respective counterparts in 19 and 20
leads to two key observations. First, the canonical and regularized positive gradients remain exactly the same after considering the BYOL-specific heuristics. Second, as already mentioned in Section 3.2 the negative gradient of the canonical BYOL now has a multiplicative factor Tr that does not impact the additional force exerted by the regularizer. Thus, the motivation in favor of NRCČ that we had mentioned earlier Section heuristics. We present the schematic workflow of BYOL+NRCC in Figure 5 We also describe NRCC and NYOL+ respectively, in com The code base is available at https://githin www.:aw....oobiiii...i..a.c.ccce http///aabbiieeea///////CC/
02
3.2 remains unaltered even if we account for the implementattion-speeeeee
the following Algorithms and
3.6.2 Clustering in the NRCC Regularized CL Embedding Space
After optimizing the InfONCE-NRCC or BYOL+NRCC loss, we can safely discard everything except the fa i.e., the original encoder. In the case of BYOL+NRCC, only retain the from the target branch. Now, at the end of training, then we can apply parametric technique like k-mcans to the learned f space. Otherwise, can project the embeddings to a lower-dimensional space through methods like UMAP or I-SNE (Van der Maaten & Hinton 2008) that retain the local neighborhoods. Now, NRCC, along with SGHMC-based augmentation, manages to properly identify and separate the clusters the embodding space. Thus, the revealed cluster structures are not distorted after additional of mode-seeking clustering methods like GridShift that do not require knowledge of the number of clusters.
get the cluster-friendly embedding. If we know the number of clusters beforehand,
betboorhood-prseevviie dimension reduction. This further enables the use
14

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 3: The proposed BYOL+NRCC.
Input: Inppu dataset (x1, X2, xN); batch size n, learning rate 8, weight. of NRCC regularizer in the total loss X, temperature parameter т; number of maximum iterations Ç, and pool of transformations T.
Output: The learned encoder f.
Randomly initialize the parameters for online network f, 9, and q and target network f and g- while Marimum iteration is not reached do
Sample random batch N of n. instances from the dataset.
for all E {1,2. nn} do
Sample transformations T E T and create x; = (x;; and x = (x,,)
end
Create positive pairs as xx,xx"), for all i 21,2,... ,n].
Find SGHMC transformations xf for all i E f1,2, .22....,) by Algorithm| Create NRCC negative pairs as (x², x5) where i, E {1, (1,2,-.. , and 7 j.
Calculate 1 using (6).
10
Update online network J, 9, and using the gradients of _
12
Update target network f and g by momentum-bason running average of online parameters.
13 end
14 Discard everything except f and return that as the feature encoder.
X
f
g
q
online
X₁
target
f
g
BB 1,a
X
Figure 5: For BYOL+NRCC, the network architecture remains the same while the algorithm (original denoted in black) adds an extra pass (shown by the blue line) of the negative pair through the target network (the branch green). negative pair is created by taking an augmentation of x; as x and the SGHMC transformation of another sample Xj as x5. To calculate the NRCC regularizer, the x is passed through the online network (the branch groen) to get q(z)) while x goes through target be mappod to zf. The loss then be calculated by 6 for updating online while ensuring stop gradient on larget. The larget then updated using the momentum-driven running average of the online parameters in Grill al. 2000
4 Experiments
4.1
Experimental Protocol
To ensure a fair comparison, we refrain from using pre-trained models in all our experiments. We use ResNet-34 as our backbone network following the conventional recommendation. Morcover, as per standard protocol (Kim & Ye 2022 Chen al. (020b), we train the backbone for 1000 epochs with batch size 256. The number of iterations of SGHMC set to as increasing it did not provide any considerable
15

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Ablation study on CIFAR-10 dataset in terms of NMI, ARI, ACC, highlighting how the regularization by NRCC, negative pair generation by SGHMC, data dimension reduction by UMAP, and clustering by
GridShift, all when coupled together in the proposed framework, improve the performance for CL methods like InfoNCE and BYOL (best is boldfaced, second best underlined). Please note, DR: Dimension Reduction technique, CA: Clustering Algorithm, and Aug: traditional Augmentations.
NRCC
DR
Performance Indices
Comment
CL
Aug
SGHMC
t-SNE
UMAP
I-means
GridShift
NMI
ACC
ARI
C-+n-maans
IafoNCE
x
x
x
x
77.6
77.8
71.7
BYOL
x
x
81.7
89.4
79.0
CL++sSNEEGrrdSiitt
IafoNCE
x
X
v
78.1
78.3
72.5
BYOL
8
81.9
89.5
80.1
Ch-iMMAP-Gridhhin
IafoNCE
7
/
78.7
78.9
74.5
BYOL
82.5
89.9
82.3
CL+N w/o SCHMC+k-means
Info
x
x
80.4
85.1
80.2
BYOL
x
x
84.5
90.6
84.1
CL+N w/o SCHMCC+-NB--Grdhhii
InfoNCE
x
80.7
85.9
80.5
BYOL
x
x
84.9
91.1
84.8
CLANNCCC w/o
InfoNCE
/
x
81.5
86.7
82.4
CGHMC-UMAP++riCCE
BYOL
/
85.
92.2
85.1
CL+NRCC+
InfoNCE
x
86.8
92.3
85.2
k-means
BYOL
x
87.6
92.8
87.0
CL+NRCC++
InfoNCE
87.0
92.7
85.9
L-SNE+GridShit
BYOL
88.1
93.3
87.2
CL+NRCC+
InfoNCE
/
87.1
93..4
86.7
UMAP+GridShitt (Ours)
BYOL
88.4
93.8
87.9
performance boost. The rest of the hyper-paramotors for the proposed method are tuned by grid scarch, and we detail the same in Appendix C All the contending algorithms follow their originally recommended settings. For evaluating the clustering performance of the proposed methods, consider four types of datasets, namely largo-scale moderate resolution CIFAR-10, CIFAR-100 (Krizhevsky & Hinton 2009). and STL-10 (Coates el al. 2011), moderate scale higher resolution subsets of ImageNet such as ImageNet- and
ImageNet- (Russakovsky et al., 2015), large-scale higher resolution TinyImageNet (Le & Yang 2015).
and large scale moderate resolution long-tailed CIFAR-10-LT (Tang et al.
2020 and CIFAR-20-LT (Tang
ct al. 2020 Further details on datasets are provided in Appendix []] For CIFAR variants, we follow a similar architectural modification to NesNet-34 as suggested by SimCLR (Chen et al., (020b)). All contenders map the imput to 256-dimensional embedding space while data reduction, if used, further decreases it to three such that mode-seeking algorithms can be applied. The code base for the proposed techniques can be found at https:://itthbb ctt. http///biihhhheaa//////O00
4.2 Ablation Study
We begin with an ablation study on the CIFAR-10 dataset to understand how the different components of the proposed framework impact its clustering performance. We break down the ablation study into four stages, First, setting the baseline performances of InfoNCE and BYOL when a k-means clustering is used. Second, evaluating the improvement of baseline clustering performances t-SNE (Van der Maaten & Hinton 2008 or UMAP is used for dimension reduction, followed by GridShift. Third, validate the significance of NRCC for obtaining a friendl embedding space by applying the regularizer with standard augmentation techniques. Fourth, gathering evidence in support of SGHMC
fitting augmentation technique in the NRCC framework. In Table 2 we detail our findings in terms of three clustering performance evaluation metrics, namely Normalized Mutual Information (NMI) (Strchl & Ghosh 2002), Clustering Accuracy (ACC) Xie el al. 2016), and Adjusted Rand Index (ARI) (Hubert & Arabie 1985). From Table 2 we can observe that the baseline performances in terms all three indices improve when GridShift is applied on the lower dimensional space instead of k-means on the CL-ombedding space. Morcover, the improvements by GridShift. are more pronounced when UMAP is used for dimension reduction. We can make two critical conclusions from these findings. First, the mode-secking algorithm a lower-dinensional space not only removes the demand
16

---

Published in Transactions on Machine Learning Research (11/2024)
for the number of clusters & but also provides performance boost. Second, if performed by conserving local neighborhoods, data dimensionality reduction may further improve representation quality by decreasing the redundancy of noisy features still present in the embedding space. 'he rest of the ablation study provides evidence in favor of NRCC, i.c., the proposed regularizer, by demonstrating how it, in general, improves the clustering performance of baseline algorithms. However, the true potential of NRCC only realized when SGHMC is used for negative pair generation instead of traditional augmentation. This reaffirms SGHMC as an integral part of NRCC and highlights the necessary characteristics of transformation that best suits CL task. Moreover, UMAP is consistently found to be better choice than t-SNE for translating the cluster-friendly embedding space onto a lower-dimensional one, i.e,, suitable for effectively employing GridShift. Following these empirical observations, we safely fix the choice for our dimensionality reduction method to UMP and the clustering Lechnique to GridShift. Thus, our prescription for an optimal DC technique takes the shape of Baseline+NNCCC GridSSift.
Table 3: Comparison of clustering performance of the proposed WRCC+GridShiftt coupled with InfoNCE and BYOL, against 20 other State-Of-The-i (SOTA) DC methods in terms of ACC on six benchmark image dalasets of varying scale and resolution. The best boldfaced, the second best underlined, and the Margin is defined as the difference between the best proposed and its nearest existing SOTA.
Method
CIFAR-10
CIFAR-100
STL-10
ImageNet-10
ImageNet-Dogs
TinylmageNet
IIC (Ji 2019
61.7
25.7
DCCM Wil al. 2010
62.3
32.7
48.2
PICA (uung
2020
64.5
32.2
CC [T. al. 202
79.0
42.9
85.0
89.3
42.9
14.0
MICB
2021
83.5
75.2
GCC
Zhong et 01
2021
85.6
47.2
78.8
90.1
52.6
13.8
TCC
Ta al.
2122
90.6
49.1
81.4
89.7
59.5
(Saregg 21.
2022
45.1
94.2
43.4
14.1
SimCLR [Chen et nl.
2020h
77.8
42.6
71.6
90.8
34.6
14.8
MoCo (He
SimSiam 1 Cleen & T1
2021
85.6
71.6
92.1
67.4
20.3
BYOL
ct
82.5
93.9
19.9
EBCLR Kim
2022
79.3
42.9
72.3
81.2
31.9
13.2
PCL [T]
2020
87.4
41.0
90.7
41.2
15.9
IDFD
1ao et 2021
81.5
42.5
75.6
95.4
59.1
SCAN
Dang et ml
2021
88.3
44.0
80.9
NMM
Daug et 81
2021
17.7
80.8
CCES
(Yin et al.
2023
81.2
44.2
84.7
19.6
SACC
Dong
2028
IaoNCEE (Ours)
93.4
49.8
88.2
92.1
69.6
28.5
NYOL+ GridShift (Ours)
93.8
63.9
88.2
95.9
72.6
31.4
Margin
+-10.8
+1.8
+0.5
+11.1
4.3
Performance Comparison on Benchmark Datasets
For the comparative study of clustering performance, we take six benchmark datasets, namely, CIFAR-10, CIFAR-20, STL-10, and the three ImageNet subsets. As competing algorithmss we consider 20 state-of-the-art (SOTA) methods spread across six different DC strategies. First, the non- CL--techiiques, such as, IIC (Ji ct al. 2019), DCCM (Wu et al. 2019) and PTCA (Van Gansbeke et al.] (020). Second, the methods that directly produce cluster assignments, like CC (Li al. 2021), MiCE (Tsai et al., 2021), GCC (Zhong al.) 2021), TCL (Li et al. 2022 TCC (Shen et al. 2021). and C3 (Sadeghi et al. 2022p. Third, CL-methods that are not tailored for clustering but produce generalized embedding that can be efficiently clustered, for example, SimClr (Chen et al.] 2020b). EBCLR (Kim & Ye 2022), MoCo (He et al. 2020), SimSiam (Chen & He 2021), and BYOL (Grill et. al. 2020). Fourth, CL methods tailored for clustering tasks, such as IDED (Tao et al. 2021) and PCL (Li et al. 2020a). Fifth, multi-stage methods where a pre-training is followed by a clustering task-specific fine-tuning, like SCAN (Van Gansbeke et al., (2020) and NMM (Dang (et al.] 2021). Sixth, a couple of recent contrastive clustering methods explore the importance of better augmentation strategies in the context, namely, CCES (Yin et al. 2023) and SACC (Deng al. 2023
17

---

Published in Transactions on Machine Learning Research (11/2024)
Table 4: Comparison of clustering performance of the proposed and BYOL, against 20 other State-Of-The-Ae (SOTA) DC methods in terms of NMI six benchmark image datasets of varying scale and resolution. The best is boldfaced, the second best underlined, and the Margin is defined as the difference between the best proposed and its nearest existing SOTA.
coupled with InfoNCE
Method
CIFAR-10
CIFAR-100
STL.100
ImageNet-10
ImageNet-Deg Dogs
TinyImageNet
ICC (Ji et al
2019
51.3
43.1
DOOM
28.5
PICA
(uuaug
2020
56.1
29.6
CC [Li
2021
70.5
43.1
76.4
85.9
44.5
34.0
MICE Teat et al
2021
73.7
13.6
63.5
12.3
GCC
Zhonng
688
TCL
La 0E al. 21/2
81.9
52.9
79.9
87.5
62.3
TT
C3
(Sutegn
2022
74.8
43.4
90.5
44.8
33.5
SimCIR Chen nl.
2020h
77.6
40.9
62.4
80.6
35.9
33.8
EBCLR Iin & 2022
78.1
41.3
64.6
78.4
32.4
29.5
MoCo (Ho al.
SimSlam Chen &
2021
78.6
65.9
83.1
08.3
36.1
BYOL [Grlll et 01 2020
81.7
71.3
86.6
63.5
36.5
PCL [Li et al 2020
80.2
71.8
84.1
44.0
35.0
IDFD
64.3
54.6
Dang
2021
NMM Dang et
2021
71.8
18.4
69.4
CCES Yin 2023
72.4
43.6
77.5
38.2
SACC Deng
2028
76.5
44.8
69.1
87.7
45.5
NRCNCCE-NCCCGGral (Ours)
87.1
61.5
80.5
84.3
64.2
43.5
NBCC- (Ours)
88.4
65.8
80.7
90.6
68.4
47.8
Margin
+6.5
+12.9
+0.8
+0.1
+4..9
+9.66
Table 5: Comparison of clustering performance of the proposed BRRC++GriSShine coupled with InfoNCE and BYOL, against 20 other State-OffhheAr (SOTA) DC methods in terms of ART on six benchmark image sestunndeine
datascts of varying scale and resolution. The best is boldfaced, the second best underlined, and the Margin is defined as the difference between the best proposed and its nearest existing SOTA.
Method
CIFAR-10
CIFAR-100
STL-10
ImageNet-10
ImageNet-Dogs
TinyImageNet
ICC (Ji al. 2019
41.1
29.5
DOCM
40.8
17.3
26.2
PICA
(Huang et al 2020
46.7
15.9
CC LL al 2021
63.7
26.6
72.6
82.2
27.4
7.1
MiCB
Tisa ec al 2021
69.8
28.0
37.5
28.6
GOC
Zhong
72.8
30.5
63.1
82.2
36.2
7.5
TCL
Li et aL
2022
78.0
35.7
75.7
83.7
51.6
TCC
C3
Südeggh
al. 21122
70.7
27.5
86.1
28.0
6.5
SimCLR Chen et ul.
2020b
71.7
25.8
51.6
82.1
23.8
6.7
FBCLR [Km & Ye 122
72,4
25.9
53.8
77.6
19.3
5.8
MoCo (He
2020)
60.8
24.2
02.4
19.7
8.0
SimStam Cieen & Hc
2021
73.6
57.2
83.3
50.1
9.4
BYOL
10.0
PCL [Li et al.
2020a
76.6
67.0
82.2
29.9
8.7
IDFD
66.3
26.4
57.5
90.1
41.3
SCAN
Dang
NMM
Jang ct. al.
2021
70.9
31.6
65.0
CCES
Yin al.
2023
69.4
30.1
73.1
12.5
SACC
Dong ot
2023
72.4
28.2
62.6
84.3
28.5
CRCCC--NCCCGGriee (Ours)
86.7
34.5
76.0
83.4
54.1
15.7
RYOL+
(Ours)
90.2
17.66
Margin
+88.9
+13.5
+0.4
+0.1
+4.6
+5.1
We take three performance indices, namely ACC, NMI, and ARI, measure the chustering quality of an algorithm. In Table report the performance in terms of ACC while Tables and respectively tabulates
18

---

Published in Transactions on Machine Learning Research (11/2024)
Table 6: Comparison of clustering performance of the proposed GridShff,,, coupled with InfoNCE and BYOL, against MoCo and bascline BYOL, in terms of NMI, ACC, and ARI, two long-tailed image
datasets. The best is boldfaced and Margin is defined as the difference between the best proposed and its nearest existing SOTA,
CIPAR-1O-LT
CIFAR-20-17
Method
NMI
ACC
ARI
NM1
ACC
ARI
MoCo
46.7
33.4
27.7
31.2
28.2
16.1
BYOL
51.6
41.3
30.8
41.9
31.6
22.3
BYOL+NRCC+-ridShii (Oues)
55.6
44.2
35.3
45.3
38.6
28.3
Margin
+5.0
+22
+..5
+3.4
+4.0
+5.0
NMI and ARI measures. Closer inspection of Table 3 reveals that the proposed BYOL+NRCC++rddSie obtains the best performance all the six datasets, improving accuracy by 5.10% average from the nearest contenders among the 20 SOTA methods. The proposed SRGNNEE+NRCC+rrllSS second position on four datasets. This is an expected outcome given that BYOL is known to be a superior representation learner (Grill et al.] 2020). Similar behavior is reflected in Tables 4 and 5 as well. Specifically, BYOL+NRCC++riddee GridShift maintains its best position in terms of both NMI and ARI over all six datasets, leading ahead from its nearest contender, respectively, by 5.80% and 5.43% on average. The other NRCC variant, InfoNCE+NRCC++ GridShift, also performs consistently, standing second over five datasets in terms of NMI and four datasets when ARI is considered. These findings unanimously testify that NRCC indoed provides cluster-friendi embedding space that, when preserved by data dimensionality reduction through local neighborhood preserving UMAP, can significantly improve performance by GridShift without requiring the knowledge of the number of clusters. The generally improved representations learned by BYOL over InfoNCE are also further validated through their clustering performance.
also allains the
4.4 Performance Comparison on Long-tailed Datasets
Up to this point, we have only validated the performance of the proposed BYOL-NRCC-Grüdhhif on datasets with uniform-sized clusters. However, in reality, it is common to find datascts with imbalanced clusters long- taigt cluster distributions. Thus, consider two imbalanced datasets, namely, CIFAR-10-II and CIFAR-20-I.T, to validate NRCC's effectiveness. In Table 6. we compare the clustering performance of the AYOL+NROC+GridShIe
Table G we can observe that against MoCo and baseline BYOL in terms of NMI, ACC, and ARI. From retains its best performance, followed by BYOL, while MoCo performs the worst among the three. Specifically, applying ACC, and ARI, respectively, by 4.2%, 3.45%, and 4.75%, on average, over the bascline algorithm. Thus, per the empirical evidence, we can safely conclude that even in the case of inbalanced clusters, NRCC is still capable of finding embedding space where independent cluster structures accurately preserved.
GridShift improves NMT,
Table 7: Time taken in hours train BYOL and BYOL+NRCC on the same device with four V100 GPUs.
Dataset
BYOL
BYOL+NRCC (Ours)
Increment in Time
Increment in ACC
CIFAR-10
5h 33m
8h 17m
1.49x
4.9%
STL-10
2h 37m
3h 42m
1.41x
6.9%
ImageNet-10
5h 12m
7h 18m
1.40x
2.1%
ImageNet-Dogs
3h 17m
4h 48m
1.46x
4.6%
Tiny Imageeaeee
37h
52h 17m
1.40x
57.8%
Average Inerement Time for NRCG
1.43x
Average Inerement in ACC for NRCC
7.5%
4.5 Computational Overhead of the Proposed Clustering Methods
We have established the improved performance of NYCC+ but to be sure of its usefulness, need to study its computational feasibility compared to the baseline BYOL. Specifically, want to know
19

---

Published in Transactions on Machine Learning Research (11/2024)
if the probable increment in the proposed technique's training cost is negligible compared to the obtained gain in performance. We use the same computing setup with four V100 GPUs while calculating the time in hours to ensure fairness. Specifically, in Table 7 we compare the training time of baseline BYOL against the proposed BYOL+NRCC on five benchmark datascts, namely, CIFA-10, STL-10, and three ImageNet variants. We use the same computing setup with four V100 GPUs while calculating the time in hours to ensure fairness. Table 7 suggests that NRCC only puts 1.45x times computational overhead on the bascline on average, largely outmatched by the commendable performance improvement of 7.5% in clustering accuracy.
We undertake a follow-up study with similar computational settings to understand the potential speed and performance benefits of using SGHMC in NRCC. Specifically, we investigate the applicability of the two predecessors of SGHMC, namely Hamiltonian Monte Carlo (HMC) and the regular Monte Carlo (MC) sampling techniques. In Table [ we summarize our findings regarding the training time and Accuracy of the three BYOL+ variants when trained on C1FAR-10. We recall from Section [4. that in our implementation, SGHMC has only been iterated once. Hence, in our first experiment, we use a single itcration of HMC and MC to compare their performance against SGHMC. From Table g we can observe that in both cases SGHMC offers better performance at slightly higher computational cost. Regular MC though fastest among the three performs the worst by lagging 5.01% in Accuracy compared to SGHMC. On the other hand, though HMC and SGHMC share similar comparable costs, SGHMC outperforms HMC by 2.66%. Subsequently, to establish a fair setting, we increase the number of iterations for MC and HMC such that their performances closely match that of SGHMC. In this case, the performance gap between the three sampling strategies decreases while SGHMC becomes the most economical option by large margin. Specifically, both HMC and MC reach lower Accuracy compared to SGHMC while accruing increased costs of 1.89 and 1.37x, respectively. In conclusion, SGHMC offers more effective trade-off between performance and computational cost compared to its predecessors, when used in NRCC.
Table 8: Time taken in hours to train BYOL and BYOL+NROC on the same device with four V100 GPUs the CIFAR-10 dataset when NRCC uses SGHMC, Hamiltonian Monte Carlo (HMC), and regular Monte Carlo (MC) for generating negative pairs. Accuracy is abbreviated as ACC.
Method
Comment
Time
ACC
Against Baseline
Time
ACC
BYOL Grill al.
2020
For reference
5h 33m
89.4
NYOL+ (single iteration SCHMC)
Proposed method, here Baseline
8h
93.8
BYOL+NRCC (single iteration HMC)
7h 36m
91.3
+41m (0.91x)
-2.66%
BYOL+NRCC (single iteration MC)
5h 47m
89.1
+2h 30m (0.69x)
-5.01%
BYOL+NRCC (multiple iterations HMC)
Closest: baseline ACC'
15h 44m
93.6
-7h 27m (1.80x)
-0.21%
BYOL+NROC (multiple Iteration MC)
Closest to baseline ACC*
11h 24m
93.2
-3h 7m (1.37x)
-0.63%
Further increment in iterations did not improve ACC.
Better than baseline. +:
Wonse than Baseline.
4.6 Coupling NRCC with Other ofate-of-the-arts CL Methods
Up to this point, we restrict our discussion to InfoNCE and BYOL, two distinct foundational CL methods. Over the past few years, some other CL methods such as SimSiam Chen & He (2021) and Barlow Twins Zbontar et al. (2021) also gained popularity due to their superiority over the predecessors in some downstream tasks like classification and object detection. SimSiam can be thought of as a variant of BYOL that discards momentum-basod updates. However, Chen & He| (2021) p agree that such a change may sacrifice accuracy. This may be caused by slightly irregular feature embedding that does not preserve the cluster structures to the expected extent. On the other hand, Barlow Twins may compromise discriminative features in an attempt to reduce redundancy. This may negatively impact clustering performance which can worsen under incompatible choices of augmentation. Hence, coupling NRCC may not be proven beneficial for such basclines that are not canonically supportive of clustering performance. To validate this intuition, we consider an ablation study on the CIFAR-10 dataset in the following Table 9 We can see from Table g that SimSiam and Barlow Twins both perform better than SimCLR but worse than BYOL. Morcover, after coupling NRCC, both SimSiam and Barlow Twins perform worse than the two proposed techniques. Thus, NRCC aids both SimSiam and Barlow Twins in improving their clustering performance but fails to totally compensate for
20

---

Published in Transactions on Machine Learning Research (11/2024)
their bias against such task. In conclusion, NRCC improves the clustering performance of CL methods but the gain may be limited by the baseline algorithm.
Table 9: Coupling NRCC with other stato-of-tho-an CL mothods.
Method
ACC
ARI
NMI
SimSiam
73.6
78.6
Performance compared to SimCLR
+10%
+2.6%
+1.2%
Porformance compared to BYOL
-4.2%
-6.8%
-3.7%
BarlowTwins
87.3
74.2
78.4
Parformance compared to SimCIR
12.2%
+33.4%
+ +1.0%
Performance compared to BYOL
-2.3%
-6.0%
-4.0%
MuuSinn-NRCC++rraaaaa
87.1
75.4
80.3
Performance compared to LnoNCE++NICC+-rraS (Ouns)
-6.7%
-1.00%
-7.8%
Porformance compared to BYOL--NBCC+GrrdShit (Ours)
-7.1%
-14.2%
-9.1%
BarlowTwiis NRCC+ NNRCC+GridShiii
82.7
Performance compared to InfoNCT:--RRCC-Grrddi (Ours)
-3.2%
-7.3%
-5.0%
Performance compared to BYOL-NRCC+-rrdShit (Ours)
-0.4%
-1.3%
-1.4%
4.7 The Scalability of NRCC
To evaluate the scalability of the proposed NRCC we conduct a couple of experiments. The first experiment is focused on large-scale datasets with high number of clusters such as the full ImageNet-1k. 1k. The sccond experiment investigates the applicability of NRCC with larger backbone networks such as the deeper variants of ResNet and Visual Transformers (ViT). In both cases, we consider the BYO++NRCC+GrrdSSin variant as our proposed method of choice given maintains a consistent performance improvement over InfoNCE-NRCC+
The following Table I0 documents the findings of these two experiments.
In the first experiment with ImageNet-Ik, 1k, follow the experimental protocol described in Li et al. and train a ResNet-50 for 200 epochs. To ensure fair comparison, we only consider the state-of -otofftheeee the-ah art methods that have a reported result (either in the original paper or in later reference such as Li et al. (2020a) on ImageNet-1k in terms of Adjusted Mutual Information (AMI) under the common protocol (Li ltt al. 2020a).. From the top half of Table 10 we can observe that even when the number of clusters is increased in ImageNet-1k the proposed SYOL+NRCC+GrrdSan
2020a
sustains a commendable lead in Adjusted Mutual Information (AMI) over the the contenders. Specifically, the proposed NYC++ method achieves 5.9% improvement in AMI from its closest and most recent contender PIPCDR (Kumar & Lee 2025). The experiment also shows that if replace the traditional ResNett-50 back with a better performing ViT-Small network then a further 1.2% performance gain can be achieved for our method. Moreover, in comparison to the BYOL baseline, using ViT--mmall the proposed NRCC regularizer can provide a 11.9% boost in AMI. Thus we can safely establish NRCC as scalable to a large number of clusters.
In the second experiment, we consider the TntgeNet- dataset and vary the backbone network of BYOL+NRCC+Gridse GridShift between two ResNet and three ViT variants. Specifically, we take ResNet-34 and ResNet-50, the two widely popular residual backbones in contrastive learning literature, which are also used throughout this paper. We further consider the more recent ViT-Tiny, Vit-Small, and ViT-Base networks as backbones that gradually increase the number of parameters from 5.8M to 86M. For fairness, all five networks are trained under the same experimental protocol detailed in Section 4.1 and Appendix C If we summarize the bottom part of Table I0 that presents the results of this second experiment then we can observe four interesting factors. First, changing the backbone from ResNet-34 to the larger ResNet-50 does not improve the performance but slightly decreases all three metrics. Second, Vit-Tiny suffers a somewhat limited performance deterioration of 2.7% (compared to the best performer Vit-Small) average with only 26% of parameters. Third, Vit-Small and Res.Net-34 though share an almost equal number of parameters the ViT has an improved performance in terms of all three indices, making it the best among the five choices. Fourth, similar to ResNet increasing the number of parameters from Vit-Small to ViT-Base also acts against the performance. We can make two conclusions from these observations. First, VIT owing to its generally better performance than ResNet has the potential to become a primary backbone for contrastive clustering. Second, at least for most clustering tasks, blindly increasing model capacity without accounting for the data availability and problem complexity may not be straightforward strategy for finding success. This is evident.
21

---

Published in Transactions on Machine Learning Research (11/2024)
from the consistent performance of similar-sized models such ViT-Small and ResNet-34 on moderate-scale problems like ImageNet subsets while the larger ResNet-50 finds success on more complex ImageNet-1K as seen in the previous experiment.
The scalability of the proposed BYOL-VRCC+rrdShie contrastive clustering method. The best result is boldfaced while the second best is underlined. Note that the abbreviation NoP stands for the Number of Parameters in the backbone network.
Table 10:
The performance of BYOL--NRCC+ Gridhhif ImageNet- Ik compared the state-of-tho-art.
Experiment
Dataset
Method
Backbone (NoP)
AMI
DeepCluster Li et al.
2020a
28.1
MoCo [Li al.
2020
28.5
Large dataset
ImagoNet-II Ik
PCL [LT
2022
ResNet-50 [25.6M]
41.0
ProPos [Huang ot al
2022
52.5
PIPCDR [Kumar
2025
51.2
NYOL+NNcCC+GradSii (Ous)
57.4
BYOL BYOL+NRCC+ GridStift (Ours)
ViT-Small (22.2M)
51.9
58.1
DeepCluster Original article Caron et al. 2018).
MoCo original article He et al. (020))
The performance of NROL--NRCC-GrdBBut ImageNet-Doge when larger backbone networks used.
Experiment
Dataset
Method
Backbone (NoP)
NMI
ACC
ARI
ResNet-31 (22.7M)
68.4
72.6
59.4
ResNet-50 (25.6M)
67.6
72.2
59.3
Large backbone
ImagoNet-
BYOL--NRCC++rrddsii (Ours)
ViT-Tiny (5.8M)
66.9
72.0
58.3
VIT-Small (22 (2)2MM
68.8
73.8
60.1
VIT-Base (86M)
68.5
72.6
59.5
5 Conclusion and Future Works
We recognize the uncontrolled interplay between positive pair-induced attraction and, if present, negative pair-influenced repulsion as critical factors leading to collapsed or fragmented clusters in CL-bbsed DC methods. In response, we introduce the NRCC regularizer, which carefully balances these two counteracting forces while enhancing them ensure that the inherent cluster structure in the dataset remains intact in the embedding space. To achieve this, we re-envision SGHMC, a sampling method initially designed improve the computational efficiency of Hamiltonian Monte Carlo for online inference, as an approximately invariant augmentation strategy in CL. As per knowledge, this is the first work that aims to repurpose sampling method such as SGHMC, as a data augmentation strategy. To further justify the aptitude of SGHMC as an augmentation in the CL framework we theoretically establish its approximately invariant property and empirically demonstrate its ability of curating hard negative pairs following our critcria. Utilizing SGHMC- generated iegalive pairs in NRCC roduces density-aware repulsive force, facilitating the learning of a cluster- frintdr-freedly embedding. Consequently, the representation acquired by NRCC can be effectively projected to lower-dimensional space using a local neighborhood- pressooviidrrrriingggg method. This opens up opportunities to apply mode-seeking algorithms that do not necessitate prior knowledge of the mumber of clusters.
There are two limitations of the current work that open potential future avenues of research. First, unlike end- to-end methods (Huang et al. 2022), we take a modular approach, where three subsequent steps for embedding learning, dimensionality reduction, and mode-seeking clustering are used. On one hand, such an approach offers flexibility allowing the user to employ state-of-the-art neighborhood-preeee preservig dimensionality reduction techniques and clustering algorithms of choice to further improve application-sppeiite performance. Moreover, unlike existing approaches, this modularity enables our framework to allow mode-seeking clustering algorithms that remove the dependency the number of clusters. On the other hand, if perfect harmony among the different choices for the components is not met, then that may restrict the framework from performing at its true polential. To alleviate this issue, a balance can be achieved by simultaneously performing the dimensionality reduction task in the CL embedding learning framework. Specifically, instead of directly using an external UMAP-type technique, one may emulate the same through an additional auxiliary loss in the CL objective itself. Such an approach may further benefit from shifting the problem to the graph contrastive learning domain, where representing the neighborhood and the modes can become easier. However, this
22

---

Published in Transactions on Machine Learning Research (11/2024)
may not be straightforward given the additional computational overhead, algorithmic complexity, modified definition of positive and negative pairs, and the enhanced difficulty of directly finding the cluster structures in a low-dimensional embedding space. Second, even though SGHMC stands out as an excellent choice for augmentation in CL methods for DC, it still poses a potential threat to the time complexity due to its iterative nature. Our comparative study in Table suggests that NRCC only introduces a minute compulational overhead of 1.43x times over BYOL while performing 7.5% better on average than the canonical method. However, considering there may still be room for improvements, future research direction may investigate better alternatives to SGHMC that can produce approximately invariant augmentations for hard negative pairs in more comparatively economical way.
Acknowledgment
This research has been conducted with the financial support of the European Union under the RE- FRESH Research Excellence For Region Sustainability and High-tech Industries project number C7.10.03.01 00000000000000200000000000000000 003/0000048 via the Operational Program Just Transition.
References
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint orxi 1902.09290, 2019.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.05577 2019.
Deyu Bo, Yuan Fang, Yang Liu, and Chuan Shi. Graph contrastive learning with stable and scalable spectral encoding. Advances in Neural Information Processing Systems, 36, 2024.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-dlistrbbution-aware margin loss. Advances in neural information processing systems, 32, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Huy N Chau and Miklós Rásonyi. Stochastic gradient hamiltonian monte carlo for non-convex learning. Slochastic Processes and their Applications, 149:341 368, 2022.
Shuxiao Chen, Edgar Dobriban, and Jane H Tee. group-theoretic framework for data augmentation. The Journal of Machine Learning Research, 21 (1):99888 9955, 2020a.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference machine learning, pp. 1683-1691. PMLR, 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-13758, 2021.
Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence, 1798))790-799 1995.
Ching Yao Chuang, Joshua Robinson, Yen- CanCChe Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Aduances in neural information processing systems, 33:8765 (487665-7775, 8775, 2020.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth internalional conference on artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp- 13693 13702, 2021.
Xiaozhi Deng, Dong Huang, Ding- Hua Chen, Chang-Dong Wang, and Jian-Huare Lai. Strongly augmented contrastive clustering. Pattern Recognition, 139:109470, 2023.
Xuefeng Gao, Mert Gürbüzbalaban, and Lingjiong Zhu. Global convergence of stochastic gradient hamiltonian monte carlo for nonconvex stochastic optimization: Nonasymptotic performance bounds and momentum- based acceleration. Operalions Research, 70(5):2931-22) 29477, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in newral information processing systems, 33:21271 18:21471-21284, 2020.
Xifeng Guo, Long Gao, Xinwang Liu, and Jiamping Yin. Improved deep embedded clustering with local structure preservation. In Proceedings of the Twenty- Siath International Joint Conference on Artificial Intelligence, pp. 1753-1759, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEFF/CWV Conference Computer Vision and Pottern Recognition, pp. 1074 1083, 2021.
Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion, pp. 8849 8858, 2020.
Zhizhong Huang, Jie Chen, Junping Zhang, and Hongming Shan. Learning representation for clustering via prototype scattering and positive sampling. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification,
1985.
Jennifer Jang and Heinrich Jiang. Meanshift++:: Extremely fast mode-seeking with applications to segmenta- tion and object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 4102-4113, 2021.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9865-9874, 2019.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzacpfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in neural information processing systems, 33:21798 21809, 2020.
Boomsu Kim and Jong Chul Ye. Energy-based contrastive learning of visual representations. arXie preprint ar Xii::2202.04933, 2022.
24

---

Published in Transactions on Machine Learning Research (11/2024)
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Abhishek Kumar and Dong-Gyu Lee. Enhancing clustering representations with positive proximity and cluster dispersion learning. Information Sciences, 686:121277, 2025.
Abhishek Kumar, Oladayo S Ajami, Swagatam Das, and Rammohan Mallipeddi. Gridshift: A faster mode- secking algorithm for image segmentation and object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 8131 8139, 2022.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervisod representations. ar preprint urXiv:2005. (tle:20000004466,, 2020a.
Xuelong Li, Rui Zhang, Qi Wang, and Hongyuan Zhang. Autoencoder constrained clustering with adaptive
neighbors. IEEE transactions on neural networks and learning systems, 32(1):443-4455 2020b.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAT Conference Artificial Intelligence, volume 35, pp. 8547-8555, 2021.
Yumfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. Twin contrastive learning for online clustering. International Journal of Computer Vision, 130(9):2205 2221, 2022.
Jiexi Liu and Songcan Chen. Timesurl: Self-supervised contrastive learning for universal time series representation learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 2024.
Kanti V Mardia and Peter Jupp. Directional statistics. John Wiley Sons, 2009.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1542:080003422 2018.
Kenta Oono and Taiji Suzuki. Approximation and non-parametric estimation of resnet-type convolutional neural networks. In International conference machine learning, pp. 4922 4931. PMLR, 2019.
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory, pp. 1674 1703. PMLR, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet. large scale visual recognition challenge. International journal of computer vision, 115:211 252, 2015.
Mohammadreza Sadeghi, Hadi Hojjati, and Narges Armanfard. C3: Cross-instance guided contrastive clustering, arXiv preprint Xrx::22211.07133 2022.
Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip Torr, and Ling Shao. You never cluster alone. Advances in Neural Information Processing Systems, 34:27734-27746, 2021.
Bharath K. Sriperumbudur Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert R. G. Lanckriet. On the empirical estimation of integral probability metrics. Electronic Journal of Stalistics, 6:1550 1599, 2012.
Alexander Strehl and Joydeep Ghosh. Cluster ensembles knowledge reuse framework for combining multiple partitions. Journal of machine learning research, 3(Dec):583-611, -117 2002.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33: 1513-1524, 2020.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Yaling Tao, Kentaro Takagi, and Kouta. Nakata. Chustrringfriennlly representation learning via instance discrimination and feature decorrelation. arXiv preprinl urXiv:2166.000 2021.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Aduances in newral information processing systems, 1856827-6830 2020.
Tsung Wei Tsai, Chongxuan Li, and Jun Zhu, Mice: Mixture of contrastive experts for unsupervised image clustering. In International conference on learning representations, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1), 2008
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Lue Van Gool. Scan: Learning to classify images without labels. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Parl X, pp. 268 285. Springer, 2020.
Tao Wang, Guangpin Tao, Wanglong Lu, Kaihao Zhang, Wenhan Luo, Xiaoqin Zhang, and Tong Lu. Restoring vision in hazy weather with hierarchical contrastive learning. Pattern Recognition, 145:109956, 2024.
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance. Bernoulli, 25(4A):262) 2648, 2019.
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8150 8159, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non con-maametrric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733 3742, 2018.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487. PMLR, 2016.
Bo Yang, Xiao Fu, Nicholas Sidiropoulos, and Mingyi Hong. Towards k-means- -eenn-- spaces: Simultaneous deep learning and clustering. In international conference on machine learning, pp. 3861-3870. PMLR, 2017.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5147-5156, 2016.
Jun Yin, Haowei Wu, and Shiliang Sun. Effective sample pairs based contrastive learning for clustering. Information Fusion, 99:101899, 2023.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 13:5112-5823, 2020.
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning. Science, :79(663)) 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann TeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
Jiaqi Zeng and Pengtao Xic. Contrastive self-supervised Icarning for graph classification. In Proceedings of the AAAI conference on Artificial Intelligence, volume 35, pp. 10824 10832, 2021.
Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11289 10-111122 11297, 2023.
26

---

Published in Transactions on Machine Learning Research (11/2024)
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10042 10042-10051, 2021.
Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nic, Zhouchen Lin, and Xian-Sheng Hua. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp.
2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral- network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 9719-9728, 2020.
A
Notations
The following Table 11 describes the different mathematical notations used throughout the paper. Apart from these, for the proofs of Proposition 3.1 and Theorem 3.6 (in Appendix and
and B.2 respectively) any
additionally required notations and definitions are clearly described topically and are not carried forward elsewhere. Thus, for simplicity we discard them from Table 11
Table 11: Mathematical notations used this article according to the order of appearance.
Notation
Description
N.
samples residing the native space The set of all positive integers and the set of real numbers.
N,,R
d,
The dimensionslity of the native space X' and the embedding space.
The embedding space of dimensionality d. An encoder mapping from X' H.
f
x; h
A sample in native space from the set and the embedding space H. possible augmentations and two augmentations in The sample transformed by (T"). A mini-batch N of size
T,T".
The set
(xc"
N.n
PP(P-)
The set all positive (negative) pairs in mini-batch. The negative pairs w.r.t. (T")").
(L(P))
9
Additional dense layer map the embedding space another space
CChen
2020b
2.
The mapped sample o(h) and the space spanned by the mapping g-
Temperature parameter controlling the extent similarity simm e.g.
product (x-).
InfoNCE loss function Additional prediction block in target branch of BYOL (Grill et al.
i-th sample and augmentation T (T(T")
2020
BYOL loss function (w.r.t. i-th somple and augmentation 7 Mappings and outputs for target branch of BYOL Grill al. 2020 NRCC regularized loss functions. Hyper-paramett
70
relative wright Arbitrary augmented view of x E N.
NRCC regularizer.
P,
The probability distribution on space X, random variahle following P, and empirical distribution
N observations of Xs.
A map T x X X.
e
An identity augmentation Non-identically distributed.
Conv w,8. C, FCW.
Couvolution operation with weight w, bias b, and activation a mapping from C channels to C'. Fully connected layer with activation a, weight W, and bias A typical padding operation.
M, L
There are M residual block each of depth f. in the deep network under concern. The embedding function realized through a deep network with the parameter space Q. Idlentity mapping.
ld
B", B",B D.
Constants greater than zero, used for various bounds. Note that coutrolled by two non-uegative real sequences {anhnen (b, Jaen. The probability distribution T, random variable following Q-augmentod distrihution, and empirical distribution based on N L..dd coples of Y.
Q Pr.
Po
Prohability distribution over pairwise distance characterized by Q.
6.61,
Control parameters of SGHMC.
p.s
Intermediate vectors associated with SGHMC. empirical expectation and 2. Wassorstein distancrs.
Wi. W/
27

---

Published in Transactions on Machine Learning Research (11/2024)
B Notes on Theoretical Analysis
In this Section, we provide the detailed proofs of Proposition 3.1 and Theorem 3.6
B.1 Proof of Proposition 3.1
Let start by obscrving that,
Ex. Un (P(P.N)
- Ex (PP) - WE 2
where Wi is the 1-Wasserstein distance, defined Wi 8) ( JH ||x ylldy(x, y), given H. the underlying Polish embedding space. Also, r is the class of measure couples whose marginals turn out to be a and B. The inequality is due to the Henttoooric-Rubiistte
duality (Chen et al. 2020a).
Now, let E denote the class of n-Hölder functions mapping from to H, where π > 0. Given / LYN and measures a, 8 defined on the input space, using the triangle inequality we get,
(V vn(3)) Wi(((a), & (VG(a))) + Wi((sS) v((3).
Also, Wi (V))))) S 1 lo where the suppressed constant is dž. Moreover, since there exists a constant A > such that H is subset of the class of A-Lisschitz functions, one can argue,
inf yET' (a.8) Xxx / [(x) - 1(y) dy(x,)) s
/ TET' (a..)) (a.8)) XxX
(i(()))
inf
||x ylldy(x,y) Wi(a,
We now combine the findings till this point to get Wi (V))( (2(3) A (i 3) S 11 - -
Given any / C, and the activation function being ReLU, there exists Va with O(M) residual blocks, each with (1)) channels and of depth Ollog M), such that || 21 M OOno & Suzuki, 2019 The suppressed constant in the inequality involves d, while the result holds for any filter size {2, Considering our case,
Wi (1 ( - A (I M-E.
II exact invariance holds, Lee TX= X then, iTT, acts as consistent estimator of P under the Wasserstein distance. In other words, the error will plunge to zero as N increases. However, the departure from such scenario would hinder the shrinkage of the discrepancy even in large sample regime. To demonstrate the same, we write, Wi (PT,N,, S Wi (IP,,, + Wi (PNN, P), where the term Wi (P a.s; 0 as N - (Weed & Bach 2019). Further, in non-asymptotic sense, it is upper bounded by O(N max (d.2)), with high probability. Therefore, essentially the error committed during estimation is the disagreement between the two plug-in estimators. This will give us the extent of deviation due to augmentation. Following Sriperumbudur et al. (2012), we can also write,
Wi (i sup SEC
Σ W.f(Z:)
(21)
where C2 is the class of real-valued 1-Lipschitz maps on X. Also, W; N: Z, Y, for = 1,... N and
WN+i - + ZN++ X; for i I,... N. The solution to (21) can be achieved by solving linear program (Sriperumbudur et al. 2012). As such, suffer the most deterministic error due to particular choice of augmentation. Now, combining the results obtained so far and taking expectation, we get.
EV (P)(P) - YN S O(M O(PN (d.7)
where 7N = AEx, supjEct
EiM W.)(Z))
Hence the proof.
28

---

Published in Transactions on Machine Learning Research (11/2024)
B.2 Proof of Theorem 3.6
The proof follows as corollary to Raginsky et al. (2017) that reiterate with additionally required context. for smooth read. To put our reparametrized formulation into perspective, let us first define
g(x, Un)
Van (x);
where Un = (. ,II,) given that I,,... In are random permutations of 1, n. Also, F(x) ff (()())) where y denotes the law 92 follows such that F² minxex F(x). The corresponding empirical risk realized set of parametric values is given as Fn(x) 1 -== Pa, F (x). Under this setup, our update rules, conditional on given value of f2 corresponding to SGHMC, boil down to
Pj+1 (1 -1))p 629(X,, Uas) x,=s, + 0355+1,
8j+1 8 + d2P;
where the initializations So.80 and (en, (ryhjen are mutually independent.
Now, let sj be the output after j N iterations, and (8*, pD) are instances that follow D(s", p'[12 = 1l, namely the Gibbs distribution as dcfined carlier. Fragmenting the excess risk realized at j into distinct sources of variation yields
B(F(s,)) - == -(LF))))))E((())++++)) + [(Fn(s*))
E(Fn(s))) F
(22)
where ε denotes the excess population risk due to SGHMC, taking Gibbs algorithm as the baseline. The terms &, respectively signily the generalization error associated with the Gibbs algorithm and the extent. of expected suboptimality due to the same.
Let us concentrate on &g first. Given that F is achieved the value x",
B(Fo(8")) F* E Fs")
min Fa (x) xEX
+
min Fn(x) Fx((*))
xEX
< E
Fs()
min Fa(x) xE
eA
< 6' log
+
m
28
where 5'
d62
The last incquality is due to Proposition 11 in Raginsky al. (017))
48,
To upper bound &z, observe that
E(F(s*) - B(Fa(5))) = E sy (s*) - FF(($)]
n n (8") - PPn.
where St' (2/)2 n Rn are i.i.d. replicates drawn independently of Ω and $* Now, expanding the term
inside the sunmation we get
1 1 (dSt) f ((da)) / (sp(s) (Paç(s) -Pr, (8)
E (8*) - Pa, (8*)
(dS2,, n
"up ... (uen) m(ds)) Teee Ry (d) (ds) (s) ... tu
"u, ... (ien)) u(aa))) / T ai. .e..) (cls) (s)
(dD) 1
( ((nnn (eopuot boleet
(
(ds) (ds) (
(d)
(23)
29

---

Published in Transactions on Machine Learning Research (11/2024)
Now, applying Proposition 12 of Raginsky et al
2017
23
obtain upper bound on & given
2d
+ 20"
E2 5 8'n
A
+ B
where B 2 is such that [[FP Σ B and the suppressed constant is the one associated with the logarithmic Sobolev inequality followed by aa (Raginsky al. (2017), Proposition 9).
To obtain an upper bound for the error E1, first, observe that by Assumption 3.2 for any given Q2 |[VPn(x)]] I A x B, vx E X' (Raginsky et al. 2017) Lemma 2). Denoting the marginal distribution of Sj, given ft, by Lj
D(s,) we can write
B(F(s,)) - B(F(s*)) = f |no |f F(s)) F(s)
The term on the write possesses a natural upper bound based on the following lemmaa. Lemma B.1 (Raginsky et al. (2017). Lemma 6; Chau & Rásonyi 2022). Lemma 5.2). Let a, B E (2)
(distributions having finite second moments) and h: X + IR such that h E C2 salisfies
(VM(x)ll$<sill x1||x|| + K2, VxeX
where K1 > O and K2 2 0. Then we have
1 h do -
1 h d3 s (xi0 + (2))))(a,)))
where 0² = f a[xxx)]xx / /(xx)|/x112
Hence, in our case
V F(s)eg(ds) / F(s)tn(ds) s (Ao B))ss mq))
given == [s [ss) v
[|s)))]|s < 00. Moreover, Theorem 2.8 and 4.1 in Chau & Rásonyj 2022 imply that there exists constants c K1, K2 and K3 that satisfy
Wa (2 ma) S Ki +c) + (2(M)(po(s), ma)) exp (-jK382);
(24)
where the semi-metric based on -(--) is defined as Wp(a, = inf infawass ~e s' )}, and po(s) is the s-marginal of the initial state distribution. As such,
E[p((s
E, (Ao +B) [1 +e) + K2 Wo mz) exp (-jKgd2)
To show that the term the right of (24). and hence the error & becomes arbitrarily small, choose
+ _ and (2X₁)* K3 log
2K, (, given > 0. A simple reparametrization gives us the
desired constants C1, 02 and C3 in the Theorem.
C Implementation Details
We have rigorously trained all models for 1,000 epochs, following the conventional recommendations (Tao
ct al. 2021 Tsai et al., (202)). We have utilized the Stochastic Gradient Descent. (SGD) optimizer with a
cosine learning rate scheduler, that includes a warm up for the initial 50 updates. For MoCo (He al.) 2020) BYOL Grill et al. 2020), and NRCC, we set the base learning rate 0.05, dynamically scaling it with the
30

---

Published in Transactions on Machine Learning Research (11/2024)
batch size (3 0.05 x n/256). Notably, we have multiplied the learning rate 3 for the feature extractor by 10 for the predictor networks of BYOL, and NRCC, crucial step discussed in Chen & He 2021 Grill et al. (2020) for achieving satisfactory performance.
For the other hyperparameters of NRCC, we have conducted grid search over the possible choices each to find the combination that worked best on average. Finally, we set the temperature T (searched between {0.1,1, I, 10} and the regularization weight À (varied between {0.1, 0.5, 1} 0.1 each. In the case of SGHMC, set 51. 62, 83, and as 0.1, 0.05, 0.99, and 1 (inereasing the mumber of updated did not provide any considerable improvement), respectively. Following conventional guidelines, the mini-batch size was 512 for MoCo and 256 for the remaining models, including NRCC.
We have constructed the long-tailed versions of CIFAR-10 and CIFAR-20, namely CIFAR-10-LT and CIFAR- 20-LT, by employing the codes from Tang et al. (020). We have followed the guidelines of Zhou et al. 2020 and Cao al. (2019) while constructing the LTT variants of CIFAR, that resulted in two datasots both with a cluster imbalance ratio (the number of samples in the most crowded cluster divided by the number of the
sample in the smallest one) of 10. Specifically, the long-tailed datasets predominantly contained samples from the majority clusters (head), while only few samples were present in the other clusters (tail). Table 12
highlights the key properties of the datasets used in this study.
Table 12: Properties of the eight datasets used in this study.
Datasets
Number of Sample
Number of
Resolution
Cluster
Clusters
Uniformity
CIPAR-10
Krizhevsky Hinton
2009
60000
32
Uniform
CIFAR- 100 Krighevsky & Hinkon
2009
60000
100
32 x 32
Uniform
STL-10 (Coates et 4L. 2011
13000
10
96
Uniform
ImageNet-IU Husensovsyy et al.
2015
13000
10
224
224
Uniform
ImageNet-Dogg Ruseakowsky et
2015
19500
15
224
224
Uniform
TinylmageNet
[To Yang 2015
100000
200
224
224
Uniform
CIFAR-10-4 1--LT
Tanng
60000
32
Non-mnifurm, imbalance ratio Non-uniform, imbalance ratio 10
CIFAR-20-LT
Tanng at.
2020
60000
20
32 x 32
The Number of Iterations in SGHMC
Throughout the article, we have theoretically and empirically established how SGHMC is immensely useful for NRCC as an approximately invariant augmentation strategy. However, to effectively balance the trade-off between computational overhead and performance gain (see Table 8 we only iterate SGHMC once. This follow-up study focuses empirically understanding the behavior of SGHMC--aased augmentation when the algorithm is iterated multiple timos. In Table 13 take path similar to that in Table by considering the mean and standard deviation of Euclidean distances between 1000 negative pairs from the Ieage dataset that are generated by SGHMC over gradually increasing iterations of Table 13 reveals that the SGHMC-augmented samples progressively converge over iterations towards a stationary benchmark as indicated theoretically (see Theorem 3.6). This is visually supported by Figure that demonstrates the augmented instances for two samples (the same ones previously used in Fig 2) over progressive SGHMC iterations. In both cases, the visible dissimilarity between the augmentations corresponding to distinct samples diminishes over iterations and almost becomes identical at the end of the 11th step, signifying near convergence. From Table 13 and 1 we also observe that the negative pairs generated by SGHMC even at iteration 13 still lie at a distance 34% greater compared to the maximum attainable by traditional transformations.
to 13. A closer scrutiny
31

---

Published in Transactions on Machine Learning Research (11/2024)
Table 13: Mean and Standard Deviation of Euclidean distance in the proposed BYOL+NRCC embedding space between 1000 ImageNet-10 samples and their corresponding transformed views obtained by SGHMC (ours) over progressively increasing steps.
SCHMC iterations
Distance between negative pair in
Difference of mean distance with the previous recorded iteration
embedding space SMeanaSttanaard Deviation)
Iteration
0.01 0.000
Iteration 3
0.0092::00000 0.005
-2.73%%
Iteration 5
0.0071::0.00 0036
-29.57%
Iteration
0.00G0=::000 0.0021
-18.33%
lteration 9
0.0055:1:0 0021
-9.09%
Iseration
0.0053110.002
-3.77%
Ieeration 13
0.005 10000000000000
-3.92%
Original
SGHMC Augmented Instances
Iteration1
Itcration
Iteration 5
Iteration 7
Iteration 9
Itcration 11
Figure 6: SGHMC augmented instances over progressively increasing iterations from 1-11 for two samples from the ImageNet-10 dataset.
32


Reviews:
[{"review_id": "BWdnaBX1tm", "reviewer": "Reviewer_GpT5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nMost of the concerns are addressed in the experiments, and the explanations are sufficiently detailed.\n\n1. There is extensive analysis on the effectiveness of NRCC, which strongly supports the claims made.\n\n2. A general regularization loss term is proposed, which can be easily applied to other methods.\n\n3. The combination of NRCC + SGHMC + GridShift shows significantly stronger performance compared to other state-of-the-art methods.\n\n## Weaknesses\n\n1. There is a lack of analysis on the effects of GridShift.\n\n2. When using SGHMC, performing multiple sampling steps incurs additional time costs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper demonstrates that the imbalanced learning signals from positive and negative pairs in traditional contrastive learning and clustering methods can lead to poorly differentiated embedding spaces. To address this issue, the authors propose a regularization method called Negative Pair-based Regularizer (NRCC) and a negative pair data augmentation method using Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) sampling. Authors experimentally show that NRCC and SGHMC complement each other and, when combined with other methods, achieve superior performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could authors provide examples of consecutive images as they change over several SGHMC steps and analyze how the distances in the embedding space vary for those images?", "broader_impact_concerns": "There are no specific concerns to address.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iO2Dk0n3J2", "reviewer": "Reviewer_5DJk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The author provided a detailed mathematical process of argumentation.\n- The experiments are extensive and the results are promising.\n- According to what the author said in the submission, it is the first time to investigate the Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) sampling method as a potential data augmentation technique in contrastive learning.\n\nWeaknesses\n\n- There are some writing errors in the manuscript, and the text has not been carefully proofread. e.g. in the introduction section, the last sentence of the second paragraph is repeated in the second sentence of the fourth paragraph. \n- The mathematical formulas are too complex and lack necessary explanations, making the content difficult to understand. Additionally, there are some minor writing issues present in the formulation of some mathematical formulas. i.e. n Formula 7, there is a missing ‘]’ symbol and TX ≠ d X. in chapter 3.3.\n- The paper only analyzes the combination of the NRCC method with InfoNCE and BYOL, which are classical contrastive learning methods. Looking forward to the results of its combination with the latest contrastive learning methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission contributes a Contrastive-Learning-based deep clustering framework named Negative pair-based Regularizer for Contrastive Clustering(NRCC), which generates cluster-friendly embeddings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors have added many detailed mathematical definitions and argumentation details, which are crucial for understanding the method proposed in the article.", "broader_impact_concerns": "I did not identify any concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cCbf3RQPZh", "reviewer": "Reviewer_wfDD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- One of the very strong points of the manuscript is the mathematical derivations of the gradients of the modified loss, and their associated interpretation in terms of impact on the learned representations. This is of great value for the reader.\n- The experimental comparison are extensive, on various datasets, using different metrics, and against several baselines. The ablation studies also show the interest of each of the proposed components.\n\n**Weaknesses**\n- The theoretical assumptions and proof for SGHMC are not intuitive (the reader is refered to Raginsky et al. (2017)), and it is unclear if the proofs in Appendix A bring something new, or they are just a mere replica of the ones in Raginsky et al. (2017). Either way, this part is very difficult to read, and the authors should provide more intuition.\n- Regarding the overhead computational evaluation, we are missing the impact of the Monte-Carlo sampling: it would be interesting to report that as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript makes the intuitive point that in high-dimensional representation spaces, Contrastive learning (CL) methods can create deformed clusters, due to imposing the push away of two samples that are in the same cluster or lie in a local neighborhood. In addition, it is pointed out that current combinations of CL and deep clustering (DC) always require the number of clusters in advance. This motivates the proposed method: negative pair-based regularser for contrastive clustering (NRCC). The highlighted contributions are:\n- New regularisation framework using hard negatives coupled with existing CL paradigms, leading to a clustering-friendly embedding space.\n- Theoreical characterisation of negative pairs, facilitating cluster-friendly representation learning.\n- Relieve the user from defining the number of clusters in advance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- what are cluster--friendly embeddings formally? is there a proper definition?\n- section 3.1 you mention a complex space but say that $\\mathcal{X}$ is a subset of a real vector space. This is confusing.\n- section 3.1: I believe your indices in $\\mathcal{P}\\_+$ are incorrect. The way it is written you always take the first n elements of the dataset. Perhaps you should use $x\\_i\\in\\mathcal{N}$ in the definition of $\\mathcal{P}\\_+$ (or define $\\mathcal{N}$ as the subset of indices of the mini-batch). The same issue holds for the othe rdefinitions in the paper (e.g. $\\mathcal{P}\\_{-}$). Perhaps the best solution is to talk exclusively about the mini-batch and avoid all this hassle.\n- the theoretical analysis of BYOL is done in a simplified case, while one of BYOL's strengths is all the \"engineering and tricks\" that made it successful. I would like to see the theoretical analysis wihtout the simplification, to see if the same interpretation and reasoning hold.\n- In equation 15, the $\\Psi$ function you define corresponds to the $f$ function you defined in 3.1, right? Why the two notations? Why is it necessary to specify all the convolutional and fully connected layers? Can't you just say that you consider networks with bounded parameters?\n- After (15) it is written \"square integrable on $\\mathbb{P}\\times\\mathbb{Q}$: what does it mean to be square integrable on a probability distribution?\n- following up: what is a plug-in estimator? and what are these two estimators estimating (they are estimators of what)?\n- After equation 16, you redefine $\\Omega$ (that was the set of parameters) as a random variable. Please correct this. Also it is not clear what \"a random variable responsible for characterizing the distribution\" means.\n- In assumption 3.2, you refer to $\\mathbb{P}_{\\Omega}$ as a function, but you defined it as a probability distribution: do you imply the pdf? Please rephrase.\n- In assumption 3.4, what does the inequality with tilde sign mean? Also, what do you mean that $I_1,\\ldots,I_n$ are random permutations (in plural)? Do you mean A random permutation?", "broader_impact_concerns": "I do not think this is required for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the proposed framework seems to be technically sound and moreover supported by extensive empirical validation. Most concerns raised by the reviewers are addressed in the revision with better description and additional experiments. Based on the consensus among the reviewers, I would recommend the paper to be accepted.

However, it is still necessary to clarify the main motivation and algorithm for the proposed regularization and representation learning. For example, it is unclear that why the proposed regularization losses in Eq. (5) and (6) can lead to good clustering and what is the role of an another augmentation x^c or z^c related to a hard negative. Most of all, it is hard to figure out the whole processing of the proposed representation learning. Algorithm 1, 2 and 3 in Appendix C should be located in the main body, and a figure to describe the overall processing is necessary. In addition, the performance evaluation with larger networks and datasets such as 
ViTs and full ImageNet is also necessary.
Decision: Accept (poster)