ConMix: Contrastive Mixup at Representation Level for Long-tailed Deep Clustering
Authors: Zhixin Li
Yuheng Jia
Venue: ICLR 2025 Poster (2025)
Keywords: deep clustering
long-tailed deep clustering
unsupervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We innovatively propose a ConMix method that can effectively address long-tailed deep clustering.

Abstract: Deep clustering has made remarkable progress in recent years. However, most existing deep clustering methods assume that distributions of different clusters are balanced or roughly balanced, which are not consistent with the common long-tailed distributions in reality. In nature, the datasets often follow long-tailed distributions, leading to biased models being trained with significant performance drop. Despite the widespread proposal of many long-tailed learning approaches with supervision information, research on long-tailed deep clustering remains almost uncharted. Unaware of the data distribution and sample labels, long-tailed deep clustering is highly challenging. To tackle this problem, we propose a novel contrastive mixup method for long-tailed deep clustering, named ConMix. The proposed method makes innovations to mixup representations in contrastive learning to enhance deep clustering in long-tailed scenarios. Neural networks trained with ConMix can learn more discriminative representations, thus achieve better long-tailed deep clustering performance. We theoretically prove that ConMix works through re-balancing loss for classes with different long-tailed degree. We evaluate our method on widely used benchmark datasets with different imbalance ratios, suggesting it outperforms many state-of-the-art deep clustering approaches. The code is available at https://github.com/LZX-001/ConMix.

Published as a conference paper at ICLR 2025
CONMIX: CONTRASTIVE MIXUP AT REPRESENTATION LEVEL FOR LONG-TAILED DEEP CLUSTERING
Zhixin Li', Yuheng Jia
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China
2 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China (lizhixin, yhjia (onjiaesseee seu. edu. cn
ABSTRACT
Deep clustering has made remarkable progress in recent years. However, most existing deep clustering methods assume that distributions of different clusters are balanced or roughly balanced, which are not consistent with the common long- tailed distributions in reality. In nature, the datasets often follow long-tailed dis- tributions, leading biased models being trained with significant performance drop. Despite the widespread proposal of many long-tailes -ailed learning approaches with supervision information, research on long-tailed deep elustering remains al- most uncharted. Unaware of the data distribution and sample labels, long-tailed deep clustering highly challenging. To tackle this problem, we propose a novel contrastive mixup method for long-tailes tailetailed deep clustering, named ConMix. The proposed method makes innovations to mixup representations contrastive learn- ing to enhance deep clustering in long-tailed scenarios. Neural networks trained with ConMix can learn more discriminative representations, thus achieve better long-tailed deep clustering performance. We theoretically prove that ConMix works through re-balancing loss for classes with different long-tailed degree. We evaluate method widely used benchmark datasets with different imbalance ratios, suggesting it outperforms many state-of-the-arr deep clustering approaches. The code is available at https: /tt:://gtttttt com/ com/LZX L2X 001 (001/Cont ConMix.
INTRODUCTION
Despite the rapid advancements in deep clustering in recent years, these methods often struggle to be directly applied in real-world scenarios due to lack of consideration for the long-tailed distribu- tions. Unlike the commonly used uniform distributed datasets in labs, the datasets in nature usually present Zipf long-tailed distributions over classes, where major classes (head classes) have more samples compared to minor classes (tail classes) (Feldman, 2020). Training with class-imbalanced datasets naturally leads to biased models with significant performance drop (Hou et al., 2023). Meth- ods with supervision information have been continuously proposed to eliminate the negative impact of long-taile tailed distributions on model training. Re-sampling (Ando and Huang, 2017; Han et al., 2005; He and Garcia, 2009; Shi et al., 2023) balances the number of samples for each class partic- ipating in model training. Re-weighting (Cui et al., 2019; Park et al., 2021; Lin et al., 2017: Ren et al., 2020; Tan al., 2020) aims to re-balance loss of different classes through different weights. Logit adjustment (Menon et al., 2020; Tian et al., 2020; Wu et al., 2021) seeks to adjust prediction logits to correct biased models. Although these methods and their sequels can alleviate the perfor- mance drop caused by long-tailed distributions, they are all label-dependent and unable to directly use for unsupervised learning. As without the supervision information, it will be difficult to curately distinguish between head class and tail class samples. Inspired by (Hooker et al., 2019), SDCLR (Jiang al., 2021) prunes network identify liffeoilttoommemrrize samples, usually atyp- ical and rare samples. In long-tailed self-supervised learning, SDCLR achieves good performance, but atypical and rare samples cannot be equated with tail class samples. Due to the large number of samples, there are also many atypical samples in the head classes, and tail class samples may
"Corresponding author; Yuheng Jia.

---

Published as a conference paper at ICLR 2025
not necessarily be rare. Meanwhile, the common pseudo labeling in deep clustering is not accu- rate under long-tailed distributions, as distance- based labeling methods like K-means (Hartigan and Wong, 1979) generally lead to uniform distribution results. Therefore, difficulties in differentiating head class and tail class samples pose challenges in extending lahel-dependent long-tailed learning to long-tailed deep clustering.
In recent years, deep clustering has been witnessed sustained development due to the advancement of self-supervised representation learning (Van Gansbeke et al., 2020; Niu al., 2022; Huang al., 2023; Li et al., bb2 2022; Yu et al., 2023; Shen et al., 2021). But unlike self-supervised learning that can be further fine-tuned with labels, deep clustering lacks supervision information, thus is more severely affected by long-tailed tailed distributions. Given long-tailed deep clustering is naturally more challenging than deep clustering on uniform datasets, few methods propose solutions on long-tailed deep clustering. We notice very recent work P2OT (Zhang et al., 2024) makes contributions to express pseudo labeling by progressive partial optimal transport for imbalance clustering. However, the method is based on pre-trained DINO (Caron et al., 2021) and is to fine-tune the last block of V6T-B16 (Dosovitskiy et al., 2020) instead of training from scratch. Hence, the bias affected by the long-tailed distribution in the overall model is relatively weak. In contrast, the purpose of our work is to propose a method that can train a model from scratch while alleviating the affect of the long-tailed distributions.
The challenges in long-tailed deep clustering is primarily on three main points. (i) The lack of label information renders unable to directly differentiate between samples from head classes and tail classes, as well as leading to lack evidence that are conducive to distinguishing samples of different classes. (ii) While experimental observations can identify difficul-to-mmmorize (e.g.. SDCLR (Jiang et al., 2021)) instances, these do not necessarily equate to tail class samples. (iii) Prior research (Zhang et al., 2023) has highlighted the feature space occupied by samples from head classes is often larger than that of tail classes, whereas existing clustering algorithms tend to produce clusters that are balanced in sizes. Under such extremely constrained conditions, we only know that tail class samples are scarce compared to the abundance of head class samples. The core of the problem becomes whether we can implicitly leverage the inherent characteristics of long- distributions to enhance deep clustering.
Mixup (Zhang et al., 2018) always leads to robustness and generalization supervised learning (Zhang et al., 2021). In long-tailed supervised learning, mixup has been proved to benefit represen- tation learning by improving complexity and diversity of datasets (Zhong et al., 2021). However, in unsupervised manner, input-level mixup may harm representation learning due the inability to learn semantically meaningful representations for clustering. Therefore, Manifold Mixup (Verma al., 2019), supervised method which interpolates representations in the feature space for better representations, is referable.
Inspired by Manifold Mixup (Verma al., 2019), we propose to extend multi-sample mixup at representation level to long-tailed deep clustering. In detail, we multi-sample in every batch and interpolate representations learned in contrastive learning to synthesize new representations. Unlike previous works (Kalantidis et al.. 2020; Zhang et al., 2022) only using the synthesized represen- tations to form negative pairs with the original representations, we directly use synthesized repre- sentations for further learning. Neural networks learned in this manner can obtain discriminative representations to benefit long-tailed deep clustering. We name the proposed method ConMix, a shorthand for Contrastive Mixing.
In summary, our main contributions as follows:
We propose a contrastive mixup at representation level for long-tailed deep clustering. In this man- ner, models can effectively alleviate the bias caused by long-tailed distributions and obtain compet- itive clustering performance.
Most existing mixup methods primarily focus on supervised learning and input-level mixing. We have found an effective multi-sample representation-levee mixup method that can be used for directly training networks in unsupervised learning thus extend it to deep clustering.
We provide theoretical proof of the proposed method. In unsupervised conditions, it very difficult to distinguish between head class and tail class samples and find a right antidote. But our method
2

---

Published as a conference paper at ICLR 2025
can implicitly re-balance the losses of head and tail classes, which is similar to many taigetalled supervised learning approaches.
2 RELATED WORK
2.1 DATA IMBALANCE
In reality, data distributions always follow long-tailed distributions where head classes are dominate and tail classes are minor. In long-tailed distributions, supervised learning methods tend to overfit samples from head classes and underfit those from tail classes, leading to poor generalization perfor- mance. To address this issue, numerous methods have been proposed, including re- sampling (Ando and Huang, 2017; Han et al., 2005; He and Garcia, 2009; Shi et al., 2023), re- re-weighting (Cui et al., 2019; Park et al., 2021; Lin al., 2017; Ren al., 2020; Tan et al., 2020), and logit adjustment (Menon et al., 2020; Tian et al., 2020; Wu et al., 2021: Jia et al., 2024). Besides, methods (K (Kang al., 2020; Zhong al., 2021; Hou et al.. 2023; Chu al., 2020) based on two-stage decoupled training scheme, which decouple representation learning and classifier training, have achieved ex- cellent performance in recent years. However, these methods rely on the prior provided by labels in long-tailed datasets, which is absent in unsupervised scenarios. Very few self-supervised methods, including (Jiang et al., 2021; Zhou et al., 2022; Liu et al., 2021), propose solutions for addressing class imbalance specifically. Usually, they implicitly increase the weights for rare samples training. These prove effective under self-suue frameworks but do not readily translate to long- tailed deep clustering, where the aim is to enhance discriminability for all samples, spreading out the dis- tribution of different classes while those from the same class to cluster closely together, rather than merely improving the learning of underrepresented ones. As for long-tailed clustering, the first work is (Zhang et al., 2024), which trains and fine-tunes on pre- pre-tedined DINO (Caron et al., 2021). How- ever, to our knowledge, no method trained on long-tailed datasets from scratch has been proposed. Our objective is to delve deeper into the implications of long-tailed distributions on deep clustering and figure out how to solve Therefore, it is necessary to train models from scratch on long-tailes datasets that are specifically designed for this task.
2.2 DEEP CLUSTERING
Deep clustering aims to learn good representations and cluster samples in an unsupervised manner. DEC (Xie et al., 2016) and its variants (Li et al., 2018; Yang et al., 2017; Peng al., 2023; 2021) train the model iteratively by autoencoder reconstruction loss. Although these methods can achieve better performance than traditional clustering, deep clustering methods based on autoencoders find it difficult to capture the complex structural features of input samples. Some methods based sim- ilarity between samples (Chang et al., 2017; et al., 2019; Tao et al., 2020) are proposed for more accurate partitions. Pseudo-labeling based methods (Van Gansheke et al., 2020; Tian et al., 2017; Niu et al., 2022; Xu et al., 2023) employ pseudo labels in deep clustering to guide network training. But how obtain pseudo labels of both high quality and quantity is a problem that needs to be con- sidered. Due to the advancement of contrastive learning framework, including SimCLR (Chen et al., 2020), Moco (He et al., 2020), BYOL (Grill al., 2020), etc. contrastive-bast been massive proposed, including (Van Gansbeke et al., 2020: Niu et al.. 2022; Huang et al., 2023; Li al., 20216: 2022: Yu et al.. 2023; Shen et al., 2021). They either simply use contrastive leamn- ing as representation learning, then train soadoollabl-baseee classifiers good features, or propose -errasttveelearingg based algorithms to make representations more discriminative, or directly out- put clustering results based on contrastive learning. For example, SCAN (Van Gansbeke al., 2020) first trains a feature extraction network by SimCLR for subsequent pseudo-labeling: ProPos (Huang et al., 2023) innovatively integrates prototype-level SimCLR and instance-level B BOL: CC (Li et al., 2021bb is based on SimCLR and directly train a prediction network for clustering.
deep clustering has
2.3 MIXUP
Mixup (Zhang et al., 2018) has demonstrated its efficacy in improving model generalization in su- pervised learning. Generally, Mixup regularizes the model by interpolations of the inputs and corre- sponding labels. Its variants (Uddin et al., 2021; Yun et al., 2019; Yao et al., 2022) make advance- ments the form of interpolation or the strategy for sample selection. On the other hand, Mani-
3

---

Published as a conference paper at ICLR 2025
fold Mixup (Verma et al., 2019) interpolates in the feature space to guarantee smoother decision boundary. Interpolations in the feature space inspire some mixup-based methods in self-supervised learning. In some methods (Kalantidis et al., 2020; Zhang et al., 2022), feature space mixups are used to generate hard negative samples for contrastive learning.
3 METHOD
3.1 PRELIMINARIES
Contrastive Learning. Contrastive Learning, including SimCLR (Chen et al., 2020) and SDCLR (Jiang et al., 2021) learns instance-level visual representations via pulling positive pairs close and pushing negative pairs away. We assume an instance-level representation vs 0, forms its positive pair and Di V. are included negative pairs, where Vi a set of negative samples for u, Then contrastive loss in SimCLR can be defined as:
1 LCL. = N Σ log (v; 2,,
8 (U;;
7) + Ev, EV, (V). Ui (T)
where 7 is a temperature parameter and s(u, T) = exp(u' u/T) when and are no-nomall representations. Usually, v, and v." are different augmented versions of the same input sample and v; are other representations in the batch, excluding v; and v,'
Manifold Mixup. Manifold Mixup (Verma et al., 2019) linearly interpolates feature in hidden layers to encourage flatter class representations that possess better generalization. As a supervised method, the mixup of feature 2; and 2, defined as:
syy 2, (1 A) AA)2)
(2)
V = A Vn. + (1- A) yzzy
where l/ represents the label vector of and mixing coefficient X ~ Bela(a,
We extend Manifold Mixup contrativeeleaming--aaee deep clustering to reduce model bias caused by long-tailed distributions. Considering that mixup the input level may mislead unsu- pervised models, we instead sample semantic representations to synthesize new mixed ones. Mean- while, obtain more diverse synthetic features, choose multi mampi-sa instead of pairwise sam- pling.
3.2 MOTIVATION
In recent years, deep long-tailed learning has made great progress, for the purpose that deep learning models can be better suited to real-world application scenarios (Zhang et al., 2023). However, the majority of proposed methods focus on learning with supervision information. Without label information, it becomes hugely challenging to perceive the long-tailed distributions. Hence, very few works are dedicated long-tailed unsupervised learning, especially long-tailed deep clustering, as introduced in Section and 2. As long-tailed sule-supe methods focus training hard-to- memorize samples, a method for deep clustering should aim at enhancing the handling of overall long-tailed distributions therefore mitigating model bias. Inspired by Manifold Mixup (Verma al., 2019) in Eq. (2), as well as previous practice of mixup in long-tailed supervised learning (Zhong al., 2021), we propose mixup-based method under the framework of contrastive learning, which significantly enhances the model training on long-tailed distributions, even outperforms state-of-the- art deep clustering methods.
3.3 CONMIX
In method, we follow the framework of SimCLR (Chen et al., 2020), including its network structure, data augmentation, and loss function. Consider a unlabeled long-tailed dataset of size N: 22, .. In SimCLR framework, each input æ; is augtene twice, and the two augmented versions are fed into two different network branches SimCLR. Given N inputs, the network will output 2N representations (01, 02 .2,n. Let us assume, for each [1, N). v;
4

---

Published as a conference paper at ICLR 2025
and U positive pairs from the same input while others are negative pairs. In representa- tions from distinct inputs {21, U2....u we randomly generate M synthesized representations {z1, 22, 23, .M. } in below manner:
2m
Zm
where
Zm
Uk-
(3)
[Um
where || l/2 is the Ey-norm, enabling the synthesized representations available for contrastive learn- ing, Umm a set of indexes belonged to original representations which synthesize Zm and | denotes the number of elements in the set. In each batch, we randomly assign tags within [1, .M] to original representations from the same network branch (21, V2 Uyyy }. The generation of tags follows a uni- form distribution with equal probabilities N7 and original representations with the same tag are used to synthesize one particular representation in the manner of Eq. (3). This multi-sampling schedule results in synthesized representations with different numbers of vy Not only does it obtain more diverse synthesized representations, but it also assigns different weights to different samples, simi- lar to how mixup does. We use the exact same method synthesize {=1+M, 2 M3+M, }. including the same random tags, ensuring that and 2++ M are generated from the same samples of different augmentations. In above manner, we extend the origi- nal positive and negative sample pairs to the synthesized positive and negative sample pairs: and Zi4-M are the new positive pairs synthesized from the same input samples while others are negative. For simplicity, z represents the positive sample of 2;. Then the contrastive loss in ConMix can be formulated as below:
from (UI+N, +,
CCM
1 2M
Σ log
2M
(Zi: (Z):2,
2M > kkll 1 [ (Zi, Tk,
where 1 [[=)|]] is an indicator function. It equals 1 when & * i, otherwise 0. In the ConMix frame- work, the loss contrast average features of multiple representations instead of individual representa- tions. It can be seen as a augmented form of mixup at representation level.
4 THEORY
ConMix can improve the performance of long-tailed deep clustering without the need to distin- guish between head and tail samples in advance because it can re-balance the loss between head and tail classes. In the existing work on long-tailed supervised learning (Hou et al., 2023; Kang al., 2020; Menon et al., 2020), many approaches focus on balancing the loss across different classes, that is, artificially reducing the loss for head class samples and increasing the loss for tail class samples, order to correct for biased models. These methods are relatively simple to im- plement when supervised information is available, but it becomes much more challenging in an unsupervised setting. As ConMix mixups representations from different classes, it can implicitly achieve the loss-balance via improving the loss of tail class samples and reducing the loss of head class samples. We have uncovered this perspective through theoretical analysis. The following is proof.
We first rewrite the Ler. in a form alike LConMix and then simplify it (the detailed derivation is in Appendix C.I):
2N
LCL
log Σ 1 1x 2k. v, )/T),
2N
(5)
where (v,, rk. o, v. (mk w²))
In Eq. (5), fC, U2 vit) calculates the similarity difference between each sample e; with its particular negative sample iy and with positive sample D.* For a specific v; its loss value depends on the difference between the similarity of all negative sample pairs and the similarity of positive pair. However, since fC1. (V,, tk. of)) cannot measure the contribution of individual representation to the total loss, so we propose new function lCt. (v:):
2N
2N
hc((v:) = Σ 1essfeL 0k 0+, =
Σ
1 u (ok - 2tt).
(6)
5

---

that in a long-tailed Published as a conference paper at ICLR 2025
In the Eq. (6), the larger hCL (v;) ( is, the larger the sum of fcL( v," associated with the specific v; is, thus the greater the loss associated with v, is in LCL. In other word, (CL(U,) (v:) indicates the contribution of representation v; the total CCL. From the Eq. (6), distribution, head class representations always have greater loss for more similar negative pairs from the same class, leading biased model. On the contrary, samples from tail classes, due their scarcity of similar instances, tend to incur smaller loss and are consequently overlooked during training.
As for ConMix, we also need to measure the impact of individual vi but the situation is slightly different. LcM can be similarly rewritten as form of fcM 2k: 2,
2M
2M
CCM
Σ log
Σ
1
Aett exp (Zi, 2k. 2, 1/T),
2M
k=1
2N
where SCM 2k. 2, =
Σ
1 (2)- - =,²);
To measure the impact of an individual v; on LCM, we assume that zn is the synthesized represen- tation corresponding to vi, and calculate the difference in all fcM (2M, 2 when e; synthesizes 2x versus when does not (the detailed derivation is described in Appendix C.2):
2M
2M
(cM Σ [eM Zyz 2x
ScM (2'z Zz, z1t)
y=1
(8)
2NN
D. D.'RR Uk (2M +1) Σ 0, Uk:
2
Uk
k=1
kES.
KEST
where zle synthesized from all representations composed of 2, except v;, S, and 8 are the sets contain indexes of original representations from &g and 25. respectively, as u; synthesizes 2, and of synthesizes = denotes the assignment operator. Note that we have made simplification to omit the normalization of the synthesized representations and the proposed lCM is adapted to align with hCL for better comparison. (c,) indicates the contribution of vs LCM, and larger hCM(v;) leads to larger LCM- Based on the distinction in similarity (inner product) of representations across different classes, now between head and tail class samples.
propose new theorem explain how ConMix re-balances the loss
Let the similarity between representations within the same class is roughly the same 8w and the similarity between samples of different class is also roughly the same 8b- > And ConMix, the number of original representations composing each synthesized representation roughly equal, and their class distribution aligns with the overall class distribution. Then in the long-tailed distributions, for a head class representation Un and a tail class representation vs hcM (v%.) (xt) hCL(v)) hCL(v.)-
Theorem 1.
The detailed proof of Theorem 1 is in the Appendix C.3, Given that hcM(v)) (CM((V)) < her (vr) hct. (v:), the loss of U; is relatively lower in ConMix compared to contrastive learning. and v. has larger loss in ConMix relatively compared to contrastive learning. That is, ConMix can relatively lower the loss of head class samples and increase the loss of tail class samples, thus achieve loss re-balance implicitly.
5 EXPERIMENTS
5.1
EXPERIMENTAL SETUP
5.1.1 DATASETS AND LONG-TAILED SETTINGS
We conduct experiments on three benchmark datasets, including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-20 (Krizhevsky al., 2009) and STL-10 (Coates et al.. 2011). Among them, CIFAR- 20 a version of CIFAR- 100 that uses 20 super-classes. Since these three datasets are all balanced datasets, manually generated total of six long-tailed distributions using two imbalance ratios:

---

Published as a conference paper at ICLR 2025
5 and 10. The imbalance ratio, which is the ratio of the maximum and minimum classes, controls the long-tailed degree of data distributions. We generate the sample number for each class following the setting in (Tang al., 2020: Zhou al., 2020; Cao et al., 2019), where the number of each class in long-tailed distributions exponentially decreases. We use the train dataset in CIFAR-10 and CIFAR-20, both train and test dataset in STL.-10 (due its small train dataset) for long-tailed deep clustering, following (Tao et al., 2020). Besides, we have also conducted experiments on CIFAR- 10 with an imbalance ratio of 100 and on Tiny ImageNet (Le and Yang, 2015) with an imbalance ratio of 10, further demonstrating the effectiveness of ConMix in scenarios with more long-tailed distributions and greater number of classes. We also report clustering results on ImageNet-LI (Liu et al., 2019) in Appendix F to demonstrate the effectiveness and generalization of ConMix.
5.1.2 IMPLEMENTATION DETAILS
We use ResNet-18 18 (He et al., 2016) as the backbone for all experiments, unless otherwise specified. We train all methods for 1000 epochs and report results of the last epoch for fair comparisons. We use batch size of 512 for all methods unless otherwise specified. We hope to mixup semantically meaningful representation, thus train our model only using SDCLR (Jiang et al., 2021) for the first 200 epochs. Then we train our model only with ConMix. We set synthesized representation number M 100 in all experiments, although applying diverse M on different datasets may lead to performance improvements. Following compared methods (Huang et al., 2023; Yu et al., 2023; Tao et al., 2020; Li et al., 2023), we use K-means for final representation clustering. Other specific training settings can be found in the Appendix A.
5.2
MAIN RESULTS
In this section, we evaluate the proposed ConMix with both baseline and state-of-the- art deep clus- tering methods on various benchmarks with different imbalance ratio, including 5 and 10. We com- pare ConMix with nine methods. SimCLR (Chen et al., 2020), MoCo (He et al., 2020), and BYOL (Grill et al., 2020) SDCLR (Jiang et al., 1) is long-tailed self-lcarning method specially proposed for long-tailed distributions. CC (Li et al., 2021b) a deep clustering method based on contrastive learning, and it outputs clustering assignment directly. IDFD (Tao et al., 2020), CoNR (Yu et al., 2023), ProPos (Huang et al., 2023) and DMICC (Li et al., 2023) improve better representations for deep clustering.
self-supervised learning methods which learn general representation.
We evaluate the effectiveness of long-tailes deep clustering using four distinct metrics, including accuracy (ACC), class-averaged accuracy (CAA), normalized mutual information (NMI), adjusted rand index (ARI). Aside from CAA, all are common metrics for evaluating clustering performance. Due to the fact that in long-tailed deep clustering, the same distribution used for training and test- ing, accurate head class prediction may lead to undeservedly high ACC. Thus, We introduce CAA (class-averaged accuracy), which is the average accuracy of each class, to compare the performance of different methods fairly. The results of comparison of our method with diverse approaches shown in Table and 2, and the best result are displayed in bold. In Table and 2, we can observe
Table 1: Clustering results (in percent %) of various methods on three benchmark datasets with imbalance ratio = 5.
Datasets
CIFAR-10 10
CIFAR-20
STL-10
Metric
ACC
CAA
NMI
ARI
ACC
CAA
NMI
ARI
ACC
CAA
NMI
ARI
SimCLR
41.4
46.1
40.5
24.6
39.7
38.7
40.4
23.8
28.6
30.3
23.9
11.8
MoCo
38.8
42.1
36.1
23.8
26.8
25.7
24.5
12.1
38.4
38.9
34.9
22.2
BYOL
51.8
52.5
52.2
34.1
35.9
34.7
36.4
21.5
39.2
40.7
33.6
22.9
SDCLR
44.1
50.5
43.4
38.0
40.2
38.7
40.3
23.8
35.8
37.4
34.4
19.3
CC
25.2
20.9
18.3
1.47
16.0
12.0
18.0
00.0
41.5
30.0
46.2
22.4
IDFD
56.7
63.0
51.8
37.1
31.2
30.7
30.5
16.2
41.6
41.2
39.6
24.3
CoNR
41.4
43.5
34.9
23.3
23.8
23.4
21.6
12.1
32.4
31.3
29.0
18.2
ProPos
51.4
59.2
52.2
34.1
40.7
39.3
42.6
26.4
37.2
39.3
34.5
21.8
DMICC
40.6
42.5
36.9
25.4
25.2
23.1
20.6
7.5
45.9
45.6
45.3
35.5
ConMix
61.6
65.4
59.8
45.6
42.8
41.0
43.9
27.7
48.9
49.7
48.8
35,9
that ConMix achieves comprchensively superior performance compared to other comparative meth- ods. General representation learning methods do not necessarily yield inferior results compared to

---

Published as a conference paper at ICLR 2025
Table 2: Clustering results (in percent %) of various methods on three benchmark datasets with
imbalance ratio 10.
Datasets
CIPAR-10
CIPAR-20
STL-10
Metric
ACC
CAA
NMI
ARI
ACC
CAA
NMI
ARI
ACC
CAA
NMI
ARI
SimCLR
39.4
42.5
38.4
23.7
34.4
33.7
36.9
19.8
27.7
28.1
22.5
11.5
MoCo
37.0
40.8
34.7
23.0
26.7
25.0
24.0
12.1
38.1
32.8
34.8
23.7
BYOL
46.0
45.8
51.8
36.6
36.4
34.7
38.4
21.9
37.3
35.4
33.8
24.1
SDCLR
38.9
44.3
42.5
26.5
37.8
35.9
39.6
22.9
34.6
37.9
32.2
17.3
CC
27.5
43.9
19.9
14.3
21.9
1.1
43.0
35.3
44.7
25.4
IDFD
47.5
54.9
48.4
33.1
28.7
27.2
28.6
15.1
38.6
34.7
36.8
22.8
CoNR
31.4
44.3
29.2
17.8
20.3
17.9
17.3
8.0
34.8
32.5
30.7
21.0
ProPos
46.1
49.3
52.5
34.2
36.8
33.6
40.1
22.5
35.6
38.2
37.2
23.9
DMICC
36,6
39.5
36.8
25.9
24.7
21.9
20.7
10.1
41.3
41.9
38.7
30.3
ConMix
53.3
58.2
57.1
40.8
41.7
39.3
43.6
27.0
47,4
48.7
48.2
33,9
deep clustering methods, as they merely focus on learning good individual- representations. Some deep clustering methods may collapse to very poor performance on long-tailed distributions, primarily because the majority of samples are predicted to belong few dominant classes (e.g.. CoNR and CC on CIFAR-20). Therefore, to apply deep clustering methods effectively to the natu- rally occurring tangttilld distributions, it is essential to adequately address long-tailed distributions. Moreover, some methods achieve high ACC but not so high ARI (c.g, IDFD on CIFAR-10) which is due to insensitivity class imbalance, leading some class being dispersed across multiple clusters or multiple classes being incorrectly grouped into a single cluster.
5.3
A ABLATION STUDY
We have conducted an extensive series of ablation studies on CIFAR-10 with imbalance ratio of 10 comprehend the underlying reasons for the experiments' effectiveness, shown in Table 3. We have conducted an experiment to employ input-level mixup (Zhang et al., 2018) under the SimCLR framework in unsupervised manner, with all settings consistent with ConMix. We can observe that in the context of long-tailed tailed unsupervised learning, mixup does not enhance the model gener- alization. In fact, it slightly diminishes the model performance. We think that the model is unable to learn certain original features from the mixed images, which results in decline in the model representation learning capability. Besides, we conducted unsupervised Manifold Mixup, which interpolates hidden features in the random hidden layer as described in (Verma et al., 2019). We have also conducted pairwise ConMix, where synthesized representations are generated by pairing samples instead of using multi-sample combinations. Note that input-level mixup, unsupervised Manifold Mixup and pairwise ConMix all sample mixing coefficients from beta distributions and utilize 200-epoch SDCLR warmup. Their experimental results show that both mixup only the representation level and multi-sampling strategy work effectively. We apply SDCLR to warmup ConMix for 200 epochs to ensure meaningful representations to mixup later. However, the periments suggest that the effect of this warmup is marginal and does not play decisive role. It can be inferred that mixup at representation level is more robust compared input level, because representation-levd mixup does not impede the model to learn good representations. Our attempt to substitute SDCLR with SimCLR for warmup also yields desirable results. Note that the method we adopted in experiments is ConMix w/ SDCLR warmup described in Section 5.1.2.
Table 3: Ablation study (in percent %) of various methods CIFAR-10 with imbalance ratio 10.
Metric
ACC
CAA
NMI
ARI
Input-level mixup
36.8
37.8
28.3
19.5
SimCLR
39.4
42.5
38.4
23.7
Unsupervised Manifold Mixup
49.4
54.3
54.4
38.2
Pairwise ConMix
50.7
56.1
56.8
39.8
ConMix w/o SDCLR warmup
50.6
56.1
55.8
39,6
ConMix w/ SimCLR warmup
51.3
56.4
56.4
39.8
ConMix w/ SDCLR warmup
53.3
58.2
57.1
40.8
8

---

Published as a conference paper at ICLR 2025
5.4 CLUSTERING MORE
SCENARIOS
We have also conducted experiments under more long-tailed conditions, where ConMix outperforms several approaches well. We compare ConMix with state-of-the-art deep clustering methods when the imbalance ratio reaches 100 on CIFAR-10. How- ever, have neither explored scenarios with even longer tails nor conducted experiments across multiple datasets at this imbalance ratio, primarily because there remains considerable room for improvement when the imbalance ratio is less severe. Table the clustering performance on CIFAR-10 with imbalance ratio = 100 is shown. Although the advantages of ConMix over other methods are marginal in terms ACC and CAA, still demonstrates significantly improved per- formance in NMI and ARI. This suggests that the distribution obtained by ConMix is closer to the true underlying distribution.
representation learning methods and
Table 4: Clustering results (in percent %) of various methods on CIFAR-10 with imbalance ratio 100.
Methods
SimCLR
SDCLR
BYOL
CoNR
ProPos
DMICC
ConMix
ACC
32.2
32.8
33.5
28.1
38.2
31.3
40.4
CAA
31.6
32.3
35.5
20.9
38.3
26.1
38.5
NMI
36.4
39.4
43.1
25.6
42.4
41.1
53.4
ARI
20.2
22.2
25.3
14.6
24.9
22.4
33.5
5.5 CLUSTERING ON THE LARGE-SCALE DATASET
Following (Huang et al., 2023; Yu et al., 2023), we have evaluated on Tiny ImageNet, which contains 200 classes. Since Tiny ImageNet imbalance ratio of 10, as described in Section 5.1.1. The results are shown in Table 5. It can be seen that our method still performs better on large-scale long-tailed datasets. Some deep clustering methods perform worse than the baseline methods. We believe this is because these methods assume class balance and perform better under balanced conditions, do not work as effectively under imbalanced conditions. This also demonstrates the necessity of studying long long-tailed deep clustering.
balanced dataset, created long-tailed version with an
Table 5: Clustering results (in percent %) various methods on Tiny ImageNet with imbalance
ratio = 10.
Methods
SimCLR
SDCLR
BYOL
CoNR
ProPos
DMICC
ConMix
ACC
16.0
14.6
16.1
8.8
15.0
9.2
16.5
CAA
14.2
12.9
14.0
8.2
13.2
8.8
14.5
NMI
33.9
35,2
34.2
28.2
34.4
28.7
34.9
ARI
7.9
8.6
8.6
3.9
7.5
3.7
8.3
5.6 CONMIX BENEFITS REPRESENTATION LEARNING
To further investigate the impact of ConMix on representation learning, we test the distributions of representations in each class for CIFAR- 10 when the imbalance ratio is 10. We gauge the compact- ness of spasssseccific representations by the average similarity within each class, and the separability from other classes by the average dissimilarity between representations within class and those of other classes. Ideally, former should be high, indicating a tight grouping of representations within the same class, while the latter should be low, suggesting clear distinction between representations of different classes. As shown in Figure 1, the proposed ConMix can definitely improve the sim- ilarity within the same class and dissimilarity between different classes, jointly indicating that the representations become more discriminative and the biased model has been corrected effectively.
5.7 CONMIX WORKS ON UNSEEN BALANCED DATASETS
Experiments above are conducted by training imbalanced datasets and testing the same datasets, which common practice in deep clustering. Although we employ the CAA to indi- cate the average clustering performance across different classes, the employed K-means tends to

---

Published as a conference paper at ICLR 2025
Class-wise Similarity Comparison
Class-wise Dissimilarity Comparison
SDCLR
Avenge Discimilarity
Avenige Similarity
ConMix
0.4
0.2
10.02
SDCLR
ConMix
Class Index
Class Index
Figure 1: Class-wise similarity and dissimilarity CIFAR-10 with imbalance ratio 10. The index of classes ranges from 0 to 9, and the number of samples is decreasing. Left: class-wise similarity indicates the average similarity between all samples within the same class, the higher the better. can be seen that ConMix can significantly improve this indicator, especially for tail class samples. Thus tail class samples are more compact in high-dimensional space. Right: Class-wise dissimilarity indicates the average similarity between samples within and outside the indexed class, the lower the better. It can be seen that ConMix can significantly lower this indicator, especially for head class samples. It can inferred that head class samples are more discriminative from tail class ones after ConMix.
form clusters based on the overall data distributions. To further examine the practical performance of these biased models trained on long-tailed distributions, we apply models to cluster on balanced datasets. Specifically, we train models CIFAR-10 with an imbalance ratio of 10 and then evaluate them using the test set of CIFAR-10 10 as the balanced benchmark. In test set of CIFAR- 10, each class has 1,000 samples, which not present in the train set. Given that the data used for testing is unfamiliar to the models, the clustering results indirectly reflect model generalization, as shown in Table 6. We also train ConMix balanced CIFAR-10 dataset, using an equivalent total number of samples as in the long-tailed CIFAR-10 10 (name as ConMix-B B in Table 6). The performance of this balanced model was likewise evaluated on a test set, revealing that the performance drop induced by the long-tailed distribution is approximately 5%, highlighting that ConMix is robust to imbalanced data distributions.
Table Clustering results on balanced test set by models trained on CIFAR-10 with imbalance ratio 10 balanced CIFAR-10 10 with equal number of samples.
Methods
SimCLR
SDCLR
BYOL
CoNR
ProPos
DMICC
ConMix
ConMix-B -B
ACC
53.8
54.6
63.2
36.1
55.1
40.7
67.9
72.3
NMI
46.4
47.2
51.9
27.7
53.9
33.2
59.5
62.3
ARI
36.0
36.4
41.8
16.5
41.5
20.0
52.1
56.3
CONCLUSION
In this paper, we introduce a novel method for long-tailed deep clustering, aiming at mitigating the bias caused by training models long-tailed distributions. The proposed method, ConMix, performs multi-sampling linear interpolation at the representation level, effectively extending mixup deep clustering and contrastive learning. This simple yet effective approach achieves remarkable performance, when compared to state-of-the- deep clustering methods on several benchmarks datasets. ConMix effectively enhances representation learning, yielding more compact within-class and more separated between-class representations, which are highly conducive to clustering. We also theoretically demonstrate that ConMix implicitly re-balances the loss between head and tail classes in long-tailed learning, thus avoid the common issue in contrastive learning where losses for deep
head classes tend to be larger and those for tail classes tend to be smaller. However, clustering still lags significantly in performance compared to both long-tailed supervised learning and balanced supervised learning. We posit that there is need for more research to focus on this largely unexplored realm, in order advance the development and application of deep clustering.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant U24A20322, and the Big Data Computing Center of Southeast University.
REFERENCES
Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Machine Learning and Knowledge Discovery in Databases, pages 770-785, 2017.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with Label-distributionawwees margin loss. Advances in Neural Information Processing Systems, 32, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9650-9660, October 2021.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In Proceedings of the IEEE International Conference on Computer Vision,
pages 5879-5887 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In European Conference on Computer Vision, pages 694-710. Springer, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15, pages 223. PMLR, 1 11-13 Apr 2011.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268-9277) 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
Scan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271 122212211-22844 2020.
Hui Han, Wen-Yuan Yuan Wang, and Sing-Huan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In Advances in Intelligent Computing, pages 878-887, 2005.
J. A. Hartigan and M. A. Wong. Algorithm 136: A k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):000-108, 1979.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21): 1284-1284, 2009.

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint arXiv:1911.05248. 2019.
Chengkai Hou, Jieyu Zhang, Haonan Wang, and Tianyi Zhou. Subelass-balandd contrastive learn- ing for long-tailed recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5395-5407, October 2023.
Zhizhong Huang, Jie Chen, Jumping Zhang, and Hongming Shan. Learning representation for clus- tering via prototype scattering and positive sampling. IEEE Transactions on Pattern Analysis and Machhne Intelligence, (5(6))7509--555) 2023.
Xu Ji, Joao Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference Computer Vision, pages 9865-9874, 2019.
Yuheng Jia, Xiaorui Peng, Ran Wang, and Min-Ling Zhang. Long-tailed partial label learning by head classifier and tail classifier cooperation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12857-12865, 2024.
Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pages 4927-4939. PMLR, 2021.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 33:21798 21809, 2020.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Fengfu Li, Hong Qiao, and Bo Zhang. Discriminatively boosted image clustering with fully convo- lutional auto-encoders. Pattern Recognition, 83:161-173, 2018.
Hongyu Li, Lefei Zhang, and Kehua Su. Dual mutual information constraints for discriminative clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8571-8579, 2023.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In International Conference on Learning Representation, 2021a.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8547- 8555, 2021b.
Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. Twin contrastive learning for online clustering. International Journal of Computer Vision, 23009)2205--221, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980- 2988, 2017.
12

---

Published as a conference paper at ICLR 2025
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations. 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, June 2019.
Alena Lukasová. Hierarchical agglomerative clustering procedure. Pattern Recognition, 11(5-6):
365-381, 1979.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2020.
Chuang NIu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions Image Processing, 31 31:7264-7278. 2022.
Sculki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for imbal- anced visual classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 735-744) 2021.
Zhihao Peng, Hui Liu, Yuheng Jia, and Junhui Hou. Attention-driven graph clustering network. In Proceedings of the 29th ACM International Conference on Multimedia, page 935-943, 2021.
Zhihao Peng, Hui Liu, Yuheng Jia, and Junhui Hou. Egrc-net: Embeddingginduceed graph refinement clustering network. IEEE Transactions Image Processing, 32:6457-6466. 2023.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.
Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, (61(666))33), 2009.
Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip Torr, and Ling Shao. You never cluster alone. Advances in Neural Information Processing Systems, 34:27734-2777 2021.
Jiang-Xin Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learn- ing? Advances in Neural Information Processing Systems, 36, 2023.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11662-11677 2020.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Loigetailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33:1513-1524, 2020.
Yaling Tao, Kentaro Takagi, and Kouta Nakata. Clustering-friendly representation learning via instance discrimination and feature decorrelation. In International Conference on Learning Rep- resentations, 2020.
Junjiao Tian, Yen- Ben-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re- calibration for imbalanced datasets. Advances in Neural Information Processing Systems, 33: 8101-811 13, 2020.
Kai Tian, Shuigeng Zhou, and Jihong Guan. Deepcluster: A general clustering framework based deep learning. In Machine Learning and Knowledge Discovery in Databases, pages 809-825, 2017.
A F M Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung- Ho Bae. Saliencymix: saliency guided data augmentation strategy for better regularization. In International Conference Learning Representations, 2021.
13

---

Published as a conference paper at ICLR 2025
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis. Mare Proesmans, and Lue Van Gool. Scan: Learning to classify images without labels. In European Conference on Com- puter Vision, pages 268 285. Springer, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 4438-6447, PMLR, 2019.
Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long- tailed distribution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8659-8668, 8668, 2021.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pages 478-487. PMLR, 2016.
Mingyuan Xu, Haiyun Guo, Yuheng Jia, Zhitao Dai, and Jinqiao Wang. Pseudo label rectification with joint camera shift adaptation and outlier progressive recycling for unsupervised person re- identification. IEEE Transactions on Intelligent Transportation Systems, 24(3):3395-3406, 2023.
Bo Yang. Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning, pages 3861-3870. PMLR, 2017.
Huaxiu Yao, Yiping Wang. Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35:3361-3376, 2022.
Chunlin Yu. Ye Shi, and Jingya Wang. Contextually affinitive neighborhood refinery for deep clus- tering. Advances in Neural Information Processing Systems, 36, 2023.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, October 2019.
Chuyu Zhang, Hui Ren, and Xuming He. p OT: Progressive partial optimal transport for deep imbalanced clustering. International Conference on Learning Representations, 2024.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representa- tions, 2021.
Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang, Xiaokang Yang, and Pinyan Lu. M-mix: Generating hard negatives via multi-sample mixing for contrastive learning- In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2461-2470, 2022.
Yifan Zhang, Bingyi Kang. Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16489-16499) 2021.
-ailed recogni-
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9719-9728. 2020.
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference Machine Learning pages 27367-27377 PMLR, 2022.
14

---

Published as a conference paper at ICLR 2025
A TRAINING SPECIFICS
Here we provide the training specifics that are omitted in the main text due space limitations. Following (Chen et al., 2020), we set the first convolution layer with kernel size 3x3 and stride 1, and remove the first max-pooling layer for all experiments on CIFAR-10 and CIFAR-20 due their small image sizes. For ConMix, we adopt the stochastic gradient descent (SGD) optimizer, whose learning rate is 0.5, weight decay 0.0001 and momentum is 0.9, We adopt the cosine decay learning rate schedule to update the learning rate by step, with 10 epochs for learning rate warmup. The temperature NT-Xent is 0.2. We adopt the data augmentation methods described in (Chen et al., 2020).
To avoid the uncertainty caused by K-means random initialization, all methods evaluated by K- means will be averaged over 10 trials with different random seeds. Regarding the parameters set- tings for other comparative methods, we directly adopt the configurations as specified within their respective papers.
Our experiments are based on Pytorch and models are trained on NVIDIA GeForce RTX 4090 GPUs. It takes approximately five to seven hours single GPU to train model under different long-tailed distributions in the Section 5.2, with the actual time depending on factors such as the size of the dataset.
B MULTI-SAMPLE STRATEGY BENEFITS CONMIX
values ditferent metrics (%)
50
100
50
200
500
40
ACC
CAA
NMI
ARI
Figure 2: Clustering Results (in percent %) of Different Tag Numbers on CIFAR-10 when Imbalance Ratio 10.
We demonstrate through ablation studies that the pairwise ConMix remains effective, albeit less so than multi-sample methods. Herein, we further investigate the impact of the tag number M on clustering performance. In our experiments, we employ the long-taie distribution using CIFAR-IO, with an imbalance ratio of 10. We set the tag number M to 50, 100, 200, and 500 respectively. The results indicate that the clustering performance not significantly influenced by the tag number, unless the multi-sample mixup strategy itself is severely hindered. Due to the batch size of 512, when M set to 500, the multi-sample approach diminishes considerably, leading to a drop in experimental performance.
C DERIVATIONS AND PROOFS OF THE FORMULAS
Due to space limitations, we have omitted the derivation and proof of the formula the main text. In this section, we will provide detailed derivations and proofs of formulas that appear in the paper.
15

---

Published as a conference paper at ICLR 2025
C.I DERIVATION FOR fCL
The LCr. in Eq. (1) can be rewritten into a form of fcr. in Eq. (5):
1
2N
(0s, DJ ,T)
1
2N
2k=1 k= 1 1R (U,, Uk. T)
2N
Ler.
log
log
2N
E [iAA (Vi, 7):
2N
Uii 1, T)
:=1
1
2N
2N
8 (Ui, Uk:
2N
2N
exp(e, I Uk/TT
log
[egfil
log
[
2N
(0), 07 .T)
2N
exp(v, vi /T)
k=1
1=1
1
(9)
Σ
log
Σ
[x=sexx(11) (wk //T)/T)
2N
k=1
2N
2N
1
log
[x=x:)epp(fcL(( Uk 2(²)/
2N
1=1
where (u,, Uk. wx) = v. (vx: - wt))
C.2 DERIVATION FOR hcM
To measure the contribution of the individual vs to the total loss LCM, we propose a new function hCM (v:):
2M
2M
hCM (v:) =
Σ
SCM
2z)
Σ
(2, 2n, z)t
y=1
y=1
2
Uk
U
Upy
U
y=1
jEU,
kEU,
jEUL
2:M
Σ
Σ vf)
Uk
y=1
JEU, (i)
KEU,
jeuł {
(10)
Σ
0. Uk - 2M
0 Uk
US!
keSt
2N
Ui Uk Σ
U. Uk 2M Σ u. 0k
U. Uk
1=1
KEB
kEBt
2N
U.
Σ KES.
0. Uz (2M +1)
Σ 1. Uk:
1=1
REST
where Uz, U, and Ut denotes the set contains indexes of representations synthesizing =2, Ry and = respectively. |-| denotes the number of elements the set. zz, is synthesized from all representations composed of 2. except U. S, and St are the sets contain indexes of original representations from =e and _ respectively, as v, synthesizes 2e and di synthesizes x := denotes the assignment operator. To simplify the problem, Eq. (6), we do not take into account the weighted average coefficients needed for each similarity. These changes do not affect the measurement of the impact of individual samples the overall loss, as they are random factors in the experiments. Instead, these changes make the theory more straightforward explain.
have excluded the effects of normalization. To align with
C.3 PROOF FOR THEOREM 1
To prove that (CM(()) hCM((;)) < (cr.(vn) (v,) for head class representation U, and tail class representation Us, first define some constants: No.. is the number of representations
16

---

Published as a conference paper at ICLR 2025
that belongs to the same class the head class representation vy; Mu is the number of repre- sentations that are not of the same class as the head class representation thi No., is the number of representations that belongs to the same class the tail class representation Uzi NNt.u is the number of representations that are not of the same class as the tail class representation et. Given intra-class similarity Sw and inter-class similarity Sb, compute the outcome of (ic((1)) hcr. (1,) as below:
(CL( (v)(
2N
2N
1 (KAN((1) ox - U/ -, 0)))
Σ
[k (vg Up -0, v,²)
k=1
2N
2N
(0% 1xe - 1/2 1% 11)
(1) Uk ve uz²) - (11 27 - U/ 1/) + (m, Us
T.
nz vt))
k=1
k=1
2N
(0% Uk -1% 0/
(0) U2 v. u[) (w) + (1 kwm
(11)
2N
2N
Σ (o) Uk - 2/ 2
Σ
( Up - v. v,
-Nin Sw + Nhoue 86 - 2N (Nein Sw + Noov 86 2N
=N Sw + Nhaue 56
Ne Sw Nioue 3b
=2Ns Bw
Nnoue Sw
Nhon Sb - sw + Noone Sw - Noons Sb
=(Ntaut Nhont )8w
(Noous Nhons) )86
= (Naaue Nhoue (8 8b)-
The hCM((N) hCM( can be derived in similar manner below:
hcM (wn) hcm(
2N
2N
Un Uk
Σ
0 0k - (2M Σ
vn Us -
1 Uk Σ
1t Tuk
- (2M + +1)
Σ
u vk)
k=1
kEat
1=
kES.
KES,
2N
1/ Uk Un Un 2/2 2/ (2M +1) Σ
1/ Uk
2n Uk
kEat
KE
T
Σ kest
Vr Tu: + 0. Uz v. 0, - (2M + 1) Σ 0._ vs)
1. Uk
1=1
kE
2N
2N
U% Ub
Σ
2/l Uk (2M + 11) Σ
2 Uk
U Uk Σ
1t Tup - (2M + +1)
Σ
2a vv)
MEST
KEB
k=
KEST
REST
2N
UA Uk
(2M 2)
Σ Up Uk
0 Up (2M +2)
Σ 0 vJTv))
KESt
kESt
2N
2N
UA Uk
T
V.
T
Uk (2M + 2)
Σ
Ut Us + (2M +2)
Σ kest
1.
Uk.
k=1
k=1
KER
(12)
Note that in the above Eq. (12), we have made equivalent substitutions: Exes, io U with ΣkEst U/ and EkES, v. un with ΣkEst v. vz. And from Eq. (11), we can derive that 2NN u. U2 (Nem. Nomn (ow sb). So further connect (CM
MZN 1i Uk
17

---

Published as a conference paper at ICLR 2025
hcm((r) and (CL( hot (2r) as below:
hcM(vn) hcM (vt)
=(Noou Noou (sw 9b) (2M +2
Σ
vn. Uh + (2M + +
Σ
vi Uk
KE
keRt
(13)
(wC)( - (w) + (2M + 2)
0 Uk (2M + 2)
Σ
1/ Uh.
kESt
KE
As the number of original representations composing each synthesized representation is roughly equal, and their class distribution aligns with the overall class distribution in the assumption, S usually has more representations within the same class as v, than S, has within the same class as vy- Thus, (2M + 2) EkeS, v, (2M + 2) ENER, 2% Us O. Along with Eq. (13), we can conclude that
(v.) h)M(vr) < hCL( hCL(
(14)
Then the Theorem 1 is proven and ConMix is proven theoretically to relatively reduce the loss of head class samples and increase the loss of tail class samples, thus achieving loss re-balance.
D CLUSTERING RESULTS ON BALANCED DATASETS
We further trained and tested the performance of different methods on balanced datasets. The dataset configurations and experimental setups refer (Huang al., 2023). Specifically, we trained differ- ent methods using ResNet-11 18 for 1000 epochs on CIFAR-10 and CIFAR-20. Since ConMix does not leverage some advanced techniques suitable for balanced clustering, we propose an updated version of ConMix called "ConMix+ train the model with the loss of ConMix for 500 epochs, then train it with Propos for another 500 epochs. The total number of training epochs for this updated version is the same other methods.
that embeds ConMix into Propos (Huang et al., 2023). We first
We have reported the results of the balanced dataset in Table 7. We have also provided the clustering performance on the datasets with imbalance ratio of 10 for better comparisons.
Table 7: Clustering results (in percent %) of various methods on balanced datasets and imbalance datasets with an imbalance ratio 10.
Datasets
CIFAR-I 10
CIFAR
Data Type
Balanced
Imbalanced
Balanced
Imbalanced
Metric
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NM1
ARI
ACC
NMI
ARI
SimCLR
72.8
63.9
56.7
39.4
38.4
23.7
45.4
43.8
28.8
34.4
36.9
19.8
SDCLR
71.4
62.4
54.8
38.9
42.5
26.5
44.6
43.3
27.6
37.8
39.6
22.9
CC
79.0
70.5
40.6
43.9
42.9
43.1
26.6
19.9
21.9
1.1
IDFD
81.5
71.1
66.3
48.4
33.1
42.5
42.6
26.4
28.6
15.1
Propos
91.6
85.1
83.5
52.5
34.2
$8.2
42.3
40.1
22.5
ConMix
80.9
70.7
65.6
53.3
57.1
40.8
46.0
45.5
29.8
41.7
43.6
27.0
ConMix+Propos
92.0
85.3
83.3
53.8
58.8
42.8
59.2
58.4
42.5
43.8
47.4
30.3
Compared to the baseline methods (SimCLR, SDCLR), ConMix demonstrates significant perfor- mance improvements on both the balanced datasets and the long-tailed datasets.
Compared to recent deep clustering methods (CC, IDFD, Propos), ConMix shows improvements on long-tailed datasets but may not perform as effective as Propos on balanced datasets. Note that, ConMix still outperforms CC and IDFD most cases even on the balanced datasets.
However, the updated version "ConMix+Propoo performs the best on both balanced datasets and long-tailed datasets, showing that adding some recent deep clustering techniques "ConMix" will further improve its performance and robustness.
Moreover, we also notice that existing deep clustering algorithms perform well on balanced datasets but suffer severe performance degradation on long-tailes datasets. We believe this due to these methods making assumptions that are aligned with balanced datasets. While they can achieve good laboratory performance, they are less suitable for realistic long-tailed distributions. This further underscores the importance of research long-tailed deep clustering.
18

---

Published as a conference paper at ICLR 2025
E CLUSTERING RESULTS OF DIFFERENT CLUSTERING TECHNIQUES
Our primary consideration for using K-means as the clustering method is that most of our compared methods utilize K-means (except for Contrastive Clustering which directly outputs the clustering assignments). So the use K-means allows us to achieve a fair comparison.
However, we have investigated the impact of different clustering methods the performance of ConMix. tried the Gaussian Mixture Model (GMM) (Reynolds et al., 2009) on the learned embedding of ConMix to obtain cluster assignments. We use two methods initialize GMM: one to initialize using K-means, and the other is random initialization. We denote these two cases as GMM-K and GMM-r. Besides, tested the performance of agglomerative clustering (Lukasová, 1979) on ConMix (we refer to the method briefly as AC). The detailed results are in Table 8
Table 8: Clustering results (in percent %) of different clustering techniques imbalance datasets with an imbalance ratio = 10.
Datasets
CIFAR-I0
CIFAR-20
STL.-10
Metric
ACC
CAA
NMI
ARI
ACC
CAA
NMI
ARI
ACC
CAA
NMI
ARI
K-means
53.3
58.2
57.1
40.8
41.7
39.3
43.6
27.0
47.4
48.4
48.2
33.9
GMM-k
63.6
48.1
59.1
53.1
37.2
35.4
42.7
23.7
50.6
40.7
47.3
34.1
GMM-r
50.3
49.0
58.7
44.7
31.3
22.2
34.9
18.1
45.9
46.0
46.2
31.5
AC
59.8
62,2
58.8
46.5
39.0
37.0
42.0
20.0
47.4
51.4
49.7
33.5
Compared with these methods, formance, validating the effectiveness of our method ConMix. However, for different datasets, the methods achieving the best performance may vary. For example, CIFAR-IO, GMM-K achieved the highest ACC, NMI and ARI, while AC achieved the highest CAA. On CIFAR-20, K-means per- formed the best. On 10, GMM-K obtained the best results in terms of ACC and ARI, while AC achieved the highest CAA and NMI.
find that different clustering methods can all lead good per-
F CLUSTERING RESULTS ON IMAGENET-LT
To further validate the effectiveness and generalization capability of ConMix, we conducted ex- periments on the long-tailed dataset ImageNet-LT LT (Liu et al., 2019). It is the long-tailed subset of ImageNet- 1K (Deng ct al., 2009), consisting of 15.8K images spanning 1,000 classes, with sample number ranging from 1280 to 5. Following (Li et al., 2021a), we trained ResNet-50 for 200 epochs and reported the results of the last epoch. We compare ConMix with three baseline methods (Sim- CLR, SDCLR, BYOL) and three recent superior deep clustering methods (IDFD, CoNR, DMICC). The results are the Table 9, The experimental results demonstrate that our method still performs
Table 9; Clustering results (in percent %) of various methods on ImageNet-LT.
SimCLR
SDCLR
BYOL
IDFD
CoNR
DMICC
ConMix
ACC
14.7
13.7
14.8
4.42
6.38
5.24
15.4
CAA
11.3
10.5
10.9
5.58
6.85
5.94
12.2
NMI
51.4
50.3
50.6
35.6
38.9
37.9
51.6
ART
9.14
9.30
10.2
1.17
2.19
1.83
11.4
well on ImageNet-LT, proving its effectiveness and generalization capability. At the same time, may be surprising that recent state-of-the-art methods perform worse than the baseline methods. The reason that they make assumption that data are balanced distributed, which conflicts with the real distribution of the data. This phenomenon indicates the limitations of current deep clustering methods in handling long-tailed data and highlights the urgent need for research into long-tailed deep clustering.
G EMPIRICAL ANALYSIS ON THE IMPACT OF CONMIX
We conducted experiments on CIFAR-10 with imbalance ratios (IR) of 1, 2, 5, 10. 20, 50, and 100. And provide the empirical analysis from the perspective of the compactness of the learned
19

---

Published as a conference paper at ICLR 2025
representations for different classes. Specifically, train the baseline SimCLR and ConMix CIFAR-10 under different imbalance ratios and calculate the class-wise similarity for each class. Class-wise similarity denotes the average similarity of samples within the same class and can indi- cate the compactness of representations of different classes in the feature space. Thus, to facilitate the description, will refer this metric "compactness" below. We categorize the 10 classes of CIFAR-I0 into Many, Medium, and Few categories based on the number of samples, following 3:4:3 ratio. We first calculate the compactness for each class, then compute the average compactness for the three categories.
Moreover, we propose new metric, denoted F/My, which represents the ratio of the Few category compactness to the Many's ss to measure the balance between head classes (Many) and tail classes (Few). The larger F/My is, the greater the discrepancy compactness between head classes and tail classes is, indicating more severe impact of the long-tailed distribution. The results are shown in the Table 10.
Table 10: The comparison of compactness and F/My on CIFAR- 10 with different imbalance ratios.
Method
Many
Medium
Few
F/My
SimCLR
0.08
0.08
0.11
1.39
IR=1
ConMix 0.16
0.16
0.20
1.25
SimCLR
0.05
0.08
0.11
2.25
IR=2
ConMix
0.13
0.16
0.26
1.98
SimCLR
0.03
0.08
0.16
4.74
IR=5
ConMix
0.10
0.17
0.35
3.36
SimCLR
0.03
0.08
0.22
7.74
IR=10
ConMix
0.08
0.18
0.43
4.85
SimCLR 0.02
0.09
0.26
10.36
IR=20
ConMix
0.08
0.20
0,46
5.69
SimCLR
0.02
0.12
0.27
13.11
IR=50
ConMix
0.12
0.27
0.36
5.08
SimCLR
0.02
0.13
0.25
12.55
IR=100
ConMix
0.07
0.26
0.30
4.22
From the table, we can derive the following empirical analysis:
(1) The head classes (Many) typically have smaller compactness values, while tail classes (Few) have larger compactness values. This suggests that due to long-tailed effect, head classes occupy more of the feature space than tail classes.
(2) When the distribution is long-tailed, the compactness for Few category is relatively higher, while the compactness for Many category is relatively lower, leading larger F/My ratio. This is negative effect of the long-tailed distribution. However, ConMix can reduce the F/My ratio com- pared SimCLR under different imbalance ratios, indicating its ability mitigate the impact the long-tailed distributions.
(3) Regardless whether the classes are Few, Medium, or Many, ConMix improves compactness across different imbalance ratios. This indicates that samples within the same class become more compact, which is beneficial for clustering.
(4) When the imbalance ratio is 1, the dataset is balanced and differences in compactness across different classes are due varying difficulty of learning each class.
The above analysis empirically demonstrates that ConMix benefits long-tailed clustering.
20


Reviews:
[{"review_id": "biYQkFlyY1", "reviewer": "Reviewer_z1DH", "summary": "This paper proposes a new method called ConMix for dealing with the long-tailed problem of deep clustering. A major challenge in long-tailed deep clustering is how to deal with class imbalance in a dataset without label information. ConMix solves this problem through an innovative approach to mixed representations in contrastive learning to enhance deep clustering performance in the case of long-tailed distributions.", "questions": "1. The result of pairwise ConMix shown in Table3 on CIFAR-10 is better than the result of ConMix with M=500 presented in Figure 2.Is this reasonable? In my understanding, the former is equivalent to ConMix with a larger M on CIFAR-10.\n2. Have you conducted additional experiments on balanced models of other methods as ConMix-B to support the opinion about robustness?\nQuestion3. Are there other clustering methods being studied besides k-means?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe author has conducted comprehensive experiments, compared with multiple clustering algorithms, and prove the effectiveness of ConMix under long-tailed distribution.\n2.\tReasonable theoretical analysis is given to verify that ConMix can implicitly achieve the loss-balance.\n3.\tContributions of different elements in ConMix are studied through extensive experiments.", "weaknesses": "The representation synthesis part is supposed to be represented more intuitively, which may be a little confusing at first reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IeUhpwSPdp", "reviewer": "Reviewer_EuZq", "summary": "The paper presents a novel method, ConMix, aimed at addressing the challenges of long-tailed distributions in deep clustering. The authors argue that existing deep clustering approaches typically assume balanced class distributions, which is not the case in many real-world datasets. ConMix leverages a contrastive mixup strategy to enhance representation learning, theoretically proving its effectiveness in rebalancing class losses without the need for label information. The method is evaluated on benchmark datasets, demonstrating superior performance over existing state-of-the-art approaches.", "questions": "Conduct additional experiments on large and complex datasets like ImageNet to validate the effectiveness and generalization capability of ConMix.\n\nEnhance the discussion on method interpretability by providing more empirical analysis regarding its impacts.\n\nProvide detailed descriptions of the experimental setup and hyperparameter selections to improve transparency and reproducibility of the research.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The introduction of ConMix as a contrastive mixup method specifically designed for long-tailed deep clustering is a notable contribution to the field. The approach is innovative, extending mixup techniques into the realm of unsupervised learning.\n\nThe authors provide a theoretical foundation for their method, demonstrating how it can implicitly balance losses across head and tail classes. This theoretical insight is valuable and adds depth to the paper.\n\nThe evaluations on various benchmark datasets and the assertion of outperforming existing methods lend credibility to the proposed approach. The performance metrics presented seem robust.", "weaknesses": "Diversity of Datasets: The experiments are limited to a few benchmark datasets, lacking validation of the method’s effectiveness on more complex and diverse datasets. It is recommended to conduct experiments on larger image classification datasets such as ImageNet to thoroughly evaluate the model’s generalization ability and practicality.\n\nInterpretability of the Method: Although theoretical proofs are provided, the interpretability of how ConMix specifically affects the model learning process remains insufficient. Consider adding comparative experiments to illustrate the specific impacts of ConMix under varying conditions (e.g., different long-tail ratios) to enhance the depth of the paper.\n\nDetails of Experimental Setup: The experimental section lacks detailed descriptions of hyperparameter choices and training specifics, which could affect the reproducibility of results. It is suggested to include these details in the methodology section to assist other researchers in understanding and replicating the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6eII6aCEj", "reviewer": "Reviewer_RUUY", "summary": "The paper proposes to leverage mixup to improve deep clustering approaches for imbalanced datasets. In particular, a multi-sample mixup is incorporated into the SimCLR loss. Instead of just contrasting two augmentations of the same sample, a random subset of samples are selected and the mean representations of their two augmentations are contrasted. A theoretical analysis is performed that shows, under a certain set of simplifications, that this procedure increases the loss of the underrepresented classes. Further, empirical evaluation demonstrate that the scheme can outperform alternative approaches in the imbalanced setting.", "questions": "Could the authors elaborate on the statement in Line 220: “…. Equivalent to implicitly sampling different weights from the beta distribution”. Do the authors refer to the mixing coefficient in the original mixup formulation? While the reviewer understands that the cardinality of the set U_m follows a beta distribution, the final mixup representation will be the mean representations of these samples, which is different from the mixing coefficients. \n\nFurther, could the authors comment on the performance of the proposed approach in the balanced setting and on the results in Table 3 with regards to the pairwise mixup and the variability of the results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Limited work has been done on considering imbalance in deep clustering, with most approaches adopting a balanced assumption, and approaches addressing this shortcomings are thus of significance to the community.\n\nWhile mixup on a representations has previously been integrated into contrastive learning (also mixing multiple samples), there is a certain novelty of leveraging this in the clustering setting to address class imbalance, which is further supported by the theoretical analysis.\n\nThe proposed approach is simple and appears to be effective in the settings considered in this work.", "weaknesses": "While the author’s main focus is on the imbalanced setting, it would be beneficial to also include comparisons in a balanced setting to be able to judge the overall ability of the method.\n\nThe overall clarity of Section 3.3. can be improved. How are the “stochastically assigned tags” selected? Does each sample have a certain probability of being included (independent of each other)? If that is the case, what is the probability set to? Also, in line 215, the notation of the cardinality of the set is not aligned with Eq. 3.\n\nThe comparison to pairwise ConMix (standard manifold mixup) in Table 3 is not clear. It appears that the pairwise mixup obtains equivalent results to ConMix w/o SDCLR warmup and it is unclear if pairwise ConMix leverages SDCLR warmup here. Also, is this pairwise ConMix directly leveraging the mean representation of the pair or do the authors create a convex combination with weights sampled from a beta distribution? \n\nAs deep clustering methods tend to be a bit less stable than supervised models, some measures of variability/statistical significance would be beneficial in Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)