ConMix: Contrastive Mixup at Representation Level for Long-tailed Deep Clustering

Authors: Zhixin Li, Yuheng Jia
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: deep clustering, long-tailed deep clustering, unsupervised learning

Abstract:
Deep clustering has made remarkable progress in recent years. However, most existing deep clustering methods assume that distributions of different clusters are balanced or roughly balanced, which are not consistent with the common long-tailed distributions in reality. In nature, the datasets often follow long-tailed distributions, leading to biased models being trained with significant performance drop. Despite the widespread proposal of many long-tailed learning approaches with supervision information, research on long-tailed deep clustering remains almost uncharted. Unaware of the data distribution and sample labels, long-tailed deep clustering is highly challenging. To tackle this problem, we propose a novel contrastive mixup method for long-tailed deep clustering, named ConMix. The proposed method makes innovations to mixup representations in contrastive learning to enhance deep clustering in long-tailed scenarios. Neural networks trained with ConMix can learn more discriminative representations, thus achieve better long-tailed deep clustering performance. We theoretically prove that ConMix works through re-balancing loss for classes with different long-tailed degree. We evaluate our method on widely used benchmark datasets with different imbalance ratios, suggesting it outperforms many state-of-the-art deep clustering approaches. The code is available at https://github.com/LZX-001/ConMix.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730573850346

thread_type:
official_review

<thread_object>
creator: reviewer_ruuy

<content>
summary:
The paper proposes to leverage mixup to improve deep clustering approaches for imbalanced datasets. In particular, a multi-sample mixup is incorporated into the SimCLR loss. Instead of just contrasting two augmentations of the same sample, a random subset of samples are selected and the mean representations of their two augmentations are contrasted. A theoretical analysis is performed that shows, under a certain set of simplifications, that this procedure increases the loss of the underrepresented classes. Further, empirical evaluation demonstrate that the scheme can outperform alternative approaches in the imbalanced setting.

strengths:
Limited work has been done on considering imbalance in deep clustering, with most approaches adopting a balanced assumption, and approaches addressing this shortcomings are thus of significance to the community.

While mixup on a representations has previously been integrated into contrastive learning (also mixing multiple samples), there is a certain novelty of leveraging this in the clustering setting to address class imbalance, which is further supported by the theoretical analysis.

The proposed approach is simple and appears to be effective in the settings considered in this work.

weaknesses:
While the author’s main focus is on the imbalanced setting, it would be beneficial to also include comparisons in a balanced setting to be able to judge the overall ability of the method.

The overall clarity of Section 3.3. can be improved. How are the “stochastically assigned tags” selected? Does each sample have a certain probability of being included (independent of each other)? If that is the case, what is the probability set to? Also, in line 215, the notation of the cardinality of the set is not aligned with Eq. 3.

The comparison to pairwise ConMix (standard manifold mixup) in Table 3 is not clear. It appears that the pairwise mixup obtains equivalent results to ConMix w/o SDCLR warmup and it is unclear if pairwise ConMix leverages SDCLR warmup here. Also, is this pairwise ConMix directly leveraging the mean representation of the pair or do the authors create a convex combination with weights sampled from a beta distribution? 

As deep clustering methods tend to be a bit less stable than supervised models, some measures of variability/statistical significance would be beneficial in Table 3.

questions:
Could the authors elaborate on the statement in Line 220: “…. Equivalent to implicitly sampling different weights from the beta distribution”. Do the authors refer to the mixing coefficient in the original mixup formulation? While the reviewer understands that the cardinality of the set U_m follows a beta distribution, the final mixup representation will be the mean representations of these samples, which is different from the mixing coefficients. 

Further, could the authors comment on the performance of the proposed approach in the balanced setting and on the results in Table 3 with regards to the pairwise mixup and the variability of the results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714430503

thread_type:
official_review

<thread_object>
creator: reviewer_euzq

<content>
summary:
The paper presents a novel method, ConMix, aimed at addressing the challenges of long-tailed distributions in deep clustering. The authors argue that existing deep clustering approaches typically assume balanced class distributions, which is not the case in many real-world datasets. ConMix leverages a contrastive mixup strategy to enhance representation learning, theoretically proving its effectiveness in rebalancing class losses without the need for label information. The method is evaluated on benchmark datasets, demonstrating superior performance over existing state-of-the-art approaches.

strengths:
The introduction of ConMix as a contrastive mixup method specifically designed for long-tailed deep clustering is a notable contribution to the field. The approach is innovative, extending mixup techniques into the realm of unsupervised learning.

The authors provide a theoretical foundation for their method, demonstrating how it can implicitly balance losses across head and tail classes. This theoretical insight is valuable and adds depth to the paper.

The evaluations on various benchmark datasets and the assertion of outperforming existing methods lend credibility to the proposed approach. The performance metrics presented seem robust.

weaknesses:
Diversity of Datasets: The experiments are limited to a few benchmark datasets, lacking validation of the method’s effectiveness on more complex and diverse datasets. It is recommended to conduct experiments on larger image classification datasets such as ImageNet to thoroughly evaluate the model’s generalization ability and practicality.

Interpretability of the Method: Although theoretical proofs are provided, the interpretability of how ConMix specifically affects the model learning process remains insufficient. Consider adding comparative experiments to illustrate the specific impacts of ConMix under varying conditions (e.g., different long-tail ratios) to enhance the depth of the paper.

Details of Experimental Setup: The experimental section lacks detailed descriptions of hyperparameter choices and training specifics, which could affect the reproducibility of results. It is suggested to include these details in the methodology section to assist other researchers in understanding and replicating the experiments.

questions:
Conduct additional experiments on large and complex datasets like ImageNet to validate the effectiveness and generalization capability of ConMix.

Enhance the discussion on method interpretability by providing more empirical analysis regarding its impacts.

Provide detailed descriptions of the experimental setup and hyperparameter selections to improve transparency and reproducibility of the research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730811326904

thread_type:
official_review

<thread_object>
creator: reviewer_z1dh

<content>
summary:
This paper proposes a new method called ConMix for dealing with the long-tailed problem of deep clustering. A major challenge in long-tailed deep clustering is how to deal with class imbalance in a dataset without label information. ConMix solves this problem through an innovative approach to mixed representations in contrastive learning to enhance deep clustering performance in the case of long-tailed distributions.

strengths:
1.	The author has conducted comprehensive experiments, compared with multiple clustering algorithms, and prove the effectiveness of ConMix under long-tailed distribution.
2.	Reasonable theoretical analysis is given to verify that ConMix can implicitly achieve the loss-balance.
3.	Contributions of different elements in ConMix are studied through extensive experiments.

weaknesses:
The representation synthesis part is supposed to be represented more intuitively, which may be a little confusing at first reading.

questions:
1. The result of pairwise ConMix shown in Table3 on CIFAR-10 is better than the result of ConMix with M=500 presented in Figure 2.Is this reasonable? In my understanding, the former is equivalent to ConMix with a larger M on CIFAR-10.
2. Have you conducted additional experiments on balanced models of other methods as ConMix-B to support the opinion about robustness?
Question3. Are there other clustering methods being studied besides k-means?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733285095779

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Our Responses

comment:
We thank all the reviewers (**z1DH**, **EuZq**, **RUUY**) for their efforts in improving this work.

Overall, the reviewers acknowledged the following strengths of our work:

(1) A notable contribution for a method specially designed for long-tailed deep clustering, 
a field that has received little attention in previous research. (**EuZq**, **RUUY**)

(2) The innovation of extending mixup to unsupervised learning and long-tailed learning. (**EuZq**, **RUUY**)

(3) Reasonable theoretical analysis which adds depth to the paper. (**z1DH**, **EuZq**, **RUUY**)
 
(4) Comprehensive experiments which prove the effectiveness of ConMix. (**z1DH**, **EuZq**, **RUUY**)

Meanwhile, following the reviewers' suggestions, our main revisions are as follows:

(1) We have clarified content in the paper that might cause misunderstandings and revised the manuscript accordingly
to enhance its readability. (**z1DH**, **RUUY**)

(2) We have added results on balanced datasets to prove the robustness of ConMix. (**z1DH**, **RUUY**)

(3) We investigated the performance of clustering techniques besides K-means. (**z1DH**)

(4) We have added results on ImageNet to further demonstrate the effectiveness and generalization capability 
of ConMix. (**EuZq**)

(5) We conducted experiments on datasets with varying imbalance ratios and provide empirical analysis for 
understanding the actual impact of ConMix on the learning process. (**EuZq**)

We believe our extensive experiments and detailed responses have addressed the reviewers' concerns.

Above is the summary of our responses. Thank you for your reading!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a deep clustering method for imbalanced data. Such a long-tail problem is less explored in clustering research but is common in real-world applications. The proposed method is simple yet effective, with detailed theoretical analyses provided by the authors. As all three reviewers gave positive scores, I decided to accept this paper. The authors should further improve the clarity of the manuscript in its camera-ready version to help readers interpret and reproduce the proposed method.