Title: Diversity-oriented Deep Multi-modal Clustering
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-modal clustering

diversity learning

contrastive learning
Abstract: Deep multi-modal clustering (DMC) aims to explore the correlated information from different modalities to improve the clustering performance. Most existing DMCs attempt to investigate the consistency or/and complementarity information by fusing all modalities, but this will lead to the following challenges: 1) Information conflicts between modalities emerge. 2) Information-rich modalities may be weakened. To address the above challenges, we propose a diversity-oriented deep multi-modal clustering (DDMC) method, where the core is dominant modality enhancement instead of multi-modal fusion. Specifically, we select the modality with the highest average silhouette coefficient as the dominant modality, then learn the diversity information between the dominant madality and the remaining ones with diversity learning, and finally enhance the dominant modality for clustering. Extensive experiments show the superiority of the proposed method over several compared DMC methods. To our knowledge, this is the first work to perform multi-modal clustering by enhancing the dominant modality instead of fusion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the two challenges in most existing deep multi-modal clustering methods, the authors design a novel diversity-oriented multi-modal clustering method whose main idea is to enhance the dominant modality. They first select the dominant modality by prior knowledge or silhouette coefficient and then learn the diversity information between it and the other modalities. The resulting ACC and NMI values on various datasets show the superiority of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed diversity-oriented multi-modal clustering method is novel, and the motivation is interesting which inspires the insights of dominant modality enhancement. The diversity learning and optimization method is described thoroughly, with a short but proper theoretical analysis, enhancing reader comprehension. The organization is quite clear and the experiment involving the enhancement or fusion is necessary to show the effectiveness of the diversity learning based method. \n\nHowever, there are also few weaknesses in the manuscript. \n\na) To enhance the paper's quality, it is better to cite some latest multi-modal clustering references that reflect recent research developments.\n\nb) What are the differences of the proposed method with existing methods except the diversity learning mechanism? The authors are suggested to give a clear description.\n\nc) In section 2.2, the author should detail when the method needs use the SI to select the dominant modality. \n\nd) The improvement in Table 2 is not clearly shown, and the authors should give a clear presentation.\n\ne) Small issue: the authors are better to improve the paper format: e.g., the proof for the theoretical analysis in section 2.6 is given in one paragraph, which is not good to understand the process. It is better to split them into more paragraphs to organize them in a more readable fashion.\n\nQUESTIONS:\nSee the strengths and weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a deep multi-modal clustering framework based on dominant modality enhancement named DDMC. This approach aims to solve two challenges in the deep multi-modal clustering field, i.e., information conflicts between modalities and modality quality imbalance of many exiting multi-modal fusion and clustering methods. Experiments on different datasets validate the effectiveness of the proposed DDMC, also clearly demonstrating its feasibility and stability in multi-modal scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper proposes a deep multi-modal clustering framework based on dominant modality enhancement, and lots of experiments on different datasets validate the effectiveness of the proposed method, which reflects the stability in multi-modal clustering scenarios. Generally, this paper proposes a novel multi-modal clustering method, and the method introduction is clear, written in a clear and logic way. The general work flow is fluent and easy to understand. The effectiveness and feasibility of DDMC are clearly validated in the experiments from the main and appendix parts of the paper.\n\n- Please find the specifics in Questions section.\n\nQUESTIONS:\n(1) It seems that the proposed DDMC is the first work using dominant modality enhancement to solve the multi-modal clustering problem. Are there any possibilities for generalizing the idea for other related problems like multi-task or transfer learning. \n\n(2) What are the sources of the compared methods in the experimental parts? They are reproduced by authors or given in the original paper? If it is from the corresponding papers, can the authors give the websites of them in the paper? \n\n(3) It is good to see that the authors give the limitations of existing methods in two aspects, but it is unclear which the kind of related works the limitation corresponds to. It is recommended for the authors to give more details concerning the analysis in the\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

comments: SUMMARY:
This work takes a different direction by learning diversity information between the dominant modality and other modalities. The proposed method first tackles information conflicts among modalities. Comprehensive experiments across multiple dimensions demonstrate the superior performance of this approach compared to existing state-of-the-art methods.

STRENGTHS AND WEAKNESSES:
Strength:

S1: This paper introduces a novel multi-modal clustering framework, which uses a dominant-modality enhancement strategy to mitigate noise from conventional feature fusion. 

S2: This strategy significantly improves the clustering performance when the data is unevenly distributed or has large quality differences.

Weaknesses:

W1: Technical Limitations. The authors propose two diversity learning paradigms: feature-level and cluster-level. What is the relationship between these paradigms? Do they operate through mutual enhancement, or how do they interact with each other?

W2: Methodological Issues. What is the original purpose of the silhouette coefficient? Given that there are various metrics similar to SI, could these alternative metrics also be applied for dominant modality selection? Please clarify the rationale behind choosing this specific metric.

W3: In Algorithm 1, the title 'Enhancement instead of fusion diversity learning for deep multi-modal clustering' appears inconsistent with the proposed method name. Additionally, the parameter description 'Number of categories of clusters K' is imprecise.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper a novel diversity-oriented deep multi-modal clustering method that focuses on dominant modality enhancement rather than modality fusion is proposed. The method first selects the modality with the highest average silhouette coefficient as the dominant modality and then leverages diversity learning to extract the differential information between the dominant modality and the remaining modalities. Experimental results show that the proposed method outperforms various compared single-modal and multi-modal methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper proposes a novel diversity-oriented deep multi-modal clustering method that focuses on dominant modality enhancement rather than fusing different modalities. It introduces an effective and efficient method for multi-modal recognition community. In particular, the idea concentrating on improving dominant modality instead of fusing different modalities is quite different from the related works, which may be a new paradigm for exploring the correlated information among modalities. Additionally, many kinds of experiments including ablation study, parameter analysis and convergence analysis have been conducted to show the superiority of the proposed method. \nWeakness: There are some weaknesses still exist in this paper listed in the following issues: 1) The authors mentioned that they decided the dominant modality method by prior knowledge or the Silhouette coefficient fraction, I am curious that what does the prior knowledge is. Can the authors give more details or information about the prior knowledge. 2) The font size in Figure 2/3 is too small, it is suggested to modify it to further improve the readability. 3) Are there any shortcomings of the proposed method? A more comprehensive analysis of the proposed method is needed.\n\nQUESTIONS:\nPlease see the weaknesses for details.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}