Diversity-oriented Deep Multi-modal Clustering
Authors: Yanzheng Wang
Xin Yang
Yujun Wang
Shizhe Hu
Mingliang Xu
Venue: NeurIPS 2025 poster ()
Keywords: Multi-modal clustering
diversity learning
contrastive learning
Primary area: general_machine_learning
TL;DR: We propose a diversity-oriented deep multimodal clustering method.

Abstract: Deep multi-modal clustering (DMC) aims to explore the correlated information from different modalities to improve the clustering performance. Most existing DMCs attempt to investigate the consistency or/and complementarity information by fusing all modalities, but this will lead to the following challenges: 1) Information conflicts between modalities emerge. 2) Information-rich modalities may be weakened. To address the above challenges, we propose a diversity-oriented deep multi-modal clustering (DDMC) method, where the core is dominant modality enhancement instead of multi-modal fusion. Specifically, we select the modality with the highest average silhouette coefficient as the dominant modality, then learn the diversity information between the dominant madality and the remaining ones with diversity learning, and finally enhance the dominant modality for clustering. Extensive experiments show the superiority of the proposed method over several compared DMC methods. To our knowledge, this is the first work to perform multi-modal clustering by enhancing the dominant modality instead of fusion.

Diversity-oriented Deep Multi-modal Clustering
Yanzheng Wang Xin Yang Yujun Wang, Shizhe Hu',
Mingliang Xu"
School of Computer and Artificial Intelligence, Zhengzhou University, China
Abstract
Deep multi-modal clustering (DMC) aims explore the correlated information from different modalities to improve the clustering performance. Most existing DMCs attempt to investigate the consistency or/and complementarity information by fusing all modalities, but this will lead to the following challenges: 1) Informa- tion conflicts between modalities emerge. 2) Information-rich modalities may be weakened. To address the above challenges, we propose a diversity-orienter deep multi-modal clustering (DDMC) method, where the core is dominant modality enhancement instead of muulti-moda fusion. Specifically. we select the modality with the highest average silhouette coefficient as the dominant modality, then learn the diversity information between the dominant madality and the remaining ones with diversity learning. and finally enhance the dominant modality for clustering. Extensive experiments show the superiority of the proposed method over several compared DMC methods. To our knowledge, this the first work to perform multi-modal clustering by enhancing the dominant modality instead of fusion.
1 Introduction
Deep multi-modal clustering (DMC) aims to integrate data from multiple modalities (e.g., image, text, audio, etc.) to classify the data through unsupervised learning. DMC combines the feature extraction ability of deep learning and the clustering idea of unsupervised learning, which is an important direction of multi-modal data analysis present, and has achieved excellent performance in many fields, such as the medical field [13), autonomous driving and intelligent transportation 5, and recommendation systems 66-8
Related Works. At present, almost all DMCs cluster by fusion method. Based on different fusion stages, DMCs can be roughly divided into the following three categories: 1) Feature-level fusion: these methods [013 extract the features of different modalities and connect them into single high-dimensional feature vector, which is used as a single input for clustering or feature learning. For example: Zhou and Shen [10] propose method that first use adversarial regularizer to align modalities, and then perform an attention fusion on all modalities, so as to quantify the importance of different modalities. 2) Decision-leve fusion: these methods [6] first obtain independent modeling of each modality, obtain their own output or clustering results, and then fuse these results in the final stage. For example, Meng al. [1] conduct the features of different modalities in their respective clustering layers for depression regression training, and linearly weighted sums the prediction results of each modality obtain the fusion result as the final output result. 3) Mixed-level fusion: these methods 17-119] combine the output of a single mode prediction through feature-level fusion. For example, Morales al. [17 train separate model for each modality, then get the predictions for each modality, and finally train new model on these new vectors to output the final prediction.
Equal contribution "Correponning author (iehiizhhuu@gmen @mmalllcommmmmmmmmmmm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Motivations. Although the above methods have achieved excellent performance, due to the use of different levels of fusion, the following challenges arise: 1): Inconsistency of multi-modal data: different modalities may provide conflicting or inconsistent information about the same thing. 2): Weight assignment and modalities importance: modalities fusion can cause some informative high- To address the
quality modalities to be forced to align with low-quality high-noise modalities [20]. above challenges, we drew inspiration from [21]. As shown in Figure a).. people perceive the world in the process of the world about 83% comes from the vision, and the
VISUAL
remaining ways (such smell, touch, taste and hearing) add up to only 17%. However, if the information obtained by the other ways is supplemented with the information obtained by the visual way, a more complete understanding will be obtained. Taking driving as example, most of the information on the road condition comes from the vision, if at the same time through the radio, speakers and other information, we will have a more correct judgment on the road condition. Inspired by this, we propose a new multi- modal clustering method in Figure 1b). in which modality 3 is the dominant modality. The diversity learning approach is employed to extract the diversity information between the dominant modality and the remaining modalities (modality 1, modality 2) and then the diversity information is spliced to the dominant modality for enhancement. Finally, the enhanced dominant modality is clustered.
SMELL
63%
TOUCH
TASTE
HEARING
1%
11%
Diversity Learning
Modality
Contributions. In this paper, we propose diversity- oriented deep multi-modal clustering (DDMC) method. Specifically, our innovation lies proposing dominant modality enhancement strategy that 'enhancement instead of fusion'. In this method, we first select an informative modality as the dominant modality, and subsequently em- ploy diversity learning to extract the diversity information from other modalities from the dominant modality. Fi- nally, we concatenate these dissimilarity information to the dominant modality and perform the final clustering output through the clustering module. Compared with the latest DMCs method, our method achieves significant perfor- mance improvement on benchmark datasets, which verifies its effectiveness and ad advantages. The main contributions of our work can be summarised follows:
Modaliry 3(Dominant)
Modality 2
Enhanced Features
Canarrncyy Faraex
Dinvaly Forei
(b)
Figure I: (a) The way humans perceive the world and its proportion. (b) The general idea of our method. Inspired by multi-modal assisted vision in human perception of the world.
Ours is the first work to investigate molti-modal clustering by enhencing dominant modalities rather than fusing modalities.
We propose inveerity--oiennee deep multi-modal clustering method by dominant modality enhancement rather than modality fusion, which can maximally retain important information in the raw modality and has the advantages of both single-modas and multi-modal clustering.
We can effectively enhance the dominant modality by simultaneously mining the diversity information of the remaining modality relative to the dominant modality at both the feature- level and the cluster-level. The results on multiple multi-modal datasets can validate the effectiveness of DDMC.
Methodology
A multi-modal dataset X = (XI..... .X. ......MM)) contains N samples of M modalities, where X E R Nxd, denotes the samples of dimension dm, from the m-th modality. Our purpose is to correctly divide N' samples into K clusters by learning information between multiple modalities. To provide more intuitive overview of the proposed method, we present it in Figure 2 Taking three modalities as an example, the dominant modality (modality 2 is shown the figure) first determined through the selection of dominant modality module. Subsequently, each modality is

---

Encoders
Fraton-leed Diversity Tearning
Clastering Module
A Fredhakk
X
X
X²
Comppgire kammg
Comppgieer loening
X
Min
Momee
I (D,22 (Z))
Divenity
Inforration
Guslance
2
Chaster-lvee Diversity Learning
Clustering Layers
Enhanced Fecaure
X
UDC
(Dominant)
1
Mar (4P.44 )
Ps
UDC
Max ((t,
A
X
UDC
Final Clustering
A
Crosse Entrogy
Figure 2: Details of the proposed framework. Firstly, the dominant modality feature Ddm consistency features {Crm AM mizdddm and diversity features (Zth M encoders. Then, through feature-level diversity learning and cluster-level diversity learning, the diversity information of the remaining modality is learned. Finally, the diversity information is enhanced for the dominant modality, and the final clustering completed through the clustering module. Among them, UDC is Uniform Distribution Constraint, which forces the edge distribution of each cluster to be as close to uniform as possible. DDC is Deep Divergence- based Clustering.
mitddm are obtained through modality specific
modeled with its original features through independent variational encoders. Next, diversity learning is performed at both the feature-level and the cluster-level, and the diversity information obtained from the non-dominant modalities is used to enhance the representation of the dominant modality. Finally, the final clustering is performed based on the enhanced dominant modality features to obtain the clustering results.
2.1 Motivation: Modality Fusion Leads to Lower Clustering Efficiency
In order to learn the information between modality, re- searchers capture the connections of heterogeneous modal- ity by fusing each modality. However, this also brings corresponding problems: unfair weights in the fusion lead to the degradation of the results. Specifically, we illustrate it the popular multi-modal dataset Flickr (22). as shown in Figure 3 We conducted 20 iterations of K- means 23 clustering for each modality and the fusion modality, representing the quality of individual modality and the fusion modality through its unsupervised accuracy, As shown in the figure, it can be clearly seen that modality 3 belongs to the high-quality modality, while modality 1 and 2 belong to the low-quality modality. Due to the unequal relationship between the weights allocated each modality and the quality in the fusion, the clustering effect of the fused modality is worse. However, these low-quality modality are not completely useless, we hope select a high-quality modality as the dominant modal- ity, and then learn useful information from the remaining modality enhance the dominant modality. Therefore, unfair alignment of high-qualij modality to low-quality modality can be avoided.
Poddlly
Mocilyy 1
Pocellty
Figure 3: The accuracy rates of each modality and the fused modality after K-means clustering.
3

---

2.2 Selection of Dominant Modality
We decide that the dominant modality method prior knowledge or the Silhouette coefficient (SI) fraction. Prior knowledge is that the modality weights in the dataset have been measured in previous work, or the importance of certain modality in some datasets is obvious. For these important modalities, the dominant modality is selected. For example, the importance weights of each modality are clearly given in [T0]. In the absence of reliable prior information, the SI fraction adopted to determine the dominant modality. SI is metric used to evaluate the quality of clustering results. It combines the intra-cluster closeness and inter-cluster separation, and calculates a SI for each data point to measure its similarity to its own cluster and its nearest neighbor cluster. In the raw features, the clustering results are calculated by the K-means method, and then the SI score of each modality can be derived. For each sample T."h the average SI of each modality is calculated as follows:
1
SI
SI (I Σ
(1)
N
TP EX
m
where:
(xx)") a (xx
ST (x?") =
max {b (x²"") a (x>"
C Eay EOT a." #a. (ar," m²)) to calculate the average distance between this sample point and all other points in the same cluster, and b (xx"" = min CTT Eay eaT d (2x" x="") is to calculate the average distance between this sample point and all points in the nearest neighboring cluster. |CT") is the number of all sample points in the cluster where x is located, and d(.) is the distance between the two sample points.
where a (xx"")
The modality with the highest SI is the dominant modality Xdm is given by:
dm = max
(I(XX)),.,,S((XX1)
(3)
2.3 Diversity Learning
After selecting the dominant modality, each modality is encoded by independent variational encoder. The dominant modality generates feature Dem, and the remaining modalities generate two sets of features, diversity features {Z" JM mm and consistent features {C'm) my/dmm Then, after two levels of diversity learning, the diversity information of the remaining modalities relative to the dominant modality can be learned while removing redundant information.
Feature-level diversity learning can extract the diversity features between non-dominant and dominant modalities through mutual information and contrastive learning, compress redundant information and separate consistent information from diverse information. Cluster-level diversity learning ensures balanced distribution of each cluster through Uniform Distribution Constraint and mutual information, and enhances the expression of dominant modality from the clustering level. Therefore, the two are synergistically enhanced, enhancing low-level feature expression through feature-level learning, aligning high-level semantic structures through cluster-level learning, and cluster-level can guide feature-level learning through pack-propagation to further promote clustering diversity information.
Feature-level Diversity Learning In order to leamn the diversity information at the feature level, we propose the following objective function:
M min L. = I(xdm Ddm)+ + Σ
M + Σ ; Ddm). m.f.dm
(4)
Where I(:) ) is the mutual information measurement. The first two terms describe the compression loss of the raw modalities after passing through the encoders, and the third term aims to learn the diversity information of the dominant modality. Inspired by the Information Bottleneck (IB) theory [24 25). according to Eq. 4 this method can effectively screen out the minimum sufficient representation that is most critical to the subsequent tasks of each modality under the guidance of the dominant modality, thereby suppressing redundant modal noise and highlighting the diversity information between the modalities.

---

In order to further effectively strip away the potential diversity information in the non-dominant modal- ity, the dominant modality D the non-dominant modality. Since the encoder network on which the consistency features { {Cm M and the diversity features { m rely share parameters, the alignment process can curately decouple and extract the correct diversity features from the non-dominant modality while maintaining the consistency semantics, thereby improving the expressiveness the dominant modal- ity. The contrastive loss between the dominant modality Ddm and the consistency features C is calculated as follows:
Dam dm is compared and aligned with the consistency features {C maddmm of
of ac-
ca 10 )
(deffr) d
pdm.. Σ log Σ ExeNeg(af cz es'iTi'
Am,m ij
(5)
where Ti temperature hyperparameter, .eg (ditm c.r)) represents the similarity set between negative sample pairs, ddm and ea represent the i-th and j-th samples of features Ddm and C"". Contrastive learning introduced between the dominant modality and other modalities to facilitate the screening of consistent features and further improve the efficiency of mining diverse information. The common way calculate contrast loss is to accumulate the contrast loss between all modalities, so the contrast loss function in this section can be expressed as:
M
C2
edmn. +
(6)
to sum up, the loss function of the feature- diversity learning is expressed as follows:
LFDL = C1 C2.
Cluster-level Diversity Learning In order to fully capture the semantic coherence between the dominant modality and the non-dominant modality and improve the effectiveness of diversity features in clustering tasks, DDMC further introduces diversity learning mechanism at the cluster level. Specifically, the diversity features extracted from the non-dominant modality and the dominant modality are input into independent cluster assignment layers obtain the cluster assignment [m=1, so as to mine the diversity information between them in the cluster structure, thereby enhancing the clustering expression ability of the dominant modality. To this end, we designed the following objective function:
representation {A "
M Σ mfdddmm
max
C3
at the same time, in order to improve the stability and balance of the clustering results, we further apply the UDC method to adjust the cluster assignment representation {A m) m=1 to obtain (Au" m) "
m==1-
The objective function of this item is:
M min L1= Σ CE(A")AA"))
where C' the cross entropy
26 The core of this method is to carry out standardization processing 27 This method can convert the soft probability prediction of each
similar to Sinkhorn-Knope sample belonging to each cluster into a more balanced hard label assignment, so that each sample is assigned to only one cluster and the distribution of the number of samples in each cluster is kept as balanced as possible, thereby improving the discriminability and robustness of clustering. The specific implementation details are shown in Appendix A.I Therefore, the overall objective loss function of cluster-level diversity learning LCDI. is:
LCDL= = L4
(10)
After being processed by the two-levels diversity learning module, the diversity information contained in the remaining modalities can be effectively extracted compared to the dominant modality.
5

---

Algorithm -Diverstty-oriented Deep Multi-modal Clustering
mm=1> Number of clusters K; Trade-off parameters and 8;; Epoch number E; Temperature parameters 71-
Input:
Multi-modal datasets {X" [X"
Initializing the network:
Select the dominant modality by Eq. for i E do
The dominant modality feature D consistency features {Cr .M (
mizddm and diversity features
Z /m obtained through modality specifie encoders;
The clustering results {.A m MMM.
of various modality are obtained through cluster layers;
Calculate LEDS. with Eq. (29 Eq.
(3I). Eq. (6) and Eq. 77)
Calculate LCDL. with Eq. (31 Eq.
and Eq. 10
Calculate LDDC by Eq. (12
Optimize all parameters by minimizing Eq. end for
13
Output::ultiimmodal clustering assignment Q.
2.4 Clustering Module
In order to alleviate the information interference and feature confusion problems that may be caused by direct modal fusion, this paper first enhances the dominant modality. The enhanced dominant modality is expressed as:
den
Enhanced Concat D, 7 ,?.,, 7"h (m # dm).
(11)
Then, the enhanced dominant mode is clustered by DDC to obtain the cluster assignment matrix Q. DDC (28] an effective unsupervised clustering method in deep learning, which aims to improve clustering performance by optimizing the dissimilarity measure between sample distributions. It consists of three parts, the first part is intra-cluster compression and inter-cluster separation, the second part is the orthogonality constraint, and the third part is the assignment of simple simplex corners, the loss optimization function of the clustering module is as follows:
K-1 CDDC
Kg K6 Ko,
K ΣE Di AF ΛTК
XKA,,
triu(Q"
(12)
K
K
1=1 f>i
Where K the Gaussian matrix kernel, 8, 8, denotes the column vectors of the cluster assignment matrix Q, and K is the number of clusters. X is the column vector of B [Bat], defined as:
Bab cxp lla e2.
2.5 Optimization
In summary, DDMC is achieved by minimizing the following loss function:
Liotal = LDDC + &LFDL + BLCDL-
(13)
Where 9 are trade-off parameters. To optimize the mutual information loss term in Eq. 4 we introduce a variational optimization strategy. Specifically, we estimate the lower bound of the mutual information term Li through variational method and use this its optimization target, because maximizing its variational lower bound can achieve an unbiased estimate L1. The specific implementation process is presented Appendix A.2 and the training steps of this method summarized in Algorithm
2.6 Theoretical Analysis
Theorem 1. By optimizing Eq. 7 is possible to remove the private redundant information of each modality while obtaining the diversity information of the remaining modality relative to the dominant modality.

---

For any three modalities (m E {u, v)), where dm is the dominant modality. We introduce a simplified assumption: when the training fully converged, the encoder can accurately capture the joint information. According to the data processing inequality, the mutual information between representations upper-bounded by that between the original modalities:
Proof:
T(Z"; Ddm) T(X"; X T(C"; Ddm) S T(X"; X
(14)
because Z" ++ X ++ X Xdm ++ Ddm forms a Markov chain. Similarly,
I(2"; Ddm) S x( I(C"; & 1Xx";
(15)
At best, the information that can be shared between each modality and the dominant modality satisfies:
(Z"; dm)) + (C"; Ddm) = 7 (x"; xdm), T (Z"; Ddm) + T (C"; Ddm) = T (x"; Xdm).
(16)
Since the optimization objective tends to maximize T(Z"; Ddm) and 7 T(C"; Ddm), while the com- pression loss enforces the representation to remain concise, the optimal strategy is to assign redundant shared information to low-dimensional subspace. Therefore, we let the encoding satisfy:
I (C"h,
Ddm Ddm)
= I (x"; x"),, (Z"; =0 0.
(17)
That is, C carries all common information between X and X while zu is independent ofis independent of Dam and contains only modality-speciffe information. Similarly,
I Dmm I (x"; I (mm) =0
(18)
At this time, C" and Ddm share the redundant information between all modalities, while Zu, 7 each only carries the remaining independent information. According to the chain rule of mutual information:
(X"
(X"; Ddm) + Z
(19)
When 1(2";) = 0, it follows that 1(X"; Z4 = 1(X"; 2²). Assuming complete informa- tion preservation, have:
T(X"; Ddm) = ("); I 1xx; 2u, Ddm) H (x"),
(20)
where H() is the entropy. Finally, we obtain:
I TZ"; = H (X") (x"; Xdm) = / xX"; Xm)
(21)
That is, 7u captures the amount of information that X still has given Xdm, which is exactly the diversity information of X relative to d The same is true for Z". The above derivation shows that introducing modal contrast loss and mutual information loss under the compression loss constraint of the variational autoencoder will decouple the latent variables of the non-dominant modality from the latent variables of the dominant modality: the shared information is reflected in C C", while Z", z" only contains the diversity information relative the dominant modality.
3 Experiment
In this segment, we carry out a series of experiments to verify the efficacy of the framework we have proposed. For more experimental data and experimental details, please refer Appendix(A.3 A.3
3.1 Datasets
We evaluate the effectiveness of our proposed method by employing five well-known datasets, including Caltech-3V, Caltech-4V, ESP-Game, Flickr and IAPR. The Caltech image dataset [29], which comprises 1440 samples distributed across 7 distinct classes, is available three multi-modal variants. Caltech-3V incorporates features of Wavelet moments 30), CENsus TRansform hISTogram
7

---

Table Clustering results in terms of ACC and NMI on the multi-moda datasets. (The bold and underline value are the best and second best result, respectively)
Methods
Caltech-3V
Caltech
ESP- PaP-Game
Flickr
IAPR
ACC
ACC
KM
46.3
31.3
54.6
46.7
43.2
29.4
40.9
22.5
38.9
17.2
Ncuts(TPAMM
42.6
25.4
67.8
47.6
41.0
25.9
48.4
26.1
41.9
18.9
AmKM
46.9
31.5
44.9
30.6
49.9
34.7
41.0
21.6
40.4
17.0
AINNus(TPAMM'0))
48.2
18.9
CVSC (NeurIPS'11) I1)
54.4
45.3
64.9
54.5
40.1
28.8
41.0
26.8
35.1
18.4
RMKMC (3CAT(13
59.5
49.4
65.5
60.3
44.7
29.7
42.3
23.4
36.4
15.9
SwMC (UCAI
30.2
23.1
43.7
44,2
43.7
44.2
34.3
34.5
30.2
23.1
ONMSC (AAAI'20)
SMCMB (TBD'23)
67.2
54,5
74.4
67.0
54.9
40.5
52.8
32.1
34,8
16.4
FAMC (CVPR'20)
38,9
21.4
29.6
16.5
27.1
6.5
30.5
9.1
37.1
16.4
DEMVC (InfoSci'2
38.7
27.0
48.4
39.7
35.5
21.6
44.8
25.2
30.1
13.8
SiMVC (CVPR'21)
16.2
18.5
CoMVC (CVPR221)
54.1
50.4
56.8
56.8
51.8
38.2
49.3
30.6
46.7
21.5
MFLVC (CVPR'22)
56.6
73.3
65.2
52.1
39.4
53.8
32.8
47.3
22.6
SEM (NeurIPS'23)
69.2
59.2
82.6
75.3
36.6
23.5
53.1
30,9
42.2
18.9
DIVIDE (AAAI'24)
60.9
53.8
64.3
57.9
46.5
27.0
52.3
33.5
45.6
23.0
SCMVC (TMM224)
75.9
66.3
84.4
72.9
36.1
24.8
54.2
32.3
46.5
24.1
SSLNMVC (TMM 25)
64.4
58.3
82,1
72.8
44.8
32.3
31,2
33.0
46.4
24.0
DDMC
76.7
68.8
90.3
82.7
60.9
40,9
58.7
36.5
49,5
28.3
Ours vs BestCompared
0.87
2.51
5.91
7.47
6.07
0.47
4.51
3.01
2.21
4.21
(CENTRIST) 31 and Local Binary Pattern 32. treating each feature type a separate modality. Caltech-4V builds upon Caltech-3V by introduc- ing additional feature, namely Gen- eralized Search Trees [33]. ESP-
Table I: Details about the Multi mollil Datasets
Dataset
Samples
Clusters
Dimension
Caltech-3V
1440
7
40/254/928
Caltech-4V
1440
7
40/254/928/512
ESP-Game
11032
7
300/300/300
Game [34] dataset is derived from an online image tagging game, contain- ing 11,032 images across categories, with each sample having three modal- ity descriptions. Flickr [22] dataset is widely used multi-modal dataset for image retrieval,
Flickr
12154
6
100/100/100
IAPR
7855
6
100/100
containing 12154 sample across 6 categories. It utilizes the same three modalities as ESP-Game.
IAPR (35] is comprehensive multi-modal image dataset, containing categories with 7,855 samples and two features.
3.2 Compared Methods
To further substantiate the merits of our proposed approach, conducted comparative experiments with a comprehensive set of eighteen baseline methods, classified into three types: Single-Mors
Clustering Methods: (-MMens(KKV), Normalized Cuts(Neuts) [36], All-modalities "MMeansAAKMM
All-modal Normalized Cuts(AvNNuts):: Traditional Multi-modal Clustering Methods: CVSC
37]. RMKMC B3), SwMC (39], ONMSC [40], SMCMB 41]; Deep Multi-modal Clustering
Methods: MVSCN [42), DEMVC 43 SiMVC (44), CoMVC [44), MFLVC (45), SEM (20).
DIVIDE 46), SCMVC [47, SSLNMVC (48). All comparison method codes are obtained from the
original authors through their official GitHub repositories or personal homepages.
The clustering performance is meticulously assessed by employing two extensively utilized metrics: Accuracy (ACC) and Normalized Mutual Information (NMI) [12]. 49
Evaluation Metrics
The higher the value of these two indicators, the better the performance of our model in terms of clustering accuracy and consistency.
3.3 Clustering Performance Analysis
Table demonstrates the clustering effectiveness of our proposed method on five publicly available datasets, with the clustering outcomes in terms of ACC and NMI being reported. It is observed that

---

Table 3: Ablation Study different multi-mon modal datasets (The bold and underline value are the best and second best result, respectively).
Caltech-3V ACC
Caltech-4V ACC
ESP-Game ACC
Flickr ACC
IAPR
Methods
NMI
NMI
NMI
NM1
ACC
NMI
(1) CDDC Conc LeDI.
70.2
53.2
77.8
69.6
35.3
14.4
49.0
30.7
39.1
18.0
71.4
73.5
35.5
48.8
(3) Cooc + LEDL
73.4
64.3
82.1
81.8
36,6
22.0
55.3
36.3
45.4
24.0
All Modules (The Proposed Method)
76.7
68.8
90.3
82.7
60.9
40.9
58.7
36.5
49.5
28.3
method significantly outperforms the compared single/all-view, traditional and deep multi-modal clustering methods. The remarkable results demonstrate the strong clustering capability our method, which is attributed to the two levels diversity learning that is capable of acquiring diverse knowledge and eliminating redundant information.
The proposed method in this study demonstrates remarkable adaptability. On the Caltech dataset, as the number of modalities increases, the ACC and NMI of the model show a steady upward trend, owing to the rich complementary information contained in multi-modal data. As the number of modalities increases, the model can obtain data features from more dimensions. This indicates that the proposed method is capable of efficiently integrating information from different modalities and fully leveraging the unique features and clustering information contained each modality, thereby significantly enhancing the accuracy and consistency of clustering. However, such an expected improvement in clustering performance is conspicuously absent when observing the outcomes of the EAMC and DEMC methods. These findings indicate that these methods might not be sufficiently robust when confronted with variations in the number of modalities. Compared with the best competitors, our method also obtains excellent performance. For example, the ESP-Game dataset, method demonstrates a substantial improvement in both ACC and NMI compared to the second-best method (SMCMB), achieving increases of 6.0% and 0.5% respectively. In terms of ACC and NMI results on the five tested multi-modality datasets, the proposed method achieves average improvements of 3.88% and 3.52% the second-best comparison method, respectively. This highlights the stability of our method across different datasets.
3.4 Ablation Study
In this part, give ablation study to further show the effectiveness of different components of our method. The DDMC includes a clustering module and two diversity learning modules, which means there are four possible combinations. The outcomes of these combinations The results indicate that the performance generally improves as more modules are incorporated into the model. For instance, the Caltech-3V dataset, the baseline model achieved an ACC of 70.2% and NMI of 53.2%. In contrast, the model that includes all modules saw a significant enhancement in performance, with ACC of 76.7% and NMI of 68.8%. Similar trends are observed across other datasets, with particularly notable performance enhancements in the Caltech-4V and IAPR datasets when all modules are integrated into the model. Specifically, the model equipped with all modules achieved an ACC of 90.3% and a NMI of 82.7% on the Caltech-4V dataset, and an ACC of 49.5% and NMI of 28.3% on the IAPR dataset. These results underscore the superior performance of the model when all components are utilized.
presented in Table 3
3.5 Enhancement us Fusion
Enhancement
Fusion
In addition, we replaced the final enhancement step with the commonly used modality fusion strategy, and the comparison results are shown in Figure 4 Experiments show that the enhancement strategy outperforms the fu- sion method on all datasets; especially on the ESP-Game dataset, the ACC is improved by about 11% This re- sult shows that compared with modality fusion, dominant modality enhancement can significantly improve cluster- ing performance, while the noise introduced in the fusion process will weaken the role of the dominant modality and reduce the clustering effect, further verifying the effective- ness of the DDMC.
0013
1.501
0,982
0A83
Caltech-4/
Flikk
JAPR
Figure 4: Enhancement us Fusion

---

(a) Caltech-3V
(b) Caltech-4V
(c) ESP-Game
(d) Flickr
(e) IAPR
Figure 5: Parameter analysis of DDMC on different datasets.
3.6 Parameter Sensitivity
In the proposed method, thoroughly investigate the sensitivity of these parameters, we conducted extensive experiments on all datasets with various parameter configurations. Specifically, we utilized a grid search strategy to optimize the values of and / within the range [10-*, 10-3 10-2, 1001 100 -10", 10 10"]. The clustering ACC results are illustrated in Figure As can be seen, our method exhibits robust performance across all datasets, with minimal performance degradation under most parameter settings. This suggests that our method relatively insensitive to parameter fluctuations and maintains stable performance overall. Therefore, DDMC can have good performance in a large range of parameter values.
trade-off parameters a, to balance LEDL, LCDL- To
4 Conclusion and Limitations
clustering framework that leverages dominant- modality enhancement strategy to mitigate noise from conventional feature-fusion. Rather than fusing all modalities indiscriminately, level diversity learning to extract diversity information from the remaining modalities, and augment the dominant modality accordingly. This strategy significantly improves the clustering performance, especially when the multi-modal data is unevenly distributed or has large quality differences.
This paper introduces novel deep
identify the highest-quality modality dominant, perform two-
In addition, method is not without limitations. Although SI is excellent indicator, there may still exist other theoretically feasible indicators. will continue to study them in the future and introduce more robust dominant modality selection mechanisms. We also realize that this method performs poorly when dealing with incomplete modal data, especially when some modalities are missing or the data is unbalanced. In future research, We will explore more powerful selection strategies and effective strategies for extracting diversity information, such as combining multiple indicators adaptive optimization methods, ensure that the selection of the dominant modality is more convincing.
Acknowledgment
The authors thank anonymous reviewers for their constructive comments. This work was supported by Henan Province Outstanding Youth Science Fund Program under Grant 252300421223, National Natural Science Foundation of China under Grant 62206254, 62325602, 62036010, and China Postdoctoral Science Foundation under Grant 2024T170843 and 2023M743186.
References
A. Roy, P. Rivas, and M. Tayba, "Unsupervised machine learning methods for diagnosing autism spectrum disorder using multimodal data: A survey," Proc. Int. Conf. Comput. Sci. Comput. Intell., 2022, pp. 41-52.
[2] D. Townsell, Banerjee, L. Chen, and M. Raymer, "Advancing chest x-ray diagnostics via multi-modal neural networks with attention," in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2024, pp. 1-4.
[3] A. Kumar, "Deep learning for multi-mode medical imaging fusion: Enhancing diagnostic accuracy in complex disease detection." Int Eng Technol Res Manag, vol. no. 11, p. 183, 2022.
10

---

[4] D. Feng, C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Gläser, F. Timm, W. Wiesbeck, and K. Dietmayer. "Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges." IEEE Truns. Intell. Trunsp. Syst., vol. 22, no. 3, pp. 1341-1360, 2021.
[5] X. Zhang. Y. Gong. Lu, J. Wu, Z. Li, Jin, J. Li, "Multi- modal fusion technology based on vehicle information: A survey," IEFE Trans. Intell. Veh. vol. 8, no. 6, pp. 3605-3619, 2023.
[6] Z. Huang, X. Xu, J. Ni, H. Zhu, and C. Wang. "Multimodal representation learning for recommendation in internet of things," " IEEE Internet Things J., vol. 6, no. 6, pp. 10675-10685, 2019.
[7] Y. Tong, "Research on deep clustering and multimodal deep learning service recommendation system for large-scale user data," in Int. Conf. Integr. Circuits Commun. Syst., 2025, pp. 1-5.
[8] J. Li, C. Yang. G. Ye, and Q. V. H. Nguyen. "Graph neural networks with deep mutual learning for designing multi- modal recommendation systems," Iuf. Sci., vol. 654, p. 1 118815, 2024.
[9] A. Kumar and H. D. III. "A co-training approach for multi-view spectral clustering." in Proc. Int. Conf. Mach. Learn., 2011, pp. 393-400.
[10] R. Zhou and Y. Shen, "End-to-end adversarial-atteniine network for multi-modal clustering," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 107--144616 616.
[11] Y. and Liao, "Multi-view clustering via adversarial view embedding and adaptive view fusion," Appl. Intell. vol. 51, pp. 1201-1212, 2021.
[12] S. Hu, G. Zou, C. Zhang, Z. Lou, R. Geng, and Y. Ye, "Joint contrastive triple-learning for deep multi-view clustering," Inf. Process. Manag,, vol. 60, 103284, 2023.
[13] G. Ke, G. Chao, X. Wang, C. Xu, Y. Zhu, and Y. Yu, "A clusteringgguided contrastive fusion for multi-view representation learning," IEFE Trans. Circuits Syst. Video Technol., vol. 34, pp. 2056-2069, 2024.
[14] H. Meng, D. Huang, H. Wang, H. Yang, M. AI-Shuraifi, and Y. Wang, "Depression recognition based on dynamic facial and vocal expression features using partial least square regression." in Pruc. Int. Workshop Audio/Vis. Vis. Emot, Chail., 2013, pp. 21-30.
[15] A. Samareh, Jin, Z. Wang, X. Chang, and Huang. "Detect depression from communication: How computer vision, signal processing, and sentiment analysis join forces," IISE. Trans. Healthe. Syst. Eng., vol. 8, pp. 1-42, 2018.
[16] X. Liu, L. Liu, Q. Liao, S. Wang, Y. Zhang, W. Tu, C. Tang, J. Liu, and E. Zhu, "One pass late fusion multi-view clustering," in Proc. Int. Conf. Mach. Learn., 2021.
[17] M. Morales. S. Scherer. and R. Levitan. "A inguustically-infomme fusion approach for multimodal depression detection," in Proc. Workshop Comput. Linguist. Clin. Psychol.: From Keyboard Clin., 2018 pp. 13-24.
[18] S. Alghowinem, R. Goecke, J. Cohn, M. Wagner, G. Parker, and M. Breakspear, "Cross-culuuue cultural detection of depression from nonverbal behaviour," in Proc. IEEE Int. Conf. Auto, Face Gesture Recognit. vol. 1, 2015, pp. 1-8.
[19] H. Shalu, H. P. H. S. CN, A. Das, S. Majumder, A. Datar, S. M. MS, A. Das, and J. Kadiwala, "Depression status estimation by deep learning based hybrid multi-modal fusion model," CoRR, vol. abs/2011. 14966. 2020.
[20] J. Xu, S. Chen, Y. Ren, X. Shi, H. Shen, G. Niu, and X. Zhu, weightee contrastive learning among multiple views for mitigating representation degeneration," in Proc. Ady. Neural Inf. Process. Syst., 2023.
[21] D. Qiao, G. Liu, T. Lv, W. Li, and Zhang, "Marine vision-based situational awareness using discriminative deep lcarning: A survey," J. Mar. Sei. Eng, vol. 9, no. 4, 2021.
[22] M. J. Huiskes and M. S. Lew, "The mir flickr retrieval evaluation," in Proc. Int. ACM Conf. Multimedia Retr. MIR., 2008, pp. 39-43.
[23] J. A. H. A. Wong, "Algorithm as 136: A k-means clustering algorithm." J. R. Stat, Soc., vol. 28, no. 1, pp. 100-108, 1979.
[24] N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," in Annu, Allerton Conf. Commun., 1999, pp. 368-377.

---

[25] S. Hu, Z. Lou, X. Yan, and Y. Ye, "A survey on information bottleneck," IEEE Trans. Pattern Anal. Much. Intell., vol. 46, 8, pp. 5325-5344 2024.
[26] A. Mao, M. Mohri, and Y. Zhong. "Cross-entropy loss functions: theoretical analysis and applications." in Proc. Int. Conf. Mach. Learn., 2023.
[27] P. A. Knight, "The sinkhorn knop algorithm: Convergenoe and applications. SIAM J. Matrix Anal. Appl..
vol. 30, no. 1, pp. 261-275, 2008.
[28] M. Kampffmeyer, S. Løkse, F. M. Bianchi, L. Livi, A. Salberg, and R. Jenssen, "Deep divergence-based approach to clustering," Neural Netw., vol. 113, pp. 91-101, 2019.
[29] L. Fei-Fei, R. Fergus, and P. Perona, "Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories." in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2004, pp. 178-178.
[30] D. Shen and H. H. 1p. "Discriminative wavelet shape descriptors for recognition of 2-d patterns." Pattern Recognit, vol. 32, no. 2, pp. 151-165, 1999.
[31] J. Wu and J. M. Rehg, "Centrist: A visual descriptor for scene categorization," " IEEE Trans. Pattern Anal. Mach. Intell,, vol. 33, no. 8, pp. 1489-1501, 2011.
[32] L. Wolf, T. Hassner, and Y. Taigman, "Descriptor based methods in the wild," Proc. Eur. Conf. Comput. Vis., 10 2008
[33] A. Friedman, "Framing pictures: The role of knowledge in automatized encoding and memory for gist," J. Exp. Psychol.-Gen. Gen., vol. 108, pp. 316-55, 09 1979.
[34] T.-S. Chua, J. Tang, R. Hong. H. Li, Z. Luo, and Y. Zheng, "Nus-wide; a real-world web image database from national university of singapore," in Proc. ACM Inf. Conf. Image Video Retr, 2009.
[35] M. Grubinger, P. Clough, H. Müller, and T. Deselaers, "The iapr tc12 benchmark: new evaluation resource for visual information systems," Jr. Int. Conf. Comput. Linguist., 10 2006.
[36] Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Trans. Puttern Anal. Mach. Intell., vol. 22, no. 8, pp. 888 905, 2000.
[37] A. Kumar, Rai, and H. Daume, "Co-regularized multi-view spectral clustering," in Adv. Neural Inf. Process. Syst. vol. 24, 2011.
[38] X. Cai, F. Nie, and H. Huang, "Multi-view k-means clustering on big data," in Proc. Int. Joint Conf. Artif. Intell, 2013, pp. 2598-2604.
[39] F. Nie, J. Li, and X. Li, "Self-weighted multiview clustering with multiple graphs," in Proc. Int. Joint Conf. Artif. Intell., 2017, pp. 2564-2570.
[40] S. Zhou, X. Liu, J. Liu, X. Guo, Y. Zhao, E. Zhu, Y. Zhai, J. Yin, and W. Gao, "Multi-view spectral clustering with optimal neighborhood laplacian matrix," " in Proc. AAAI Conf. Artif. Intell., 2020, pp- 6965-6972.
[41] J. Lao, D. Huang, C.-D. Wang, and J.-H. Lai, "Towards scalahle multi-view clustering via joint learning of many hipartite graphs," IEEE Trans. Big Dota. vol. 10, no. 1, pp. 77-91, 2024.
[42] Z. Huang, J. T. Zhou, X. Peng, C. Zhang, H. Zhu, and J. Lv, "Multi-view spectral clustering network," in Proc. Int. Joint Conf. Artif. Intell., 2019, pp. 2563-2569.
[43] J. Xu, Y. Ren. G. Li, L. Pan, C. Zhu, and Z. Xu, "Deep embedded multi-view clustering with collaborative training." " Inf. Sci., vol. 573, pp. 279-22998 290, 2021.
[44] D. J. Trosten, S. Lokse, R. Jenssen, and M. Kampffmeyer, "Reconsidering representation alignment for multi-view clustering," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2021, pp. 1255-1265.
[45] J. Xu, H. Tang, Y. Ren, L. Peng, X. Zhu, and L. He, "Multi-level feature learning for contrastive multi-view clustering," " in Proc. IEEE Conf. Comput. Vis, Pattern Recognit., 2022, pp. 16 030-16039.
[46] Y. Lu. Y. Lin. M. Yang. D. Peng, P. Hu, and X. Peng. "Decoupled contrastive multi-view clustering with high-order random walks," " in Proc. AAAI Conf. Artif. Intell., 2024.
[47] S. Wu, Y. Zheng, Y. Ren, J. He, X. Pu, S. Huang, Z. Hao, and L. He, "Self weighted contrastive fusion for deep mullii-view clustering." IEEE Trans. Multimedia. vol. 26, pp.
2024.
12

---

[48] W. Yan. T. Yang, and C. Tang, "Self-supervised semantic soft label learning network for deep multi-view clustering," IEEE Trans. Multimedia., pp. 1-13, 2025.
[49] Y. Gan, Y. You, J. Huang, S. Xiang, C. Tang, W. Hu, and S. An, "Multi-view clustering via multi-stage fusion," " IEEE Trans. Multimed., pp. 1-13, 2025.
[50] R. Jonker and T. Volgenant, "Improving the hungarian assignment algorithm." Oper. Res. Lett., vol. 5, 4, pp. 171-175, 1986.
[51] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradhury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopt, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and Chintala, "Pytorch: An imperative style, perforeefer deep learning library," Proc. Adv. Neural Inf. Process. Syst., 2019.
13

---

A Appendix
A.1
The Details of Uniform Distribution Constraint
The role of UDC is to force the edge distribution of each cluster to be close to uniform as possible when generating hard labels from the soft clustering distribution. Firstly: Firstly, perform exponentiation operations on the original cluster distribution matrix [Am]M to enhance the
Im=l
diagonal weights, Then, define two edge vectors r E R K and c RN The goal is to make P diag(r) diag(c) satisfy the following requirements:
Plk
IN, PTIN= = K 1K. N
(22)
That is, P approximates 'coupling matrix' that is uniform for both rows and columns. Then iterative calculation carried out through the Sinkhorn- Knopp algorithm:
K k A(O)
+IN
10+1)
(1+1)
(23)
(1+1)
Until convergence, transform the normalized matrix Pback to the N x K shape and take the maximum index of the n-th row obtain the discrete label: Aun = arg max;, Pr, k The above process not only ensures the discreteness of the output labels, but also approximately satisfies the uniform distribution constraint of each cluster globally.
A.2 Optimization
Taking modality X as an example, based on the definition of mutual information, we can get the following formula:
(x²:2²)= = 2'.x!
log p(=))( / p (2 log
p p
(x x²)) (z11)
(24)
Since the posterior distribution p (z),x²) cannot be solved directly, another scalable distribution ((22) is used to approximate p(z*). By minimizing the Kullback-Leibler (KL) divergence between
the two distributions, q(=²) optimized step by step make it similar to p(=²). Specifically, since
KL is non-negative, we can get:
KT. p (2') (=')] 1 (=2')
p (x=
log q (21)
(25)
[ (()) logo (+t) > / p(x*)( log (=t)
Now, Eq. 24 can be estimated as:
(xt; < [ (2) log
p (2 q (211)
J 21 p (z*1x=t) p(x*) log
(2*1x²)
(26)
(2t)
Since m MM Imsdddm does not contain the dm-th term and the dominant modality is not distinguished
in the calculation of compression loss, for the sake of simplicity, we temporarily let 2dm Ddm that the compression loss in (4) can be estimated as:
so
M
M
p (2 q (z²)
Σ middm
1(x"h lT + dm D)
Σ/ mm
p (x" log
(27)
In order remove unnecessary items, Monte Carlo sampling is used replace MM M p(.²). There- fore, Eq. (27) can be further expressed as:
M
p (2"'l20 (2" " p p>>"11>") [.m)
Σ dyddm
1(xT + 1(X D)
log
(28)
q (2
m
14

---

where N is the number of data samples. Assuming that Σm p(2"'<>"" conforms to the Gaussian distribution, the mean μ and variance a each modality can be learned through the encoder. For the convenience of calculation, we reparameterize ΣM n" by Σm zt = _ux=") + o(r") * where is the standard normal distribution. Therefore, the objective loss function for calculating the compression loss in Eq. can be expressed as:
M
P ((=T|<>")
1(x"; T(Xdm; D)
E. log
N
9 ((2
(29)
E
K KTIp(=""
where E, represents mathematical expectation. For the calculation of diversity information loss in Eq.44). we use the discrete joint probability estimation for calculation:
M 1 Ddm)= =
M
dm
dm
mm (i.
m.dm (i,j) log
(30)
pm perre
m-ddm
d
==1
=1
where:
dm
dm
pm mm (1,5). (j) >
m mm,
(31)
pmd dm (i.j)
D (j) + D (1) (1)
2NV
==1
Similarly, the calculation of cluster assignment mutual information L3 can also be optimized by 31.
A.3 Further Analysis
Evaluation Metries ACC used to evaluate the degree correspondence between the assigned cluster labels and the true labels of the data points, essentially measuring the extent to which the clustering algorithm correctly identifies the inherent groupings within the dataset, which is defined as:
ACC ACC=Eis 8(p,, map(q.))
E
(32)
n
where denotes the number of samples, 8(i.)) 1if (8(i,) O, otherwise), and map(4:) represents the clustering result being matched to the ground truth through Hungarian algorithm|SL (50]. NMI quantifies the mutual dependence between the clustering results and the ground truth, taking into account the distribution of data points across different clusters, formulated by:
(S2C)
NMI (H + (C))/2
(33)
where Ω indicates the original ground truth, and C' denotes the clustering information, H( represents the entropy of the clustering result, and [(;) indicates mutual information between clustering results.
Details of the Compared Methods K-Menas: An unsupervised learning algorithm that divides data into K clusters so that the similarity of data points within clusters is high and the similarity between clusters is low.
All-modal K-Means: This algorithm concatenate all modal features and then executes KMenas
algorithm
136 An image segmentation algorithm based graph theory is proposed to minimize the normalized cut value to optimize the image region partitioning, while considering the inter-group dissimilarity and intra-group similarity.
Normailzed Cuts:
All-modal Normalized Cuts: The method uses Normalized Cuts for the joint representation each data point to achieve clustering.
15

---

CoregMVSC: [37 A co-regularized view spectral clustering method that enhance accuracy by enforcing clustering consistency across views.
RMKMC: 38 A multi-view multiple kernel clustering framework based on the restarted strategy
and self-guiding mechanism.
SwMC: [39] self-weighted mult-iview clustering algorithm that iteratively optimizes the target similarity matrix and view weights to effectively integrate multi-view data for clustering without requiring additional parameters.
ONMMSC: [40] A multi-view spectral clustering algorithm searches for the optimal matrix within the neighborhood of the linear combination of Laplacian matrices, breaking through the limitations of traditional methods to enhance clustering performance.
SMCMB: 41
This method mining rich information in multi-view data by joint learning of mul- tiple bipartite graphs, and maintaining high efficiency on large-scale data sets, the time and space complexity is close linear.
EAMC: (0] An end-to-end adversarial attention network that aligns potential feature distributions and quantifies modal importance, respectively, through adversarial learning and attention mechanisms.
DEMVC: 43
A multi-view clustering algorithm that utilizes common and complementary in- formation from multiple views to achieve better clustering performance through deep embedded representation learning and collaborative training mechanisms.
SiMVC: 44 A deep multi-view clustering baseline model that does not require alignment of representation distributions
CoMVC: [4] A deep multi-view clustering algorithm that selectively aligns representations of different views at the sample level by contrasting learning frameworks.
MFLVC: 45
multi-level feature learning framework is proposed, which can effectively reduce the interference of view private information by separating reconstruction target and consistency target.
DIVIDE; 46 A novel robust multi-view clustering method, which identifies global data pairs via high-order random walks and employs a decoupled contrastive learning framework to perform intra-view and inter-view contrastive learning in separate embedding spaces, thereby enhancing clustering performance and robustness against missing views.
48 A deep multi-view clustering method that enhances the consistency of multi- SSLNMVC:
View features through a consensus high-level feature learning module and aligns view-specific and view-consensus semantic labels using self- supervueere semantic calibration module.
SEM:
[0 This method solves the representation degradation problem caused by contrast learning in multi-view scenarios through self-weighting and information reconstruction strategies.
The method establishes hierarchical feature fusion framework and self-weighted contrastive fusion approach, effectively separating the consistency objective from the reconstruction objective.
SCMVC: 47
Implementation Details Our experiments were conducted on a Windows 10 operating system, utilizing powerful configuration equipped with 96 GB of system memory and high-performance NVIDIA GeForce RTX 4090D GPU. We implemented the proposed framework using the Py Torch platform (5). For all datasets, the training batch size was uniformly set to 512, and we utilized the Adam optimization algorithm with an initial learning rate of 0.0003. The configuration of parameters in the proposed model detailed follows. The hyperparameters a and B are tuned to values ranging from 0,0001 to 1000, with each value being a power of 10. Given that 100 epochs proved to be ample for the training convergence of algorithm, we accordingly trained the model from the beginning up to 100 epochs. To enhance robustness and circumvent local minima, we trained the proposed model 10 times, reporting the clustering outcome with the minimal clustering loss. For all datasets, we utilized modal-specific variational encoders comprising three fully connected layers, each layer consists of a batch normalization layer and a RELU layer. The second layer and the output layer are set to 512. The clustering layer adopts fully connected layer with a softmax layer. The parameter of the dropout layer is set to O. The temperature hyperparameter in the comparative learning is set to 0.5.
16

---

Caltech 3V
(h) Caltech 4V
(c) ESP Game
(d) Flickr
(e) TAPR
Figure 6: Convergence analysis of DDMC on the datasets in the order of Caltech 3V Caltech 4V, ESP- Game, Flickr and IAPR, respectively.
(a) 0 epoch
(b) 25 epochs
(c) 50 epochs
(d) 75 epochs
(e) 100 epochs
Figure 7: Evolution of cluster assignments during training on the Caltech-4V dataset.
Convergence Analysis To evaluate the convergence properties of our proposed approach, we conducted a comprehensive series of experiments across diverse range of datasets, including Caltech-3V, Caltech-4V, ESP-Game, Flickr, and IAPR datasets. As depicted in Figure [ we present the objective loss values in conjunction with the clustering performance metrics, specifically ACC and NMI, throughout the training epochs. It is evident that the loss values experience pronounced decline during the initial stages of training, particularly within the first 25 epochs. This rapid decrease indicates that the model is effectively learning and adjusting its parameters to minimize the loss function. Subsequently, the loss values begin to stabilize, suggesting that the model has reached a point of diminishing returns in terms of further loss reduction. The results consistently demonstrate that convergence for all metrics is achieved after the 100-th epoch threshold. This uniform convergence pattern across the diverse datasets highlights the robustness our algorithmic iterations in attaining balanced state, thereby ensuring consistent performance metrics irrespective of dataset variations.
T-SNE Visualization Analysis In order to further substantiate the efficacy of our proposed frame- work, we employed the widely recognized t-sne tool to visualize the clustering outcomes of DDMC on the Caltech-4V dataset. The joint training of approach yields satisfactory clustering results. To provide an intuitive depiction of the training evolution, we carried out t-sne visualizations of the clustering results at different training epochs, namely the 0-th, 25-th, 50-th, and 100-th epoch, as illustrated in Figure 7 In this figure, distinct colors are assigned to represent the various clusters generated by the DDC module.
evident that as the number of training epochs increases, the clustering assignments tend to become more compact and well-separated from each other. This observation indicates the effectiveness of our proposed approach enhancing clustering performance throughout the training process.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction sections accurately reflect the contribution of the paper. For details, please refer to the abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the paper are explained in Section 4. Please see Section 4 for details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: The paper provides a detailed description of the proposed theory and optimiza- tion. For details, please see Section 2.6 and Appendix A.I.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The experiments in Section 3 are all real and effective. Please see Section for details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Data and code are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttpe//gguudeeeeeeouuuuumbbbba ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section and Appendix A.3 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix A.3 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes],
Justification:
This article complies with the NeurlPS ethical standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The research work this paper will not have any impact on society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk in the paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper involved no crowdsourcing and no research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper involved no crowdsourcing and no research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The methodological development of this study did not involve LLM as any sigmificant, original, or nonstandard component.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
24


Reviews:
[{"review_id": "LvDxp6VFBp", "reviewer": "Reviewer_MBDR", "summary": "In this paper a novel diversity-oriented deep multi-modal clustering method that focuses on dominant modality enhancement rather than modality fusion is proposed. The method first selects the modality with the highest average silhouette coefficient as the dominant modality and then leverages diversity learning to extract the differential information between the dominant modality and the remaining modalities. Experimental results show that the proposed method outperforms various compared single-modal and multi-modal methods.", "questions": "Please see the weaknesses for details.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: This paper proposes a novel diversity-oriented deep multi-modal clustering method that focuses on dominant modality enhancement rather than fusing different modalities. It introduces an effective and efficient method for multi-modal recognition community. In particular, the idea concentrating on improving dominant modality instead of fusing different modalities is quite different from the related works, which may be a new paradigm for exploring the correlated information among modalities. Additionally, many kinds of experiments including ablation study, parameter analysis and convergence analysis have been conducted to show the superiority of the proposed method. \nWeakness: There are some weaknesses still exist in this paper listed in the following issues: 1) The authors mentioned that they decided the dominant modality method by prior knowledge or the Silhouette coefficient fraction, I am curious that what does the prior knowledge is. Can the authors give more details or information about the prior knowledge. 2) The font size in Figure 2/3 is too small, it is suggested to modify it to further improve the readability. 3) Are there any shortcomings of the proposed method? A more comprehensive analysis of the proposed method is needed.", "was_revised": true, "final_justification": "Authors have addressed all my concerns.", "rebuttal": "Thank you very much for your careful review and precious suggestions. We attach great importance to the raised issues and have carefully revised and improved the problems such as the incomplete method description, the small font size in the pictures and the inadequate description of the method's shortcomings based on the comments. The following is our detailed response and explanation to each comment:\n\nComment 1：The authors mentioned that they decided the dominant modality method by prior knowledge or the Silhouette coefficient fraction, I am curious that what does the prior knowledge is. Can the authors give more details or information about the prior knowledge？\n\nThanks for your comment. We give priority to using prior knowledge to determine the dominant modality in our method because in many practical tasks, we can quickly judge based on experience that the information of a certain modality is richer or more critical. For example, \"ESP-Game\" is a dataset about online image annotation games. Then, vision is the most direct and efficient modality in annotation, while text plays an auxiliary role in recognition, helping the model better learn the detailed information in the pictures. Therefore, selecting the modality with obvious advantages in this task as the dominant modality and integrating the diverse information of other modalities will help improve the overall learning effect. However, when the dominant modality cannot be clearly judged, we use the Silhouette coefficient score as an auxiliary basis and select the modality with high intra-cluster compactness and high inter-cluster separation as the dominant modality.\n\nComment 2：The font size in Figure 2/3 is too small, it is suggested to modify it to further improve the readability. \n\nThanks for your comment. We have noted the small font size issue in the figure, and we have enlarged the font size in this figure as well as all other figures throughout the paper to ensure better readability in both print and electronic formats.  In addition, we have carefully adjusted the proportion between the font size and graphical elements to achieve a more balanced and visually clear presentation. \n\nComment 3：Are there any shortcomings of the proposed method? A more comprehensive analysis of the proposed method is needed.\n\nThanks for your comment. The method proposed in this paper mainly improves the performance of multi-modal clustering by selecting the dominant modality and combining the diversity information of other modalities. However, our method still has the following shortcomings: (1) There is still room for improvement in the dominant modality selection strategy. Although SI is an excellent indicator, there may still exist other theoretically feasible indicators. We will continue to study them in the future and introduce more robust dominant modality selection mechanisms. (2) We also realized that this method performs poorly when dealing with incomplete modal data, especially when some modalities are missing or the data is unbalanced. We will add a clearer description of the limitations of this method in the paper."}, {"review_id": "rY4MXNvPGi", "reviewer": "Reviewer_a5KX", "summary": "This work takes a different direction by learning diversity information between the dominant modality and other modalities. The proposed method first tackles information conflicts among modalities. Comprehensive experiments across multiple dimensions demonstrate the superior performance of this approach compared to existing state-of-the-art methods.", "questions": "Please refer to the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nS1: This paper introduces a novel multi-modal clustering framework, which uses a dominant-modality enhancement strategy to mitigate noise from conventional feature fusion. \n\nS2: This strategy significantly improves the clustering performance when the data is unevenly distributed or has large quality differences.\n\nWeaknesses:\n\nW1: Technical Limitations. The authors propose two diversity learning paradigms: feature-level and cluster-level. What is the relationship between these paradigms? Do they operate through mutual enhancement, or how do they interact with each other?\n\nW2: Methodological Issues. What is the original purpose of the silhouette coefficient? Given that there are various metrics similar to SI, could these alternative metrics also be applied for dominant modality selection? Please clarify the rationale behind choosing this specific metric.\n\nW3: In Algorithm 1, the title 'Enhancement instead of fusion diversity learning for deep multi-modal clustering' appears inconsistent with the proposed method name. Additionally, the parameter description 'Number of categories of clusters K' is imprecise.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I keeped my original score.", "rebuttal": "Thank you very much for your careful review and precious suggestions. We attach great importance to the issues raised and have carefully revised and improved the problems such as the relationship between learning paradigms in the method, the selection of metrics, and the conflicts described in the article based on the comments. The following is our detailed response and explanation to each comment:\n\nComment 1：Technical Limitations. The authors propose two diversity learning paradigms: feature-level and cluster-level. What is the relationship between these paradigms? Do they operate through mutual enhancement, or how do they interact with each other?\n\nThanks for your comment. In our method, feature-level diversity learning can extract the difference features between non-dominant and dominant modalities through mutual information and contrastive learning, compress redundant information and separate consistent information from diverse information. Cluster-level diversity learning ensures balanced distribution of each cluster through Uniform Distribution Constraint and mutual information, and enhances the expression of dominant modality from the clustering level. Therefore, the two are synergistically enhanced, enhancing low-level feature expression through feature-level learning, aligning high-level semantic structures through cluster-level learning, and cluster-level can guide feature-level learning through back-propagation to further promote clustering diversity information.\n\nComment 2： Methodological Issues. What is the original purpose of the silhouette coefficient? Given that there are various metrics similar to SI, could these alternative metrics also be applied for dominant modality selection? Please clarify the rationale behind choosing this specific metric.\n\nThanks for your comment. Silhouette coefficient (SI) is an indicator used to evaluate the quality of clustering, which measures the closeness of data points in the same cluster and the degree of separation from other clusters. SI is usually used to evaluate the effectiveness of clustering methods such as K-means and hierarchical clustering. We also considered whether other indicators can replace SI, such as mutual information, reconstruction loss, Davies-Bouldin index (DBI) and other indicators, but none of them are suitable as indicators for selecting dominant modality. Specifically, the modality with the largest mutual information with the remaining modalities indicates that the modality has the highest correlation, and the modality with the smallest reconstruction error indicates that the information retention ability is the strongest. Neither of them can effectively select the modality with the richest information. DBI is easily disturbed by outliers and has poor stability. Therefore, SI is a suitable metric for selecting the dominant modality, as it can comprehensively consider intra-class compactness and inter-class separation, and also has good numerical stability and cross-mode comparison ability. In addition, there must be many theoretically feasible metrics. In our future work, we will conduct more extensive research to improve the selection strategy.\n\nComment 3：In Algorithm 1, the title 'Enhancement instead of fusion diversity learning for deep multi-modal clustering' appears inconsistent with the proposed method name. Additionally, the parameter description 'Number of categories of clusters K' is imprecise.\n\nThanks for your comment. We have noticed that the algorithm title deviates from the actual title and the description is inaccurate. We will conduct a detailed check of the full text in subsequent versions to ensure that the full text is consistent and the wording is correct. Thank you again for your careful review."}, {"review_id": "OvEIWj3mQ3", "reviewer": "Reviewer_85Ru", "summary": "This paper introduces a deep multi-modal clustering framework based on dominant modality enhancement named DDMC. This approach aims to solve two challenges in the deep multi-modal clustering field, i.e., information conflicts between modalities and modality quality imbalance of many exiting multi-modal fusion and clustering methods. Experiments on different datasets validate the effectiveness of the proposed DDMC, also clearly demonstrating its feasibility and stability in multi-modal scenarios.", "questions": "(1) It seems that the proposed DDMC is the first work using dominant modality enhancement to solve the multi-modal clustering problem. Are there any possibilities for generalizing the idea for other related problems like multi-task or transfer learning. \n\n(2) What are the sources of the compared methods in the experimental parts? They are reproduced by authors or given in the original paper? If it is from the corresponding papers, can the authors give the websites of them in the paper? \n\n(3) It is good to see that the authors give the limitations of existing methods in two aspects, but it is unclear which the kind of related works the limitation corresponds to. It is recommended for the authors to give more details concerning the analysis in the", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This paper proposes a deep multi-modal clustering framework based on dominant modality enhancement, and lots of experiments on different datasets validate the effectiveness of the proposed method, which reflects the stability in multi-modal clustering scenarios. Generally, this paper proposes a novel multi-modal clustering method, and the method introduction is clear, written in a clear and logic way. The general work flow is fluent and easy to understand. The effectiveness and feasibility of DDMC are clearly validated in the experiments from the main and appendix parts of the paper.\n\n- Please find the specifics in Questions section.", "was_revised": true, "final_justification": "Thanks to the author for solving my concerns, and I decided to keep the original score.", "rebuttal": "Thank you very much for your careful review and precious suggestions. We attach great importance to the issues raised and have carefully revised and improved the problems such as the application areas, the sources of the comparison methods and limitations of related work in the paper based on the comments. The following is our detailed response and explanation to each comment:\n\nComment 1：It seems that the proposed DDMC is the first work using dominant modality enhancement to solve the multi-modal clustering problem. Are there any possibilities for generalizing the idea for other related problems like multi-task or transfer learning.\n\nThanks for your comment. Your suggestion is an important inspiration for our future work. Multi-task learning aims to complete multiple related tasks simultaneously by sharing feature representations, but in practical applications, it often faces challenges such as task conflicts and unreasonable sharing methods. Drawing on the idea of \"dominant modality guides diversity learning\" in this study, tasks can be divided into dominant tasks and non-dominant tasks, thereby effectively alleviating conflicts and interference between tasks while enhancing the performance of dominant tasks. Transfer learning is committed to transferring the knowledge obtained in the source task to the target task, and often faces problems of negative transfer and large differences between the source/target domains. If combined with our method framework, by selecting the source domain with the richest information as the \"dominant\" and combining it with a differential guidance strategy for transfer, it is expected to improve the efficiency and stability of knowledge transfer. In future work, we will further explore the application potential and expansion methods of this idea in multi-task learning, transfer learning and other scenarios.\n\nComment 2： What are the sources of the compared methods in the experimental parts? They are reproduced by authors or given in the original paper? If it is from the corresponding papers, can the authors give the websites of them in the paper?\n\nThanks for your comment. The comparison methods used in the experimental section are all representative and high-performing approaches in the field of deep multimodal clustering. For fairness and reproducibility, we obtained the implementations directly from the authors’ official GitHub repositories, as referenced in their original papers. To facilitate future research and ensure transparency, we will clearly provide the URLs of these codebases in the revised version of the paper, so that other researchers can easily access and utilize them.\n\nComment 3：It is good to see that the authors give the limitations of existing methods in two aspects, but it is unclear which the kind of related works the limitation corresponds to. It is recommended for the authors to give more details concerning the analysis.\n\nThanks for your comment. First of all, almost all existing deep multi-modal clustering methods use modal fusion to find consistent information between modalities. All the methods mentioned in the related work use different levels of fusion, which corresponds to the limitations of multi-modal data conflicts and unfair fusion weights. For some of the few methods that do not use modal fusion to find cross-modal information, they also face the challenge of being unable to effectively resolve information conflicts. We will further supplement the relevant work in the article based on your suggestions."}, {"review_id": "ZqilzyQRvJ", "reviewer": "Reviewer_FbZy", "summary": "To address the two challenges in most existing deep multi-modal clustering methods, the authors design a novel diversity-oriented multi-modal clustering method whose main idea is to enhance the dominant modality. They first select the dominant modality by prior knowledge or silhouette coefficient and then learn the diversity information between it and the other modalities. The resulting ACC and NMI values on various datasets show the superiority of the proposed method.", "questions": "See the strengths and weaknesses", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The proposed diversity-oriented multi-modal clustering method is novel, and the motivation is interesting which inspires the insights of dominant modality enhancement. The diversity learning and optimization method is described thoroughly, with a short but proper theoretical analysis, enhancing reader comprehension. The organization is quite clear and the experiment involving the enhancement or fusion is necessary to show the effectiveness of the diversity learning based method. \n\nHowever, there are also few weaknesses in the manuscript. \n\na) To enhance the paper's quality, it is better to cite some latest multi-modal clustering references that reflect recent research developments.\n\nb) What are the differences of the proposed method with existing methods except the diversity learning mechanism? The authors are suggested to give a clear description.\n\nc) In section 2.2, the author should detail when the method needs use the SI to select the dominant modality. \n\nd) The improvement in Table 2 is not clearly shown, and the authors should give a clear presentation.\n\ne) Small issue: the authors are better to improve the paper format: e.g., the proof for the theoretical analysis in section 2.6 is given in one paragraph, which is not good to understand the process. It is better to split them into more paragraphs to organize them in a more readable fashion.", "was_revised": true, "final_justification": "The authors have well solved my concerns and I decide to raise the rating.", "rebuttal": "Thank you very much for your careful review and precious suggestions. We attach great importance to the issues raised and have carefully revised and improved the problems such as the recent advances in multi-modal clustering, the details of the methodology and Overall article structure based on the comments. The following is our detailed response and explanation to each comment:\n\nComment 1：To enhance the paper's quality, it is better to cite some latest multi-modal clustering references that reflect recent research developments.\n\nThanks for your comment. In the field of deep multi-modal clustering, many excellent methods have also been proposed recently. These methods mainly enhance the performance of multi-modal clustering by optimizing the contrastive learning mechanism, improving the extraction accuracy of consistency information between modality, and effectively removing redundant information within modality. However, it has not effectively solved the problems of information conflicts and unfair weights brought about by modality fusion. We will elaborate in detail on multiple deep multi-modal clustering methods that can reflect the latest progress in the paper.\n\nComment 2：What are the differences of the proposed method with existing methods except the diversity learning mechanism? The authors are suggested to give a clear description.\n\nThanks for your comment. In addition to the diversity mechanism, this paper has two major differences from existing methods: (1) It uses the dominant modality-guided method for clustering. Compared with existing methods that consider all modalities at the same time and extract consistent information from them, it can effectively avoid the problem of information conflict between modalities. (2) There is no fusion throughout the process. Modal fusion will weaken the importance of high-quality modalities due to weight issues, while this method can well retain all useful information in high-quality modalities and enhance high-quality modalities through differential learning.\n\nComment 3：In section 2.2, the author should detail when the method needs use the SI to select the dominant modality.\n\nThanks for your comment. In our method, we give priority to using prior knowledge to select the dominant modality. This is because in many real-world tasks, domain experience can be used to quickly determine which modality is more representative and effective in expressing information. For example, \"ESP-Game\" is a dataset about online image annotation game, then vision is the most direct and efficient modality in annotation, and text plays the role of auxiliary recognition, which can help the model to better learn the detailed information in the picture. When there is a lack of clear prior judgment basis, we introduce the SI score as an auxiliary criterion and select the modality with a higher clustering SI score, that is, the modality with compact intra-class and good inter-class separation, as the dominant modality reference to enhance the robustness of the model in unsupervised situations.\n\nComment 4：The improvement in Table 2 is not clearly shown, and the authors should give a clear presentation.\n\nThanks for your comment. We will modify and optimize all the tables in the paper to ensure that the experimental results are clearer and more intuitive. These improvements aim to enhance the readability of the results and to more comprehensively demonstrate the performance gains and overall strengths of the proposed model across various evaluation metrics.  By improving the layout and clarity of the tables, we hope to make it easier for readers to interpret the experimental findings and appreciate the contributions of our method.\n\nComment 5：Small issue: the authors are better to improve the paper format: e.g., the proof for the theoretical analysis in section 2.6 is given in one paragraph, which is not good to understand the process. It is better to split them into more paragraphs to organize them in a more readable fashion.\n\nThanks for your comment. We will restructure the proof in Section 2.6 by breaking it down into logically organized and clearly separated paragraphs. This adjustment is intended to make the mathematical derivation more transparent and easier to follow.  Additionally, we will conduct a comprehensive review of the entire paper to further enhance its overall readability, including improvements in the clarity of explanations, formatting consistency, and presentation flow. These efforts aim to ensure that readers from diverse backgrounds can better understand and engage with the content of the paper.  Once again, we sincerely thanks for your valuable feedback."}]

Author rebuttal:

Meta-review: This paper proposes a diversity-oriented deep multi-modal clustering method in which the differences with existing methods lie in the dominant modality enhancement instead of modality fusion. Generally, all the reviewers reach a consensus that the proposed method is novel, with effectiveness validated by extensive experiments. Meanwhile, reviewers also raised some concerns in the initial review stage on the method description/working issues, algorithm analysis, generality to other fields, and experimental details. With the rebuttal provided by the authors, the reviewers suggest that the concerns have been well addressed, reaching final positive ratings. By carefully reading the paper, initial reviews, and rebuttal, I agree with the reviewers and recommend accepting this paper.
Decision: Accept (poster)