A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Authors: Zhengzheng Lou
Hang Xue
Chaoyang Zhang
Shizhe Hu
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal clustering
Information bottleneck
Primary area: general_machine_learning->clustering
TL;DR: We in this paper give a new peer-review look on the multi-modal clustering problem and propose to iteratively treat one modality as ``author" and the remaining modalities as ``reviewers" so as to reach a peer-review score for each modality.

Abstract: Despite the superior capability in complementary information exploration and consistent clustering structure learning, most current weight-based multi-modal clustering methods still contain three limitations: 1) lack of trustworthiness in learned weights; 2) isolated view weight learning; 3) extra weight parameters. Motivated by the peer-review mechanism in the academia, we in this paper give a new peer-review look on the multi-modal clustering problem and propose to iteratively treat one modality as "author" and the remaining modalities as "reviewers" so as to reach a peer-review score for each modality. It essentially explores the underlying relationships among modalities. To improve the trustworthiness, we further design a new trustworthy score with a self-supervision working mechanism. Following that, we propose a novel Peer-review Trustworthy Information Bottleneck (PTIB) method for weighted multi-modal clustering, where both the above scores are simultaneously taken into account for accurate and parameter-free modality weight learning. Extensive experiments on eight multi-modal datasets suggest that PTIB can outperform the state-of-the-art multi-modal clustering methods.

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Zhengzheng Lou' Hang Xue
Chaoyang Zhang'
Shizhe Hu
Abstract
the perspective of strategies, current MMCs can be classified into weighted MMC (Wang et al., 2020a), shared feature subspace learning based MMC (Li et al., 2022), multi-modal consensus clustering (Liu et al., 2021; Liang et al., 2024), multi-mode co-clustering (Kumar et al., 2011; Zhang et al., 2024), multi-modal subspace clustering (Li et al., 2022), and tensor representation learning based MMC (He Atia, 2022; Gu et al., 2024). With the rapid development of vari- ous kinds of methods, MMC has been successfully applied into many practical applications, such as object recogni- tion (Lou et al., 2013), human action recognition (Hu et al., 2020) and coherent groups detection in crowd scenes (Wang et al., 2020b).
Despite the superior capability in complementary information exploration and consistent cluster- ing structure learning, most current weight-based multi-modal clustering methods still contain three limitations: 1) lack of trustworthiness in learned weights; 2) isolated view weight learning; 3) extra weight parameters. Motivated by the peer-review mechanism in the academia, we in this paper give new peer-review look the multi-modal clus- tering problem and propose iteratively treat one modality as "author" and the remaining modali- ties as "reviewers" so as to reach peer-review score for each modality. It essentially explores the underlying relationships among modalities. improve the trustworthiness, we further design new trustworthy score with a self-supervision working mechanism. Following that, we propose novel Peer-review Trustworthy Information Bot- tleneck (PTIB) method for weighted multi-modal clustering, where both the above scores are si- multaneously taken into account for accurate and parameter-free modality weight learning. Exten- sive experiments on eight multi-modal datasets suggest that PTIB can outperform the state-of-the- art mulli-modal clustering methods.
Related Works. Investigated from the above methods, the weighted MMCs have exhibited remarkable clustering per- formance in the past decade, which is mainly due to its superior capability in complementary relationship discov- ery and consistent clustering structure learning. For the example of some typical ones, Xu et al. (2016) incorpo- rated feature selection into weighted MMC for solving the high -ith-dimnnsional data clustering, so that the discriminative representations and accurate clustering results can be jointly learned. Further, to address the challenging large-scale data clustering, Zhang et al. (2019) proposed an efficient binary MMC method, worked by collaboratively conducting the discrete representation learning and binary clustering as- signment discovery. Different from most weighted MMCs, Zhao et al. (2020) designed a cluster weights learning based method to find the more fine-grained cluster relationships compared to the cult parameter learning issue, Wang et al. (2020a) focused on automatically tuning the weight parameters to obtain a refined unified graph fusion matrix. In more recent year, Zhou & Shen (2020) leveraged adversarial learning and at- tention mechanism to align the latent feature distributions and quantify the importance of modalities respectively. Xia et al. (2023b) introduced an adaptive fusion layer to adap- tively sense the importance of modalities. Hu et al. (2025) introduced a simple fusion mechanism that dynamically updates modality-specife weights via backpropagation, sim- ilar to parameter optimization. Chen et al. (2023) explored the higher-order relations across modalities by designing a low-rank Tensor based proximity learning method for MMC
1. Introduction
methods. For the diffi-
Learning consistent clustering result from different modal- ities by fully mining the underlying modality correlations is the essence of multi-modal clustering (MMC) (Raya et al., 2024). From the view of basic method, existing MMC methods are generally based on k-means, canonical cor- relation analysis, spectral clustering, matrix factorization, information bottleneck and deep learning methods. From
'School of Computer Science and Artificial Intelligence, Zhengzhou University, Zhengzhou, China. Correspondence to: Shizhe Hu <ieshizhehue @eaaeegmmaaaaoo com, https://shzzhchuugge httphhhhhhhuiituiioooooooo github.io/>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
or quality of the "author" modality. By mimicking this inter- esting mechanism in Figure 1, we can employ the feedback review scores to effectively integrate the complementary dis- criminative information across modalities so as to promote multi-modal clustering performance.
G General peer-review
A specific example
Author Reeilower
resicw
Raciawrr
Sccepe Revis Reject
Author
Author Rarieneer
/Restawer
rexiew
reall
Ascept Bevise Roject
Revisser
Contribution. this paper, we propose to address the multi- clustering problem from the new peer-review look. It allows the remaining modalities as "reviewers" to review one "author" modality, and provides peer-review scores for each modality. Additionally, the correspond- ing trustworthy score is designed to measure the reliabil- ity of different "reviewer" modalities, thus ensuring the trustworthiness of the peer-review scores learned from the "peer-review" process. With the remarkable performance of the popular information bottleneck theory (Tishby et al., 1999) in multi-modal learning (Gao et al., 2007; Lou et al., 2013; Hu et al., 2022), we in this paper propose Peer- review Trustworthy Information Bottleneck (PTIB) method for solving the weighted multi-modal clustering problem. The clustering result obtained in each iteration is used to update the trustworthy score of each "reviewer" modality, working in a self-supervised mechanism. Thus, a more rea- sonable trustworthy evaluation of the peer-review score for each modality is reached. The peer-review scores given by "reviewer" modalities and the corresponding trustworthy scores for evaluating these "reviewer" modalities are com- bined to determine the importance or contribution (using modality weights in this paper) of each "author" modal- ity. Moreover, we solve the optimization problem of PTIB by an effective k-means-like algorithm. Experiments on eight multi-modal datasets demonstrate the superiority and effectiveness of the proposed method.
Peer-review" on mulli- moddl clustering
A specific case
Modality
Trevien
reSRiK
Modellty
The
Mledallyy
Modallty
Modality 3
Frssew
Modality
The
Figure 1. The upper box indicates the general peer-review process box shows the peer-review look on multi-modal clustering, where each modality can be either an "author" or a "reviewer".
in academia, while the lower
problem.
Investigated from the above closely-related weighted MMC works, we observed there mainly lies three limitations. First, all the existing weighted MMCs only focus on learning the modality weights using different meth- ods, ignoring whether the learned weights are trustworthy or not. Second, the weight learning mechanisms of most weighted MMCs is separated and essentially relied on the "single-more modal information", c.g., w" , where w' and Fti indicate the modality weight and the objective function value of the i-th modality respectively. Thus, the modality correlations are not fully explored and exploited, which may degrade the clustering performance. Third, most weighted MMCs require one or more parameters for controlling the weight distribution, where these parameters are usually dif- ficult to be tuned by hand without any prior knowledge.
Limitations.
The major contributions are summarized as follows:
A novel pecr-review trustworthy information bottle- neck (PTIB) method is proposed for multi-modal clus- tering. which performs by jointly learning the peer- review scores on different modalities and the trustwor- thy scores for quantifying the trustworthiness of the peer-review scores.
Motivation. In academia, peer-review mechanism is always adopted to mutually assess the contribution of one specific work to the community, discuss the significance of work, and provide communication to peers. Similarly, different modalities jointly contribute the multi-modal clustering task, making it necessary to assess their individual contri- butions. Inspired by this, naturally give a peer-review look on the multi-modal clustering problem. As illustrated in Figure 1. in general peer-review process, a person may be an author or a reviewer. When a person acts as an au- thor, the work will be assigned to different reviewers by Editor in Chief (EIC) Associate Editor (AE) to provide recommendations and suggestions. The peer-review mech- anism facilitates the improvement and refinement of work, ensuring high-quality and reliable publications. From the "peer-review" look on multi-modal clustering, one modality can either be an "author" or a "reviewer". The "reviewer" modalities review the work of the "author" modality, and produce feedback review scores to evaluate the contribution
We give new peer-review look on the multi-modal clustering problem, and thus design a peer-review score for evaluating the quality of each modality.
A corresponding trustworthy score is newly designed to evaluate the trustworthiness of peer-review score, ensuring the reliability of multi-modal peer-review.
Rich experiments on eight multi-modal datasets demon- strate the superiority and effectiveness of the proposed PTIB.
Relations Between Multi-modal and Multi-view Clus- tering. Generally, the aim of multi-modal clustering and
2

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
multi-view clustering is similar, especially in integrating different sources of information for improving clustering performance. The differences between them are as follows: Multi-view learning focuses on diverse feature representa- tions of the same object, while multi-modal learning deals with complex relationships between heterogeneous modal- ities, which is more complicated to handle. In practical applications, overlaps between them may exist (e.g., multi- modal data can also be considered as generalized multi-view data). However, technical solutions should be selected based on data characteristics (feature homogeneity and semantic consistency) to ensure methodological compatibility. In this paper, we use a more general expression of multi-modal clustering instead of multi-view clustering.
(2024) propose multi-modal IB method that uses shared representations of multiple modalities eliminate private information of single modality. But the modality-private information is eliminated as much as possible during the process of data compression, only exploring the shared in- formation of modalities without taking advantage of the complex relationship between modalities. Hu et al. (2024a) learn embeddings on two distinct feature spaces, reconstruct semantic information in parallel manner, and IB theory is further used to reduce representation noise. However, its final clustering result is obtained by directly averaging the local clusters from the modal high- nigh-dimensional features.
Different from the existing multi molti-modal clustering methods based on IB theory, the proposed method considers the com- plex relationship between modalities, where the designed multi-modal peer-review process is used to reasonably score the contribution of each modality, and the trustworthiness of it is ensured in self-supervised manner.
2. Revisit: Information Bottleneck
Information bottleneck (IB) principle (Tishby et al., 1999) originated from the rate-distortion theory, and its details can be referred from our survey (Hu et al., 2024b).
3. The Proposed Method
For clustering problem, IB considers as a data compres- sion process, which attempts to learn an optimal compressed representation T of X while maximally maintaining the rel- evant information about variable Y. It is formally described as
In this section, we first give the problem formulation and overall framework, then elaborately picture the details of the PTIB method.
Problem Formulation. Given the random variable X = (2,,42,.... .n.) denoting the set of n data samples from the dataset X. we have the random variable {Y'), denot- ing the data feature representation of m different modalities, where YJ E Rm==d indicates that the feature dimension of the j-th modality is d², Then, we reach the corresponding co-occurrence matrix {X, Y y= and its joint probability distribution {p(X, Y>)m } by adopting the popular Bag- of- Words model (Fei-Feii & Perona, 2005).
R(D) = min
(2()))))(()))))2 X);
(1)
where the compact representation T compresses the source variable X while maximally capturing the relevant infor- mation with respect to the variable Y, p(t]x) indicates the probability of data point x being assigned to 1-th cluster, X) and (T;; Y) denote the mutual information be- tween the compressed variable T and variable X and Y respectively.
By adopting a positive 3, we have the following Lagrange version of the IB method
Overall Framework. As the overall framework illustrated in Figure 2, PTIB first captures the local clustering struc- ture of each modality, i.e., T T IB method, and then obtains the peer-review scores for each "author" modality with the evaluation of remaining "reviewer" modalities. Meanwhile, the final clustering re- sult is adopted for assessing the trustworthiness of each modality in a self-suuervision fashion. With the peer-review (1² u,² ... nn' and trustworthy {o-, _....... omm) score for each modality, more accurate modality weights and im- proved clustering result are learned in each iteration.
,T",, with the
Emin [ X(T;X)- - BI(T;Y),
(2)
where 8 E (0,+oc) is a Lagrange multiplier which serves as trade-off parameter between data compression and rele- vant information preservation. And a formal iterative solu- tion for the above Eq. (2) is given as
p(t) @Dxrleewllelle<r11
p(t|z)= Z(x, 3)
(3)
Regarding the trustworthiness of multiple modalities, al- most all existing methods focus on trustworthy multi-mode classification(Han et al., 2021: 2023; Zheng et al., 2023; Zou et al., 2023). Han et al. (2023) introduce the variational Dirichlet characterize the distribution of the class proba- bilities, parameterized with evidence from different views and integrated with the Dempster-Shafer theory, thus pro- moting both classification reliability and robustness. Zheng
where p(()= = Σ. Ex (())))()) Z(x, B) is a normalization function, p(y|t)= = M3) Σ. p(t)=))((x, y). and DeL (Cover & Thomas, 2006) is the Kullback-Leible divergence.
In recent years, IB theory has been widely used in various multi-modal clustering tasks. For example, Federici et al. (2020) propose a multi-modal IB method that can identify non-shared information between two modalities. Yan al.
3

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Peer-review Score
(42, 15)
Local Clustering
(4, v
AAAAA
W Weight Learning
T
Ch, m
w" u²aa*
X
Information Bottleneck
W
Σ M o.
Trustworthy Score
wºx
X
o
AAAAA
(0)10/)
Update
Figure 2. The framework of PTIB. Each modality is treated as an "author" or "reviewer", with its local clustering result serving as the "work" or "criteria". A peer-review is conducted in the multi-modal setting to obtain peer-review score. Additionally, the final clustering result played "FIC /AE" quantifies the reliability of "reviewer" modality obtain trustworthy score. Both the two scores taken into account for modality weight learning, leading to more accurate weights and improved clustering result through iteration.
et al. (2023) proposes trustworthy multi-modal classifi- cation network via multi-level confidence learning, which integrates both feature and label-level confidence learning for trustworthy multi-mode classification. Zou et al. (2023) induces a transparent fusion strategy based on the modality confidence estimation strategy track information variation within different modalities for dynamical fusion. Different from them, the proposed method aims to guarantee the trust- worthiness of the learned modal weights in self-supervised manner. To the best of our knowledge, none of the exist- ing weighted MMCS employ the trustworthy strategy in the weight learning process.
criteria
peer-review
result
score
Reviewer
Modality
work
mutual information
Author
Modality 2
Figure 3. The local clustering results from the modality and modality 2 are regarded the criteria and the work, respectively. The pecr-review score is the mutual information between them.
criteria. The peer-review score depends on how similar it is to the review criteria. Apparently, the general metric of mutual information is a good choice for quantifying them. For clarity, we give two-modal example in Figure To be more accurate, we adopt the normalized mutual information version and formally define it as follows
3.1. Peer-review Score
Here, we give a new peer-review look on the multi-modal clustering, resorting the advantages of this mechanism to learn modal weight through modal interaction.
In general peer-review process, there often some spe- cific review criteria conducted to assess and score the thor's work. By mimicking this, we reach the peer-review score in similar manner under the multi-mode modal peer-review process. First of all, it is necessary to establish the peer- review criteria for different reviewer modalities. The feature of each modality contains unique characteristics, but their different feature dimensions make it hard quantify their discrepancy. Therefore, we instead adopt the local cluster- ing result of the reviewer modality as review criteria. Then, the author's 's work is also represented with the local cluster- ing result, which can be directly compared with the review
2x I(Ti, T k = H(T*) H(T*)
(4)
where n4 the peer-review score given by the reviewer modality / to the author modality V T and T are the local clustering results obtained from the reviewer modality yi and the author modality V* respectivell. H (o) denotes the entropy of one variable. Naturally, all of the peer-review scores of author modality / can be represented as vector
1 = -.i,,.., ......., - ,..... H
(5)

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Remark. Actallyy there are many ways attain the final peer-review scores, while we only select the normalized mutual information a typical metric. Additionally, al- though the peer-review score provides a quantitative anal- ysis for exploring the modality correlations, it still has its limitations, Due to the varying quality of multiple modali- ties, low-quality "reviewer" modality may give an inaccu- rate peer-review score to a relatively high-quality "author" modality. This may probably lead wrong evaluation of the importance of modalities and eventually do harm to the clustering performance. Hence, it is imperative to analyze and evaluate the trustworthiness of the peer-review scores given by reviewer modalities to ensure their reliability.
Hipl- -laa 1--ahann- -aam
label
cluster
p
dlupc)=11
label
p Hipl -Sau 3- Goloa - OSEB
cluster
Figure 4. A typical example of the "symmetry crisis" issue that the poor and better clusters may have the same cluster uncertainty.
(0.5 1)
(0.5.
H(p)
(5
3.2. Trustworthy Score
0:27
In this part, we propose to regard the final clustering assign- ment as the "EIC / AE", which is then used to evaluate the trustworthiness of the peer-review score in each iteration with superviuenv fashion. The final clustering assign- ment improves the iteration increases, thus leading to more accurate trustworthy scores.
6.71
0:21
0.1
0.15
p
Figure 5. The solution of handling the "symmetry crisis" problem.
entropy of it gets, thus resulting in high uncertainty of the cluster and interference in information judgment.
Clustering is to divide samples into different clusters accord- ing to their features, which is similar to a two-layer decision tree process. By treating the input sample set as the root node of decision tree, each leaf node thus corresponds to one specific cluster of the reviewer modality. Thus, we assess the trustworthiness of the reviewer modality by measuring cluster uncertainty, as defined follow, much like evaluating the quality of decision by measuring leaf purity.
Eq. (6) leads to an issue that a poor cluster and a better cluster obtain the same value of Eq. (6), where we call it "symmetry crisis". A typical example is illustrated in Figure 4. To address this crisis, we assume that a cluster with more than half incorrectly assigned samples (i.e., less than half major category) is of low quality, and we directly set its cluster uncertainty as the max value, as shown our handling of it in Figure 5. Finally, based on Definition 3.2, we define the following concept of cluster distortion to evaluate the quality of the reviewer modality.
Definition 3.1 (Major/Minor Category). Given a multi- modal dataset, if the local clustering result of modality super- vised by the final clustering result, the category of correctly assigned samples in cluster of specific local clustering result is called major category, the set of categories of incor- rectly assigned samples in it is called minor categories. Definition 3.2 (Cluster Uncertainty). For one reviewer modality, if the probability of major category in a cluster p. then the cluster uncertainty can be measured by the following information entropy
Definition 3.4 (Cluster Distortion). Let U (u1, U2.... } denote the set of clusters in each reviewer modality and C = (...,, Crui denote the set of clusters the final clustering result in each iteration, then the cluster distortion from U to C' is given by
if ifosp<b 1 d(uy. c.) Hp),,
[1.
(7)
(p) -llog p- (1 p) logg(1 p).
(6)
1
The final clustering result cannot make a completely ac- curate judgment on the local clustering result like the true label. We can only expect that the two are as similar as pos- sible, that is, the probability of major category is as larger as possible. In a self- supervieed scenario. the mixing of minor categories introduces interference when judging the infor- mation of the current cluster, so we only use the p and 1 to represent the cluster uncertainty. The following theorem can prove this claim and its proof is given in Appendix A. Theorem 3.3. For arbitrary cluster of a reviewer modality, the more mixed the minor categories gain, the higher the
where My (1 q S (UT) denotes the arbitrary cluster in the cluster set U, c, denotes the major category of each cluster uy in U. and p = J |u n c./. Note that d(u, c.) 1 indicates that a cluster dominated by incorrectly assigned samples have the largest distortion.
Based on the distortion measurement Eq. (7) with respect to a cluster, the distortion between U and C is defined as follows
10
D(U,C) = U
d(ug Cw).
(8)
9=1
5

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
The above clustering distortion works by measuring how much the local clustering result of a single reviewer modality distorts from the final clustering result, similar to the evalu- ation from EIC/ / AE to reviewer in peer-review mechanism. is noted that this approach is coincidentally consistent with the self-supervised learning. Generally, the smaller the clustering distortion is, the more reliable the reviewer modality is. Formally, the trustworthy score is defined by
T
T
0.5
w 12 0.1
T
T
02 = 0.2
(a)
T
T
/4 = 0.5
W 14 oj 0.4
T
T
0.8
of D(T".TF)
(9)
T
T
0.5
where of is the trustworthy score of reviewer modality Vi, T is the local clustering result of reviewer modality V and T' is the final clustering result. Similarly, all the trustworthy scores for reviewer modalities of reviewing the author modality V can represented as a vector
= 1/ =04
T
oj
(b)
T
T
1 0.5
W M 02 0.1
T
= -of.....,
(10)
3.3. Modality Weight Learning
Figure 6. Example of two-modal prejudice processing. (a) unreasonable that high-quality modality with high trustworthy score learned small weight. (b) T gives the final modality weights by directly adopting its own review.
By jointly considering the peer-review and trustworthy score, we obtain the final modality weight with the inner product as follows
Based on above, we summarize the final modality weight learning as follows
" = 1t t²
Σ i=1,ie
u..o..m m m2 2.
(11)
p of if = 2, 1=1,ik
Σ
Note that Eq. (11) is only applicable when there are more than two modalities. Next, we will discuss in detail the cir- cumstance where two modalities are given in the following.
(12)
of, if 2.
i=1,ik
Two-modal Prejudice Processing. In academic peer- review, unreliable reviewers can give unreasonable dis- like or preference for the author's work. Similarly, in multi- modal setting, reviewer modalities tend to have modality prejudice. we have more than two modalities, the modal- ity prejudice will be climinated by multiple reviewers and their trustworthiness modality weight learning. However, it can not be solved like this if only two modalities.
3.4. The Objective Function
Finally, with the above modality weight learning mecha- nism, we have the objective function with the popular infor- mation bottleneck realization follows
Fmoz(p(((3))] = Σ w²
(13)
1=1
Assume only two modalities, it inevitable that one modal- ity will be better or worse than another, so they may have different trustworthy scores. As shown in Figure 6(a), due to the single reviewer, the modality prejudice can not be eliminated, leading to unreasonable modality weight. Simi- lar to the scenario that the EIC/ / AE makes the final decision by directly adopting his/her own review instead of using the comments from the unreliable reviewer, it can learn the modality weight by directly leveraging the trustworthy score given by the final clustering result to the author modality, as shown in Figure 6(b). In this case, the peer-review score has no effect, as they are all the same value due to the symmetric scoring function.
when 8 E (0, +00) balances the information compression and preservation. Note that 3 equal +oc represents the extreme compression, and in this case it is parameter-free.
The advantages and possible weaknesses of the proposed method are discussed in Appendix B.
3.5. Optimization Method
We solve the optimization problem of PTIB with an effective sequential k-means-lite like draw-and -nd-rer algorithm (Lou et al., 2013; Hu et al., 2020; 2022), where each sample is sequentially drawn from the old cluster and assigned to

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
an optimal new cluster that minimizes the merger cost to maximize the objective function. It is shown as follows:
Table 1. Brief Information of the Datasets
Dataset
Type
Modality
Samples
Clusters
Weight Initialization. Modality weights are initialized with the initial peer-review and trustworthy scores.
20NG
Text
3
500
5
COIL.20
Image
3
1440
20
Random Clustering, The initial input, i.c. X, is randomly partitioned to [T] data clusters.
Event
Image
3
1579
8
Soccer
Image
3
280
17Flowers
Image
3
1360
17
Draw. Each sample x is sequentially drawn from its "old" cluster pold of the i-th modality, which is then taken as separate cluster {zr.. Now, it leads to [T] + 1 data clusters.
75Flowers
Image
2
5514
75
COIL100
Image
2
7200
100
MMI
Video
2
1760
22
Merger. To make the number of clusters recover to [T', a "new" cluster fnew is required to be selected from the existing data clusters for merger, which meanwhile satisfies the minimal cost for the separate cluster {ar) This ensures that the separate cluster } will be merged to the optimal candidate data cluster.
4.2. Results and Analysis
We present the comparison clustering results with of the-art methods on 8 multi-modal datasets in terms of Acc and NMI on Table 2 and 3. Observed from both the tables, we have the following discoveries.
Appendix A.2 shows a formal definition of the above "merger" and "merger cost" process. Appendix A.3 shows the details of algorithm and its computational complexity.
Single-modal VS All-modal. Intuitively, concatenating the features from different modalities may improve the clus- tering quality in comparison to the single-modal clustering. However, from both tables, the clustering results degrade sig- nificantly on the multi-modal COIL20 and Soccer datasets. For instance, compared to Neuts method on COIL20 dataset, the Acc and NMI values of Neuts-All decrease by 28.55 % and 26.08%, respectively. This clearly shows the instability of the all-modal methods and also reveals the necessity of clustering.
4. Experiments
4.1. Experiments Setup
Eight multi-modal datasets are used, including 20NG, COIL200, Event, Soccer, 17Flowers, 75Flowers, COIL100 and MMI. The brief information of them is summarized in Table 1 and their details are shown in the Appendix C.I.
Single/all-modae VS Compared multi-modal. Overall, the compared MMCs can beat the single/all-modal methods on most multi-modal datasets. Additionally, the second best results are always reached by the compared multi-modal method for all the datasets. However, is also seen that on some datasets the MMCs is inferior to the single/all-modal methods. This is probably because that some compared MMCs tend to be trapped into local optimal solution and thus lead to unsatisfactory results.
We compare with 4 traditional single-modal clustering methods, including KM, Neuts, KM-ALL, Neuts-ALL, and 13 state-of-the-art multi-modal clustering methods, in- cluding MVIB, Co(reg), MilB, RMSC, LMSC, MLAN, GMC, DMIB, FPMVS-CAG, MCMLE, TBGL, TIM and SMVAGC-SF: Their details are shown in the Appendix C.2.
For all the compared methods, we adopt the parameter value settings from their papers to attain the best clustering results with optimal parameter setting on each dataset. For the pro- posed method, we search the parameters from the settings in the following parameter analysis section. Afterwards, we conduct the experiments for 10 times and report the av- erage results and standard deviation in terms of Accuracy (Ace) and Normalized Mutual Information (NMI). All the compared methods and the proposed method are conducted in the same experimental environment, which is desktop computer with Windows 10 operating system, 32GB RAM., and MATLAB 20
Single/all- -odall VS Ours. The proposed method out- performs the single/all modall methods by a large margin all the involved multi-modal datasets. For a notable example, our PTIB obtains improvement of 57% and 42.07% in terms of Acc and NMI compared to the best values of singlc/all-mode methods on 20NG dataset. This phenomenon clearly demonstrates the superiority of the proposed method.
Compared multi-modal VS Ours. For the compared multi- modal IB-based methods, they have reached the second best results by two times on 17 flowers and MMI datasets. This mainly lies in the fact that the compared IB-based methods have the strong ability of the mutual information on the vari- able correlation quantization, while the remaining compared methods fail to do that. The compared multi-modal non- IB-based methods achieve the most second best clustering
Due to the limited space, the T-SNE visualization analysis is shown in Appendix C.3.
7

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Table Clustering Results (+/-Standard Deviation) on First Four Datasets with SOTA Methods ( denotes the best result, o denotes the second best.) )
ZING
COIL20
Event
Soccer
Method
Ace
NMI
Acc
NMI
Acc
NMI
Acc
NMI
KM
22.28 148
4,45+2.32
53.06:+33.00
65.06=2.12
33.93=4.10
19.84:2.00 09
25.82+45.06
18.70+7.97
Neuts (TPAMTOD)
42.80++2.40
27.65:+2001
7..69+11.30
84.01=0.54
34.10-11.28
14.9740.40
48.21+1114
45.024-2.21
KM-AII
21,46=00.68
1,76+00665
46. 14:6.88
60,70=4.51
28.85=2.29
11.37+2.10
22.46=33994
8. 8.14+3.59
Ncuts-All (0))
71.20++0.17
57.234-0.10 10
46. 14.14-0.52
57.93-0.23
35.06+00.69
20.
39.75-40.94
34.04+00.57
MVIB (DASFAA'07)
94,22
83.21 18
10.74
73.65=6.63
40.02+2.04
13.71=1.56
35.79:3.96
21.42:44225
Co(reg) (N
20.02:+0.62
3.15+0055
64.33+-1.68
83.79-0.45
38.58-0.92
24.30+0.55
24.13++0.53
11,43+-0.99
MΩB (UCAA113)
93.76.12.89
14.54
13,81
92.39-1199
48.5811.50
13.41.11.35
$3.64.12.76
49.74.3..4
RMSC (AAAT14)
17.26:10.91
05.70:10.84
65.43-1..31
2..56
36.58 1.26
21.02-10.88
28.96-11.90
12.16:12.18
LMSC (CVPR'17)
16.16:10
15.3711154
719412272
82.18=2.337
43.9222.84
27.53:2.558
31.25:16.53
15.8511.71
MLAN (TIP'18)
%644==0111
19.18+0.11
87,22=2.30
14
19.90=0.72
6,66=0.80
28.21=0.01
21,27+0.
GMC (TKDF220)
98.20++0.00
97.92-40.00
60,90+0.00
84.67-0.00
18.1 18.11-+0.00
10.74++0.00
29.29++0.00
25. 25.82-00.00
DMIB (TCYB'22)
98.30=0.114
97.56+0.49
65.90+44.03
77.70=2.46
80-3.02
32.97-22338
50.68++2.23 54.07+43.67
FPMVS CAG (TIP"22)
13.80
59.23
19.17
85.11
48.8910.00
31.99.1.00
50.14.10.00
49.56100.00
MCMLE (I)
77. 07.40:20.00
69.96110000
85
93.48=0.00
44.46:0.000
30.2421.000
56.07700.00 P
50.06110000
TBGL (TPAMI 23)
89.11=0.00
83,45+0.00
86 10=0.00
92.41=0.00
42.84=0.00
28. 28.40:0.00
54.39+0.000
49.78+0000
TIM (T1P'23)
99.40=0.00
98.08=0.00
56.7014.08
71.39=0.29
54.60=2.50
36.8621.77
48,93:0.51
41.42144099
SMVAGC-SF (TIP'24)
86,07=66.40
72.61 +3.59
75.66+55.10 10
89,43=2.11
1..76+1.270
36,97-0.65
15,14+1.56
1..61++1855
PTIB
0.000=0.000
99,30:20.00
93.33=00000
96,4620.00
60.24::016 -
45.36:10.28
62.8610017
53.23:1.16-
Improve (+ VS
0.40 (?)
1,22 (t)
6.11 (t)
2.11 (t)
5.48 (t)
8.39
(.79
2.55 (1)
Table Clustering Results (t/-Standard Deviation) on Remaining Four Datasets with SOTA Methods (+ the second best.)
denotes the best result, denotes
17Flowers
75Flowers
COILI00
MMI
Method
Ace
NMT
Acc
NMI
Acc
NMI
Acc
NMI
KM
22.41=11.67
24.31+1.14
19.48:40.85
35,21=0.75
27.96=1.78
58. 18.13:11.52
26,89:22.95
44.15+1160
Neuts (TPAMPOD)
27.71+40.72
26.43++0.40
24.80++0.58
41.50-0.19
40.97+11.28
58.5244.59
38.43-40.47
53,17+0.43
KM-AII
17.63=#1.27
13.55++1.86
21.13#0.88
32,57=0.71
29.25=1.57
50.55=22.15
27.11=1181
38
Ncuts-All (TPAMF'0)
28.77+40.63
26.31
27.41+0.31
42.41-0.21
48.63++0.97
64.74-0..56
40.53++1.52
52.77+0..2
MVIB (DASFAA'07)
21.32:#1.05
18.28:11488
18.49:0.61
33.05=0.45
46.71=2230
70.29: 1.10
44.95+2..0 e
$4.65=11.49
Co(reg) (NeurIPS'II)
26.28-+0.49
27. 12:40200
28. 16+0036
44.95=0.09
48.35+0044
70.86++0.15
34.72+00.53
+0,31
MMB (JCAT
38,52+22.08
37,24=1.40
24.57=00.2
40,79=0.37
50.52+0.08 08
72881=0446
40,14+2.09
12.50+11.69
RMSC (AAAT14)
19.70-00666
17.86-+0.38
26.42-40.97
42.95-0.30
46.32-00.8
69.33-+0.45
30.28++1.00
43.94++0.89
LMSC (CVPR'17)
33.29:12.29
31.4911.60
24.58.0.90
42.50-0.59
48.7611.45
66.7411.85
40.1711.88
51.6211.29
MLAN (TIP'18)
24.32=1191
22.21+11.24
23.58=0.53
34.16=1.15
45.05=0.41
09.55=00.33
18.15=0.05
52.68+0.04
GMC (MCCCTDE'S (TKDE'20) 20)
6.76=0.00
4,78:10.00
18.52-10.00
30.06=0.00
38.66200000
67.55:20.00
35.60:20.00
55.65=00000 o
DMIB (TCYB'22) 22)
35.48:26.04
32.56+5.447
26.72-+1.13
43.13=00.79
50.33+1 88
72.57=00.77
41.10=2..6
52.96+2.10
FPMVS- CAG (2))
30.51-10.00
27.27
23.83+00.00
38.24-0.00
45.03--0.00
7..58+10.00
36.77+0.000
51.03-+0.00
MCMLE (TPAMI22)
32.13:10.00
32.11110.00
28.76:10.00
47.03-0.000
50.47:00.00
74.59.1.000
42.04:10.00
52.9710.000
TBGL (TPAMI22)
31.07.1..00
32.46.1..00
26.52.10.00
47.0910.00
50.66
67.82.10.00
43.15.10.00
13.27
TIM (TIP'23)
32.9822.228
29.3613.60
21.320.60
26.23=1.24
514421.72
74.9820.70
28.98111.57
39,56222.9
SMVAGC-SF (TIP'24)
42.41=2.07=
36.40+1.43
31.92+0.633
46,89=0.22
56.78+11.33.
76,78=0.55p
40,93+2,14
13,03++1225
PTIB
45.29:0.055
42.49-10.08
38.73-0.366
51.91:0,20
61.1710.23 *
82.86:10.19
48.30
60.39:10.49
Improve (+ o)
2.88 (?)
5.25
3.81 (t)
4.82 (1)
4.39 (1)
6.08 (f)
3,35
4.74
results, but still underperform the proposed method.
4.3. Parameter Analysis
Improvement Analysis. Compared with the TIM, MLAN, SMVAGC-SE MCMLE and MVIB that obtained the sec- ond best results, the proposed method has attained notable improvements about 0.4%, 6.11%, 5.48%, 6.79%, 2.88%, 3.81 %, 4.39% and 3.35% on the eight multi-modal datasets in terms of Acc values respectively. For weighted MMCs, the trustworthiness of the learned modality weights is sig- nificant for ensuring better clustering performance. The proposed method explicitly considers this vital factor by jointly incorporating the peer-review and trustworthy score.
There is only one parameter / in the proposed PTIB method, where we give the parameter settings as [10, 50, 100, 500, 700, 900, 1000). Then we conduct exten- sive experiments on all the datasets with different settings to investigate the parameter sensitivity, and show the clustering performance with both Ace and NMI results in Figure 7.
From the figure, can clearly obtain the best parameter set- tings corresponding to the optimal clustering results for dif- ferent datasets, which are [10, 50, 100, 500, 700, 900, 1000]. [500, 700, 900, 1000], [50]. [50, 100), [700, 900, 1000].
8

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
20NG
COIL20
100
98
60
Clustoring Ragalt (%)
69
09.
04
99.6
50
92
09.4
10
88
Aue NMI
NXI
NMI
20
99
86
3
17Flowers
75Plowers
COIL100
MMI
Chustrrig Resth (%))
45
50
80
-5
70
4u
60
40
30
Acc KMI
NMI
NMI
30
8
Figure 7. Parameter study of our PTIB method eight multi-modal datasets, Note that the values in the horizontal axis indicate the seven parameters [10, 50, 100, 500, 700, 900, 1000].
the table, we observe that the versus margin value always reaches roughly -2%, and leads to about -4% on only few datasets (e.g., obtaining - -6.87% on COIL20 datasets in terms of Acc value). This phenomenon demonstrates that the proposed method has a huge potential for more real- world applications, especially in the fields where we hardly
Table Clustering Results (ta-Sraddadd Deviation) multi- modal Datasets with Faremeter-free Version
PTBB
Parameter-five PTIB
Versus Margin
Datasets
Acc
NMI
Acc
NMI
Acc
NMI
20NG
99.80=00000
09.00=0.00
99.80=0.00
99.30=0.01
0.00
0.00
COIL.0
93.33-1:0.00
96.46:0..00
86.46-0.00
93.80-00.00
-6.87
-2.66
Eveat
60.24:0.11
15.36=0.28
59.01=0.62
44.39=0.50
-1.23
-0.97
tune the parameters without any prior knowledge.
Soccer
62.86:20.17 53.23=00.16
59.64=0000
51.65=0001
-3.22
-1.58
17Flowers
45.29:10.05 42.49 12.49.00.08
42.74.1.38
40.92.10.22
.2.55
-1.57
75Flnwers
35.73:10.36 50.20
14.57=0.36
51.23:10.24
-1.16
-0.68
5. Conclusion
COILI00
61.17=20.23 82 09
59.93=0.61
82.24=0.30
-1.24
-0.62
MMI
48.30=#0.80 60.39=0044
44.26=0.01
58.38=0000
-4.04
-2.01
Motivating by the peer-review mechanism in academia, in this article we propose a novel peer-review trustworthy infor- mation bottleneck (PTIB) method for addressing weighted multi-moda clustering problem. PTIB measures the modal- ity correlations by iteratively learning the peer-review score of each modality, which works by alternately playing the role of "author" and "reviewer" for each modality. Addition- ally, we further design trustworthy score for improving the reliability of the learned peer-review score, thus leading to more accurate modalities weights. Rich experiments on 8 datasets reveal the superiority and effectiveness of the pro- posed PTIB. The proposed method also with some possi- ble weaknesses. It is designed for fully aligned molti-modal clustering and complete multi-modal clustering, where none of the data samples across modalitin modalities are unaligned, missing, or damaged. And it requires the number of clusters of the dataset in advance, like almost all the existing multi-modal clustering methods. future, we will focus on solving the discussed possible weaknesses of PTIB and apply it into more real-world applications. Moreover, this "peer-review" idea may be considered using for other problems, such as federated learning.
[500]. [50), and [100], respectively. It is observed that there are more than one best parameters for most datasets, such as 20NG, COIL20, and 17Flowers. Additionally, for all the datasets, the better results are obtained approximately from the range [500, 700, 900, 1000). which is a wide range for practical parameter selection. All these observations reveal that the parameter tuning in practice is not a heavy burden and also shows the great potential of the proposed method for real-world applications.
4.4. Potential for Parameter-free Version
To further investigate the potential of the proposed method in practical applications, we make parameter 3 as +00, leading to an elegant parameter-free version of our PTIB, formulated as Fmaarppd [pt[==] Em w². I Y*). Then we conduct series of experiments involved eight multi- modal datasets, as shown in Table
We also add a new column called versus margin, using the value of parameter-fi free version minus that of the PTIB method to compare the difference between them. From

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Acknowledgements
Hu, S., Yan, X., and Ye, Y. Joint specific and correlated information exploration for multi-view action clustering. Inf. Sci., 524: 2020.
The authors thank anonymous reviewers for their construc- tive comments. This work was supported by National Nat- ural Science Foundation of China under Grant 62206254, Henan Province Outstanding Youth Science Fund Program under Grant 252300421 and China Postdoctoral Science Foundation under Grant 2024T1
Hu, S., Shi, Z., and Ye, Y. DMIB: dual-correlated multi- variate information bottleneck for multiview clustering. IEEE Trans. Cybern., 52(6):4200-4274 2022.
2024T170843 70843 and 2023M743186.
Hu, S., Lou, Z., Yan, X., and Ye, Y. A survey on information bottleneck. IEEE TPAMI, 46(8):5335-5334 2024b.
Impact Statement
Hu, S., Fan, J., Zou, G., and Ye, Y. Multi-aspect self self-guided deep information bottleneck for multi-modal clustering. In AAAI, pp.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
2025.
Huang, Z., Ren, Y., Pu, X., Huang, S., Xu, Z., and He, Self-supervised graph attention networks for deep weighted multi-view clustering. In AAAI, pp. 7936-7943, 2023.
References
Bay, H., Tuytelaars, T., and Gool, L. V. SURF: speeded up robust features. In ECCV, pp. 404-417, 2006.
Khan, F. S., van de Weijer, J., and Vanrell, M. Top-down color attention for object recognition. ICCV, pp. 979- 986, 2009.
Chen, M., Wang, C., and Lai, Low-rank tensor based prox- imity learning for multi-view clustering. IEEE TKDE, 35 (5):5076-5099 2023.
Kumar, A., Rai. P., and III, H. D. Co-regularized multi-view spectral clustering. In NeurlPS, pp. 1413-1421, 2011. 2011
Cover, T. M. and Thomas, J. A. Elements of information theory. Wiley, 2006.
Li, Z.,, Tang, C.,, Liu, X., Zheng, X., Zhang, W., and Zhu, E. Consensus graph learning for multi-view clustering. IEEE TMM, 24:2461-2477 2022.
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view infor- mation bottleneck. In ICLR, 2020.
Liang. W., Zhu, E., Yu, S., Xu, H., Zhu, X., and Liu, X. Scalable multiple kernel clustering: Learning clustering structure from expectation. In ICML, 2024.
Fei-Fei, L. and Perona, P. A bayesian hierarchical model for learning natural scene categories. In CVPR, pp. 524-531, 2005.
Lin, J. Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theor., 37(1):145 151, 2006.
Gao, Y., Gu, S., Li, J., and Liao, Z. The multi-view infor- mation bottleneck clustering. In DASFAA, pp. 912-917, 2007.
Liu, X., Liu, L., Liao, Q., Wang. S., Zhang, Y., Tu, W., Tang, C., Liu, and Zhu, E. One pass late fusion multi-view clustering. In ICML, pp. 6850-6859, 2021.
Gu, Z., Li, Z., and Feng. S. EDISON: enhanced dictionary- induced tensorized incomplete multi-view clustering with gaussian error rank minimization. ICML, 2024.
Lou, Z.. Ye, Y., and Yan, X. The tion bottleneck with application to unsupervised image categorization. In IJCAI, pp.
informa-
2013.
Han, Z., Zhang, C., Fu, H., and Zhou, J. T. Trusted multi- view classification. In ICLR, 2021.
Lowe, D. G. Distinctive image features from scale- iave-invariant keypoints. IJCV, 60(2)91-110, 2004.
Han, Z., Zhang, C., Fu, H., and Zhou, J. T. Trusted multi- view classification with dynamic evidential fusion. IEEE TPAMI, 45(2):2551-256 2023.
Nie, F., Cai, G., Li, J., and Li, X. Auto-weighted multi- view learning for image clustering and semi-supervised classification. IEEE TIP, 27(3):10111151, 2018.
He, Y. and Atia, K. Multi-mode tensor space clustering based on low-tensor-rank representation. In AAAI, pp. 4893-6901, 2022.
Raya, S., Orabi, M., Afyouni, I., and Aghbari, Z. A. Multi- modal data clustering using deep learning: A systematic review. Nearocomputing, 607:128348, 2024.
Hu, J., Yang, C., Huang, K., Wang, H., Peng, B., and Li, T. Information bottleneck fusion for deep multi-view clustering. KBS, 289:1 1551, 2024a.
Shi, J. and Malik, J. Normalized cuts and image segmenta- tion. IEEE TPAMI, 22(8))888 888-8058888888808888880811011 2000.
10

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. In Proc. Annual Allerton Conf. Communnication Control Computing, pp. 368-377, 1999.
Zhang, C., Hu, Q. Fu, H., Zhu, P., and Cao, X. Latent multi-view subspace clustering. In CVPR, pp. 4333-4341, 2017.
Wang, H., Yang, Y., and Liu, B. GMC: graph-based multi- view clustering. IEEE TKDE, (2(6):1116-112) 2020a.
Zhang. C.,, Lou, Z., Zhou, Q. and Hu, S. Multi-view clus- tering via triplex information maximization. IEEE TIP. 32:4299-4313, 2023.
Wang, Q. Chen, M., Nie, F., and Li, X. Detecting coherent groups in crowd scenes by multiview clustering. IEEE TPAMI, 4261))46-58, 2020b.
Zhang. C., Xue, H., Nie, K., Wu, X., Lou, Z., Yang, S., Zhou, Q., and Hu, S. Nice to meet images with big clusters and features: A cluster-weighted multi-modal co-clustering method. IPM, 61(5):103735, 2024.
Wang, S., Liu, X., Zhu, X., Zhang, P., Zhang, Y., Gao, F., and Zhu, E. Fast parameter-free multi-view subspace clustering with consensus anchor guidance. IEEE TIP, 31:556-568, 2022.
Zhang, Z., Liu, L.,, Shen, F., Shen, H. T., and Shao, L. Binary multi-view clustering. IEEE TPAMI, 4177):7744-77 2019.
Wang, S., Liu, X., Liu, S., Tu, W., and Zhu, E. Scalable and structural multi-view graph clustering with adaptive anchor fusion. IEEE TIP, 33:4627 46346627-669,,,, 2024.
Zhao, Q.. Zong, L., Zhang, X., Liu, X., and Yu, H. Multi- view clustering via clusterwise weights learning. KBS, 193:105459, 2020.
Wen, J., Zhang, Z., Fei, L., Zhang, B., Xu, Y., Zhang, Z., and Li, A survey on incomplete multiview clustering. IEEE Trans. Syst. Man Cybern. Syst., 1362)11 2023.
Zheng, X., Tang. C., Wan, Z., Hu, C., and Zhang, W. Multi- level confidence learning for trustworthy multimodal clas- sification. In AAAI, pp. 11381-1138) 2023.
Zhong, G. and Pun, C. Improved normalized cut for multi- view clustering. IEEE TPAMI, 44(12):10244-10251, 2022.
Wen, J., Xu, G., Tang, Z., Wang, W., Fei, L., and Xu, Y. Graph regularized and feature aware matrix factoriza- tion for robust incomplete multi-view clustering. IEEE TCSVT, 34(5):3728-3744 - 2024.
Zhou, R. and Shen, Y. End-to-end adversarial-attention network for multi-modal clustering, In CVPR, pp. 14607- 14616, 2020.
Wolf, L.,, Hassner, T., and Taigman, Y. Descriptor based methods in the wild. In Real- Life Images workshop at the Eur. Conf. Computer Vis., 2008
Zou, X., Tang, C., Zheng, X., Li, Z., He, X., An, S., and Liu, X. DPNET: dynamic poly-attention network for trustworthy multi-modal classification. In ACM MM, pp. 3550-3559, 2023.
Xia, R., Pan, Y., Du, L., and Yin, J. Robust multi-view spectral clustering via low-rank and sparse decomposition. In AAAI, pp. 2149-2155, 2014.
Xia, W., Gao, Q. Wang. Q., Gao, X., Ding, C., and Tao, D. Tensorized bipartite graph learning for multi-view clustering. IEEE TPAMI, 45(4):5187-5202, 2023a.
Xia, W., Wang, T., Gao, Q. Yang, M., and Gao, X. Graph embedding contrastive multi-modal representation learn- ing for clustering. IEEE TIP, 32:1 10011701111 1183, 2023b.
Xu, J., Ren, Y., Tang, H., Yang. Z., Pan, L., Yang, Y., Pu, X., Yu, P. S., and He, L. Self-supervised discriminative feature learning for deep nulti-view clustering. IEEE TKDE, 3577) 2023.
Xu, Y., Wang, C., and Lai, J. Weighted multi view clustering with feature selection. PR, 53:25-35, 2016.
Yan, X.,
Mao, Y., Ye, Y., and Yu, H. Cross-modal cluster- ing with deep correlated information bottleneck method. IEEE TNNLS, 35(10): (3510)13508-11 1350--1352, 2024.

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
A. Theorem Proof and Optimization
A.1. Proof of Theorem 3.3
Proof. Given a dataset with its true category of clusters and a specific clustering result, for a certain cluster, the probability
of its major category is denoted as and the sum of the probabilities of minor categories is a, leading to the first probability distribution {a, 1 a). It is assumed that the minor categories obey an arbitrary probability distribution B ), i.e., E, by
(br, , ,,,bb)1) Then, we use the entropy of the second distribution to minus that of the first distribution as follows
leading the second probability distribution {a, b1 (1 a), h2(1 -......, bo a))
H2((a, b1 (1 - 62. ................ a))) H1 (1a, 1 -aa)
60-1
0-1
--lloge a
Σ
(1 - -a]] logg (1 a]] + a logz a + (1 - a) logz - a)
5=1
-(1 a) log2(1 a)
[by(1 a) 000g bj + log2(1 - -a)]
0-1
0-1
=(1 a) logz(1 aa) 1
(1-
-a)by log2 by +
by (1 a) log2 a)]
(14)
1=1
==1
0-1
-11 a) logg(1 - a) - (1 - a)
by loga bj - (1 - a) logz (1 - a)
0-1
=(1 a){- Σ
by logg by]
5=1
-(1 (B)
- 1). which clearly shows that, the more mixed the minor categories are, the higher the entropy of it gets, thus resulting in high uncertainty of the cluster. Actually, (B) is the noisy and uncertain information that the minor categories convey, which interferes with the judgment of useful information.
Since 1 1 a 2 0 and 0 < HBB) log2 1), we have 0 S H2 H1 S (1- (1-a)loga (0
This proves the theorem.
A.2. The Formal Definition of the Optimization Process
Proposition A.1. [Merger] (Hu et al., 2020) If the separate cluster {Ø} is merged into one specific cluster t in the i-th modality, a new merged cluster reached, named f. This process is formulated as the following:
p(f) = p(z) + p(t)
(15)
p(y'lt) mi ply" (x) + π2 p(z"(t)
where ply (x:) indicates the feature conditional distribution of the i-th modality, p(y indicates the cluster centroid of the i-th modality, and the merger function II = {TT1, m2)
(p(r) pit) }. p(8) p(t)
Proof. The detailed proof can be referred from the Proposition I in the work (Hu et al., 2020).
To obtain the maximal value of the function (13),,,, we attempt to select the optimal cluster tne for each merger, and ensure the "merger cost" i.e., the value change of function Eg.() always minimal. This is formulated by
new arg min min(AFFFF))) (AFaa)) = arg min(FFhef Faft )
(16)
maz
man
where Fbef and mar indicate the value of (13) before and after the merger process respectively.
Actually, the above formulation has a significant impact on whether a good or bad new cluster tner is selected. We thus have the merger cost formulation with the following theorem.
12

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Algorithm 1 The Proposed PTIB
1: Input: m joint distributions (p(X, Y*) the number of clusters [T), the balance parameter B.
2: Output: Final clustering result p(t|æ=).
3: Modality Weight Initialization; Compute the initial modality weights with initial peer-review and trustworthy score;
4: Random Clustering: T t- Random partition of X into [T'] clusters;
5: repeat
6: for all x E X' do
7: Draw: Draw x: from the "old" cluster gold become separate cluster {x);
8: Merger: Select "new" cluster (new for the separate cluster {ær to merge corresponding to the minimal merger
cost in Theorem A.2;
9: end for
10: Update the trustworthy score using the clustering result in each iteration;
Update the weight for each modality;
until Samples in different clusters remain unchanged fixed number of iterations.
Theorem A.2. [Merger Cost] (Ilu al., 2020) Given two clusters {x} and t, we have the merger cost as
AF max ((x), p(t) dist((x),t).
(17)
where
dist({x),1) Σ w/ (STTS -- --------Snnnnoo))))
where JS is the Censen-Shannoe divergence (Lin, 2006).
Proof. The detailed proof can be referred from the Theorem the work (Hu al., 2020).
A.3. Algorithm and Computational Complexity
The Algorithm 1 shows the details of the optimization process and we investigate the computational complexity of it. At step 8 in Algorithm 1, given sample x, we have the merger cost as AF (fz), t) for every "new" cluster t to reach the minimal one, and it takes ([/T)||(((X))) | |Y unchanged or after fixed number of iterations, it takes O(rIT])XX(( [(X |Y? + 1)), where is the number of repetitions. Generally, the number of clusters is treated as constant. Hence, the overall computational complexity takes O(r X|XX)( |Y y))))).
[Y*)). When the samples in different clusters remain
B. Discussion
In this section, we first discuss and show the differences or advantages of the proposed method with existing clustering methods. Then, we further analyze the weaknesses of our method.
The main strengths of the PTIB method in comparison with existing methods are as follows.
Trustworthy weight learning. To our knowledge, none of the existing weighted MMCs employ the trustworthy strategy in the weight learning process, which may probably leads to inaccurate modality weights. Unlike them, we attempt to learn trustworthy modality weights in an iterative optimization process.
Correlation quantization based learning. We in this paper focus modality correlation quantization using mutual information, while most related existing methods generally measure themselves for weight learning, e.g., using objective function value of each modality for complementary information learning.
Parameter-free weight learning. Most weighted MMCs usually adopt one or more regularization parameters to control the learned weight distribution, where the parameters are difficult to tune in practice. In contrast, the weight learning process in this paper completely parameter- free without need any prior knowledge.
13

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
20
H
Socoar
17Flowers
MMI
Figure 8. Some typical images and videos from COIL20, Soccer, 17Flowers and MMI datasets.
Self-supervision mechanism. Self-supervised learning mechanism (Xu et al., 2023) incorporated into the proposed method for guiding the modality weight learning. Thus, in this way both the clustering structure learning and the weight learning can mutually benefit from win-win cooperation.
Some possible weaknesses of the proposed method are revealed in the following.
Fully aligned multi-modal clustering. The proposed method works under the assumption that each data sample across different modalities is fully aligned. For example of the multi-fcature images, the first sample described with one kind of feature, e.g., shape, must be aligned with the same sample described with another feature, e.g., color.
Complete multi-modal clustering. The proposed method can only solve the complete multi-modal clustering problem where none of the data samples across modalities are missing or damaged. Incomplete multi-modal clustering (Wen et al., 2023; 2024) or its self-supervised version (Huang et al., 2023) has attracted lots of attention and is worth considering in the future.
Given number clusters advance. Like almost all the existing multi-moda. clustering methods, the proposed method requires the number of clusters of the dataset in advance, which may limit its wide applications in completely unknown areas.
C. More Experimental Details
C.1. The Detail of Datasets
Here, we provide detailed description of the dataset used in the experimental part of this paper and exemplar samples from the image and video data shown in Figure 8.
20NG dataset is composed of 500 newsgroup documents extracted from the 20 Newsgroup dataset. Three different pre-processing methods provide the modalities for each document.
COIL20 dataset has 1440 images about 20 objects, where each object has 72 images taken at 5 intervals in its 360 horizontal rotation. Three features are adopted for shape, color and texture representation, i.e., SIFT (Lowe, 2004), Color Attention (Khan al., 2009), and TPLBP (Wolf al., 2008) respectively. Every feature represents one single modality.
Event dataset contains eight kinds of sports event classes with 1579 images. The features are the same as those of the COIL20 dataset.
ca. http://wwwwaaaaaaaa///////////aa httt::///isao...aafffrraaaaaaaas
1-20 1-20.php
14

---

A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method
Soccer dataset contains 280 images of 7 soccer teams captured from the websites. The features extracted by three ways, including SIFT (Lowe, 2004), Color Attention (Khan et al., 2009), and TPLBP (Wolf al., 2008), are used as three modalities.
17Flowers dataset $ consists of 1360 images of flowers belonging to 17 different classes, each of which has 80 images. Three kinds of features, i.e., SURF (Bay et al., 2006), Color Attention (Khan et al., 2009), and TPLBP (Wolf et al., 2008), are used as three modalities. This dataset is challenging for clustering because there are classes with large variations within each class and close similarity across classes.
75Flowers dataset is selected from the 102 Flowers dataset. The images contain large variation the posture and the light. This dataset contains two modalities by using SIFT (Lowe, 2004) and Color Attention (Khan et al., 2009) as shape and color feature extractors respectively.
COIL100 dataset consists of images from 100 objects, where each object has 72 images. We use two features extracted by SIFT (Lowe, 2004) and SURF (Bay et al., 2006) as two modalities.
MMI dataset has 1760 samples with challenging 22 multi-modal (RGB, depth and Skeleton) and multi-modal (front and side modalities) interactive human actions, which are collected in cluttered and unclear places. Note that we adopt the RGB action videos with two modalities for experiments.
C.2. The Detail of Comparison Methods
1. KM, Neuts (Shi & Malik, 2000): KM (k-means) and Neuts traditional single-modal clustering methods, and the best results are reported among different modalities for each dataset.
2. KM-AII,, Neuts-All: Both of the methods are built by applying their single-moda version the multi-modal datasets
with concatenated features.
3. MVTB (Gao et al., 2007): It is the first multi-view IB method proposed to address the document clustering problems from the websites, working by designing a compatible constraint ensure the consistency among view assignments.
4. Co(reg) (Kumar al., 2011): It co-regularizes the data clustering hypotheses among views to learn consistent assignments based on spectral model.
5. MfIB (Lou al., 2013): It is a weighted multi-feature IB method designed for solving the unsupervised image classification, where the weights set manually.
6. RMSC (Xia et al., 2014): It solves the noisy multi-view clustering problem by designing a robust spectral method.
7. LMSC (Zhang et al., 2017): It learns latent shared representations among views make the feature subspace more robust and accurate.
8. MILNN (Nie et al., 2018): It jointly learns the local structure and clustering assignments, and then automatically tunes the view weights without using parameters.
9. GMC (Wang et al., 2020a): It is a mash- weighted multi niew clustering method by automatically tuning the algorithm parameters.
10. DMIB (Hu et al.. 2022): It jointly takes account into the dual correlations about the cross-feature and cross-cluster view correlations for multi-view clustering based IB theory.
11. FPMVS-CAG (Wang et al., 2022): It deals with the multi-view subspace clustering problem by a fast parameter-free free method with the guidance of selected consensus anchors.
12. MCMLE (Zhong& & Pun, 2022): improves the traditional Neuts method for multi-view clustering by Laplacian embedding to learn shared binary assignment matrix among different modalities.
htm1
1-
hhttp://media.fe
htt.:/pp......mi...e.mm. wtw.c///iiii... html
15

---

A Peer-review Look Multi-modal Clustering: An Information Bottleneck Realization Method
20NG:
20NG: View 2
20NG: View
COIL20: View2
COIL20: View3
MML Viewl
MMI: View2
Figure 9. T-sne visualization results on 20NG, COIL20 and MMI datasets.
13. TBGL (Xia et al., 2023a): It focuses on learning tensorized bipartite graphs for clustering multi-view datasets by simultaneously considering the intra/inter- -ieww similarities.
14. TIM (Zhang al., 2023): is information-theorrtiee. method for solving the multi-view clustering problem, and works by following three principles, i.c., contained, complementary and compatible principle.
15. SMVAGC-SF (Wang et al., 2024): It jointly optimizes anchor graph construction and graph alignment, and adaptively fuses multiple anchor graphs with different magnitudes to improve the quality of multi-view clustering.
C.3. T-SNE Visualization Analysis
To further illustrate the learned clustering structure, vividly show the t-SNE visualization of the clustering results in Figure 9 with three typical multi-modal datasets, i.e., 2ONG, COIL20 and MMI. From this figure, it is observed that the visualization of most modalities from the involved datasets illustrate relatively compact and separated clustering structure. For typical example of the modality 2 and 3 in COIL20 dataset, the data clusters with different colors are quite clear, and data samples in most clusters are densely distributed while samples from different clusters are in a long distance.
16


Reviews:
[{"review_id": "EImiGCmnFV", "reviewer": "Reviewer_ZBoD", "summary": "This paper proposes a new peer-review trustworthy information bottleneck method. It designs a multimodal peer review process, in which the modality will iteratively act as an \"author\" or \"reviewer\" to conduct peer review to explore the potential relationship, which is quantified as the peer-review score; and the trustworthiness of the modality as a \"reviewer\" is judged in a self-supervised manner. Extensive experiments on 8 datasets show that the proposed method is superior to existing cutting-edge methods in terms of clustering performance indicators.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below.\n\n**1. Generally, in a peer-review process, an EiC is also involved. I am interesting about which part stands for the EiC role in the clustering process.**\n\n***Response:*** Thanks for your comments. Indeed, an EIC is also involved in an academic peer-review process and we argue that both the EIC and AE can play the same supervisory role. This is consistent with our intention that use the final clustering result to ensure the trustworthiness of the peer-review score in a self-supervision fashion. In the submitted manuscript, AE is mentioned to help readers better understand how to self-supervise and evaluate the trustworthiness of the multimodal peer-review process. In summary, the final clustering result can be regarded as EIC/AE.\n\n**2. The author should detail the ‘k-means-like draw-and-merger algorithm’ in Sec 3.5 using one or two sentences to enhance clarity and readability.**\n\n***Response:*** Thanks for your comments. The detail of the ‘k-means-like draw-and-merger algorithm’ is that Each sample is sequentially drawn from the old cluster and assigned to an optimal new cluster that minimizes the merger cost to maximize the objective function.\nIn the following, we will outline the draw-and-merger algorithm and the optimization process of k-means, demonstrating why the draw-and-merger algorithm works like k-means.\nThe optimization process of $k$-means mainly includes the following steps:\n\n(a) Initialize $k$ centroids by randomly selecting $k$ samples.\n\n(b) Assign each data point to the optimal new cluster corresponding to the nearest centroid, which will reduce the $k$-means loss (i.e. Sum of Squared Errors, SSE), and recalculate the centroid of each cluster.\n\n(c) Loop through step (b) until convergence or the maximum number of iterations is reached.\n\nThe draw-and-merger algorithm mainly includes the following steps:\n\n(1) Initialize $k$ clusters by randomly assigning samples.\n\n(2) Sequentially reassign each data point to the optimal new cluster corresponding to the minimum merger cost.\n\n(3) Loop through step (2) until convergence or the maximum number of iterations is reached.\n\nObviously, the steps of both algorithms are similar, and their key step is to reassign each data point to the optimal new cluster. The difference is that the draw-and-merger algorithm reduces the computational complexity by formalizing the merging loss.\n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "The claims in the manuscript are supported by the clear and convincing evidence from the extensive experiments and the results from different aspects.", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria (including the datasets, compared methods, performance indicators) make sense for the problem.", "theoretical_claims": "I have checked the correctness of proofs for theorem 3.3 and its proof in the manuscript and appendix.", "experimental_designs_or_analyses": "I have checked the soundness/validity of the experimental designs or analyses from the Sec 4.", "relation_to_broader_scientific_literature": "This paper proposes a new peer-review trustworthy information bottleneck method, which is the key contribution of this manuscript.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The authors provide a clear explanation of the proposed method, enhancing the reader's understanding of its importance. The paper presents a clear motivation and is well-structured, which also offers some valuable insights. My comments are shown as: First, generally, in a peer-review process, an EiC is also involved. I am interesting about which part stands for the EiC role in the clustering process. Second, the author should detail the ‘k-means-like draw-and-merger algorithm’ in Sec 3.5 using one or two sentences to enhance clarity and readability.", "other_comments_or_suggestions": "Refer to the weakness", "questions_for_authors": "Refer to the weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W0HBO69ctw", "reviewer": "Reviewer_Qckw", "summary": "Most existing methods in multimodal clustering have three core challenges on the trustworthiness, weight learning and parameter learning. Motivated by the peer-review mechanism, this paper solves the multimodal clustering problem and realizes mutual review of different modalities by rotating the roles of \"author\" and \"reviewer\" to explore the potential relationship. Moreover, a trustworthy score is further given in a self-supervised manner. By considering the two aspects, a new peer-review trustworthy information bottleneck method is proposed. Comparative experiments on eight multimodal datasets show the proposed method outperforms existing state-of-the-art methods.\n\n## update after rebuttal\n\nThe authors have addressed my most concerns and I keep my rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below. \n\n**1. Lack of discussion on existing trustworthy multimodal clustering approaches. Are there prior weighted multimodal clustering methods that incorporate trustworthiness in weight learning?**\n\n***Response：*** Thanks for your comments. Regarding the trustworthiness of multiple modalities, almost all existing methods [1-4] focus on trustworthy multimodal classification. To ensure the reliability of both the multi-modal integration and the final decision, the trustworthy multimodal classification methods produce a stable and reasonable uncertainty estimation for each modality and thus promote both classification reliability and robustness. For example, Han et al. [2] introduce the variational Dirichlet to characterize the distribution of the class probabilities, parameterized with evidence from different views and integrated with the Dempster-Shafer theory, thus promoting both classification reliability and robustness. Zheng et al. [3] propose a trustworthy multimodal classification network via multi-level confidence learning which integrates both feature and label-level confidence learning for trustworthy multimodal classification. Zou et al. [4] induce a transparent fusion strategy based on the modality confidence estimation strategy to track information variation within different modalities for dynamical fusion.\n\nDifferent from them, the proposed method aims to guarantee the trustworthy of the learned modal weights in a self-supervised manner. To the best of our knowledge, none of the existing weighted MMCs employ the trustworthy strategy in the weight learning process.\n\n[1] Z. Han, C. Zhang, H. Fu, and J. T. Zhou, Trusted multi-view classification, in Proceedings of the International Conference on Learning Representations, 2021, pp. 1–11.\n\n[2] Han, Z., Zhang, C., Fu, H., and Zhou, J. T. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on pattern analysis and machine intelligence,2023, 45(2), 2551-2566.\n\n[3] Zheng, X., Tang, C., Wan, Z., Hu, C., and Zhang, W. Multi-level confidence learning for trustworthy multimodal classification. In Proceedings of the AAAI conference on artificial intelligence,2023, pp. 11381-11389.\n\n[4] Zou, X., Tang, C., Zheng, X., Li, Z., He, X., An, S., and Liu, X. Dpnet: Dynamic poly-attention network for trustworthy multi-modal classification. In Proceedings of the 31st ACM International Conference on multimedia, 2023, pp. 3550-3559.\n\n**2. Unclear details regarding implementation of baseline comparisons. Where was the code for the compared methods obtained? A fair comparison is necessary to validate the performance improvements.**\n\n***Response：*** Thanks for your comments. The codes of the all compared methods are obtained from the GitHub websites published in the original papers, and their parameter settings also strictly follow the implementation details in the original papers to ensure correct reproduction. All the compared methods and the proposed method are conducted in the same experimental environment, which is a desktop computer with Windows 10 operating system, 32GB RAM, and MATLAB 2021a.\n\n**3. Many cited works focus on multi-view clustering, while this paper targets multimodal clustering. What are the key differences, and how do multimodal clustering methods differ from multi-view clustering approaches? The paper should include recent works on multimodal clustering for a more comprehensive discussion.**\n\n***Response：*** Generally, the aim of multi-view clustering and multimodal clustering is similar especially in integrating different sources of information for improving clustering performance. The differences between them are as follows: Multi-view learning focuses on diverse feature representations of the same object, while multi-modal learning deals with complex relationships between heterogeneous modalities, which is more complicated to handle. In practical applications, overlaps between them may exist (e.g., multi-modal data can also be considered as generalized multi-view data). However, technical solutions should be selected based on data characteristics (feature homogeneity and semantic consistency) to ensure methodological compatibility. In this paper, we use a more general expression of multi-modal clustering instead of multi-view clustering. In the final version, we will include and discuss more recent works on multi-modal clustering. Moreover, we will also discuss the relationships and differences between them as well as some information theory based methods, e.g., 'Dual Contrastive Prediction for Incomplete Multi-View Representation Learning', in the final version.\n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence, as demonstrated by the comparative experiments.", "methods_and_evaluation_criteria": "After carefully checking the manuscript, the proposed methods and evaluation criteria make sense for the multi-modal clustering problem. The proposed peer-review trustworthy information bottleneck method is interesting, and comparative experiments on eight multimodal datasets show the proposed method outperforms existing state-of-the-art methods.", "theoretical_claims": "The correctness of Theorem 3.3 and its proof has been reviewed and found to be valid.", "experimental_designs_or_analyses": "The experimental design and analyses in Section 4 have been reviewed, and their soundness and validity are confirmed.", "relation_to_broader_scientific_literature": "The paper contributes to the broader literature by introducing a peer-review-based multimodal clustering framework, which has not been extensively explored in previous works. The trustworthiness score and information bottleneck approach further enhance its novelty.", "essential_references_not_discussed": "The introduction sufficiently covers essential related works, but additional discussion on the differences between multimodal and multi-view clustering would provide better context for its contributions. Some information theory based methods could also be reviewed, e.g., Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI 2023.", "other_strengths_and_weaknesses": "This paper proposes a novel multi-modal clustering method grounded in robust theory and practical applicability. The paper has a clear structure. The proposed method is validated with well-designed experiments. \n\nThe paper also has some limitations:\n1.\tLack of discussion on existing trustworthy multimodal clustering approaches. Are there prior weighted multimodal clustering methods that incorporate trustworthiness in weight learning? \n2.\tUnclear details regarding implementation of baseline comparisons. Where was the code for the compared methods obtained? A fair comparison is necessary to validate the performance improvements.\n3.\tMany cited works focus on multi-view clustering, while this paper targets multimodal clustering. What are the key differences, and how do multimodal clustering methods differ from multi-view clustering approaches? The paper should include recent works on multimodal clustering for a more comprehensive discussion.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d9rsYOBrNp", "reviewer": "Reviewer_3BK8", "summary": "In this paper, the authors propose a new multi-modal clustering method by information bottleneck method with an interesting peer-review look. This method work in a weighted mechanism with two learning scores, including peer-review and trustworthy score. It is noted that the weight learning process is conducted without parameter tuning, which is good for practical applications. Many experiments on benchmark datasets show the effectiveness and superiority of the proposed method.\n\n## update after rebuttal\n\nThe authors have addressd my all concerns and I have reviewed the comments from the other three reviewers. So, I have raised my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below.\n\n**1. The peer-review look on multi-modal clustering is interesting, have the authors considered using this idea for other problems? The authors are encouraged to give some of them in the future work so that it may help the readers to explore more possibilities of this ideas into dealing with more problems.**\n\n***Response:*** Thanks for your comments. It is possible to try to migrate the idea of \"peer review\" to federated learning. Federated Learning is a distributed machine learning technique. Its core concept is to enable multiple participants (e.g., devices or institutions) to collaboratively train a globally shared machine learning model without sharing local data, thereby achieving data privacy preservation and collaborative model building. A critical step in this process is secure aggregation: clients upload their locally trained model updates to a central server, which then merges these updates using aggregation algorithms (e.g., FedAvg aggregates local model updates from multiple clients via weighted averaging) to generate a new global model. In this “peer-review” framework, each client can be viewed as an \"author\" submitting its update, while the remaining clients act as \"reviewers\" to score the quality of that update. If a client's update has a low score, it may indicate that the client is a low-quality client or an external malicious node. By leveraging these scores, the server can reduce the importance of low-scoring updates or exclude low-scoring updates during aggregation, ensuring the robustness of the global model.\n\n**2. Some equations in the figure are too small, which may influence the readability.**\n\n***Response:*** Thanks for your comments. We will use a more legible font size for the figures in the revised version. And we checked all the figures in the manuscript to ensure that they were clear and readable enough.\n\n**3. In section 2, it is not only to introduce IB method, some multi-modal clustering works using IB needs to be added here.**\n\n***Response:*** Thanks for your comments. Some newly added multi-modal clustering works using IB are in the following:\nIn recent years, IB theory has been widely used in various multi-modal clustering tasks. For example, Federici et al. [1] proposed a multi-modal IB method that can identify non-shared information between two modalities. Yan et al. [2] proposed a multi-modal IB method that uses shared representations of multiple modalities to eliminate private information of a single modality. But the modality-private information is eliminated as much as possible during the process of data compression, only exploring the shared information of modalities without taking advantage of the complex relationship between modalities. Hu et al. [3] conduct the information bottleneck theory in the origin data, the learned features of high-dimensional and the learned features of low-dimensional to fuse the module information. However, its final clustering result is obtained by directly averaging the local clusters from the modal high-dimensional features.\n\nDifferent from the existing multi-modal clustering methods based on IB theory, the proposed method considers the complex relationship between modalities, where the designed multi-modal peer review is used to reasonably score the contribution of each perspective, and the self-supervised trustworthy score is used to ensure the reliability of the process.\n\n[1] Federici, M., Dutta, A., Forre, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020.\n\n[2] Yan, X., Mao, Y., Ye, Y., and Yu, H. Cross-modal clustering with deep correlated information bottleneck method. IEEE Transactions on Neural Networks and Learning Systems, 2023.\n\n[3] Hu, J., Yang, C., Huang, K., Wang, H., Peng, B., and Li, T. Information bottleneck fusion for deep multi-view clustering. Knowledge-Based Systems, 2024, 289, 111551.\n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "The claims made in the submission can be supported by clear and convincing experiments, which provide different levels of validation on the proposed method.", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria make sense for the multi-modal clustering problem, especially the peer-review learning mechanism and many experimental results.", "theoretical_claims": "I have checked the correctness of proofs for theoretical claims in the submission, i.e., theorem 3.3 and its detailed proof.", "experimental_designs_or_analyses": "I have checked the soundness/validity of experimental designs or analyses in the submission. Many experiments on benchmark datasets show the effectiveness and superiority of the proposed method.", "relation_to_broader_scientific_literature": "The key contributions are: the authors propose a new multi-modal clustering method by information bottleneck method with an interesting peer-review look. It is new to the area.", "essential_references_not_discussed": "Essential References are discussed in the paper.", "other_strengths_and_weaknesses": "Strengths\n(1) novelty: novel for the ICML conference; notable advantages compared with state-of-the-art methods.\n(2) soundness: technically sound, under a very rigorous framework.\n(3) significance: the problem is significant and also the method may bring impact on the related community.\n\nWeaknesses\n(1) The peer-review look on multi-modal clustering is interesting, have the authors considered using this idea for other problems? The authors are encouraged to give some of them in the future work so that it may help the readers to explore more possibilities of this ideas into dealing with more problems. \n(2) Some equations in the figure are too small, which may influence the readability. \n(3) In section 2, it is not only to introduce IB method, some multi-modal clustering works using IB needs to be added here.", "other_comments_or_suggestions": "I have not any other comments or suggestions here. I have given all my comments in the “Other Strengths And Weaknesses”.", "questions_for_authors": "I have not any other questions here. I have given all my comments in the “Other Strengths And Weaknesses”.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wZf3b5jBV1", "reviewer": "Reviewer_b4ga", "summary": "For the mentioned three limitations faced by most current weighted multimodal clustering methods, this paper, inspired by the peer-review mechanism in academia, iteratively considers one modality as the \"author\" and the remaining modalities as \"reviewers\" to obtain the peer-review score for each modality. To improve the reliability, a trustworthy score with a self-supervised working mechanism is further designed. Finally, a new PTIB method is proposed, and lots of experimental results show its effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below.\n\n**1. There are many trustworthy multi-modal classification or clustering methods published recent years. This paper also mentioned the 'trustworthy'. What are the differences between this paper and existing trustworthy multi-modal classification or clustering methods?**\n\n***Response:*** Thanks for your comments. Regarding the trustworthiness of multiple modalities, almost all existing methods [1-4] focus on trustworthy multimodal classification. To ensure the reliability of both the multi-modal integration and the final decision, the trustworthy multimodal classification methods produce a stable and reasonable uncertainty estimation for each modality and thus promote both classification reliability and robustness. For example, Han et al. [2] introduce the variational Dirichlet to characterize the distribution of the class probabilities, parameterized with evidence from different views and integrated with the Dempster-Shafer theory, thus promoting both classification reliability and robustness. Zheng et al. [3] propose a trustworthy multimodal classification network via multi-level confidence learning which integrates both feature and label-level confidence learning for trustworthy multimodal classification. Zou et al. [4] induce a transparent fusion strategy based on the modality confidence estimation strategy to track information variation within different modalities for dynamical fusion.\n\nDifferent from them, the proposed method aims to guarantee the trustworthy of the learned modal weights in a self-supervised manner. To the best of our knowledge, none of the existing weighted MMCs employ the trustworthy strategy in the weight learning process.\n\n[1] Z. Han, C. Zhang, H. Fu, and J. T. Zhou, Trusted multi-view classification, in Proceedings of the International Conference on Learning Representations, 2021, pp. 1–11.\n\n[2] Han, Z., Zhang, C., Fu, H., and Zhou, J. T. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on pattern analysis and machine intelligence,2023, 45(2), 2551-2566.\n\n[3] Zheng, X., Tang, C., Wan, Z., Hu, C., and Zhang, W. Multi-level confidence learning for trustworthy multimodal classification. In Proceedings of the AAAI conference on artificial intelligence,2023, pp. 11381-11389.\n\n[4] Zou, X., Tang, C., Zheng, X., Li, Z., He, X., An, S., and Liu, X. Dpnet: Dynamic poly-attention network for trustworthy multi-modal classification. In Proceedings of the 31st ACM International Conference on multimedia, 2023, pp. 3550-3559.\n\n**2. Methodological limitations are only mentioned in Appendix B and are not summarized in the conclusion section.**\n\n***Response:*** Thanks for your comments. We will summarize the limitations of the proposed method in the conclusion section:\nThe proposed method is also with some possible weaknesses. It is designed for fully aligned multi-modal clustering and complete multi-modal clustering, where none of the data samples across modalities are unaligned and missing or damaged. And it requires the number of clusters of the dataset in advance like almost all the existing multi-modal clustering methods.\n\n**3. Lack of description of the experimental environment, which may help readers to better reproduce the main parts of the methods.**\n\n***Response:*** Thanks for your comments. All the compared methods and the proposed method are conducted in the same experimental environment, which is a desktop computer with Windows 10 operating system, 32GB RAM, and MATLAB 2021a. \n\n**4. The font size in some pictures is small, as shown in Figure 2.**\n\n***Response:*** Thanks for your comments. We will use a more legible font size for the figures in the revised version. And we checked all the figures in the manuscript to ensure that they were clear and readable enough.\n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. Lots of experimental results show its effectiveness over many compared methods.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the problem. The proposed method iteratively considers one modality as the \"author\" and the remaining modalities as \"reviewers\" to obtain the peer-review score for each modality. Then, a trustworthy score with a self-supervised working mechanism is further designed to improve the reliability. The evaluation criteria such as datasets, evaluation metrics, compared methods in the experiments are frequently used in the community.", "theoretical_claims": "I have checked the correctness of the proofs for theoretical claims, including the theorem 3.3 and its proof in Appendix A.1.", "experimental_designs_or_analyses": "I have checked the soundness/validity of the experimental designs or analyses, including subsection 4.1, 4.2, 4.3 and 4.4 and its corresponding analysis.", "relation_to_broader_scientific_literature": "The proposed method is novel by integrating peer-review idea to address multi-modal clustering problem.", "essential_references_not_discussed": "There are no related works that are not currently discussed in the paper.", "other_strengths_and_weaknesses": "The authors are inspired by the peer-review mechanism in academia and propose a new and well-organized framework named PTIB method. The introduction, methods and experimental sections are articulated clearly. This paper is well written and the novelty is quite enough, and the problem is well addressed. \n\nHowever, there are also some weaknesses below:\n1. There are many trustworthy multi-modal classification or clustering methods published recent years. This paper also mentioned the 'trustworthy'. What are the differences between this paper and existing trustworthy multi-modal classification or clustering methods?\n2. Methodological limitations are only mentioned in Appendix B and are not summarized in the conclusion section.\n3. Lack of description of the experimental environment, which may help readers to better reproduce the main parts of the methods. \n4. The font size in some pictures is small, as shown in Figure 2.", "other_comments_or_suggestions": "Please see the above, and I have no other comments. I am looking forward to the authors’ reply to my comments, which may help me to give a final rating.", "questions_for_authors": "Please see the above, and I have no other comments. I am looking forward to the authors’ reply to my comments, which may help me to give a final rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on three issues in multi-modal clustering, i.e., lack of trustworthiness in learned weights, isolated view weight learning, extra weight parameters. To solve these issues, motivated by the peer-review mechanism, the authors treat one modality as author and the remaining modalities as reviewers to reach a peer-review score. Then, to boost the trustworthiness, the authors devise a trustworthy score with self-supervision mechanism. Further, they take into account the above scores for accurate and parameter-free modality weight learning. 
Experimental results demonstrate the effectiveness of the present algorithm. 


After rebuttal, all reviewers give positive scores for this manuscript and acknowledge its contributions. After reading this paper and the posted responses, I agree with the reviewers and recommend an acceptance for this paper.
Decision: Accept (poster)