Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Authors: Zhengzheng Lou
Ke Zhang
Yucong Wu
Shizhe Hu
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal clustering
Information bottleneck
contrastive learning
Primary area: general_machine_learning->clustering
TL;DR: This paper proposes a method called super deep contrastive information bottleneck for MMC (SDCIB), which aims to explore and utilize all types of latent information to the fullest extent.

Abstract: In an era of increasingly diverse information sources, multi-modal clustering (MMC) has become a key technology for processing multi-modal data. It can apply and integrate the feature information and potential relationships of different modalities. Although there is a wealth of research on MMC, due to the complexity of datasets, a major challenge remains in how to deeply explore the complex latent information and interdependencies between modalities. To address this issue, this paper proposes a method called super deep contrastive information bottleneck (SDCIB) for MMC, which aims to explore and utilize all types of latent information to the fullest extent. Specifically, the proposed SDCIB explicitly introduces the rich information contained in the encoder's hidden layers into the loss function for the first time, thoroughly mining both modal features and the hidden relationships between modalities. Moreover, the proposed SDCIB performs dual optimization by simultaneously considering consistency information from both the feature distribution and clustering assignment perspectives, the proposed SDCIB significantly improves clustering accuracy and robustness. We conducted experiments on 4 multi-modal datasets and the accuracy of the method on the ESP dataset improved by 9.3\%. The results demonstrate the superiority and clever design of the proposed SDCIB. The source code is available on https://github.com/ShizheHu.

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Zhengzheng Lou't
Ke Zhang'
Yucong Wu Shizhe Hu
Abstract
multi-modal data, while the scale and complexity of the data have significantly increased (Srivastava & Salakhutdi- nov, 2012; Baltrusaitis et al., 2019; Ramesh et al., 2021), "Modality" refers to different types of data obtained from various perspectives of information acquisition and process- ing (Hacquard, 2010). For example, a video can capture data information from both the sound and image perspectives. However, traditional sinllemodal data analysis methods often struggle comprehensively handle multi-modal data. Multi-modal clustering (MMC), key technology, can fully mine and integrate information from different modali- ties, revealing the potential complementarity between them. It has been widely applied in areas such as cross-modal re- trieval (Hu et al., 2019; Chun al., 2021; Yuan et al., 2022), intelligent recommendation (Liu al., 2021). and biomedi- cal science (Acosta et al., 2022; Si et al., 2023). Therefore, research on MMC methods provides vast prospects for prac- tical applications.
In an era of increasingly diverse information sources, multi-modal clustering (MMC) has be- come key technology for processing multi- modal data. It can apply and integrate the feature information and potential relationships of differ- ent modalities. Although there is a wealth of re- search on MMC, due to the complexity of datasets, major challenge remains in how to deeply ex- plore the complex latent information and interde- pendencies between modalities. To address this issue, this paper proposes a method called super deep contrastive information bottleneck (SDCIB) for MMC, which aims to explore and utilize all types of latent information to the fullest extent. Specifically, the proposed SDCIB explicitly in- troduces the rich information contained in the en- coder's ' hidden layers into the loss function for the first time, thoroughly mining both modal features and the hidden relationships between modalities, Moreover, the proposed SDCIB performs dual optimization by simultaneously considering con- sistency information from both the feature distri- bution and clustering assignment perspectives, the proposed SDCIB significantly improves cluster- ing accuracy and robustness. We conducted exper- iments on multi-mode datasets and the accuracy of the method on the ESP dataset improved by 9.3%. The results demonstrate the superiority and clever design of the proposed SDCIB. The source code is available on https://iitbbb.oo//Sizeeett
Existing traditional MMC methods mainly focus on three categories: subspace learning. graphical models and matrix decomposition (Cai et al., 201 1; Xia et al., 2023). How- ever, as multi-modal datasets become increasingly complex, traditional MMC methods are prone to the "curse of dimen- sionality" in high-dimensional spaces and struggle to handle high-dimensional data effectively.
At the same time, with the continuous development of deep learning, deep MMC methods fully integrate the ad advantages of deep neural networks, providing more solutions for MMC (Ngiam et al., 2011; Andrew al., 2013; Xie al., 2016; Liu al., 2023; Gao al., 2020). For example, Wang et al. (Wang et al., 2021) combined the self-supervised t-SNE module with the self-expression layer to learn shared low- dimensional representation. Mao et al. (Mao et al., 2021) adopted the idea of contrastive learning to maximize the shared information between modalities and minimize the redundancy within each modality, achieving efficient clus- tering by integrating
1. Introduction
In recent years, with the rapid advancement of information acquisition technology, data storage capacity has also devel- oped quickly. The sources of information have gradually diversified, evolving from traditional single-modal data to
multi-modal shared encoder with variational optimization. In recent years, Rong et al. (Rong et al., 2022) utilized variational autoencoder architecture based on the autoencoder and incorporated an attention mechanism to extract cluster- -riendrr representations from multi-omics data. However, the aforementioned methods fail deeply understand the complex relationships within data samples across modalities, neglecting the close connec-
'School of Computer and Artificial Intelligence, Zhengzhou University, Zhengzhou, China. Correspondence to: Shizhe Hu < ieshizhchu @ gmm, https://shizhchugittub gith....i/.......///......
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
tions between the data, and may even disrupt the internal associations.
improving clustering accuracy and robustness.
In the specific experiments, we selected several clas- sic multi-modal datasets and compared the proposed SDCIB with the state-of-the-an traditional and deep MMC methods. The proposed SDCIB consistently demonstrated significant advantages in clustering per- formance.
Therefore, although a large number of MMC methods have emerged in both traditional and deep learning fields in recent years, there are still some limitations in the actual clustering process, On the one hand, most existing MMC methods simply achieve a consensus between different modalities to fuse the data. While this improves clustering perfor- mance to some extent, it fails to deeply explore and capture the complex latent information and interdependencies be- tween modalities in multi-modal data. On the other hand, many deep multi-modal methods often only focus on sin- gle data information such as clustering or features, without considering the two simultaneously. As result, much of the inherent latent structural information in the data sam- ples is ignored or even disrupted, failing capture the more important heterogeneity between modalities. In this study, we propose a method called super deep contrastive information bottleneck for MMC (SDCIB) to address the aforementioned issues. The proposed SDCIB efficiently mines the latent information between modalities through the hidden layers. Meanwhile, it simultaneously considers both feature distribution and clustering assignment to better cap- ture the inherent structure of the data. Our main approach is designed for multi-modal data, where we first combine the information bottleneck theory to design a variational IB encoder, then incorporate the concept of contrastive to optimize the computation of consistency information. And finally obtain the final clustering results through a clustering layer. The proposed SDCIB is the first to explicitly intro- duce the hidden layer information from the encoder into the clustering task. On one hand, the proposed SDCIB fully incorporates the concept of "compression"" from the infor- mation bottleneck theory to eliminate redundant information in the modalities. On the other hand, the proposed SDCIB simultaneously focuses on consistency information at both the feature and clustering levels, deeply connecting the hid- den layer's latent information uncover the data's Mata's potential feature information, thereby better assisting clustering. Fi- nally, The proposed SDCIB designs a clustering layer to obtain accurate clustering results. The main contributions of this work are as follows:
2. Method
2.1. Revisit: Information Bottleneck
The information bottleneck (IB) principle (Tishby et al., 1999) is based on information theory and is widely used in various real-world applications, such as image clustering (Hu et al., 2021) and classification tasks (Lou et al., 2013). A detailed discussion of the IB principle can be found in our survey (Hu et al., 2024). Its core concept is to control the balance between the amount of information compres- sion and relevance. Suppose there is a source variable X and a label Y. The goal of the information bottleneck to find a compressed representation T through the source variable X, which compresses the source variable X as much as possible while retaining the relevance between the compressed representation T and the label Y as much as possible. Therefore, the objective function of the IB theory can be formulated as:
Cmin = -[TT=X - -(T:X)- (I))
(1)
where [(T;X) represents the mutual information between the compressed representation T and the source variable X. The smaller it is, the greater the compression. I(T; Y) rep- resents the mutual information between the compressed rep- resentation T and the label Y. The larger it is, the stronger the retained correlation. 3 is used to control the balance between compression and retention.
In recent years, IB theory has been widely used in various MMC tasks (Federici et al., 2020; Yan et al., 2024; 2025). For example, Federici et al. (Federici et al., 2020) proposed multi-modal IB method that can identify non-shared in- formation between two modalities, but it only explores the correlation of different modalities through feature distribu- tion, ignoring the consistency of cluster assignment, making the learned feature representation unfriendly to downstream clustering tasks. Yan et al. (Yan et al., 2024) proposed a multi-mode IB method that uses shared representations of multiple modalities to eliminate private information of a single modality. Yan et al. (Yan et al., 2025) further pro- posed an incremental IB method that builds acknowledge base to solve the clustering problem of incremental modali- ties. Both of the above works considered the consistency of feature distribution and cluster assignment at the same time, but they failed to consider the correlation between feature
For the MMC problem, we proposed SDCIB, which is the first to explicitly introduce the information con- tained in the encoder's ' hidden layers into the loss func- tion, fully exploring the implicit information in the modal features.
The proposed SDCIB performs dual optimization by simultancously considering consistency information from both the feature distribution and clustering assign- ment perspectives. This approach can better capture the latent information between modalities, significantly
2

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
2.3. Network Architecture
Supervied Learning
Modelity!
Our network framework is shown in Figure 1, which can be divided into three parts. 1) Variational IB encoder: The original features X' of the ith modality the variational IB encoder we designed to obtain the hidden layer features Hi. Hg..... H and the compressed rep- resentation Hi, where is the number of encoder layers and Hi represents the features output by the encoder of the ith modality at the Lth layer. Although encoders of different modalities have the same structure, their training processes are independent of each other. 2) Contrastive learning loss: By comparing features Hi, H:..... ..... Hi between different modalities and the clustering results Y the commonalities between different modalities are explored to learn features H that are more conducive to clustering. 3) Clustering layer: The clustering result Y of i-th modal- ity is obtained directly through the compressed feature Hi through linear layer and softmax function.
Rettlookk
passed through
H
Y
11
H
H
Medilit
Disbmnoo
Contractive Learning
costle sands the hovnnfl sea, thom a.s banch-on
H
H
H
Batkanck
X'
A oe helske
Feihbit
Self-Sapervied Learning
2.4. Hidden-layer Information
Figure 1. Framework of the proposed SDCIB. Given original features X of different modalities, they are input into the varia- tional IB encoder oblain the hidden layer features Hi, H2g .... Mi-1 and the compressed representation Hi Hi, passes through the clustering layer to obtain the clustering result Y*, which in turn guides the encoder to compress X In order to make different modalities as consistent as possible, we compare them from the perspectives of feature distribution and cluster allocation.
After the original feature X enters the encoder, it will be gradually compressed into a more compact representation HL. The hidden layer representation Hi. H: ... Hi- is the intermediate result of the compression behavior. Explic- itly introducing reasonable hidden layer representations Hi, H Hi into the objective function has the following advantages:
Richer feature representations: As the network depth increases, each hidden layer gradually refines the fea- ture representation. This makes the representation more informative and structured, which is particularly beneficial for clustering tasks.
distribution and clustering results. All the above MMC IB methods ignore the rich information contained in the hidden layers of the encoder and fail to explicitly utilize it. The above limitations motivate us to propose SDCIB.
2.2. Problem Definition
Enhanced model interpretability: Hidden layer rep- resentations provide insight into how the model grad- ually transforms the input. This transparency helps understand the learned hierarchical structure, facilitat- ing model analysis and fine-tuning.
For a given MMC task, suppose there are M modalities and N data samples to be clustered, which can eventually be clustered into clusters. Our goal is to learn better feature representations Hi in M modalities through the proposed SDCIB, to obtain good clustering results Y' for each modality, and finally combine the clustering results y' for each modality to obtain the final clustering result Y by:
Optimization benefits of the information bottleneck framework: Introducing hidden layer information in the information bottleneck framework helps regulate the trade-off between compression and retention of relevant information, resulting in more discriminative and generalizable clustering results.
Σ I = =)
V: arg max
(E.1.2....KK K)
2.5. Objective Function
where indicates which cluster i-th sample belongs to in the final clustering partition, and y," indicates which cluster i-th sample belongs to in the clustering partition of m-th modal.
Information Bottleneck. IB compresses the original fea- tures X to obtain a good compression representation Hi. so as to obtain a better clustering result Y The hidden layer information Hi, H Hi can participate in this
3

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
compression process, helping us obtain better compres- sion representation H Therefore, Our method explicitly introduces the hidden layer information into IB and obtains the following loss function:
2.6. Optimization
In high-dimensional continuous space, the calculation of mutual information has always been difficult problem. In order obtain good representation of mutual information, we adopted three different strategies to estimate mutual information, namely the variational method, Information Noise-Covee Contrastivee Estimation (InfoNCE)(ong & Ermon, 2020) and Mutual Information Neural Estimation (MINE) (Belghazi et al., 2018).
L1 E -
(3)
where T(X;; Hi) is the mutual information between the original feature X' and the feature Hj. I(Y'; Hi) is the mu- tual information between the feature Hf and the clustering result Y', and the parameter pression and retention of feature information. The smaller 3, the higher the degree of information compression.
Variational method is used to calculate the amount of in- formation / (x'; Hff)) retained by the compressed variable Hi of the original feature X'. From the definition of mutual information, we can get the calculation formula of mutual information as follows:
used control the com-
Consistency Information. In MMC tasks, it is a common method to explore the consistency information between mul- tiple modalities to seek reasonable data parition. In order to maximize the use of the consistency information between different modalities, we not only consider the consistency information at the feature level, but also the consistency information at the cluster level. At the same time, in or- der to ensure that the compressed features Hi, of different modalities are as consistent as possible, the proposed SD- CIB keeps the feature information Hi. H2, Hi of different modalities consistent as much as possible in the hidden layer. Therefore, the loss function of the proposed SDCIB in terms of consistency information is:
1(x;; Je plat, hi)log p(x', hif)
p(())
(6)
= -Se plat, hiylog p(hff (xt) p(hf)
Since p(h)) is difficult obtain, q(hf) approxi-
mate p(h)). According to the non-negativity of Kullback- Leibler (KL) divergence, we can get:
KLIp(b)la(bh11 f p(hi)log p(h)) (hht)
L2 =
Les (eX))((Y + Σ (uiii (4) 1=1
| p(hi)log Jm
S p(hi)log q(h))
where 1HH;; Hz) is the mutual information between fea- tures Hj and Hy of different modalities, which measures the consistency of Hi and H.. I(Y;; YJ) the mutual information between the clustering results Yt and y of different modalities, representing the consistency of y and yJ,
Rewrite 1(*;HH) into the following inequality form:
[ p(a', hi)log p(hilz') p(hf)
1(x:: 11)1
[ pla hi)log p(hilet))
(8)
((hi)
Overall Objective Function. Therefore, the proposed SD- CIB can compress the original features while maintaining the consistency between modalities and the correlation be- tween the compressed features and data partitions. More- over, compared with the ordinary deep IB method, it further explicitly utilizes the hidden layer in the encoder. Therefore, we call it Super Deep Contrastvie Inforamtion Bottleneck.
[ a p(hil=") p(bllzz)) ((h))
To simplify the formula, we use Monte Carlo sampling (Von Ahn & Dabbish, 2005) to replace p(x*) with & and get:
Its overall loss function is as follows:
Espers = cLaie (1 Geon
N T(X;, < N Σ / p(hil=') (((>)"))('()))))))))))) )p(>))og log
p p(h (x))
(9)
Σ> -
((h))
(5)
Where N is the number of data samples.
- ΣΣ L
Assuming that p(hil=') obeys a Gaussian distribution, its mean μ and standard deviation o can be obtained by the variational IB encoder, where c obeys a standard normal
where a is used to weigh the importance of IB and consis- tency information obtaining compressed features.

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Algorithm Algorithm for Optimizing the proposed SD- I 1: Input: Dataset with M modalities {X*) the num- ber of clusters K, the parameter B.
thus affecting the accuracy of the estimation. In contrast, MINE provides a more robust mutual information estima- tion method that obtains the lower bound of the mutual information by optimizing the loss:
CIB
2: Output: Final partition Y.
LINE -= (Tr Eog (Txp y))
3: Random Initialization: Randomly initialize the pa- rameters of M modality-speciffi encoders and M L mutual information estimators.
(12)
where T is the function that the neural network needs to fit, and 9 is the learnable parameter.
4: repeat
5: Calculate LAINE by Eq. (12)
For the specific optimization process, see Algorithm 1.
6:
Optimize the parameters of mutual information esti-
mators
3. Experiment
7:
Obtain (Y Hff) by mutual information estimators
8:
Calculate T(X:; Hi)) by Eq. (8)
3.1. Datasets
9:
Calculate Hf) and (1''Y1) and by Eq. (11)
We selected 4 datasets of different scales for experimental evaluation, with detailed datasets information presented in Table 1.
10:
Optimize the parameters of mopality-specific en-
coders
11:
until Convergence
Obtain partition Y
Caltech-2V(Fei--Fei et al., 2004) contains 1,440 image sam- ples, categorized into 7 classes based on WM and CEN- TRIST modalities. Event (Li Fei-Fei, 2007) encompasses 1,579 sports event image samples, divided into 8 categories based on 3 modalities: Color Attention, SIFT, and TPLBP. IAPR (Grubinger et al., 2006) includes 7,855 image sam- ples, accompanied by natural language descriptions, and is divided into 6 categories using SIFT representation and BoW model modalities. ESP (Von Ahn Dabbish, 2005) sourced from social image collection on an image an- notation game website, comprises 11,032 image samples, categorized into 7 classes.
13:
Obtain final partition Y by Eq. (2)
distribution. Now 1(X:; Hi)
can be rewritten as:
1
N
Eelog p(hilaz)) g(1i)
T(X:: H7) <
N
(10)
1
E, K. (LLp)()))) ((t)
N
InfoNCE is used when calculating mutual information T(H); H²)) between compressed variables of different modalities and mutual information L(Y'; YJ) between clus- tering results. InfoNCE estimates the mutual information by contrastive loss:
3.2. Compared Methods
To evaluate the effectiveness of the proposed SDCIB, we adopt fourteen baseline methods comparisons. Fourteen of these methods span three major categories of modality clustering approaches.
(YY = Eptrtar) log exp(folv', ΣN cxp(fo(v', vai)
HlggN (11)
Single-Modal Clustering/ / Full-Modal Clustering Meth- ods: K-Means (KM) and Normalized Cuts (Ncuts) are two classic and widely used single-modal clustering methods. Based on single-modal clustering, full-modal versions are developed by splicing multiple modalities into unified representation, ultimately leading All-modal K-Means (AmKM) and All-modal Neuts (AmNcuts).
where So represents the similarity, (v', y ) is positive ple pair, (y', vA)) is negative sample pair, and N is the total number of samples. Similarly, the mutual information I(H;; H)) can also be calculated by Eq. (11).
MINE is used to calculate the amount of information that the compressed variable Hi retains about the label y', The dimensions of Hf and Y are not consistent, where the di- mension of H1 is higher than that of Y*, The variational method requires manual aligmment of the dimensions of ii and Y', which may cause information loss during the alignment process. The InfoNCE method requires the con- struction of large number of negative samples, and due to the small dimension of y which may lead to insuffi- cient discrimination between positive and negative samples,
Traditional MMC Methods: Below are representative traditional MMC methods. These methods enhance the
Table I. Details of various kinds of multi-modal datascts.
DATASET
MODALITIES
SAMPLES
CLUSTERS
DIMENSIONALITY
CALTECH-2V
2
1440
7
(40,254)
EVENT IAPR ESP
1579
(1000,1000,1000
2
7855
5
(1200.500)
3
11032
7
(300,300,300)
5

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
robustness of clustering from different methodological per- spectives.
contrastive calibration network.
(4) ICMVC (Chao et al., 2024): An end-to-end clustering method that handles missing data through multi-modal con- sistency transfer and graph convolutional networks, and combines contrastive learning.
(1) CoS (Kumar et al., 201 I) multi-modal spec- tral clustering method that applies co-regularization to the clustering results.
(2) RMKMC (Cai et al., 2013): multi-modal k-means clustering method that adaptively adjusts modality weights to handle differences in modal quality.
(5) DIVICE (Lu et al., 2024): A multi-modal clustering method based on decoupled contrastive learning and high- order random walks, and integrates the idea of contrastive learning improve clustering performance.
(3) SwMC (Nie al., 2017): A totally self-weighted multi- modal clustering method for automatic modality weighting.
3.3. Settings of Experiments
(4) ONMSC (Zhou et al., 2020): multi-modal clustering method that integrates the neighborhood information of first- order and high-order Laplacian matrices.
The experiments adopts two commonly used clustering eval- uation metrics, accuracy (ACC) and normalized mutual in- formation (NMI), to assess clustering performance. Higher values of these metrics indicate better clustering quality. For single-modal clustering methods, the clustering result of the best-performing modality is selected as the final result. For other MMC methods, the parameter configurations in experiments follow the original settings provided in their respective papers, and the optimal results under the recommended parameters are chosen for comparison.
Below are 6 representative state-of- the-art methods in the field of deep MMC, demonstrating significant progress in recent years.
Deep MMC Methods:
(1) SiMVC and CoMVC (Trosten et al., 2021): The paper introduces two multi-modal clustering models. SiMVC is simple baseline model for deep clustering that performs clus- tering by learning weighted fusion of representations from different modalities via linear combination. CoMVC builds on this by introducing contrastive alignment module to overcome the limitations of traditional alignment methods.
In the implementation of the proposed SDCIB, parame- ters a and 3 are defined. a takes values from the range (0.1,0.2, 0.4, 0.6, 0.8), and 3 takes values from the range 10, 100, 1000, 10000}. The specific choices of a and will be presented in detail in subsequent chapters. The tire training process of the experiment is completed within 40 epochs, with batch size of 32. The proposed SDCIB consists of M mopelity- encoders, 4 * M mutual information estimators, and M clustering layers. Each
(2) MFLVC (Xu et al., 2022): A hierarchical feature learn- ing clustering method that efficiently integrates multi-level feature learning and contrastive learning.
(3) De (Yang al., 2023): A clustering method that ensures the consistency of similar samples using dual
Table 2. Clustering performance with Acc and NMI on various kinds of datasets (the bold and underlined values in the table represent the best and second-best results respectively).
CALTECH-2V
EVENT
IAPR
ESP
METHODS
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
KM NCUTS (TPAMP'0))
41.6
30.5
34.7
20.7
38.9
17.2
48.4
33.5
39.9
31.2
34.8
15.5
41.9
18.9
45.7
29.8
AMKM
46.4
31.4
28.7
11.6
40.4
17.0
35.0
20.7
AMNCUTS (TPAMI'00)
42.8
25.2
35.2
20.3
42.2
18.9
32.5
19.0
COREGMVSC (NIPS'11)
49.2
39.6
35.5
22.2
35.1
18.4
45.2
30.7
RMKMC (ICAAI13)
51.4
33.5
39.5
25.1
36.4
15.9
35.1
20.8
SWMC (ICCAI17)
34.2
26.6
16.7
2.2
30.2
23.1
37.8
22.8
ONMSC (AAAI'20)
34.2
26.6
48.6
33.8
21.6
11.1
21.2
12.2
SIMVC (CVPR'2)
$1.1
36.9
36.8
23.1
42.7
18.5
33.6
14.6
COMVC (CVPR'21
59.2
49.2
49.1
35.5
46.7
21.5
43.4
27.3
MFLVC (CVPR') (22)
61.5
53.6
48.5
34.9
47.3
22.6
52.1
36.9
DEALMVC (ACM MM'23)
47.6
37.9
26.5
9.3
35.0
10.8
43.4
27.4
ICMVC (AAAI'24)
49.6
37.9
36.4
30.3
37.1
16.8
46.7
30.0
DIVICE (AAAT'24)
64.1
52.9
31.4
12.4
45.6
23.0
47.2
28.8
SDCIB
67.5
59.2
56.5
36.4
52.9
28.7
61.4
44.7
OURS VS BEST COMPARED
3.41
5.61
7.41
0.91
5.61
5.61
9.31
7.8t
6

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
modalityysseecinn encoder contains 4 fully connected layers with dimensions of 1024, 1024, 1024, and 128, respectively. Each fully connected layer is followed by a BatchNorm layer for representation normalization and a ReLU layer as the activation function. MINE continuously optimizes the correlation between two features to obtain a more accurate mutual information measure, laying a solid foundation for subsequent clustering. The clustering layer consists of fully connected layer and softmax layer to obtain the final clustering results. Meanwhile, we use the Adam optimizer for parameter optimization, with an initial learning rate set to 0.0001.
which does not fully represent the entire dataset.
Compared to traditional MMC methods, the proposed SDCIB demonstrates a clear advantage in clustering performance. For example, on the Caltech-2V and IAPR datasets, the proposed method improves the best ACC results of traditional methods by 16. % and 16.5%, respectively. At the same time, tradi- tional MMC methods generally outperform full-modal methods, which suggests that the simple full-modal ap- proach, formed by splicing individual modalities, does not properly apply the information within the modality,
Compared to deep MMC methods, the proposed SD> CIB still demonstrates outstanding clustering perfor- mance. For example, on the Event dataset, the pro- posed method outperforms the deep MMC methods SiMVC, CoMVC, MFLVC, DealMVC, ICMVC, and DIVICE in ACC by values of 19.7%, 7.4%, 8.0%, 30.0%, 20. 10,1, and 25.1%. This is attributed to our approach of fully leveraging the internal information within the data and innovatively using hidden layer information to assist in clustering, thereby better ex- ploiting the advantages of deep learning. Additionally, the proposed SDCIB incorporates the information bot- tleneck principle and introduces contrastive learning to jointly guide the clustering process.
3.4. Results and Analysis
In this chapter, we conduct comprehensive experiments on 4 datasets of different sizes, with the ACC and NMI values for fourteen comparison methods and the proposed SDCIB are listed in Table 2. In the specific numerical comparison, the proposed SDCIB demonstrates strong competitiveness.
Compared to single-modal clustering and full-modal clustering methods, the proposed SDCIB significantly outperforms in clustering performance. Specifically, on the Caltech-2V dataset, the ACC metric of the pro- posed SDCIB exceeds the single-modal KM method and Neuts method by 25.9% and 27.6%, respectively. The ESP dataset also outperforms the full- full-modality AmKM and AmNcuts methods in terms of the NMI metric by 24.0% and 25.7%, in turn. Meanwhile, com- pared to the performance of single-modal and full- modal methods, there is no significant advantage in numerical terms. This is because in single-modal clus- tering, the optimal modal clustering result is selected,
3.5. Parameter Selection
Hyperparameters and [3. We analyze the the parameters a and |3 of the proposed SDCIB. The values of parame- ter a are {0.1, 0.2, 0.4, 0.6, 0.8). and the values of 3 are {1, 10, 100, 1000, 10000]. The parameters form all possi- ble combinations through the Cartesian product. For each
Callech-2V ACC
EvenLACC
IAPR ACC
ESP ACC
ACC
ACC
Callech-2V NMI
Event
IAPR NMI
ESP NMI
Figure 2. The parameter analysis of our extended the proposed SDCIB on the Caltech-2V, Event, IAPR and ESP datascts is presented, with each bar in the figure corresponding to specific pair of (a, (9) values.
7

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Table Model network layer evaluation Caltech-2V and Event datasets (the bold values in the table represent the best).
CALTECH-2V
EVENT
IAPR
ESP
HIDDEN LAYERS
ACC
NMI
NMI
ACC
NMI
ACC
NMI
62.6
54.8
50.1
30.3
52.7
27.2
62.1
42.5
64.4
54.2
55.0
36.6
28.0
61.6
44.7
67.5
59.2
56.5
36.4
52.9
28.7
61.4
44.7
5
65.7
58.3
47.1
30.3
49.4
26.1
58.8
42.1
64.5
57.1
50.5
31.3
52.4
38.5
62.0
43.8
65.3
58.3
44.5
27.6
27.3
61.2
43.7
Table Ablation results Caltech-2V, Event, IAPR, and ESP datasets (the bold values the table represent the best).
CALTECH-2V
EVENT
IAPR
ESP
METHODS
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
REMOVING I(Y
41.8
33.2
30.3
15.5
38.5
20.1
36.3
27.6
REMOVING T(H;; HF)
63.1
52.2
45.0
25.3
48.8
23.5
62.3
43.4
REMOVING I(Y')") AND H(H
32.2
23.6
21.6
9.1
30.0
6.8
27.1
16.3
SDCIB
67.5
59.2
56.5
36.4
52.9
28.7
61.4
44.7
of parameters, record the ACC and NMI values, shown in Figure 2. It is observed that when the parameters 0.1 and [ = 1, the ACC and NMI values across various datasets generally outperform other parameter settings. This indicates that the proposed SDCIB demonstrates universal- ity and stability under these parameters. The combination of = I and 3 effectively balances the key factors within the method, highlighting the effectiveness and rationality of the proposed SDCIB design.
lowing conclusions.
First, when remove term (1), the ACC and NMI values for the 4 datasets decrease significantly, especially for the Event dataset, where the ACC value drops by 26.2%. This is mainly because the T(Y'; Y term considers clustering in- formation between different modalities, which is beneficial for modality fusion. Second, when we remove term (2), both ACC and NMI show noticeable decline, except for the ESP dataset, where the ACC value increases slightly by 0.9% compared to the proposed SDCIB. Finally, when (1) and (2) are deleted at the same time, the clustering performance of the four datasets is almost halved. For example, on the Caltech-2V dataset, the ACC drops by 35.3%. The above experiments validate component in the proposed SDCIB the final clustering performance, fully proving its effectiveness.
Number of encoder layers. The hidden layers in the pro- posed SDCIB represent highly ingenious design. To in- vestigate the impact of the number of network layers on clustering performance, we conducted experiments on the Caltech-2V, Event, IAPR, and ESP datasets, varying the number network layers 2, 3, imental results shown in Table 3. The results indicate that having more or fewer layers does not necessarily lead to better performance. Notably, the optimal results for the four datasets are generally achieved with a 4-layer network. Therefore, in the architecture of the proposed SDCIB, we adopt a 4-layer network with dimensions set to 1024, 1024, 1024, and 128.
significant contribution of each
6, and The exper-
3.7. Necessity of hidden layer information
To verify the necessity of hidden layer information, we con- ducted experimental investigations in this chapter. Under the condition that the parameter settings consistent, designed two groups of experiments. One experiment ig- nores the information from the hidden layers and focuses only on the final output layer Hi, while the other experi- ment retains the hidden layer information extraction part of the proposed SDCIB. The specific experimental results are presented in Table where it can be observed that the proposed SDCIB demonstrates varying degrees of improve- ment across different datasets. Notably, the Event dataset, the ACC value increased by 10.6%, and the NMI value im- proved by 7.3%. The experiments provide strong evidence
3.6. Ablation Study
In this chapter, we conduct ablation studies to evaluate the contribution of different components in the objective func- tion of the proposed SDCIB. Specifically, two terms subject removal: (1) mutual information between Y and Yi, denoted as (Y), and (2) mutual information be- tween Hi and H/ denoted as [HH;; HF)). Table 4 presents the results of the ablation experiments, leading to the fol-
8

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Table Experimental results the Caltech-2V, Event, IAPR, and ESP datasets with and without the application of hidden layer information (the bold values in the table represent the best).
CALTECH-2V
EVENT
IAPR
ESP
METHODS
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
SDCIB-NO-HIDDEN
63.6
55.8
45.9
29.1
48.9
26.9
57.9
40.0
SDCIB
67.5
59.2
56.5
36.4
52.9
28.7
61.4
44.7
IMPROVEMENT 3.9
3.41
10.61
7.31
4.01
1.81
3.51
4.71
for the crucial role of hidden layer information. Moreover, they indicate that fully mining and utilizing hidden layer information helps to more deeply explore the intrinsic re- lationships and latent structures among modalities, thereby enhancing the accuracy and robustness of clustering results.
3.8. Convergence Analysis
Ca
To study the convergence of the proposed SDCIB, we present the total loss function variation curves of the Caltech- 2V and IAPR datasets in chart. As shown in Figure 3, We observe its loss value, which shows an overall downward trend. At about 500 epochs, the function basically converges to a relatively stable value. Therefore, the proposed SD- CIB demonstrates rapid convergence within certain range, which not only validates the effectiveness of the proposed SDCIB but also demonstrates the reliability and stability of the proposed SDCIB.
GAPP
ESP
Figure 4. Visualization of the Caltech-2V, Event, IAPR, and ESP datasets (different colors represent different clusters).
3.9. Visualization Validation
4. Conclusion
In this chapter, we use t-distributed stochastic neighbor em- bedding (t-SNE) to visualize the Caltech-2V, Event, IAPR, and ESP datasets. As shown in Figure 4 different colors represent different clusters. It is evident that the proposed SDCIB shows a more compact distribution of data sam- ples within the same cluster, while the samples from dif- ferent clusters are more dispersed. Overall, the distribution clearer and more intuitive. This visualization directly demonstrates the advantages of the proposed SDCIB clus- tering performance, highlighting its ability to capture fine- grained patterns in complex datasets.
This paper innovatively proposes the super deep contrastive information bottleneck (SDCIB) for MMC method ad- dress the challenges currently faced by MMC, enabling more thorough extraction of data information. Specifi- cally, the proposed SDCIB not only incorporates the rich information contained in the encoder's hidden layers into the clustering process, but also performs dual optimization from consistency information perspectives: feature dis- tribution and clustering assignment. Compared to fourteen advanced MMC methods, the superiority of the proposed SDCIB has been repeatedly validated on the Caltech-2V, Eyent, IAPR, and ESP datasets.
Caltech-2V
IAPR
However, the proposed SDCIB still faces several challenges, such as limited performance on incomplete data samples (Liu al., 2024), reliance on predetermined number of clusters, and the use of batch learning strategy. These factors may limit the flexibility of the proposed SDCIB in practical applications, especially in complex scenarios involving unknown structures, missing data, or real-time requirements. Addressing these issues will be the focus of our future work.
-15
-304
260
400
Figure 3. The loss function variation curves of the Caltech-2V and
IAPR datasets.

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Acknowledgements
Gao, Q.. Lian, H., Wang, Q. and Sun, G. Cross-modal sub- space clustering via deep canonical correlation analysis. In AAAI, pp. 3938-3945, 2020.
The authors thank anonymous reviewers for their construc- tive comments. This work was supported by National Nat- ural Science Foundation of China under Grant 62206254, Henan Province Outstanding Youth Science Fund Program under Grant 252300421 and China Postdoctoral Science Foundation under Grant 2024T1 70843 and 2023M743186.
Grubinger, M.. Clough, P., Müller, H., and Deselaers, T.
The iapr tc- benchmark: A new evaluation resource for visual information systems. In Proceedings International workshop ontolmage, volume 2, 2006.
Hacquard, V. Modality, Language, 86(3):739-745 741, 2010.
Impact Statement
Hu, D., Nie, F., and Li, X. Deep multimodal clustering for unsupervised audiovisual learning. In CVPR, pp. 9248- 9257, 2019.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hu, S., Yan, X., and Ye, Multi-task image clustering through correlation propagation. IEEE TKDE, 33(3): 1113-1127, 2021.
References
Hu, S., Lou, Z.,, Yan, X., and Ye, A survey on information bottleneck. IEEE TPAMI, 46(8):5325-5344, 2024.
Acosta, J. N., Falcone, G. J., Rajpurkar, P., and Topol, E. J. Multimodal biomedical ai. Nat. Med., 2879):177--1784, 2022.
Kumar, A., Rai, P., and III, H. D. Co-regularized multi-view spectral clustering. In NeurIPS, pp. 1413-1421, 2011.
Andrew, G., Arora, R., Bilmes, A., and Livescu, K. Deep canonical correlation analysis. In ICML. pp. 1247-1255, 2013.
Li, L. and Fei-Fei, L. What, where and who? classifying events by scene and object recognition. In ICCV, pp. 1-8, 2007.
Baltrusaitis, T., Ahuja, C., and Morency, L. Multimodal machine learning: A survey and taxonomy. IEEE TPAMI, 41(2):423 443, 2019.
Liu, C., Wen, J., Wu, Z., Luo, X., Huang, C., and Xu, Y. Information recovery-driven deep incomplete multiview clustering network. IEEE Trans. Neural Networks Learn.
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Hjelm, D., and Courville, A. C. Mutual information neural estimation. In ICMI., pp. 530-539, 2018.
Syst. 35(1))115442-15522 2024.
Liu, J., Mao, Y., Huang, Z., and Ye, Y. bottleneck network with light attention for multimodal clustering. KBS, 280: 111037, 2023.
Cai, X., Nic, F., Huang. H., and Kamangar, F. Heteroge- neous image feature integration via multi-modal spectral clustering. In CVPR, pp. 1977-1984, 2011.
Liu, Y.. Lyu, C., Liu, Z., and Cao, Exploring a large- scale multi-modal transportation recommendation system. Transp. Res. Part C Emerg. Technol., 126: 103070, 2021.
Cai, X., Nie, F., and Huang, H. Multi-view k-means cluster- ing on big data. In IJCAI, pp. 2598-2604, 2013.
Lou, Z., Ye, Y., and Yan, X. The multi-fcature informa- tion bottleneck with application to unsupervised image categorization. IJCAL, pp. 1508-1514 2013.
Chao, G., Jiang, Y., and Chu, D. Incomplete contrastive multi-view clustering with high-confidence guiding. In AAAI, pp. 11221-11229, 2024.
Lu, Y., Lin, Y., Yang, M., Peng. D., Hu, P., and Peng, X.
Decoupled contrastive multi-view clustering with high- order random walks, In AAAI, pp. 14193-14201, 2024.
Chun, S., Oh, , de Rezende, R. S., Kalantidis, Y., and Larlus, D. Probabilistic embeddings for cross-modal retrieval. CVPR, pp. 8415-8424, 2021.
Mao, Y., Yan, X., Guo, Q., and Ye, Y. Deep mutual infor- mation maximin for gross-modal clustering. In AAAI, pp. 8893-8901, 2021.
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view infor- mation bottleneck. In ICLR, 2020.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y. Multimodal deep learning. ICML, pp. 689 696, 2011.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In CVPR workshops, pp. 178, 2004.
Nie, F., Li, J., and Li, X. welghteighted multiview clustering with multiple graphs. In IJCAI, pp. 2574-2570, 2017.
10

---

Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In ICML, pp. 8821-8831, 2021.
Yang, X., Jin, J., Wang, S., Liang, K., Liu, Y., Wen, Y., Liu, S., Zhou, S., Liu, X., and Zhu, E. Dealmve: Dual con- trastive calibration for multi-view clustering, In ACMMM, 2023.
Rong, Z., Liu, Z.., Song, J., Cao, L., Yu, Y., Qiu, M., and Hou, Y. Mcluster-vaes: An end-to-end variational deep learning- based clustering method for subtype discovery using multi-omics data. Comput. Biol. Med., 150:106085, 2022.
Yuan, Z., Zhang, W., Tian, C., Mao, Y., Zhou, R., Wang, H., Fu, K., and Sun, X. MCRN: A multi-source cross-modal retrieval network for remote sensing. Int. J. Appl. Earth Obs. Geoinf., 115:103071, 2022.
Si, J.. Tian, Z., Li, D., Zhang, L., Yao, L., Jiang, W., Liu, J., Zhang, R., and Zhang, X. A multi-modal clustering method for traditional chinese medicine clinical data via media convergence. CAAI Trans. Intell. Technol., 8(2): 390-400, 2023.
Zhou, S., Liu, X., Liu, J., Guo, X., Zhao, Y., Zhu, E., Zhai Y., Yin, J., and Gao, W. Multi-view spectral clustering with optimal neighborhood laplacian matrix. In AAAI, pp. 6965-6977 2020.
Song. J. and Ermon, S. Multi-label contrastive predictive coding. In NeurlPS, pp. 8161-8173, 2020.
Srivastava, N. and Salakhutdinov, Multimodal learning with deep boltzmann machines. In NeurIPS, pp. 2231- 2239, 2012.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. Proceedings Annual Aller- ton Conference Communnication Control Computing, pp. 368 083-377,11 1999.
Trosten, D. Løkse, Jenssen, R., and Kampffmeyer, M. Reconsidering representation alignment for multi-view clustering. In CVPR, pp. 1255-1265, 2021.
Von Ahn, L. and Dabbish, L. ESP: labeling images with computer game. In AAAI, pp. 91-98, 2005.
Wang, Q., Xia, W., Tao, Z., Gao, Q., and Cao, X. Deep self-supervised t-sne for multi-modal subspace clustering. In ACMMM, pp. 1748-1755, 2021.
Xia, W., Wang. T., Gao, Q. Yang, M., and Gao, X. Graph embedding contrastive multi-modal representation learn- ing for clustering. IEEE TIP, 32:1 170-1183, 2023.
Xie, J., Girshick, R. B., and Farhadi, A. Unsupervised deep embedding for clustering analysis. In ICMI., pp. 478-487, 2016.
Xu, J., Tang, H., Ren, Y., Peng, L., Zhu, X., and He, L. Multi-level feature learning for contrastive multi-view clustering. In CVPR, pp. 16030-16039, 2022.
Yan, X.. Mao, Y., Ye, Y., and Yu, H. Cross-modal cluster- ing with deep correlated information bottleneck method. IEEE TNNLS, 35(10): 13508-1352222 2024.
Yan, X., Mao, Y., Ye, Y., and Yu, H. Incremental multiview clustering with continual information bottleneck method. IEEE TSMC, 55(1):995-3066 2025.


Reviews:
[{"review_id": "BHfsBMETKO", "reviewer": "Reviewer_t5hp", "summary": "In multi-modal clustering, effectively capturing the complex relationships between modalities remains a challenge. For solving this, this paper propose a new super deep contrastive information bottleneck method to maximize the utilization of latent information in multi-modal data. It firsts introduces hidden layer information from the encoder into the clustering process to enhance modality feature representation; then, it proposes a dual contrastive learning optimization strategy The experimental results demonstrate that the method not only enhances clustering performance but also exhibits strong applicability in multi-modal data modeling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below.\n\n**Q1: It is good to see that the authors give some recent IB works on MMC problem, it is suggested to give a deeper analysis on the limitations of them. Although the differences with the proposed method is given now, some more analysis is also needed.**  \n***Response：*** Thank you for the insightful comments on the information bottleneck work in this paper.  \nBased on your suggestions, we conducted a more in-depth analysis of the limitations of recent IB works on MMC problems. These analyses were added to Section 2.1 of the manuscript as follows:\nFederici et al. [1] proposed a multi-modal IB method that can identify non-shared information between two modalities, but it only explores the correlation of different modalities through feature distribution, ignoring the consistency of cluster assignment, making the learned feature representation unfriendly to downstream clustering tasks. Yan et al. [2] proposed a multi-modal IB method that uses shared representations of multiple modalities to eliminate private information of a single modality. Yan et al. [3] further proposed an incremental IB method that builds acknowledge base to solve the clustering problem of incremental modalities. Both of the above works considered the consistency of feature distribution and cluster assignment at the same time, but they failed to consider the correlation between feature distribution and clustering results. All the above MMC IB methods ignore the rich information contained in the hidden layers of the encoder and fail to explicitly utilize it.  \nThe above limitations motivate us to the proposed SDCIB.  \n**References**  \n[1]: Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020.  \n[2]: Yan, X., Mao, Y., Ye, Y., and Yu, H. Cross-modal clustering with deep correlated information bottleneck method. IEEE Transactions on Neural Networks and Learning Systems, 2023.  \n[3]: Yan, X., Mao, Y., Ye, Y., and Yu, H. Incremental multiview clustering with continual information bottleneck method. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024.  \n\n**Q2: Some equation references is not proper, such as Eq. 2, Eq. 8. A bracket is missing throughout the whole manuscript.**  \n***Response：*** Thank you for the detailed review. In response to your specific suggestion regarding the equation references format, we have made thorough revisions to the whole manuscript. First, based on your feedback, we corrected Eq. 2 and Eq. 8 to the standardized format, Eq. (2) and Eq. (8), respectively. Additionally, we conducted a comprehensive review of the whole manuscript to ensure consistency and accuracy in all equation references. Furthermore, in order to further improve the quality of the manuscript, we not only addressed the equation references issue but also meticulously checked and adjusted all formatting details throughout the whole manuscript (including formulas, symbols, references, etc.) to ensure greater precision and rigor.\n\n**Q3: Some English usage about the writing details is suggested to be improved, such as 'cluster number K'.**  \n***Response：*** Thank you for the attention to the language details in our manuscript and for the helpful suggestions. We have carefully reviewed the whole manuscript for language issues and made detailed corrections where needed. As you suggested with *'cluster number $K$'*, we have corrected it to *'the number of clusters $K$'*, making it more standard. Additionally, we have checked for similar language issues throughout the manuscript and have made the necessary adjustments, such as *'the parameter $α$, and the parameter $β$'*, which has now been adjusted to a more fluent *'the parameter $α, β$'* to improve the accuracy and naturalness of the expression. To ensure the overall language quality, we have conducted an in-depth review of the whole manuscript's English expressions, optimizing wording and grammar to make the overall presentation more standardized and clear.  \n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "No theoretical claims here.", "experimental_designs_or_analyses": "Yes, I have checked. By conducting experiments on four multi-modal datasets, the method significantly outperforms existing methods.", "relation_to_broader_scientific_literature": "This paper propose a new super deep contrastive information bottleneck method to solve the multi-modal clustering problem.", "essential_references_not_discussed": "No essential related works are missing in the paper.", "other_strengths_and_weaknesses": "The proposed method is based on an information-theoretical method called information bottleneck which is named SDCIB and is also organized in a rigorous, theoretical sounded way. The descriptions is clear and well written, and is novel enough for this conference. I have my comments below: \n\n1) It is good to see that the authors give some recent IB works on MMC problem, it is suggested to give a deeper analysis on the limitations of them. Although the differences with the proposed method is given now, some more analysis is also needed. \n\n2) Some equation references is not proper, such as Eq. 2, Eq. 8. A bracket is missing throughout the whole manuscript.\n\n3) Some English usage about the writing details is suggested to be improved, such as 'cluster number K'.", "other_comments_or_suggestions": "I have no other comments and suggestions.", "questions_for_authors": "I have no other questions for this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SK38FEUTW", "reviewer": "Reviewer_BTqg", "summary": "This paper proposed an information bottleneck based method named SDCIB for addressing the multi-modal clustering problem, which aims to mine the complex correlations and interdependencies among modalities. It mainly contains two aspects, first, it incorporates the different hidden layers into loss functions to fully mine the relationships among modalities. Then, it also explores the consistency information among clustering assignments of modalities. The experimental results show the superiority of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below.  \n\n**Q1: The full name of some abbreviations is missing, such as KL, which may influence the readability and the understanding on the paper.**  \n***Response：*** Thank you for pointing out this issue. We have verified the full names and abbreviations you mentioned one by one and made the following improvements. For the term *'KL divergence'* you pointed out, we have provided its full form as *'Kullback-Leibler divergence'* when it first appeared to ensure the professionalism and accuracy. In addition, we conducted a thorough check of other related terms and discovered that the abbreviation *'MINE'* was also not expanded the first time it appeared. Its full name is *'Mutual Information Neural Estimation'* and we have added this clarification in the manuscript. We will ensure that all key terms in the manuscript provide their full names when they first appear and are clearly presented in the final version.  \n\n**Q2: It is seen that the improvement of the proposed method over other methods is significant. Will the source code be released to the public to enhance the development of the multi-modal clustering community？**  \n***Response：*** We sincerely appreciate your recognition of the effectiveness of the proposed SDCIB. We fully understand the importance of code availability in promoting research transparency and advancing the field of multi-modal clustering. Currently, we have organized and optimized the code to ensure its clarity and readability. We plan to release the source code after the official publication of the paper, hoping that it will contribute to the research community and support further advancements in this field.  \n\n**Q3: Are there any limitations of the proposed method? The authors are encouraged to give them in the conclusion.**  \n***Response：*** We sincerely appreciate your valuable suggestion. In response, we have revised the conclusion to include the following limitations of the proposed SDCIB:  \n* The proposed SDCIB shows limited performance when handling incomplete data samples, particularly when certain modalities or features are missing. In such cases, the model may struggle to accurately learn the relationships between the modalities, which could lead to suboptimal clustering results.  \n* The number of clusters must be known in advance as most existing multi-modal clustering methods. This requirement can be restrictive, as it assumes prior knowledge of the data's underlying structure.  \n* The proposed SDCIB primarily relies on batch learning, which may not be suitable for certain applications, such as streamed multi-modal data.  \n\nAdditionally, we acknowledge that while these limitations may affect certain scenarios, we will explore solutions to address them, with the aim of further enhancing the method’s robustness and applicability.  \n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No proofs and theoretical claims in this paper.", "experimental_designs_or_analyses": "I have carefully checked the soundness and validity of the experimental designs and their analysis, including all the subsections in Sec. 3.", "relation_to_broader_scientific_literature": "This paper proposed an information bottleneck based method named SDCIB for addressing the multi-modal clustering problem. I find the above is the key contributions, and is not proposed in prior findings or results.", "essential_references_not_discussed": "The related works are essential to understand the key contributions and no important ones are not discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a well-articulated motivation, with its effectiveness rigorously validated through multiple experiments. \n2. This paper exhibits a notable advancement in multi-modal clustering field, demonstrating superior performance over existing methods. \n3. The authors offer a comprehensive and well-structured explanation of the method, effectively highlighting its significance. \n\n\nWeaknesses:\n1. The full name of some abbreviations is missing, such as KL, which may influence the readability and the understanding on the paper. \n2. It is seen that the improvement of the proposed method over other methods is significant. Will the source code be released to the public to enhance the development of the multi-modal clustering community？\n3. Are there any limitations of the proposed method? The authors are encouraged to give them in the conclusion.", "other_comments_or_suggestions": "I have no other comments or suggestions. Please see my comments above.", "questions_for_authors": "I have no other questions for authors, all my questions are shown Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XF2zFpJdac", "reviewer": "Reviewer_aWGf", "summary": "This paper proposes a Super Deep Contrastive Information Bottleneck (SDCIB) for multi-modal clustering, designed to fully exploit the latent information in multi-modal data. SDCIB integrates the rich information from the hidden layers of the encoder into the clustering process, optimizing both feature distribution and clustering assignments through contrastive learning. Experimental results on four multi-modal datasets demonstrate that SDCIB outperforms existing approaches.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions.\n\n**Q1: The paper discusses the use of deeper hidden layers to explore relationships, but the specific advantages of this approach are not explicitly outlined.**  \n***Response：*** Thank you for the insightful comment. The use of hidden-layer information provides the following advantages:  \n* Richer Feature Representations: As the depth of the network increases, each hidden layer progressively refines the feature representations, capturing both low-level and high-level structures. This enables a more informative and structured representation, which is particularly beneficial for clustering tasks.  \n* Improved Feature Compression: Deeper layers allow for a more effective compression process, where redundant information is filtered out while preserving essential details. By leveraging intermediate hidden-layer information, the model learns a more compact and meaningful feature representation.  \n* Optimization Benefits in the Information Bottleneck Framework: Introducing hidden-layer information in the IB framework helps regulate the trade-off between compression and retention of relevant information, leading to more discriminative and generalizable clustering results.  \n\nWe will incorporate a more detailed discussion of these aspects in Section 2 to clarify the advantages of using deeper hidden layers.  \n\n\n**Q2: The authors utilize MINE to estimate mutual information. It would be helpful if the paper included a discussion of why MINE was chosen over other potential estimation methods. Were alternative methods considered and, if so, why were they not chosen?**  \n***Response：*** Thank you for the suggestion. During the experiment, we did consider other methods such as variational mutual information estimation and InfoNCE. However, for mutual information $I(A;B)$, $A$ is a feature and $B$ is the label of the cluster, which results in the dimension of $A$ being much larger than that of $B$. We found that:  \n* The variational method requires manual alignment of the dimensions of $A$ and $B$, which may cause information loss during the alignment process.  \n* The InfoNCE method requires the construction of a large number of negative samples, and due to the small dimension of $B$, we found that this may lead to insufficient discrimination between positive and negative samples, thus affecting the accuracy of the estimation.  \n\nIn contrast, MINE provides a more robust mutual information estimation method, especially when there is a significant dimensional difference between the two variables. This method can estimate more efficiently and accurately without the need for dimensional alignment or worrying about insufficient discrimination of negative samples. Therefore, we chose MINE. \n\n**Q3: There are minor formatting and writing inconsistencies. For instance, ensure consistency in the usage of terms like \"Information\" on Page 3, and correct the reference to \"IJCAL\" in the bibliography.**  \n***Response：*** Thank you for the careful review. We have carefully reviewed the manuscript and made the necessary revisions. For example, the two instances of *'Information'* on page 3 had different formatting, and we have standardized them to title case as *'Hidden-layer Information'* and *'Consistency Information.'* Additionally, regarding the reference to *'IJCAL'* that you mentioned, we have corrected it in the references to: Nie, F., Li, J., and Li, X. Self-weighted multiview clustering with multiple graphs. In *Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI2017)*, pp. 2564–2570, 2017. Furthermore, we have checked all other references to enhance its overall precision. \n\n**Q4:The authors are encouraged to provide quantitative descriptions of the results. For example, \"The experiment shows that the method is effective\" should be changed to \"The accuracy of the method on the X dataset is improved by 5%\"**  \n***Response：*** Thank you for the detailed feedback on the content of the abstract. We have revised the abstract accordingly. Specifically, we changed the sentence *'We conduct experiments on 4 multi-modal datasets.'* to *'We conducted experiments on 4 multi-modal datasets and the accuracy of the method on the ESP dataset improved by 9.3%.  The results demonstrate the superiority and clever design of the proposed SDCIB.'* This adjustment enhances the precision of the expression and more intuitively highlights the effectiveness of the proposed SDCIB.  \n \nThanks again. The modifications will be added to the final version.", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The experimental results on multiple multi-modal datasets consistently demonstrate that the SDCIB method outperforms existing approaches.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the multi-modal clustering problem. SDCIB effectively leverages the rich information from the encoder's hidden layers, optimizing both feature distribution and clustering assignments through contrastive learning, which aligns to improve clustering performance in multi-modal settings.", "theoretical_claims": "The paper does not present any formal proofs or theoretical claims.", "experimental_designs_or_analyses": "I have reviewed the soundness and validity of the experimental designs and analyses in Section 3, including subsections 3.1 to 3.9. The experimental setup, dataset selection, evaluation metrics, and comparative analysis are appropriately designed to support the paper's claims.", "relation_to_broader_scientific_literature": "This paper proposes a Super Deep Contrastive Information Bottleneck (SDCIB) method for multi-modal clustering in the broader scientific literature. It improves upon existing ideas in multi-modal clustering, contrastive learning, and information bottleneck, offering more efficient and powerful clustering solutions for multi-modal data.", "essential_references_not_discussed": "Critical references have been included.", "other_strengths_and_weaknesses": "The paper presents a well-structured argument and is written with clarity. Below are my detailed comments: \n1. The paper discusses the use of deeper hidden layers to explore relationships, but the specific advantages of this approach are not explicitly outlined. I recommend the authors provide a more detailed explanation of the benefits of using deeper layers in Section 2.\n2. The authors utilize MINE to estimate mutual information. It would be helpful if the paper included a discussion of why MINE was chosen over other potential estimation methods. Were alternative methods considered and, if so, why were they not chosen?\n3. There are minor formatting and writing inconsistencies that could be addressed to improve the paper. For instance, ensure consistency in the usage of terms like \"Information\" on Page 3, and correct the reference to \"IJCAL\" in the bibliography.", "other_comments_or_suggestions": "I have one suggestion regarding the abstract: the authors are encouraged to provide quantitative descriptions of the results. For example, \"The experiment shows that the method is effective\" should be changed to \"The accuracy of the method on the X dataset is improved by 5%\".", "questions_for_authors": "Please refer to the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6wvkmQ0lzx", "reviewer": "Reviewer_kMYo", "summary": "To fully explore the complex latent information and interdependencies among multi-modal data, this paper propose a super deep contrastive information bottleneck for multi-modal clustering method. It incorporates the rich information from the hidden layers of the encoder into the clustering process to comprehensively capture modality features and their associations. Furthermore, a dual contrastive learning strategy is designed to ensure more precise and stable clustering performance. Experimental results show the promising performance by the method, which fully validate its advantages and well-designed framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and constructive suggestions. We have carefully revised the whole manuscript and provided detailed responses to each point below.\n\n**Q1: Hidden layer information integration mechanism: Is it a shared encoder or an independent encoder? How to fuse the hidden layer information of different modalities?**  \n***Response：*** Thank you for the insightful question. Below, we clarify the integration mechanism of hidden layer information in our approach.  \nInstead of a shared encoder, we adopt an independent encoder for each modality. Different modalities usually have different statistical properties, making it difficult to directly fuse their hidden layer information at an early stage. This design allows each modality to learn modality-specific feature representations, which is beneficial given the different statistical properties of various modalities.  \nInstead of performing direct fusion at the hidden layer level, we compare the hidden layer representations of different modalities to extract shared and complementary information. Specifically, each encoder learns to align its representations by evaluating the relationship between its hidden layer features and the hidden layer features of other modalities. We achieve this by maximizing the mutual information of hidden layer features between different modalities, which encourages aligned representations while preserving modality-specific features.  \nBy adopting this strategy, we ensure that the final multimodal representations are both consistent and rich, leading to a more efficient feature learning process. We hope this explanation clarifies our integration mechanism.  \n\n**Q2: Improper language expression: The lengthy sentence structure affects readability.**  \n***Response：*** Thank you for the constructive comments on sentence structure. To enhance the readability of the manuscript, we have made extensive revisions. We focused on checking redundant sentences and reducing them through segmentation or conciseness to improve clarity. For example, the sentence in the *'Introduction'* section, *'The proposed SDCIB can efficiently and meticulously mine the latent information between modalities through the hidden layers, and simultaneously focus on both feature distribution and clustering assignment to better capture the inherent structure of the data,'* has been revised to: *'The proposed SDCIB efficiently mines the latent information between modalities through the hidden layers. Meanwhile, it simultaneously considers both feature distribution and clustering assignment to better capture the inherent structure of the data.'* In the final version, we will adjust lengthy expressions into clearer and more readable sentences.\n\n**Q3: The experimental data is not public: it is recommended to provide a data download link to improve the reproducibility of the paper.**  \n***Response：*** We appreciate your suggestion and realize the importance of reproducibility in scientific research. Currently, our experimental section already cites the original source of the dataset to provide transparency of its origin. However, we also understand that providing a direct download link can further improve the accessibility of readers. We will revise the manuscript to add specific download links to ensure that it is publicly available and reproducible.\n\n**Q4: Shortcoming is missing: The shortcomings of the proposed method are not given in the paper. Generally, for a conference paper, advantage and limitation analysis are the major parts.**  \n***Response：*** Thank you for the insightful comment. Based on your suggestion, we have updated the conclusion to include the following limitations of the proposed SDCIB:  \n* The proposed SDCIB demonstrates limited effectiveness when dealing with incomplete data, particularly when certain modalities or features are missing. In these cases, the model may face difficulties in accurately capturing the relationships between modalities, which could result in less optimal clustering outcomes.  \n* The method requires the number of clusters to be predetermined, which may be a limitation as it assumes prior knowledge of the data's structure, making it less flexible in scenarios where this information is unavailable.  \n* The proposed SDCIB primarily relies on batch learning, which may not be suitable for certain applications, such as streamed multi-modal data.\n\nWhile we recognize that these limitations could impact certain scenarios, we believe that future research can explore ways to address them, with the goal of improving the method’s robustness and expanding its range of applications.  \n\nThanks again for the valuable suggestions provided by the reviewer. The modifications will be added to the final version.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence in the experimental parts in section 3.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem in this paper. This paper propose a super deep contrastive information bottleneck for multi-modal clustering method, and the experimental results show the promising performance by the method, which fully validate its advantages and well-designed framework.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "I have checked the soundness/validity of any experimental designs or analyses in the experimental sections.", "relation_to_broader_scientific_literature": "This paper proposes a novel super deep contrastive information bottleneck for multi-modal clustering method, and it is different from prior related findings.", "essential_references_not_discussed": "There is no essential references not discussed in this paper.", "other_strengths_and_weaknesses": "This paper propose a novel super deep contrastive information bottleneck for multi-modal clustering, which is new and well-organized. The experiments are well-designed and have shown its advantages over existing methods. However, there are still some concerns shown below:\n1.Hidden layer information integration mechanism: Is it a shared encoder or an independent encoder? How to fuse the hidden layer information of different modalities? \n2.Improper language expression: The lengthy sentence structure affects readability.\n3.The experimental data is not public: it is recommended to provide a data download link to improve the reproducibility of the paper.\n4. Shortcoming is missing: The shortcomings of the proposed method are not given in the paper. Generally, for a conference paper, advantage and limitation analysis are the major parts.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "Please see the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on how to deeply explore the complex latent information and interdependencies between modalities in multi-modal clustering. The authors introduce the rich information contained in the encoder's hidden layers into the loss function to mine both modal features and the hidden modal  relationships. They perform dual optimization by concurrently considering consistency information from both the feature distribution and clustering assignment perspectives to improve the clustering accuracy and robustness.  Experimental results illustrate the effectiveness of the presented method.

After rebuttal, the raised concerns are solved, all reviewers are positive about this manuscript and recognize its contributions.  After reading the manuscript and responses, I agree with reviewers' opinions and recommend an acceptance for this manuscript.
Decision: Accept (poster)