A Novel Approach for Effective Multi-View Clustering with Information-Theoretic Perspective
Authors: Chenhang Cui
Yazhou Ren
Jingyu Pu
Jiawei Li
Xiaorong Pu
Tianyi Wu
Yutao Shi
Lifang He
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-view learning， clustering

Abstract: Multi-view clustering (MVC) is a popular technique for improving clustering performance using various data sources. However, existing methods primarily focus on acquiring consistent information while often neglecting the issue of redundancy across multiple views.
This study presents a new approach called Sufficient Multi-View Clustering (SUMVC) that examines the multi-view clustering framework from an information-theoretic standpoint. Our proposed method consists of two parts. Firstly, we develop a simple and reliable multi-view clustering method SCMVC (simple consistent multi-view clustering) that employs variational analysis to generate consistent information. Secondly, we propose a sufficient representation lower bound to enhance consistent information and minimise unnecessary information among views. The proposed SUMVC method offers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data. 
 To verify the effectiveness of our model, we conducted a theoretical analysis based on the Bayes Error Rate, and experiments on multiple multi-view datasets demonstrate the superior performance of SUMVC.

A Novel Approach for Effective Multi-View Clustering with Information-Theoreti Perspective
Chenhang Cui'; Yazhou Ren" Jingyu Pu', Jiawei Li', Xiaorong Pul, Tianyi Wu', Yutao Shi', Lifang He² University Electronic Science and Technology of China, China
2 Department of Computer Science and Engineering, Lehigh University, USA chenhangeuie @ gmail.cm, yazhou.ren @ westc..du..nn, pujingyu0105e 163.com, puxiaor@ uestc.edu.en, sixteen.l.jw @ gmail.com. syt 59421 @ coml tianyi- wu @ outlook.com, lih3 19@leighheeed
Abstract
Multi-view clustering (MVC) is popular technique for improving clustering per- formance using various data sources. However, existing methods primarily focus acquiring consistent information while often neglecting the issue of redundancy across multiple views. This study presents a new approach called sufficient multi- view clustering (SUMVC) that examines the multi-view clustering framework from an information-tee -heorretic standpoint. Our proposed method consists of two parts. Firstly, we develop simple and reliable multi-view clustering method SCMVC (simple consistent multi-view clustering) that employs variational analysis to gener- ate consistent information. Secondly, propose sufficient representation lower bound to enhance consistent information and minimise unnecessary information among views. The proposed SUMVC method offers a promising solution the problem of multi-view clustering and provides new perspective for analyzing multi-view data. To verify the effectiveness of our model, we conducted a theoreti- cal analysis based on the Bayes Error Rate, and experiments on multiple multi-view datasets demonstrate the superior performance of SUMVC.
Introduction
Clustering is fundamental unsupervised learning task with broad applications across various fields. Traditional clustering methods primarily focus on analyzing single-view data. However, the rapid progress in multimedia technology has led to the collection of real-world data from multiple sources and perspectives, resulting in the emergence of multi-view data. Recently, multi-view clustering has gamnered significant attention due its ability to leverage information from multiple views to improve clustering performance and robustness 24] Multi-view data arise in many applications where multiple sources of information or perspectives clustering aims to group these objects into clusters by considering all available views jointly. This technique has proven be highly effective in various domains, including social network analysis (6). image and video analysis [50]. and bioinformatics [4]. Nevertheless, the main challenge in multi-view clustering lies in extracting valuable information from the views for clustering purposes while simultaneously minimizing view redundancy.
the same set of objects are available. Multi-view
Information bottleneck (IB) theory provides a theoretical framework for understanding the optimal trade-off between accuracy and compression in data processing [36]. The IB method suggests that
'These authors contributed equally.
*Coresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the optimal representation of the data can be achieved by maximizing the mutual information be- tween the input and the output while minimizing the mutual information between the input and the representation. Although this principle provides a theoretic method to learn less superfluous information for downstream tasks, existing methods can still be improved in efficiently extracting embedded features from multi-view unlabeled data. In recent years, many information-theonee ods have been applied to multi-view scenarios. Specifically, [44] proposes a generic pretext-aware residual relaxation technique to enhance multi-view representation learning (MVRL). [S]] propose novel approach to MVRL through an information-theorrtic framework grounded in total correlation. Meanwhile, variational analysis has gained wide usage in machine learning and optimization for approximating complex probability distributions [3] This technique offers a robust framework for solving optimization problems involving probability distributions, which are frequently encountered in multi-view clustering and information bottleneck. By approximating the probability distributions, variational analysis can effectively reduce the data dimensionality and enhance the efficiency of clustering algorithms. Variational autoencoders (VAEs) are a popular type of generative models that uses variational inference learn compact representations of high-dimensional data (18). How- ever, the vast majority of them are designed for single-view data and cannot capture consistent and complementary information in multi-view setting.
meth-
To tackle the aforementioned concerns, we delve deeper into the principles of the information bottleneck and variational analysis in the realm of multi- view clustering. We propose simple consistent multi-view clustering method SCMVC and a novel multi-view self-supervised method SUMVC from an information -theoretic standpoint. SUMVC could strengthen consistency across views for clustering while concurrently decreasing view redundancy.
Our contributions are three-fold:
We propose a consistent variational lower bound to explore the consistent information among views for multi-view clustering. We introduce SCMVC, simple consistent multi-view clustering approach, utilizing the proposed lower bound. Despite being a simple approach, experiments demonstrate the effectiveness of SCMVC.
We extend the information bottleneck principle the multi-view self supersupe learning and propose a sufficient representation lower bound. We expand SCMVC to improve the consistency and reduce the redundancy. This mainly aims to acquire predictive information and reduce redundancy among views. We refer to this new model as SUMVC (sufficient multi-view clustering).
We adopt the collaborative training strategy for SUMVC to better adapt to multi-view scenarios. Additionally, a theoretical analysis of the generalization error of representations is conducted to evaluate the efficacy of SUMVC for clustering tasks. Experiments on real-world multi-view data show the superior performance of the proposed model.
2 Related Work
Information Bottleneck The information bottleneck (IB) principle is a widely used framework in information theory and machine learning that provides rational way understand the balance between compression and prediction. Introduced by [36], the IB principle formalises the intuition that successful learning algorithms need find balance between retaining relevant information and discarding irrelevant information. Since then, the IB principle has been widely applied to various machine learning tasks, including unsupervised learning (37), classification 3. and clustering 2 5T) Recently, the IB principle has also been used to study the information processing capacity of the brain By constraining the learned representations of deep neural networks, the IB principle was shown to improve the generalization performance of these models and reduce their computational complexity (2). In addition, the IB principle has been used to develop new techniques for training deep neural networks, such as the deep variational information bottleneck (2). The theoretical foundation of the IB principle is built on concepts such as Markov chains, entropy, and conditional entropy [5]. It has been widely applied in various fields, such as data mining, image processing, natural language The IB principle has also been applied in control theory to quantify the amount of [31] Overall, the IB principle has become an important tool for designing more efficient and effective machine learning algorithms. Its widespread applications various fields highlight its relevance and practicality.
33 34
processing, and computer vision 10 16 35
2

---

Varitional Autoencoder Variational autoencoders (VAEs) are a popular generative model class that learns compact representation input data through an encoder-decoder architecture. Unlike traditional autoencoders, VAFs are designed to model the latent space distribution of input data, enabling them to generate new data samples consistent with the input distribution. Early work on VAES by [8]] introduced the VAE model and derived its objective function, while [9 proposed method for approximating gradients by sampling the latent variables. Since then, VAEs have been applied to various applications, including image generation, data compression, and anomaly detection. For instance, [T combined VAEs with generative adversarial networks (GANs) to generate high- quality images, while (25) applied VAEs detect anomalies in seasonal key performance indicators (KPIs). Moreover, VAEs have been combined with other deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to improve their performance various tasks. For example, [28] combined VAES with CNNs to extract hierarchical features from images. It is worth mentioning the characteristic of variational autoencoders is incorporated into our model. In summary, VAEs are powerful for generative modeling and have contributed significantly to the field of deep learning. Deep learning with nonparametric clustering is a pioneer work in applying deep belief network to deep clustering. But in deep clustering based on the probabilistic graphical model, more research comes from the application of variational autoencoder (VAB), which combines variational inference and deep autoencoder together.
Multi- View Clustering In practical clustering tasks, the input data usually have multiple views. Multi- vieti clustering (MVC) methods are proposed to address the limitations of traditional clustering methods by leveraging the complementary information provided by different views of the same instance. Numerous algorithms and techniques have been proposed for MVC, ranging from classic methods such as spectral clustering more recent deep learning and probabilistic modeling develop- ments. One popular approach is co-training, which was first introduced by 4 Co-training involves training multiple models, each using a different view of the data, and then using the models to label additional unlabeled data. This approach has been successfully applied in various domains, including text classification [56), image recognition (2T). and community detection 244 Another popular approach is low-rank matrix factorization, which aims to learn a shared latent representation of the data across multiple views. This approach has been explored in various forms, such as structured low-rank matrix factorization [55], censor-based factorization [4]]. and deep matrix factorization [54]. Subspace methods have been utilized in MVC in the previous few years. c.g., [ 26 30
integrated multiple affinity graphs into consensus one with the topological relevance considered.
In recent years, deep learning has emerged as a powerful tool for MVC, with various deep neural network architectures proposed for the task. For example, [3 proposed a deep adversarial MVC method, which learns a joint representation of the data across multiple views using adversarial training. [46] proposed deep embedded clustering (DEC) to learn a mapping from the high original feature space lower-dimensional one in which pratical objective optimized. [48] proposed a framework of multi-level feature learning for contrastive multi-view clustering (MFLVC), which combines MVC with contrastive learning improve clustering effectiveness.
3 The Proposed Method
Self-supervised learning methods for MVC 52 57 58 aim at reducing redundant information and achieving sufficient representation from multiple views. In the multi-view scenario, there are also many methods [3 49 that explore consistent information between views. Despite great efforts, the task of minimizing redundancy and extracting valuable information for subsequent processes while maintaining coherence across perspectives continues to pose significant challenges. In this section, inspired by [9] 39], we extend the principle of the information bottleneck, utilize the reparameterization trick for solving the proposed lower bound and introduce the collaborative training strategy to the MVC scenario. Moreover, we conduct bayes error rates for learned representations to validate the generalization performance of our method.
Problem Statement Given the multi-view data, X, which consists of y subsets such that x
[x|:...: x E .R is the x ddddimensional data of the i-th view, where n is the number of instances and d is the number of dimensions of the i-th view. MVC aims to partition the n instances into K clusters.
3

---

3.1 Preliminaries
Analysis of redundancy According to 19. we can assumee that z contains all necessary information for predicting y without having prior knowledge of y. This can be achieved by maintaining all the shared information between the two views x' and x/. This assumption is grounded on the belief that both views provide similar predictive information. The following formal definition elucidates this concept (See details in Appendix B.2.).
Definition 3.1 191 xf considered redundant with respect to x! for y if and only if the mutual information x(yxxxx>)+ O.
Intuitively, x' is considered redundant for task if it is irrelevant for the prediction of y when x! already observed. Whenever x' and x'are mutually redundant with respect to one another (x' is redundant with respect to x for y), we have the following:
Corollary 3.1 Let x² and x² be two mutually redundant views for a target and let z' be a represen- tation of xt. If is sufficient for x! (1(xx:x1 0) then ef as predictive for y as the combination of the two views (I(x'x/:y)= = I(y:z)).
Variational bound The variational bound [8]. also known as the Evidence Lower Bound (ELBO), widely used optimization objective in Bayesian inference and machine learning. It provides lower bound on the marginal likelihood of the observed data. Variational bound provides an efficient approximate posterior inference of the latent variable z' given an observed value x,, which can be formulated as:
log qo (x)) 2 L(o'. ,';xx') = Bp log (e((()x)) + log g (2.2))
(1)
where of and d' are generative and variational parameters of the i-th view respectively. Through optimizing the lower bound L(6', x'),,, we can approximate the posterior inference of the latent variable accurately based on an observed value of x.. As such, we can obtain more representative embedded features.
3.2 SCMVC: Learning Consistent Information via Varational Lower Bound
Variational inference has greatly contributed representation learning. Introducing variational inference allows representation learning to be performed in probabilistic manner thereby providing a principled framework for capturing uncertainty in the learned representation. Variational inference can optimize the generative model, resulting in more informative data representation. In this section, we analyze the consistent information among views from the vartional inference perspective and propose SCMVC (simple consistent multi-view clustering). We introduce a consistent varitional lower bound to model the latent space distribution input data more consistent with all views. Given z"), v-views and global embedded features غ = (2'.2,... .'.....z'' we formalize this concept the following theorem (Proofs are given in Appendix B.3.):
Theorem 3.1 (Consistent Variational Lower Bound) Define the generative model qo(x1(z, y)), and posterior as po(Z, y'(x') (o.(ar) The ELBO for multi-view model can be formulated
max L. Ep qo ytt../tott..yu
(2)
the first term is to maximize the reconstruction probability of input x', i.e.. maximize the decoding probability from latent variable i to x'. The constraint term the KL divergence of the approximate from the true posterior to ensure that the distribution of acquired features and labels is consistent with their intractable true posterion It can be intuitively understood that our goal to obtain the unique distribution of the embedded feature Ž of for different categories.
where the y is the pseudo-label x2 1t
For better understanding, Eq. can be expanded out as:
po(t)x')
max Loom BB (o
poo' log +KUppo//EEMM (3) (a))))

---

Overall, the objective function of SCMVC is:
[ "uee -
im1
where Lice is the reconstruction term and k is the constraint term which approximates the true posterior.
3.3 SUMVC: Learning Sufficient Representation via Reducing Redundancy
Although SCMVC ensures consistency among views, there is lack of consideration for reducing mutual redundancy and achieving sufficiency among views, which is important to obtain latent representations suitable for downstream tasks. Next, advance work and extend the information bottleneck principle to the multi-view setting. By decomposing the mutual information between x² and e (See Appendix B.I.), we can identify two components:
(x(x = (, xxxx) + (x(xx)
where the first term denotes superfluous information and the second term denotes signal-relovant information. We take view self-supervised signals to guide the learning of view i. To ensure that the representation sufficient for the task, we maximize the mutual information between and z!. As for Lu xxx"), this term indicates the information in z! that is exclusive to x' and cannot be predicted by observing x². As we assume mutual redundancy between the two views, this information is deemed irrelevant to the task and could be safely discarded [9]. Overall, to extract sufficient representation of z', we maximize the following formulation:
max -o (xxx) Ia, (x).2).
(5)
which can be optimized via maiximazing the lower bound in Theorem 3.2
Theorem 3.2 (Sufficient Representation Lower Bound) Considering a multi-view model, the gen- erative model is (x12.,y)) y"). The sufficient representation lower bound can be defined as:
max suf = -Dxx(Po(e))x>) (=(() Iow((''.2))
(6)
Proof Considering zl and z! on the same domain Z, we have
Po z|x² xf) P =//// = x²)
=1((222())) pot Bo-p
log
(7)
Po =|x²² (z) dx' = x²)
log
(8)
Parlzl = z|x' = = xx)
=-Be ot
(9)
2 (
(10)
For 1x(x)22) ) )), it can be lower bounded as:
1g(x), 2)) (P2) - Inw(deex Iow((())) (ew(t:x
2 (2:0(22)2))
where Iaa(te x/100//0)))))000 when z! is sufficient for z!. This occurs whenever contains all the necessary information about z!. Therefore, Eq. can be optimized via maximizing the lower bound:
-Dxalpo(elx')
(e z/).
In addition, we adopt collaborative training to allow mutual supervision to better adapt to multi-view scenes. The views are selected in turn to act as the supervisory view, while the remaining views serve as the supervised views. This learning approach could efficiently use the common information and complementary information of multiple views.
5

---

SUMVC is derived by incorporating a lower bound on sufficient representation into SCMVC, the objective of which can be defined as follows:
( zon
B1
(11)
jeljiii
where the j-th view can be regarded as the supervisory view, while the j-th view is the supervised view. Through the collaborative training we could reduce redundancy and obtain consistent information among views. What's s more, we align the embedded features and category distribution with the original distribution obtain information conducive clustering. Finally, in order to better utilize global features, we run K-means on 2 to obtain the clustering results.
3.4 The Reparameterization Trick for Problem Solving
In order to solve the problem, let z! be a continuous random variable, and z! pa(('(x)) be some conditional distribution of the i-th view. It is then often possible express the random variable ef as a deterministic variable 전' = (c', xx), where is an auxiliary variable with independent marginal g(t) and function (a(() parameterized by f¹. We let ट' (<(l)) N(r,, o²) In this case, valid reparameterization is z' = μ¹ o'é' where e' is an auxiliary noise variable N(0, 1).
4 Discussion
4.1 Connection to VaDE
Variational deep embedding (VaDE) 17 is an unsupervised generative clustering algorithm, which combines VAE and Gaussian Mixture Model (GMM) model the data generation process. In our method, from Eq. 3 it can be found that SCMVC has taken into account the consistency information among views compared with the VaDE for the single-view scenario. Moreover, we introduce the SUMVC to help obtain a more sufficient representation, and we propose new methods to solve the problem based on the reparameterization trick.
4.2 Bayes Error Rate for Downstream Task
In this section, inspired by 8 39
present theoretical analysis of the generalization of representations, assuming T to be a categorical variable. We take view j as self-supervised signals to guide the learning of view i example. To represent the irreducible error when inferring labels from the representation via an arbitrary classifier, we use Bayes error rate Pe == Be- max P(ř rl2'),, where Pe the Bayes error of arbitrary learned representations 2. and is the estimated label T from our classifier. Analyzing the Bayes error rate allows to gain insights into the generalization performance of our propose model. We present general form of the complexity of the sample with mutual information I(z'; x²) estimation using empirical samples from the distribution Pe.v. Let dx denote the uniformly sampled empirical distribution of P.... We ((ell)))
define 1t (c': x as Epm [log
ple)
Proposition 1 (Mutual Information Neural Estimation (0).) Let 5 < 1. There exists d e N and a family of neural networks F := fo C Rd, where θ is compact, so that 30 e O with probability at least - over the draw of (z A ~ pan
+ log(
1 - 0
(12)
This proposition shows that there exists neural network 0*, with high probability, (ef: can approximate x/) with samples rate O Vn Under this network 6 and the same parameters d and 6, are ready to present main results the Bayes error rate.

---

Theorem 4.1 (Bayes Error Rates for Arbitrary Learned Representations (391.) Let (T) be T's cardinalitiy and Th(x) min/max(x, 0), m Given Pe Th(p.). we have a thresholding function:
+ log( )
De s -xx-- (eni)++(ee:ee (e:x)e) (2:.)) +
(13)
n
Given arbitrary learned representations (c'). Theorem 4.1 suggests the corresponding Bayes error rate (Pe) is small when: 1) the estimated mutual information (c': is large; 2) larger number of samples are used for estimating the mutual information; and 3) the task- iiseeirreeevant information xxx', 7) is small. The first and the second results support the claim that maximizing (z';x)) may learn the representations beneficial to downstream tasks. The third result implies the learned representations may perform better on the downstream task when the task-irrelevant information is small. We evaluate the quality of our self-supervised method's learned representations using Bayes Error Rates in Theorem 4.2 The proofs are given in Appendix B.4.
Theorem 4.2 The Bayes Error Rates for arbitrary learned representations can be minimized by maximizing 14 + formally:
max Leon + BL sas + min P..
(14)
Experiments
5.1 Experimental Setup
Table 1: The statistics of experimental datasets.
Dataset
#Samples
#Views
#Clusters
Multi-MNIST
70000
2
10
Multi-Fashion
10000
3
10
Multi-COIL-10
720
3
10
Multi-COIL-20 1440
3
20
Datasets As shown in Table [ we use the following four real-world multi-view datasets our study. MNIST 20 is widely used dataset of handwritten digits from 0 to 9. The Fashion dataset [45] comprises images various fashion items, including T-shirts, dresses, coats, etc. The COIL dataset [29 contains images of various objects, such as cups, ducks, and blocks, shown in different poses. We use multi-view datasets derived from origin datasets: Multi-COIL-10, Multi-COIL-29 Multi-MNIST and Multi-Fashio Each dataset includes multiple views of each example, all randomly sampled from the same category. In Multi-COIL- 10 (K = 10) and Multi-COIL-20 (K = 20), different views of an object correspond to various poses, but retain the same label. In Multi-MNIST. different views of a digit represent the same digit written by different individuals. Multi Multi-Fashion, different views of a product category signify different fashionable designs for that category.
Comparison Methods The comparison methods include three single-view clustering methods: K- means (1967) (27). B -VAR (B- VAF: learning basic visual concepts with a constrained variational framework (2017) (2),, and VaDE (variational deep embedding: an unsupervised and generative approach to clustering (2016) (177), the input of which is the concatenation of all views, and five sfate-of-the-an MVC methods: BMVC (binary multi-view clustering (2018) [53] ). SAMVC (self- paced and auto- weighted multi-view clustering (2020) (2)). RMSL (reciprocal multi-layer subspace learning for multi-view clustering (2019) (22). DEMVC (deep embedded multi-view clustering with
collaborative training (2021) 47 FMVACC (fast multi-view cocoooccoreespondence clustering (2022) (31).
Evaluation Mcasures Three measures are used to evaluate the clustering results, including, clustering accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). larger value of these metries indicates better clustering performance.
7

---

Table 2: Clustering results of all methods four datasets. The best result in each row is shown in bold and the second-best is underlined.
Datasets
K-means B-V VaDE BMVC SAMVC RMSL DEMVC FMVACC SCMVC SUMVC
ACC
Muli-MMNIST
53.9
49.3
30.7
89.3
98.2
55.6
94.5
99.8
Multi-Fashion
47.6
51.3
40.6
62.2
77.9
77.9
78.6
77.4
84.9
91.4
Mulii-COI.-10
73.3
59.8
32.5
66.7
67.8
96.4
89.1
93.2
98.1
100.0
Multi-COIL-20
41.5
53.1
20.3
57.0
83.4
66.5
85.0
75.8
83.7
100.0
NMI
Multi-MNIST
48.2
43.6
35.4
90.2
98.9
48.2
87.3
99.3
Multi-Fashion
51.3
51.0
53.7
75.6
68.8
75.6
90.3
73.8
80.9
87.5
Malti-COIL-10
76.9
68.5
44.8
68.1
82.6
92.5
89.7
93.4
96.7
100.0
Multi -00i
64.5
66.7
36,9
90.0
79.1
76.3
96.5
84.9
90.8
100.0
ARI
Multi-MNIST
36.0
29.1
8.5
85.6
98.6
45.2
88.5
99.5
Multi-Fashion
34.8
33.7
22.8
68.2
55.7
68.2
77.2
70.3
75.6
84.1
Multi-COIL-10
64.8
51.4
18.1
53.0
62.1
92.1
89.7
92.5
95.8
100.0
Mult--COIL-20
38.4
45.0
9.0
81.3
55,4
58.7
86.0
79.1
79.9
100.0
Table 3: Ablation studies of SUMVC on loss components.
Components
Multi-MNIST
Multi-COIL-21
Tree
LxLL
Lanf
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
d
x
x
70.5
72.9
62.6
56.7
57.6
47.7
62.5
81.1
63.3
x
94.5
87.3
88.5
84,9
80.9
75.6
83.7
90.8
79.9
/
11.5
0.2
0.1
11.3
0.1
0.1
9.9
4.6
0.0
/
99.8
99.3
99.5
91.4
87.5
84.1
100.0
100.0
100.0
5.2
Comparison with State-of-the-Art Methods
Table 2 shows the quantitative comparison between the proposed methods and baseline models for several datasets. The best result is highlighted in bold, while the second-best result underlined in each line. Despite its simplicity, the SCMVC method demonstrates similar performance to existing methods in most cases. This primarily because our method focuses on extracting global information from multiple distributions. Moreover, the results reveal that SUMVC outperforms existing methods terms of three evaluation methods. Notably, SUMVC achieves remarkable improvements on Multi-COIL- and Multi-COIIL-20 The reason is that the proposed lower bound could explore consistent predictive information and reduce redundancy among views, which benefits clustering results.
5.3 Ablation Studies
To further verify the contributions of the proposed method, we conduct ablation studies on Eq. 11] which consists of Lyee LKI, and Lan-- As shown in Table 3] it is found that the performance of clustering is significantly influenced by LxL. We can also find that combining Loon and Layf yields better results than using either Lyon or L alone, which illustrates that each loss function plays crucial role in the overall performance of the model. We find that the clustering performance is not ideal when only Lay is used for training. The reason is that the latent features extracted by the lack of reconstruction constraints cannot accurately reflect the information of the original features.
5.4 Visualization of Feature Representation
We use 1-SNE 42
to visualize the learning process of embedded features. The non-linear dimen- sionality reduction technique is used for visualizing and clustering high-dimensional data, which has been widely used in machine learning and data analysis to visualize nigh-dimensional data

---

(a) epoch=50
(b) epoch=100 100
(c) epoch=300
(d) epoch=400
(e) epoch=100
(I) epoch=200
g epoch=400
(h) epoch=800
Figure 1: Visualization of the latent representations in training on Multi-COIL-10 and Multi-COli 20 in the first and second rows, respectively.
Figure 2: Clustering performance w.r.t. different parameter settings on Multi-COIL-10 and Multi- COIL-20 respectively.
in lower-dimensional spaces for easier understanding and interpretation. We show the embedded
features of Multi-COIL-10 and Multi-COIL-20 Fig. [ At the start the training stage, the embedded features are non-separable. As the training progresses, the clustering structures of the embedded features become more apparent while their centroids gradually separate. This visual demonstration highlights the effectiveness of our proposed model.
5.5 Parameter Sensitivity Analysis
SUMVC has primary hyperparameterrs, i.e., y and B. We the clustering performance with different settings to demonstrate the stability of SUMVC. Results are presented in Fig. 2 Additionally, higher values of y are required to encourage disentangling but result in a sedlity-ddsenttnlllmmee trade-off between the quality of reconstructions and its latent features disentanglement, / that is either too large or too small also leads reduced clustering performance due to the imbalance between reducing redundant information and obtaining consistent information. The recommended parameters for y and B are both in our experiments.
Conclusion
In this paper, propose simple consistent approach and informatiin--teorreii approach for multi-view clustering to address the challenge of obtaining useful information from multiple views while reducing redundancy and ensuring consistency among views. In particular, leverage the information bottleneck principle and variational inference to the multi-view scenarios. The proposed SCMVC and SUMVC are extensively evaluated multiple real- wall-world multi-view datasets. Extensive experiments demonstrate the clustering performance of the proposed models.

---

Acknowledgement
This work supported in part by Medico- Enniieeinggn Cooperation Funds from University of Elec- tronic Science and Technology of China (ZYGX2021 YGLH022), Lifang He is partially supported by the National Science Foundation grants (MRI-2215789, IIS-1909879, IIS-2319451) and Lehigh's grants under Accelerator and CORE.
References
[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. JMLR, 19:1947-1980, 2018.
[2] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. TPAMI, 40:2897-2905,, 2018.
[3] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 2016.
[4] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pages 92-100, 1998.
[5] TM Cover and JA Thomas. Joint entropy and conditional entropy. Elements of Information Theory, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, page 16, 2006.
[6] lain J Cruickshank. Multi-view clustering of social-based data. PhD diss., Carnegie Mellon University, 2020.
[7] Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applica- tions. TPAMI, 35(11):2765-2788 2013.
[8] Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transac- tions on Information Theory, 40:259-266, 1994.
[9] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In ICLR, 2020.
[10] Jacob Goldberger, Hayit Greenspan, and Shiri Gordon. Unsupervised image clustering using the information bottleneck method. In DAGM-Symposinee pages 158-165, 2002.
[11] Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63:139 1444,,444, 2020.
[12] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[13] Shudong Huang, Hongjie Wu, Yazhou Ren, Ivor Tsang, Zenglin Xu, Wentao Feng, and Jiancheng Lv. Multi-view subspace clustering on topological manifold. In NeurIPS, pages 25883-25894, 2022.
[14] Zhanpeng Huang, Jiekang Wu, Jinlin Wang, Yu Lin, and Xiaohua Chen. A unified multi-view clustering method based non-negative matrix factorization for cancer subtyping. IJDWM, 19:1-19, 2023.
[15] HycongJoo Hwang, Geon Ccon-Hyeong Kim, Seunghoon Hong, and Kee- Eung Kim. Multi-view representation learning via total correlation objective. In NeurIPS, pages 12194-12207, 2021.
[16] Senerath Mudalige Don Alexis Chinthaka Jayatilake and Gamage Upeksha Ganegoda. In- volvement of machine learning tools in healthcare decision making. Journal of Healthcare Engineering, 2021.
10

---

[17] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: an unsupervised and generative approach to clustering. In IJCAI, pages 1965-1972. 2017.
[18] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[19] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi Semi-supervised learning with deep generative models. In NeurIPS, pages 3581-3588 2014.
[20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Hafnner Gradient-bant based learning applied to document recognition. Proceedings of the IEEE, 86:2278-22 2324,, 1998.
[21] Chung-Kuei Lee and Tyng-Luh Liu. Guided co-training for multi-view spectral clustering, In ICIP, pages 4042-444400 4042-4046, 2016.
[22] Ruihuang Li, Changqing Zhang, Huazhu Fu, Xi Peng, Tianyi Zhou, and Qinghua Hu. Reciprocal multi-layer subspace learning for multi-view clustering. In ICCV, pages 8 8172-8180, 2019.
[23] Zhaoyang Li, Qianqian Wang, Zhiqiang Tao, Quanxue Gao, Zhaohua Yang, et al. Deep adversarial multi-view clustering network. In IJCAI, pages 2952-2958, 2019.
[24] Jialu Liu, Chi Wang, Jing Gao, and Jiawei Han. Multi-view clustering via joint nonnegative matrix factorization. In SDM, pages 252-260, 2013.
[25] Yunxiao Liu, Youfang Lin, QinFeng Xiao, Ganghui Hu, and Jing Wang. Self-adversarial varia- tional autoencoder with spectral residual for time series anomaly detection. Neurocomputing, 458:349-333 363, 2021.
[26] Can-Yi Lu, Hai Min, Zhong- MoongQQiu Zhao, Lin Zhu, De-Shuang Huang, and Shuicheng Yan. Robust and efficient subspace segmentation via least squares regression. In ECCV, pages 347-360. Springer, 2012.
[27] James MacQueen. Classification and analysis of multivariate observations. In BSMSP, pages 281-297, 1967.
[28] Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. Stacked convolutional auto-encoders for hierarchical feature extraction. In ICANN, pages 52-59, 2011.
[29] Sameer A Nene, Shree K Nayar, Hiroshi Murase, al. Columbia object image library (coil-20). Technical Report, CUCS-005-96 1996.
[30] Feiping Nie, Hua Wang, Heng Huang, and Chris Ding. Unsupervised and semi-supervised learning via / 1 --norm graph. In ICCV, pages 1268-2273. IEEE, 2011.
[31] Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. An information bottleneck approach for controlling conciseness in rationale extraction. In EMNLP, pages 1938-1952, 2020.
[32] Yazhou Ren, Shudong Huang, Peng Zhao, Minghao Han, and Zenglin Xu. Self-paced and auto-weighted multi-view clustering. Neurocomputing, 383:248-256, 2020.
[33] Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116:11537-1156,, 2019.
[34] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. Information Flow in Deep Neural Networks, page 24, 2022.
[35] Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingcheng Fu, Cheng Ji, and Philip S Yu. Graph structure learning with variational information bottleneck. In AAAI, pages 4165-4174, 2022.
[36] TISHBY. The information bottleneck method. Computing Research Repository (CoRR), 2000.
11

---

[37] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop, pages 1-5, 2015.
[38] Daniel Trosten, Sigurd Lokse, Robert Jenssen, and Michael Kampffmeyer. Reconsidering representation alignment for multi-view clustering. In CVPR, pages 1255-1265, 2021.
[39] Y-H Tsai, Y Wu, R Salakhutdinov, and L-P Morency. Self-supervised learning from a multi-view perspective. In ICLR, 2021.
[40] Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Russ R Salakhutdinov. Neural methods for point-wise dependency estimation. In Neur/PS, pages 62-72. 2020.
[41] Tim Van de Cruys, Thierry Poibeau, and Anna Korhonen. A tensor-based factorization model of semantic compositionality. Conference of the North American Chapter of the Association of Computational Linguistics (HTL-NAACL), pages 1142-1151, 2013.
[42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9:2579-2600 2008.
[43] Siwei Wang. Xinwang Liu, Suyuan Liu, Jiaqi Jin, Wenxuan Tu, Xinzhong Zhu, and En Zhu, Align then fusion: Generalized large-scale multi-view clustering with anchor matching corre- spondences. In NeurIPS, pages 5882-5895, 2022.
[44] Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurlPS, pages 12104-12115, 2021.
[45] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 170..077777 2017.
[46] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, pages 478-487, 2016.
[47] Jie Xu, Yazhou Ren, Guofeng Li, Lili Pan, Ce Zhu, and Zenglin Xu. Deep embedded multi-view clustering with collaborative training. Information Sciences, 573:279-200 2021.
[48] Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. Multi-vae: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In ICCV, pages 9234-9243, 2021.
[49] Jie Xu, Yazhou Ren, Huayi Tang, Zhimeng Yang. Lili Pan, Yang Yang, Xiaorong Pu, Philip S Yu, and Lifang He. Self-supervised discriminative feature learning for deep multi-view clustering. TKDE, pages 4470-7482, 2023.
[50] Yuanjin Xu and Ming Wei. Multi-view clustering toward aerial images by combining spectral analysis and local refinement. Future Generation Computer Systems, 1 117:138 114--44, 2021.
[51] Wenbiao Yan, Jihua Zhu, Yiyang Zhou, Yifei Wang, and Qinghai Zheng. Multi-view semantic consistency based information bottleneck for clustering. arXiv preprint arXiv:2303.00000,, 2023.
[52] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, pages 12310-12320, 2021.
[53] Zheng Zhang, Li Liu, Fumin Shen, Heng Tao Shen, and Ling Shao. Binary multi view clustering. TPAMI, 41:1774-1782, 2018.
[54] Handong Zhao, Zhengming Ding, and Yun Fu. Multi-viow clustering via deep matrix factoriza- tion. In AAAI, pages 2921-2927, 2017.
[55] Shuai Zheng, Xiao Cai, Chris Ding, Feiping Nie, and Heng Huang. closed form solution to multi-view low-rank regression. In AAAI, pages 1973-1979, 2015.
12

---

[56] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS, pages 321-328, 2003.
[57] Yiyang Zhou, Qinghai Zheng, Shunshun Bai, and Jihua Zhu. Semantically consistent multi-view representation learning. arXiv preprint arXiv:2303.043300 2023.
[58] Yiyang Zhou, Qinghai Zheng. Wenbiao Yan, Yifei Wang, and Jihua Zhu. Mcoco: Multi-level consistency collaborative multi-view clustering. arXiv preprint arXiv:2302. 13339, 2023.
13


Reviews:
[{"review_id": "H7hfmrOHYX", "reviewer": "Reviewer_79CP", "summary": "A consistent variational lower bound is provided to explore the consistent information among views for multi-view clustering, based on which SCMVC (simple consistent multi-view clustering) is proposed. To enhance consistent information and minimize unnecessary information among views, a sufficient representation lower bound is further proposed.", "questions": "Please see ‘Weaknesses’.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The main idea is novel, necessary preliminary knowledge and sufficient theoretical analysis is given.\n\n(2) Experiments conducted on real multi-view data demonstrate the good performance of proposed methods. The codes are available. \n\n(3) The findings of this study have the potential to contribute to the advancement of multi-view clustering techniques.", "weaknesses": "(1) It is said the proposed model does not perform well on datasets with strong heterogeneity between views, but no evidence or experiments support this statement. How does the heterogeneity affect the performance should be explained.\n\n(2) The paper shows quantitative results of the proposed SUMVC and SCMVC. However, I expect some visualization results to show the difference between these two methods.\n\n(3) A thorough analysis of potential drawbacks, and practical considerations would enhance the overall strength of the paper.\n\n(4) The tested data sets in this paper contain a small number (<=3) of views. It is suggested to add data sets with more than three views for discussion.\n\n(5) The writing should be further improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": " We thank the reviewer for the thoughtful comments and feedback. Below, we provide our responses to the key questions that were raised by the reviewer.\n\n1. It is said the proposed model does not perform well on datasets with strong heterogeneity between views, but no evidence or experiments support this statement. How does the heterogeneity affect the performance should be explained.\n\nThank you for your insightful comment. We have conducted additional experiments on the BDGP dataset  to substantiate our initial statement. BDGP contains 5 different types of drosophila. Each sample has visual and textual views. It has high heterogeneity between views. To mitigate the heterogeneity issue, we explored the use of a Multilayer Perceptron (MLP) to harmonize dimensions. We also tested the efficacy of sharing parameters between views. Both strategies have shown promise in alleviating the performance decrement in heterogeneous datasets. We  made a comparison between four models: (1) SCMVC-NS (not-share): the original SCMVC  without consideration of heterogeneity between views, (2) SCMVC-MLP: the improved  SCMVC  using a single layer MLP as a consideration of unifying dimensions and sharing parameters between views, (3) SUMVC-NS: the original SUMVC without consideration of heterogeneity between views, and (4) SUMVC-MLP: the improved SUMVC using a single layer MLP as a consideration of unifying dimensions and sharing parameters between views.\n\n | Method | SCMVC-NS | SCMVC-MLP | SUMVC-NS| SUMVC-MLP |SAMVC |FMVACC |\n| --- | --- | --- | --- | ---  | ---  | --- |\n| ACC | 49.9   |  55.8| 55.3 |71.4|51.3|58.6|\n| NMI| 44.8    |   48.6| 39.9 |60.3|45.2|36.8|\n| ARI |   29.6|   35.1| 27.3|45.3|19.6|44.3|\n\nThe experimental outcomes demonstrate that while our model does not reach the ideal level on the BDGP dataset with strong heterogeneity between views, it delivers a comparable performance to the existing models, i.e.,  (Fast multi-view anchor-correspondence clustering, NeurIPS 2022) and SAMVC (Self-paced and auto-weighted multi-view clustering, NC 2020).\nWe believe these additional experiments and proposed solutions provide a clearer understanding of our model's behavior under varying degrees of heterogeneity. We hope this addresses your concerns satisfactorily and provides a more comprehensive view of our work.\n\n \n2. The paper shows quantitative results of the proposed SUMVC and SCMVC. However, I expect some visualization results to show the difference between these two methods.\n\nThank you for your insightful suggestion. We agree that providing visual results could further illustrate the differences between the SUMVC and SCMVC methods, in addition to the quantitative results we have already provided. \nTo this end, we will provide additional visualizations to better elucidate the distinctions between the two methods. \n\n3.  A thorough analysis of potential drawbacks, and practical considerations would enhance the overall strength of the paper.\n\nThank you for your constructive feedback. We discussed potential drawbacks in the Appendix:\nour model does not perform well on datasets with particularly strong heterogeneity between views, such as huge differences in dimensions of different views.\nWe will add a new paragraph in the Conclusion section to address this issue. In this section, we critically analyze the limitations of our approach, discuss possible scenarios where our method may not perform optimally, and outline practical considerations for researchers and practitioners intending to adopt our methodology. Please see the details below:\n\nThe heterogeneity between views (i.e., the distributions of different views are obviously different) can make it more difficult for the VAE to learn a meaningful latent representation of the data. When the views are highly dissimilar such as views of BDGP, it may be challenging for the VAE to find a shared low-dimensional representation that captures the important features of both views. This can lead to suboptimal performance and poor reconstruction quality.  \n\nWe believe that this addition will provide the reader with a more comprehensive understanding of our work, and more importantly, it will stimulate further research to overcome the highlighted limitations.  \n \nFor practical considerations, multi-view clustering has broad prospects in the fields of data analysis and pattern recognition. The main idea behind multi-view clustering is to utilize multiple data sources or feature sets to perform clustering analysis, thereby improving the accuracy and robustness of clustering results.\nMulti-view clustering can be applied in various domains such as bioinformatics, social network analysis, image processing, and text mining. In bioinformatics, multi-view clustering can be used for the analysis of gene expression data, combining different experimental platforms and data types to discover more accurate gene expression patterns and biological features. In social network analysis, multi-view clustering can integrate users' social relationships, interest tags, and behavioral data to achieve more refined user segmentation and community discovery.\n\n\n4. The writing should be further improved.\n\nWe appreciate your feedback on our writing. We will review and revise our manuscript. We focus on improving the clarity, coherence, and conciseness of our writing, ensuring our arguments are well-structured, and that our language is precise. \n\n"}, {"review_id": "0fHqcFsiM8", "reviewer": "Reviewer_NC94", "summary": "This paper considers the problem of multi-view clustering from an information theoretic perspective. It focuses on representation learning, and optimizes said representation to improve down-stream clustering performance (with k-means). It introduces an Information Bottleneck based loss function, which considers consistency between views, redundancy and sufficiency of representations in addition to the traditional likelihood based reconstruction loss. It introduces 2 methods SCMVC and SUMVC based on some or all of these additional loss terms.\n\nThe paper then analyzes the model effectiveness by establishing a connection to the Bayes Error Rate, and also shows good experimental results across some multi-view datasets.", "questions": "Overall questions:\n\n- This paper is more about representation learning for clustering than clustering itself, right? Similar to spectral clustering. Not that that's a bad thing, it's just a little misleading. You spend a lot of the paper talking about MVC but the actual clustering is just k-means on top of a learned representation. It might help to be clear about this up front.\n- As I mentioned in the second point of the Limitations, it seems like this method is also weak to only partial information existing in views. How would you remedy this? Would ensemble variational methods work here as well?\n- For the MNIST experiments, I am confused about your choice to use pairs of the same digit (but written by different people) as two views. The data distribution of both the views are very similar, then. Also, if you pair up digits, shouldn't your dataset size be 35000 (halved) and not 70000? Or are you using both (A, B) and (B, A) for each pair? If so, this doesn't really seem like a good multi-view dataset since the views themselves are basically indistinguishable overall.\n- In table 3, the 3rd row seems to not have any useful information. It's clear that $L_{rec}$ would be required since you're using an auto-encoder. Instead, maybe you can have $L_{rec} + L_{suf}$ which you don't have here. That would be interesting to see.\n\n\nI will leave line-by-line comments here (since there doesn't seem to be a better place for this). There are a few typos here and there, but I won't bother too much with those:\n\n- [Line 32] Did you mean maximizing MI between representation* and output (not input and output)\n- [Line 82] \"... to quantify amount of\" -- sentence is not complete.\n- [Section 3] You use y, z without explaining/defining what they are. Clustering is an unsupervised task, so what does this mean in this context? I'm guessing it means cluster assignment but it isn't clear. \n- [Line 164] I may be wrong but isn't $\\phi^i$ the generational parameter set, and not $\\theta^i$?\n- [Line 180] What is a \"pseudo-label?\" A cluster assignment? You should define this earlier.\n- [Line 164] What do you mean by unique distribution?\n- [Line 186] This equation is very hard to parse. For cleaner appearance, you should consider using \\left( and \\right) to have larger parantheses. You could also remove the superscripts just for these long equations where there is no $j$, and just leave a note below. Also, shouldn't there be a conditioning on y in the first term? Lastly, $KL$ -> $D_{KL}$.\n- [Line 198] maximizing*\n- [Equation 9] There should be no expectation here, right? The $D_{KL}$ absorbs it, I think. Also, for the second term. the first distribution uses $z^j$ instead of $z^i$.\n- [Line 243] Should it be $z^j_m$ here? Also, what is $P^{\\otimes n}$?", "limitations": "- [Author mentioned] Heterogeneity in the data (eg. in terms of dimensions of features) affects performance significantly.\n- Restrictive assumptions are made on mutual redundancy between views. Multi-view data often has only partial information available in each view. I.e. you might need more than one view to get the complete picture. This also seems like a weakness of the methods.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Introduces a novel approach for representation learning for Mmulti-View Clustering (MVC) based on information theoretic criteria.\n- Proposes two separate methods, both of which show good experimental performance.\n- Has mathematical rigor in both breaking down the loss function, as well as in the theoretical analysis that follows.\n- Compares against multiple state-of-the-art MVC methods and shows superior performance on the chosen datasets.\n- Conducts a wide range of experiments, including ablation studies and parameter sensitivity analysis.", "weaknesses": "- The presentation of the mathematical parts of the preliminaries, discussion and analysis is quite opaque. Variables and terms are often explained only after they have already been used in equations. The equations tend to be quite cluttered and difficult to parse.\n- The datasets chosen are less than ideal, and are mostly derivative from single-view datasets. There are other multi-view datasets such as NUS-WIDE and 3-Source News which are potential candidates for publicly available MV datasets.\n- The assumptions (and thus applicability) of the methods are restrictive. I.e. assuming mutual redundancy across all views. As they mention in the supplementary, their methods perform poorly on heterogeneous MV data, limiting their applicability.\n- The paper was not self-contained; the supplementary material was essentially required to understand the a lot of the details. For eg. the limitations of the methods were only mentioned in the supplementary material. The supplementary material must not be used as additional pages, and the paper itself should be able to stand alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thoughtful comments and feedback. Below please find our detailed responses to the questions.\n\n1.\tThis paper is more about representation learning for clustering than clustering itself, right?\n\nWe appreciate your insightful observation. You're correct in noting that our paper primarily focuses on the aspect of representation learning for clustering, which is analogous to spectral clustering. The use of the k-means method on top of the learned representation is indeed a subsequent step, and the Multi-View Representation Learning (MVRL) method forms the core of our discussion.\nIn response to your feedback, we will revise our manuscript to make this point more explicit upfront. We believe this adjustment better frames the main contributions of our work and minimizes any potential misunderstandings about the paper's focus.\nThank you again for pointing out this nuance. \n\n2. As I mentioned in the second point of the Limitations, it seems like this method is also weak to only partial information existing in views. How would you remedy this? Would ensemble variational methods work here as well?\n\nThank you for your insightful comments. We appreciate your mention of the issue of partial information existing in views, which indeed is a challenge that our method currently faces. The original intention of MVC is that a single view only contains partial information, and it is difficult to depict the complete clustering structure. MVC combines complementary information from multiple views to achieve better clustering results. If \"partial information\" refers to missing data, one possible strategy could involve integrating additional data preprocessing steps, such as imputation methods for handling missing data, which could potentially enhance the robustness of our method when dealing with partial views.\n\nRegarding your suggestion of using ensemble variational methods, we believe it's a very promising direction. Ensemble methods could indeed provide a solution to this issue by leveraging the consensus among multiple models, each trained on a different subset of the data. This approach could help to address the inherent uncertainty and variability in the data, and thus potentially improve the performance of our method when dealing with partial views.\nWe will follow your suggestion to add a discussion about these potential strategies to the manuscript. We believe that this addition will stimulate further research on this topic and provide a roadmap for improving the current limitations of our method.\nWe hope this adequately addresses your concerns. We greatly value your suggestions and look forward to any further feedback you may have.\n\n3. For the MNIST experiments, I am confused about your choice to use pairs of the same digit (but written by different people) as two views. The data distribution of both the views are very similar, then. Also, if you pair up digits, shouldn't your dataset size be 35000 (halved) and not 70000? Or are you using both (A, B) and (B, A) for each pair? If so, this doesn't really seem like a good multi-view dataset since the views themselves are basically indistinguishable overall.\n\nThank you for your thoughtful feedback, which will undoubtedly improve the quality of our paper.  The MNIST-MV data we used in the experiments is a public available dataset and has been widely used in previous multi-view studies, e.g., \"Deep safe multi-view clustering: Reducing the risk of clustering performance degradation caused by view increase,  CVPR 2022\", \"Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clustering, ICCV 2021\", and \"Multi-view Semantic Consistency based Information Bottleneck for Clustering, arxiv 2023\".\n\nRegarding our choice to use pairs of the same digit written by different people as two views, the rationale behind this decision was to examine the ability of our model to identify and learn from subtle differences in similar-looking data. While the two views may seem indistinguishable at a macro level, they can contain minute differences at a micro level. These differences are due to the unique handwriting styles of different individuals, which our model aims to capture and learn from.\n\nAs for your question about the dataset size, you are correct in your understanding. We indeed used both (A, B) and (B, A') pairs (A' may not be equal to A), effectively maintaining the original dataset size of 70,000. The reason behind this approach was to increase the diversity of our training data and further test the robustness of our model. However, we understand your concern about the potential impact on the multi-view nature of the dataset.\n\nIn light of your comments, we will ensure that our choices are properly justified in the revised manuscript.\n\n\n4. In table 3, the 3rd row seems to not have any useful information. It's clear that $L_{rec}$ would be required since you're using an auto-encoder. Instead, maybe you can have  $L_{rec} +L_{suf} $ which you don't have here. That would be interesting to see.\n\n\nThank you for your insightful comments.  We will remove the 3rd row to avoid any confusion and the results of the ablation experiment with the inclusion of $L_{rec} + L_{suf}$ can be found in the table below. \n\n|Dataset | Multi-MNIST | Multi-Fasion|Multi-COIL-20 |  \n|---|---|---|---|\n| ACC|  98.4| 84.6 |86.9| \n| NMI    |  96.0| 80.8 |91.0| \n| ARI |   96.6| 75.2 |83.1| \n\nWe found that model with only the $L_{rec} + L_{suf}$ terms  perform worse than SUMVC. This is because $L_{suf}$ helps the model learn the distributional features of the latent layer. Therefore, the lack of this constraint makes it challenging for the model to effectively learn these features.   We hope these adjustments and explanations address your concerns satisfactorily.\n"}, {"review_id": "NZwvogfijO", "reviewer": "Reviewer_ugfB", "summary": "This work introduces a new approach called sufficient multi-view clustering (SUMVC) to improve clustering performance using multiple data sources. Existing methods often focus on acquiring consistent information while neglecting the issue of redundancy across multiple views. By contrast, the proposed SUMVC provides a promising solution to the problem of multi-view clustering and offers a new perspective for analyzing multi-view data. The effectiveness of the model is verified through theoretical analysis and experiments on multiple multi-view datasets, showing superior performance compared to other methods.", "questions": "The paper mentions that the superiority of SUMVC is demonstrated through experiments on multiple multi-view datasets. However, there is no detailed discussion or analysis provided on the characteristics of these datasets or how they were selected. What are the criteria for dataset selection, and how representative are these datasets?", "limitations": "No potential negative societal impact of this work exists.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By examining the multi-view clustering framework from an information-theoretic standpoint, SUMVC offers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data. \n2. The effectiveness of the proposed model is demonstrated through theoretical analysis based on the Bayes Error Rate and experiments on multiple multi-view datasets, highlighting its superior performance compared to other methods.\n", "weaknesses": "1. This paper is lack of in-depth discussion or evaluation of the proposed approach's limitations. \n2. The authors could provide more insights into the computational complexity of the proposed method and its scalability to large-scale multi-view datasets. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the review for the constructive comments and feedback. Below we provide our responses to the key questions made by the reviewer.\n\n1. The paper mentions that the superiority of SUMVC is demonstrated through experiments on multiple multi-view datasets. However, there is no detailed discussion or analysis provided on the characteristics of these datasets or how they were selected. What are the criteria for dataset selection, and how representative are these datasets?\n\nThank you for your insightful comments. We appreciate your interest in the dataset selection process for our experiments.\nThe datasets used in our experiments, including Multi-COIL-10 (K = 10), Multi-COIL-20 (K = 20), Multi-MNIST, and Multi-Fashion, were chosen based on several criteria. These criteria include the presence of multi-view data, variety in the type of data (ranging from object images in different poses to handwritten digits by different individuals), and their public availability and widespread use in the research community.\nMore specifically:\n\n•\tIn Multi-COIL-10 and Multi-COIL-20, different views of an object correspond to various poses, but retain the same label. This allows for an assessment of how well the SUMVC can handle variations in perspective.\n\n•\tIn Multi-MNIST, different views of a digit represent the same digit written by different individuals, testing the SUMVC's ability to recognize the same object despite stylistic differences.\n\n•\tIn Multi-Fashion, different views of a product category signify different fashionable designs for that category, challenging the SUMVC to identify similar categories despite variations in design.\n\nThese datasets have been extensively used in the field, which allows for a reliable comparison of our method with existing ones. Moreover, the diversity of these datasets ensures a robust evaluation of our method across different types of multi-view data. These Multi-view datasets are commonly used for evaluating MVC methods and widely applied in MVC research papers, such as “MCoCo: Multi-level Consistency Collaborative Multi-view Clustering”, “Contrastive multi-view hyperbolic hierarchical clustering” and “Deep safe incomplete multi-view clustering: Theorem and algorithm”. \n\n2. This paper is lack of in-depth discussion or evaluation of the proposed approach's limitations.\n\nThank you for your constructive feedback. We will add  a new paragraph in the Conclusion section to address this issue. In this section, we critically analyze the limitations of our approach, discuss possible scenarios where our method may not perform optimally, and outline practical considerations for researchers and practitioners intending to adopt our methodology. Please see the details below:\nThe heterogeneity can make it more difficult for the VAE to learn a meaningful latent representation of the data. When the views are highly dissimilar such as views of BDGP, it may be challenging for the VAE to find a shared low-dimensional representation that captures the important features of both views. This can lead to suboptimal performance and poor reconstruction quality.\n\n\n3. The authors could provide more insights into the computational complexity of the proposed method and its scalability to large-scale multi-view datasets.\n\nThank you for your insightful suggestion. We will include a new section in our manuscript entitled \"Computational Complexity Analysis.\" In this section, we discuss the algorithmic complexity of our method, outline the key factors that influence its computational cost, and analyze its scalability with respect to the size of the dataset and the number of views. Please see the details below:\nAssuming $T_1$ represents the number of iterations for training SCMVC,  $T_2$ represents the epochs training SUMVC based on SCMVC, $l$ represents the dimensionality of the embedding for each view, $n$ is the number of instances and $V$ is the number of views. Then the whole training process requires $O(V n T_1) $ to train the variational autoencoders via $L_{con}$, $O(V(V-1) nlT_2) $ to calculate the $L_{suf}$. \nOverall, the time complexity of the model is $O(Vn((V-1)l T_2+ T_1 ))$. Similar to commonly used deep MVC methods, the computational complexity of our approach is also linear to the data size, making it easier to apply for large-scale MVC data clustering.\n\n"}, {"review_id": "AFMMII0QwJ", "reviewer": "Reviewer_89XX", "summary": "The paper proposed Sufficient Multi-View Clustering , SUMVC, which is composed of two main components. The first component is a simple and reliable multi-view clustering method called SCMVC (simple consistent multi-view clustering), which utilizes variational analysis to generate consistent information. The second component proposes a lower bound on sufficient representation, aiming to enhance consistent information and reduce unnecessary information among views.", "questions": "see above.", "limitations": "It can be found in Appendix of this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Analyze the effectiveness of multi-view clustering from the information-theoretic perspective is interesting and valuable.\n- The proposed method outperforms the state of the arts. The paper has good reproducibility with the provided codes.\n- The paper is very well written. The theoretical and empirical analyzes are convincing.", "weaknesses": "- There are few comparative methods, and more comparative methods need to be selected reasonably to illustrate the effectiveness of the proposed methods.\n- The theoretical analysis (based on the Bayes Error Rate) should be elaborated more. The key findings or insights from this analysis should also be emphasized.\n- Did the experimental results demonstrate any limitations or potential challenges of the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. There are few comparative methods, and more comparative methods need to be selected reasonably to illustrate the effectiveness of the proposed methods.\nWe appreciate your suggestion. We have conducted additional comparisons with several other relevant methods, i.e., FMR\n(Flexible multi-view representation learning for subspace\nclustering, IJCAI 2019), LMVSC (Large-scale\nmulti-view subspace clustering in linear time, AAAI 2020) and CSMSC (Consistent and specific multi-view subspace clustering, AAAI 2018). These methods are chosen based on their relevance, popularity, and the availability of implementation details, which ensure a fair comparison.               We demonstrate the results using the multi-coil-10 dataset as an example. Again, our method outperforms all these methods.\n \n| Method | FMR | LMVSC|CSMSC| SCMVC |SUMVC|\n| --- | --- | --- | --- | ---  | ---  |\n| ACC | 78.1 |  63.8| 97.6   | 98.1|100.0|\n| NMI| 80.0 |   75.8| 96.2  | 96.7|100.0|\n| ARI |   70.6|   55.1| 94.9 | 95.8|100.0|\n\n2. The theoretical analysis (based on the Bayes Error Rate) should be elaborated more. The key findings or insights from this analysis should also be emphasized.\n\nThank you for your constructive feedback. The Bayes Error Rate, defined as the probability of misclassifying a data point when the true underlying distribution of the data is known, serves as a pivotal indicator of the performance of learning algorithms. It offers valuable insights into the effectiveness of the feature extraction conducted by our model. In particular, the Bayes Error Rate is an important index in understanding the overall performance of our model and reflects the impact of representations on the performance of downstream tasks.\n\nIn response to this comment, we will expand our discussion of the Bayes Error Rate and its implications in the final version. We delve deeper into its theoretical underpinnings and provide a clearer connection between a lower Bayes Error Rate and the efficacy of the features extracted by our model for downstream tasks.\n \n\n3. Did the experimental results demonstrate any limitations or potential challenges of the proposed method?\n\nThank you for your insightful question. We discussed limitations in the Appendix.\nOne limitation we found is  the proposed model exhibits poor performance when applied to datasets that have significant heterogeneity across different perspectives. This could potentially impact the robustness and generalization of the model. \n \n"}, {"review_id": "DtgeWbRZez", "reviewer": "Reviewer_XaYq", "summary": "This paper proposes two methods for multi-view clustering, which aims at grouping data from multiple sources or perspectives. The first part, SCMVC uses a consistent variational lower bound to learn consistent information among views. The second part, SUMVC extends the information bottleneck principle to reduce redundancy and achieve sufficient representation among views. The proposed SUMVC consists of two terms: a consistent variational lower bound and a sufficient representation lower bound to enhance consistency and minimize redundancy among views. The authors leverage information bottleneck theory and variational analysis to develop the model. The paper also provides a theoretical analysis of the generalization error of the learned representations based on the Bayes error rate. The paper evaluates the proposed methods on four real-world multi-view datasets.", "questions": "The authors mention that the methods discussed in this article cannot be effectively applied to datasets with high heterogeneity. Could authors provide a detailed analysis?", "limitations": "Please see weakness. Also, the complexity analysis is missing.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper addresses an important and challenging problem of MVC with an information-theoretic perspective. This paper proposes novel lower bounds to address the issue of view redundancy and consistency. The authors utilize Bayes Error Rate to provide a theoretical explanation of the effectiveness of the proposed method. \n3.\tThe paper presents extensive experiments on four real-world multi-view datasets and demonstrates the superior performance of the proposed methods over existing methods.\n", "weaknesses": "1.\tThe proposed objective function in Eq. (4) shares similarities with that of VAE [1], and it would be beneficial if the author could provide further explanations on this matter. How does this loss function differ from that of VAE? Is it merely an extension of the single-view approach to multi-view data? Additionally, what is the reason for SCMVC achieving significantly better performance than VAE-based methods in Table 2?\n2.\tIn SCMVC, the fusion representation $\\vec{Z}$ is directly optimized, while it is not in SUMVC. Could this approach negatively impact the performance of SUMVC?\n3.\tThe methods discussed in this article may not be effectively applied to datasets with high heterogeneity. \n4.\tThe introduction section contains some complex sentences. Simplifying the language and structure could improve clarity.\n5.\tIt would be beneficial for the authors to provide a more comprehensive discussion of some missed information-based multi-view methods [2-5] in the related work section.\n\n[1] Auto-Encoding Variational Bayes, NIPS’13\n\n[2] COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction, CVPR'21\n\n[3] Rethinking Minimal Sufficient Representation in Contrastive Learning, CVPR’ 22\n\n[4] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'23\n\n[5] Multi-view information-bottleneck representation learning, AAAI’21\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your invaluable comments and suggestions. We have addressed the points you raised as follows.\n\n1. The proposed objective function in Eq. (4) shares similarities with that of VAE [1], and it would be beneficial if the author could provide further explanations on this matter. How does this loss function differ from that of VAE? Is it merely an extension of the single-view approach to multi-view data? Additionally, what is the reason for SCMVC achieving significantly better performance than VAE-based methods in Table 2?\n\nThank you for your thoughtful comments. You are correct in noticing the similarity between the proposed objective function in our paper and that of the VAE. The objective function in our method is indeed a variant of the VAE loss function. However, it is specifically designed to handle multi-view data, while the VAE loss function is typically used for single-view data. This introduces significant differences in terms of functionality and application between the two. \nOur method is not merely an extension of the single-view approach to multi-view data. It takes into account the unique characteristics of multi-view data, such as inter-view correlations and view-specific features, which are not considered in standard VAE-based methods. This ability to exploit multi-view information is a key factor distinguishing our loss function from that of VAE.\nIn terms of the superior performance of SUMVC over VAE-based method (i.e., \\beta-VAE) as presented in Table 2, it can be attributed to several factors. Firstly, the SUMVC model is designed to capture both shared and view-specific representations, while VAE-based model only generates a single shared representation. This additional flexibility in SUMVC allows for better handling of multi-view data and contributes to its enhanced performance.\n\n2. In SCMVC, the fusion representation $\\overrightarrow{Z}$ is directly optimized, while it is not in SUMVC. Could this approach negatively impact the performance of SUMVC?\n\nThank you for raising this valuable point. In the proposed SUMVC method, the fusion representation is indeed not directly optimized as it is in SCMVC. However, this approach is by design and serves a specific purpose in the context of our research.\nIn SUMVC, the goal is to utilize the complementary information from different views without explicitly enforcing a shared representation. This allows SUMVC to maintain the distinctiveness of each view, which can be beneficial in cases where there is considerable heterogeneity across views. \nOn the other hand, SCMVC directly optimizes the fusion representation to encourage greater interaction and integration between views. This approach is particularly suitable when there is a high level of consistency or overlap among the views.\nThus, while the direct optimization of $\\overrightarrow{Z}$ in SUMVC could theoretically enhance performance in some cases, it could also compromise the ability of the model to handle datasets where preserving the distinctiveness of each view is crucial. Therefore, the decision to not directly optimize $\\overrightarrow{Z}$ in SUMVC was an intentional design choice made with these considerations in mind.\nWe appreciate your attention to these details and your insightful question. We hope this explanation provides a clearer understanding of the design and intended applications of SUMVC and SCMVC.\n\n3. The methods discussed in this article may not be effectively applied to datasets with high heterogeneity.\n\nWe discussed in the Appendix that the improvement in performance is not very significant when heterogeneity is high compared to single-view clustering. For experimental results, please refer to the response to Reviewer 79CP. \nWe believe that incorporating these additions will offer a more thorough understanding of the applicability and limitations of our methods. We value your critical feedback and are confident that it will contribute to strengthening our paper.\n\n4. The introduction section contains some complex sentences. Simplifying the language and structure could improve clarity.\n\nThank you for your valuable feedback. We will revise the introduction section and worked on simplifying the language and sentence structure. We will strive to break down complex sentences into more manageable parts and to convey our ideas in a straightforward manner without compromising the depth of our research.\n\n\n5. It would be beneficial for the authors to provide a more comprehensive discussion of some missed information-based multi-view methods [2-5] in the related work section.\n\nThank you for your thoughtful suggestion. We will these papers and added an enhanced review of these methods in the related work section of our manuscript. We will  discussed their methodologies, findings, and how they relate to and contrast with our own work  in the final version. The discussion of the methods mentioned by the reviewer is as follows:\n\n“Multi-view information-bottleneck representation learning” aims to develop a model that effectively explores the common latent structure and view-specific intrinsic information in multi-view data while discarding irrelevant information to enhance generalization capability.\n\n “Completer: Incomplete multi-view clustering via contrastive prediction” addresses two challenging problems in incomplete multi-view clustering analysis: learning an informative and consistent representation among different views without labels, and recovering missing views from the data. “Dual contrastive prediction for incomplete multi-view representation learning” provides a new perspective on the relationship between cross-view consistency learning and data recovery and propose a method that jointly addresses these challenges in multi-view representation learning.\n\n\n\n"}]

Author rebuttal:

Meta-review: This well-written paper has been reviewed by five knowledgeable reviewers who all recommended its acceptance (three straight accepts and two weak accepts). The authors have engaged the reviewers in productive discussions which addressed the key concerns raised by the reviewers. The presented work involves multi-view clustering and it will be of interest to the NeurIPS audience.
Decision: Accept (poster)