Title: A Novel Approach for Effective Multi-View Clustering with Information-Theoretic Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-view learning， clustering
Abstract: Multi-view clustering (MVC) is a popular technique for improving clustering performance using various data sources. However, existing methods primarily focus on acquiring consistent information while often neglecting the issue of redundancy across multiple views.
This study presents a new approach called Sufficient Multi-View Clustering (SUMVC) that examines the multi-view clustering framework from an information-theoretic standpoint. Our proposed method consists of two parts. Firstly, we develop a simple and reliable multi-view clustering method SCMVC (simple consistent multi-view clustering) that employs variational analysis to generate consistent information. Secondly, we propose a sufficient representation lower bound to enhance consistent information and minimise unnecessary information among views. The proposed SUMVC method offers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data. 
 To verify the effectiveness of our model, we conducted a theoretical analysis based on the Bayes Error Rate, and experiments on multiple multi-view datasets demonstrate the superior performance of SUMVC.

A Novel Approach for E ff ective Multi-View Clustering with Information-Theoretic Perspective

Abstract

Multi-view clustering (MVC) is a popular technique for improving clustering performance using various data sources. However, existing methods primarily focus on acquiring consistent information while often neglecting the issue of redundancy across multiple views. This study presents a new approach called su ffi cient multiview clustering (SUMVC) that examines the multi-view clustering framework from an information-theoretic standpoint. Our proposed method consists of two parts. Firstly, we develop a simple and reliable multi-view clustering method SCMVC (simple consistent multi-view clustering) that employs variational analysis to generate consistent information. Secondly, we propose a su ffi cient representation lower bound to enhance consistent information and minimise unnecessary information among views. The proposed SUMVC method o ff ers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data. To verify the e ff ectiveness of our model, we conducted a theoretical analysis based on the Bayes Error Rate, and experiments on multiple multi-view datasets demonstrate the superior performance of SUMVC.

1 Introduction

Clustering is a fundamental unsupervised learning task with broad applications across various fields. Traditional clustering methods primarily focus on analyzing single-view data. However, the rapid progress in multimedia technology has led to the collection of real-world data from multiple sources and perspectives, resulting in the emergence of multi-view data. Recently, multi-view clustering has garnered significant attention due to its ability to leverage information from multiple views to improve clustering performance and robustness [24]. Multi-view data arise in many applications where multiple sources of information or perspectives on the same set of objects are available. Multi-view clustering aims to group these objects into clusters by considering all available views jointly. This technique has proven to be highly e ff ective in various domains, including social network analysis [6], image and video analysis [50], and bioinformatics [14]. Nevertheless, the main challenge in multi-view clustering lies in extracting valuable information from the views for clustering purposes while simultaneously minimizing view redundancy.

Information bottleneck (IB) theory provides a theoretical framework for understanding the optimal trade-o ff between accuracy and compression in data processing [36]. The IB method suggests that

the optimal representation of the data can be achieved by maximizing the mutual information between the input and the output while minimizing the mutual information between the input and the representation. Although this principle provides a theoretic method to learn less superfluous information for downstream tasks, existing methods can still be improved in e ffi ciently extracting embedded features from multi-view unlabeled data. In recent years, many information-theoretic methods have been applied to multi-view scenarios. Specifically, [44] proposes a generic pretext-aware residual relaxation technique to enhance multi-view representation learning (MVRL). [15] propose a novel approach to MVRL through an information-theoretic framework grounded in total correlation. Meanwhile, variational analysis has gained wide usage in machine learning and optimization for approximating complex probability distributions [3]. This technique o ff ers a robust framework for solving optimization problems involving probability distributions, which are frequently encountered in multi-view clustering and information bottleneck. By approximating the probability distributions, variational analysis can e ff ectively reduce the data dimensionality and enhance the e ffi ciency of clustering algorithms. Variational autoencoders (V AEs) are a popular type of generative models that uses variational inference to learn compact representations of high-dimensional data [18]. However, the vast majority of them are designed for single-view data and cannot capture consistent and complementary information in a multi-view setting.

To tackle the aforementioned concerns, we delve deeper into the principles of the information bottleneck and variational analysis in the realm of multi-view clustering. We propose a simple consistent multi-view clustering method SCMVC and a novel multi-view self-supervised method SUMVC from an information-theoretic standpoint. SUMVC could strengthen consistency across views for clustering while concurrently decreasing view redundancy.

Our contributions are three-fold:

We propose a consistent variational lower bound to explore the consistent information among views for multi-view clustering. We introduce SCMVC, a simple consistent multi-view clustering approach, utilizing the proposed lower bound. Despite being a simple approach, our experiments demonstrate the e ff ectiveness of SCMVC.

We extend the information bottleneck principle to the multi-view self-supervised learning and propose a su ffi cient representation lower bound. We expand SCMVC to improve the consistency and reduce the redundancy. This mainly aims to acquire predictive information and reduce redundancy among views. We refer to this new model as SUMVC (su ffi cient multi-view clustering).

We adopt the collaborative training strategy for SUMVC to better adapt to multi-view scenarios. Additionally, a theoretical analysis of the generalization error of representations is conducted to evaluate the e ffi cacy of SUMVC for clustering tasks. Experiments on real-world multi-view data show the superior performance of the proposed model.

2 Related Work

Information Bottleneck The information bottleneck (IB) principle is a widely used framework in information theory and machine learning that provides a rational way to understand the balance between compression and prediction. Introduced by [36], the IB principle formalises the intuition that successful learning algorithms need to find a balance between retaining relevant information and discarding irrelevant information. Since then, the IB principle has been widely applied to various machine learning tasks, including unsupervised learning [37], classification [3], and clustering [2, 51]. Recently, the IB principle has also been used to study the information processing capacity of the brain [33, 34]. By constraining the learned representations of deep neural networks, the IB principle was shown to improve the generalization performance of these models and reduce their computational complexity [2]. In addition, the IB principle has been used to develop new techniques for training deep neural networks, such as the deep variational information bottleneck [2]. The theoretical foundation of the IB principle is built on concepts such as Markov chains, entropy, and conditional entropy [5]. It has been widely applied in various fields, such as data mining, image processing, natural language processing, and computer vision [10, 16, 35]. The IB principle has also been applied in control theory to quantify the amount of [31]. Overall, the IB principle has become an important tool for designing more e ffi cient and e ff ective machine learning algorithms. Its widespread applications in various fields highlight its relevance and practicality.

Varitional Autoencoder Variational autoencoders (VAEs) are a popular generative model class that learns a compact representation of input data through an encoder-decoder architecture. Unlike traditional autoencoders, VAEs are designed to model the latent space distribution of input data, enabling them to generate new data samples consistent with the input distribution. Early work on VAEs by [18] introduced the VAE model and derived its objective function, while [19] proposed a method for approximating gradients by sampling the latent variables. Since then, VAEs have been applied to various applications, including image generation, data compression, and anomaly detection. For instance, [11] combined VAEs with generative adversarial networks (GANs) to generate highquality images, while [25] applied V AEs to detect anomalies in seasonal key performance indicators (KPIs). Moreover, VAEs have been combined with other deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to improve their performance on various tasks. For example, [28] combined VAEs with CNNs to extract hierarchical features from images. It is worth mentioning the characteristic of variational autoencoders is incorporated into our model. In summary, VAEs are powerful for generative modeling and have contributed significantly to the field of deep learning. Deep learning with nonparametric clustering is a pioneer work in applying deep belief network to deep clustering. But in deep clustering based on the probabilistic graphical model, more research comes from the application of variational autoencoder (V AE), which combines variational inference and deep autoencoder together.

Multi-View Clustering In practical clustering tasks, the input data usually have multiple views. Multi-view clustering (MVC) methods are proposed to address the limitations of traditional clustering methods by leveraging the complementary information provided by di ff erent views of the same instance. Numerous algorithms and techniques have been proposed for MVC, ranging from classic methods such as spectral clustering to more recent deep learning and probabilistic modeling developments. One popular approach is co-training, which was first introduced by [4]. Co-training involves training multiple models, each using a di ff erent view of the data, and then using the models to label additional unlabeled data. This approach has been successfully applied in various domains, including text classification [56], image recognition [21], and community detection [24]. Another popular approach is low-rank matrix factorization, which aims to learn a shared latent representation of the data across multiple views. This approach has been explored in various forms, such as structured low-rank matrix factorization [55], tensor-based factorization [41], and deep matrix factorization [54]. Subspace methods have been utilized in MVC in the previous few years, e.g., [7, 13, 26, 30] integrated multiple a ffi nity graphs into a consensus one with the topological relevance considered.

In recent years, deep learning has emerged as a powerful tool for MVC, with various deep neural network architectures proposed for the task. For example, [23] proposed a deep adversarial MVC method, which learns a joint representation of the data across multiple views using adversarial training. [46] proposed deep embedded clustering (DEC) to learn a mapping from the high original feature space to a lower-dimensional one in which a pratical objective is optimized. [48] proposed a framework of multi-level feature learning for contrastive multi-view clustering (MFLVC), which combines MVC with contrastive learning to improve clustering e ff ectiveness.

3 The Proposed Method

Self-supervised learning methods for MVC [9, 52, 57, 58] aim at reducing redundant information and achieving su ffi cient representation from multiple views. In the multi-view scenario, there are also many methods [38, 49] that explore consistent information between views. Despite great e ff orts, the task of minimizing redundancy and extracting valuable information for subsequent processes while maintaining coherence across perspectives continues to pose significant challenges. In this section, inspired by [9, 39], we extend the principle of the information bottleneck, utilize the reparameterization trick for solving the proposed lower bound and introduce the collaborative training strategy to the MVC scenario. Moreover, we conduct bayes error rates for learned representations to validate the generalization performance of our method.

Problem Statement Given the multi-view data, X , which consists of v subsets such that X i = [ x i 1 ; . . . ; x i n ] ∈ R n × di is the n × di -dimensional data of the i -th view, where n is the number of instances and di is the number of dimensions of the i -th view. MVC aims to partition the n instances into K clusters.

3.1 Preliminaries

Analysis of redundancy According to [1, 9], we can assume that z contains all necessary information for predicting y without having prior knowledge of y . This can be achieved by maintaining all the shared information between the two views x i and x j . This assumption is grounded on the belief that both views provide similar predictive information. The following formal definition elucidates this concept (See details in Appendix B.2.).

Definition 3.1 [9] x i is considered redundant with respect to x j for y if and only if the mutual information I ( y ; x i | x j ) = 0 .

Intuitively, x i is considered redundant for a task if it is irrelevant for the prediction of y when x j is already observed. Whenever x i and x j are mutually redundant with respect to one another ( x i is redundant with respect to x j for y ), we have the following:

Corollary 3.1 Let x i and x j be two mutually redundant views for a target y and let z i be a representation of x i . If z i is su ffi cient for x j ( I ( x i ; x j | z i ) = 0) then z i is as predictive for y as the combination of the two views ( I ( x i x j ; y ) = I ( y ; z i )) .

Variational bound The variational bound [18], also known as the Evidence Lower Bound (ELBO), is a widely used optimization objective in Bayesian inference and machine learning. It provides a lower bound on the marginal likelihood of the observed data. Variational bound provides an e ffi cient approximate posterior inference of the latent variable z i given an observed value x i , which can be formulated as:

θ

where θ i and ϕ i are generative and variational parameters of the i -th view respectively. Through optimizing the lower bound L ( ϕ i , θ i ; x i ), we can approximate the posterior inference of the latent variable z i accurately based on an observed value of x i . As such, we can obtain more representative embedded features.

3.2 SCMVC: Learning Consistent Information via Varational Lower Bound

Variational inference has greatly contributed to representation learning. Introducing variational inference allows representation learning to be performed in a probabilistic manner thereby providing a principled framework for capturing uncertainty in the learned representation. Variational inference can optimize the generative model, resulting in a more informative data representation. In this section, we analyze the consistent information among views from the vartional inference perspective and propose SCMVC (simple consistent multi-view clustering). We introduce a consistent varitional lower bound to model the latent space distribution of input data more consistent with all views. Given v -views and global embedded features ⃗ z = { z 1 , z 2 , . . . , z v } , we formalize this concept as the following theorem (Proofs are given in Appendix B.3.):

Theorem 3.1 (Consistent Variational Lower Bound) Define the generative model q ϕ ( x i | ⃗ z , y i ) , and a posterior as p θ ( ⃗ z , y i | x i ) = p θ ( y i | ⃗ z ) p θ ( ⃗ z | x i ) . The ELBO for multi-view model can be formulated as

where the y i is the pseudo-label of x i , ⃗ z = { z 1 , z 2 . . . , z v } , the first term is to maximize the reconstruction probability of input x i , i.e., maximize the decoding probability from latent variable ⃗ z to x i . The constraint term is the KL divergence of the approximate from the true posterior to ensure that the distribution of acquired features and labels is consistent with their intractable true posterior. It can be intuitively understood that our goal is to obtain the unique distribution of the embedded feature ⃗ z of x for di ff erent categories.

For better understanding, Eq. (2) can be expanded out as:

Overall, the objective function of SCMVC is:

where L i rec is the reconstruction term and L i KL is the constraint term which approximates the true posterior.

3.3 SUMVC: Learning Su ffi cient Representation via Reducing Redundancy

Although SCMVC ensures consistency among views, there is a lack of consideration for reducing mutual redundancy and achieving su ffi ciency among views, which is important to obtain latent representations suitable for downstream tasks. Next, we advance our work and extend the information bottleneck principle to the multi-view setting. By decomposing the mutual information between x i and z i (See Appendix B.1.), we can identify two components:

where the first term denotes superfluous information and the second term denotes signal-relevant information. We take view j as self-supervised signals to guide the learning of view i . To ensure that the representation is su ffi cient for the task, we maximize the mutual information between x j and z i . As for I θ i ( z i , x i | x j ), this term indicates the information in z i that is exclusive to x i and cannot be predicted by observing x j . As we assume mutual redundancy between the two views, this information is deemed irrelevant to the task and could be safely discarded [9]. Overall, to extract su ffi cient representation of z i , we maximize the following formulation:

which can be optimized via maiximazing the lower bound in Theorem 3.2.

Theorem 3.2 (Su ffi cient Representation Lower Bound) Considering a multi-view model, the generative model is q ϕ i ( x i | z i , y i ) . The su ffi cient representation lower bound can be defined as:

Proof 1 Considering z i and z j on the same domain Z , we have

For I θ i ( x j , z i ) , it can be lower bounded as:

θ

θ

where I θ i θ j ( z i ; x j | z j ) = 0 when z j is su ffi cient for z i . This occurs whenever z j contains all the necessary information about z i .

Therefore, Eq. (5) can be optimized via maximizing the lower bound:

In addition, we adopt collaborative training to allow mutual supervision to better adapt to multi-view scenes. The views are selected in turn to act as the supervisory view, while the remaining views serve as the supervised views. This learning approach could e ffi ciently use the common information and complementary information of multiple views.

SUMVC is derived by incorporating a lower bound on su ffi cient representation into SCMVC, the objective of which can be defined as follows:

where the i -th view can be regarded as the supervisory view, while the j -th view is the supervised view. Through the collaborative training we could reduce redundancy and obtain consistent information among views. What's more, we align the embedded features and category distribution with the original distribution to obtain information conducive to clustering. Finally, in order to better utilize global features, we run K -means on ⃗ z to obtain the clustering results.

3.4 The Reparameterization Trick for Problem Solving

In order to solve the problem, let z i be a continuous random variable, and z i ∼ p θ i ( z i | x i ) be some conditional distribution of the i -th view. It is then often possible to express the random variable z i as a deterministic variable z i = f θ i ( ϵ i , x i ), where ϵ i is an auxiliary variable with independent marginal q ( ϵ ) and function f θ i ( · ) parameterized by θ i . We let z i ∼ q ϕ i ( z i | x i ) = N ( µ i , σ i 2 ). In this case, a valid reparameterization is z i = µ i + σ i ϵ i , where ϵ i is an auxiliary noise variable ϵ i ∼ N (0 , 1).

4 Discussion

4.1 Connection to VaDE

Variational deep embedding (VaDE) [17] is an unsupervised generative clustering algorithm, which combines VAE and Gaussian Mixture Model (GMM) to model the data generation process. In our method, from Eq. (3), it can be found that SCMVC has taken into account the consistency information among views compared with the VaDE for the single-view scenario. Moreover, we introduce the SUMVC to help obtain a more su ffi cient representation, and we propose new methods to solve the problem based on the reparameterization trick.

4.2 Bayes Error Rate for Downstream Task

In this section, inspired by [8, 39], we present a theoretical analysis of the generalization error of representations, assuming T to be a categorical variable. We take view j as self-supervised signals to guide the learning of view i as an example. To represent the irreducible error when inferring labels from the representation via an arbitrary classifier, we use Bayes error rate Pe : = E z i ∼ P z i [1 -max t ∈ T P ( ˆ T = t | z i )], where Pe is the Bayes error rate of arbitrary learned representations z i , and ˆ T is the estimated label T from our classifier. Analyzing the Bayes error rate allows us to gain insights into the generalization performance of our propose model. We present a general form of the complexity of the sample with mutual information I ( z i ; x j ) estimation using empirical samples from the distribution Pz i , x j . Let P ( n ) z i , x j denote the uniformly sampled empirical distribution of Pz i , x j . We define ˆ I ( n ) θ ∗ ( z i ; x j ) as E P ( n ) z i x j [log p ( x j | z i ) p ( x j ) ].

,

Proposition 1 (Mutual Information Neural Estimation [40].) Let 0 < δ < 1 . There exists d ∈ N and a family of neural networks F : = ˆ f θ : θ ∈ Θ ⊆ R d , where Θ is compact, so that ∃ θ ∗ ∈ Θ , with probability at least 1 -δ over the draw of { z x i m , x j m } n m = 1 ∼ P ⊗ n z i x j ,

,

This proposition shows that there exists a neural network θ ∗ , with high probability, ˆ I ( n ) θ ∗ ( z i ; x j ) can approximate I ( z i ; x j ) with n samples at rate O GLYPH<18> 1 √ n GLYPH<19> . Under this network θ ∗ and the same parameters d and δ , we are ready to present our main results on the Bayes error rate.

Theorem 4.1 (Bayes Error Rates for Arbitrary Learned Representations [39].) Let | T | be T's cardinalitiy and Th ( x ) = min { max { x , 0 } , 1 -1 | T | } . Given pe = Th ( ¯ pe ) , we have a thresholding function:

Given arbitrary learned representations ( z i ), Theorem 4.1 suggests the corresponding Bayes error rate ( Pe ) is small when: 1) the estimated mutual information ˆ I ( n ) θ ∗ ( z i ; x j ) is large; 2) a larger number of samples are used for estimating the mutual information; and 3) the task-irrelevant information I ( z i ; x i | x j , T ) is small. The first and the second results support the claim that maximizing I ( z i ; x j ) may learn the representations beneficial to downstream tasks. The third result implies the learned representations may perform better on the downstream task when the task-irrelevant information is small. We evaluate the quality of our self-supervised method's learned representations using Bayes Error Rates in Theorem 4.2. The proofs are given in Appendix B.4.

Theorem 4.2 The Bayes Error Rates for arbitrary learned representations can be minimized by maximizing L i 1 + β L i j 2 , formally:

5 Experiments

5.1 Experimental Setup

Datasets As shown in Table 1, we use the following four real-world multi-view datasets in our study. MNIST [20] is a widely used dataset of handwritten digits from 0 to 9. The Fashion dataset [45] comprises images of various fashion items, including T-shirts, dresses, coats, etc. The COIL dataset [29] contains images of various objects, such as cups, ducks, and blocks, shown in di ff erent poses. We use multi-view datasets derived from origin datasets: Multi-COIL-10, Multi-COIL-20, Multi-MNIST and Multi-Fashion. Each dataset includes multiple views of each example, all randomly sampled from the same category. In Multi-COIL-10 (K = 10) and Multi-COIL-20 (K = 20), di ff erent views of an object correspond to various poses, but retain the same label. In Multi-MNIST, di ff erent views of a digit represent the same digit written by di ff erent individuals. In Multi-Fashion, di ff erent views of a product category signify di ff erent fashionable designs for that category.

Comparison Methods The comparison methods include three single-view clustering methods: K -means (1967) [27], β -VAE ( β -VAE: learning basic visual concepts with a constrained variational framework (2017) [12]), and VaDE (variational deep embedding: an unsupervised and generative approach to clustering (2016) [17]), the input of which is the concatenation of all views, and five state-of-the-art MVC methods: BMVC (binary multi-view clustering (2018) [53]), SAMVC (selfpaced and auto-weighted multi-view clustering (2020) [32]), RMSL (reciprocal multi-layer subspace learning for multi-view clustering (2019) [22]), DEMVC (deep embedded multi-view clustering with collaborative training (2021) [47]), FMVACC (fast multi-view anchor-correspondence clustering (2022) [43]).

Evaluation Measures Three measures are used to evaluate the clustering results, including, clustering accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). A larger value of these metrics indicates a better clustering performance.

5.2 Comparison with State-of-the-Art Methods

Table 2 shows the quantitative comparison between the proposed methods and baseline models for several datasets. The best result is highlighted in bold, while the second-best result is underlined in each line. Despite its simplicity, the SCMVC method demonstrates similar performance to existing methods in most cases. This is primarily because our method focuses on extracting global information from multiple distributions. Moreover, the results reveal that SUMVC outperforms existing methods in terms of three evaluation methods. Notably, SUMVC achieves remarkable improvements on Multi-COIL-10 and Multi-COIL-20. The reason is that the proposed lower bound could explore consistent predictive information and reduce redundancy among views, which benefits clustering results.

5.3 Ablation Studies

To further verify the contributions of the proposed method, we conduct ablation studies on Eq. (11), which consists of Lrec , LKL , and Lsuf . As shown in Table 3, it is found that the performance of clustering is significantly influenced by LKL . We can also find that combining Lcon and Lsuf yields better results than using either Lcon or Lsuf alone, which illustrates that each loss function plays a crucial role in the overall performance of the model. We find that the clustering performance is not ideal when only Lsuf is used for training. The reason is that the latent features extracted by the lack of reconstruction constraints cannot accurately reflect the information of the original features.

5.4 Visualization of Feature Representation

We use t -SNE [42] to visualize the learning process of embedded features. The non-linear dimensionality reduction technique is used for visualizing and clustering high-dimensional data, which has been widely used in machine learning and data analysis to visualize high-dimensional data

(a) epoch

=

50

(b) epoch

=

100

(c) epoch

=

300

(d) epoch

=

400

(e) epoch

=

100

(f) epoch

=

200

(g) epoch

=

400

(h) epoch

=

800

ACC

0.5

IWN

0.5

ACC

0.5

NMI

0.5

0.01

0.01

0.01

0.01

0.1

0.3

β

9'0

0.7

0.01

0.1

人

0.3

0.5

0.7

0.1

0.3

β

0.5

0.7

0.01

0.1

人

0.3

0.5

0.7

0.1

0.3

β

0.5

0.7

0.01

0.1

0.3

0.5

0.7

0.1

°0

B

0.5

0.7

0.01

0.1

0.3

0.5

0.7

in lower-dimensional spaces for easier understanding and interpretation. We show the embedded features ⃗ z of Multi-COIL-10 and Multi-COIL-20 in Fig. 1. At the start of the training stage, the embedded features are non-separable. As the training progresses, the clustering structures of the embedded features become more apparent while their centroids gradually separate. This visual demonstration highlights the e ff ectiveness of our proposed model.

5.5 Parameter Sensitivity Analysis

SUMVC has two primary hyperparameters, i.e., γ and β . We test the clustering performance with di ff erent settings to demonstrate the stability of SUMVC. Results are presented in Fig. 2. Additionally, higher values of γ are required to encourage disentangling but result in a fidelity-disentanglement trade-o ff between the quality of reconstructions and its latent features disentanglement. β that is either too large or too small also leads to reduced clustering performance due to the imbalance between reducing redundant information and obtaining consistent information. The recommended parameters for γ and β are both 0.1 in our experiments.

6 Conclusion

In this paper, we propose a simple consistent approach and an information-theoretic approach for multi-view clustering to address the challenge of obtaining useful information from multiple views while reducing redundancy and ensuring consistency among views. In particular, we leverage the information bottleneck principle and variational inference to the multi-view scenarios. The proposed SCMVCandSUMVCareextensively evaluated on multiple real-world multi-view datasets. Extensive experiments demonstrate the clustering performance of the proposed models.

References

Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. JMLR , 19:1947-1980, 2018.

Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. TPAMI , 40:2897-2905, 2018.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory , pages 92-100, 1998.

TM Cover and JA Thomas. Joint entropy and conditional entropy. Elements of Information Theory, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA , page 16, 2006.

Iain J Cruickshank. Multi-view clustering of social-based data. PhD diss., Carnegie Mellon University , 2020.

Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. TPAMI , 35(11):2765-2781, 2013.

Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions on Information Theory , 40:259-266, 1994.

Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In ICLR , 2020.

Jacob Goldberger, Hayit Greenspan, and Shiri Gordon. Unsupervised image clustering using the information bottleneck method. In DAGM-Symposium , pages 158-165, 2002.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63:139-144, 2020.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta -vae: Learning basic visual concepts with a constrained variational framework. In ICLR , 2017.

Shudong Huang, Hongjie Wu, Yazhou Ren, Ivor Tsang, Zenglin Xu, Wentao Feng, and Jiancheng Lv. Multi-view subspace clustering on topological manifold. In NeurIPS , pages 25883-25894, 2022.

Zhanpeng Huang, Jiekang Wu, Jinlin Wang, Yu Lin, and Xiaohua Chen. A unified multi-view clustering method based on non-negative matrix factorization for cancer subtyping. IJDWM , 19:1-19, 2023.

HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view representation learning via total correlation objective. In NeurIPS , pages 12194-12207, 2021.

Senerath Mudalige Don Alexis Chinthaka Jayatilake and Gamage Upeksha Ganegoda. Involvement of machine learning tools in healthcare decision making. Journal of Healthcare Engineering , 2021.

Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: an unsupervised and generative approach to clustering. In IJCAI , pages 1965-1972, 2017.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NeurIPS , pages 3581-3589, 2014.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Ha ff ner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86:2278-2324, 1998.

Chung-Kuei Lee and Tyng-Luh Liu. Guided co-training for multi-view spectral clustering. In ICIP , pages 4042-4046, 2016.

Ruihuang Li, Changqing Zhang, Huazhu Fu, Xi Peng, Tianyi Zhou, and Qinghua Hu. Reciprocal multi-layer subspace learning for multi-view clustering. In ICCV , pages 8172-8180, 2019.

Zhaoyang Li, Qianqian Wang, Zhiqiang Tao, Quanxue Gao, Zhaohua Yang, et al. Deep adversarial multi-view clustering network. In IJCAI , pages 2952-2958, 2019.

Jialu Liu, Chi Wang, Jing Gao, and Jiawei Han. Multi-view clustering via joint nonnegative matrix factorization. In SDM , pages 252-260, 2013.

Yunxiao Liu, Youfang Lin, QinFeng Xiao, Ganghui Hu, and Jing Wang. Self-adversarial variational autoencoder with spectral residual for time series anomaly detection. Neurocomputing , 458:349-363, 2021.

Can-Yi Lu, Hai Min, Zhong-Qiu Zhao, Lin Zhu, De-Shuang Huang, and Shuicheng Yan. Robust and e ffi cient subspace segmentation via least squares regression. In ECCV , pages 347-360. Springer, 2012.

James MacQueen. Classification and analysis of multivariate observations. In BSMSP , pages 281-297, 1967.

Jonathan Masci, Ueli Meier, Dan Cire¸ san, and Jürgen Schmidhuber. Stacked convolutional auto-encoders for hierarchical feature extraction. In ICANN , pages 52-59, 2011.

Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). Technical Report, CUCS-005-96 , 1996.

Feiping Nie, Hua Wang, Heng Huang, and Chris Ding. Unsupervised and semi-supervised learning via l 1-norm graph. In ICCV , pages 2268-2273. IEEE, 2011.

Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. An information bottleneck approach for controlling conciseness in rationale extraction. In EMNLP , pages 1938-1952, 2020.

Yazhou Ren, Shudong Huang, Peng Zhao, Minghao Han, and Zenglin Xu. Self-paced and auto-weighted multi-view clustering. Neurocomputing , 383:248-256, 2020.

Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences , 116:11537-11546, 2019.

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. Information Flow in Deep Neural Networks , page 24, 2022.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingcheng Fu, Cheng Ji, and Philip S Yu. Graph structure learning with variational information bottleneck. In AAAI , pages 4165-4174, 2022.

N TISHBY. The information bottleneck method. Computing Research Repository (CoRR) , 2000.

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop , pages 1-5, 2015.

Daniel J Trosten, Sigurd Lokse, Robert Jenssen, and Michael Kamp ff meyer. Reconsidering representation alignment for multi-view clustering. In CVPR , pages 1255-1265, 2021.

Y-H Tsai, Y Wu, R Salakhutdinov, and L-P Morency. Self-supervised learning from a multi-view perspective. In ICLR , 2021.

Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Russ R Salakhutdinov. Neural methods for point-wise dependency estimation. In NeurIPS , pages 62-72, 2020.

Tim Van de Cruys, Thierry Poibeau, and Anna Korhonen. A tensor-based factorization model of semantic compositionality. In Conference of the North American Chapter of the Association of Computational Linguistics (HTL-NAACL) , pages 1142-1151, 2013.

Laurens Van der Maaten and Geo ff rey Hinton. Visualizing data using t-sne. JMLR , 9:2579-2605, 2008.

Siwei Wang, Xinwang Liu, Suyuan Liu, Jiaqi Jin, Wenxuan Tu, Xinzhong Zhu, and En Zhu. Align then fusion: Generalized large-scale multi-view clustering with anchor matching correspondences. In NeurIPS , pages 5882-5895, 2022.

Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurIPS , pages 12104-12115, 2021.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML , pages 478-487, 2016.

Jie Xu, Yazhou Ren, Guofeng Li, Lili Pan, Ce Zhu, and Zenglin Xu. Deep embedded multi-view clustering with collaborative training. Information Sciences , 573:279-290, 2021.

Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. Multi-vae: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In ICCV , pages 9234-9243, 2021.

Jie Xu, Yazhou Ren, Huayi Tang, Zhimeng Yang, Lili Pan, Yang Yang, Xiaorong Pu, Philip S Yu, and Lifang He. Self-supervised discriminative feature learning for deep multi-view clustering. TKDE , pages 7470-7482, 2023.

Yuanjin Xu and Ming Wei. Multi-view clustering toward aerial images by combining spectral analysis and local refinement. Future Generation Computer Systems , 117:138-144, 2021.

Wenbiao Yan, Jihua Zhu, Yiyang Zhou, Yifei Wang, and Qinghai Zheng. Multi-view semantic consistency based information bottleneck for clustering. arXiv preprint arXiv:2303.00002 , 2023.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In ICML , pages 12310-12320, 2021.

Zheng Zhang, Li Liu, Fumin Shen, Heng Tao Shen, and Ling Shao. Binary multi-view clustering. TPAMI , 41:1774-1782, 2018.

Handong Zhao, Zhengming Ding, and Yun Fu. Multi-view clustering via deep matrix factorization. In AAAI , pages 2921-2927, 2017.

Shuai Zheng, Xiao Cai, Chris Ding, Feiping Nie, and Heng Huang. A closed form solution to multi-view low-rank regression. In AAAI , pages 1973-1979, 2015.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS , pages 321-328, 2003.

Yiyang Zhou, Qinghai Zheng, Shunshun Bai, and Jihua Zhu. Semantically consistent multi-view representation learning. arXiv preprint arXiv:2303.04366 , 2023.

Yiyang Zhou, Qinghai Zheng, Wenbiao Yan, Yifei Wang, and Jihua Zhu. Mcoco: Multi-level consistency collaborative multi-view clustering. arXiv preprint arXiv:2302.13339 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two methods for multi-view clustering, which aims at grouping data from multiple sources or perspectives. The first part, SCMVC uses a consistent variational lower bound to learn consistent information among views. The second part, SUMVC extends the information bottleneck principle to reduce redundancy and achieve sufficient representation among views. The proposed SUMVC consists of two terms: a consistent variational lower bound and a sufficient representation lower bound to enhance consistency and minimize redundancy among views. The authors leverage information bottleneck theory and variational analysis to develop the model. The paper also provides a theoretical analysis of the generalization error of the learned representations based on the Bayes error rate. The paper evaluates the proposed methods on four real-world multi-view datasets.\n\nSTRENGTHS:\n1.\tThe paper addresses an important and challenging problem of MVC with an information-theoretic perspective. This paper proposes novel lower bounds to address the issue of view redundancy and consistency. The authors utilize Bayes Error Rate to provide a theoretical explanation of the effectiveness of the proposed method. \n3.\tThe paper presents extensive experiments on four real-world multi-view datasets and demonstrates the superior performance of the proposed methods over existing methods.\n\nWEAKNESSES:\n1.\tThe proposed objective function in Eq. (4) shares similarities with that of VAE [1], and it would be beneficial if the author could provide further explanations on this matter. How does this loss function differ from that of VAE? Is it merely an extension of the single-view approach to multi-view data? Additionally, what is the reason for SCMVC achieving significantly better performance than VAE-based methods in Table 2?\n2.\tIn SCMVC, the fusion representation $\\vec{Z}$ is directly optimized, while it is not in SUMVC. Could this approach negatively impact the performance of SUMVC?\n3.\tThe methods discussed in this article may not be effectively applied to datasets with high heterogeneity. \n4.\tThe introduction section contains some complex sentences. Simplifying the language and structure could improve clarity.\n5.\tIt would be beneficial for the authors to provide a more comprehensive discussion of some missed information-based multi-view methods [2-5] in the related work section.\n\n[1] Auto-Encoding Variational Bayes, NIPS’13\n\n[2] COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction, CVPR'21\n\n[3] Rethinking Minimal Sufficient Representation in Contrastive Learning, CVPR’ 22\n\n[4] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'23\n\n[5] Multi-view information-bottleneck representation learning, AAAI’21\n\nQUESTIONS:\nThe authors mention that the methods discussed in this article cannot be effectively applied to datasets with high heterogeneity. Could authors provide a detailed analysis?\n\nLIMITATIONS:\nPlease see weakness. Also, the complexity analysis is missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

comments: SUMMARY:
The paper proposed Sufficient Multi-View Clustering , SUMVC, which is composed of two main components. The first component is a simple and reliable multi-view clustering method called SCMVC (simple consistent multi-view clustering), which utilizes variational analysis to generate consistent information. The second component proposes a lower bound on sufficient representation, aiming to enhance consistent information and reduce unnecessary information among views.

STRENGTHS:
- Analyze the effectiveness of multi-view clustering from the information-theoretic perspective is interesting and valuable.
- The proposed method outperforms the state of the arts. The paper has good reproducibility with the provided codes.
- The paper is very well written. The theoretical and empirical analyzes are convincing.

WEAKNESSES:
- There are few comparative methods, and more comparative methods need to be selected reasonably to illustrate the effectiveness of the proposed methods.
- The theoretical analysis (based on the Bayes Error Rate) should be elaborated more. The key findings or insights from this analysis should also be emphasized.
- Did the experimental results demonstrate any limitations or potential challenges of the proposed method?

QUESTIONS:
see above.

LIMITATIONS:
It can be found in Appendix of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This work introduces a new approach called sufficient multi-view clustering (SUMVC) to improve clustering performance using multiple data sources. Existing methods often focus on acquiring consistent information while neglecting the issue of redundancy across multiple views. By contrast, the proposed SUMVC provides a promising solution to the problem of multi-view clustering and offers a new perspective for analyzing multi-view data. The effectiveness of the model is verified through theoretical analysis and experiments on multiple multi-view datasets, showing superior performance compared to other methods.

STRENGTHS:
1. By examining the multi-view clustering framework from an information-theoretic standpoint, SUMVC offers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data. 
2. The effectiveness of the proposed model is demonstrated through theoretical analysis based on the Bayes Error Rate and experiments on multiple multi-view datasets, highlighting its superior performance compared to other methods.

WEAKNESSES:
1. This paper is lack of in-depth discussion or evaluation of the proposed approach's limitations. 
2. The authors could provide more insights into the computational complexity of the proposed method and its scalability to large-scale multi-view datasets.

QUESTIONS:
The paper mentions that the superiority of SUMVC is demonstrated through experiments on multiple multi-view datasets. However, there is no detailed discussion or analysis provided on the characteristics of these datasets or how they were selected. What are the criteria for dataset selection, and how representative are these datasets?

LIMITATIONS:
No potential negative societal impact of this work exists.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of multi-view clustering from an information theoretic perspective. It focuses on representation learning, and optimizes said representation to improve down-stream clustering performance (with k-means). It introduces an Information Bottleneck based loss function, which considers consistency between views, redundancy and sufficiency of representations in addition to the traditional likelihood based reconstruction loss. It introduces 2 methods SCMVC and SUMVC based on some or all of these additional loss terms.\n\nThe paper then analyzes the model effectiveness by establishing a connection to the Bayes Error Rate, and also shows good experimental results across some multi-view datasets.\n\nSTRENGTHS:\n- Introduces a novel approach for representation learning for Mmulti-View Clustering (MVC) based on information theoretic criteria.\n- Proposes two separate methods, both of which show good experimental performance.\n- Has mathematical rigor in both breaking down the loss function, as well as in the theoretical analysis that follows.\n- Compares against multiple state-of-the-art MVC methods and shows superior performance on the chosen datasets.\n- Conducts a wide range of experiments, including ablation studies and parameter sensitivity analysis.\n\nWEAKNESSES:\n- The presentation of the mathematical parts of the preliminaries, discussion and analysis is quite opaque. Variables and terms are often explained only after they have already been used in equations. The equations tend to be quite cluttered and difficult to parse.\n- The datasets chosen are less than ideal, and are mostly derivative from single-view datasets. There are other multi-view datasets such as NUS-WIDE and 3-Source News which are potential candidates for publicly available MV datasets.\n- The assumptions (and thus applicability) of the methods are restrictive. I.e. assuming mutual redundancy across all views. As they mention in the supplementary, their methods perform poorly on heterogeneous MV data, limiting their applicability.\n- The paper was not self-contained; the supplementary material was essentially required to understand the a lot of the details. For eg. the limitations of the methods were only mentioned in the supplementary material. The supplementary material must not be used as additional pages, and the paper itself should be able to stand alone.\n\nQUESTIONS:\nOverall questions:\n\n- This paper is more about representation learning for clustering than clustering itself, right? Similar to spectral clustering. Not that that's a bad thing, it's just a little misleading. You spend a lot of the paper talking about MVC but the actual clustering is just k-means on top of a learned representation. It might help to be clear about this up front.\n- As I mentioned in the second point of the Limitations, it seems like this method is also weak to only partial information existing in views. How would you remedy this? Would ensemble variational methods work here as well?\n- For the MNIST experiments, I am confused about your choice to use pairs of the same digit (but written by different people) as two views. The data distribution of both the views are very similar, then. Also, if you pair up digits, shouldn't your dataset size be 35000 (halved) and not 70000? Or are you using both (A, B) and (B, A) for each pair? If so, this doesn't really seem like a good multi-view dataset since the views themselves are basically indistinguishable overall.\n- In table 3, the 3rd row seems to not have any useful information. It's clear that $L_{rec}$ would be required since you're using an auto-encoder. Instead, maybe you can have $L_{rec} + L_{suf}$ which you don't have here. That would be interesting to see.\n\n\nI will leave line-by-line comments here (since there doesn't seem to be a better place for this). There are a few typos here and there, but I won't bother too much with those:\n\n- [Line 32] Did you mean maximizing MI between representation* and output (not input and output)\n- [Line 82] \"... to quantify amount of\" -- sentence is not complete.\n- [Section 3] You use y, z without explaining/defining what they are. Clustering is an unsupervised task, so what does this mean in this context? I'm guessing it means cluster assignment but it isn't clear. \n- [Line 164] I may be wrong but isn't $\\phi^i$ the generational parameter set, and not $\\theta^i$?\n- [Line 180] What is a \"pseudo-label?\" A cluster assignment? You should define this earlier.\n- [Line 164] What do you mean by unique distribution?\n- [Line 186] This equation is very hard to parse. For cleaner appearance, you should consider using \\left( and \\right) to have larger parantheses. You could also remove the superscripts just for these long equations where there is no $j$, and just leave a note below. Also, shouldn't there be a conditioning on y in the first term? Lastly, $KL$ -> $D_{KL}$.\n- [Line 198] maximizing*\n- [Equation 9] There should be no expectation here, right? The $D_{KL}$ absorbs it, I think. Also, for the second term. the first distribution uses $z^j$ instead of $z^i$.\n- [Line 243] Should it be $z^j_m$ here? Also, what is $P^{\\otimes n}$?\n\nLIMITATIONS:\n- [Author mentioned] Heterogeneity in the data (eg. in terms of dimensions of features) affects performance significantly.\n- Restrictive assumptions are made on mutual redundancy between views. Multi-view data often has only partial information available in each view. I.e. you might need more than one view to get the complete picture. This also seems like a weakness of the methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
A consistent variational lower bound is provided to explore the consistent information among views for multi-view clustering, based on which SCMVC (simple consistent multi-view clustering) is proposed. To enhance consistent information and minimize unnecessary information among views, a sufficient representation lower bound is further proposed.

STRENGTHS:
(1) The main idea is novel, necessary preliminary knowledge and sufficient theoretical analysis is given.

(2) Experiments conducted on real multi-view data demonstrate the good performance of proposed methods. The codes are available. 

(3) The findings of this study have the potential to contribute to the advancement of multi-view clustering techniques.

WEAKNESSES:
(1) It is said the proposed model does not perform well on datasets with strong heterogeneity between views, but no evidence or experiments support this statement. How does the heterogeneity affect the performance should be explained.

(2) The paper shows quantitative results of the proposed SUMVC and SCMVC. However, I expect some visualization results to show the difference between these two methods.

(3) A thorough analysis of potential drawbacks, and practical considerations would enhance the overall strength of the paper.

(4) The tested data sets in this paper contain a small number (<=3) of views. It is suggested to add data sets with more than three views for discussion.

(5) The writing should be further improved.

QUESTIONS:
Please see ‘Weaknesses’.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06