Generalized Information-theoretic Multi-view Clustering
Authors: Weitian Huang
Sirui Yang
Hongmin Cai
Venue: NeurIPS 2023 poster (2023)
Keywords: information bottleneck
multi-view clustering
variational autoencoders
TL;DR: This work extends the information bottleneck theory to multi-view scheme in the unsupervised setting.

Abstract: In an era of more diverse data modalities, multi-view clustering has become a fundamental tool for comprehensive data analysis and exploration. However, existing multi-view unsupervised learning methods often rely on strict assumptions on semantic consistency among samples. In this paper, we reformulate the multi-view clustering problem from an information-theoretic perspective and propose a general theoretical model. In particular, we define three desiderata under multi-view unsupervised learning in terms of mutual information, namely, comprehensiveness, concentration, and cross-diversity. The multi-view variational lower bound is then obtained by approximating the samples' high-dimensional mutual information. The Kullback–Leibler divergence is utilized to deduce sample assignments. Ultimately the information-based multi-view clustering model leverages deep neural networks and Stochastic Gradient Variational Bayes to achieve representation learning and clustering simultaneously. Extensive experiments on both synthetic and real datasets with wide types demonstrate that the proposed method exhibits a more stable and superior clustering performance than state-of-the-art algorithms.

Generalized Information-theorrtic Multi-view
Clustering
Weitian Huang
Sirui Yang
School of Computer Science and Engineering South China University of Technology Guangzhou, 510006, China
School of Computer Science and Engineering
South China University of Technology Guangzhou, 510006, China
Hongmin Cai*
School of Computer Science and Engineering South China University of Technology Guangzhou, 510006, China
Abstract
In an era more diverse data modalities, multi-view clustering has become a fundamental tool for comprehensive data analysis and exploration. However, exist- ing multi-view unsupervised learning methods often rely on strict assumptions on semantic consistency among samples. In this paper, reformulate the multi-view clustering problem from an information- -onnation-thooretic perspective and propose a gen- eral theoretical model. In particular, unsupervised learning in terms of mutual information, namely, comprehersiveness concentration, and cross-diversity. The multi-view variational lower bound is then obtained by approximating the samples' high-dimensional mutual informa- tion. The Kullback-Leible divergence utilized deduce sample assignments. Ultimately the information-uun multi-view clustering model leverages deep neu- ral networks and Stochastic Gradient Variational Bayes to achieve representation learning and clustering simultaneously. Extensive experiments on both synthetic and real datasets with wide types demonstrate that the proposed method exhibits more stable and superior clustering performance than tate-of-the-an algorithms.
define three desiderata under multi-view
Introduction
Multiple views of data are often collected by different measurement methods to represent various properties of objects in many practical applications. For example, for images or videos, various visual features such HOG (1. SIFT (2), and LBP 3 can be acquired by traditional filters and form kind multi-view data. These features can be collected from different domains or generated from various sensors. In general, different views can complement each other due limitations or biases in measurement methods that would result in insufficient information contained in individual views [4].
The ever-increasing network activity brings massive unlabeled data, which makes clustering, the traditional task machine learning, one of the hot topics in unsupervised learning. Importantly, in clustering task, learning the comprehensive and discriminative representations is crucial for partitioning the samples. Compared to single-view clustering, multi-view clustering (MVC) is widely studied as it exploits complementary information between different views to improve the ultimate performance. Most multi-view clustering algorithms learn the consistent representation among multiple views realize clustering. The benchmark techniques include view co-regularization (5)
Hongmin Cai is the Corresponding Author, whose E-mail is:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Canonical Correlation Analysis (CCA) 66 7 and low-rank tensor decomposition [8] Although
different methods have achieved superior clustering performance, they lack unified theoretical framework to help us further understand multi-view clustering.
Recently, Information Bottleneck (IB) 9 have attracted increasing attention in deep neural network research. The information bottleneck is an information theory principle that helps explain the puzzling success of today's artificial intelligence algorithms. This motivates to explain multi-view learning using information bottleneck theory. In this paper, we extend from supervised information bottleneck theory to unsupervised learning, and construct a general information theorettt multi-vier clustering framework. The main contributions of our work are illustrated as follows,
In unsupervised scheme, we define three requirements for multi-view representation learning from the view of information theory, comprehenssveeess, concentration, and cross-diversiny Consequently, a general information -aasad multi view clustering model (IMC) is proposed achieve representation learning and clustering simultaneously.
We present multi-view VAE scheme solve IMC by leveraging deep neural networks and Stochastic Gradient Variational Bayes (SGVB). Specifically, the multi-view mutual information approximated by variational lower bound, which can be expressed unified multi-view representation obtained through multiple variational autoencoders and sampling, followed by self- self-training strategy for clustering.
Experimental results show that the proposed method is effective learning informative yet minimal representations and exhibits satisfactory generalization ability.
2 Variants of Information Bottleneck
2.1 Information bottleneck
The information bottleneck principle 99] aims to obtain the most compressed representation of data while retaining task relerrelvant information. To achieve this, for given input data x with label y, the corresponding codeword z decided by p(z|x) can be obtained by optimizing the objective of the information bottleneck,
min p(x (2
T(Z; X) max I(Z - BI(Z; = LIB-
(1)
2
where I (Z; X) represents the mutual information between two random variables X and Z, which implicitly measures the level of compression in the representation LıB to identify the optimal encoding p(z|x) that minimizes [Z; X.) while ensuring / (Z; Y) exceeds a minimal threshold value I" This constrained problem can then be transformed into non-constraint problem by incorporating trade-off factor 3 that balances T(Z; Y) and I(Z; X). Such scheme yields sufficient and minimal representations that maximize task-relevant information while minimizing task- irreleaal information, thus balancing prediction and generalization performance.
2.2 Unsupervised information bottleneck
When the sample labels are unavailable, our aim is to compress each sample's description to the greatest extent possible while simultaneously maximizing the retention of the principal component information of the data. The unsupervised information bottleneck objective is described in [10).
max I(Z (Z; 2 Ep(x)) [En())>) [log g(x|z)] 3D (pLz)((z)))l := CUIB. (2) z
where (a) forms the lower bound of LUTB, which is achieved by substituting the intractable posterior p(x[z) and marginal p(z) with respective variational posterior (x|z)) and marginal g(z). Note that the lower bound is essentially identical 3VAE [T]. The detailed derivation process is shown in Appendix A.
In clustering tasks, one aims to discover the underlying structure of data and divide it into distinct clusters. The current deep clustering methods firstly extract deep low-dimensional embeddings and then apply standard clustering techniques. Theoretically, the deep clustering method is expected to

---

follow the information bottleneck principle while preserving the original data clustering structure. Therefore, one can tailor Eq. (2) to realize deep clustering.
max (2; (Z)
2Pe(x) [pp(x)x) [log ((x(2)] 3D 3Dxx(p(2)x))lleeeeeee - PRaaaPPaalllex)lldece =LIICC = LUIB- - pxp()) (xrtx))/Paet(l(cxlllaaaaaaa
Here we introduce a discrete variable represent the cluster and denote p(c[x) as the cluster distribution implied by the data. We aim for the cluster inferred by embedding to be close to the original cluster distribution. To measure this, we add KL divergence term with the balance factor It is worth noting that the gap between LB and LUIB lies in the preservation of the clustering structure in the process information compression.
Connection to DEC. Deep Embedding for Clustering (DEC) [2] begins with pre-training an auto- encoder learn low-dimensional representations, and the soft cluster assignment computed by the embeddings (as q( ((clz) in Eq. (3) is determined by the Student's (-distribution. The clustering loss is designed via KL divergence to approach the auxiliary target distribution (as p(c|x) in Eq. (3)) By setting 3 = 0 and y = 1, and using the pre- pre-training strategy, we obtain the DEC model.
Connection to VaDE. Variational Deep Embedding (VaDE) [3] combines VAEs and Gaussian mixture models and encodes the initial data distribution as a learned Gaussian mixture distribution in the latent variable space. By setting 3 and y get the VaDE model.
3 Information-based Multi-view Clustering
Let x { be dataset consisting of m views, where each view collection comprises J==
n independent and identically distributed (i.i.d) samples, ( = [x] (1) x ..... xx, <")).. Consider Z = (z) {m presenting view-peculias compressed representations for and c E {1,.... K} designating categorical random variable and represent the index of the actual cluster. For multi- view clustering, we first learn a unified latent representation across all views, which satisfies comprehensiveness, concentration and cross-diversity. Additionally, the multi-view representation is needed to preserve the original data structure information related to clusters followed by performing clustering.
3.1 Unified multi-view representation learning
We first define the desiderata for multi-view representation as follows.
Definition 3.1 (Comprehensive, concentrative and cross-diverse multi-view representation).
Comprehensiveness. Given the multi-vies observations X, representation, Z, is comprehensive each view observation can be predicted by Z.
Z arg max (Z:
(4)
Concentration. For each view data (*), their view-peculiar compressed representation ((0) concentrative if the redundant information for each view is eliminated,
Z (v)+ arg min 1 (Z (0) (e)).
(5)
Cross-diversity. View-peculiar latent representations Z can be cross-diverse if they avoid duplication of information within the view,
Z = arg max [ZZ Z).
2
To more intuitively grasp the definition of multi-view representations, we depict the three requirements for representation learning from the perspective of information theory. As illustrated in Fig I taking dual-view data as an example, the information carried by (L) and (2) are (x) and (X respectively, which are fixed. effectively convey the combination of the two views, the unified representation Z achieves informativeness by maximizing the mutual information between Z and
3

---

X(X
x
7.
2
(Z (X0)
min min/ (200) 1100)
(a) Comprehensiveness
(b) Concentration
max (2
z(1)
Z
max (2((2)2)
2
7
min (2(2
(c) Cross-diversity
Figure Illustration of three desiderata for learning multi-view representations.
X (2), Additionally, to remove redundant information from each view, minimize (( x and (( ( respectively for maximum compression of the respective view information. Ideally, semantically irrelevant information specific to each view is removed while complete semantically relevant information retained.
To balance comprehensiveness and concentration, we introduce mutual information between Z and Z (2) as a cross-diversity indicator,
I(Z: ( z (2) = I(Z: z + (Z I(Z ()) z()) I(
(7)
where we denote Dz () 2)) I(Z ( 2) I(Z ((() Z), representing the duplicate
information between views. As showed in Fig (c). one can approach cross cross-diversit by maximizing
[Z; ()), (Z; (2) and minimizing Dz ; (2)))
Proposition 3.1 (Cross-diversity is sufficient for comprehensiveness and concentration, respec- tively). Given two Markov chains ( - ( - Z and ( - ( - Z, maximization of I(Z (1) ()) sufficient for maximization of I(Z (1), (2)) when Z(1) ((2) are fixed and
minimization of "3 ((Z()); (()) when fixed, respectively.
Proof. Under the two Markov chains accompanied by loss of information, we have,
I (1) ( (b) ((Z ( x(2)),
Next rewrite the cross-diversity indicator,
I( H(Z (1) 2(2) (2)
-HZZ")+ ((Z
(1 (()) (1)
S H(ZZ)))
Since I(Z; (1, (()) achieves the maximum value, i.e. ( H(Z())+H(Z*)) 2))
We denote T(Z; (1) Z(2)) T,
/ (Zz (1) X (2) = -(T; (2))))
= =H(T)- - H x!R)
(c)
- H(Z + H( 2x1x), x(2)
= =H(Z()) (1 x(()))))) + - (x
= ((Zz))(1)) + x(21),

---

X(X)
X
2(0)
7
p(24)x00)
q(x
((1)
Z
X(I)
p(tp2i))2
(0.1)
(a) Special case
Z
p(z(2)x(22)
((x(a)
X(X
2
4
2
2
C
x
x
(0,1))
(b) General case
The architecture of dual-view instance of IMC
Figure 2: (a) Cluster information is only related to common information between multiple views. (b) View-peculiar information also includes semantic features. (c) Multi-view variational autoen- coders and fusion network are incorporated to perform representation learning and clustering simultaneously.
To satisfy both inequalities (b) and (c), the equation holds, i.e., IZZ (1) x = II
T(Z (())) Therefore, when T(z ( Z maximized, fixing z z(), T(Z: (1) ())
maximization holds, and conversely, fixing Z, ( x minimization holds.
With these desiderata, the unified multi-view representation learning can be formulated as follows,
max max 2,2
Σ T(Z x (T;;
E (x)
En(n|2)) llog q( (x(2))
- - DKz(p(z x ( ( + (); (8)
where (d) leverages the variational technique construct the lower bound of the first two terms (deviation can be found in Appendix B), the 3 is the balance coefficient that controls the cross-diversity of representation and balances the comprehensiveness and concentration. Mutual information between Z and multiple representations (11) ,..., Z (m) can further be expressed by,
m
T(Z Σ Z(
Dz Dz (0) (1).
3.2 Generalized multi-view clustering framework
For the multi-view clustering task, previous works 14 15 considered category-related or semantic- related information to be relevant only to the multi-view consensus part. As shown in Fig. 2 (a), the part where two views intersect contains all the information related cluster c the multi-view view data. However, this is only special case. More generally, the respective views contain specific semantic information that can complement each other. This is why a comprehensive understanding of an object event requires different views. As shown in Fig. (b), cannot determine if the view-common or view-pe -wwcueuuaar parts of the views contain more semantic information, we can only assume that they all contain this information.
Next, we seek to learn the unified multi-view representation without losing the original clustering structure of the data. For examples, assume that the data obeys Gaussian mixture distribution, or that the data has a tight intra-class structure and relaxed inter-class structure. We add a clustering term to Equation ( for measuring the distance between the posterior distribution of given original

---

data to clusters and the posterior distribution of a given multi-view representation to clusters. Then, the objective of information based mieti-view clustering can be formulated as,
LIMC =Bp(x) Ep(aax)) [log [log)(
(x (1g(((1))
data reconstruction
multi-regulaizationn
7 p( (Der)((c()<ll((c)e)))
2Z;
(10)
clustering
information shift
be regarded four items representing specific roles: data reconstruction, multi-regularization, information shift and clustering. The data reconstruction item maximizes the expectation of the log-likelihood to achieve comprehensive so that the multi-view representation retain more information of the data. The nulti-regularization term makes the multi-view representation forget the information of the original data by minimizing the KL divergence of the posterior distri- bution and the variational distribution to meet the requirement of concentration. Information shift item adjusts the information relationship between multi-view representations achieve cross-diverse. The clustering item measures the degree to which the data retains the original data structure after dimensionality reduction.
Note that Equation 10)
3.3 Numerical scheme to solve IMC
Following VAE [16], we use a deep neural network to estimate the parameters of the unknown distribution to instantiate IMC, as shown in Fig. (c). According to the Markov chain X(t) -e- ())) - ( the posterior p(z) X) and likelihood (()(z) can be factorized as, respectively,
p(z|x)= / /.. Pv pol (z
m
II
Patio (z (x )d dg
TF PF (2(1)) be(1)) Rap
Rey (xx(2) (e(2)) ....))...EE prmos (z((m) (e(m)) (2)(22(2())) )m)))
x Agtnn (xx()) (z),
where Pace) (z) (0) denotes the view-peculiar encoder with parameter ((0), Iow (x()))((() de-
notes the view-peculiar decoder with parameter @f(e), To obtain the unified multi-view representations from view-peculiar representations, we design fusion module (z) (x(0)) ) parameter with multiple fully connected layers v).
For clustering item, we adopt the same strategy as DEC 12. replace g(clz) with Student's t- distribution, and set target p(c) with auxiliary distribution,
gek: = (1+ ||z un 2
E, + Izz 18,--u1177 1
/ E, qik
Pdk E9 "e)
(11)
where lik measures the similarity between the representation z; and cluster center u4- Pik enhances the data points assigned with high confidence to improve the purity of the cluster.
For optimization IMC using Stochastic Gradient Variational Bayes (SGVB), the sampling op- erations of ((z(*) m 1= z) should be mapped the deterministic functions. According to the reparameterizatioo trick [16] for the continuous variable, we sample the t-th latent representations
by,
z (n) Mglh) + R )) where R.toR E Eon,, e ((..))
(n)
(12)
z, pu + Re where R R = En, 10,1
(13)

---

Algorithm 1 Optimization Procedure of IMC
Input: Multi-view dataset X'; Setting T = 1 and dimensionality of the latent variables.
Parameter: Initialize parameters { {0(1),, vb with random values, cluster centers u {ui,.., 3 } by K-means.
1: while not reaching the maximal epochs do
2: for in samples do
3: for e in m views do
4: Calculate (Magio) Egte)) through v-th encoder and then sample zit by Equation
5: end for
Calculate (Hyp Eg) through the fusion module and then sample Ziz by Equation 13 6: for e in m views do
7: Generate x(") by v-th decoders.
8: end for
9:
end for
10: Update {,)(0)) (B(1),, v, by maximizing Equation (14)
end while
Output: The cluster for each sample c, by selecting the index of maximum probability value q(c, 2z.)
where Ma(n) and Rgoss are the outputs of the v-th encoder, My and R.g. are the outputs of the fusion module, respectively.
By using Monte-Carlo estimators, the objective of IMC can be further written as,
T
max (at0) (e09).(( (o00
LIMC
log Mateo (x{ [zit)
(ztt) - llog Pate (Z (1))
uml
log.N(O,,))
Σ 1=1
Piu log
Pik
+ (L; 2),
(14)
Qik
where T denotes the number of Monte Carlo samples and is usually set to be Additionally. variational distribution g(2(") is set to multivariate standard normal distribution, i.e., g(z()))
(O, I). The mutual information between representations [z; Z) can be maximized by using sample- based differentiable mutual information lower bound, -hasnn-Shnnon estimator [17]. The partial
derivatives of each parameter combination are then calculated for the stochastic backpropagation technique. The concrete optimization procedure of IMC summarized in Algorithm
Related Work
Recently, several works have been devoted to the problem of multi-viww representation learning based on information theory. For supervised methods, (18) analyses the consistency and complementary characteristics of multi-view features from the information bottleneck theory, and strengthen the discriminative power of the encoder through the margin maximization method balance the accuracy and complexity of the multi-view model. [19] seeks to fuse the features of each view to obtain joint representation through a deep neural network, and maximize the mutual information between labels and joint representations to preserve the target information. Given a graph and its labels, (20) proposes Graph Information Bottleneck (GIB) framework to efficiently infer the most informative yet compressed subgraph for recognition.
While in the unsupervised scheme. [14] proposes multi-view representation learning method based on information bottleneck theory, Multi-view Information Bottleneck (MIB), MIB assumes that the shared part of the two views the target information, and the part shared by the two views is the superfluous information, followed by leveraging off-the-shelf information bottleneck technique to learn robust representations. [21 explores the potential relationship and intrinsic information between different views, and discarded redundant information from multi-view data with the help of the information bottleneck principle to balance the consistency and complementarity between multiple views. Under the framework of information theory. Completer [22) maximizes multi-view shared mutual information for consistent learning, and minimizes conditional entropy to recover missing view with available data. 23 regards each view source data self-supervised signal for
7

---

learning latent representations, removing redundant information from the view itself while retaining consistent information for shared features across multiple views. multivariate mutual information term is employed to decompose view-specific representations, which further ensures that the multi- view fusion self-expression matrix via view-common features is compact and informative. The aforementioned unsupervised multi-view representation learning methods aim to learn consistent information shared across multiple views, as shown in Fig. 2 (a). These methods either have limitations in practical applications or do not consider the duplicate information among views. It motivates us to construct a general information-theerrtie framework and introduce three requirements for multi-vies representation learning.
unsupervised multi-view clustering
5
Experiments
5.1 Experimental settings
Model setup: The architectures of Pateo (z(²) (2("))((') (x) and (ate) (x(") (z) are fully connected networks with with 6--500-500-1224--55 and 256 155-1004--00-000-d, neurons, respectively, where d, is the dimensionality of each view. The fusion network P(e) (z) {z(") ==)) concatenates multiple view- peculiar latent representations, followed by fully connected layer. Adam optimizer (24) is utilized to maximize the objective function, and set the learning rate be 0.001 with decay of 0.9 for every 10 epochs.
Datasets: We adopt four real-world datasets listed follows, (1) UCI-digits 255 contains 2000 examples of ten numerals from 0 to 9 with five views which are respectively extracted by Fourier coefficients, profile correlations, Karhunen-Love coefficient, Zernike moments, and pixel average extractors. (2) Notting-Hii [26] is widely used video face dataset for clustering, which collects 4660 faces across 76 tracks of the 5 main actors from the movie 'Notting Hill'. We use the multi-view version provided in (27). consisting of 550 images with three kind features, i.c., LBP, gray pixels,
and Gabor features. (3) BDGP [28] contains 2500 images in 5 categories, and each sample is described by 1750-D image vector and a 79-D textual feature vector. (4) Caltech20 is a subset of the object recognition dataset (35) containing 20 classes with six different views, including Gabor
features, wavelet moments, CENTRIST features, histogram of oriented gradients, GIST features and
local binary patterns.
Four state-of-the-an algorithms and two ablation models are used to compare the clustering performance, including: (1) DMVAE (36) assumes that the data and multi-view latent representation obey a Gaussian mixture distribution, which can be considered as a special case of IMC, i.c.. let 3 0 and 7 1. (2) MIB [4] considers the parts shared by both views target information and the view-specific parts as superfluous information. (3) CMiB-Nets (2]] proposes collaborative multi-view information bottlenecks learn the information inherent within view and the shared structure between views, while reducing redundant information. (4) Completer [15] introduces maximization of mutual information between views and minimization of conditional entropy of different views to achieve dual prediction and representation learning. For faimeess, we repeatedly perform all compared algorithms 10 times on four datasets and the mean and standard deviation were recorded.
Compared algorithms:
Ablation models:
For ablation study, construct two variants of IMC comparison. (1) IMC-v1
indicates that the objective function Equation (10) reduces the item of information shift, i.c., 3 = O. (2) IMC-v2 sets y 0 to discard the KL divergence of the clustering item, and uses k-means to perform clustering on the unified multi-view representation.
Evaluation metrics: For comprehensive analysis, we use three popular clustering metrics including Normalized Mutual Information (NMI), Accuracy (ACC) and adjusted rand index (ART). The higher the values of these indicators, the better the clustering performance.
5.2 Performance evaluation
We tested seven methods on four multi-view datasets. The experimental results are summarized in Table [] It can be observed that i) combining the four datasets in terms of three indicators, IMC has obtained the best performance times and the second-best performance times, which is the most stable and accurate multi-view clustering model. ii) Although the Completer model has achieved the
8

---

Table 1: Clustering performance comparison four datasets (mean=sstandard deviation). The optimal and suboptimal results are in bold and underlined, respectively.
Datasets
Metrics
DMVAE
MIB
CMIB-Naa Completer
IMC-v
IMC-v2
IMC
ACC
90.95=0.62
83.30=1.27
85.70=1.11 91.28+1.41
90.01=0.60
84,01=1.15
92.13:10.55
UCT-digits
NMI
85.54+1.06 75.43+
1..31+1.41 0..34
84.79+0032
19.01
88.01
ARI
85.40+1.554 1..5
76.97+
86.67+0086
84.78+0.43
78.1840.88
07,83++0.25
ACC
76.22:1.21 81.65+1.37
85,40112.36
80.17:2.79
77.7912.00
14.8311.90
87.10:11.35
Notting-Hij
NMI
72.97:1,96 75.95+2.52
78.65:22.57
76.11
74.93+11.44
78.7911.08
8..67:1.42
ARI
69.50=0.77 71.91+1.61
80.46+1.92
71.48+3329
70.30+2.15
79.10+2.32
10.19+1.74
ACC
10 86.82:10.65
85.82.10.58
1..3
88.7011142
80.46.1
91.46.10.82
BDGP
NMI 85,3 80.82.10.86
81.6010..6
74.25110.99
81.4711.12
73.31-2.45
84.40.1
ARI
78.58. 13.90.11.65
74.25.1.19
7..44.1.445
77.65.1..15
69.31
80.25.1
ACC
61.50=0.54 56.12:2.55
55.26+3.18
62.31=2.65
58.05=1.28
52,42+3.15
60.82
Caltech20
NMI
68.32+1.23
63.28+2.66
62.44+2.56
70.25+2.20
65.75+2.20
61.76+33.4
69.20+
ARI
19.86
18.90
26.45512.74 61.14
2..52 53.30:12.82
6..42:12.12
Table 2: Comparison of NMI at noise ratios of 10%, 30%, and 50% optimal and suboptimal results are in bold and underlined, respectively.
deviation). The
Datasets
Noise
DMVAE
MIB
CMIB-Nets Completer
IMC-vl
IMC-v2
IMC
10%
82.35 1.24 1.76
75.55++2.35 85.28 1.84 81.12422.6
78.46 1.45 86.82.1 1.05
UCI-digits
30%
75,84
66.35
65,5012.88
73,40=2.56
1.36 56.8443.40 25.30=2.66
72.18 69.9011.25
50%
63.80 58.46:22.80
56.4611.64 60.76=3.65
70.35:
best NMI performance in caltech20, its NMI performance lags behind by 12.97% in BDGP. This instabiliiyy can also be found in the MIB model. iii) The end-to-end model (IMC, IMC-v1, DMVAE) that considers the clustering loss in the process of learning multi-view representation will achieve better clustering results than the two-stage model (MIB, CMIB-nets, Completer), iv) Comparing IMC, IMC-vI and IMC-v2, the performance of using KL divergence clustering term has been significantly improved, while the use of information shift term has small improvement.
We therefore draw three conclusions: First, the IMC developed based on the general definition of multi-view information theory obtains more stable clustering results. Second, end-to-end models help to learn more discriminative representations. Third, in the process of learning multi-view representations, three requirements (comprehensiveness, concentration, cross-diversity) play role in improving the clustering performance.
5.3 Robustness Assessment
To evaluate the robustness of the model, we add different scales sparse noise the dataset UCI- digits. Specifically, we randomly select 10%, 30%, and 50% pixels of the original image size, and add pepper noise or salt noise each pixel with probability 0.5, i.c., set 255.
As shown by the clustering results in Table the performance of all algorithms decreases as the noise ratio increases, but the performance of our algorithm is always the best. It can be observed that the decline of IMC is the smallest, while the decline of the Completer model is obvious. This may be due to the assumption of multi-view semantic consistency is more sensitive as the noise ratio increases, the information shared by multiple views will also incorporate more noise. is worth noting that the performance of IMC-22 is better than that of IMC-v1, which contrary to the elustering experiment in Table [ It can be concluded that only pursuing inter-view consistent information leads to semantic bias when the dataset contains noise, while more informative representations can be obtained for robustness based on the general multi-view information bottleneck theory.
5.4 Parameter analysis
The proposed method contains two balance parameters, the information shift balance coefficient B8, and the clustering loss trade-off parameter 7, Although IMC at fixed values of some parameters has

---

MMI
20
0.37
8.0
0m
0.1
0.1
0.1
10
10
10
10
10
10
100
100
100
Figure 3: Parameter analysis of 3 and y on Caltech20 dataset.
shown satisfactory performance, it is still important to explore the effect of varying these parameters on the performance and mine the inner mechanism this model. We evaluate the parameter sensitivity of ACC, NMI, ARI metrics on Caltech20 dataset, as shown in Figure 3 we choose the values of 3 and y in the range of [0.01, 0.1, 1, 10, 100). can be seen from the results that our method is not sensitive to the coefficient of clustering loss, but how to adjust the parameter 3 is the key to improving clustering performance. As expressed in Equation (8). 3 controls the degree of cross-diversity of multi-view representation. Therefore, the selection strategy of the beta parameter can be based on prior knowledge of the dataset. In our experience, if there large degree of consistency in the semantics of multiple views, then the B is small and vice versa.
Conclusion
In this paper. we propose an information- theorretc multi-view clustering framework that avoids the assumption of semantic consistency across multiple views. Specifically, extend the information bottleneck theory to unsupervised multi-view learning through defining three requirements for multi- view representations via mutual information. By constructing multi-view variational lower bound and introducing KL. divergence as a clustering loss, the entire framework is finally optimized by deep neural network and stochastic gradient variational Bayes. Experiments verify the superiority and robustness of the generalized information-based multi-view clustering on four benchmark datasets and noisy data.
The practical limitations of the proposed model lie in the choice parameters. The mathematical strategy for selecting the optimal parameters is direction that can be studied in the future. In addition, constructing tighter lower bound or estimator to approximate high-dimensional mutual information also place for further improvement in this work.
Acknowledgment
We thank the anonymous reviewers for their helpful feedbacks. This work was supported part by the National Key Research and Development Program of China (2022 12200), the Key-Area Research and Development of Guangdong Province (2022A0055050014, 2022B 10500002, the Key-Area Research and Development Program of Guangzhou City (202206030009, 2023B01J0002), the National Natural Science Foundation of China (U21A20520 62172112), Guangdong Key Labo-
ratory of Human Digital Twin Technology (202281212010004)
References
[1] Navneet Dalal, and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR pages 886-893, 2005.
[2] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR pages 248-255, 2009.
10

---

[3] Timo Ojala, Matti Pietikainen, and Topi Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell., 24(7):971-98) 2002.
[4] Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learning. 1883, 2018.
IEEE Trans. Knowl. Data. Eng.,
[5] Abhishek Kumar, Piyush Rai and Hal Daume. Co-regularized multi-view spectral clustering. Adv. Neural Inf. Process. Syst. 24:1413-1421, 2011.
[6] Galen Andrew, Raman Arora, Jeff Bilmes and Karen Livescu. Deep canonical correlation analysis. In ICML, pages 1247-1255, 2013.
[7] Wang, Weiran, Xinchen Yan, Honglak Lee, and Karen Liveseu. Deep variational canonical correlation analysis. CoRR, abs/1610.03454, 2016.
[8] Changqing Zhang, Huazhu Fu, Si Liu, Guangcan Liu and Xiaochun Cao. Low-rank tensor constrained multiview subspace clustering. In ICCV, pages 1582-1590, 2015.
[9] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Annu. Allerton Conf. Commun. Control Comput., pages 368-377, 1999.
[10] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
[11] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[12] Xie, Junyuan, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, pages 478-487, 2016.
[13] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In IJCAI, 2017
[14] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In ICLR, 2021.
[15] Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
[16] Diederik Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2014.
[17] R. Devon Hjelm, Alex Fedorov, Samuel Lavcic-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
[18] Xu, Chang, Dacheng Tao, and Chao Xu. Large-margin multi-viewinformation bottleneck. IEEE Trans. Pattern Anal. Mach. Intell., 36(8):1559- 1572, 2014.
[19] Qi Wang, Claire Boudreau, Qixing Luo, Pang-Ning Tan, and Jiayu Zhou. Deep multi-view information bottleneck. In SIAM ICDM, pages 37-45, 2019.
[20] Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Graph information bottleneck for subgraph recognition. In ICLR, 2021.
[21] Zhibin Wan, Changqing Zhang, Pengfei Zhu, and Qinghua Hu. Multi-view information- bottleneck representation learning. In AAAI, pages 10085-10092, 2021.
[22] Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng I.v, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In CVPR, pages 11174-11183, 2021.

---

[23] Shizhe Hu, Zenglin Shi, Xiaoqiang Yan, Zhengzheng Lou, and Yangdong Ye. Multiview Clustering With Propagating Information Bottleneck. IEEE Trans. Neural Netw. Learn. Syst. 2023.
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[25] Arthur Asuncion and David Newman.
Uci
machine
learning
repository.
http://arcciiveeess..cc..dd/// 2007.
[26] Yi-Fan Zhang, Changsheng Xu, Hanqing Lu,, and Ych-Min Huang. Character identification in feature-length films using global face-name matching. IEEE Trans. Multimedia, 11(7):127--1288, 2009.
[27] Xiaochun Cao, Changqing Zhang, Huazhu Fu, Si Liu, and Hua Zhang. Diversity-induced multi-view subspace clustering. In CVPR, pages 586-594, 2015.
[28] Xiao Cai, Hua Wang, Heng Huang, and Chris Ding. Joint stage recognition and anatomical anmotation of drosophila gene expression patterns. Bioinformatics, 28(12)ii6-i24, 2012.
[29] Karpathy Andrej, and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, pages 3128-3137, 2015.
[30] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771-1800, 2002.
[31] Menglei Hu and Songcan Chen. Doubly aligned incomplete multi-view clustering, In IJCAI, pages 2262-2268, 2018.
[32] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In ICML, pages 1083-1092, 2015.
[36] Ming Yin, Weitian Huang, and Junbin Gao. Shared generative latent representation learning for multi-view clustering. In AAAI, pages 6688-6695, 2020.
[34] Menglei Hu and Songcan Chen. One-pass incomplete multi-view clustering, In AAAI, pages 3838-3845, 2019.
[35] Fei-Fei Li, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop. 2004.
[36] Ming Yin, Weitian Huang and Junbin Gao. Shared generative latent representation learning for view clustering, In AAAI, pages 6688-6695, 2020.
12

---

Appendix A. Derivation of Equation 2
By substituting the intractable posterior p(x(z) and marginal p(z) with respective variational posterior (x((z) and marginal q(z), we form a variational lower bound for unsupervised information bottleneck,
max! (Z; X) - (I; 2
= / p(2, x) loy p(x(z) dxdz 3 Σ p(x)
/
p(z, i) log p(zli) dz p(2)
=H(x) + -Epc)) If (0g)x))og((xxx2)
-3 Σ / p(2)x;) log p(z)
p(z))x;) dz
-Bp( I ((g (x of
Σ p(z/)x;) log p(z|xi) dz q(z)
B n
/ p(z)x)) log (xx22) 3 p(z|x) log p(z)x) dz g(z)
=Ep(x)
-Bp(x) ([pptl))) (x|2) 3D (p(z)x))l1 x))lq(2))
where H(x) is a constant and we take p(i) = and p(zli) = p(z|x:). When the dataset containing independent and identically distributed (i.i.d) samples is large enough, the expectation of p(x) can be regarded as uniform sampling from the dataset to get x,. Note that the lower bound is essentially identical to VAE. The former represents the data reconstruction process and the latter is a regularization term for forgotten identity information, driving the acquisition of informative and generative representations, respectively.
Appendix B. Derivation of Equation (
we substitute the intractable posterior p(X |z) and marginal p(z()) with respective variational posterior q( X 'z) and marginal (z)()), and then the variational lower bound for the first two terms of equation 8 yields as follows,
Same as Equation 2
771
max T(Z; 2,Z
Σ (( (())
=/ / p(z, x)
p(((z)
dX Σ)
/ (( log p(z(v)) dxx dbx(")
p(2(0)(((o))
log
p(x)
-x(X) + Expx)) / p((2|2) log (2)2
Σ Fipextis) // (xfo)) log ((2())((())
p(z(v)) d2((1)
2E 2 f p(z|x) log (())))())))) Σ E poccii) If p( log p(alo)/p(()) dz(")
{(z(1))
-Ep(x) Bp(x)x) [log xxx Σ Dkz(p(z(") ( ""x"aala""""")"aa"a"""""")
where (x) is constant. Given the unified multi-view representation z, multi-view datasets X' can be viewed as conditionally independent, i.e., p(x') p(x))))
The first term maximizes the expectation of the likeliiiddood to achieve comprehensive so that the multi- view representation retain more information of the data. The second term makes the multi-view representation forget the information of the original data by minimizing the KL divergence of the posterior distribution and the variational distribution to meet the requirement of concentration.
13


Reviews:
[{"review_id": "PA5sai5X4t", "reviewer": "Reviewer_i2fg", "summary": "In this paper, the authors introduce representation learning with the unsupervised information bottleneck to multi-view clustering. Based on the framework of information bottleneck, the authors theoretically summarize 3 key properties (comprehensiveness, concentrate, and cross-diversity) required by multi-view clustering representation. Finally, a DEC module is added to obtain the clustering assignment. ", "questions": "- (The concept of comprehensiveness) In Definition 3.1, the definition of comprehensive seems wrong: \" can be predicted by $Z$\". \n- It may be somewhat confusing to connect $\\min I(Z, X)$ and \"eliminating redundant information of each view\". It should minimize the irrelevant information between $X$ and $Z$.  \n- In Line 116, why is Eq. 4 used for *comprehensiveness* and *concentrate*? Is it the proposed cross-diversity?  \n\nOverall, the paper has some merits but requires significant proofreading. I believe the paper could be better but the quality of this version is unsatisfactory. I'd like to update my score after reading the response and other reviews. ", "limitations": "The limitations are not discussed in the main paper. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper is well-motivated and easy to follow. The pointed three desiderata are convincing although there may be some typos in the definition.  \n- The paper is technically sound. ", "weaknesses": "- There are many typos, especially in key concepts. For example, \n    - there may be plenty of typos of notations in Definition 3.1, which causes my major concern about the soundness of this paper. \n    - In Line112-113, \"minimize $I(Z^{(1)}, X^{(2)})$ and $I(Z^{(1)}, X^{(2)})$\" seems wrong. \n- The experiment may be a little insufficient. For example,\n    - The ablation experiments to study which 3 different desiderata are more important are missing. \n    - The visualization of learned $Z$ is also helpful to improve the quality. Since the primary contribution is the representation learning and there seems no apparent contribution to the clustering module, it is important to show whether the quality of representation is better.  \n    - The running time is missing.\n- The used mathematical techniques to derive the variational bound lack novelty (widely used in VIB, GIB, etc.).  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful and constructive feedback.\n\n**Q1:** In Line112-113, \"minimize and $I(\\textbf{Z}^{(1)}; \\textbf{X}^{(2)})$ and $I(\\textbf{Z}^{(1)}; \\textbf{X}^{(2)})$\" seems wrong.\n\n**A1:** I am sorry for the serious typos, here should be to minimize $I(\\textbf{Z}^{(1)}; \\textbf{X}^{(1)})$ and $I(\\textbf{Z}^{(2)}; \\textbf{X}^{(2)})$, for maximum information compression of each view.\n\n**Q2:** The ablation experiments to study which 3 different desiderata are more important are missing.\n\n**A2:** I'm sorry that I did the ablation experiment and analysis, but I didn't set up a chapter on ablation study in the manuscript, which made it easy to be ignored. We will highlight the ablation study more clearly in the new version.\n\nIn line 242 of Section 5.1 Experimental Settings, we construct two variants IMC-v1 and IMC-v2 to compare with IMC, and showed the experimental results in Table 1 and Table 2. The performance of using KL divergence clustering term has been significantly improved, while the use of information shift term has a small improvement. And in Section 5.2 Performance Analysis, we concluded that end-to-end models help to learn more discriminative representations.\n\n**Q3:** The visualization of learned $\\textbf{Z}$ is also helpful to improve the quality. Since the primary contribution is the representation learning and there seems no apparent contribution to the clustering module, it is important to show whether the quality of representation is better.\n\n**A3:** Thanks for your constructive suggestions. The representation visualization of multi-view learning can intuitively see the latent space structure after data dimensionality reduction, which is usually strongly related to the clustering metrics. We visualize the latent representations of all datasets and compare them with other multi-view representation learning methods, and we will present the results in the supplementary material due to the page limit of the main text.\n\n**Q4:** The running time is missing.\n\n**A4:** We tested the running time of 10 runs for 20 epochs on the UCI-digits dataset on a computer with an NVIDIA RTX 2070 GPU. The results show that Completer has the fastest running time, followed by our proposed model, which is close to MIB, because the multidimensional Jensen-Shannon estimator with high computational complexity is used.\n\n|   | DMVAE  | MIB  | Completer | IMC |\n|  ----  | ----  | ----  | ----  | ----  |\n| Running Time /s | 12.58 $\\pm$ 0.12 | 8.41 $\\pm$ 0.15 | 3.65 $\\pm$ 0.20 | 7.20 $\\pm$ 0.12|\n|\n\nThe results show that Completer has the fastest running time, followed by our proposed model, which is close to MIB, because the multivariate Jensen-Shannon estimator with high computational complexity is used.\n\n**Q5:** (The concept of comprehensiveness) In Definition 3.1, the definition of comprehensive seems wrong: \" can be predicted by \n$\\textbf{Z}$\".\n\n**A5:** There may be a bit of ambiguity here. The comprehensiveness means that multi-view observations $ \\textbf{X}^{(1)}, \\textbf{X}^{(2)}, ..., \\textbf{X}^{(V)}$ can be predicted/generated by the unified representation $\\textbf{Z}$.\n\n**Q6:** It may be somewhat confusing to connect $\\min I(\\textbf{Z}; \\textbf{X})$ and \"eliminating redundant information of each view\". It should minimize the irrelevant information between $\\textbf{X}$ and $\\textbf{Z}$.\n\n**A6:** In the unsupervised setting, without labels, it's not known which information in the data is irrelevant to the task. It is desired to keep the principal and intrinsic information about the sample and eliminate redundant information such as background and noise.\n\n**Q7:** In Line 116, why is Eq. 4 used for comprehensiveness and concentrate? Is it the proposed cross-diversity?\n\n**A7:** Like the unsupervised information bottleneck theory ($\\beta$VAE), it is desired to retain information about the data distribution ($\\max I(\\textbf{Z}; \\textbf{X})$) and forget the information for identity samples ($\\min I(\\textbf{Z}; i)$). For the multi-view scenario, we seek to retain the multi-view data distribution information ($\\max I(\\textbf{Z}; \\textbf{X}^{(1)},\\textbf{X}^{(2)})$), and want to forget the single-view information ($\\min I(\\textbf{Z}^{(v)}, \\textbf{X}^{(v)})$), and in order to match $\\textbf{Z}$ and $\\textbf{Z}^{(v)}$, we propose $\\max I(\\textbf{Z}; \\textbf{Z}^{(1)}, \\textbf{Z}^{(2)})$ as a trade-off glue (related proof in Proposition 3.1). There are two cases, when the information entropy of $\\textbf{Z}$ is fixed, maximizing $I(\\textbf{Z};\\textbf{Z}^{(1)}, \\textbf{Z}^{(2)})$ will increase the information entropy of $\\textbf{Z}^{(1)}$ and $\\textbf{Z}^{(2)}$. When the information entropy of $\\textbf{Z}^{(1)}$, and $\\textbf{Z}^{(2)}$ is fixed, maximizing $I(\\textbf{Z};\\textbf{Z}^{(1)}, \\textbf{Z}^{(2)})$  will reduce the information entropy of $\\textbf{Z}$. However, both cases are learning cross-view diversity."}, {"review_id": "zaMqESBOHl", "reviewer": "Reviewer_YKQa", "summary": "This paper presents an innovative information-theoretic framework for multi-view clustering, which overcomes the limitations of existing methods that rely on strict semantic consistency assumptions. By leveraging deep neural networks, the proposed method achieves more stable and superior clustering performance on several datasets.", "questions": "See weaknesses.", "limitations": "The authors have discussed the limitations of this paper.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The idea of incorporating the information bottleneck into multi-view clustering is intriguing and enlightening.", "weaknesses": "1. The importance of Eq. 3 in the proposed method is evident, as it provides the IB-based objective for the clustering approach. However, the paper lacks an explanation of how the original IB objective (Eq. 1) is transformed into the clustering objective, and why it takes the specific form presented. Additionally, if the information bottleneck based clustering is not an original contribution of this paper, it should be properly referenced.\n2. The concept of the three desiderata is unclear. From Fig. 1, it appears that the final result of optimizing the three desiderata is to maximize I(Z; X^{1,2}), minimize I(Z; X{1,2}), and maximize I(Z; Z^{1,2}). This optimization seems contradictory, particularly when it aims to maximize the mutual information between Z and view-specific Z^{1,2} while minimizing that between raw data X and Z^{1,2}. Furthermore, the paper lacks an explanation of why these three desiderata are useful for learning cross-view representation, and there are no ablation studies to investigate their influence.\n3. The proposed generalized multi-view clustering framework seems to differ from Eq. 3 only in the information shift term. This should be clarified and elaborated upon.\n4. The experiments conducted on relatively simple and small-scale datasets limit the persuasiveness of the results. Additionally, the comparison baselines appear outdated, raising concerns about the effectiveness of the proposed method.\n5. There are some typos, such as line 112: \"I (Z^{(1)}; X^{(2)}).\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful and detailed feedback.\n\n**A1:** Under the information bottleneck principle, [1] introduced the unsupervised information bottleneck objective Eq.(2), which is essentially the same as $\\beta$VAE [2]. By summarizing the deep clustering methods DEC [3] and VaDE [4], we introduce the KL divergence term on the basis of Eq.(2) to discover the cluster structure of the data. The extent to which the clustering structure of the data is maintained while reducing dimensionality is the gap between Eq.(3) and Eq.(2).\n\n[1] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In *ICLR*, 2017.\n\n[2] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In *ICLR*, 2017.\n\n[3] Xie, Junyuan, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering. In *ICML*, pages 478-487, 2016.\n\n[4] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In *IJCAI*, 2017.\n\n**A2:** Actually, the optimization is a trade-off. Like the unsupervised information bottleneck theory ($\\beta$VAE), it is desired to retain information about the data distribution ($\\max I(\\textbf{Z}; \\textbf{X})$) and forget the information for identity samples ($\\min I(\\textbf{Z}; i)$). For the multi-view scenario, we seek to retain the multi-view data distribution information ($\\max I(\\textbf{Z}; \\textbf{X}^{(1)},\\textbf{X}^{(2)})$), and want to forget the single-view information ($\\min I(\\textbf{Z}^{(v)}, \\textbf{X}^{(v)})$), and in order to match $\\textbf{Z}$ and $\\textbf{Z}^{(v)}$, we propose $\\max I(\\textbf{Z}; \\textbf{Z}^{(1)}, \\textbf{Z}^{(2)})$ as a trade-off glue (related proof in Proposition 3.1). There are two cases, when the information entropy of $\\textbf{Z}$ is fixed, maximizing $I(\\textbf{Z};\\textbf{Z}^{(1)}, \\textbf{Z}^{(2)})$ will increase the information entropy of $\\textbf{Z}^{(1)}$ and $\\textbf{Z}^{(2)}$. When the information entropy of $\\textbf{Z}^{(1)}$, and $\\textbf{Z}^{(2)}$ is fixed, maximizing $I(\\textbf{Z};\\textbf{Z}^{(1)}, \\textbf{Z}^{(2)})$  will reduce the information entropy of $\\textbf{Z}$. However, both cases are learning cross-view diversity.\n\n**For ablation study**, in line 242 of Section 5.1 Experimental Settings, we construct two variants IMC-v1 and IMC-v2 to compare with IMC, and showed the experimental results in Table 1 and Table 2. The performance of using KL divergence clustering term has been significantly improved, while the use of information shift term has a small improvement. And in Section 5.2 Performance Analysis, we concluded that end-to-end models help to learn more discriminative representations.\n\n\n**A3:** Eq. (3) is clustering based on single-view information bottleneck, while the proposed information-theoretic multi-view clustering (IMC) is for multi-view data. The maximum information shift term is used to connect the unified multi-view representation $\\textbf{Z}$ and the view-specific representation $\\textbf{Z}^{(v)}$ to achieve cross-diversity.\n\n\n**A4:** Following your suggestion, we further test our model on a large-scale multi-view dataset.\n\n**NUS-WIDE-Object (NUS)** is a dataset for object recognition that consists of 30000 images in 31 classes. We use 5 features provided by the web-site, i.e. 65 dimension color Histogram (CH), 226 dimension color moments (CM), 145 dimension color correlation (CORR), 74 dimension edge distribution, and 129 wavelet texture.\n\n|  Datasets  |  Metrics  |  DMVAE | MIB | CMIB-Nets | Completer | IMC-v1 | IMC-v2 | IMC |\n|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |\n|  | ACC |   18.24$\\pm$0.64 |   15.24$\\pm$0.56 |   14.25$\\pm$0.35 |   16.88$\\pm$0.46 |   17.60$\\pm$0.40 |   14.38$\\pm$0.36 |   $\\textbf{19.78} \\pm \\textbf{0.47}$ |  \n| NUS  | NMI |  19.87$\\pm$1.05 |   15.75$\\pm$0.42 |   14.65$\\pm$0.40 |  17.68$\\pm$0.35 |   18.67$\\pm$0.62 |   15.55$\\pm$0.74 |   $\\textbf{21.12} \\pm \\textbf{0.34}$ |  \n|  | ARI |  6.78$\\pm$0.81 |   5.83$\\pm$0.84 |  4.26$\\pm$0.74 |   6.24$\\pm$0.64 |   5.68$\\pm$0.48 |  5.68$\\pm$0.48 |   $\\textbf{8.32} \\pm \\textbf{0.41}$ |  \n|\n\nFrom the results, it can be seen that NUS is a challenging dataset, and there is still much room for improvement in the clustering performance of all the models The optimal and sub-optimal performance achieved by the proposed IMC and DMVAE model demonstrates that incorporating the clustering loss to bootstrap the representation learning can significantly improve the clustering performance, which can be consistently concluded from the ablation experiments comparing IMC, IMC-v1, and IMC-v2 .\n\n**A5:** I am sorry for the serious typos, here should be to minimize $I(\\textbf{Z}^{(1)}; \\textbf{X}^{(1)})$ and $I(\\textbf{Z}^{(2)}; \\textbf{X}^{(2)})$, for maximum information compression of each view."}, {"review_id": "XVhJnFELy9", "reviewer": "Reviewer_bG4n", "summary": "This paper reformulates the multi-view clustering problem from an information-theoretic perspective and propose a general theoretical framework. The authors extend the information bottleneck theory to unsupervised multi-view learning and achieve representation learning and clustering by leveraging deep neural networks and stochastic gradient variational Bayes.", "questions": "1.in line 113, should the $I(Z^{(1)};X^{(2)})$ be $I(Z^{(2)};X^{(2)})$?", "limitations": "As the authors state in the summary section, the model is limited by the choice of parameters, and it would be a great improvement if a mathematical strategy could be found.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper combines information theory with multi-view clustering and gives some new definitions to portray some properties of the multi-view domain, which is a very innovative idea that can contribute to the field.\n2. This paper provides a solid theory and some key proofs are detailed and complete.\n", "weaknesses": "1. The datasets used in the experimental part are a bit less and small, and there are many challenging and large datasets in the field of multi-view clustering, the authors should add more experiments to enhance the sufficiency.\n2. Parameter analysis part should show parameter changes on all datasets, you may show the figures of all datasets under one metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your compliment and constructive suggestion.\n\n**Q1:** The datasets used in the experimental part are a bit less and small, and there are many challenging and large datasets in the field of multi-view clustering, the authors should add more experiments to enhance the sufficiency.\n\n**A1:** Following your suggestion, we further test our model on a large-scale multi-view dataset.\n\n**NUS-WIDE-Object (NUS)** is a dataset for object recognition that consists of 30000 images in 31 classes. We use 5 features provided by the website, i.e. 65 dimension color Histogram (CH), 226 dimension color moments (CM), 145 dimension color correlation (CORR), 74 dimension edge distribution, and 129 wavelet texture.\n\n|  Datasets  |  Metrics  |  DMVAE | MIB | CMIB-Nets | Completer | IMC-v1 | IMC-v2 | IMC |\n|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |\n|  | ACC |   18.24$\\pm$0.64 |   15.24$\\pm$0.56 |   14.25$\\pm$0.35 |   16.88$\\pm$0.46 |   17.60$\\pm$0.40 |   14.38$\\pm$0.36 |   $\\textbf{19.78} \\pm \\textbf{0.47}$ |  \n| NUS  | NMI |  19.87$\\pm$1.05 |   15.75$\\pm$0.42 |   14.65$\\pm$0.40 |  17.68$\\pm$0.35 |   18.67$\\pm$0.62 |   15.55$\\pm$0.74 |   $\\textbf{21.12} \\pm \\textbf{0.34}$ |  \n|  | ARI |  6.78$\\pm$0.81 |   5.83$\\pm$0.84 |  4.26$\\pm$0.74 |   6.24$\\pm$0.64 |   5.68$\\pm$0.48 |  5.68$\\pm$0.48 |   $\\textbf{8.32} \\pm \\textbf{0.41}$ |  \n|\n\nFrom the results, it can be seen that NUS is a challenging dataset, and there is still much room for improvement in the clustering performance of all the models The optimal and sub-optimal performance achieved by the proposed IMC and DMVAE model demonstrates that incorporating the clustering loss to bootstrap the representation learning can significantly improve the clustering performance, which can be consistently concluded from the ablation experiments comparing IMC, IMC-v1, and IMC-v2.\n\n**Q2:** Parameter analysis part should show parameter changes on all datasets, you may show the figures of all datasets under one metric.\n\n**A2:** This suggestion is very constructive and I will post the parametric analysis of the NMI indicator for all datasets in the revised version, the results for the ACC and ARI indicators will be presented in the supplementary material.\n\n**Q3:** in line 113, should the $I(\\textbf{Z}^{(1)}; \\textbf{X}^{(2)})$ be $I(\\textbf{Z}^{(2)}; \\textbf{X}^{(2)})$?\n\n**A3:** I am sorry for the serious typos, here should be to minimize$I(\\textbf{Z}^{(1)}; \\textbf{X}^{(1)})$ and  $I(\\textbf{Z}^{(2)}; \\textbf{X}^{(2)})$, for maximum information compression of each view."}, {"review_id": "89Y52e2Va5", "reviewer": "Reviewer_Ayjx", "summary": "This paper proposes a new framework for unsupervised multi-view learning based on information bottleneck theory. The paper defines three desiderata for multi-view representation learning in terms of mutual information, namely, comprehensiveness, concentrate, and cross-diversity. The paper further introduces a clustering term to preserve the original data structure and leverages deep neural networks and stochastic gradient variational Bayes to optimize the objective function. The paper evaluates the proposed method on four real-world datasets and shows that it outperforms several state-of-the-art algorithms in terms of clustering performance.", "questions": "The authors are encouraged to address the weaknesses pointed out in the previous section, such as conducting ablation studies. These improvements would make the contribution of the paper more evident and might increase my rating.", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "•\tThis paper provides a general and principled information-theoretic framework for multi-view clustering that does not rely on strict assumptions about semantic consistency across views.\n\n•\tIt incorporates three requirements for multi-view representation learning that balance the trade-off between informativeness, compression, and diversity of the latent features.\n", "weaknesses": "•\tThe authors should clarify how their definition of comprehensive, concentrative, and cross-diverse multi-view representation differs from the one used by Completer [22], which also maximizes the mutual information between views and minimizes conditional entropy of different views. \n\n•\tThe loss function consists of four terms with different roles: data reconstruction, multi-regularization, information shift, and clustering. The authors should conduct ablation studies to show the contribution and necessity of each term for the proposed method.\n\n•\tThe balance parameters $\\beta$ and $\\gamma$ control the trade-off in the objective function. The authors should provide some theoretical or empirical guidance for choosing the optimal values of these parameters for different datasets or scenarios.\n\n•\tAs the robustness of the model is important to evaluate, what is the definition of robustness in this paper and the relation between the robustness and the proposed information bottleneck theory?\n\n•\tIn Eq.(4), what is the difference between $\\mathbf{Z}^{(v)}$ and $\\mathbf{Z}$? How to learn $\\mathbf{Z}$?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our work and constructive comments.\n\n**Q1:** The authors should clarify how their definition of comprehensive, concentrative, and cross-diverse multi-view representation differs from the one used by Completer [22], which also maximizes the mutual information between views and minimizes conditional entropy of different views.\n\n**A1:** Completer maximizes the mutual information between views and minimizes the conditional entropy across view to learn the shared information between views, thus forcing the latent representations of different views to be consistent. The shared information is assumed to contain all task-related information from the data. However, this is not a reasonable assumption in practice, e.g., different views share the same background information but depict different foreground information.\n\nIn contrast, the proposed model (IMC) is based on a general assumption that task-related information exists both in the view-shared part and in the view-peculiar part. Under the guidance of the information bottleneck principle, we seek to retain more intrinsic information (comprehensive) and compress each view as much as possible (concentrative), so we need to make each view contain various information (cross-diverse), respectively corresponding to three different mutual information. This is the motivation for proposing IMC, which is different from Completer.\n\n**Q2:** The loss function consists of four terms with different roles: data reconstruction, multi-regularization, information shift, and clustering. The authors should conduct ablation studies to show the contribution and necessity of each term for the proposed method.\n\n**A2:** I'm sorry that I did the ablation experiment and analysis, but I didn't set up a chapter on ablation study in the manuscript, which made it easy to be ignored. We will highlight the ablation study more clearly in the new version.\n\nIn line 242 of Section 5.1 Experimental Settings, we construct two variants IMC-v1 and IMC-v2 to compare with IMC, and showed the experimental results in Table 1 and Table 2. The performance of using KL divergence clustering term has been significantly improved, while the use of the information shift term has a small improvement. And in Section 5.2 Performance Analysis, we concluded that end-to-end models help to learn more discriminative representations.\n\n**Q3:** The balance parameters and control the trade-off in the objective function. The authors should provide some theoretical or empirical guidance for choosing the optimal values of these parameters for different datasets or scenarios.\n\n**A3:** You point out one of the main limitation of this approach, we claim in chapter 5.4 that empirical guidance can be based on prior knowledge of the dataset. In our experiences, if there is a large degree of consistency in the semantics of multiple views, then the β is set to small and vice versa. As we mentioned in Chapter 6 there are practical limitations on parameter selection. Theoretical guidance for choosing the optimal parameters is a direction that can be researched in the future.\n\n**Q4:** As the robustness of the model is important to evaluate, what is the definition of robustness in this paper and the relation between the robustness and the proposed information bottleneck theory?\n\n**A4:** We believe that the robustness of the model is its ability to adapt to data generated in various complex scenarios, including noisy data. To evaluate the robustness of the proposed method, we added sparse noise of different scales to the dataset and the clustering results are shown in Table 2. It can be observed that the decline of IMC is the smallest, while the decline of the Completer model is obvious. This may be due to the assumption of multi-view semantic consistency is more sensitive as the noise ratio increases, the information shared by multiple views will also incorporate more noise.\n\nWe are looking forward to adding other indicators to measure model robustness. If you have any suggestions on robustness metrics, it will greatly improve the quality of our work.\n\n**Q5:**  In Eq.(4), what is the difference between $\\textbf{Z}^{(v)}$ and $\\textbf{Z}$? How to learn $\\textbf{Z}$?\n\n**A5:** $\\textbf{Z}$ is the unified latent representation and$\\textbf{Z}^{(v)}$ is the latent representation of the $v$-th view. We seek to learn a uniform multi-view representation that satisfies three desiderata, so we introduce the representations for each view. $\\textbf{Z}$ can be obtained by the posterior $p(\\textbf{z}|\\textbf{z}^{(1)}, \\textbf{z}^{(2)}, …, \\textbf{z}^{(v)})$ which is modeled by a multiple fully connected layer (line 166). Intuitive understanding can be seen in Figure 2(c)."}]

Author rebuttal:

Meta-review: This paper proposes a general multi-view clustering framework based on the information bottleneck principle [9], and put the framework into effect via utilizing deep neural networks and stochastic gradient variational Bayes. After the author rebuttal, the majority of the reviews agree that the paper is contributive and suggest acceptance. Reviewer i2fg acknowledges the author rebuttal that has partially addressed their concerns, but continues complaining the novelty of the proposed theories. Based on the author-review communications, the review-review discussions, and my own read, my assertion is that the technical novelty is significant enough, though the presented framework and theories are not completely new. Thus I would suggest accepting the paper.
Decision: Accept (poster)