SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weitian Huang
Guangzhou, 510006, China
Sirui Yang
Hongmin Cai*
In an era of more diverse data modalities, multi-view clustering has become a fundamental tool for comprehensive data analysis and exploration. However, existing multi-view unsupervised learning methods often rely on strict assumptions on semantic consistency among samples. In this paper, we reformulate the multi-view clustering problem from an information-theoretic perspective and propose a general theoretical model. In particular, we define three desiderata under multi-view unsupervised learning in terms of mutual information, namely, comprehensiveness, concentration, and cross-diversity. The multi-view variational lower bound is then obtained by approximating the samples' high-dimensional mutual information. The Kullback-Leibler divergence is utilized to deduce sample assignments. Ultimately the information-based multi-view clustering model leverages deep neural networks and Stochastic Gradient Variational Bayes to achieve representation learning and clustering simultaneously. Extensive experiments on both synthetic and real datasets with wide types demonstrate that the proposed method exhibits more stable and superior clustering performance than state-of-the-art algorithms.
Multiple views of data are often collected by different measurement methods to represent various properties of objects in many practical applications. For example, for images or videos, various visual features such as HOG [1], SIFT [2], and LBP [3] can be acquired by traditional filters and form a kind of multi-view data. These features can be collected from different domains or generated from various sensors. In general, different views can complement each other due to limitations or biases in measurement methods that would result in insufficient information contained in individual views [4].
The ever-increasing network activity brings massive unlabeled data, which makes clustering, the traditional task of machine learning, one of the hot topics in unsupervised learning. Importantly, in clustering task, learning the comprehensive and discriminative representations is crucial for partitioning the samples. Compared to single-view clustering, multi-view clustering (MVC) is widely studied as it exploits complementary information between different views to improve the ultimate performance. Most multi-view clustering algorithms learn the consistent representation among multiple views to realize clustering. The benchmark techniques include view co-regularization [5],
*Hongmin Cai is the Corresponding Author, whose E-mail is: hmcai@scut.edu.cn.
Canonical Correlation Analysis (CCA) [6, 7] and low-rank tensor decomposition [8]. Although different methods have achieved superior clustering performance, they lack a unified theoretical framework to help us further understand multi-view clustering.
Recently, Information Bottleneck (IB) [9] have attracted increasing attention in deep neural network research. The information bottleneck is an information theory principle that helps explain the puzzling success of today's artificial intelligence algorithms. This motivates us to explain multi-view learning using information bottleneck theory. In this paper, we extend from supervised information bottleneck theory to unsupervised learning, and construct a general information-theoretic multi-view clustering framework. The main contributions of our work are illustrated as follows,
- In unsupervised scheme, we define three requirements for multi-view representation learning from the view of information theory, comprehensiveness, concentration, and cross-diversity. Consequently, a general information-based multi-view clustering model (IMC) is proposed to achieve representation learning and clustering simultaneously.
- We present a multi-view VAE scheme to solve IMC by leveraging deep neural networks and Stochastic Gradient Variational Bayes (SGVB). Specifically, the multi-view mutual information is approximated by a variational lower bound, which can be expressed as a unified multi-view representation obtained through multiple variational autoencoders and sampling, followed by a self-training strategy for clustering.
- Experimental results show that the proposed method is effective in learning informative yet minimal representations and exhibits satisfactory generalization ability.
2 Variants of Information Bottleneck
2.1 Information bottleneck
The information bottleneck principle [9] aims to obtain the most compressed representation of data while retaining task-relevant information. To achieve this, for given input data  $\mathbf{x}$  with label  $\mathbf{y}$ , the corresponding codeword  $\mathbf{z}$  decided by  $p(\mathbf{z}|\mathbf{x})$  can be obtained by optimizing the objective of the information bottleneck,
$$
\min  _ {p (\mathbf {z} | \mathbf {x}): I (\mathbf {Z}; \mathbf {Y}) \geq I ^ {*}} I (\mathbf {Z}; \mathbf {X}) = \max  _ {\mathbf {Z}} I (\mathbf {Z}; \mathbf {Y}) - \beta I (\mathbf {Z}; \mathbf {X}) := \mathcal {L} _ {I B}. \tag {1}
$$
where  $I(\mathbf{Z};\mathbf{X})$  represents the mutual information between two random variables  $\mathbf{X}$  and  $\mathbf{Z}$ , which implicitly measures the level of compression in the representation  $\mathbf{Z}$ .  $\mathcal{L}_{IB}$  is to identify the optimal encoding  $p(\mathbf{z}|\mathbf{x})$  that minimizes  $I(\mathbf{Z};\mathbf{X})$  while ensuring  $I(\mathbf{Z};\mathbf{Y})$  exceeds a minimal threshold value  $I^{*}$ . This constrained problem can then be transformed into a non-constraint problem by incorporating a trade-off factor  $\beta$  that balances  $I(\mathbf{Z};\mathbf{Y})$  and  $I(\mathbf{Z};\mathbf{X})$ . Such scheme yields sufficient and minimal representations that maximize task-relevant information while minimizing task-irrelevant information, thus balancing prediction and generalization performance.
2.2 Unsupervised information bottleneck
When the sample labels are unavailable, our aim is to compress each sample's description to the greatest extent possible while simultaneously maximizing the retention of the principal component information of the data. The unsupervised information bottleneck objective is described in [10],
$$
\max _ {\mathbf {Z}} I (\mathbf {Z}; \mathbf {X}) - \beta I (\mathbf {Z}; i) \stackrel {(a)} {\geq} \mathbb {E} _ {p (\mathbf {x})} \left[ \mathbb {E} _ {p (\mathbf {z} | \mathbf {x})} \left[ \log q (\mathbf {x} | \mathbf {z}) \right] - \beta D _ {K L} (p (\mathbf {z} | \mathbf {x}) | | q (\mathbf {z})) \right] := \mathcal {L} _ {U I B}. \quad (2)
$$
where  $(a)$  forms the lower bound of  $\mathcal{L}_{UIB}$ , which is achieved by substituting the intractable posterior  $p(\mathbf{x}|\mathbf{z})$  and marginal  $p(\mathbf{z})$  with respective variational posterior  $q(\mathbf{x}|\mathbf{z})$  and marginal  $q(\mathbf{z})$ . Note that the lower bound is essentially identical to  $\beta$  VAE [11]. The detailed derivation process is shown in Appendix A.
In clustering tasks, one aims to discover the underlying structure of data and divide it into distinct clusters. The current deep clustering methods firstly extract deep low-dimensional embeddings and then apply standard clustering techniques. Theoretically, the deep clustering method is expected to
follow the information bottleneck principle while preserving the original data clustering structure. Therefore, one can tailor Eq. (2) to realize deep clustering,
$$
\begin{array}{l} \max _ {\mathbf {Z}} I (\mathbf {Z}; \mathbf {X}) - \beta I (\mathbf {Z}; i) \\ \geq \mathbb {E} _ {p (\mathbf {x})} \left[ \mathbb {E} _ {p (\mathbf {z} | \mathbf {x})} \left[ \log q (\mathbf {x} | \mathbf {z}) \right] - \beta D _ {K L} \left(p (\mathbf {z} | \mathbf {x}) | | q (\mathbf {z})\right) - \gamma \mathbb {E} _ {p (\mathbf {z} | \mathbf {x})} \left[ D _ {K L} \left(p (\mathbf {c} | \mathbf {x}) | | q (\mathbf {c} | \mathbf {z})\right) \right] \right] \\ := \mathcal {L} _ {I B C} = \mathcal {L} _ {U I B} - \gamma \mathbb {E} _ {p (\mathbf {x})} \left[ \mathbb {E} _ {p (\mathbf {z} | \mathbf {x})} [ D _ {K L} (p (\mathbf {c} | \mathbf {x}) | | q (\mathbf {c} | \mathbf {z})) ] \right]. \tag {3} \\ \end{array}
$$
Here we introduce a discrete variable  $c$  to represent the cluster and denote  $p(c|\mathbf{x})$  as the cluster distribution implied by the data. We aim for the cluster inferred by embedding  $\mathbf{z}$  to be close to the original cluster distribution. To measure this, we add KL divergence term with the balance factor  $\gamma$ . It is worth noting that the gap between  $\mathcal{L}_{IBC}$  and  $\mathcal{L}_{UIB}$  lies in the preservation of the clustering structure in the process of information compression.
Connection to DEC. Deep Embedding for Clustering (DEC) [12] begins with pre-training an autoencoder to learn low-dimensional representations, and the soft cluster assignment computed by the embeddings (as  $q(c|\mathbf{z})$  in Eq. (3)) is determined by the Student's  $t$ -distribution. The clustering loss is designed via KL divergence to approach the auxiliary target distribution (as  $p(c|\mathbf{x})$  in Eq. (3)). By setting  $\beta = 0$  and  $\gamma = 1$ , and using the pre-training strategy, we obtain the DEC model.
Connection to VaDE. Variational Deep Embedding (VaDE) [13] combines VAEs and Gaussian mixture models and encodes the initial data distribution as a learned Gaussian mixture distribution in the latent variable space. By setting  $\beta = 1$  and  $\gamma = 1$ , we get the VaDE model.
3 Information-based Multi-view Clustering
Let  $\mathcal{X} = \{\mathbf{X}^{(v)}\}_{v=1}^m$  be a dataset consisting of  $m$  views, where each view collection comprises  $n$  independent and identically distributed (i.i.d) samples,  $\mathbf{X}^{(v)} = \{\mathbf{x}_1^{(1)}, \mathbf{x}_2^{(2)}, \dots, \mathbf{x}_n^{(v)}\}$ . Consider  $\mathcal{Z} = \{\mathbf{Z}^{(v)}\}_{v=1}^m$  presenting view-peculiar compressed representations for  $\mathcal{X}$  and  $c \in \{1, \dots, K\}$  designating a categorical random variable and represent the index of the actual cluster. For multi-view clustering, we first learn a unified latent representation  $\mathbf{Z}$  across all views, which satisfies comprehensiveness, concentration and cross-diversity. Additionally, the multi-view representation is needed to preserve the original data structure information related to clusters followed by performing clustering.
3.1 Unified multi-view representation learning
We first define the desiderata for multi-view representation as follows.
Definition 3.1 (Comprehensive, concentrative and cross-diverse multi-view representation). Comprehensiveness. Given the multi-view observations  $\mathcal{X}$ , a representation,  $\mathbf{Z}$ , is comprehensive if each view observation can be predicted by  $\mathbf{Z}$
$$
\mathbf {Z} ^ {*} = \arg \max  _ {\mathbf {Z}} I (\mathbf {Z}; \boldsymbol {\chi}). \tag {4}
$$
Concentration. For each view data  $\mathbf{X}^{(v)}$ , their view-peculiar compressed representation  $\mathbf{Z}^{(v)}$  is concentrative if the redundant information for each view is eliminated,
$$
\mathbf {Z} ^ {(v) *} = \arg \min  _ {\mathbf {Z} ^ {(v)}} I \left(\mathbf {Z} ^ {(v)}; \mathbf {X} ^ {(v)}\right). \tag {5}
$$
Cross-diversity. View-peculiar latent representations  $\mathcal{Z}$  can be cross-diverse if they avoid duplication of information within the view,
$$
\boldsymbol {\mathcal {Z}} ^ {*} = \arg \max  _ {\boldsymbol {\mathcal {Z}}} I (\boldsymbol {\mathcal {Z}}; \boldsymbol {\mathcal {Z}}). \tag {6}
$$
To more intuitively grasp the definition of multi-view representations, we depict the three requirements for representation learning from the perspective of information theory. As illustrated in Fig 1, taking dual-view data as an example, the information carried by  $\mathbf{X}^{(1)}$  and  $\mathbf{X}^{(2)}$  are  $H(\mathbf{X}^{(1)})$  and  $H(\mathbf{X}^{(2)})$  respectively, which are fixed. To effectively convey the combination of the two views, the unified representation  $\mathbf{Z}$  achieves informativeness by maximizing the mutual information between  $\mathbf{Z}$  and
(a) Comprehensiveness
(b) Concentration
(c) Cross-diversity
Figure 1: Illustration of three desiderata for learning multi-view representations.
$\mathbf{X}^{(1)}\mathbf{X}^{(2)}$ . Additionally, to remove redundant information from each view, we minimize  $I(\mathbf{Z}^{(1)};\mathbf{X}^{(1)})$  and  $I(\mathbf{Z}^{(2)};\mathbf{X}^{(2)})$  respectively for maximum compression of the respective view information. Ideally, semantically irrelevant information specific to each view is removed while complete semantically relevant information is retained.
To balance comprehensiveness and concentration, we introduce mutual information between  $\mathbf{Z}$  and  $\mathbf{Z}^{(1)}\mathbf{Z}^{(2)}$  as a cross-diversity indicator,
$$
I (\mathbf {Z}; \mathbf {Z} ^ {(1)}, \mathbf {Z} ^ {(2)}) = I (\mathbf {Z}; \mathbf {Z} ^ {(1)}) + I (\mathbf {Z}; \mathbf {Z} ^ {(2)}) - \left[ I (\mathbf {Z} ^ {(1)}; \mathbf {Z} ^ {(2)}) - I (\mathbf {Z} ^ {(1)}; \mathbf {Z} ^ {(2)} | \mathbf {Z}) \right], \tag {7}
$$
where we denote  $D_{\mathbf{Z}}(\mathbf{Z}^{(1)};\mathbf{Z}^{(2)}) = I(\mathbf{Z}^{(1)};\mathbf{Z}^{(2)}) - I(\mathbf{Z}^{(1)};\mathbf{Z}^{(2)}|\mathbf{Z})$ , representing the duplicate information between views. As showed in Fig 1 (c), one can approach cross-diversity by maximizing  $I(\mathbf{Z};\mathbf{Z}^{(1)})$ ,  $I(\mathbf{Z};\mathbf{Z}^{(2)})$  and minimizing  $D_{\mathbf{Z}}(\mathbf{Z}^{(1)};\mathbf{Z}^{(2)})$ .
Proposition 3.1 (Cross-diversity is sufficient for comprehensiveness and concentration, respectively). Given two Markov chains  $\mathbf{X}^{(1)}\rightarrow \mathbf{Z}^{(1)}\rightarrow \mathbf{Z}$  and  $\mathbf{X}^{(2)}\rightarrow \mathbf{Z}^{(2)}\rightarrow \mathbf{Z}$ , maximization of  $I(\mathbf{Z};\mathbf{Z}^{(1)},\mathbf{Z}^{(2)})$  is sufficient for maximization of  $I(\mathbf{Z};\mathbf{X}^{(1)},\mathbf{X}^{(2)})$  when  $\mathbf{Z}^{(1)},\mathbf{Z}^{(2)}$  are fixed and minimization of  $\sum_{v = 1}^{2}I(\mathbf{Z}^{(v)};\mathbf{X}^{(v)})$  when  $\mathbf{Z}$  is fixed, respectively.
Proof. Under the two Markov chains accompanied by loss of information, we have,
$$
I (\mathbf {Z}; \mathbf {X} ^ {(1)}, \mathbf {X} ^ {(2)}) \stackrel {(b)} {\leq} I (\mathbf {Z} ^ {(1)}; \mathbf {X} ^ {(1)}) + I (\mathbf {Z} ^ {(2)}; \mathbf {X} ^ {(2)}),
$$
Next we rewrite the cross-diversity indicator,
$$
\begin{array}{l} I (\mathbf {Z}; \mathbf {Z} ^ {(1)}, \mathbf {Z} ^ {(2)}) = H (\mathbf {Z} ^ {(1)}, \mathbf {Z} ^ {(2)}) - H (\mathbf {Z} ^ {(1)}, \mathbf {Z} ^ {(2)} | \mathbf {Z}) \\ = H (\mathbf {Z} ^ {(1)}) + H (\mathbf {Z} ^ {(2)}) - I (\mathbf {Z} ^ {(1)}; \mathbf {Z} ^ {(2)}) - H (\mathbf {Z} ^ {(1)}, \mathbf {Z} ^ {(2)} | \mathbf {Z}) \\ \leq H (\mathbf {Z} ^ {(1)}) + H (\mathbf {Z} ^ {(2)}), \\ \end{array}
$$
Since  $I(\mathbf{Z};\mathbf{Z}^{(1)},\mathbf{Z}^{(2)})$  achieves the maximum value, i.e.  $I(\mathbf{Z};\mathbf{Z}^{(1)},\mathbf{Z}^{(2)}) = H(\mathbf{Z}^{(1)}) + H(\mathbf{Z}^{(2)})$ . We denote  $I(\mathbf{Z};\mathbf{Z}^{(1)},\mathbf{Z}^{(2)})$  as  $\mathbf{T}$ ,
$$
\begin{array}{l} I (\mathbf {Z}; \mathbf {X} ^ {(1)}, \mathbf {X} ^ {(2)}) = I (\mathbf {T}; \mathbf {X} ^ {(1)}, \mathbf {X} ^ {(2)}) \\ = H (\mathbf {T}) - H (\mathbf {T} | \mathbf {X} ^ {(1)}, \mathbf {X} ^ {(2)}) \\ \stackrel {(c)} {\geq} H (\mathbf {Z} ^ {(1)}) + H (\mathbf {Z} ^ {(2)}) - H (\mathbf {Z} ^ {(1)}, \mathbf {Z} ^ {(2)} | \mathbf {X} ^ {(1)}, \mathbf {X} ^ {(2)}) \\ = H (\mathbf {Z} ^ {(1)}) - H (\mathbf {Z} ^ {(1)} | \mathbf {X} ^ {(1)}) + H (\mathbf {Z} ^ {(2)}) - H (\mathbf {Z} ^ {(2)} | \mathbf {X} ^ {(2)}) \\ = I \left(\mathbf {Z} ^ {(1)}; \mathbf {X} ^ {(1)}\right) + I \left(\mathbf {Z} ^ {(2)}; \mathbf {X} ^ {(2)}\right), \\ \end{array}
$$
(a) Special case
(b) General case
Figure 2: (a) Cluster information is only related to common information between multiple views. (b) View-peculiar information also includes semantic features. (c) Multi-view variational autoencoders and a fusion network are incorporated to perform representation learning and clustering simultaneously.
(c) The architecture of dual-view instance of IMC
To satisfy both inequalities (b) and (c), the equation holds, i.e.,  $I(\mathbf{Z}; \mathbf{X}^{(1)}, \mathbf{X}^{(2)}) = I(\mathbf{Z}^{(1)}; \mathbf{X}^{(1)}) + I(\mathbf{Z}^{(2)}; \mathbf{X}^{(2)})$ . Therefore, when  $I(\mathbf{Z}; \mathbf{Z}^{(1)}, \mathbf{Z}^{(2)})$  is maximized, fixing  $\mathbf{Z}^{(1)}, \mathbf{Z}^{(2)}, I(\mathbf{Z}; \mathbf{X}^{(1)}, \mathbf{X}^{(2)})$  maximization holds, and conversely, fixing  $\mathbf{Z}, I(\mathbf{Z}^{(1)}; \mathbf{X}^{(1)}) + I(\mathbf{Z}^{(2)}; \mathbf{X}^{(2)})$  minimization holds.
With these desiderata, the unified multi-view representation learning can be formulated as follows,
$$
\begin{array}{l} \max  _ {\mathbf {Z}, \boldsymbol {\bar {z}}} I (\mathbf {Z}; \boldsymbol {\mathcal {X}}) - \sum_ {v} ^ {m} I \left(\mathbf {Z} ^ {(v)}; \mathbf {X} ^ {(v)}\right) + \beta I (\mathbf {Z}; \boldsymbol {\mathcal {Z}}) \\ \stackrel {(d)} {\geq} \mathbb {E} _ {p (\boldsymbol {x})} \left[ \mathbb {E} _ {p (\mathbf {z} | \boldsymbol {x})} [ \log q (\boldsymbol {x} | \mathbf {z}) ] - \sum_ {v} ^ {m} D _ {K L} (p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)}) | | q (\mathbf {z} ^ {(v)})) \right] + \beta I (\mathbf {Z}; \boldsymbol {\mathcal {Z}}), \tag {8} \\ \end{array}
$$
where  $(d)$  leverages the variational technique to construct the lower bound of the first two terms (deviation can be found in Appendix B), the  $\beta$  is the balance coefficient that controls the cross-diversity of representation and balances the comprehensiveness and concentration. Mutual information between  $\mathbf{Z}$  and multiple representations  $\mathbf{Z}^{(1)},\dots,\mathbf{Z}^{(m)}$  can further be expressed by,
$$
I (\mathbf {Z}; \boldsymbol {\mathcal {Z}}) = \sum_ {v = 1} ^ {m} I (\mathbf {Z}; \mathbf {Z} ^ {(v)}) - \sum_ {v = 1} ^ {m} \sum_ {t \neq v} ^ {m} D _ {\mathbf {Z}} \left(\mathbf {Z} ^ {(v)}, \mathbf {Z} ^ {(t)}\right). \tag {9}
$$
3.2 Generalized multi-view clustering framework
For the multi-view clustering task, previous works [14, 15] considered category-related or semantic-related information to be relevant only to the multi-view consensus part. As shown in Fig. 2 (a), the part where two views intersect contains all the information related to cluster  $c$  in the multi-view data. However, this is only a special case. More generally, the respective views contain specific semantic information that can complement each other. This is why a comprehensive understanding of an object or event requires different views. As shown in Fig. 2 (b), we cannot determine if the view-common or view-peculiar parts of the views contain more semantic information, we can only assume that they all contain this information.
Next, we seek to learn the unified multi-view representation without losing the original clustering structure of the data. For examples, we assume that the data obeys a Gaussian mixture distribution, or that the data has a tight intra-class structure and a relaxed inter-class structure. We add a clustering term to Equation (8) for measuring the distance between the posterior distribution of a given original
data to clusters and the posterior distribution of a given multi-view representation to clusters. Then, the objective of information-based multi-view clustering can be formulated as,
$$
\begin{array}{l} \mathcal {L} _ {I M C} = \mathbb {E} _ {p (\boldsymbol {x})} \left[ \underbrace {\mathbb {E} _ {p (\boldsymbol {z} | \boldsymbol {x})} [ \log q (\boldsymbol {x} | \boldsymbol {z}) ]} _ {\text {d a t a r e c o n s t r u c t i o n}} - \underbrace {\sum_ {v} ^ {m} D _ {K L} (p (\boldsymbol {z} ^ {(v)} | \boldsymbol {x} ^ {(v)}) | | q (\boldsymbol {z} ^ {(v)}))} _ {\text {m u l t i - r e g u l a r i z a t i o n}} \right. \\ \left. - \gamma \underbrace {\mathbb {E} _ {p (\mathbf {z} \mid \boldsymbol {\chi})} \left[ D _ {K L} \left(p (\boldsymbol {c} \mid \boldsymbol {\mathcal {X}}) \right| | q (\boldsymbol {c} \mid \boldsymbol {\mathbf {z}})\right)} _ {\text {c l u s t e r i n g}} \right] + \beta \underbrace {I (\mathbf {Z} ; \boldsymbol {\mathcal {Z}})} _ {\text {i n f o r m a t i o n s h i f t}}. \tag {10} \\ \end{array}
$$
Note that Equation (10) can be regarded as four items representing specific roles: data reconstruction, multi-regularization, information shift and clustering. The data reconstruction item maximizes the expectation of the log-likelihood to achieve comprehensive so that the multi-view representation retain more information of the data. The multi-regularization term makes the multi-view representation forget the information of the original data by minimizing the KL divergence of the posterior distribution and the variational distribution to meet the requirement of concentration. Information shift item adjusts the information relationship between multi-view representations to achieve cross-diverse. The clustering item measures the degree to which the data retains the original data structure after dimensionality reduction.
3.3 Numerical scheme to solve IMC
Following VAE [16], we use a deep neural network to estimate the parameters of the unknown distribution to instantiate IMC, as shown in Fig. 2 (c). According to the Markov chain  $\mathbf{X}^{(v)}\rightarrow$ $\mathbf{Z}^{(v)}\rightarrow \mathbf{Z}\rightarrow \mathbf{X}^{(v)}$ , the posterior  $p(\mathbf{z}|\boldsymbol {\mathcal{X}})$  and likelihood  $q(\boldsymbol {\mathcal{X}}|\mathbf{z})$  can be factorized as, respectively,
$$
\begin{array}{l} p (\mathbf {z} | \boldsymbol {\mathcal {X}}) = \int \int \dots \int p _ {\psi} (\mathbf {z} | \{\mathbf {z} ^ {(v)} \} _ {v = 1} ^ {m}) \prod_ {v = 1} ^ {m} p _ {\theta^ {(v)}} (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)}) d _ {\mathbf {z} ^ {(1)}} d _ {\mathbf {z} ^ {2}} \dots d _ {\mathbf {z} ^ {m}}, \\ = \mathbb {E} _ {p _ {\theta (1)} (\mathbf {z} ^ {(1)} | \mathbf {x} ^ {(1)})} \mathbb {E} _ {p _ {\theta (2)} (\mathbf {z} ^ {(2)} | \mathbf {x} ^ {(2)}) \dots \mathbb {E} _ {p _ {\theta (m)} (\mathbf {z} ^ {(m)} | \mathbf {x} ^ {(m)})} \left[ p _ {\psi} (\mathbf {z} | \{\mathbf {z} ^ {(v)} \} _ {v = 1} ^ {m}) \right], \\ \end{array}
$$
$$
q (\boldsymbol {\mathcal {X}} | \mathbf {z}) = \prod_ {v = 1} ^ {m} q _ {\phi^ {(v)}} \left(\mathbf {x} ^ {(v)} | \mathbf {z}\right),
$$
where  $p_{\theta^{(v)}}(\mathbf{z}^{(v)}|\mathbf{x}^{(v)})$  denotes the view-peculiar encoder with parameter  $\theta^{(v)}, q_{\phi^{(v)}}(\mathbf{x}^{(v)}|\mathbf{z}^{(v)})$  denotes the view-peculiar decoder with parameter  $\phi^{(v)}$ . To obtain the unified multi-view representations from view-peculiar representations, we design a fusion module  $p_{\psi}(\mathbf{z}|\{\mathbf{z}^{(v)}\}_{v=1}^{m})$  parameter with multiple fully connected layers as  $\psi$ .
For clustering item, we adopt the same strategy as DEC [12], replace  $q(\pmb{c}|\mathbf{z})$  with Student's t-distribution, and set target  $p(\pmb{c}|\pmb{x})$  with auxiliary distribution,
$$
q _ {i k} = \frac {\left(1 + \| \mathbf {z} _ {i} - \mathbf {u} _ {k} \| ^ {2}\right) ^ {- 1}}{\sum_ {j} \left(1 + \| \mathbf {z} _ {i} - \mathbf {u} _ {j} \| ^ {2}\right) ^ {- 1}},
$$
$$
p _ {i k} = \frac {q _ {i k} ^ {2} / \sum_ {i} q _ {i k}}{\sum_ {j} \left(q _ {i j} ^ {2} / \sum_ {i} q _ {i j}\right)}, \tag {11}
$$
where  $q_{ik}$  measures the similarity between the representation  $\mathbf{z}_i$  and cluster center  $\mathbf{u}_k$ .  $p_{ik}$  enhances the data points assigned with high confidence to improve the purity of the cluster.
For optimization IMC using Stochastic Gradient Variational Bayes (SGVB), the sampling operations of  $\left(\{\mathbf{z}^{(v)}\}_{v=1}^{m}, \mathbf{z}\right)$  should be mapped to the deterministic functions. According to the reparameterization trick [16] for the continuous variable, we sample the  $t$ -th latent representations by,
$$
\mathbf {z} _ {t} ^ {(v)} = \boldsymbol {\mu} _ {\phi^ {(v)}} + \boldsymbol {R} _ {\phi^ {(v)}} \boldsymbol {\epsilon} _ {t} ^ {(v)}, \text {w h e r e} \boldsymbol {R} _ {\phi^ {(v)}} \boldsymbol {R} _ {\phi^ {(v)}} ^ {T} = \boldsymbol {\Sigma} _ {\phi^ {(v)}}, \boldsymbol {\epsilon} _ {t} ^ {(v)} \in \mathcal {N} (\mathbf {0}, \mathbf {I}) \tag {12}
$$