SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {z} _ {t} = \boldsymbol {\mu} _ {\psi} + \boldsymbol {R} _ {\psi} \boldsymbol {\epsilon} _ {t}, \quad \text {w h e r e} \boldsymbol {R} _ {\psi} \boldsymbol {R} _ {\psi} ^ {T} = \boldsymbol {\Sigma} _ {\psi}, \boldsymbol {\epsilon} _ {t} \in \mathcal {N} (\mathbf {0}, \mathbf {I}) \tag {13}
$$
Algorithm 1 Optimization Procedure of IMC
Input: Multi-view dataset  $\mathcal{X}$ ; Setting  $T = 1$  and dimensionality of the latent variables.  
Parameter: Initialize parameters  $\{\phi^{(v)}\}$ ,  $\{\theta^{(v)}\}$ ,  $\psi$  with random values, cluster centers  $\mathbf{u} = \{\mathbf{u}_1,\dots,\mathbf{u}_K\}$  by  $K$ -means.  
1: while not reaching the maximal epochs do  
2: for  $i$  in  $n$  samples do  
3: for  $v$  in  $m$  views do  
4: Calculate  $(\pmb{\mu}_{\phi^{(v)}},\pmb{\Sigma}_{\phi^{(v)}})$  through  $v$ -th encoder and then sample  $\mathbf{z}_{it}^{(v)}$  by Equation (12);  
5: end for  
Calculate  $(\pmb{\mu}_{\psi},\pmb{\Sigma}_{\psi})$  through the fusion module and then sample  $\mathbf{z}_{it}$  by Equation (13);  
6: for  $v$  in  $m$  views do  
7: Generate  $\mathbf{x}^{(v)}$  by  $v$ -th decoders.  
8: end for  
9: end for  
10: Update  $\{\phi^{(v)}\}$ ,  $\{\theta^{(v)}\}$ ,  $\psi$ ,  $\mathbf{u}$  by maximizing Equation (14).  
11: end while
Output: The cluster for each sample  $\mathbf{c}_i$  by selecting the index of maximum probability value  $q(\mathbf{c}_i|\mathbf{z}_i)$ .
where  $\pmb{\mu}_{\phi^{(v)}}$  and  $\pmb{R}_{\phi^{(v)}}$  are the outputs of the  $v$ -th encoder,  $\pmb{\mu}_{\psi}$  and  $\pmb{R}_{\psi}$  are the outputs of the fusion module, respectively.
By using Monte-Carlo estimators, the objective of IMC can be further written as,
$$
\begin{array}{l} \max  _ {\{\theta^ {(v)} \}, \{\phi^ {(v)} \}, \psi , \mathbf {u}} \mathcal {L} _ {\mathrm {I M C}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \frac {1}{T} \sum_ {v = 1} ^ {m} \sum_ {t = 1} ^ {T} \left[ \log q _ {\phi^ {(v)}} \left(\mathbf {x} _ {i} ^ {(v)} | \mathbf {z} _ {i t}\right) - \log p _ {\theta^ {(v)}} \left(\mathbf {z} _ {i t} ^ {(v)} | \mathbf {x} _ {i} ^ {(v)}\right) \right. \right. \\ \left. + \log \mathcal {N} (\mathbf {0}, \mathbf {I}) \right] - \gamma \sum_ {k = 1} ^ {K} p _ {i k} \log \frac {p _ {i k}}{q _ {i k}} \Bigg ] + \beta I _ {J S} (\mathbf {Z}; \boldsymbol {\mathcal {Z}}), \tag {14} \\ \end{array}
$$
where  $T$  denotes the number of Monte Carlo samples and is usually set to be 1. Additionally, variational distribution  $q(\mathbf{z}^{(v)})$  is set to multivariate standard normal distribution, i.e.,  $q(\mathbf{z}^{(v)}) = \mathcal{N}(\mathbf{0},\mathbf{I})$ . The mutual information between representations  $I(\mathbf{z};\mathcal{Z})$  can be maximized by using sample-based differentiable mutual information lower bound, Jensen-Shannon estimator [17]. The partial derivatives of each parameter combination are then calculated for the stochastic backpropagation technique. The concrete optimization procedure of IMC is summarized in Algorithm 1.
Recently, several works have been devoted to the problem of multi-view representation learning based on information theory. For supervised methods, [18] analyses the consistency and complementary characteristics of multi-view features from the information bottleneck theory, and strengthen the discriminative power of the encoder through the margin maximization method to balance the accuracy and complexity of the multi-view model. [19] seeks to fuse the features of each view to obtain a joint representation through a deep neural network, and maximize the mutual information between labels and joint representations to preserve the target information. Given a graph and its labels, [20] proposes a Graph Information Bottleneck (GIB) framework to efficiently infer the most informative yet compressed subgraph for recognition.
While in the unsupervised scheme, [14] proposes a multi-view representation learning method based on information bottleneck theory, Multi-view Information Bottleneck (MIB). MIB assumes that the shared part of the two views is the target information, and the part not shared by the two views is the superfluous information, followed by leveraging off-the-shelf information bottleneck technique to learn robust representations. [21] explores the potential relationship and intrinsic information between different views, and discarded redundant information from multi-view data with the help of the information bottleneck principle to balance the consistency and complementarity between multiple views. Under the framework of information theory, Completer [22] maximizes multi-view shared mutual information for consistent learning, and minimizes conditional entropy to recover missing view with available data. [23] regards each view source data as a self-supervised signal for
learning latent representations, removing redundant information from the view itself while retaining consistent information for shared features across multiple views. A multivariate mutual information term is employed to decompose view-specific representations, which further ensures that the multi-view fusion self-expression matrix via view-common features is compact and informative. The aforementioned unsupervised multi-view representation learning methods aim to learn consistent information shared across multiple views, as shown in Fig. 2 (a). These methods either have limitations in practical applications or do not consider the duplicate information among views. It motivates us to construct a general information-theoretic-based unsupervised multi-view clustering framework and introduce three requirements for multi-view representation learning.
Model setup: The architectures of  $p_{\phi^{(v)}}(\mathbf{z}^{(v)}|\mathbf{x}^{(v)})$  and  $q_{\theta^{(v)}}(\mathbf{x}^{(v)}|\mathbf{z})$  are fully connected networks with  $d_v$ -500-500-1204-256 and 256-1024-500-500- $d_v$  neurons, respectively, where  $d_v$  is the dimensionality of each view. The fusion network  $p_{(\psi)}(\mathbf{z}|\{\mathbf{z}^{(v)}\}_{v=1}^m)$  concatenates multiple view-peculiar latent representations, followed by a fully connected layer. Adam optimizer [24] is utilized to maximize the objective function, and set the learning rate to be 0.001 with a decay of 0.9 for every 10 epochs.
Datasets: We adopt four real-world datasets listed as follows, (1) UCI-digits [25] contains 2000 examples of ten numerals from 0 to 9 with five views which are respectively extracted by Fourier coefficients, profile correlations, Karhunen-Love coefficient, Zernike moments, and pixel average extractors. (2) Notting-Hill [26] is widely used video face dataset for clustering, which collects 4660 faces across 76 tracks of the 5 main actors from the movie 'Notting Hill'. We use the multi-view version provided in [27], consisting of 550 images with three kind of features, i.e., LBP, gray pixels, and Gabor features. (3) BDGP [28] contains 2500 images in 5 categories, and each sample is described by a 1750-D image vector and a 79-D textual feature vector. (4) Caltech20 is a subset of the object recognition dataset [35] containing 20 classes with six different views, including Gabor features, wavelet moments, CENTRIST features, histogram of oriented gradients, GIST features and local binary patterns.
Compared algorithms: Four state-of-the-art algorithms and two ablation models are used to compare the clustering performance, including: (1) DMVAE [36] assumes that the data and multi-view latent representation obey a Gaussian mixture distribution, which can be considered as a special case of IMC, i.e., let  $\beta = 0$  and  $\gamma = 1$ . (2) MIB [14] considers the parts shared by both views as target information and the view-specific parts as superfluous information. (3) CMIB-Nets [21] proposes collaborative multi-view information bottlenecks to learn the information inherent within a view and the shared structure between views, while reducing redundant information. (4) Completer [15] introduces maximization of mutual information between views and minimization of conditional entropy of different views to achieve dual prediction and representation learning. For fairness, we repeatedly perform all compared algorithms 10 times on four datasets and the mean and standard deviation were recorded.
Ablation models: For ablation study, we construct two variants of IMC as a comparison. (1) IMC-v1 indicates that the objective function Equation (10) reduces the item of information shift, i.e.,  $\beta = 0$ . (2) IMC-v2 sets  $\gamma = 0$  to discard the KL divergence of the clustering item, and uses  $k$ -means to perform clustering on the unified multi-view representation.
Evaluation metrics: For a comprehensive analysis, we use three popular clustering metrics including Normalized Mutual Information (NMI), Accuracy (ACC) and adjusted rand index (ARI). The higher the values of these indicators, the better the clustering performance.
We tested seven methods on four multi-view datasets. The experimental results are summarized in Table 1. It can be observed that i) combining the four datasets in terms of three indicators, IMC has obtained the best performance 8 times and the second-best performance 3 times, which is the most stable and accurate multi-view clustering model. ii) Although the Completer model has achieved the
Table 1: Clustering performance comparison on four datasets (mean±standard deviation). The optimal and suboptimal results are in bold and underlined, respectively.
Datasets	Metrics	DMVAE	MIB	CMIB-Nets	Completer	IMC-v1	IMC-v2	IMC
UCI-digits	ACC	90.95±0.62	83.30±1.27	85.70±1.15	91.28±1.41	90.01±0.60	84.01±1.15	92.13±0.55
NMI	85.54±1.06	75.43±1.04	78.31±1.41	86.34±0.60	84.79±0.32	79.01±1.54	88.01±0.73
ARI	85.40±1.54	76.16±1.55	76.97±1.64	86.67±0.86	84.78±0.43	78.18±0.88	87.83±0.25
Notting-Hill	ACC	76.22±1.21	81.65±1.37	85.40±2.36	80.17±2.79	77.79±2.00	84.83±1.90	87.10±1.35
NMI	72.97±0.96	75.95±2.52	78.65±2.57	76.11±2.27	74.93±1.94	78.79±1.08	80.67±1.42
ARI	69.50±0.77	71.91±1.61	80.46±1.92	71.48±3.29	70.30±2.15	79.10±2.32	80.19±1.74
BDGP	ACC	90.59±1.45	86.82±0.65	85.82±0.58	79.31±1.55	88.70±1.42	80.46±1.85	91.46±0.82
NMI	85.32±0.53	80.82±0.86	81.60±0.66	74.25±0.69	81.47±1.12	73.31±2.45	84.40±1.20
ARI	78.58±2.54	73.90±1.65	74.25±2.19	71.44±1.45	77.65±2.15	69.31±2.80	80.25±1.62
Caltech20	ACC	61.50±0.54	56.12±2.54	55.26±3.18	62.31±2.65	58.05±1.28	52.42±3.15	60.82±1.66
NMI	68.32±1.23	63.28±2.66	62.44±2.56	70.25±2.20	65.75±2.20	61.76±3.40	69.20±1.48
ARI	59.86±1.46	58.10±2.90	56.45±2.74	61.14±2.55	57.52±2.56	53.30±2.82	61.42±2.12
Table 2: Comparison of NMI at noise ratios of  $10\%$ ,  $30\%$ , and  $50\%$  (mean  $\pm$  standard deviation). The optimal and suboptimal results are in bold and underlined, respectively.
Datasets	Noise	DMVAE	MIB	CMIB-Nets	Completer	IMC-v1	IMC-v2	IMC
UCI-digits	10%	82.35±1.24	74.60±1.75	75.55±2.35	85.28±1.84	81.12±2.26	78.46±1.45	86.82±1.05
30%	75.84±2.26	66.35±3.45	65.50±2.82	73.40±2.56	69.90±3.25	72.18±2.48	78.56±1.36
50%	63.80±1.58	58.46±2.80	56.46±1.64	60.76±3.65	56.84±3.40	65.30±2.66	70.35±2.28
best NMI performance in caltech20, its NMI performance lags behind by  $12.97\%$  in BDGP. This instability can also be found in the MIB model. iii) The end-to-end model (IMC, IMC-v1, DMVAE) that considers the clustering loss in the process of learning multi-view representation will achieve better clustering results than the two-stage model (MIB, CMIB-nets, Completer). iv) Comparing IMC, IMC-v1 and IMC-v2, the performance of using KL divergence clustering term has been significantly improved, while the use of information shift term has a small improvement.
We therefore draw three conclusions: First, the IMC developed based on the general definition of multi-view information theory obtains more stable clustering results. Second, end-to-end models help to learn more discriminative representations. Third, in the process of learning multi-view representations, three requirements (comprehensiveness, concentration, cross-diversity) play a role in improving the clustering performance.
5.3 Robustness Assessment
To evaluate the robustness of the model, we add different scales of sparse noise to the dataset UCI-digits. Specifically, we randomly select  $10\%$ ,  $30\%$ , and  $50\%$  pixels of the original image size, and add pepper noise or salt noise to each pixel with probability 0.5, i.e., set 0 or 255.
As shown by the clustering results in Table 2, the performance of all algorithms decreases as the noise ratio increases, but the performance of our algorithm is always the best. It can be observed that the decline of IMC is the smallest, while the decline of the Completer model is obvious. This may be due to the assumption of multi-view semantic consistency is more sensitive as the noise ratio increases, the information shared by multiple views will also incorporate more noise. It is worth noting that the performance of IMC-v2 is better than that of IMC-v1, which is contrary to the clustering experiment in Table 1. It can be concluded that only pursuing inter-view consistent information leads to semantic bias when the dataset contains noise, while more informative representations can be obtained for robustness based on the general multi-view information bottleneck theory.
5.4 Parameter analysis
The proposed method contains two balance parameters, the information shift balance coefficient  $\beta$ , and the clustering loss trade-off parameter  $\gamma$ . Although IMC at fixed values of some parameters has
Figure 3: Parameter analysis of  $\beta$  and  $\gamma$  on Caltech20 dataset.
shown satisfactory performance, it is still important to explore the effect of varying these parameters on the performance and to mine the inner mechanism of this model. We evaluate the parameter sensitivity of ACC, NMI, ARI metrics on Caltech20 dataset, as shown in Figure 3, we choose the values of  $\beta$  and  $\gamma$  in the range of  $\{0.01, 0.1, 1, 10, 100\}$ . It can be seen from the results that our method is not sensitive to the coefficient of clustering loss, but how to adjust the parameter  $\beta$  is the key to improving clustering performance. As expressed in Equation (8),  $\beta$  controls the degree of cross-diversity of multi-view representation. Therefore, the selection strategy of the beta parameter can be based on prior knowledge of the dataset. In our experience, if there is a large degree of consistency in the semantics of multiple views, then the  $\beta$  is set to small and vice versa.
In this paper, we propose an information-theoretic multi-view clustering framework that avoids the assumption of semantic consistency across multiple views. Specifically, we extend the information bottleneck theory to unsupervised multi-view learning through defining three requirements for multi-view representations via mutual information. By constructing a multi-view variational lower bound and introducing KL divergence as a clustering loss, the entire framework is finally optimized by deep neural network and stochastic gradient variational Bayes. Experiments verify the superiority and robustness of the generalized information-based multi-view clustering on four benchmark datasets and noisy data.
The practical limitations of the proposed model lie in the choice of parameters. The mathematical strategy for selecting the optimal parameters is a direction that can be studied in the future. In addition, constructing a tighter lower bound or estimator to approximate high-dimensional mutual information is also a place for further improvement in this work.
We thank the anonymous reviewers for their helpful feedbacks. This work was supported in part by the National Key Research and Development Program of China (2022YFE0112200), the Key-Area Research and Development of Guangdong Province (2022A0505050014, 2022B1111050002), the Key-Area Research and Development Program of Guangzhou City (202206030009, 2023B01J0002), the National Natural Science Foundation of China (U21A20520, 62172112), Guangdong Key Laboratory of Human Digital Twin Technology (2022B1212010004).
[1] Navneet Dalal, and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR pages 886-893, 2005.
[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR pages 248-255, 2009.
[23] Shizhe Hu, Zenglin Shi, Xiaogiang Yan, Zhengzheng Lou, and Yangdong Ye. Multiview Clustering With Propagating Information Bottleneck. IEEE Trans. Neural Netw. Learn. Syst., 2023.
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[25] Arthur Asuncion and David Newman. Uci machine learning repository. [http://archive.ics.uci.edu/ml], 2007.
[26] Yi-Fan Zhang, Changsheng Xu, Hanqing Lu, and Yeh-Min Huang. Character identification in feature-length films using global face-name matching. IEEE Trans. Multimedia, 11(7):1276-1288, 2009.
[27] Xiaochun Cao, Changqing Zhang, Huazhu Fu, Si Liu, and Hua Zhang. Diversity-induced multi-view subspace clustering. In CVPR, pages 586-594, 2015.
[28] Xiao Cai, Hua Wang, Heng Huang, and Chris Ding. Joint stage recognition and anatomical annotation of drosophila gene expression patterns. Bioinformatics, 28(12):i16-i24, 2012.
[29] Karpathy Andrej, and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, pages 3128-3137, 2015.
[30] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771-1800, 2002.
[31] Mengelei Hu and Songcan Chen. Doubly aligned incomplete multi-view clustering. In *IJCAI*, pages 2262-2268, 2018.
[32] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In ICML, pages 1083-1092, 2015.
[36] Ming Yin, Weitian Huang, and Junbin Gao. Shared generative latent representation learning for multi-view clustering. In AAAI, pages 6688-6695, 2020.
[34] Mengelei Hu and Songcan Chen. One-pass incomplete multi-view clustering. In AAAI, pages 3838-3845, 2019.
[35] Fei-Fei Li, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, 2004.
[36] Ming Yin, Weitian Huang and Junbin Gao. Shared generative latent representation learning for multi-view clustering. In AAAI, pages 6688-6695, 2020.
Appendix A. Derivation of Equation (2)
By substituting the intractable posterior  $p(\mathbf{x}|\mathbf{z})$  and marginal  $p(\mathbf{z})$  with respective variational posterior  $q(\mathbf{x}|\mathbf{z})$  and marginal  $q(\mathbf{z})$ , we form a variational lower bound for unsupervised information bottleneck,
$$
\begin{array}{l} \max _ {\mathbf {Z}} I (\mathbf {Z}; \mathbf {X}) - \beta I (\mathbf {Z}; i) \\ = \int \int p (\mathbf {z}, \mathbf {x}) \log \frac {p (\mathbf {x} | \mathbf {z})}{p (\mathbf {x})} d \mathbf {x} d \mathbf {z} - \beta \sum_ {i} \int p (\mathbf {z}, i) \log \frac {p (\mathbf {z} | i)}{p (\mathbf {z})} d \mathbf {z} \\ = H (\mathbf {x}) + \mathbb {E} _ {p (\mathbf {x})} \left[ \int p (\mathbf {z} | \mathbf {x}) \log p (\mathbf {x} | \mathbf {z}) d \mathbf {z} \right] - \beta \frac {1}{n} \sum_ {i} \int p (\mathbf {z} | \mathbf {x} _ {i}) \log \frac {p (\mathbf {z} | \mathbf {x} _ {i})}{p (\mathbf {z})} d \mathbf {z} \\ \geq \mathbb {E} _ {p (\mathbf {x})} \left[ \int p (\mathbf {z} | \mathbf {x}) \log q (\mathbf {x} | \mathbf {z}) d \mathbf {z} \right] - \beta \frac {1}{n} \sum_ {i} \int p (\mathbf {z} | \mathbf {x} _ {i}) \log \frac {p (\mathbf {z} | \mathbf {x} _ {i})}{q (\mathbf {z})} d \mathbf {z} \\ = \mathbb {E} _ {p (\mathbf {x})} \left[ \int p (\mathbf {z} | \mathbf {x}) \log q (\mathbf {x} | \mathbf {z}) d \mathbf {z} - \beta \int p (\mathbf {z} | \mathbf {x}) \log \frac {p (\mathbf {z} | \mathbf {x})}{q (\mathbf {z})} d \mathbf {z} \right] \\ = \mathbb {E} _ {p (\mathbf {x})} \left[ \mathbb {E} _ {p (\mathbf {z} | \mathbf {x})} [ \log q (\mathbf {x} | \mathbf {z}) ] - \beta D _ {K L} (p (\mathbf {z} | \mathbf {x}) \| q (\mathbf {z})) \right], \\ \end{array}
$$
where  $H(\mathbf{x})$  is a constant and we take  $p(i) = \frac{1}{n}$  and  $p(\mathbf{z}|i) = p(\mathbf{z}|\mathbf{x}_i)$ . When the dataset containing  $n$  independent and identically distributed (i.i.d) samples is large enough, the expectation of  $p(\mathbf{x})$  can be regarded as uniform sampling from the dataset to get  $\mathbf{x}_i$ . Note that the lower bound is essentially identical to  $\beta \mathrm{VAE}$ . The former represents the data reconstruction process and the latter is a regularization term for forgotten identity information, driving the acquisition of informative and generative representations, respectively.
Appendix B. Derivation of Equation (8)
Same as Equation 2, we substitute the intractable posterior  $p(\mathcal{X}|\mathbf{z})$  and marginal  $p(\mathbf{z}^{(v)})$  with respective variational posterior  $q(\mathcal{X}|\mathbf{z})$  and marginal  $q(\mathbf{z}^{(v)})$ , and then the variational lower bound for the first two terms of equation 8 yields as follows,
$$
\begin{array}{l} \max  _ {\mathbf {Z}, \mathbf {\Sigma}} I (\mathbf {Z}; \mathbf {\Sigma} \mathcal {X}) - \sum_ {v} ^ {m} I (\mathbf {Z} ^ {(v)}; \mathbf {X} ^ {(v)}) \\ = \int \int p (\mathbf {z}, \boldsymbol {\mathcal {X}}) \log \frac {p (\boldsymbol {\mathcal {X}} | \mathbf {z})}{p (\boldsymbol {\mathcal {X}})} d \boldsymbol {\mathcal {X}} d \mathbf {z} - \sum_ {m} ^ {v} \int \int p (\mathbf {z} ^ {(v)}, \mathbf {x} ^ {(v)}) \log \frac {p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)})}{p (\mathbf {z} ^ {(v)})} d \mathbf {x} ^ {(v)} d \mathbf {z} ^ {(v)} \\ = H (\boldsymbol {\mathcal {X}}) + \mathbb {E} _ {p (\boldsymbol {\mathcal {X}})} \left[ \int p (\mathbf {z} | \boldsymbol {\mathcal {X}}) \log p (\boldsymbol {\mathcal {X}} | \mathbf {z}) d \mathbf {z} \right] - \sum_ {m} ^ {v} \mathbb {E} _ {p (\mathbf {x} ^ {(v)})} \left[ \int p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)}) \log \frac {p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)})}{p (\mathbf {z} ^ {(v)})} d \mathbf {z} ^ {(v)} \right] \\ \geq \mathbb {E} _ {p (\boldsymbol {\chi})} \left[ \int p (\mathbf {z} | \boldsymbol {\chi}) \log q (\boldsymbol {\chi} | \mathbf {z}) d \mathbf {z} \right] - \sum_ {m} ^ {v} \mathbb {E} _ {p (\mathbf {x} ^ {(v)})} \left[ \int p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)}) \log \frac {p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)})}{q (\mathbf {z} ^ {(v)})} d \mathbf {z} ^ {(v)} \right] \\ = \mathbb {E} _ {p (\boldsymbol {\chi})} \left[ \mathbb {E} _ {p (\mathbf {z} | \boldsymbol {\chi})} [ \log q (\boldsymbol {\chi} | \mathbf {z}) ] - \sum_ {v} ^ {m} D _ {K L} (p (\mathbf {z} ^ {(v)} | \mathbf {x} ^ {(v)}) | | q (\mathbf {z} ^ {(v)})) \right], \\ \end{array}
$$
where  $H(\mathcal{X})$  is a constant. Given the unified multi-view representation  $\mathbf{z}$ , multi-view datasets  $\mathcal{X}$  can be viewed as conditionally independent, i.e.,  $p(\mathcal{X}) = p(\mathbf{x}^{(1)})p(\mathbf{x}^{(2)})\dots p(\mathbf{x}^{(m)})$ . The first term maximizes the expectation of the log-likelihood to achieve comprehensive so that the multiview representation retain more information of the data. The second term makes the multi-view representation forget the information of the original data by minimizing the KL divergence of the posterior distribution and the variational distribution to meet the requirement of concentration.