SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Variational deep embedding (VaDE) [17] is an unsupervised generative clustering algorithm, which combines VAE and Gaussian Mixture Model (GMM) to model the data generation process. In our method, from Eq. (3), it can be found that SCMVC has taken into account the consistency information among views compared with the VaDE for the single-view scenario. Moreover, we introduce the SUMVC to help obtain a more sufficient representation, and we propose new methods to solve the problem based on the reparameterization trick.
4.2 Bayes Error Rate for Downstream Task
In this section, inspired by [8, 39], we present a theoretical analysis of the generalization error of representations, assuming  $T$  to be a categorical variable. We take view  $j$  as self-supervised signals to guide the learning of view  $i$  as an example. To represent the irreducible error when inferring labels from the representation via an arbitrary classifier, we use Bayes error rate  $P_{e} \coloneqq \mathbb{E}_{z^{i} \sim P_{z^{i}}} [1 - \max_{t \in T} P(\hat{T} = t | z^{i})]$ , where  $P_{e}$  is the Bayes error rate of arbitrary learned representations  $z^{i}$ , and  $\hat{T}$  is the estimated label  $T$  from our classifier. Analyzing the Bayes error rate allows us to gain insights into the generalization performance of our propose model. We present a general form of the complexity of the sample with mutual information  $I(z^{i}; x^{j})$  estimation using empirical samples from the distribution  $P_{z^{i}, x^{j}}$ . Let  $P_{z^{i}, x^{j}}^{(n)}$  denote the uniformly sampled empirical distribution of  $P_{z^{i}, x^{j}}$ . We define  $\hat{I}_{\theta^{*}}^{(n)}(z^{i}; x^{j})$  as  $\mathbb{E}_{P_{z^{i}, x^{j}}^{(n)}}[\log \frac{p(x^{j} | z^{j})}{p(x^{j})}]$ .
Proposition 1 (Mutual Information Neural Estimation [40].) Let  $0 < \delta < 1$ . There exists  $d \in \mathbb{N}$  and a family of neural networks  $F := \hat{f}_{\theta} : \theta \in \Theta \subseteq \mathbb{R}^d$ , where  $\Theta$  is compact, so that  $\exists \theta^{*} \in \Theta$ , with probability at least  $1 - \delta$  over the draw of  $\{z_{x_m^i}, x_m^j\}_{m=1}^n \sim P_{z^i, x^j}^{\otimes n}$ ,
$$
\left| \hat {I} _ {\theta^ {*}} ^ {(n)} \left(z ^ {i}; x ^ {j}\right) - I \left(z ^ {i}; x ^ {j}\right) \right| \leq O \left(\sqrt {\frac {1 + \log \left(\frac {1}{\delta}\right)}{n}}\right). \tag {12}
$$
This proposition shows that there exists a neural network  $\theta^{*}$ , with high probability,  $\hat{I}_{\theta^{*}}^{(n)}(z^i;x^j)$  can approximate  $I(z^{i};x^{j})$  with  $n$  samples at rate  $O\left(\frac{1}{\sqrt{n}}\right)$ . Under this network  $\theta^{*}$  and the same parameters  $d$  and  $\delta$ , we are ready to present our main results on the Bayes error rate.
Theorem 4.1 (Bayes Error Rates for Arbitrary Learned Representations [39].) Let  $|T|$  be  $T$ 's cardinality and  $Th(x) = \min \{\max \{x,0\}, 1 - \frac{1}{|T|}\}$ . Given  $p_e = Th(\bar{p}_e)$ , we have a thresholding function:
$$
\bar {p} _ {e} \leq 1 - \exp \left(- (H (T) + I \left(x ^ {i}; x ^ {j} \mid T\right) + I \left(z ^ {i}; x ^ {i} \mid x ^ {j}, T\right) - \hat {I} _ {\theta^ {*}} ^ {(n)} \left(z ^ {i}; x ^ {j}\right) + O \left(\sqrt {\frac {1 + \log \left(\frac {1}{\delta}\right)}{n}}\right)\right). \tag {13}
$$
Given arbitrary learned representations  $(z^i)$ , Theorem 4.1 suggests the corresponding Bayes error rate  $(P_e)$  is small when: 1) the estimated mutual information  $\hat{I}_{\theta^*}^{(n)}(z^i;x^j)$  is large; 2) a larger number of samples are used for estimating the mutual information; and 3) the task-irrelevant information  $I(z^i;x^i|x^j,T)$  is small. The first and the second results support the claim that maximizing  $I(z^i;x^j)$  may learn the representations beneficial to downstream tasks. The third result implies the learned representations may perform better on the downstream task when the task-irrelevant information is small. We evaluate the quality of our self-supervised method's learned representations using Bayes Error Rates in Theorem 4.2. The proofs are given in Appendix B.4.
Theorem 4.2 The Bayes Error Rates for arbitrary learned representations can be minimized by maximizing  $L_1^i + \beta L_2^{ij}$ , formally:
$$
\max  L _ {c o n} ^ {i} + \beta L _ {s u f} ^ {i j} \leftrightarrow \min  \bar {p} _ {e}. \tag {14}
$$
Table 1: The statistics of experimental datasets.
Dataset	#Samples	#Views	#Clusters
Multi-MNIST	70000	2	10
Multi-Fashion	10000	3	10
Multi-COIL-10	720	3	10
Multi-COIL-20	1440	3	20
Datasets As shown in Table 1, we use the following four real-world multi-view datasets in our study. MNIST [20] is a widely used dataset of handwritten digits from 0 to 9. The Fashion dataset [45] comprises images of various fashion items, including T-shirts, dresses, coats, etc. The COIL dataset [29] contains images of various objects, such as cups, ducks, and blocks, shown in different poses. We use multi-view datasets derived from origin datasets: Multi-COIL-10, Multi-COIL-20, Multi-MNIST and Multi-Fashion. Each dataset includes multiple views of each example, all randomly sampled from the same category. In Multi-COIL-10 (K = 10) and Multi-COIL-20 (K = 20), different views of an object correspond to various poses, but retain the same label. In Multi-MNIST, different views of a digit represent the same digit written by different individuals. In Multi-Fashion, different views of a product category signify different fashionable designs for that category.
Comparison Methods The comparison methods include three single-view clustering methods:  $K$ -means (1967) [27],  $\beta$ -VAE ( $\beta$ -VAE: learning basic visual concepts with a constrained variational framework (2017) [12]), and VaDE (variational deep embedding: an unsupervised and generative approach to clustering (2016) [17]), the input of which is the concatenation of all views, and five state-of-the-art MVC methods: BMVC (binary multi-view clustering (2018) [53]), SAMVC (self-paced and auto-weighted multi-view clustering (2020) [32]), RMSL (reciprocal multi-layer subspace learning for multi-view clustering (2019) [22]), DEMVC (deep embedded multi-view clustering with collaborative training (2021) [47]), FMVACC (fast multi-view anchor-correspondence clustering (2022) [43]).
Evaluation Measures Three measures are used to evaluate the clustering results, including, clustering accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). A larger value of these metrics indicates a better clustering performance.
Table 2: Clustering results of all methods on four datasets. The best result in each row is shown in bold and the second-best is underlined.
Datasets	K-means	β-VAE	VaDE	BMVC	SAMVC	RMSL	DEMVC	FMVACC	SCMVC	SUMVC
ACC
Multi-MNIST	53.9	49.3	30.7	89.3	-	-	98.2	55.6	94.5	99.8
Multi-Fashion	47.6	51.3	40.6	62.2	77.9	77.9	78.6	77.4	84.9	91.4
Multi-COIL-10	73.3	59.8	32.5	66.7	67.8	96.4	89.1	93.2	98.1	100.0
Multi-COIL-20	41.5	53.1	20.3	57.0	83.4	66.5	85.0	75.8	83.7	100.0
NMI
Multi-MNIST	48.2	43.6	35.4	90.2	-	-	98.9	48.2	87.3	99.3
Multi-Fashion	51.3	51.0	53.7	75.6	68.8	75.6	90.3	73.8	80.9	87.5
Multi-COIL-10	76.9	68.5	44.8	68.1	82.6	92.5	89.7	93.4	96.7	100.0
Multi-COIL-20	64.5	66.7	36.9	90.0	79.1	76.3	96.5	84.9	90.8	100.0
ARI
Multi-MNIST	36.0	29.1	8.5	85.6	-	-	98.6	45.2	88.5	99.5
Multi-Fashion	34.8	33.7	22.8	68.2	55.7	68.2	77.2	70.3	75.6	84.1
Multi-COIL-10	64.8	51.4	18.1	53.0	62.1	92.1	89.7	92.5	95.8	100.0
Multi-COIL-20	38.4	45.0	9.0	81.3	55.4	58.7	86.0	79.1	79.9	100.0
Table 3: Ablation studies of SUMVC on loss components.
Components	Multi-MNIST	Multi-Fasion	Multi-COIL-20
\( L_{rec} \)	\( L_{KL} \)	\( L_{suf} \)	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI
✓	✘	✘	70.5	72.9	62.6	56.7	57.6	47.7	62.5	81.1	63.3
✓	✓	✘	94.5	87.3	88.5	84.9	80.9	75.6	83.7	90.8	79.9
✘	✘	✓	11.5	0.2	0.1	11.3	0.1	0.1	9.9	4.6	0.0
✓	✓	✓	99.8	99.3	99.5	91.4	87.5	84.1	100.0	100.0	100.0
5.2 Comparison with State-of-the-Art Methods
Table 2 shows the quantitative comparison between the proposed methods and baseline models for several datasets. The best result is highlighted in bold, while the second-best result is underlined in each line. Despite its simplicity, the SCMVC method demonstrates similar performance to existing methods in most cases. This is primarily because our method focuses on extracting global information from multiple distributions. Moreover, the results reveal that SUMVC outperforms existing methods in terms of three evaluation methods. Notably, SUMVC achieves remarkable improvements on Multi-COIL-10 and Multi-COIL-20. The reason is that the proposed lower bound could explore consistent predictive information and reduce redundancy among views, which benefits clustering results.
5.3 Ablation Studies
To further verify the contributions of the proposed method, we conduct ablation studies on Eq. (11), which consists of  $L_{rec}$ ,  $L_{KL}$ , and  $L_{suf}$ . As shown in Table 3, it is found that the performance of clustering is significantly influenced by  $L_{KL}$ . We can also find that combining  $L_{con}$  and  $L_{suf}$  yields better results than using either  $L_{con}$  or  $L_{suf}$  alone, which illustrates that each loss function plays a crucial role in the overall performance of the model. We find that the clustering performance is not ideal when only  $L_{suf}$  is used for training. The reason is that the latent features extracted by the lack of reconstruction constraints cannot accurately reflect the information of the original features.
5.4 Visualization of Feature Representation
We use  $t$ -SNE [42] to visualize the learning process of embedded features. The non-linear dimensionality reduction technique is used for visualizing and clustering high-dimensional data, which has been widely used in machine learning and data analysis to visualize high-dimensional data
(a) epoch=50
(b) epoch  $= 100$
(c) epoch=300
(d) epoch=400
(e) epoch=100
(f) epoch=200
(g) epoch=400
(h) epoch=800
Figure 1: Visualization of the latent representations  $\vec{z}$  in training on Multi-COIL-10 and Multi-COIL-20 in the first and second rows, respectively.
Figure 2: Clustering performance w.r.t. different parameter settings on Multi-COIL-10 and Multi-COIL-20 respectively.
in lower-dimensional spaces for easier understanding and interpretation. We show the embedded features  $\vec{z}$  of Multi-COIL-10 and Multi-COIL-20 in Fig. 1. At the start of the training stage, the embedded features are non-separable. As the training progresses, the clustering structures of the embedded features become more apparent while their centroids gradually separate. This visual demonstration highlights the effectiveness of our proposed model.
5.5 Parameter Sensitivity Analysis
SUMVC has two primary hyperparameters, i.e.,  $\gamma$  and  $\beta$ . We test the clustering performance with different settings to demonstrate the stability of SUMVC. Results are presented in Fig. 2. Additionally, higher values of  $\gamma$  are required to encourage disentangling but result in a fidelity-disentanglement trade-off between the quality of reconstructions and its latent features disentanglement.  $\beta$  that is either too large or too small also leads to reduced clustering performance due to the imbalance between reducing redundant information and obtaining consistent information. The recommended parameters for  $\gamma$  and  $\beta$  are both 0.1 in our experiments.
In this paper, we propose a simple consistent approach and an information-theoretic approach for multi-view clustering to address the challenge of obtaining useful information from multiple views while reducing redundancy and ensuring consistency among views. In particular, we leverage the information bottleneck principle and variational inference to the multi-view scenarios. The proposed SCMVC and SUMVC are extensively evaluated on multiple real-world multi-view datasets. Extensive experiments demonstrate the clustering performance of the proposed models.
This work is supported in part by Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China (ZYGX2021YGLH022). Lifang He is partially supported by the National Science Foundation grants (MRI-2215789, IIS-1909879, IIS-2319451) and Lehigh's grants under Accelerator and CORE.
[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. JMLR, 19:1947-1980, 2018.
[2] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. TPAMI, 40:2897-2905, 2018.
[3] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
[4] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pages 92-100, 1998.
[5] TM Cover and JA Thomas. Joint entropy and conditional entropy. Elements of Information Theory, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, page 16, 2006.
[6] Iain J Cruickshank. Multi-view clustering of social-based data. PhD diss., Carnegie Mellon University, 2020.
[7] Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. TPAMI, 35(11):2765-2781, 2013.
[8] Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions on Information Theory, 40:259-266, 1994.
[9] Marco Federici, Anjan Dutta, Patrick Forre, Nate Kushner, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In ICLR, 2020.
[10] Jacob Goldberger, Hayit Greenspan, and Shiri Gordon. Unsupervised image clustering using the information bottleneck method. In DAGM-Symposium, pages 158-165, 2002.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63:139–144, 2020.
[12] Irina Higgins, Loic Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
[13] Shudong Huang, Hongjie Wu, Yazhou Ren, Ivor Tsang, Zenglin Xu, Wentao Feng, and Jiancheng Lv. Multi-view subspace clustering on topological manifold. In NeurIPS, pages 25883-25894, 2022.
[14] Zhanpeng Huang, Jiekang Wu, Jinlin Wang, Yu Lin, and Xiaohua Chen. A unified multi-view clustering method based on non-negative matrix factorization for cancer subtyping. IJDWM, 19:1-19, 2023.
[15] HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view representation learning via total correlation objective. In NeurIPS, pages 12194-12207, 2021.
[16] Senerath Mudalige Don Alexis Chinthaka Jayatilake and Gamage Upeksha Ganegoda. Involvement of machine learning tools in healthcare decision making. Journal of Healthcare Engineering, 2021.
[56] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS, pages 321-328, 2003.
[57] Yiyang Zhou, Qinghai Zheng, Shunshun Bai, and Jihua Zhu. Semantically consistent multi-view representation learning. arXiv preprint arXiv:2303.04366, 2023.
[58] Yiyang Zhou, Qinghai Zheng, Wenbiao Yan, Yifei Wang, and Jihua Zhu. Mcoco: Multi-level consistency collaborative multi-view clustering. arXiv preprint arXiv:2302.13339, 2023.