SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenhang Cui $^{1*}$ , Yazhou Ren $^{1*†}$ , Jingyu Pu $^{1}$ , Jiawei Li $^{1}$ , Xiaorong Pu $^{1}$ , Tianyi Wu $^{1}$ , Yutao Shi $^{1}$ , Lifang He $^{2}$
<sup>1</sup> University of Electronic Science and Technology of China, China
$^{2}$  Department of Computer Science and Engineering, Lehigh University, USA chenhangcui@gmail.com, yazhou.ren@uestc.edu.cn, pujingyu0105@163.com,
puxiaor@uestc.edu.cn, sixteen.l.jw@gmail.com, syt_59421@outlook.com,
tianyi-wu@outlook.com, lih319@lehigh.edu
Multi-view clustering (MVC) is a popular technique for improving clustering performance using various data sources. However, existing methods primarily focus on acquiring consistent information while often neglecting the issue of redundancy across multiple views. This study presents a new approach called sufficient multiview clustering (SUMVC) that examines the multi-view clustering framework from an information-theoretic standpoint. Our proposed method consists of two parts. Firstly, we develop a simple and reliable multi-view clustering method SCMVC (simple consistent multi-view clustering) that employs variational analysis to generate consistent information. Secondly, we propose a sufficient representation lower bound to enhance consistent information and minimise unnecessary information among views. The proposed SUMVC method offers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data. To verify the effectiveness of our model, we conducted a theoretical analysis based on the Bayes Error Rate, and experiments on multiple multi-view datasets demonstrate the superior performance of SUMVC.
Clustering is a fundamental unsupervised learning task with broad applications across various fields. Traditional clustering methods primarily focus on analyzing single-view data. However, the rapid progress in multimedia technology has led to the collection of real-world data from multiple sources and perspectives, resulting in the emergence of multi-view data. Recently, multi-view clustering has garnered significant attention due to its ability to leverage information from multiple views to improve clustering performance and robustness [24]. Multi-view data arise in many applications where multiple sources of information or perspectives on the same set of objects are available. Multi-view clustering aims to group these objects into clusters by considering all available views jointly. This technique has proven to be highly effective in various domains, including social network analysis [6], image and video analysis [50], and bioinformatics [14]. Nevertheless, the main challenge in multi-view clustering lies in extracting valuable information from the views for clustering purposes while simultaneously minimizing view redundancy.
Information bottleneck (IB) theory provides a theoretical framework for understanding the optimal trade-off between accuracy and compression in data processing [36]. The IB method suggests that
$\dagger$  Corresponding author.
the optimal representation of the data can be achieved by maximizing the mutual information between the input and the output while minimizing the mutual information between the input and the representation. Although this principle provides a theoretic method to learn less superfluous information for downstream tasks, existing methods can still be improved in efficiently extracting embedded features from multi-view unlabeled data. In recent years, many information-theoretic methods have been applied to multi-view scenarios. Specifically, [44] proposes a generic pretext-aware residual relaxation technique to enhance multi-view representation learning (MVRL). [15] propose a novel approach to MVRL through an information-theoretic framework grounded in total correlation. Meanwhile, variational analysis has gained wide usage in machine learning and optimization for approximating complex probability distributions [3]. This technique offers a robust framework for solving optimization problems involving probability distributions, which are frequently encountered in multi-view clustering and information bottleneck. By approximating the probability distributions, variational analysis can effectively reduce the data dimensionality and enhance the efficiency of clustering algorithms. Variational autoencoders (VAEs) are a popular type of generative models that uses variational inference to learn compact representations of high-dimensional data [18]. However, the vast majority of them are designed for single-view data and cannot capture consistent and complementary information in a multi-view setting.
To tackle the aforementioned concerns, we delve deeper into the principles of the information bottleneck and variational analysis in the realm of multi-view clustering. We propose a simple consistent multi-view clustering method SCMVC and a novel multi-view self-supervised method SUMVC from an information-theoretic standpoint. SUMVC could strengthen consistency across views for clustering while concurrently decreasing view redundancy.
Our contributions are three-fold:
- We propose a consistent variational lower bound to explore the consistent information among views for multi-view clustering. We introduce SCMVC, a simple consistent multi-view clustering approach, utilizing the proposed lower bound. Despite being a simple approach, our experiments demonstrate the effectiveness of SCMVC.
- We extend the information bottleneck principle to the multi-view self-supervised learning and propose a sufficient representation lower bound. We expand SCMVC to improve the consistency and reduce the redundancy. This mainly aims to acquire predictive information and reduce redundancy among views. We refer to this new model as SUMVC (sufficient multi-view clustering).
- We adopt the collaborative training strategy for SUMVC to better adapt to multi-view scenarios. Additionally, a theoretical analysis of the generalization error of representations is conducted to evaluate the efficacy of SUMVC for clustering tasks. Experiments on real-world multi-view data show the superior performance of the proposed model.
Information Bottleneck The information bottleneck (IB) principle is a widely used framework in information theory and machine learning that provides a rational way to understand the balance between compression and prediction. Introduced by [36], the IB principle formalises the intuition that successful learning algorithms need to find a balance between retaining relevant information and discarding irrelevant information. Since then, the IB principle has been widely applied to various machine learning tasks, including unsupervised learning [37], classification [3], and clustering [2, 51]. Recently, the IB principle has also been used to study the information processing capacity of the brain [33, 34]. By constraining the learned representations of deep neural networks, the IB principle was shown to improve the generalization performance of these models and reduce their computational complexity [2]. In addition, the IB principle has been used to develop new techniques for training deep neural networks, such as the deep variational information bottleneck [2]. The theoretical foundation of the IB principle is built on concepts such as Markov chains, entropy, and conditional entropy [5]. It has been widely applied in various fields, such as data mining, image processing, natural language processing, and computer vision [10, 16, 35]. The IB principle has also been applied in control theory to quantify the amount of [31]. Overall, the IB principle has become an important tool for designing more efficient and effective machine learning algorithms. Its widespread applications in various fields highlight its relevance and practicality.
Varitional Autoencoder Variational autoencoders (VAEs) are a popular generative model class that learns a compact representation of input data through an encoder-decoder architecture. Unlike traditional autoencoders, VAEs are designed to model the latent space distribution of input data, enabling them to generate new data samples consistent with the input distribution. Early work on VAEs by [18] introduced the VAE model and derived its objective function, while [19] proposed a method for approximating gradients by sampling the latent variables. Since then, VAEs have been applied to various applications, including image generation, data compression, and anomaly detection. For instance, [11] combined VAEs with generative adversarial networks (GANs) to generate high-quality images, while [25] applied VAEs to detect anomalies in seasonal key performance indicators (KPIs). Moreover, VAEs have been combined with other deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to improve their performance on various tasks. For example, [28] combined VAEs with CNNs to extract hierarchical features from images. It is worth mentioning the characteristic of variational autoencoders is incorporated into our model. In summary, VAEs are powerful for generative modeling and have contributed significantly to the field of deep learning. Deep learning with nonparametric clustering is a pioneer work in applying deep belief network to deep clustering. But in deep clustering based on the probabilistic graphical model, more research comes from the application of variational autoencoder (VAE), which combines variational inference and deep autoencoder together.
Multi-View Clustering In practical clustering tasks, the input data usually have multiple views. Multi-view clustering (MVC) methods are proposed to address the limitations of traditional clustering methods by leveraging the complementary information provided by different views of the same instance. Numerous algorithms and techniques have been proposed for MVC, ranging from classic methods such as spectral clustering to more recent deep learning and probabilistic modeling developments. One popular approach is co-training, which was first introduced by [4]. Co-training involves training multiple models, each using a different view of the data, and then using the models to label additional unlabeled data. This approach has been successfully applied in various domains, including text classification [56], image recognition [21], and community detection [24]. Another popular approach is low-rank matrix factorization, which aims to learn a shared latent representation of the data across multiple views. This approach has been explored in various forms, such as structured low-rank matrix factorization [55], tensor-based factorization [41], and deep matrix factorization [54]. Subspace methods have been utilized in MVC in the previous few years, e.g., [7, 13, 26, 30] integrated multiple affinity graphs into a consensus one with the topological relevance considered.
In recent years, deep learning has emerged as a powerful tool for MVC, with various deep neural network architectures proposed for the task. For example, [23] proposed a deep adversarial MVC method, which learns a joint representation of the data across multiple views using adversarial training. [46] proposed deep embedded clustering (DEC) to learn a mapping from the high original feature space to a lower-dimensional one in which a practical objective is optimized. [48] proposed a framework of multi-level feature learning for contrastive multi-view clustering (MFLVC), which combines MVC with contrastive learning to improve clustering effectiveness.
3 The Proposed Method
Self-supervised learning methods for MVC [9, 52, 57, 58] aim at reducing redundant information and achieving sufficient representation from multiple views. In the multi-view scenario, there are also many methods [38, 49] that explore consistent information between views. Despite great efforts, the task of minimizing redundancy and extracting valuable information for subsequent processes while maintaining coherence across perspectives continues to pose significant challenges. In this section, inspired by [9, 39], we extend the principle of the information bottleneck, utilize the reparameterization trick for solving the proposed lower bound and introduce the collaborative training strategy to the MVC scenario. Moreover, we conduct bayes error rates for learned representations to validate the generalization performance of our method.
Problem Statement Given the multi-view data,  $X$ , which consists of  $\nu$  subsets such that  $X^i = [x_1^i; \ldots; x_n^i] \in \mathbb{R}^{n \times d_i}$  is the  $n \times d_i$ -dimensional data of the  $i$ -th view, where  $n$  is the number of instances and  $d_i$  is the number of dimensions of the  $i$ -th view. MVC aims to partition the  $n$  instances into  $K$  clusters.
Analysis of redundancy According to [1, 9], we can assume that  $\mathbf{z}$  contains all necessary information for predicting  $\mathbf{y}$  without having prior knowledge of  $\mathbf{y}$ . This can be achieved by maintaining all the shared information between the two views  $x^{i}$  and  $x^{j}$ . This assumption is grounded on the belief that both views provide similar predictive information. The following formal definition elucidates this concept (See details in Appendix B.2.).
Definition 3.1 [9]  $x^i$  is considered redundant with respect to  $x^j$  for  $\mathbf{y}$  if and only if the mutual information  $I(\mathbf{y};x^i|x^j) = 0$ .
Intuitively,  $x^i$  is considered redundant for a task if it is irrelevant for the prediction of  $\mathbf{y}$  when  $x^j$  is already observed. Whenever  $x^i$  and  $x^j$  are mutually redundant with respect to one another ( $x^i$  is redundant with respect to  $x^j$  for  $\mathbf{y}$ ), we have the following:
Corollary 3.1 Let  $x^i$  and  $x^j$  be two mutually redundant views for a target  $\mathbf{y}$  and let  $z^i$  be a representation of  $x^i$ . If  $z^i$  is sufficient for  $x^j$  ( $I(x^i; x^j | z^i) = 0$ ) then  $z^i$  is as predictive for  $\mathbf{y}$  as the combination of the two views ( $I(x^i x^j; \mathbf{y}) = I(\mathbf{y}; z^i)$ ).
Variational bound The variational bound [18], also known as the Evidence Lower Bound (ELBO), is a widely used optimization objective in Bayesian inference and machine learning. It provides a lower bound on the marginal likelihood of the observed data. Variational bound provides an efficient approximate posterior inference of the latent variable  $z^{i}$  given an observed value  $x^{i}$ , which can be formulated as:
$$
\log q _ {\phi^ {i}} \left(x ^ {i}\right) \geq L \left(\phi^ {i}, \theta^ {i}; x ^ {i}\right) = \mathbb {E} _ {p _ {\theta^ {i}} \left(z ^ {i} \mid x ^ {i}\right)} \left[ - \log p _ {\theta^ {i}} \left(z ^ {i} \mid x ^ {i}\right) + \log q _ {\phi^ {i}} \left(x ^ {i}, z ^ {i}\right) \right], \tag {1}
$$
where  $\theta^i$  and  $\phi^i$  are generative and variational parameters of the  $i$ -th view respectively. Through optimizing the lower bound  $L(\phi^i, \theta^i; x^i)$ , we can approximate the posterior inference of the latent variable  $z^i$  accurately based on an observed value of  $x^i$ . As such, we can obtain more representative embedded features.
3.2 SCMVC: Learning Consistent Information via Varational Lower Bound
Variational inference has greatly contributed to representation learning. Introducing variational inference allows representation learning to be performed in a probabilistic manner thereby providing a principled framework for capturing uncertainty in the learned representation. Variational inference can optimize the generative model, resulting in a more informative data representation. In this section, we analyze the consistent information among views from the vartional inference perspective and propose SCMVC (simple consistent multi-view clustering). We introduce a consistent vartional lower bound to model the latent space distribution of input data more consistent with all views. Given  $\nu$ -views and global embedded features  $\vec{z} = \{z^1, z^2, \dots, z^\nu\}$ , we formalize this concept as the following theorem (Proofs are given in Appendix B.3):
Theorem 3.1 (Consistent Variational Lower Bound) Define the generative model  $q_{\phi}(x^i|\vec{z},y^i)$ , and a posterior as  $p_{\theta}(\vec{z},y^{i}|x^{i}) = p_{\theta}(y^{i}|\vec{z})p_{\theta}(\vec{z}|x^{i})$ . The ELBO for multi-view model can be formulated as
$$
\max  L _ {c o n} ^ {i} = \mathbb {E} _ {p _ {\theta} (\vec {z}, y | x ^ {i})} [ \log q _ {\phi} (x ^ {i} | \vec {z}, y ^ {i}) ] - \gamma D _ {K L} \left(p _ {\theta} (\vec {z}, y ^ {i} | x ^ {i}) \| q _ {\phi} (\vec {z}, y ^ {i})\right), \tag {2}
$$
where the  $y^i$  is the pseudo-label of  $x^i$ ,  $\vec{z} = \{z^1, z^2, \ldots, z^v\}$ , the first term is to maximize the reconstruction probability of input  $x^i$ , i.e., maximize the decoding probability from latent variable  $\vec{z}$  to  $x^i$ . The constraint term is the KL divergence of the approximate from the true posterior to ensure that the distribution of acquired features and labels is consistent with their intractable true posterior. It can be intuitively understood that our goal is to obtain the unique distribution of the embedded feature  $\vec{z}$  of  $x$  for different categories.
For better understanding, Eq. (2) can be expanded out as:
$$
\max  L _ {c o n} ^ {i} = \mathbb {E} _ {x ^ {i} \sim \tilde {p} (x ^ {i})} [ \mathbb {E} _ {\vec {z} \sim \tilde {p} (\vec {z} | x ^ {i})} [ \log q _ {\phi} (x ^ {i} | \vec {z}) - \gamma (\sum_ {i = 1} ^ {\nu} \sum_ {y} p _ {\theta} (y ^ {i} | \vec {z}) \log \frac {p _ {\theta} (\vec {z} | x ^ {i})}{q _ {\phi} (\vec {z} | y ^ {i})} + K L (p _ {\theta} (y ^ {i} | \vec {z}) \| q _ {\phi} (y ^ {i})) ] ], \tag {3}
$$
Overall, the objective function of SCMVC is:
$$
\max  \sum_ {i = 1} ^ {v} \left(L _ {\text {c o n}} ^ {i}\right) = \sum_ {i = 1} ^ {v} \left(L _ {\text {r e c}} ^ {i} - \gamma L _ {K L} ^ {i}\right), \tag {4}
$$
where  $L_{rec}^{i}$  is the reconstruction term and  $L_{KL}^{i}$  is the constraint term which approximates the true posterior.
3.3 SUMVC: Learning Sufficient Representation via Reducing Redundancy
Although SCMVC ensures consistency among views, there is a lack of consideration for reducing mutual redundancy and achieving sufficiency among views, which is important to obtain latent representations suitable for downstream tasks. Next, we advance our work and extend the information bottleneck principle to the multi-view setting. By decomposing the mutual information between  $x^i$  and  $z^i$  (See Appendix B.1.), we can identify two components:
$$
I _ {\theta_ {i}} \left(x ^ {i}, z ^ {i}\right) = I _ {\theta_ {i}} \left(z ^ {i}, x ^ {i} \mid x ^ {j}\right) + I _ {\theta_ {i}} \left(x ^ {j}, z ^ {i}\right),
$$
where the first term denotes superfluous information and the second term denotes signal-relevant information. We take view  $j$  as self-supervised signals to guide the learning of view  $i$ . To ensure that the representation is sufficient for the task, we maximize the mutual information between  $x^j$  and  $z^i$ . As for  $I_{\theta_i}(z^i,x^i|x^j)$ , this term indicates the information in  $z^i$  that is exclusive to  $x^i$  and cannot be predicted by observing  $x^j$ . As we assume mutual redundancy between the two views, this information is deemed irrelevant to the task and could be safely discarded [9]. Overall, to extract sufficient representation of  $z^i$ , we maximize the following formulation:
$$
\max  - I _ {\theta_ {i}} \left(z ^ {i}, x ^ {i} \mid x ^ {j}\right) + I _ {\theta_ {i}} \left(x ^ {j}, z ^ {i}\right), \tag {5}
$$
which can be optimized via maximizing the lower bound in Theorem 3.2.
Theorem 3.2 (Sufficient Representation Lower Bound) Considering a multi-view model, the generative model is  $q_{\phi}(x^i |z^i,y^i)$ . The sufficient representation lower bound can be defined as:
$$
\max  L _ {s u f} ^ {i j} = - D _ {K L} \left(p _ {\theta^ {i}} \left(z ^ {i} \mid x ^ {i}\right), p _ {\theta^ {j}} \left(z ^ {j} \mid x ^ {j}\right)\right) + I _ {\theta^ {i} \theta^ {j}} \left(z ^ {i}, z ^ {j}\right). \tag {6}
$$
Proof 1 Considering  $z^i$  and  $z^j$  on the same domain  $\mathbb{Z}$ , we have
$$
\begin{array}{l} - I _ {\theta} \left(x ^ {i}; z ^ {i} \mid x ^ {j}\right) = - \mathbb {E} _ {x ^ {i}, x ^ {j} \sim p \left(x ^ {i}, x ^ {j}\right)} \mathbb {E} _ {z \sim p _ {\theta^ {i}} \left(z ^ {i} \mid x ^ {i}\right)} \left[ \log \frac {p _ {\theta^ {i}} \left(z ^ {i} = z \mid x ^ {i} = x ^ {i}\right)}{p _ {\theta^ {i}} \left(z ^ {i} = z \mid x ^ {j} = x ^ {j}\right)} \right] (7) \\ = - \mathbb {E} _ {x ^ {i}, x ^ {j} \sim p \left(x ^ {i}, x ^ {j}\right)} \mathbb {E} _ {z \sim p _ {\theta^ {j}} \left(z ^ {i} \mid x ^ {i}\right)} \left[ \log \frac {p _ {\theta^ {j}} \left(z ^ {i} = z \mid x ^ {i} = x ^ {i}\right) p _ {\theta^ {j}} \left(z ^ {j} = z \mid x ^ {j} = x ^ {j}\right)}{p _ {\theta^ {j}} \left(z ^ {j} = z \mid x ^ {j} = x ^ {j}\right) p _ {\theta^ {j}} \left(z ^ {i} = z \mid x ^ {j} = x ^ {j}\right)} \right] (8) \\ = - \mathbb {E} _ {x ^ {i}, x ^ {j} \sim p (x ^ {i}, x ^ {j})} \left[ D _ {K L} \left(p _ {\theta^ {i}} \left(z ^ {i} \mid x ^ {i}\right) \| p _ {\theta^ {j}} \left(z ^ {j} \mid x ^ {j}\right)\right) - D _ {K L} \left(p _ {\theta^ {i}} \left(z ^ {j} \mid x ^ {j}\right) \| p _ {\theta^ {j}} \left(z ^ {j} \mid x ^ {j}\right)\right) \right] (9) \\ \geq - D _ {K L} \left(p _ {\theta^ {j}} \left(z ^ {i} \mid x ^ {i}\right) \| p _ {\theta^ {j}} \left(z ^ {j} \mid x ^ {j}\right)\right). (10) \\ \end{array}
$$
For  $I_{\theta^j}(x^j,z^i)$ , it can be lower bounded as:
$$
\begin{array}{l} I _ {\theta^ {i}} (x ^ {j}, z ^ {i}) \stackrel {(P 2)} {=} I _ {\theta^ {i} \theta^ {j}} (z ^ {i}; z ^ {j} x ^ {j}) - I _ {\theta^ {i} \theta^ {j}} (z ^ {i}; z ^ {j} | x ^ {j}) = I _ {\theta^ {i} \theta^ {j}} (z ^ {i}; z ^ {j} x ^ {j}) = I _ {\theta^ {i} \theta^ {j}} (z ^ {i}; z ^ {j}) + I _ {\theta^ {i} \theta^ {j}} (z ^ {i}; x ^ {j} | z ^ {j}) \\ \geq I _ {\theta^ {\prime} \theta^ {\prime}} (z ^ {i}; z ^ {j}), \\ \end{array}
$$
where  $I_{\theta^i \theta^j}(z^i; x^j | z^j) = 0$  when  $z^j$  is sufficient for  $z^i$ . This occurs whenever  $z^j$  contains all the necessary information about  $z^i$ .
Therefore, Eq. (5) can be optimized via maximizing the lower bound:
$$
- D _ {K L} \left(p _ {\theta^ {i}} \left(z ^ {i} \mid x ^ {i}\right), p _ {\theta^ {j}} \left(z ^ {j} \mid x ^ {j}\right)\right) + I _ {\theta^ {i} \theta^ {j}} \left(z ^ {i}, z ^ {j}\right).
$$
In addition, we adopt collaborative training to allow mutual supervision to better adapt to multi-view scenes. The views are selected in turn to act as the supervisory view, while the remaining views serve as the supervised views. This learning approach could efficiently use the common information and complementary information of multiple views.
SUMVC is derived by incorporating a lower bound on sufficient representation into SCMVC, the objective of which can be defined as follows:
$$
\max  \sum_ {i = 1} ^ {v} \left(L _ {\text {c o n}} ^ {i} + \sum_ {j = 1, j \neq i} ^ {v} \beta L _ {\text {s u f}} ^ {i j}\right), \tag {11}
$$
where the  $i$ -th view can be regarded as the supervisory view, while the  $j$ -th view is the supervised view. Through the collaborative training we could reduce redundancy and obtain consistent information among views. What's more, we align the embedded features and category distribution with the original distribution to obtain information conducive to clustering. Finally, in order to better utilize global features, we run  $K$ -means on  $\vec{z}$  to obtain the clustering results.
3.4 The Reparameterization Trick for Problem Solving
In order to solve the problem, let  $z^i$  be a continuous random variable, and  $z^i \sim p_{\theta^i}(z^i|x^i)$  be some conditional distribution of the  $i$ -th view. It is then often possible to express the random variable  $z^i$  as a deterministic variable  $z^i = f_{\theta^i}(\epsilon^i,x^i)$ , where  $\epsilon^i$  is an auxiliary variable with independent marginal  $q(\epsilon)$  and function  $f_{\theta^i}(\cdot)$  parameterized by  $\theta^i$ . We let  $z^i \sim q_{\phi^i}(z^i|x^i) = N(\mu^i,\sigma^{i2})$ . In this case, a valid reparameterization is  $z^i = \mu^i + \sigma^i\epsilon^i$ , where  $\epsilon^i$  is an auxiliary noise variable  $\epsilon^i \sim N(0,1)$ .
4.1 Connection to VaDE