SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chengliang Liu $^{1}$ , Jie Wen $^{1*}$ , Yabo Liu $^{1}$ , Chao Huang $^{2}$ , Zhihao Wu $^{1}$ , Xiaoling Luo $^{3}$ , Yong Xu $^{1*}$
$^{1}$ School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen
$^{2}$ School of Cyber Science and Technology, Sun Yat-sen University (Shenzhen Campus)
<sup>3</sup>College of Computer Science and Software Engineering, Shenzhen University
{liucl1996, horatio_ng}@163.com, {jiewen_pr, huangchao_08}@126.com,
yaboliu.ug@gmail.com, xiaolingluoo@outlook.com, yongxu@gmail.com
Multi-view learning has become a popular research topic in recent years, but research on the cross-application of classic multi-label classification and multi-view learning is still in its early stages. In this paper, we focus on the complex yet highly realistic task of incomplete multi-view weak multi-label learning and propose a masked two-channel decoupling framework based on deep neural networks to solve this problem. The core innovation of our method lies in decoupling the single-channel view-level representation, which is common in deep multi-view learning methods, into a shared representation and a view-proprietary representation. We also design a cross-channel contrastive loss to enhance the semantic property of the two channels. Additionally, we exploit supervised information to design a label-guided graph regularization loss, helping the extracted embedding features preserve the geometric structure among samples. Inspired by the success of masking mechanisms in image and text analysis, we develop a random fragment masking strategy for vector features to improve the learning ability of encoders. Finally, it is important to emphasize that our model is fully adaptable to arbitrary view and label absences while also performing well on the ideal full data. We have conducted sufficient and convincing experiments to confirm the effectiveness and advancement of our model.
With the increasing popularity of multi-view learning, researchers have discovered that heterogeneous multi-source features obtained from multiple sensors or techniques can provide a richer and more diverse description of samples [1-4]. For example, features can be extracted using scale-invariant feature transform (SIFT) operator and deep neural networks from different perspectives of an image, or features can be extracted from multimedia data, such as images and texts, to express higher-level semantic objects [5-7]. Many multi-view learning methods are based on the assumptions of multiview semantic consistency and feature complementarity [8, 9]. They aim to obtain unique sample labels from multi-source features to maintain the unity of multiple views at the semantic level and to leverage the unique advantages of each view to compensate for the limitations of single-view observation [10-12]. For instance, Nie et al. emphasized the importance of learning a cross-view similarity matrix in their work [13]. Li et al. proposed to learn a common graph in the spectral embedding space for final clustering [14]. Han et al. attempted to compute confidence scores of samples on each view to facilitate classification predictions across multiple views [15]. Lin et
al. proposed dual contrastive prediction for incomplete multi-view representation learning, which performs contrastive learning at both the instance level and category level to ensure cross-view consistency and recover missing information [16]. However, the instance-level contrastive loss of this method only aggregates cross-view representations in the potential space from the perspective of consistency, ignoring the view-private information.
On the other hand, as a classical classification problem, multi-label classification has occupied an important area of pattern recognition for a long time [17]. However, under the upsurge of multi-view learning, new composite problem, i.e., multi-view multi-label classification (MvMLC), are attracting increasing attentions from researchers [18, 19]. As the name suggests, MvMLC can be seen as a supervised subtask within the scope of multi-view learning and can also be considered as an extension of traditional single-view multi-label classification with respect to feature diversity. As a result, existing MvMLC methods not only take into account the basic characteristics of multi-view learning but also consider the requirements of downstream multi-label classification [20, 18, 21]. For example, Zhao et al. proposed a framework called CDMM that mines multi-view consistency and diversity information, which uses the Hilbert-Schmidt Independence Criterion [22] to enhance the diversity of each view and obtain consistent classification results through late fusion [18]. Another advanced non-aligned MvMLC model proposed by Zhao et al. is LVSL (learn view-specific labels), which not only considers the local geometric structure in the original feature space but also introduces a learnable label correlation matrix with low-rank structure [23].
However, these ideal MvMLC methods ignore the possibility of missing multi-view features and labels, making it difficult to achieve good performance on incomplete multi-view weak multi-label data, in which sample's partial views and tags are unknown [24, 25]. In recent years, researchers have proposed some incomplete multi-view weak multi-label classification (iMvWMLC) frameworks to address the adverse effects caused by missing views and labels. For instance, Tan et al. proposed an incomplete multi-view weak label learning model (iMVWL) that decomposes multi-view features into a latent common representation and multiple view-specific basis matrices, and connects the latent representation with labels through a mapping matrix [26]. Additionally, two prior missing indicator matrices are introduced to skip the missing views and labels. Recently, Li et al. developed a non-aligned iMvWMLC model based on traditional matrix factorization, named NAIM3L, to align multiple views in the label space and impose global high-rank and local low-rank constraints on the predicted multi-label matrix [25]. For the double-missing problem, Li et al. introduced a composite indicator matrix for NAIM3L, containing view and label missing information. Another representative deep iMvWMLC framework is DICNet proposed by Liu et al., which applies the contrastive learning technique to aggregate cross-view instances of the same sample and separate instances belonging to different samples [20]. Although DICNet has achieved remarkable improvement, there is a non-negligible category collision problem, i.e., instances belonging to similar samples will be forcibly separated due to this contrastive learning mechanism.
To tackle these issues, we present the Masked Two-channel Decoupling framework (MTD for short), capable of handling cases where partial views and labels are both missing. The motivation behind this framework is that we both want the underlying representations of the same sample in multiple views to be consistent, while also wanting each view to display its own characteristics, creating a difficult balancing act [10]. This is especially challenging when each view of the same sample is represented by only one feature vector. To address this, we explicitly decouple each view's latent feature into two types of features, i.e., shared feature and view-proprietary feature, via two-channel encoders. We also propose a novel cross-channel contrastive loss that narrows the distance between positive pairs (shared instances in different views of the same sample) and expands the distance between negative pairs (any cross-view and channel instance pairs of a sample, except positive pairs). Note that our cross-channel contrastive loss does not involve the comparison among samples, so it can well avoid the above class conflict problem and improve the framework's feature decoupling ability. Additionally, we draw inspiration from famous image patch masks [27] and introduce random mask fragments in the input data to guide the encoder's learning of valuable information. Lastly, we design a weak label-guided graph regularization loss to preserve the geometric structure in the embedding space, accounting for the non-equidistance among samples in label space. In summary, our paper makes three key contributions:
1. We propose a novel masked two-channel decoupling framework (MTD) for the iMvWMLC task, which extracts shared and private features for each view separately and enforces this
Figure 1: Main architecture of our MTD. Masked data  $\{\mathbf{X}^{\prime (v)}\}_{v = 1}^{m}$  is real input of the MTD. And the cross-channel contrastive loss aims to enhance the semantic of 'shared-proprietary' channels.
design with a cross-channel contrastive loss. At the same time, this framework is compatible with arbitrary incomplete multi-view and weak multi-label data.
2. To our best knowledge, we are the first to apply random fragment mask in the field of multiview learning and achieve significant performance gains, which supports a new multi-view vector data enhancement mechanism for the communication.
3. Different from existing graph regularization based approaches, we utilize supervised label information to build a more reliable topological graph, inducing the embedding features extracted by the encoders to preserve the geometric structure among samples.
In this section, we comprehensively introduce our model from the following four aspects, namely, two-channel decoupling framework, weak label-guided graph regularization, random fragment masking mechanism, and multi-label classification. First, for the convenience of description, we briefly introduce the formal problem definition and common notations.
In this section, we define the iMvWMLC task as follows: Given multi-view multi-label data  $\{\{\mathbf{X}^{(v)}\}_{v=1}^{m}, \mathbf{Y}\}_{v=1}^{(m)}$ , where  $\mathbf{X}^{(v)}$  is  $v$ -th view's feature matrix containing  $n$  samples and  $m$  is the number of views. The  $i$ -th row of  $\mathbf{X}^{(v)} \in \mathbb{R}^{n \times d_v}$  denotes the instance of  $i$ -th sample on  $v$ -th view with dimension  $d_v$ .  $\mathbf{Y} \in \{0,1\}^{n \times c}$  is the label matrix with  $c$  categories and  $\mathbf{Y}_{i,j} = 1$  denotes the  $i$ -th sample is tagged as  $j$ -th class. For missing views, we introduce  $\mathbf{W} \in \{0,1\}^{n \times m}$  as the missing-view index matrix, where  $\mathbf{W}_{i,j} = 1$  represent  $j$ -th view of  $i$ -th sample is available, otherwise  $\mathbf{W}_{i,j} = 0$ . For missing labels, we let  $\mathbf{G} \in \{0,1\}^{n \times c}$  be the missing-label index matrix, whose element  $\mathbf{G}_{i,j} = 1$  means  $i$ -th sample's  $j$ -th label is known, otherwise  $\mathbf{G}_{i,j} = 0$ . Finally, we fill the unavailable views and the unknown labels with '0' in the data pre-processing step to get final incomplete multi-view feature  $\{\mathbf{X}^{(v)}\}_{v=1}^{m}$  and weak label matrix  $\mathbf{Y}$ . Our goal is to train a neural network that can perform multi-label classification reasoning on incomplete multi-view data. Subscript access  $\mathbf{B}_{i,j}$ ,  $\mathbf{B}_{i,:}$ , and  $\mathbf{B}_{:,j}$  mean the element, row, and column of any matrix  $\mathbf{B}$ .
2.2 Two-channel decoupling framework
As we know, due to the diversity of multi-view data acquisition methods, each view of the original data may have different feature dimensions, which is not conducive to the parallel execution of deep networks. To this end, we map the heterogeneous raw data into a unified embedding space with dimension  $d_{e}$  through stacked encoders. Different from conventional deep multi-view networks, we
employ two groups of multi-layer perceptrons as two channels, shared channel and view-private channel, to extract shared information and view-proprietary information from raw data, respectively, i.e.,  $\{\mathsf{E}_v^S:\mathbf{X}'(v)\to \mathbf{S}(v)\}_{v = 1}^m$  and  $\{\mathsf{E}_v^O:\mathbf{X}'(v)\to \mathbf{O}(v)\}_{v = 1}^m$ .  $\mathsf{E}_v^S$  and  $\mathsf{E}_v^O$  denote the shared feature encoder and view-proprietary feature encoder of view  $v$ , respectively.  $\mathbf{X}'(v)$  represents the masked input (See Section 2.4).  $\mathbf{S}^{(v)}\in \mathbb{R}^{n\times d_e}$  and  $\mathbf{O}^{(v)}\in \mathbb{R}^{n\times d_e}$  are the extracted multi-view shared feature matrix and view-proprietary feature matrix of  $n$  samples. The architectures of these two channels are the same but serve different purposes. Specifically,  $\mathsf{E}_v^S$  tries to mine common features across views, which preserve the sample's basic properties held by all views, while  $\mathsf{E}_v^O$  focuses on extracting features unique to each view. This design decomposes the discriminative information of each view into two parts, which can satisfy the multi-view consistency and complementarity assumptions at the same time. Of course, it is difficult to achieve the above purposes with only two encoding channels due to the lack of guidance, so we propose a multi-view cross-channel contrastive loss  $\mathcal{L}_{ccc}$  to guide the separation of these two kinds of features as follows:
$$
\mathcal {L} _ {c c c} = \sum_ {i = 1} ^ {n} \frac {\frac {1}{3 N ^ {2} - N} \left[ 2 \sum_ {u = 1} ^ {m} \sum_ {v = 1} ^ {m} \mathbb {1} _ {[ \Upsilon ]} \mathcal {S} \left(s _ {i} ^ {(u)} , o _ {i} ^ {(v)}\right) ^ {2} + \sum_ {u = 1} ^ {m} \sum_ {v \neq u} ^ {m} \mathbb {1} _ {[ \Upsilon ]} \mathcal {S} \left(o _ {i} ^ {(u)} , o _ {i} ^ {(v)}\right) ^ {2} \right]}{\frac {1}{N ^ {2} - N} \sum_ {u = 1} \sum_ {v = u} \mathbb {1} _ {[ \Upsilon ]} \left[ \left(\mathcal {S} \left(s _ {i} ^ {(u)}, s _ {i} ^ {(v)}\right) + 1\right) / 2 \right]}, \tag {1}
$$
where  $\mathcal{S}$  is the similarity measure function:  $S(x,y) = \frac{x^T y}{\|x\|_2\cdot\|y\|_2}$  and  $\mathbb{1}_{[\Upsilon]}$  is the condition function:  $\mathbb{1}_{[\Upsilon]} = 1$  if condition  $\{\Upsilon : \mathbf{W}_{i,u} \mathbf{W}_{i,v} = 1\}$  is true, 0 otherwise.  $N = \sum_{u,v} \mathbf{W}_{i,u} \mathbf{W}_{i,v}$  denotes the number of valid instance pairs. That is to say we only consider instance pairs that are not missing on both sides.  $s_i^{(v)}$  and  $o_i^{(v)}$  are features corresponding to  $i$ -th sample in the shared feature  $\mathbf{S}^{(v)}$  and view-proprietary feature  $\mathbf{O}^{(v)}$ , respectively. According to Eq. (1), our cross-channel contrastive loss consists of two parts, namely, the numerator is the mean of negative pairs' similarities and the denominator is that of positive pairs' similarities. To be more specific, we pair instances from  $2N$  channels together, where positive pairs consist of shared features from different views, and the remaining pairs are considered negative. We aim to maximize the similarity between the positive pairs of shared features and minimize the similarity between the negative pairs. This approach fulfills the design objective of the dual-channel model, which is to encourage consistency between shared features from different views while maintaining a clear distinction between each view's view-proprietary feature and other shared or view-proprietary features.
To further enhance the feature extraction capability of the encoders, we introduce stacked decoders to restore the embedding features extracted by the two-channel encoders to original feature space, i.e.,  $\{\mathsf{D}_v:\mathbf{S}^{(v)}\in \mathbb{R}^{n\times d_e}\to \bar{\mathbf{X}}^{(v)}\in \mathbb{R}^{n\times d_v}\}_{v = 1}^m$  , where  $\mathsf{D}_v$  is the decoder corresponding to  $v$  -th view.  $\bar{\mathbf{X}}^{(v)}$  is the reconstructed feature. Finally, we measure the reconstruction quality using the weighted mean square error loss:
$$
\mathcal {L} _ {r e} = \frac {1}{n} \sum_ {v = 1} ^ {m} \sum_ {i = 1} ^ {n} \frac {1}{d _ {v}} \left\| \bar {\mathbf {X}} _ {i,:} ^ {(v)} - \mathbf {X} _ {i,:} ^ {(v)} \right\| _ {2} ^ {2} \mathbf {W} _ {i, v}, \tag {2}
$$
where  $\mathbf{W}$  is introduced to mask the unavailable instances. With the  $\mathbf{S}^{(v)}$  and  $\mathbf{O}^{(v)}$  on each view, we can naturally perform cross-view fusion to obtain the unique shared representation  $\bar{\mathbf{S}}$  and  $\bar{\mathbf{O}}$  of all samples [28, 20]:
$$
\bar {\mathbf {S}} _ {i,:} = \sum_ {v = 1} ^ {m} \frac {\mathbf {S} _ {i , :} ^ {(v)} \mathbf {W} _ {i , v}}{\sum_ {v} \mathbf {W} _ {i , v}}, \bar {\mathbf {O}} _ {i,:} = \sum_ {v = 1} ^ {m} \frac {\mathbf {O} _ {i , :} ^ {(v)} \mathbf {W} _ {i , v}}{\sum_ {v} \mathbf {W} _ {i , v}}. \tag {3}
$$
Successively, we try to combine shared information and proprietary information to learn consistent representations of samples. Here, unlike the commonly used addition or connection operations, we adopt a novel feature interaction approach, i.e., enhancing the shared information via the proprietary information:
$$
\mathbf {Z} _ {i, j} = \theta (\bar {\mathbf {O}} _ {i, j}) \cdot \bar {\mathbf {S}} _ {i, j}, \tag {4}
$$
where  $\theta$  is sigmoid activation function and  $\mathbf{Z} \in \mathbb{R}^{n \times d_e}$  is the final fusion representation.
2.3 Weak label-guided graph regularization
Many unsupervised multi-view learning methods apply graph regularization technique to maintain the intrinsic structure of data, based on the assumption that two similar samples in the original feature
space should also be similar in the latent space, which has shown good performance in unsupervised scenarios [29-31]. When transferring this assumption to multi-label classification tasks with the characteristics of supervised tasks, we can get a new assumption: two similar samples in the label space should also be similar in the embedding space. Based on this, we attempt to guide the feature extraction process of the encoders by imposing a graph constraint with the supervised information. First, we calculate similarity matrix according to the weak label matrix  $\mathbf{Y}$ :
$$
\mathbf {T} _ {i, j} = \frac {\mathbf {C} _ {i , j} \cdot \left(y _ {i} y _ {j} ^ {T}\right)}{\mathbf {C} _ {i , j} \cdot \left(y _ {i} y _ {j} ^ {T}\right) + \eta}, \tag {5}
$$
where  $\mathbf{T} \in [0,1]^{n \times n}$  is the sample similarity graph, describing the similarity between any two samples.  $\mathbf{C}_{i,j} = \mathbf{G}_{i,:}\mathbf{G}_{j,:}^T$  is the number of labels available for both samples  $i$  and  $j$ .  $y_{i}$  and  $y_{j}$  denote the  $i$ -th and  $j$ -th rows of  $\mathbf{Y}$ . And  $\eta$  is a constant, empirically set to 100 for simplicity. Taking into account the fact that  $C_{ij}$  is much larger than  $y_{i}y_{j}^T$  on most datasets, when the number of categories is large, even if two samples only have one same label, it means that they are much more similar than other sample pairs.
With such a similarity graph, we can constrain the distance between any two samples in the embedding feature space to achieve the purpose of structure preservation by following loss:
$$
\mathcal {L} _ {g c} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \left\| \mathbf {Z} _ {i,:} - \mathbf {Z} _ {j,:} \right\| _ {2} ^ {2} \mathbf {T} _ {i, j}. \tag {6}
$$
In view of Eq. (6), the more similar two samples are, the greater the penalty weight they are imposed. In order to improve the computational efficiency of the model in the GPU, we rewrite Eq. (6) in the form of matrix product:
$$
\mathcal {L} _ {g c} = \frac {1}{n ^ {2}} T r (\mathbf {Z} ^ {T} \mathbf {L} \mathbf {Z}), \tag {7}
$$
where  $Tr(\cdot)$  is the trace operation.  $\mathbf{L}$  denotes the Laplacian matrix calculated by  $\mathbf{L} = \mathbf{D} - \mathbf{T}$ , where  $\mathbf{D}$  is a diagonal matrix with diagonal element  $\mathbf{D}_{i,i} = \sum_{j=1}^{n} \mathbf{T}_{i,j}$ .
2.4 Masking random fragment of feature
As we all know, the original data contains massive redundant information, which inevitably interferes with the feature extraction process of the model. Inspired by the well-known masked autoencoders (MAE) [27] that randomly mask image patches, we attempt to apply a masking mechanism to the input multi-view vector data. To this end, for any view  $v$ , we initialize a matrix  $\mathbf{M}^{(v)} \in \{0,1\}^{n \times d_v}$  whose elements are all 1 at the same size as the raw data  $\mathbf{X}^{(v)}$ , and randomly sample  $n$  integers  $[b_1, b_2, \dots, b_n]$  from the range of  $[1, d_v - l]$  as the mask starting point of each instance in view  $v$ , where  $l$  is the mask length. Then the  $b_i$ -th to  $(b_i + l)$ -th bits of feature  $\mathbf{M}_{i,:}^{(v)} \in \mathbb{R}^{d_v}$  are set to 0 to get the final mask matrix  $\mathbf{M}^{(v)}$ . This matrix stores consecutive mask segments starting at random, enabling local deactivation of feature vectors. We use  $m$  mask matrices to mask the original input data, and get corresponding masked feature matrices as the input of the MTD:
$$
\left\{\mathbf {X} ^ {\prime (v)} \right\} _ {v = 1} ^ {m} = \left\{\mathbf {X} ^ {(v)} \odot \mathbf {M} ^ {(v)} \right\} _ {v = 1} ^ {m}, \tag {8}
$$
where  $\odot$  means the element-wise multiplication.
At first glance, this is similar to MAE masking random image patches to encourage the model to learn useful features. The difference is that our MTD only partially masks the input features but does not do anything to the encoders, i.e., the encoders still works as if processing the unmasked data. In addition, in our model, we uniformly set the mask rate  $\sigma = l / d_v$  to 0.25 for all datasets instead of a larger masking rate like MAE, because we notice the difference that the information density of vector data is much larger than that of image data. Extensive experiments are performed in Section 3.5 to verify the effectiveness of this masking approach.
2.5 Weighted multi-label classification and overall loss function
With the fused embedding representation  $\mathbf{Z}$ , a fully connected layer is employed as a classifier to map  $\mathbf{Z}$  to the label space to get the prediction result:
$$
\mathbf {P} = \theta (\mathbf {Z} \mathcal {W} + \mathcal {B}), \tag {9}
$$
Algorithm 1 Training process of MTD
Input: Incomplete multi-view data  $\{\mathbf{X}^{(v)}\}_{v = 1}^m$ , missing-view index matrix  $\mathbf{W}$ , weak label  $\mathbf{Y}$ , missing-label index matrix  $\mathbf{G}$ .
Output: Prediction results P.
1: Initialize model parameters and set hyperparameters  $(\alpha, \beta, \gamma,$  learning rate, and training epochs  $e)$ .
2:  $t = 0$ .
3: while  $t < e$  do
4: Construct random feature mask matrices  $\{\mathbf{M}^{(v)}\}_{v = 1}^{m}$
5: Compute masked input data  $\{\mathbf{X}^{\prime (v)}\}_{v = 1}^{m}$  by Eq. (8).
6: Extract shared embedding feature  $\{\mathbf{S}^{(v)}\}_{v = 1}^{m}$  and view-private embedding feature  $\{\mathbf{O}^{(v)}\}_{v = 1}^{m}$  by two-channel encoders  $\{\mathsf{E}_v^S\}_{v = 1}^m$  and  $\{\mathsf{E}_v^O\}_{v = 1}^m$ , respectively.
7: Compute fused shared embedding feature  $\overline{\mathbf{S}}$  and fused view-proprietary feature  $\overline{\mathbf{O}}$  by Eq. (3).
8: Compute final fused representation  $\mathbf{Z}$  by Eq. (4).
9: Compute similarity graph  $\mathbf{T}$  by Eq. (5) and corresponding Laplacian matrix  $\mathbf{L}$ .
10: Compute cross-channel contrastive loss  $\mathcal{L}_{ccc}$  by Eq. (1), reconstruction loss  $\mathcal{L}_{re}$  by Eq. (2), and graph embedding loss  $\mathcal{L}_{gc}$  by Eq. (7).
11: Obtain predictions  $\mathbf{P}$  by Eq. (9) and compute classification loss  $\mathcal{L}_{mc}$  by Eq. (10).
12: Compute total loss  $\mathcal{L}_{all}$  by Eq. (11).
13: Update network parameters.
14:  $t = t + 1$
15: end while
where  $\mathcal{W} \in \mathbb{R}^{2d_e \times c}$  and  $\mathcal{B} \in \mathbb{R}^{n \times c}$  are learnable parameters of classifier and  $\mathbf{P} \in \mathbb{R}^{n \times c}$  is our predicted score matrix. In single-label classification tasks, the cross-entropy loss is usually adopted to guide the learning of model, while in multi-label classification tasks, the prediction of each category is regarded as an independent binary classification problem. Besides, considering the unknown labels in label matrix, we use following multi-label cross-entropy loss as our main classification loss: