SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ming-Kun Xie $^{1,2}$ , Jia-Hao Xiao $^{1,2}$ , Hao-Zhe Liu $^{1,2}$ , Gang Niu $^{3}$ , Masashi Sugiyama $^{3,4}$ , Sheng-Jun Huang $^{1,2*}$
$^{1}$ Nanjing University of Aeronautics and Astronautics  $^{2}$ MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
$^{3}$ RIKEN Center for Advanced Intelligence Project
<sup>4</sup>The University of Tokyo, Tokyo, Japan
{mkxie, jiahaoxiao, haozheliu, huangsj}@nuaa.edu.cn
gang.niu.ml@gmail.com sugi@k.u-tokyo.ac.jp
Pseudo-labeling has emerged as a popular and effective approach for utilizing unlabeled data. However, in the context of semi-supervised multi-label learning (SSMLL), conventional pseudo-labeling methods encounter difficulties when dealing with instances associated with multiple labels and an unknown label count. These limitations often result in the introduction of false positive labels or the neglect of true positive ones. To overcome these challenges, this paper proposes a novel solution called Class-Aware Pseudo-Labeling (CAP) that performs pseudolabeling in a class-aware manner. The proposed approach introduces a regularized learning framework incorporating class-aware thresholds, which effectively control the assignment of positive and negative pseudo-labels for each class. Notably, even with a small proportion of labeled examples, our observations demonstrate that the estimated class distribution serves as a reliable approximation. Motivated by this finding, we develop a class-distribution-aware thresholding strategy to ensure the alignment of pseudo-label distribution with the true distribution. The correctness of the estimated class distribution is theoretically verified, and a generalization error bound is provided for our proposed method. Extensive experiments on multiple benchmark datasets confirm the efficacy of CAP in addressing the challenges of SSMLL problems. The implementation is available at https://github.com/milkxie/SSMLL-CAP.
In single-label supervised learning, each instance is assumed to be associated with only one class label, while many realistic scenarios may be multi-labeled, where each instance consists of multiple semantics. For example, an image of a nature landscape often contains the objects of sky, cloud, and mountain. Multi-label learning (MLL) is a practical and effective paradigm for handling examples with multiple labels. It trains a classifier that can predict all the relevant labels for unseen instances based on the given training examples. A large number of recent works have witnessed the great progress that MLL has made in many practical applications [27], e.g., image annotation [21], protein subcellular localization [28], and visual attribute recognition [32].
Thanks to its powerful capacity, the deep neural network (DNN) has become a prevalent learning model for handling MLL examples [34]. Unfortunately, it requires a large number of precisely labeled
*Correspondence to: Sheng-Jun Huang (huangsj@nuaa.edu.cn).
examples to achieve favorable performance. This leads to a high cost of manual annotation, especially when the dataset is large and the labeling task must be carried out by an expert. Given that it is hard to train an effective DNN based on a small subset of training examples, it is rather important to exploit the information from unlabeled instances. The problem has been formalized as a learning framework called semi-supervised multi-label learning (SSMLL), which aims to train a classifier based on a small set of labeled MLL examples and a large set of unlabeled ones.
Compared to semi-supervised learning (SSL) that has made great progress [4, 38], SSMLL has received relatively less attention in the context of deep learning. Generally, there are still three main challenges towards the development of SSMLL. Firstly, since each instance is associated with multiple labels and the number is unknown, the commonly used pseudo-labeling strategy that selects the most probable label or the top- $k$  probable labels cannot be applied to the SSMLL problems. It would face the dilemma of either introducing false positive labels or neglecting true positive ones. Secondly, due to the intrinsic class-imbalance property of MLL data, it is hard to achieve favorable performance by using a fixed threshold for each instance. Thirdly, recent studies have mainly focused on multi-label learning with missing labels (MLML) [11, 17, 2] scenarios, where each training instance is assumed to be assigned with a subset of true labels. Unfortunately, these methods often fail to achieve favorable performance, or cannot even be applied to the SSMLL scenarios, since most of them were designed under the assumption of MLML.
To solve these challenges, in this paper, we propose a novel Class-Aware Pseudo-labeling (CAP) method for handling the SSMLL problems. Unlike the existing methods, we perform pseudo-labeling in a class-aware manner to avoid estimating the number of true labels for each instance, which can be very hard in practice. Specifically, a regularized learning framework is proposed to determine the numbers of positive and negative pseudo-labels for each class based on the class-aware thresholds. Given that the true class distribution is unknown, we alternatively determine the thresholds based on the estimated class distribution of labeled data, which can be a tight approximation according to our observation. Our theoretical results show the correctness of estimated class distribution and provide a generalization error bound for CAP. Extensive experimental results on multiple benchmark datasets with a variety of comparing methods validate that the proposed method can achieve state-of-the-art performance.
Thanks to the powerful learning capacity of DNNs, MLL has made great advances in the context of deep learning. Some methods designed architectures [6] or training strategies [22, 49] to exploit the label correlations. Some other methods designed sophisticated loss functions to improve the performance of MLL [34]. The last group of methods designed specific architectures to capture the objects related to semantic labels. Global-average-pooling (GAP) based models [33] and attention-based models [22, 26] are two groups of representative methods.
There are relatively few works that study how to improve the performance of deep models in SSMLL scenarios. Instead of end-to-end training, the only deep SSMLL method [43] performed the two-stage training, which first used a DNN to extract features, and then used a linear model to perform classification. [37] proposed a deep sequential generative model to handle the noisy labels collected by crowdsourcing and unlabeled data simultaneously. [20] focused on the transductive and non-deep scenario, and thus cannot be applied to our setting. The method proposed by [39] utilized the graph neural network (GNN) to deal with SSMLL data with graph structures. In contrast, there are many works that trained linear models to solve the SSMLL problems [5, 15, 42, 52, 50, 40]. M3DN was proposed to deal with SSMLL data in multi-modal multi-instance learning scenarios by adopting optimal transport technique [48].
Pseudo-labeling has become a popular method in semi-supervised learning (SSL). The idea was firstly applied to semi-supervised training of deep neural networks [23]. Subsequently, a great deal of works have been devoted to improving the quality of pseudo-labels either by adopting consistency regularization [4, 38], or by using distribution alignment [30, 3]. The contrastive learning technique has been applied to improve the performance of SSL [24]. To improve the reliability of pseudo-labels, an uncertainty-aware pseudo-labeling method proposed in [35] selected reliable pseudo-labels based on the prediction uncertainty. Recent studies have also paid attention to dealing with the class-imbalance problem of pseudo-labeling in SSL scenarios [18, 45, 14]. Unlike FixMatch that
selects unlabeled examples with a fixed threshold, Dash [47] selected unlabeled examples with a dynamic threshold, with the goal of achieving better pseudo-labeling performance. FlexMatch [51] was proposed to select unlabeled examples for every class according to the current learning status of the model. Several works have been explored the idea of selecting different thresholds for different classes to improve the performance of SLL [13, 14, 44]. However, these methods are designed for the multi-class single-label scenario, and cannot be directly applied to the multi-label scenario.
In order to reduce the annotation cost, a cost-effective strategy is to assign a subset of true labels to each instance. For example, [11] designed a partial binary cross entropy (BCE) loss that re-weights the losses of known labels. As an extreme case of MLML, single positive multi-label learning (SPML) [8, 53, 41] assumes that only one of multiple true labels can be observed during the training stage. The pioneering work [8] trains DNNs by simply treating unobserved labels as negative ones and utilizes the regularization to alleviate the harmfulness of false negative labels. [53] propose asymmetric pseudo labeling technique to recover true labels.
3 The Method
In the SSMLL problem, let  $\pmb{x} \in \mathcal{X}$  be a feature vector and  $\pmb{y} \in \mathcal{Y}$  be its corresponding label vector, where  $\mathcal{X} = \mathbb{R}^d$  is the feature space and  $\mathcal{Y} = \{0,1\}^q$  is the label space with  $q$  possible class labels. Here,  $y_{k} = 1$  indicates the  $k$ -th label is relevant to the instance, while  $y_{k} = 0$ , otherwise. Suppose that we are given a labeled dataset with  $n$  training examples  $\mathcal{D}_l = \{(x_i,y_i)\}_{i=1}^n$  and an unlabeled dataset with  $m$  training instances  $\mathcal{D}_u = \{x_j\}_{j=1}^m$ . Our goal is to train a DNN  $f(\pmb{x};\theta)$  based on the labeled dataset  $\mathcal{D}_l$  and unlabeled dataset  $\mathcal{D}_u$ , where  $\theta$  is the parameter of the network. For notational simplicity, we omit the notation  $\theta$  and let  $f(\pmb{x})$  be the predicted probability distribution over classes and  $f_k(\pmb{x})$  be the predicted probability of the  $k$ -th class for input  $\pmb{x}$ .
Typical multi-label learning methods usually train a DNN with the commonly used binary cross entropy (BCE) loss, which decomposes the original task into multiple binary classification problems. Unfortunately, BCE loss often suffers from positive-negative imbalance issue. To mitigate this problem, we adopt the asymmetric loss (ASL) [34], which is a variant of focal loss with different focusing parameters for positive and negative instances. In our experiment, we found it works better than BCE loss. Formally, given the predicted probabilities  $f(\pmb{x})$  on instance  $\pmb{x}$ , the ASL loss is defined as
$$
\mathcal {L} (f (\boldsymbol {x}), \boldsymbol {y}) = \sum_ {k = 1} ^ {q} y _ {k} \ell_ {1} \left(f _ {k} (\boldsymbol {x})\right) + \left(1 - y _ {k}\right) \ell_ {0} \left(f _ {k} (\boldsymbol {x})\right), \tag {1}
$$
Here,  $\ell_1(f_k) = -(1 - f_k)^{\lambda_1}\log (f_k)$  and  $\ell_0(f_k) = -(f_k)^{\lambda_0}\log (1 - f_k)$  represent the losses calculated on positive and negative labels, where  $\lambda_{1}$  and  $\lambda_0$  are positive and negative focusing parameters.
3.1 Instance-Aware Pseudo-Labeling
The loss function may not be the best choice to solve the SSMLL problem, since besides the labeled training examples, there still exist a large number of unlabeled training examples. To exploit the information of unlabeled data, inspired by recent SSL works [4, 38], an intuitive strategy is assigning the unlabeled instances with pseudo-labels based on the model outputs. Formally, we define the unlabeled loss  $\mathcal{L}_u$  as
$$
\mathcal {L} _ {u} (f (\boldsymbol {x}), \hat {\boldsymbol {y}}) = \sum_ {k = 1} ^ {q} \hat {y} _ {k} \ell_ {1} (f _ {k} (\boldsymbol {x})) + (1 - \hat {y} _ {k}) \ell_ {0} (f _ {k} (\boldsymbol {x})),
$$
where  $\hat{\pmb{y}} = [\hat{y}_1,\dots ,\hat{y}_j]^\top$  represents the pseudo-label vector for instance  $\pmb{x}$
In the above formulation, the most significant element is how to obtain the pseudo-labels  $\hat{\pmb{y}}$  that significantly affects the final performance of SSMLL. Most of existing pseudo-labeling methods are performed in an instance-aware manner by assigning pseudo-labels to each unlabeled instance based on its probability distribution. Below, we briefly review three instance-aware pseudo-labeling strategies that can be applied to the SSMLL problems. The most commonly used strategy adopted by the SSL method called FixMatch [38] is to select the most probable label as the ground-truth one:
$$
\hat {y} _ {k} = \left\{ \begin{array}{l l} 1 & \text {i f} k = \underset {c \in [ q ]} {\arg \max } f _ {c} (\boldsymbol {x}), \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
One advantage of the strategy is that it is likely to safely identify a true label for each unlabeled training instance. Unfortunately, it is obvious that the strategy would neglect multiple true labels. Generally, it transforms the unlabeled dataset into another learning scenario called single positive multi-label learning (SPML) [8], where only one of multiple positive labels is available for each instance. A straightforward strategy is to simply treat unobserved labels as negative ones. Although this strategy enables us to train a classifier based on SPML data, it would introduce a large number of false negative labels, leading to unfavorable performance.
The second choice is an improved version of the above strategy, which selects the top  $l$  probable labels as the true ones:
$$
\hat {y} _ {k} = \left\{ \begin{array}{l l} 1 & \text {i f} f _ {k} (\boldsymbol {x}) \geq \tau^ {l}, \\ 0 & \text {o t h e r w i s e}, \end{array} \right. \tag {3}
$$
where  $\tau^l$  is the  $l$ -th predicted probability in a descending order. The strategy conducts a competition among labels, and selects top  $l$  winners. The optimal solution is to set  $l$  as the true number of positive labels for each unlabeled instance. Unfortunately, since the true number is unknown in practice, as a compromise, we set  $l$  as the average number of positive labels per instance. Given that the true number does not always equal to the average number, it would be caught in a dilemma of either introducing false positive labels or neglecting true positive ones.
The last choice is to adopt an instance-aware threshold  $\tau_{j}$  that separates positive and negative labels for each unlabeled instance.
$$
\hat {y} _ {j k} = \left\{ \begin{array}{l l} 1 & \text {i f} f _ {k} (\boldsymbol {x} _ {j}) \geq \tau_ {j}, \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {4}
$$
Compared to the above methods, this strategy achieves a strong flexibility that allows it to assign different numbers of positive labels to different instances. A potential limitation is that it is hard to find the optimal thresholds for different instances. In practice, a feasible solution is to adopt a global threshold, that is  $\forall j\in [m],\tau_{j} = \tau$ . Obviously, it is impossible to adopt a global threshold that is optimal for all instances, especially considering the class-imbalance property of MLL data. In general, a large threshold often leads to a small recall score of tail classes, which indicates that less positive labels would be identified. While a small threshold often results in a small precision score of head classes, which indicates besides positive labels, a great deal of negative ones would be treated as positive ones. The dilemma prevents the model from obtaining favorable performance.
3.2 Class-Aware Pseudo-Labeling
As discussed above, in many real-world scenarios, it is really difficult to acquire the true number of positive labels for each instance. This leads the instance-aware pseudo-labeling methods to be caught in the dilemma of either mislabeling false positive labels or neglecting true positive labels, resulting in a noticeable decrease of the model performance.
To solve this issue, we propose a regularized learning framework to assign pseudo-labels in a class-aware manner. Formally, we reformulate the optimization problem of SSMLL as
$$
\begin{array}{l} \min  _ {\boldsymbol {y}, \theta} \sum_ {i = 1} ^ {n} \sum_ {k = 1} ^ {q} y _ {i k} \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {i}\right)\right) + \left(1 - y _ {i k}\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {i}\right)\right) \\ + \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \hat {y} _ {j k} \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) + \left(1 - \hat {y} _ {j k}\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) - \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \alpha_ {k} \hat {y} _ {j k} + \beta_ {k} \left(1 - \hat {y} _ {j k}\right), \tag {5} \\ \end{array}
$$
s.t.  $\forall j\in [m],\hat{\pmb{y}}_j = [y_{j1},\dots ,y_{jq}]^\top \in \{0,1\} ^q,$
$$
\forall k \in [ q ], \alpha_ {k} > 0, \beta_ {k} > 0,
$$
where  $\alpha_{k}$  and  $\beta_{k}$  are class-aware regularized parameters to control how many positive and negative labels would be included into model training for class  $k$ . Below, we primarily provide a solution of the optimization problem Eq.(5), and then discuss how to set parameters  $\alpha_{k}$  and  $\beta_{k}$  to capture the true class distribution of unlabeled examples.
Alternative Search It is hard to directly solve the optimization problem Eq.(5), since there are two sets of variables. A feasible solution is to adopt the alternative convex search [1, 54] strategy that optimizes a group of variables by fixing the other group of variables.
(a) Different Pseudo-Labling Strategies
Figure 1: (a) An illustration of the comparison between instance-aware and class-aware pseudolabeling methods. (b) The curves of the estimated and true class proportions on COCO and VOC. By using the CAT strategy, CAP can provide high-quality pseudo-labels by approximating the true class distribution. This can be validate by the results in (b), where the empirical and true class proportions of positive labels show high-level consistency.
(b) Class Proportions
Suppose that pseudo labels  $\hat{\pmb{y}}$  are given, then the optimization problem Eq.(5) can be transformed into an ordinary loss by treating the pseudo labels as the true ones:
$$
\min  _ {\theta} \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} \left(f \left(\boldsymbol {x} _ {i}\right), \boldsymbol {y} _ {i}\right) + \frac {1}{m} \sum_ {j = 1} ^ {m} \mathcal {L} _ {u} \left(f \left(\boldsymbol {x} _ {j}\right), \hat {\boldsymbol {y}} _ {j}\right), \tag {6}
$$
which can be solved by applying the stochastic gradient decent (SGD) method.
With the parameters  $\theta$  fixed, we reformulate the optimization problem with respect to  $\hat{\pmb{y}}$  as
$$
\min  _ {\hat {y}} \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \hat {y} _ {j k} \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) + \left(1 - \hat {y} _ {j k}\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) - \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \alpha_ {k} \hat {y} _ {j k} + \beta_ {k} \left(1 - \hat {y} _ {j k}\right). \tag {7}
$$
Consider that  $\hat{y}_k$  is assume to be one or zero, we can obtain the following solution:
$$
\hat {y} _ {k} = \left\{ \begin{array}{l l} 1 & \text {i f} f _ {k} (\boldsymbol {x}) \geq \tau \left(\alpha_ {k}\right), \\ 0 & \text {i f} f _ {k} (\boldsymbol {x}) \leq \tau \left(\beta_ {k}\right), \\ - 1 & \text {o t h e r w i s e}, \end{array} \right. \tag {8}
$$
where  $\tau (\alpha_{k}) = \exp (-\alpha_{k})$  and  $\tau (\beta_{k}) = 1 - \exp (-\beta_{k})$  are two class-aware thresholds, and  $\hat{y}_k = -1$  means that the label  $\hat{y}_k$  would not be used for model training.
3.3 Class-Distribution-Aware Thresholding
An important problem is how to set the thresholds  $\tau (\alpha_{k})$  and  $\tau (\beta_{k})$ , which determine the numbers of positive and negative pseudo-labels for every class  $k$ . In order to capture the true class distribution, we propose the Class-distribution-Aware Thresholding (CAT) strategy to determine  $\tau (\alpha_{k})$  and  $\tau (\beta_{k})$ . Suppose that we are given  $\pmb{y}_j,\forall j\in [m]$ , i.e., the true label vectors of unlabeled training instances. By solving the following equation, we can obtain  $\tau (\alpha_{k})$  and  $\tau (\beta_{k})$  that capture the true class distribution of unlabeled data.
$$
\frac {\sum_ {j = 1} ^ {m} \mathbb {I} (f _ {k} (\boldsymbol {x} _ {j}) \geq \tau (\alpha_ {k}))}{m} = \gamma_ {k} ^ {*}, \quad \frac {\sum_ {j = 1} ^ {m} \mathbb {I} (f _ {k} (\boldsymbol {x} _ {j}) \leq \tau (\beta_ {k}))}{m} = \rho_ {k} ^ {*},
$$
where  $\gamma_k^* = \frac{\sum_{j=1}^m \mathbb{I}(y_{jk} = 1)}{m}$  and  $\rho_k^* = \frac{\sum_{j=1}^m \mathbb{I}(y_{jk} = 0)}{m}$  are respectively the proportions of positive and negative labels in unlabeled data for class  $k$ . Although during the training process, the true labels of unlabeled instances are inaccessible, our observation shows that the estimated class distribution, i.e.,
the class proportions of positive and negative labels in labeled examples, can tightly approximate the true class distribution. As shown Figure 1 (b), we illustrate the proportions of positive labels in labeled examples and unlabeled examples for every class  $k$  on two benchmark datasets COCO and VOC. The proportions of labeled examples are respectively  $p = 0.05$  and  $p = 0.1$  for COCO and VOC. From the figures, it can be observed that even with a small proportion of labeled examples  $(p = 0.05)$ , it achieves a nearly complete overlap between the estimated and true curves, which validates that the estimated class distribution can be a tight approximation of the true one. This motivate us to alternatively utilize the estimated class distribution to solve the solutions for  $\tau(\alpha_k)$  and  $\tau(\beta_k)$ :
$$
\frac {\sum_ {j = 1} ^ {m} \mathbb {I} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right) \geq \tau \left(\alpha_ {k}\right)\right)}{m} = \hat {\gamma} _ {k}, \quad \frac {\sum_ {j = 1} ^ {m} \mathbb {I} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right) \leq \tau \left(\beta_ {k}\right)\right)}{m} = \hat {\rho} _ {k}, \tag {9}
$$
where  $\hat{\gamma}_k = \frac{\sum_{i=1}^n \mathbb{I}(y_{ik} = 1)}{n}$  and  $\hat{\rho}_k = \frac{\sum_{i=1}^n \mathbb{I}(y_{ik} = 0)}{n}$  are respectively the proportions of positive and negative labels in labeled data for class  $k$ . Figure 1 provides an illustration of the comparison between three instance-aware pseudo-labeling methods and the CAP method. By utilizing CAT strategy, CAP is expected to assign pseudo-labels with the class distribution that approximates the true one.
In practice, to further improve the performance of CAP, one feasible solution is to discard a fraction of unreliable pseudo-labels with relatively low confidence, which may have a negative impact on the model training. Specifically, for any class  $k \in [q]$ , we select top  $\eta_1 \cdot \hat{\gamma}_k$  and  $\eta_0 \cdot \hat{\rho}_k$  proportion probable pseudo-labels, where  $\eta_1, \eta_0 \in [0,1]$  are two parameters to control the reliable intervals of pseudo-labels. By substituting the two terms into the right sides of Eq.(9), we can obtain the thresholds correspondingly. In Section 5.4, we perform ablation experiments to study the influence of reliable intervals on the model performance.
In this section, we perform theoretical analyses for the proposed method. In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of estimated class distribution. Our work focuses on the latter. Consider an extreme case, where the model predictions are perfect, i.e., the confidences of positive labels are always greater than that of negative labels. In such a case, we still need an appropriate threshold to precisely separate the positive and negative labels. This implies that we need to capture the true class distribution of unlabeled data in order to achieve desirable pseudo-labeling performance.
4.1 Correctness of the Estimated Class Distribution