SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jun-Yi Hang, Min-Ling Zhang*
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China
Key Laboratory of Computer Network and Information Integration (Southeast University),
Ministry of Education, China
{hangjy, zhangml}@seu.edu.cn
In partial multi-label learning (PML), each training example is associated with a set of candidate labels, among which only some labels are valid. As a common strategy to tackle PML problem, disambiguation aims to recover the ground-truth labeling information from such inaccurate annotations. However, existing approaches mainly rely on heuristics or ad-hoc rules to disambiguate candidate labels, which may not be universal enough in complicated real-world scenarios. To provide a principled way for disambiguation, we make a first attempt to explore the probabilistic graphical model for PML problem, where a directed graph is tailored to infer latent ground-truth labeling information from the generative process of partial multi-label data. Under the framework of stochastic gradient variational Bayes, a unified variational lower bound is derived for this graphical model, which is further relaxed probabilistically so that the desired prediction model can be induced with simultaneously identified ground-truth labeling information. Comprehensive experiments on multiple synthetic and real-world data sets show that our approach outperforms the state-of-the-art counterparts.
Partial multi-label learning aims to induce a multi-label predictor from inaccurately annotated examples, where a set of candidate labels is assigned to each training example but only some of these candidate ones are valid. The problem of PML naturally arises in many real-world applications [39, 51, 52, 20]. For instance, as shown in Figure 1, the data collected from the crowdsourcing platform is inevitable to contain inaccurate supervision information due to potential unreliable annotators, where invalid labels exist in the set of candidate labels.
Formally, let  $\mathcal{X} = \mathbb{R}^d$  denote the input space and  $\mathcal{Y} = \{l_1, l_2, \dots, l_t\}$  denote the label space with  $t$  class labels. A partial multi-label example is denoted as  $(\mathbf{x}, S)$ , where  $\mathbf{x} \in \mathcal{X}$  is its feature vector and  $S \subseteq \mathcal{Y}$  is its set of candidate labels. As a basic assumption of PML, the ground-truth labels reside in the candidate label set, i.e.  $Y \subseteq S$ , and are concealed from the learning algorithm. Given a partial multi-label data set  $\mathcal{D} = \{(\mathbf{x}_i, S_i) | 1 \leq i \leq m\}$ , the goal of PML is to derive a multi-label predictor  $h: \mathcal{X} \to 2^{\mathcal{Y}}$  which can accurately predict all the ground-truth labels for an unseen instance.
A straightforward strategy for tackling the problem of PML is to simply treat all the candidate labels as valid ones and then apply off-the-shelf multi-label learning approaches [50, 23] to induce the predictor. This naive strategy is obviously suboptimal as the learning process will be significantly misled by the noisy labels in the candidate label set.
Candidate labels (valid ones in red)
Figure 1: An exemplary partial multi-label learning scenario. Among the set of 7 candidate labels given by crowdsourcing annotators, only 4 labels are valid including tree, light, mountain, and road.
Considering that the ground-truth labels are concealed in the candidate label set, it is a common strategy of existing approaches to learn from partial multi-label data by disambiguation, i.e. recovering the ground-truth labeling information from candidate labels. The basic idea of disambiguation is to estimate the labeling confidence for each candidate label from observed inaccurately annotated examples, which reflects how likely the candidate label can be a ground-truth one. For example, some approaches introduce the smoothness assumption to directly elicit the labeling confidence via label propagation [48, 26], label enhancement [31, 41], or cluster assignment [39, 3], etc. While some other works heuristically believe that the noisy labels in the candidate label set are generally sparse, so that the noisy label matrix [30, 29] or the noisy label identifier [35] can be learned with sparsity regularization. Although feasible results have been achieved in these approaches, a potential limitation is that they rely on heuristics or ad-hoc rules for disambiguation, which may not be widely applicable in some challenging scenarios. For example, sparsity on underlying noisy labels can hardly be satisfied in some extremely low-quality annotation scenarios.
To improve this, we explore a principled manner for disambiguation, which is regarded as a task of latent variable inference in this paper. Accordingly, a novel partial multi-label learning approach named PARD, i.e. Partial multi-label learning with probAbilistic-gRaphical Disambiguation, is presented. By regarding the concealed ground-truth labels as latent variable, a directed graphical model is tailored to describe the generative process of partial multi-label data. With the directed graphical model, a unified variational lower bound of the data log-likelihood is derived via variational inference, which allows to disambiguate the candidate labels and induce the prediction model simultaneously. Specifically, the ground-truth labeling information is identified by fitting the generative model of PML data, where the intrinsic structure information of data can be captured for principled disambiguation. While the desired prediction model is optimized with a confidence-smoothed cross entropy loss. Comprehensive experimental studies show that our approach performs better than well-established PML algorithms.
The rest of this paper is organized as follows. Section 2 briefly reviews related works. Section 3 presents details of the proposed PARD approach. Section 4 reports experimental results over a wide range of synthetic and real-world data sets. Section 5 concludes this paper.
Partial multi-label learning is an emerging weakly-supervised learning problem, which has received wide attention in recent years [39, 40, 36]. Here, we would like to first make a brief review on two closely related learning problems, i.e. multi-label learning [50, 23] and partial label learning [6, 24].
Multi-label learning (MLL) deals with the problem where an example can be associated with multiple labels simultaneously. According to the order of label correlations considered, existing MLL approaches can be grouped into three categories, including first-order approaches which treat each class label independently [2, 14], second-order approaches which explore label correlations between pairwise class labels [8, 53], and high-order approaches which consider label correlations among a subset or the whole set of class labels [32, 44]. With the same goal, both MLL and PML aim to learn a multi-label predictor from multi-semantic objects. Nevertheless, the key difference between these
two learning problems lies in whether the ground-truth labels are accessible or not during the learning process, which also makes PML more challenging than MLL since the supervision information is inaccurate.
Partial label learning (PLL) deals with the problem where each example is associated with multiple candidate labels, among which only one is valid. The problem of PLL can be tackled in several different strategies, such as transforming into well-studied supervised learning problems [49, 34], disambiguating the candidate labels [9, 45, 43], or learning with theoretically consistent algorithms [24, 10], etc. Conceptually speaking, PLL and PML possess similar form of supervision, where false positive labels reside in the candidate label set. However, the task of PML is more complicated than PLL since the desired predictor is a multi-label one in PML instead of a single-label one in PLL and the number of ground-truth labels is further unknown.
To solve the PML problem, the most commonly employed strategy is disambiguation, i.e. recovering the ground-truth labeling information from candidate labels. Existing approaches mainly rely on some heuristics or ad-hoc rules to disambiguate candidate labels, which can be roughly grouped into three categories in terms of the heuristics or rules exploited, including smoothness assumption-based approaches, low-rank constraint-based approaches, and sparsity regularization-based approaches. The first category of approaches relies on smoothness assumption that close instances may share similar labels for disambiguation. As a seminal work, an iterative procedure is proposed in [39] to optimize the labeling confidence of each candidate label and the prediction model alternatively, where the labeling confidence scores are obtained by assuming cluster structures exist in the feature space. Follow-up works employ smoothness assumption to design different disambiguation mechanisms, such as propagating labels in a graph based on instance similarity [48, 26], recovering numerical labels with label enhancement [31, 41, 42], or performing implicit disambiguation by jointly embedding [12]. Low-rank constraint-based approaches turn to the low-rank property for disambiguation. For example, low-rank matrix decomposition is exploited to recover the ground-truth labeling information in [47] and a low-rank subspace is learned in [25] to elicit credible labels from improved subspace representations. While sparsity regularization-based approaches [30, 29, 35] impose sparsity on noises hiding in candidate label set, which facilitate the disambiguation in a roundabout manner. A potential limitation of these approaches is that the above heuristics or ad-hoc rules can be hardly held in some challenging scenarios, which may lead to suboptimal performance of induced prediction model.
Therefore, it is quite natural to consider whether it is possible to perform disambiguation in a principled manner, which is still an underexplored direction with only a few attempts. PML-GAN [46] recovers ground-truth labeling information with an instance reconstruction process, corresponding to a minimax adversarial game to implicitly model instance distribution. While MILI-PML [11] disambiguates candidate labels by maximizing the mutual information between features and identified ground-truth labels. With a totally different methodology, our approach regards disambiguation as a task of latent variable inference, which is formalized in a maximum likelihood framework with a concise variational lower bound of the log-likelihood on observed PML data. As another pioneer work towards this direction, PML-MD [37] devises a meta-learning framework, where disambiguation is performed with the guidance from an accurately annotated validation set. Moving one step forward, we make a first attempt to tackle the problem via modelling the generative process of partial multi-label data with a tailored directed graphical model. Compared with PML-MD, our approach is more flexible since there is no need to include a clean validation set for disambiguation, which may not be easily accessible in many real-world scenarios. We will detail our approach in the next section.
3 The PARD Approach
To tackle the problem of partial multi-label learning, PARD disambiguates the candidate labels and induces the prediction model simultaneously by optimizing towards a unified variational lower bound of the data log-likelihood. In the following content, we will present the tailored directed graphical model which describes the generative process of partial multi-label data. Then, we derive a variational lower bound of the data log-likelihood and further explain how to relate it with disambiguation and predictor induction. Finally, essential implementation issues w.r.t. the optimization procedure are detailed.
Some necessary notations have been introduced in section 1. For notation briefness, a  $t$ -dimensional indicator vector  $\mathbf{s} \in \{0,1\}^t$  is utilized to denote the set of candidate labels  $S$ , where  $s_k = 1$  indicates  $l_k \in S$  and  $s_k = 0$  otherwise. Similarly, a  $t$ -dimensional indicator vector  $\mathbf{y} \in \{0,1\}^t$  is utilized to denote the set of ground-truth labels  $Y$ .
3.2 Probabilistic Graphical Disambiguation Framework
Considering that only the instance and its associated candidate labels can be observed in the partial multi-label training set, we tailor a directed graphical model to describe the generative process of partial multi-label data, which involves an unobserved latent variable  $\mathbf{y}$  as the ground-truth labels. The generative process consists of three steps: (1) sample an instance  $\mathbf{x}$  from the marginal distribution  $p_{\theta}(\mathbf{x})$ ; (2) sample a latent  $\mathbf{y}$  as the ground-truth labels of the instance  $\mathbf{x}$  from the ground-truth class posterior distribution  $p_{\theta}(\mathbf{y}|\mathbf{x})$ ; (3) corrupt the instance's ground-truth labels to obtain its candidate labels  $\mathbf{s}$  by  $p_{\theta}(\mathbf{s}|\mathbf{x},\mathbf{y})$ . Accordingly, the joint probability  $p_{\theta}(\mathbf{x},\mathbf{y},\mathbf{s})$  can be factorized as
$$
p _ {\theta} (\mathbf {x}, \mathbf {y}, \mathbf {s}) = p _ {\theta} (\mathbf {x}) p _ {\theta} (\mathbf {y} | \mathbf {x}) p _ {\theta} (\mathbf {s} | \mathbf {x}, \mathbf {y}). \tag {1}
$$
Given the partial multi-label training set  $\mathcal{D}$ , the above directed graphical model can be learned via maximizing the log-likelihood on the observed data. However, it is generally challenging to directly estimate the parameter of a directed graphical model by likelihood maximization due to computational intractability [17]. To make the optimization tractable, the variational lower bound of the log-likelihood $^2$  is derived as follows
$$
\log p _ {\theta} (\mathbf {s} | \mathbf {x}) \geq \mathcal {L} (\mathbf {x}, \mathbf {s}; \theta , \phi) = \mathbb {E} _ {q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s})} [ \log \frac {p _ {\theta} (\mathbf {s} | \mathbf {x} , \mathbf {y}) p _ {\theta} (\mathbf {y} | \mathbf {x})}{q _ {\phi} (\mathbf {y} | \mathbf {x} , \mathbf {s})} ], \tag {2}
$$
where  $q_{\phi}(\mathbf{y}|\mathbf{x},\mathbf{s})$  is the introduced variational posterior to approximate the true posterior  $p_{\theta}(\mathbf{y}|\mathbf{x},\mathbf{s})$ .  $\mathcal{L}(\mathbf{x},\mathbf{s};\theta ,\phi)$  is the derived variational lower bound as a surrogate loss function of the log-likelihood, which can be rewritten as
$$
\mathcal {L} (\mathbf {x}, \mathbf {s}; \theta , \phi) = \mathbb {E} _ {q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s})} [ \log p _ {\theta} (\mathbf {s} | \mathbf {x}, \mathbf {y}) ] - K L [ q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s}) | | p _ {\theta} (\mathbf {y} | \mathbf {x}) ], \tag {3}
$$
where  $KL[\cdot ||\cdot ]$  denotes the KL-divergence between two distributions. Here, the meaning of the ground-truth class posterior distribution  $p_{\theta}(\mathbf{y}|\mathbf{x})$  is comprehensible, which is actually the desired prediction model<sup>3</sup>. While the variational posterior  $q_{\phi}(\mathbf{y}|\mathbf{x},\mathbf{s})$  (a.k.a. inference model) attempts to disambiguate the candidate label set by inferring the most probable ground-truth labels from which the candidate labels could have been corrupted given the instance  $\mathbf{x}$ . Correspondingly,  $p_{\theta}(\mathbf{s}|\mathbf{x},\mathbf{y})$  (a.k.a. generative model) corrupts the identified ground-truth labels to recover the observed candidate labels.
To move one step further for comprehending the learning behavior of the above three models, we reformulate the variational lower bound via unfolding the KL-divergence term
$$
\begin{array}{l} \mathcal {L} (\mathbf {x}, \mathbf {s}; \theta , \phi) = \mathbb {E} _ {q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s})} \left[ \log p _ {\theta} (\mathbf {s} | \mathbf {x}, \mathbf {y}) \right] + H \left[ q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s}) \right] \\ + \mathbb {E} _ {q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s})} [ \log p _ {\theta} (\mathbf {y} | \mathbf {x}) ], \tag {4} \\ \end{array}
$$
where  $H[\cdot ]$  denotes the entropy of a distribution. In the above objective, the first two terms present an entropy-regularized autoencode process (w.r.t. s), with which the intrinsic structure information of data can be captured and utilized for disambiguation. While the last term is actually a cross entropy loss for inducing the prediction model with identified ground-truth labeling information  $q_{\phi}(\mathbf{y}|\mathbf{x},\mathbf{s})$ . By optimizing all these terms in a unified variational lower bound, the models can gradually capture the underlying generative process of partial multi-label data, so that the candidate label set is disambiguated and the desired prediction model is induced simultaneously.
3.3 Additional Training Techniques
In this section, we present additional techniques that allow the derived variational lower bound to be optimized efficiently.
2We omit the optimization for the marginal distribution on  $\mathbf{x}$ , as it is not the focus of our work. Detailed derivation process can be found in Appendix B.
<sup>3</sup>With a little abuse of notations, a distribution, e.g.  $p_{\theta}(\mathbf{y}|\mathbf{x})$ , is also called a model which actually denotes the model to parameterize the distribution, when the context is clear. All the distributions involved in Eq. (3) are instantiated as multivariate Bernoulli distributions and are parameterized by neural networks.
Algorithm 1 Pseudocode of the Optimization Procedure for PARD
Inputs: Training set  $\mathcal{D} = \{(\mathbf{x}_i,S_i)|1\leq i\leq m\}$ , batch size  $b$  and maximal iteration  $T$ .
1: Initialize model parameters  $\theta, \phi$ ;
2: for  $t = 1 : T$  do
3: Sample a batch of training examples  $\mathcal{B} = \{(\mathbf{x}_{i_k},S_{i_k})|1\leq k\leq b\}$  from training set  $\mathcal{D}$
4: Compute unbiased estimator of the variational lower bound on  $\mathcal{B}$  by Eq. (6) and Eq. (7);
5: Update model parameters  $\theta, \phi$  via gradient ascent.
Outputs: Model parameters  $\theta, \phi$ .
Continuous Relaxing with Gumbel-Softmax Trick. It is quite challenging to compute the first expectation term in Eq. (3). Actually, it has a high complexity of  $\mathcal{O}(2^t)$  to analytically compute the first expectation term in Eq. (3), since the ground-truth labels  $\mathbf{y}$  can take  $2^{t}$  different values. In addition, estimating it by Monte Carlo sampling is non-differentiable, since sampling from discrete variational posterior  $q_{\phi}(\mathbf{y}|\mathbf{x},\mathbf{s})$  prevents gradients from flowing through the inference model.
To circumvent this problem, we exploit Gumbel-Softmax trick [13, 27] to smooth the sampling process so that the expectation term can be efficiently estimated by Monte Carlo sampling in a differentiable manner. Specifically, the Bernoulli variable  $\mathbf{y}$  is relaxed by the binary Concrete variable  $\mathbf{c} \sim \text{BinConcrete}(\mathbf{p}, \tau)$ , which is a continuous alternative with the reparameterization form as
$$
\mathbf {c} = \frac {1}{1 + \exp [ - (\log \alpha + 1) / \tau ]}, \tag {5}
$$
where  $\alpha = \frac{\mathbf{p}}{1 - \mathbf{p}}$  and  $\mathbf{p}$  is the parameter of the multivariate Bernoulli distribution  $q_{\phi}(\mathbf{y}|\mathbf{x},\mathbf{s})$ . 1 is a sampling from Logistic distribution and  $\tau >0$  is a temperature parameter. In the limit  $\tau \rightarrow 0$ , a binary Concrete variable smoothly converges to its Bernoulli counterpart. With the above sampling trick, an unbiased estimation of the expectation term can be obtained and gradients w.r.t. the distribution parameter  $\mathbf{p}$  (or w.r.t. the parameter of the inference model equivalently) are well-defined by the chain rule
$$
\mathbb {E} _ {q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s})} [ \log p _ {\theta} (\mathbf {s} | \mathbf {x}, \mathbf {y}) ] \approx \frac {1}{L} \sum_ {i = 1} ^ {L} \log p _ {\theta} (\mathbf {s} | \mathbf {x}, \mathbf {c} ^ {(i)}) \tag {6}
$$
$$
w h e r e \quad \mathbf {c} ^ {(i)} \sim B i n C o n c r e t e (\mathbf {p}, \tau).
$$
Closed-Form Solution of the KL-Divergence Term. The KL-divergence term in Eq. (3) can not be integrated analytically in general cases. To make it tractable, we exploit mean-field approximation technique and derive a closed-form solution of the KL-divergence term as follows
$$
K L \left[ q _ {\phi} (\mathbf {y} | \mathbf {x}, \mathbf {s}) \right\rvert   \left| p _ {\theta} (\mathbf {y} | \mathbf {x}) \right] = \sum_ {k = 1} ^ {t} p _ {\phi} ^ {y _ {k}} \log \frac {p _ {\phi} ^ {y _ {k}}}{p _ {\theta} ^ {y _ {k}}} + \left(1 - p _ {\phi} ^ {y _ {k}}\right) \log \frac {1 - p _ {\phi} ^ {y _ {k}}}{1 - p _ {\theta} ^ {y _ {k}}}, \tag {7}
$$
where  $p_{\phi}^{y_k} = q_{\phi}(y_k = 1|\mathbf{x},\mathbf{s})$  and  $p_{\theta}^{y_k} = p_{\theta}(y_k = 1|\mathbf{x})$ . Detailed derivation process can be found in Appendix C.
With these training techniques, the derived variational lower bound can be straightforwardly optimized with standard stochastic gradient methods. Algorithm 1 summarizes the pseudocode of the optimization procedure, where the candidate labels and the prediction model are updated simultaneously.
4.1.1 Data Sets
For comprehensive performance evaluation, five real-world and a number of synthetic PML data sets are employed in this paper. Table 1 summarizes detailed characteristics of each data set. Specifically,
Table 1: Characteristics of the experimental data sets. The first five ones are real-world PML data sets and the last five ones are multi-label data sets employed to generate synthetic PML data sets.
Dataset	#Examples	#Features	#Class Labels	Cardinality	Domain
YeastBP	6139	6139	217	5.54	\( Biology^1 \)
YeastCC	6139	6139	50	1.35	\( Biology^1 \)
YeastMF	6139	6139	39	1.01	\( Biology^1 \)
Musicemotion	6833	98	11	2.42	\( Music^1 \)
Music_style	6839	98	10	1.44	\( Music^1 \)
corel5k	5000	499	374	3.52	\( Images^2 \)
rcv1-s1	6000	944	101	2.88	\( Text^2 \)
Corel16k-s1	13766	500	153	2.86	\( Images^2 \)
iaprtc12	19627	1000	291	5.72	\( Images^3 \)
espgame	20770	1000	268	4.69	\( Images^3 \)
1 http://palm.seu.edu.cn/zhangml/
2 http://mulan.sourceforge.net/datasets.html
3 http://lear.inrialpes.fr/people/guillaumin/data.php
the first five data sets are real-world PML data sets, where candidate labels are collected from web users and manually examined to specify the ground-truth labels. While the last five data sets, including corel5k, rcv1-s1, Corel16k-s1, iaprtc12 and espgame, are multi-label data sets.
Following [35], a synthetic PML data set is generated from one multi-label data set by adding random labeling noise. Accordingly, for data sets with more than 100 class labels, we filter out their rare labels to keep the 15 most frequent labels and remove instances without relevant labels. Then, we randomly flip the irrelevant labels of an instance until the number of irrelevant labels in the set of candidate labels equals to  $\gamma \%$  number of ground-truth labels. We vary the value of  $\gamma$  in the range of  $\{100, 150, 200, 250\}$ , so that a total of  $5 \times 4 = 20$  synthetic PML data sets are generated for experimental studies.
Five widely-used evaluation metrics for multi-label learning are employed to evaluate the performance of each approach, including Average precision, Hamming loss, One-error, Coverage and Ranking loss. Detailed definitions on these metrics can be found in [50].
4.1.3 Implementation Details