SlowGuess/NAACL_2022_OCR huggingface dataset:
Shuo Lei†, Xuchao Zhang†, Jianfeng He†*, Fanglan Chen†, Chang-Tien Lu†
†Department of Computer Science, Virginia Tech, Falls Church, VA, USA
$^{\ddagger}$ NEC Laboratories America, Princeton, NJ, USA
{slei, jianfenghe, fanglanc, ctlu} @vt.edu, xuczhang@nec-labs.com
Large-scale multilingual pre-trained language models have achieved remarkable performance in zero-shot cross-lingual tasks. A recent study has demonstrated the effectiveness of self-learning-based approach on cross-lingual transfer, where only unlabeled data of target languages are required, without any efforts to annotate gold labels for target languages. However, it suffers from noisy training due to the incorrectly pseudo-labeled samples. In this work, we propose an uncertainty-aware Cross-Lingual Transfer framework with Pseudo-Partial-Label (CLTP)<sup>1</sup> to maximize the utilization of unlabeled data by reducing the noise introduced in the training phase. To estimate pseudo-partial-label for each unlabeled data, we propose a novel estimation method, considering both prediction confidence and the limitation to the number of similar labels. Extensive experiments are conducted on two cross-lingual tasks, including Named Entity Recognition (NER) and Natural Language Inference (NLI) across 40 languages, which shows our method can outperform the baselines on both high-resource and low-resource languages, such as 6.9 on Kazakh (kk) and 5.2 Marathi (mr) for NER.
The multilingual pre-trained language models such as mBERT (Devlin et al., 2019), XLM-R (Connieu et al., 2020) and mT5 (Xue et al., 2021) are able to support zero-shot transfer from a source language to target languages. Despite the remarkable performance on direct zero-shot cross-lingual transfer tasks, one would apply semi-supervised learning on target languages to obtain more robust and accurate predictions in a practical scenario. Recent studies (Dong and de Melo, 2019; Xu et al., 2021) validate the effectiveness of self-learning in
Figure 1: For an unlabeled sample with ambiguous predictions, the standard one-hot-labeling takes the class with the highest confidence as the pseudo-one-hot-label, introducing the noise in the training phase due to the wrong prediction. Instead of choosing one among the predictions that all have low confidence, the proposed partial-labeling method takes both ambiguous classes as candidate labels, allowing the ground-truth label to be presented in the training phase.
Confidence	Pseudo-Label	Ground-Truth
Standard	low	(0,0,1,0,0)	x
Partial-Label	high	(0,1,1,0,0)	√
cross-lingual transfer tasks, utilizing predictions of unlabeled data of target languages as silver labels. Dong and de Melo (2019) iteratively grow the training set by selecting top-k percent of unlabeled data and Xu et al. (2021) boost the performance by considering prediction confidence in the pseudo-label selection. Although their self-learning frameworks significantly improve the transferring performance, it still lags far behind supervised learning. The main reason is that the model suffers from the large number of incorrectly pseudo-labeled samples used in the training phase. Even though they adopt the selection mechanism where those easy and high-confidence predictions will be firstly added into the training set, it cannot guarantee the accurate prediction for all unlabeled data. In fact, the accuracy of the predictions drops quickly in the later iterations (see A.1), since most of the remaining unlabeled data are more difficult to be classified. To avoid the false positive pseudo labels, the most naive way is to select pseudo-labels with extremely high confidence. However, in this way, a large amount of unlabeled data will be discarded due to strictly high
<sup>1</sup>We release our code at: github.com/slei109/CLTP.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1987 - 1997
confidence bars. However, the discarded data are still valuable for learning a classifier, especially in the zero-shot task. So, how to maximize the utilization of the unlabeled data while minimizing the ratio of noisy pseudo labels in the training set?
Intuitively, if we are confident that the unlabeled data belong to a candidate class set but unable to assign one-hot pseudo labels, it is more efficient to present all potential labels comparing to discarding them directly. Thus, we propose to present pseudo-partial-labels for those data to the model. As illustrated in Figure 1, for an ambiguous sample, the model believes that it belongs to one of two categories with high confidence but has difficulty determining which category it belongs to. In this case, the standard one-hot-labeling takes the class with the highest confidence as the pseudo label, increasing the ratio of noisy pseudo labels in the training phase due to the wrong prediction. By contrast, the proposed partial-labeling method takes both ambiguous classes as candidate labels, allowing the ground-truth label presented in the training phase. In this way, the model can continue to learn on the pseudo-partial-labeled data by disambiguating the candidate labels and finding the latent ground-truth.
In this work, we propose an uncertainty-aware Cross-Lingual Transfer framework with PseudoPartial-Label (CLTP) that employs partial label learning to boost cross-lingual zero-shot transfer. Specifically, our framework utilizes any multilingual pre-trained models as the backbone, and iteratively grows the training set by adding predictions of target language data as silver labels. For those difficult data samples with low prediction confidence, different from discarding them directly or introducing a single-hypothetical but incorrect pseudo-label, we associate them with pseudo-partial-labels to better maximize the data utilization. To estimate the pseudo-partial-label, we propose a novel uncertainty-aware estimation method that considers both prediction confidence and the limitation to the number of candidate labels. The model continues to learn on the pseudo-partial-labeled data by disambiguating the candidate labels and finding the latent ground-truth.
Our key contributions can be summarized as follows. 1) We design an uncertainty-aware crosslingual transfer framework with pseudo-partial-labels. 2) We propose a novel pseudo-partial-label estimation method that considers prediction confi
dences and the limitation to the number of candidate classes. 3) We evaluate the proposed framework on both NER and NLI tasks across 40 languages in total. Comprehensive experiments show that our framework achieves a strong performance of both high-resource and low-resource languages on both tasks by a sizable margin, such as 6.9 on Kazakh (kk), 5.2 Marathi (mr) for NER and  $1\%$  on Arabic (ar),  $0.8\%$  on Bulgarian (bg) for NLI.
Cross-Lingual Representation Learning. Pre-trained transformer-based models have proven effective in learning cross-lingual information. mBERT (Devlin et al., 2019) is pre-trained on raw Wikipedia texts in languages using masked language modeling and next sentence prediction tasks with no explicit cross-lingual objective. XLM-R (Conneau et al., 2020) improves over mBERT by training longer with more data from CommonCrawl, and without the NSP objective. Recently, two self-learning based methods were proposed for cross-lingual transfer. Dong and de Melo (2019) proposed a self-learning framework to incorporate the predictions of mBERT for the cross-lingual text classification task. Xu et al. (2021) improved over the XLM-R by jointly training multiple languages together and considering prediction confidence in the silver labels selection process. However, these two methods still suffer from noisy training because of the incorrect pseudo-labels.
Pseudo-Labeling. Pseudo-labeling (Lee et al., 2013; Shi et al., 2018; Iscen et al., 2019) belongs to the self-learning scenario, and it is often used in semi-supervised learning to generate pseudo-labels for unlabeled samples with a model trained on labeled data. Inspired by noise correction work (Yi and Wu, 2019), Wang and Wu (2020) attempted to update the pseudo-labels through an optimization framework. Recently, Rizve et al. (2021) selected pseudo-labels with both prediction uncertainty and calibration, allowing for negative pseudo-labels generations. However, most existing methods involve learning from noisy data and cannot generalize to partial label learning.
Partial Label Learning. Partial label learning (Cour et al., 2011), also called ambiguously label learning (Chen et al., 2017) and superset label problem (Gong et al., 2017), has subsequently attracted a lot of attention (Feng et al., 2020; Wang and Zhang, 2020; Yao et al., 2020; Yan and Guo, 2020;
Wang et al., 2021). It refers to the task where each training sample is associated with a set of candidate labels, while only one of them is assumed to be true. Existing studies on the partial label learning can be divided into two groups: average-based methods and identification-based methods. The average-based methods (Cour et al., 2011; Zhang and Yu, 2015; Zhang et al., 2016) consider each candidate label as equally important during model training, and average the outputs of all candidate labels for predictions. The identification-based methods aim at directly maximizing the output of exactly one candidate label, chosen as the truth label. Yan and Guo (2020) studied the utilization of batch label correction; Yao et al. (2020) managed to improve the performance by combining different networks. Wen et al. (2021) proposed the Leveraged Weighted (LW) loss, considering the trade-off between losses on partial labels and non-partial ones. In this work, we adopt the idea of LW loss function for partial label learning due to its effectiveness and generalization.
Uncertainty Estimation. Recently, estimating the uncertainty of deep learning models has attracted increasing attention and have been validated the effectiveness in NLP tasks (Zhang et al., 2019; He et al., 2020). There are two main uncertainty types in Bayesian modeling (Kendall and Gal, 2017; Depeweg et al., 2018): epistemic uncertainty (EU) that captures the model uncertainty itself, which can be explained with more data; aleatoric uncertainty (AU) that captures the intrinsic data uncertainty regardless of models. Another group of uncertainty estimation methods are based on belief/evidence theory through Fuzzy Logic (De Silva, 2018), Dempster-Shafer Theory (Sentz et al., 2002), and Subjective Logic (Sensoy et al., 2018). Belief theorists focus on the reasoning of the inherent uncertainty in information resulting from unreliable, incomplete, deceptive, and/or conflicting evidences. Subjective Logic considers uncertainty in subjective opinions in terms of vacuity (i.e., lack of evidence) (Sensoy et al., 2018), dissonance (i.e., conflicting evidence), and consonance (i.e., composite subsets of state values) (Shi et al., 2020).
Summary Current works on self-learning based cross-lingual transfer methods suffer from noisy training and poor generalization due to the incorrectly pseudo-labeled samples. In this work, we adopt the idea of partial label learning to maximize unlabeled data utilization while reducing the ef
fect of ambiguously pseudo-label estimations in the self-learning framework. To the best of our knowledge, it is the first time pseudo-partial-label employed in the self-learning framework for the cross-lingual transfer and estimating the pseudopartial-label with the prediction uncertainty.
In this section, we propose a partial-label based self-learning framework to boost cross-lingual transfer performance. The overview of the proposed framework is presented in Section 3.2. The technical details for the uncertainty-aware pseudopartial-label estimation and partial label learning are described in Sections 3.3 and 3.4, respectively.
Partial label learning refers to the task where each training sample is associated with a set of candidate labels, while only one of them is assumed to be true. The goal is to find the latent ground-truth for the input through observing the partial label set. Formally, given a non-empty feature space (input space)  $\mathcal{X} \subset \mathbb{R}^d$  and a supervised label space  $\mathcal{Y}^* = [C] := \{1, \dots, C\}$ , where  $C$  is the number of classes and the partial label space is denoted as  $\mathcal{Y} := \{y | y \subset \mathcal{Y}^*\}$ . For the rest of this paper, let  $y^{(i)} = [y_1^{(i)}, \dots, y_C^{(i)}] \subseteq \{0, 1\}^C$  be the binary vector representing the partial-labels of the instance  $i$ , where  $y_c^{(i)} = 1$  if class  $c$  is selected as the candidate class and  $y_c^{(i)} = 0$  if  $c$  is not selected. For convenience, we use  $k$ -hot partial labels to represent the number of candidate classes in the partial label. For example, the partial label in Figure 1 is a two-hot partial label and has two candidate classes.
3.2 Partial-Label based Self-Learning Framework
Our approach aims to improve the overall performance by maximizing the utilization of unlabeled data while reducing the noise introduced in the training phase. This can be accomplished by applying partial label learning on those highly uncertain predictions. The intuition is that if the data appears ambiguous to be classified, it will be more effective to present potential labels instead of discarding them directly or introducing a single-hypothetical but incorrect pseudo-label in the training phase.
The complete training procedure of our proposed task-agnostic framework for cross-lingual transfer is shown in Figure 2. In our proposed CLTP frame
Figure 2: Illustration of the uncertainty-aware cross-lingual transfer framework with pseudo-partial-labels.
work, we first train a pre-trained multilingual model on the gold labels of the source language. Then the model makes predictions on the unlabeled dataset of the target languages. The proposed uncertainty-aware estimation component generates the pseudopartial-labels based on the model predictions and their corresponding uncertainty estimations. After that, we adopt a selection mechanism to incorporate the unlabeled data with high confidence scores into the training phase.
The whole training process for our method is described in Algorithm 1, where  $\mathcal{D}_u$  denotes the set of tuples combining unlabeled data with corresponding pseudo-partial-labels and prediction uncertainty  $\gamma_{diss}$ . First, a model  $f(\cdot)$  is trained on the gold labels of the source language in the first iteration. Once trained, the model can make predictions and estimate the pseudo-partial-labels for all unlabeled data of target languages in  $D_u$  based on the method introduced in Section 3.3. Note that the inputs of different languages are mixed together. Next, a subset of the pseudo-partial-labels  $S_u$  is selected with the uncertainty estimation. After selection, the model goes back to the training phase using the selected pseudo-partial-labels as well as the gold labels. We repeat the process iteratively until max iteration is reached. The early stop criteria are implemented on the dev set of the source language only since the gold labels are not available for the other languages. Note that the model is trained only on the one-hot pseudo-label in the first three iterations to accelerate the convergence.
3.3 Pseudo-Partial-Label Estimation
The key point of pseudo-partial-label estimation is to guarantee that the ground-truth class of an instance resides in the candidate label set, which is the basic definition for partially supervised learning. Intuitively, if the model classifies an instance with lower confidence, the instance may be hard to dis
Algorithm 1 Self-Learning on Cross-Linguual Tasks
Input: A gold label dataset  $\mathcal{D}_L$  of source language, an unlabeled dataset of target languages  $\mathcal{U}$ .
1: repeat
2: Training  $f(\cdot)$  on  $\mathcal{D}_L$  with  $\mathcal{L}_{EVI}$
3: for each target language do
4:  $\mathcal{D}_u\gets \emptyset$
5: for  $x_{u}$  in  $\mathcal{U}$  do
6:  $(\tilde{y},\gamma_{diss})\gets f_{\theta}(x_u)$
7:  $\mathcal{D}_u\gets \mathcal{D}_u\cup \{(x_u,\tilde{y},\gamma_{diss})\}$
8: end for
9:  $\mathcal{S}_u\gets \underset {S\subset \mathcal{D}_u,|S|\leq N}{\arg \min}\sum_{(x_u,\gamma_{diss})\in S}\gamma_{diss}$
10:  $\mathcal{U}\gets \mathcal{U}\backslash \mathcal{S}_u$
11:  $\mathcal{U}_s\gets \mathcal{U}_s\cup \mathcal{S}_u$
12:  $\mathcal{D}_L\gets \mathcal{D}_L\cup \mathcal{U}_s$
13: end for
14: until max iteration arrives.
15: Training  $f(\cdot)$  on  $\mathcal{D}_L$  with  $\mathcal{L}_{\varphi}$  until converge
tinguish from several classes or cannot be identified due to the lack of knowledge. Hence, we consider prediction confidence in the pseudo-partial-label estimation to better determine the most uncertain classes for each ambiguous instance.
We define the prediction uncertainty of the instance  $x$  belonging to the partial-label  $y$  as the partial-label uncertainty, which is denoted as  $\gamma_{diss}(x,y)$ . The decomposed entropy dissonance proposed by Shi et al. (2020) is adapted to calculate the partial-label uncertainty, as it can indicate the contradiction among certain classes. Specifically, dissonance is an evidence-based uncertainty (Sensoy et al., 2018) where the softmax probability is replaced by Dirichlet distribution, and each predicted logit for class  $c$  is regarded as the evidence  $e_c$ . The expected probability  $p_c$  for class  $c$  under Dirichlet distribution is defined as follows.
$$
p _ {c} = \frac {e _ {c} + 1}{S}, \text {w i t h} S = \sum_ {c} e _ {c} + C \tag {1}
$$
where  $S$  is referred to as the Dirichlet strength. We adopt the cross-entropy loss as the training loss  $\mathcal{L}_{\mathrm{EVI}}$ , and its Bayes risk under the Dirichlet distribution can be defined as
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {E V I}} = \int \left[ \sum_ {c} - y _ {c} \log \left(p _ {c}\right) \right] \frac {1}{B e t a (\alpha)} \prod_ {c} p _ {c} ^ {\alpha_ {c} - 1} d \mathbf {p} _ {c} \\ = \sum_ {c} - y _ {c} \int \log (p _ {c}) \frac {1}{B e t a (\boldsymbol {\alpha})} \prod_ {c} p _ {c} ^ {\alpha_ {c} - 1} d \mathbf {p} _ {c} \\ = \sum_ {c} y _ {c} (\psi (S) - \psi \left(e _ {c} + 1\right)) \tag {2} \\ \end{array}
$$
where  $Beta(\alpha)$  is the multinomial beta function (Kotz et al., 2004) and  $\psi(\cdot)$  is the digamma function.  $\alpha$  is the parameters of the Dirichlet density on the predictors. As shown in Equation (2), training the model with  $\mathcal{L}_{\mathrm{EVI}}$  is to make the positive evidence close to the total evidence when the ground-truth is positive. If there are conflicts of strong evidence among certain classes, dissonance will become high to indicate the contradiction. The following describes the dissonance for each instance:
$$
\operatorname {B a l} \left(b _ {j}, b _ {k}\right) = \left\{ \begin{array}{l l} 1 - \frac {\left| b _ {j} - b _ {k} \right|}{b _ {j} + b _ {k}}, & \text {i f} b _ {j} b _ {k} \neq 0 \\ 0, & \text {e l s e w i s e} \end{array} \right. \tag {3}
$$
$$
d i s s = \sum_ {c} \frac {b _ {c} \sum_ {c ^ {\prime} \neq c} b _ {c ^ {\prime}} \operatorname {B a l} \left(b _ {c} , b _ {c ^ {\prime}}\right)}{\sum_ {c ^ {\prime} \neq c} b _ {c ^ {\prime}}} \tag {4}
$$
where  $b_{c} = e_{c} / S$  represents the belief mass for class  $c$ . Recall that each predicted logit for class  $c$  is regarded as the evidence  $e_{c}$ .
In the pseudo-partial-label estimation, the belief mass for the instance belongs to a candidate class set can be calculated via Binomial Comultiplication operator in subjective logic, which is denoted as 'V'. Let  $b_{c}$  and  $b_{h}$  be the belief mass for class  $c$  and class  $h$ , respectively. The belief mass for the instance belongs to class  $c$  or class  $h$  is defined as:
$$
b _ {c \vee h} = b _ {c} + b _ {h} - b _ {c} b _ {h} \tag {5}
$$
Thus, we can calculate the partial-label uncertainty  $\gamma_{diss}(x,y)$  via Equation (4) and (5).
However, if we simply estimate the pseudopartial-label only based on the lowest partial-label uncertainty, it will lead to an invalid partial label like  $(1,1,1,1)$ , as containing all classes in the candidate set must have the highest confidence. Furthermore, in the partial label learning, the accuracy of the model decreases with the increased number of similar labels to the true label (Lv et al., 2020; Wen et al., 2021) because it increases the learning difficulty. To remedy this problem, we leverage a penalty ratio to balance the prediction uncertainty and the number of candidate classes. Specifically, from Figure.3, we observe that as the number of candidate classes increases, the improvement in prediction recall tends to decrease. That means containing too many candidate classes in the pseudo-partial-label has limited improvement to the model. Only the most confusing candidate
Figure 3: Recall of k-hot pseudo-partial-labels in the last iteration with various uncertainty methods.
classes are important for those ambiguous samples. Motivated by this, we employ a penalty ratio to punish a larger number of candidate classes. Thus, a pseudo-partial-label  $\tilde{y}$  is obtained as follows:
$$
\tilde {y} = \underset {y \subset \mathcal {Y}} {\arg \min } \left(\left(\lambda^ {\| y \| _ {1} - 1} + \tau\right) \tau^ {\| y \| _ {1} - 2} \gamma_ {d i s s} (x, y)\right) \tag {6}
$$
where  $\mathcal{V}$  is the collection of all subsets in the partial label space and  $||y||_1$  calculates the number of candidate classes in the partial label  $y$ .  $\lambda$  and  $\tau$  are penalty ratios to punish a larger number of candidate classes, which determine the penalty strength and the preference for candidate class number, respectively.
3.4 Learning with Pseudo-Partial-Labels
The target of partial-label learning is to learn a classifier with access to the candidate label set (partial label set) by disambiguating the candidate labels and finding the latent ground-truth for the input during the training phase. We adopt Leveraged Weighted (LW) loss function (Wen et al., 2021) for partial label learning due to its effectiveness and generalization. The LW loss is a multiclass loss and searches for the latent ground-truth by assigning more weights to the loss of classes more likely to be the true and lessening weights to the confusing ones. To be specific, the LW loss function is defined as:
$$
L _ {\varphi} (\tilde {y}, f (x)) = \sum_ {\tilde {y} _ {c} = 1} \omega_ {c} \varphi \left(f _ {c} (x)\right) + \beta \sum_ {\tilde {y} _ {c} = 0} \omega_ {c} \varphi (- f _ {c} (x)), \tag {7}
$$
where  $\varphi (\cdot):\mathbb{R}\to \mathbb{R}^{+}$  denotes a binary loss and we adopt the Sigmoid loss function. We use parameter  $\beta$  to distinguish the effects between candidate classes and non-candidate ones.  $f_{c}(x)$  represents the predicted logit of instance  $x$  for class  $c$ , while  $w_{c}$  is the weighting parameters to assign weights
to the loss of classes. Since the key point is to disambiguate the candidate classes, the model is supposed to assign more weights to the loss of classes that are more likely to be the ground-truth. Thus, instead of assigning fixed values, the weighting parameters are updated by normalizing the prediction score through an iterative learning process. Specifically, at the  $t$ -th learning step,  $w_{c}^{(t)}$  is calculated as follows:
$$
\omega_ {c} ^ {(t)} = \left\{ \begin{array}{l l} \frac {\exp (f _ {c} ^ {(t)} (x))}{\sum_ {\tilde {y} _ {c} = 1} \exp (f _ {c} ^ {(t)} (x))}, & \mathrm {i f} \tilde {y} _ {c} = 1 \\ \frac {\exp (f _ {c} ^ {(t)} (x))}{\sum_ {\tilde {y} _ {c} = 1} \exp (f _ {c} ^ {(t)} (x))}, & \mathrm {i f} \tilde {y} _ {c} = 0 \end{array} \right. \tag {8}
$$
Note that  $w_{c}^{(t)}$  varies with sample instances. In this way, as the training epochs grow, the model focuses on the true class and rules out the untrue classes by penalizing large value of  $\varphi (-f_c(x))$ .
4.1 Evaluation Tasks & Datasets
NLI XNLI (Conneau et al., 2018) is an evaluation benchmark for the cross-lingual NLI task across 15 languages. Given a sentence pair of premise and hypothesis, the task is to classify their relationship as "neutral", "entailment", or "contradiction".