SlowGuess/NAACL_2022_OCR huggingface dataset:
Nuo Chen $^{1*}$ , Linjun Shou $^{2}$ , Ming Gong $^{2}$ , Jian Pei $^{3}$ , Daxin Jiang $^{2\dagger}$
<sup>1</sup> ADSPLAB, School of ECE, Peking University, China
$^{2}$ STCA, Microsoft, Beijing
$^{3}$ School of Computing Science, Simon Fraser University
nuochen@pku.edu.cn, {lisho, migon, djiang}@microsoft.com
jpei@cs.sfu.ca
Large-scale cross-lingual pre-trained language models (xPLMs) have shown effectiveness in cross-lingual sequence labeling tasks (xSL), such as cross-lingual machine reading comprehension (xMRC) by transferring knowledge from a high-resource language to low-resource languages. Despite the great success, we draw an empirical observation that there is a training objective gap between pre-training and fine-tuning stages: e.g., mask language modeling objective requires local understanding of the masked token and the span-extraction objective requires global understanding and reasoning of the input passage/paragraph and question, leading to the discrepancy between pretraining and xMRC. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap in a self-supervised manner. Second, we present ContrActive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel sequences via unsupervised cross-lingual instance-wise training signals during pre-training. By these means, our methods not only bridge the gap between pretrain-finetune, but also enhance PLMs to better capture the alignment between different languages. Extensive experiments prove that our method achieves clearly superior results on multiple xSL benchmarks with limited pretraining data. Our methods also surpass the previous state-of-the-art methods by a large margin in few-shot data settings, where only a few hundred training examples are available.
Sequence labeling (SL) tasks are common conditions and have considerable impact in natural language processing communities, such as named entity recognition (NER) (Lample et al., 2016), and
Figure 1: Mask language modeling objective in the pretraining stage and span-extraction objective of xMRC in the fine-tuning stage.
machine reading comprehension (MRC) (Hermann et al., 2015; He et al., 2018; Asai et al., 2018; Yuan et al., 2020; You et al., 2020b; You et al.), to name a few. For instance, MRC models are required to predict whether each token in the input passage is the start or end position of the ground-truth answer span given a pre-defined question. Many promising results have been achieved using deep-learning based methods (Seo et al., 2017; Liang et al., 2021b) on popular benchmarks (Rajpurkar et al., 2016, 2018; Reddy et al., 2019; You et al., 2020a). Recently, there is a growing body of literature that recognizes the importance of cross-lingual sequence labeling tasks (xSL for short) (Huang et al., 2019; Lewis et al., 2020; Artetxe et al., 2019a), where translation systems are utilized to translate high-resource languages into low-resource languages so as to enrich the training data. However, the performance of xSL models is severely affected by translation quality (Yuan et al., 2020; Liang et al., 2021a).
Recently, thanks to multilingual pre-trained language models (xPLMs) (Huang et al., 2019; Liang et al., 2020; Conneau et al., 2019; Chi et al., 2021a) pre-trained on large multilingual corpus (billion-level), the pre-training + fine-tuning paradigm
*Work done when interned at Microsoft STCA.
Human Language Technologies, pages 1909 - 1923
English(en):
List of Ottawa buildings
Viestamese(vi):
Danh sách cóc toa nhà cua Ottawa
CLISM Input:
[CLS] [QUE] of [QUE] buildings [SEP] Danh sáché céc
toña cúa Ottawa [SEP]
Selected Spans in English: [List, Ottawa]
Aligned Spans in Viestamese: [Danh sach, Ottawa]
Figure 2: The example of our proposed CLISM. Here we present the input sequence in English and Vietnamese. Selected spans in English are from off-the-shelf NER tools like Spacy and the corresponding aligned spans in Vietnamese obtained by GIZA++ toolkit (Pei et al., 2020).
starts dominating various cross-lingual understanding tasks including entity recognition (Liang et al., 2021a,b) and question answering (Asai et al., 2018; Chen et al., 2021b). Even if in zero-shot or few-shot settings, these effective xPLMs still achieve remarkable improvements during fine-tuning via transfer learning. Despite the promising results in such settings, there is an objective gap between pre-training and fine-tuning for xSL tasks. Take the xMRC task as an example. During pre-training, PLMs are always trained with mask language modeling (MLM) (Devlin et al., 2018; Lan et al., 2020), Next Sentence Prediction (NSP) (Pires et al., 2019; Conneau et al., 2019) or translation replaced token detection (TRTD) (Chi et al., 2021c) tasks. Subsequently, these state-of-the-art models are typically trained for span extraction using distantly supervised multilingual task-related examples during fine-tuning. This is somewhat expected, since local context understanding around the mask token is mainly needed to perform pre-training tasks like MLM or TRTD, while span-extraction MRC requires the model to understand the interaction between question and global context of passage, as shown in Figure 1. Suffering from different objectives, the performance of the model inevitably deteriorates (Jiao et al., 2021; Ram et al., 2021).
Motivated by these factors, we first design a cross-lingual language-informative span masking (CLISM) task for bridging the gap between pretraining and fine-tuning in the context of xSL, which can be seen as a question answering task. A primary concern of such a self-supervised task
is how to create corresponding multilingual <question, answer> training pairs. Concretely, we group the parallel corpus into several sub-groups. Each sub-group is called a language-informative group and contains the same sentence in two different languages, a source language and a target language. Considering aligned spans in the crosslingual parallel sentences can be seen as a signal for coreference and lead to information short-cut (Jiao et al., 2021). Subsequently, we mask n-grams (e.g., named entities) spans which occur in a given source language sentence, and then we take the masked sentence and the unmasked counterpart in target language as input. Thus, we form question answering by masking all selected spans with a special token [QUE], and requiring the model to find the correct start and end positions for each of such tokens based on understanding global context of the sentence in source and target languages, as shown in Figure 2. Correspondingly, each [QUE] is regarded as a question, and its corresponding span in the target language sentence that is unmasked acts as the answer. With this strategy, the model needs to predict whether each token is the answer start or end index, which is essentially consistent with xSL tasks (Ram et al., 2021). Hence the training objectives are connected, and thus, cross-lingual PLMs can be more adequate for xSL.
Considering to better capture the alignment between the same content in different languages and avoid learning representations that are covariant with the noisy parallel input (e.g., [QUE] and [MASK]), we also design a strategy called ContrAstive-Consistency Regularization (CACR) during pre-training. Our core idea behind CACR is to push representations of the same sequence in various languages to be similar, while pushing it away from other examples via contrastive learning (You et al., 2022, 2021c). The overview of our architecture is presented in Figure 3.
The resulting language model exhibits surprising performances in multiple xSL benchmarks, such as xMRC and xNER. Of note, compared with other new state-of-the-art xPLMs such as Info-XLM and XLM-Align which utilize billion-level pretraining data, our model pre-trained with limited data e.g., 1 million achieve better performances across various datasets. As an additional benefit of our methods, our model also achieves superior performances given a few training instances. For example, ours achieve  $35.9\% / 50.8\%$  EM/F1 score
Figure 3: Model Architecture: In this example, we choose English (en) as our source language and Vietnamese (vi) as our target language, separately. In the input sequence, the original sentence in English is "DRM clause", "DRM" is the selected span. We mask "DRM" with the special token [QUE]. Then the designed CLISM is required the model to predict the start and end positions of the correct answer for [QUE] ("DRM" in target language).
with only 512 examples on MLQA dataset, outperforming all baselines by a very wide margin (vs.  $15.5\% / 26.8\%$ ).
In general, we make the following technical contributions in this paper:
- We present a new cross-lingual pre-training paradigm named CLISM to bridge the training objective gap between language models and cross-lingual sequence labeling tasks.
- We design a novel pre-training strategy termed CACR, which encourages xPLMs to better capture alignment between cross-lingual representations.
- The resulting models achieve state-of-the-art performances with limited parallel pretraining data on multiple xSL benchmarks. Moreover, we demonstrate that our method also shows significant improvements in few shot data settings during fine-tuning.
Pre-trained Language Models During the past few years, PLMs (Devlin et al., 2018; Liu et al., 2019; Lan et al., 2020; Sun et al., 2020) dominate natural language understanding and generating areas due to their promising results in various downstream tasks. Taking a step forward a more real world, there is a growing body of communities extent PLMs to multi-lingual language (xPLMs), and
some representative works have been done such as XLM-R (Conneau et al., 2019) (short for XLM-Roberta), info-XLM (Chi et al., 2021a) and XLM-Align (Chi et al., 2021b). Almost these efforts trained with token-level pre-training tasks on large multi-lingual corpus. Directly utilizing xPLMs lead high performances in the fine-tuning stage, but when transferred to downstream tasks can result the objective gap between pretrain-finetune, making xPLMs sub-optimized in downstream tasks especially for xSL (Jiao et al., 2021; Ram et al., 2021). Our proposed CLISM and CACR not only alleviate the pretrain-finetune discrepancy, but also strong the ability of xPLMs to capture alignment between cross-lingual representations in sentence-level.
Sequence Labeling Sequence Labeling tasks are of great significance in web mining, such as named entity recognition (Shen et al., 2021; Cui et al., 2021), relation identification (Akhmouch et al., 2021), event extraction (Lu et al., 2021; Lyu et al., 2021), and even machine reading comprehension (Wu et al., 2021; Chen et al., 2021a; You et al., 2021a,b; Chen et al., 2021c). However, when pushing the boundary of SL to low-resource languages, xSL can be very challenging, due to limited training data. To tackle this challenge, various efforts have been proposed (Huang et al., 2019; Liang et al., 2020; Conneau et al., 2019; Liang et al., 2021a). Cui et al. (2019) and Singh et al. (2019) utilized machine translation to get parallel data as
the data argumentation method. But the performance of these methods are restricted to the quality of translation. Hence, Yuan et al. (2020) proposed to generate low-resource language weakly labeled data from Wikipedia articles. Similarly, we also collect pre-training parallel data for CLISM from web sources. We tackle the objective gap between language models and xSL tasks, an challenge overlooked by the previous methods.
Recent Works Currently, several studies have explored to bridge the gap between pre-training and fine-tuning, including two main categories: (1) knowledge-driven, which introduces external commonsense knowledge into the pre-training for improving model performance in down-stream tasks (Varkel and Globerson, 2020; Sun et al., 2020), (2) The latter is task-oriented, which includes several well designed pre-training or fine-tuning tasks (Jiao et al., 2021; Deng et al., 2021). However, none of these works target cross-lingual areas, especially for xSL tasks. Recently, concurrent with our research, Ram et al. (2021) studied aligning the pre-training and fine-tuning stages in the context of question answering via introducing a novel pre-training task. Nevertheless, Ram et al. (2021) focused on few-shot question answering, and didn't explore the relation between different languages.
In this section, we first illustrate our two pretraining strategies: CLISM and CACR in detail. Then we introduce how to apply our methods in down-stream tasks to obtain high performances. Due to the excellent versatility of our proposed methods, which can build on top of any xPLMs. Hence, we leverage  $\mathcal{M}$  to represent a series of pre-trained language models. Our goal aims at aiding  $\mathcal{M}$  to eliminate the influence of objective gap between pretrain-finetune, and enhancing the consistency between parallel data representations.
3.1 Cross-lingual Language Informative Span Masking
Definition Heading for bridging the objective gap between pre-training and fine-tuning in xSL, we formulate a new pre-training task named Cross-lingual Language Informative Span Masking (CLISM) from the unlabeled data. Given an input sentence  $\mathbf{s}^s$  in source language and its counterpart in the target language  $\mathbf{s}^t$ . We first utilize name en
tity recognition tools like Spacy to select entities $^{1}$  in source language. Considering the translation of the same phrase may be different when pushing it into translation systems separately or concat it with a whole sentence. Therefore, we leverage the off-the-shelf alignment tool like GIZA++ (Pei et al., 2020) to align the corresponding words of the selected entity in the target language. After extracting entity spans and alignment $^{2}$ , we replace these selected spans with a special token [QUE] in source language sentence  $\mathbf{s}^s$ , and the corresponding alignment spans in  $\mathbf{s}^t$  act as the "ground-truth answer" span for each masked "question" ([QUE]). For instance, as the example presented in the Figure 3 (a), the span "DRM" is the selected span, we replace it in source language with [QUE] and utilize the aligned span "DRM" in target language as the correct answer.
Pre-training After getting identical language-informative spans  $S$  in  $\mathbf{s}^s$ , where each of  $S$  is replaced with a single [QUE] token  $\mathbb{Q}$ . Hence, we can get the final input sequence  $\mathbf{X}$  via concatenating the masked source language sentence  $\hat{\mathbf{s}}^s$  and unmasked target language sentence  $\mathbf{s}^t$  with two special tokens [SEP] and [CLS], as shown in Figure 3 (a). Concretely, the goal of CLISM can be seen as to predict the correct answer (corresponding aligned span in  $\mathbf{s}^t$ ) for a given  $\mathbb{Q} \in S$  based on understanding the meaning of input  $\mathbf{X}$ . Subsequently, our method converts unlabeled text into a set of specific questions that need to be answered simultaneously. Thus, in order to find the correct answer span of each  $\mathbb{Q}$ , we pass  $\mathbf{X}$  through  $\mathcal{M}$ , producing contextualized representations for each token, and then use the representation of  $\mathbb{Q}$  to compute dynamic start and end vectors  $\mathbf{s}_q$ ,  $\mathbf{e}_q$ :
$$
\mathbf {s} _ {q} = \mathbf {W} _ {s} \mathbf {x} _ {q} \quad \mathbf {e} _ {q} = \mathbf {W} _ {e} \mathbf {x} _ {q} \quad (1)
$$
where  $\mathbf{W}_s$  and  $\mathbf{W}_e$  are learnable parameters and  $\mathbf{x}_q$  denotes the representation of  $\mathbb{Q}$ . Subsequently, we follow the standard span-extraction setting (Huang et al., 2019) to obtain the start and end position probability distributions via computing the inner product of learned vectors  $\mathbf{s}_q$ ,  $\mathbf{e}_q$  with each token representation in  $\mathbf{X}$ :
<sup>1</sup>If the extracted entities of one sentence is none, we will remove it.
2We further conduct informative span selection strategy, reducing the bias of the extracted entities. Details can be seen in the Appendix B.
$$
\mathcal {P} (s = k | \mathbf {X}, \mathbb {Q}) = \frac {\exp (\mathbf {x} _ {k} ^ {T} \mathbf {s} _ {q})}{\sum_ {j} \exp (\mathbf {x} _ {j} ^ {T} \mathbf {s} _ {q})} \tag {2}
$$
$$
\mathcal {P} (e = k | \mathbf {X}, \mathsf {Q}) = \frac {\exp (\mathbf {x} _ {k} ^ {T} \mathbf {e} _ {q})}{\sum_ {j} \exp (\mathbf {x} _ {j} ^ {T} \mathbf {e} _ {q})} \qquad (3)
$$
Finally, we optimize the model via standard cross-entropy loss:
$$
\mathcal {L} _ {\text {c l i s m}} = - \log \mathcal {P} \left(s = a _ {i} ^ {s} | \mathbf {X}\right) - \log \mathcal {P} \left(e = a _ {i} ^ {e} | \mathbf {X}\right) \tag {4}
$$
where  $a_{i}$  denotes the aligned span of  $i$ -th  $\mathcal{Q}$  in  $\mathcal{S}$ . Thus, the capability of evidence extraction across different languages is enhanced during pre-training, and will be smoothly transferred to xSL during finetuning.
3.2 ContrAstive-Consistency Regularization
We design CACR from two points of view: (1) Misalignment between parallel sequences representations still hinders xPLMs advancing in downstream tasks, (2) prior works tend to learn corrupted representations (e.g., [MASK] token) which are covariant with the noise during pre-training (Luo et al., 2021). In contrast, our CACR aims at not only aiding  $\mathcal{M}$  in learning noise invariant representations during pre-training, but also promoting  $\mathcal{M}$  to better capture alignment between representations of parallel input sequences. Therefore, the proposed CACR could support CLISM to better optimize  $\mathcal{M}$  during pre-training.
Definition Aforementioned, the input sequence  $\mathbf{X} = \{\left[\mathrm{CLS}\right]\hat{\mathbf{s}}^{s}\left[\mathrm{SEP}\right]\mathbf{s}^{t}\left[\mathrm{SEP}\right]\}$ . We then take  $\mathbf{X}$  as input to get the contextual representations of sentences:
$$
\mathcal {H} = [ \mathcal {H} _ {c l s}, \hat {\mathcal {H}} ^ {s}, \mathcal {H} ^ {t}, \mathcal {H} _ {s e p} ] = \mathcal {M} (\mathbf {X}) \quad (5)
$$
where  $\mathcal{H} \in \mathbf{R}^{l \times d}$ ,  $1$  and  $d$  are the max input sequence length and hidden size, separately.  $\hat{\mathcal{H}}^s \in \mathbf{R}^{n \times d}$ ,  $\mathcal{H}^t \in \mathbf{R}^{m \times d}$  denote the representations of  $\hat{\mathbf{s}}^s$  and  $\mathbf{s}^t$ . Also  $m$  and  $n$  represent the sequence length of  $\hat{\mathbf{s}}^s$  and  $\mathbf{s}^t$ . Note that we can not directly utilize  $\hat{\mathcal{H}}^s$  and  $\mathcal{H}^t$  as the input of our CACR because of the mismatch between  $m$  and  $n$ . Thereafter, we apply an extra aggregation layer (e.g., mean-pooling)  $\mathcal{A}$  to obtain final global semantics of  $\hat{\mathbf{s}}^s$  and  $\mathbf{s}^t$ :
$$
\hat {\mathbf {r}} ^ {s} = \mathcal {A} (\hat {\mathcal {H}} ^ {s}) \qquad \mathbf {r} ^ {t} = \mathcal {A} (\mathcal {H} ^ {t}) \qquad (6)
$$
where  $\hat{\mathbf{r}}^s$  and  $\mathbf{r}^t$  belong to  $\mathbf{R}^d$ . Intuitively, the most simple implement of our CACR is to regard  $\hat{\mathbf{r}}^s$  and
$\mathbf{r}^t$  as positives and others in mini-batch are negatives to optimize the  $\mathcal{M}$  via a standard contrastive objective (Hjelm et al., 2019):
$$
\mathcal {L} \left(\mathbf {r} ^ {t}, \hat {\mathbf {r} ^ {s}}\right) = - \log \frac {\exp \left(\Psi \left(\mathbf {r} ^ {t} , \hat {\mathbf {r} ^ {s}}\right) / \tau\right)}{\sum_ {j = 1} ^ {B} \exp \left(\Psi \left(\mathbf {r} ^ {t} , \mathbf {r} _ {j}\right) / \tau\right)} (7)
$$
where  $B$  and  $\tau$  are mini-batch and temperature,  $\Psi(,)$  denotes the cosine similarity function. Nevertheless, considering  $\hat{\mathbf{s}}^s$  is the corrupted input, which contains the [QUE] token. Inspired by (Luo et al., 2021), optimizing our  $\mathcal{M}$  only with Eq.7 may lead the model to learn representations that are covariant with the noise text, causing incorrectly represented alignment between source language and target language. Therefore, we further introduce the original unmasked sentence in source language  $\mathbf{s}^s$  as input. Similarly, the pre-trained model  $\mathcal{M}$  encodes  $\mathbf{s}^s$  to obtain the hidden representation  $\mathcal{H}^s$ . Also the designed aggregation layer  $\mathcal{A}$  is responsible for getting the final semantic representation  $\mathbf{r}^s$ . Thereafter, the positive threefold is collected:  $\{\mathbf{r}^s, \mathbf{r}^t, \hat{\mathbf{r}}^s\}$ . Each representation in this set is regarded as positive, that is, we expect each two of them should be similar as possible in the latent space. By these means,  $\mathcal{M}$  is encouraged to learn noise-invariant representations and better capture cross-lingual representations alignment, alleviating pretrain-finetune discrepancy to some extent, as shown in Figure 3 (b). The final loss of our CACR can be formulated as:
$$
\mathcal {L} _ {c a c r} = \mathcal {L} \left(\mathbf {r} ^ {t}, \hat {\mathbf {r}} ^ {s}\right) + \mathcal {L} \left(\mathbf {r} ^ {s}, \hat {\mathbf {r}} ^ {s}\right) + \mathcal {L} \left(\mathbf {r} ^ {t}, \mathbf {r} ^ {s}\right) \tag {8}
$$
3.3 Pre-training Strategy
To be consistent with previous methods (Liu et al., 2019; Chi et al., 2021a), we also pre-train  $\mathcal{M}$  with the mask language modeling (MLM) training objective. In detail, we train the model in multi-task setting during pre-training. The total objective of our model can be defined as:
$$
\mathcal {L} = \mathcal {L} _ {\text {c l i s m}} + \mathcal {L} _ {\text {c a c r}} + \mathcal {L} _ {\text {m l m}} \tag {9}
$$
3.4 Fine-tuning
Inspired by (Ram et al., 2021), to be consistent with our proposed pre-training strategies, we also leverage the special token used in pre-training stage for downstream tasks. In other words, we append the [QUE] token to the input sequence during finetuning. Extensive results show this strategy achieve
Data	<en,ar>	<en,hi>	<en,vi>	Total
0.4M	0.3M	0.3M	1M
Table 1: Sentence number we used in pre-training.
better performances in xSL tasks, especially in few-shot data setting. For example, we can get the following input sequence in xMRC task with adding [QUE] after the input question. :
$$
\bar {\mathbf {X}} = \left\{\left[ \mathrm {C L S} \right] \mathbf {Q} [ \text {Q U E} ] [ \text {S E P} ] \mathbf {P} [ \text {S E P} ] \right\} \tag {10}
$$
where  $\mathbf{Q}$  and  $\mathbf{P}$  denote the input question and passage. Due to [QUE] token acts as a question during our pre-training. Hence after pre-training, we expect it captures enough information about the question, and thus, we utilize the representation of [QUE] to select the ground-truth answer span in xMRC task. The following process is the same as Section 3.1.
In this section, we first present the pre-training data collection. Then we illustrate experiments on xSL tasks, including specific implementation and detailed results.
4.1 Pre-training Data
Aforementioned, our proposed CLISM and CACR require parallel data. Therefore we utilize MT dataset (Conneau and Lample, 2019) to construct our input sequences. In detail, we only utilize four languages including English(en), Arabic(ar), Vietnamese(vi) and Hindi(hi) from MT dataset to train our model, proving our method also achieves great improvements in other languages which we don't obtain any data. Concretely, the inputs of our model consist of two languages: source language and target language. Considering the promising performance of off-the-shelf NER tools (e.g., Spacy) in English, thus we choose English as our source language and another three languages are regarded as the target language in turn. Thereafter, the number of our pre-training data is reported at Table 1, the total amount data is 1 million.
Model Structure We initialize the parameters of our model from XLM-R and Info-XLM base version published in Hugging Face Transformers<sup>3</sup>,
showing the generalization of our methods. Hence, the resulting model  $\mathcal{M}$  contains 12 transformer layers and hidden state is set to 768.