SlowGuess/ACL_2020_OCR huggingface dataset:
Benfeng Xu $^{1*}$ , Licheng Zhang $^{1*}$ , Zhendong Mao $^{1\dagger}$ , Quan Wang $^{2}$ , Hongtao Xie $^{1}$  and Yongdong Zhang $^{1}$
$^{1}$ School of Information Science and Technology,
University of Science and Technology of China, Hefei, China
$^{2}$ Beijing Research Institute,
University of Science and Technology of China, Beijing, China
{benfeng,zlczlc}@mail.ustc.edu.cn,quanwang1012@gmail.com {zdmao,htxie,zhyd73}@ustc.edu.cn
With the great success of pre-trained language models, the pretrain-finetune paradigm now becomes the undoubtedly dominant solution for natural language understanding (NLU) tasks. At the fine-tune stage, target task data is usually introduced in a completely random order and treated equally. However, examples in NLU tasks can vary greatly in difficulty, and similar to human learning procedure, language models can benefit from an easy-to-difficult curriculum. Based on this idea, we propose our Curriculum Learning approach. By reviewing the trainset in a crossed way, we are able to distinguish easy examples from difficult ones, and arrange a curriculum for language models. Without any manual model architecture design or use of external data, our Curriculum Learning approach obtains significant and universal performance improvements on a wide range of NLU tasks.
Natural Language Understanding (NLU), which requires machines to understand and reason with human language, is a crucial yet challenging problem. Recently, language model (LM) pre-training has achieved remarkable success in NLU. Pre-trained LMs learn universal language representations from large-scale unlabeled data, and can be simply finetuned with a few adjustments to adapt to various NLU tasks, showing consistent and significant improvements in these tasks (Radford et al., 2018; Devlin et al., 2018).
While lots of attention has been devoted to designing better pre-training strategies (Yang et al., 2019; Liu et al., 2019; Raffel et al., 2019), it is also valuable to explore how to more effectively solve downstream NLU tasks in the fine-tuning stage.
Easy cases:
easy, comfortable	positive
most purely enjoyable	positive
most plain, unimaginative	negative
badly edited	negative
Hard cases:
why didn’t Hollywood think of this sooner	positive
I simply can’t recommend it enough	positive
supposedly funny movie	negative
occasionally interesting	negative
Table 1: Examples from SST-2 sentiment classification task. Difficulty levels are determined by our review method (detailed later).
Most current approaches perform fine-tuning in a straightforward manner, i.e., all training examples are treated equally and presented in a completely random order during training. However, even in the same NLU task, the training examples could vary significantly in their difficulty levels, with some easily solvable by simple lexical clues while others requiring sophisticated reasoning. Table 1 shows some examples from the SST-2 sentiment classification task (Socher et al., 2013), which identifies sentiment polarities (positive or negative) of movie reviews. The easy cases can be solved directly by identifying sentiment words such as "comfortable" and "unimaginative", while the hard ones further require reasoning with negations or verb qualifiers like "supposedly" and "occasionally". Extensive research suggests that presenting training examples in a meaningful order, starting from easy ones and gradually moving on to hard ones, would benefit the learning process, not only for humans but also for machines (Skinner, 1958; Elman, 1993; Peterson, 2004; Krueger and Dayan, 2009).
Such an organization of learning materials in human learning procedure is usually referred to as Curriculum. In this paper, we draw inspiration from similar ideas, and propose our approach
6095
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
for arranging a curriculum when learning NLU tasks. Curriculum Learning (CL) is first proposed by (Bengio et al., 2009) in machine learning area, where the definition of easy examples is established ahead, and an easy-to-difficult curriculum is arranged accordingly for the learning procedure. Recent developments have successfully applied CL in computer vision areas (Jiang et al., 2017; Guo et al., 2018; Hacohen and Weinshall, 2019). It is observed in these works that by excluding the negative impact of difficult or even noisy examples in early training stage, an appropriate CL strategy can guide learning towards a better local minima in parameter space, especially for highly non-convex deep models. We argue that language models like transformer, which is hard to train (Popel and Bojar, 2018), should also benefit from CL in the context of learning NLU tasks, and such idea still remains unexplored.
The key challenge in designing a successful CL strategy lies in how to define easy/difficult examples. One straightforward way is to simply predefine the difficulty in revised rules by observing the particular target task formation or training data structure accordingly (Guo et al., 2018; Platanios et al., 2019; Tay et al., 2019). For example, (Bengio et al., 2009) utilized an easier version of shape recognition trainset which comprised of less varied shapes, before the training of complex one started. More recently, (Tay et al., 2019) considered the paragraph length of a question answering example as its reflection of difficulty. However, such strategies are highly dependent on the target dataset itself and often fails to generalize to different tasks.
To address this challenge, we propose our Cross Review method for evaluating difficulty. Specifically, we define easy examples as those well solved by the exact model that we are to employ in the task. For different tasks, we adopt their corresponding golden metrics to calculate a difficulty score for each example in the trainset. Then based on these difficulty scores, we further design a re-arranging algorithm to construct the learning curriculum in an annealing style, which provides a soft transition from easy to difficult for the model. In general, our CL approach is not constrained to any particular task, and does not rely on human prior heuristics about the task or dataset.
Experimental results show that our CL approach can greatly help language models learn in their finetune stage. Without any task-tailored model
architecture design or use of external data, we are able to obtain significant and universal improvements on a wide range of downstream NLU tasks. Our contributions can be concluded as follows:
- We explore and demonstrate the effectiveness of CL in the context of finetuning LM on NLU tasks. To the best of our knowledge, this is one of the first times that CL strategy is proved to be extensively prospective in learning NLU tasks.
- We propose a novel CL framework that consists of a Difficulty Review method and a Curriculum Arrangement algorithm, which requires no human pre-design and is very generalizable to a lot of given tasks.
- We obtain universal performance gain on a wide range of NLU tasks including Machine Reading Comprehension (MRC) and Natural Language Inference. The improvements are especially significant on tasks that are more challenging.
We describe our CL approach using BERT (Devlin et al., 2018), the most influential pre-trained LM that achieved state-of-the-art results on a wide range of NLP tasks. BERT is pretrained using Masked Language Model task and Next sentence Prediction task via large scale corpora. It consists of a hierarchical stack of  $l$  self-attention layers, which takes an input of a sequence with no more than 512 tokens and output the contextual representation of a  $H$ -dimension vector for each token in position  $i$ , which we denote as  $\pmb{h}_i^l \in \mathbb{R}^H$ . In natural language understanding tasks, the input sequences usually start with special token  $\langle \mathrm{CLS} \rangle$ , and end with  $\langle \mathrm{SEP} \rangle$ , for sequences consisting of two segments like in pairwise sentence tasks, another  $\langle \mathrm{SEP} \rangle$  is added in between for separating usage.
For target benchmarks, we employ a wide range of NLU tasks, including machine reading comprehension, sequence classification and pairwise text similarity, etc.. Following (Devlin et al., 2018), we adapt BERT for NLU tasks in the most straightforward way: simply add one necessary linear layer upon the final hidden outputs, then finetune the entire model altogether. Specifically, we brief the configurations and corresponding metrics for different tasks employed in our algorithms as follows:
6096
Machine Reading Comprehension In this work we consider the extractive MRC task. Given a passage  $P$  and a corresponding question  $Q$ , the goal is to extract a continuous span  $\langle p_{start}, p_{end} \rangle$  from  $P$  as the answer  $A$ , where the start and end are its boundaries.
We pass the concatenation of the question and paragraph  $\left[\langle \mathrm{CLS}\rangle ,Q,\langle \mathrm{SEP}\rangle ,P,\langle \mathrm{SEP}\rangle \right]$  to the pretrained LM and use a linear classifier on top of it to predict the answer span boundaries.
For the  $i - th$  input token, the probabilities that it is the start or end are calculated as:
$$
\left[ l o g i t _ {i} ^ {s t a r t}, l o g i t _ {i} ^ {e n d} \right] ^ {T} = \boldsymbol {W} _ {M R C} ^ {T} \boldsymbol {h} _ {i} ^ {l}
$$
$$
p _ {i} ^ {\text {s t a r t}} = \operatorname {s o f t m a x} (\{l o g i t _ {i} ^ {\text {s t a r t}} \})
$$
$$
p _ {i} ^ {e n d} = \mathrm {s o f t m a x} (\{l o g i t _ {i} ^ {e n d} \})
$$
where  $W_{MRC}^{T}\in \mathbb{R}^{2\times H}$  is a trainable matrix. The training objective is the log-likelihood of the true start and end positions  $y_{start}$  and  $y_{end}$ :
$$
l o s s = - \left(\log \left(p _ {y _ {s t a r t}} ^ {s t a r t}\right) + \log \left(p _ {y _ {e n d}} ^ {e n d}\right)\right)
$$
For unanswerable questions, the probability is calculated as  $s_{\mathrm{un}} = p_{\mathrm{cls}}^{start} + p_{\mathrm{cls}}^{end}$  using  $\langle \mathrm{CLS}\rangle$  representation. We classify a question into unanswerable when  $s_{\mathrm{un}} > s_{i,j} = \max_{i\leq j}(p_i^{start} + p_j^{end})$ . F1 is used as the golden metric.
Sequence Classification We consider the final contextual embedding of  $\langle \mathrm{CLS}\rangle$  token  $h_0^l$  as the pooled representation of the whole input sequence  $S$ . The probability that the input sequence belongs to label  $c$  is calculated by a linear output layer with parameter matrix  $W_{SC} \in \mathbb{R}^{K \times H}$  following a softmax:
$$
P (c | S) = \mathrm {s o f t m a x} (\pmb {h} _ {0} ^ {l} \pmb {W} _ {S C} ^ {T}),
$$
where  $K$  is the number of classes. The log-likelihood is also used as the training objective for this task. Accuracy is considered as the golden metric.
Pairwise Text Similarity Similar to sequence classification task, final embedding of  $\langle \mathrm{CLS}\rangle$  token  $h_0^l$  is used to represent the input text pair  $(T_{1},T_{2})$ . A parameter vector  $W_{PTS} \in \mathbb{R}^{H}$  is introduced to compute the similarity score:
$$
\mathrm {S i m i l a r i t y} (T _ {1}, T _ {2}) = \pmb {h} _ {0} ^ {l} \pmb {W} _ {P T S} ^ {T}.
$$
For this task, we use Mean Squared Error (MSE) as the training objective and also the golden metric:
$$
M S E = \left(y - \mathrm {S i m i l a r i t y} (T _ {1}, T _ {2})\right) ^ {2},
$$
where  $y$  is the similarity label in continuous score.
Figure 1: Our Cross Review method: the target dataset is split into N meta-datasets, after the teachers are trained on them, each example will be inferred by all other teachers (except the one it belongs to), the scores will be summed as the final evaluation results.
3 Our CL Approach
We decompose our CL framework into two stages: Difficulty Evaluation and Curriculum Arrangement. For any target task, let  $D$  be the examples set used for training, and  $\Theta$  be our language model which is expected to fit  $D$ . In the first stage, the goal is to assign each example  $d_{j}$  in  $D$  with a score  $c_{j}$  which reflects its difficulty with respect to the model. We denote  $C$  as the whole difficulty score set corresponding to trainset  $D$ . In the second stage, based on these scores,  $D$  is organized into a sequence of ordered learning stages  $\{S_{i}: i = 1,2,\dots,N\}$  with an easy-to-difficult fashion, resulting in the final curriculum where the model will be trained on. We will elaborate these two stages in section 3.1 and 3.2 respectively.
3.1 Difficulty Evaluation
The difficulty of a textual example reflects itself in many ways, e.g., the length of the context, the usage of rare words, or the scale of learning target. Although such heuristics seems reasonable to human, the model itself may not see it the same way. So we argue that difficulty score as the intrinsic properties of an example should be decided by the model itself, and the best metric should be the golden metric of the target task, which can be accuracy, F1 score, etc., as we introduced in section 2.
To perform difficulty evaluation, we first scatter our trainset  $D$  into  $N$  shares uniformly as  $\{\widetilde{D}_i:i = 1,2,\ldots ,N\}$ , and train  $N$  corresponding models  $\{\widetilde{\Theta}_i:i = 1,2,\dots ,N\}$  on them, which are all identical to  $\Theta$  (note that each model  $\widetilde{\Theta}_i$  will only see  $1 / N$  of the entire trainset). We refer to these  $N$  models as teachers, and  $\{\widetilde{D}_i\}$  as metadata- datasets for that they are attended only to collect
6097
information (i.e. the extent of difficulty) about the original trainset  $D$ . This preparing of teacher can be formulated as:
$$
\begin{array}{l} \widetilde {\Theta} _ {i} = \underset {\widetilde {\Theta} _ {i}} {\operatorname {a r g m i n}} \sum_ {d _ {j} \in \widetilde {D} _ {i}} L (d _ {j}, \widetilde {\Theta} _ {i}) \\ i = 1, 2, \dots , N \\ \end{array}
$$
where  $L$  indicates the loss function. After every teacher is respectively trained on its meta-dataset, the evaluation of trainset  $D$  should begin.
For each example  $d_{j}$ , it should be included in one and only one meta-dataset, let's assume it's  $\widetilde{D}_k$ , then we perform inference of  $d_{j}$  on all teachers except teacher  $k$ , because the inference from teacher  $k$  is supposed to be isolated with the meta-dataset  $\widetilde{D}_k$  it has already seen during training. After all inferences finished, we calculate scores of  $d_{j}$  in the target task's metric, resulting  $N - 1$  scores from  $N - 1$  different teachers:
$$
c _ {j i} = M (\widetilde {\Theta} _ {i} (x _ {j}), y _ {j})
$$
where  $\widetilde{\Theta}_i(\bullet)$  represents the inference function,  $x_{j}$  and  $y_{j}$  is the input and label of example  $d_{j}$  respectively,  $M$  is the metric calculation formula, which can be either F1, Accuracy or MSE for different tasks as introduced in section 2, and  $c_{ji}$  is the score of  $d_{j}$  from teacher  $\widetilde{\Theta}_i$ . Finally, we define the difficulty score of  $d_{j}$  as the integration of all  $N - 1$  scores:
$$
c_{j} = \sum_{\substack{i\in (1,\ldots ,N), i\neq k}}c_{ji}
$$
with all scores calculated, we obtain the final difficulty score set  $C$  as desired. We refer to our difficulty evaluation method as Cross Review(see Fig. 1)
In the proposed method, the teacher models perform their inferences in a crossed way, which prevents the meta-dataset from contaminating the inference set. Besides, each example gets its score from multi teachers, thus the fluctuation of evaluation results is greatly alleviated. In general, our Cross Review method can address the difficulty evaluation problem in an elegant design.
3.2 Curriculum Arrangement
In this section we describe our method to arrange the training examples  $D$  into a learning curriculum according to their difficulty scores  $C$ . We design our curriculum in a multi-stage setting  $\{S_{i}:i = 1,2,\ldots ,N\}$ . Within each stage  $S_{i}$ , the
examples are still shuffled to keep local stochastics, and examples from different stages do not overlap in order to prevent overfitting.
The sampling algorithm is built upon such principle:
The proportion of difficult examples in each stage should start with 0, and gradually increase until it reaches how much it accounts for in the original dataset distribution.
We first sort all examples by their difficulty score  $C$ , and divide them into N buckets:  $\{C_i : i = 1, 2, \dots, N\}$ , so the examples are now collected into N different levels of difficulty, ranging from  $C_1$  (the easiest) to  $C_N$  (the hardest), with the proportion distribution as:
$$
\operatorname {n u m} \left(C _ {1}\right): \operatorname {n u m} \left(C _ {2}\right): \dots : \operatorname {n u m} \left(C _ {N}\right)
$$
For tasks with discrete metrics, such distribution is naturally formed by the difficulty score hierarchy, and directly reflects the intrinsic difficulty distribution of the dataset. For other tasks, we manually divide  $C$  uniformly<sup>1</sup>. Based on these buckets, we construct the learning curriculum one stage after another. For each learning stage  $S_{i}$ , we sample examples from all antecedent buckets  $\{C_{j} : j = 1,2,\ldots,i\}$  by the following proportion:
$$
\frac {1}{N} \operatorname {n u m} \left(C _ {1}\right): \frac {1}{N} \operatorname {n u m} \left(C _ {2}\right): \dots : \frac {1}{N} \operatorname {n u m} \left(C _ {i}\right)
$$
and the final curriculum  $\{S_{i}:i = 1,2,\ldots ,N\}$  is formed as such. We refer to the arrangement algorithm as Annealing method for it provides a soft transition through multi learning stages.
At each stage, the model is trained for one epoch. When the training reached  $S_N$ , the model should be ready for the original distribution in trainset  $D$  so we finally add another stage  $S_{N + 1}$  which covers the entire trainset, and the model is trained on it until converges.
In this section we briefly describe three popular NLU benchmarks on which we evaluate our CL approach: SQuAD 2.0 (Rajpurkar et al., 2018), NewsQA (Trischler et al., 2016) and GLUE (Wang et al., 2018), their scale and metrics are detailed in Table 2.
1Please refer to our implementation detail for selected tasks in section 4.2
6098
SQuAD2.0	NewsQA	MNLI-m	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B
Train	130.3k	92.5k	392.7k	104.7k	363.8k	2.5k	67.3k	3.7k	8.6k	5.7k
Dev	11.9k	5.2k	9.8k	5.5k	40.4k	277	872	408	1.0k	1.5k
Test	8.9k	5.1k	9.8k	5.5k	39.1k	3.0k	1.8k	1.7k	1.0k	1.4k
Metrics	F1/EM	F1/EM	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	F1	Matthew	Pearson
SQuAD The Stanford Question Answering Dataset (SQuAD), constructed using Wikipedia articles, is a well known extractive machine reading comprehension dataset with two tasks: SQuAD1.1 (Rajpurkar et al., 2016) and SQuAD 2.0 (Rajpurkar et al., 2018). The latest 2.0 version also introduced unanswerable questions, making it a more challenging and practical task. In this paper, We take SQuAD 2.0 as our testbed.
NewsQA NewsQA (Trischler et al., 2016) is also a MRC dataset in extractive style but is much more challenging, with human performance at 0.694 F1 score. NewsQA is collected from news articles of CNN with two sets of crowdworkers, the "questioners" is provided with the article's headline only, and "answerers" is supposed to find the answer in full article. We ignore examples flagged to be without annotator agreement for better evaluation following (Fisch et al., 2019).
GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) is a collection of nine $^2$  diverse sentence or sentence pair language understanding tasks including sentiment analysis, textual entailment, sentence similarity, etc. It is considered as a well-designed benchmark that can evaluate the generalization and robustness of NLU algorithms. The labels for GLUE test set is hidden, and users must upload their predictions to obtain evaluation results, the submission is limited to protect test set from overfitting.
We use BERT Large (Devlin et al., 2018) as our pre-trained language model to demonstrate the ef
Table 2: The number of training, development, test examples and metrics of tasks used in this work.
Method	SQuAD 2.0	NewsQA
EM	F1	EM	F1
BERT Base	73.66	76.30	-	-
BERT Base*	73.66	76.78	47.70	60.10
BERT Base+CL	74.96	77.93	47.72	60.57
BERT Large	78.98	81.77	-	-
BERT Large*	79.12	82.09	50.40	64.12
BERT Large+CL	79.43	82.66	50.50	64.42
Table 3: Results on SQuAD 2.0 and NewsQA, all on development sets. Baseline on SQuAD 2.0 is obtained from (Yang et al., 2019), * indicates our reimplementation.
fectiveness of our CL approach. For MRC, we also test on BERT Base model for more comprehensive results. Besides reported results from literature, we also provide our re-implementation on all datasets, which form a more competitive baseline for comparison. The only difference between our re-implementation and our CL approach is the arrangement of curriculum, i.e., the order of training examples.
To obtain a more comparable and stable difficulty score, we binarize the review results before sum them together if possible. For accuracy as metric, the score  $c_{ji}$  is already binary in instance level, for F1 as metric, we count any review result  $c_{ji} > 0$  as correct. For other continuous metrics (MSE in this paper), we sum  $c_{ji}$  directly. We empirically choose  $N = 10$  as the number of meta-datasets for most tasks (also the number of difficulty level and the number of stages), for three datasets with rather limited scale (RTE, MRPC, and STS-B), we change it to  $N = 3$ . The scale of all datasets employed in this work is provided in Table 2. Intuitively, we shall get better results by searching for the best  $N$ , we leave it to future works due to limited computation resource.
We implement our approach based on the PyTorch implementation of BERT (Wolf et al., 2019). We use Adam (Kingma and Ba, 2014) optimizer
2The benchmark consists of: Multi-Genre NLI (MNLI) (Williams et al., 2018), Quora Question Pairs (QQP) (Shankar Iyer, 2016), Question NLI (QNLI) (Rajpurkar et al., 2016), Stanford Sentiment Treebank (SST) (Socher et al., 2013), Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019), Semantic Textual Similarity Benchmark (STS-B) (Cer et al., 2017), Microsoft Research Paragraph Corpus (MRPC) (Dolan and Brockett, 2005), Recognizing Textual Entailment (RTE) (Bentivogli et al., 2009), Winograd NLI (WNLI) (Levesque et al., 2012)
6099
MNLI-m	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B	Avg
results on dev
BERT Large	86.6	92.3	91.3	70.4	93.2	88.0	60.6	90.0	84.1
BERT Large*	86.6	92.5	91.5	74.4	93.8	91.7	63.5	90.2	85.5
BERT Large+CL	86.6	92.8	91.8	76.2	94.2	91.9	66.8	90.6	86.4
results on test
BERT Large	86.7	91.1	89.3	70.1	94.9	89.3	60.5	87.6	83.7
BERT Large*	86.3	92.2	89.5	70.2	94.4	89.3	60.5	87.3	83.7
BERT Large+CL	86.7	92.5	89.5	70.7	94.6	89.6	61.5	87.8	84.1
Table 4: Results on GLUE benchmark, * indicates our re-implementation, baselines on dev sets are obtained from (Liu et al., 2019), baselines on test sets are obtained from the leaderboard (https://gluebenchmark.com/leaderboard) submitted by (Devlin et al., 2018), they may have taken different hyperparameters. All results are produced with single task and single model.
with eplison equals to 1e-8. The learning rates warm up over the first  $5\%$  steps and then decay linearly to 0 for all experiments. To construct our reimplementation, on both SQuAD 2.0 and NewsQA we perform hyperparameter search with batch size in  $\{16,32\}$  and learning rate in  $\{1\mathrm{e} - 5,2\mathrm{e} - 5,3\mathrm{e} - 5,$ $4\mathrm{e} - 5\}$  for Base model, and  $\{32,48,64\}$ $\{5\mathrm{e} - 5,$ $6\mathrm{e} - 5,7\mathrm{e} - 5\}$  for Large model. We reuse the best parameter setting in SQuAD 2.0 on NewsQA. We set the max length of input sequence to 512 for NewsQA task because the paragraph is much more longer. On GLUE, we implement the experiments on Large model with batch size in  $\{16,32\}$  and learning rate in  $\{1\mathrm{e} - 5,2\mathrm{e} - 5,3\mathrm{e} - 5\}$ .
4.3 MRC Results