SlowGuess/NAACL_2022_OCR huggingface dataset:
Junyi Li $^{1,5}$ , Tianyi Tang $^{1}$ , Zheng Gong $^{1}$ , Lixin Yang $^{2}$ , Zhuohao Yu $^{2}$ , Zhipeng Chen $^{2}$ , Jingyuan Wang $^{3,4}$ , Wayne Xin Zhao $^{1,5*}$  and Ji-Rong Wen $^{1,5}$
<sup>1</sup>Gaoling School of Artificial Intelligence, Renmin University of China
$^{2}$ Renmin University of China  $^{3}$ Peng Cheng Laboratory
$^{4}$ School of Computer Science and Engineering, Beihang University
$^{5}$ Beijing Key Laboratory of Big Data Management and Analysis Methods
{lijunyi,steven_tang}@ruc.edu.cn batmanfly@gmail.com
Nowadays, pretrained language models (PLMs) have dominated the majority of NLP tasks. While, little research has been conducted on systematically evaluating the language abilities of PLMs. In this paper, we present a large-scale empirical study on general language ability evaluation of PLMs (ElitePLM). In our study, we design four evaluation dimensions, i.e., memory, comprehension, reasoning, and composition, to measure ten widely-used PLMs within five categories. Our empirical results demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; (3) PLMs have excellent transferability between similar tasks. Moreover, the prediction results of PLMs in our experiments are released as an open resource for more deep and detailed analysis on the language abilities of PLMs. This paper can guide the future work to select, apply, and design PLMs for specific tasks. We have made all the details of experiments publicly available at https://github.com/RUCAIBox/ElitePLM.
Recent years have featured a trend towards Transformer (Vaswani et al., 2017) based pretrained language models (PLMs) in natural language processing (NLP) systems. By being pretrained on massive unlabeled text, PLMs can be directly fine-tuned on downstream tasks, entirely removing the need for task-specific architectures (Radford et al., 2018). This paradigm has led to significant progress on many challenging NLP tasks such as reading comprehension (Devlin et al., 2019) and text generation (Brown et al., 2020).
With rising new state-of-the-art results that approach or surpass human performance on several tasks, it is a non-trivial research topic about how
to systematically evaluate the language abilities of PLMs from a wide range of perspectives. Given a wide range of publicly released PLMs, it is particularly useful to derive principles or guidelines for selecting suitable PLMs for specific downstream tasks. However, existing works either target some single ability (Talmor et al., 2020; Zhou et al., 2020), or consider a simple mixture of multiple (small-scale) tasks that lack a comprehensive design and test (Wang et al., 2019b; Liang Xu, 2020). There has been no detailed and systematic analysis of PLM's abilities in large-scale NLP tasks. To fill the gap of PLMs evaluation, we introduce the general language ability evaluation (ElitePLM) for empirically and systematically assessing the general language abilities of PLMs.
The ideal goal behind PLMs is to create a human-like machine learner where it can understand the language and then perform any specific task related to language. In cognitive science, Wechsler Adult Intelligence Scale (WAIS) (Kaufman and Lichtenberger, 2005) is the most commonly used intelligence quotient (IQ) test for measuring the intelligence and cognitive ability of humans. This test would assess the level of individuals on verbal comprehension, perceptual reasoning, working memory, and processing speed. Thus, by imitating the intelligence test on humans, we design four evaluation dimensions in ElitePLM for measuring the abilities of PLMs, including memory, comprehension, reasoning, and composition. Following previous works (Zhou et al., 2020; Wang et al., 2019b), for each ability in ElitePLM, we elaborate and select multiple representative tasks (e.g., question answering for the comprehension ability) and commonly-used benchmarks (e.g., GLUE and SQuAD) to quantitatively evaluate the performance of PLMs. These results can serve as numerical explanations of PLMs at a specific ability.
In human intelligence tests, the background of participants (e.g., gender, race, and occupation)
Human Language Technologies, pages 3519 - 3539
should be as much as diverse. Thus, in ElitePLM, we select a diversity of PLMs to conduct generalized and meaningful comparisons. According to training objectives, PLMs can be divided into three types: bidirectional LMs (e.g., BERT (Devlin et al., 2019)) for natural language understanding (NLU), unidirectional LMs (e.g., GPT (Radford et al., 2019)) for natural language generation (NLG), and hybrid LMs (e.g., UniLM (Dong et al., 2019)) for combining these two paradigms. Furthermore, knowledge-enhanced LMs (e.g., ERNIE (Zhang et al., 2019)) and text-to-text LMs (e.g., T5 (Raffel et al., 2020)) also emerge as important branches of PLMs. Considering the variety, we finally select ten widely-used PLMs within the above five categories and evaluate their abilities on four dimensions. We show the comparisons of these PLMs in Table 7 of Appendix A.
From the ability test results, we have three salient findings. First, PLMs with varying pretraining objectives and strategies are good at different kinds of downstream tasks. Specifically, we observe that bidirectional LMs like BERT and pretraining strategies like larger training batches in RoBERTa are helpful for memorizing pretraining corpora; permutation language modeling in XLNet is beneficial for modeling the bidirectional context in language comprehension; inter-sentence coherence objective in ALBERT is suitable for sentence-level reasoning tasks; text-to-text LMs using denoising objective like BART perform better in short text generation. Second, when fine-tuning PLMs in downstream tasks, their performance is typically sensitive to the data distribution in fine-tuning stage, which can be addressed by incorporating intermediate datasets or tasks to alleviate such a discrepancy. Third, PLMs have excellent transferability between similar tasks, especially reasoning tasks. This finding will inspire future researchers to leverage data-rich tasks for improving data-scarce tasks. For more clarity, we illustrate the impact level of each factor for PLMs' abilities in Table 8 of Appendix A.
Besides ElitePLM being an evaluation benchmark of PLMs' language ability, more importantly, the predicted results of ElitePLM can be used as an open resource for more depth and granularity in analyzing PLMs performance on each ability. For example, we further analyze the comprehension test results of PLMs across answer types in QA tasks. The analysis shows that PLMs are good at simple single-token answers such as dates but more
challenged on intricate phrase answers. Moreover, by analyzing human test and Turing test results on composition, we observe that summaries with high accuracy are more likely to pass the Turing test while rich information is more important for story generation. Overall, ElitePLM can act as an analysis tool to gain more insight into PLMs. We show the details of our used datasets and predicted outputs of PLMs in Appendix B.
This paper is intended to help establish sound principles for choosing, applying, interpreting and designing PLMs for NLP tasks in practical settings. We have released the code and predicted results of each ability experiment, providing the research and industry community with off-the-shelf tools to evaluate and analyze their PLMs.
2 ElitePLM
In this section, we will detail these four kinds of language abilities, i.e., memory, comprehension, reasoning, and composition, in ElitePLM.
Memory Ability. Memory is the most basic ability of humanity, involved in how much information we recall throughout our lives (Miyake and Shah, 1999). By analogy, ElitePLM will measure how much knowledge and language patterns PLMs have memorized in pretraining, as assessed by tests of recalling words based on contexts. Based on the memorized information, PLMs can effectively adapt to downstream tasks for understanding and reasoning about the similar context in a specific text. On the other hand, efficiency is also a critical aspect of memory ability for PLMs learning from new data distribution in the fine-tuning stage. Thus, besides recalling words, we also compare the memory efficiency of PLMs in terms of memorizing the given new information.
Comprehension Ability. Comprehension is an intricate and multifaceted ability. It typically consists of understanding a text's vocabulary, background knowledge of a specific topic, and comprehension of its linguistic structures like grammar (Cain and Oakhill, 2008). In particular, background (prior) knowledge is used to comprehend a special situation, lesson, or text. For example, readers should be aware of the background knowledge of dog behavior when reading a text about dog training. In ElitePLM, we will assess PLMs' comprehension ability from three aspects, i.e., vocabulary, background knowledge, and linguistic structures.
Reasoning Ability. Based on the comprehension of a text, reasoning ability refers to the power of the processes and strategies used in drawing inferences, reaching conclusions, arriving at solutions, and making decisions (Kyllonen and Christal, 1990). In ElitePLM, we mainly focus on three types of reasoning abilities. In detail, commonsense reasoning requires PLMs to draw inferences using commonsense knowledge about the world, like the fact that "matches" plus "logs" usually equals "fire" (Sap et al., 2020); Note that subtle differences exist between commonsense knowledge and background knowledge in comprehension ability. Commonsense knowledge is broadly defined as the total accumulation of facts and information that a person has gained from previous experiences. Deductive reasoning involves PLMs drawing conclusions from a set of given premises in the form of categorical syllogisms (e.g., all  $x$  are  $y$ ) or symbolic logic (e.g., if  $p$  then  $q$ ) (Johnson-Laird, 1999); Abductive reasoning involves reaching the most likely explanation for a set of facts, such as a scientific theory to explain a set of empirical findings (Walton, 2014).
Composition Ability. In the literature (Connors, 1997), composition is a highly intelligent and synthetic process where a writer assembles words and sentences to create a coherent and meaningful work (e.g., poem, music, and novel) from scratch, which closely resembles to the text generation task in NLP (Berninger, 1999). Therefore, in ElitePLM, we introduce several text generation tasks to evaluate the composition ability of PLMs, including story generation, text summarization, and question generation. Note that, story generation is a representative composition task which needs PLMs to not only comprehend the given story background, but also reason about and create reasonable and coherent story endings (Fan et al., 2018). During the composition process, PLMs should include a good vocabulary, grammar, spelling, and punctuation knowledge, and deliberate the text structure.
In this section, we first set up baselines, and then report the results and analysis on four ability tests.
As mentioned before, we compare the performance of ten publicly released PLMs from five categories: (1) Bidirectional LMs: BERT (Devlin et al., 2019),
RoBERTa (Liu et al., 2019b), and ALBERT (Lan et al., 2020); (2) Unidirectional LMs: GPT-2 (Radford et al., 2019); (3) Hybrid LMs: XLNet (Yang et al., 2019) and UniLM (Dong et al., 2019); (4) Knowledge-enhanced LMs: ERNIE (Zhang et al., 2019); (5) Text-to-Text LMs: BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and ProphetNet (Qi et al., 2020). We implement these models and ability tests mostly on huggingface (Wolf et al., 2020), fairseq (Ott et al., 2019), and jiant (Phang et al., 2020). To reflect the true level of language abilities, we adopt the best hyper-parameter values reported in their original papers for each PLM.
3.2 Memory Tests
Datasets and Metrics. The goal of memory tests is to assess how much knowledge and language patterns PLMs have memorized during pretraining. For this purpose, we adopt two datasets for evaluation, i.e., LAMA (F. Petroni and Riedel, 2019) and English Wikipedia (2,500M words). Specifically, LAMA is a knowledge probe corpus containing a set of knowledge facts, where facts are either subject-relation-object triples or question-answer pairs. Each fact is converted into a cloze statement where the subject or object entity is masked. Wikipedia is one of the widely-used pretraining corpora for our selected PLMs (except GPT-2 and T5). Therefore, to conduct a fair comparison, we continuously train GPT-2 and T5 on Wikipedia using their pretraining objectives. Similar to LAMA, we randomly sample 100,000 texts from Wikipedia and then mask a proportion of  $15\%$  tokens following BERT. By querying PLMs with the missing tokens on Wikipedia and LAMA, we can test the language patterns and factual knowledge in PLMs' memory. For metrics, we use Mean Precision at One  $(P@1)$  of predicting missing tokens. For efficiency, we measure it as the performance w.r.t. the number of training epochs: the more efficient a model is, the fewer epochs to achieve a reference performance.
Results and Analysis. To evaluate how much text PLMs have recalled in pretraining, we directly test PLMs using Wikipedia and LAMA without finetuning, similar to zero-shot learning. The results on  $P@1$  metric are shown in Table 1. Compared with bidirectional and hybrid LMs (e.g., BERT and XLNet), GPT-2 uses auto-regressive self-attention where every token can only attend to the context to
Models	Bidirectional	Uni.	Hybrid	KE	Text-to-Text
BERT	RoBERTa	ALBERT	GPT-2	XLNet	UniLM	ERNIE	T5	BART	ProphetNet
Vocab Size	28996	50265	30000	50257	32000	28996	28996	32100	50295	30522
LAMA
Google-RE	11.0	7.1	3.3	3.9	10.0	9.6	1.3	4.0	9.4	0.1
T-REx	29.2	23.9	21.0	12.0	28.9	28.4	13.4	21.7	15.8	1.1
ConceptNet	19.1	21.6	20.0	6.4	19.5	18.3	13.0	17.1	7.7	0.3
SQuAD	17.0	21.0	20.6	5.6	20.8	17.4	8.1	11.7	3.1	0.7
Wikipedia	70.9	71.1	63.9	42.7	68.7	71.5	45.7	65.0	47.8	31.3
Table 1: Memory test results on LAMA and Wikipedia datasets (test set). These results are based on the LARGE version of each PLM and more results can be found in the Appendix C. Bold and underlined fonts denote the best and the second best performance of a PLM (the same as below).
Relation	Template	BERT	RoBERTa	GPT-2	BART	T5
<X], place_of_death, [Y]>	[X] died in [MASK].	13.98	0.46	0.15	11.09	4.19
[X] passed away in [MASK].	13.46	0.46	0.62	3.54	1.51
[X]'s place of death was [MASK].	3.27	0.00	0.00	0.00	1.51
<X], place_of_birth, [Y]>	[X] was born in [MASK].	16.07	12.52	7.53	14.77	6.32
[X] was born in the place of [MASK].	2.83	1.29	0.00	0.00	1.39
[X]'s place of birth was [MASK].	12.16	1.87	0.00	0.00	3.12
Table 2: The impact of template on eliciting PLMs' stored knowledge.
(a) Google-RE
Figure 1: Memory efficiency  $(P@1)$  of five PLMs on Google-RE and T-REx datasets.
(b) T-REx
its left. This unidirectional training objective naturally limits the performance of GPT-2 in terms of memorizing information. It has been previously reported that PLMs can remember more information by scaling up the model size (Brown et al., 2020). However, in our tests, BART-large (400M) achieves worse results than RoBERTa-base (125M) with the same training corpus and similar vocabulary sizes (50,295 vs 50,265). During pretraining, RoBERTa adopts bidirectional objectives and novel strategies like larger training batches. It can be concluded that, as opposed to model size, training objectives and strategies reflect the way that PLMs memorize information, making significant impacts on PLMs' memory ability. Besides, we can clearly observe that all PLMs achieve their best results
in T-REx (created from Wikipedia triples) among LAMA, and perform relatively well on Wikipedia. This implies that PLMs indeed remember a large proportion of knowledge and language patterns from pretraining corpora.
To test the memory efficiency, we fine-tune five models, BERT, ALBERT, GPT-2, BART, and XLNet, for several epochs. As shown in Figure 1, to achieve a reference performance, the bidirectional training objective like BERT needs fewer epochs than other kinds of objectives. This further implies that the bidirectional training objective is also helpful to facilitate the memory efficiency since bidirectional language modeling can make PLMs more quickly capture the language patterns.
Based on the memory test results, we further analyze how to effectively elicit the information from PLMs' memory. LAMA hand-crafts templates to test PLMs by filling the [MASK] token. Therefore, we conduct a pilot study on designing different templates for two relations in Google-RE. Table 2 shows that different templates can result in substantial differences in eliciting PLMs' memory. The bidirectional LMs, e.g., BERT, show relatively adaptability to varying templates, further verifying their strength in memory ability. Therefore, with large-scale knowledge stored in PLMs, how to derive an effective and appropriate method to provoke them is a key challenge.
Models	WNLI Acc.	CoLA Matt.	MNLI M./MM.	RTE Acc.	QNLI Acc.	SST-2 Acc.	QQP F1/Acc.	STS-B P/S Corr.	MRPC F1/Acc.	Avg.
BERTBASE	65.1	52.1	84.6/83.4	66.4	90.5	93.5	69.9/88.2	77.4/73.7	79.0/85.1	76.5
BERTLARGE	65.1	60.5	86.7/85.9	70.1	92.7	94.9	72.1/89.3	87.6/86.5	85.4/89.3	80.5
RoBERTaBASE	65.1	61.4	87.4/87.2	75.1	92.9	95.7	72.5/89.4	89.2/88.5	87.5/90.7	81.8
RoBERTaLARGE	89.0	67.8	90.8/90.2	88.2	98.9	96.7	74.3/90.2	92.2/91.9	89.9/92.4	88.5
ALBERTXLARGE	65.8	58.2	35.6/36.5	62.5	94.2	95.1	71.7/88.9	87.6/86.6	69.8/80.3	72.7
ALBERTXXLARGE	64.4	64.7	89.7/89.6	70.4	95.3	96.0	70.7/88.4	91.3/90.6	68.1/80.4	80.6
GPT-2SMALL	54.8	33.8	81.1/81.4	62.1	86.7	91.2	69.8/87.9	79.0/76.5	76.9/83.6	71.9
GPT-2MEDIUM	54.1	50.5	84.8/84.5	63.6	91.2	92.1	71.4/88.6	84.3/82.7	80.0/85.5	75.8
XLNetBASE	58.9	26.2	86.1/85.3	59.9	91.3	94.0	71.5/88.9	83.9/82.9	84.3/88.3	74.0
XLNetLARGE	92.5	70.2	90.9/90.9	88.5	99.0	97.1	74.7/90.4	93.0/92.6	90.5/92.9	89.5
UniLMBASE	65.1	49.0	83.0/82.2	60.3	88.7	92.3	70.7/88.4	82.3/81.4	84.3/88.7	76.2
UniLMLARGE	65.1	61.1	87.0/85.9	70.9	92.7	94.5	71.5/89.2	86.6/85.3	85.2/89.1	80.5
ERNIEBASE	65.1	52.3	84.0/83.2	68.8	91.3	93.5	70.5/88.4	85.1/83.8	80.3/85.9	70.7
T5BASE	78.8	51.1	87.1/86.2	80.1	93.7	95.2	72.6/89.4	89.4/88.6	87.5/90.7	82.7
T5LARGE	85.6	61.2	89.9/89.6	87.2	94.8	96.3	73.9/89.9	89.9/89.2	89.8/92.4	86.4
BARTBASE	65.1	52.8	85.1/84.3	69.5	92.6	94.4	72.5/89.7	87.6/86.6	86.1/89.5	79.5
BARTLARGE	58.9	62.4	90.2/89.3	83.5	94.8	96.3	73.6/90.1	91.1/90.4	87.8/91.1	83.1
ProphetNetLARGE	52.1	24.2	81.3/80.8	51.3	93.2	93.6	70.6/88.1	73.5/72.3	69.7/80.8	69.2
Table 3: Comprehension tests results on GLUE (test set). All results are scored by the GLUE evaluation server  ${}^{1}$  .
(a) CoLA
(b)QNLI
3.3 Comprehension Tests
Datasets and Metrics. In comprehension tests, we take into account three aspects of comprehension ability, including vocabulary, background knowledge, and linguistic structures. Therefore, we employ five datasets for comprehension tests, i.e., GLUE (Wang et al., 2019b), SuperGLUE (Wang et al., 2019a), SQuAD v1.1 (Rajpurkar et al., 2016), SQuAD v2.0 (Rajpurkar et al., 2018), and RACE (Lai et al., 2017). Among these datasets, GLUE and SuperGLUE are two widely-used comprehension benchmarks. Several tasks, like word sense disambiguation and coreference resolution, can assess PLMs' understanding of vocabulary meaning and grammatical structure of a text. By contrast, SQuAD v1.1&v2.0, and RACE are three popular question answering datasets. To answer the
Figure 2: Few-shot results of four PLMs on CoLA and QNLI tasks.
Figure 3: PLMs Performance on SQuAD v1.1&v2.0 stratified by five types of answer.
natural language questions, PLMs should be aware of the background knowledge about some particular topic. For example, to answer the question "what can be used as rewards for dog training?", the background knowledge "dogs like bones" will be helpful for PLMs to answer "bones". For evaluation, we report the corresponding metrics results for each task, such as the Matthews corr. metric for CoLA.
Results and Analysis. Table 3 presents the results of comprehension test in GLUE dataset (results in other four datasets can be found in Appendix D). The last column in this table indicates the average overall performance across all tasks. Interestingly, the models behaving well in memory tests (e.g., RoBERTa and XLNet) also present good results in many comprehension tasks. The results indicate that the improvement on memory ability is beneficial for the performance of comprehension ability, which is in line with our intuition. Compared with bidirectional language modeling in
'https://gluebenchmark.com/
Datasets	Bidirectional	Uni.	Hybrid	KE	Text-to-Text
BERT	RoBERTa	ALBERT	GPT-2	XLNet	UniLM	ERNIE	T5	BART	ProphetNet
CQA	55.9	72.2	80.0	60.8	62.9	62.3	54.1	69.8	75.8	21.3
ROCStories	90.2	97.4	97.1	59.9	93.8	86.9	84.7	91.4	91.7	82.2
SWAG	86.3	89.9	90.7	79.7	86.8	83.1	80.2	73.7	87.9	70.1
HellaSwag	47.3	85.2	90.1	60.4	79.7	46.7	44.5	79.1	76.6	26.4
SM-A	89.4	93.0	92.5	88.7	83.7	89.3	88.7	92.7	82.9	85.5
SM-B	85.8	92.3	92.3	73.4	88.7	86.4	87.7	88.2	67.9	78.0
ARCT	71.2	57.9	79.5	66.7	83.1	72.3	73.7	69.4	84.2	65.5
Table 4: Reasoning tests results on seven datasets (test set). CQA is short for CommonsenseQA. SM-A and SM-B denote the Task A and Task B of Sense Making, respectively. We report the results of LARGE version for each model in this table and more results can be found in the Appendix E.
BERT, permutation language modeling (relying on all permutations of the factorization order) used in XLNet enables PLMs to learn more context for enhancing PLMs' understanding of text, which seems to be effective for good comprehension ability.
Among these tasks, we observe a significant performance drop in the linguistic acceptability task (CoLA) since it has different data distribution from the pretraining corpora (Wang et al., 2021). This kind of sensitiveness to unfamiliar tasks is also reflected in Figure 2, where the model performance on CoLA shows a more volatile fluctuation (ranging from 10 to 35) than QNLI (ranging from 15 to 20). It indicates that the performance of PLMs is closely related to the similarity of data distributions in pretraining and fine-tuning. To solve this challenge, it will be better to adopt intermediate fine-tuning, which involves first fine-tuning PLMs on an intermediate dataset similar to the final target dataset and then transferring tuned PLMs to the final dataset.
To gain more insights into PLMs' comprehension ability, we choose four representative PLMs (i.e., BERT, RoBERTa, ALBERT, and BART) and humans to analyze their performance across the answer types of SQuAD v1.1&v2.0. The results in Figure 3 show that PLMs perform well on simple answers such as dates and persons. For these categories of answers, there are usually only a few plausible candidates and most answers are single tokens. The models are more challenged on other intricate answer types (e.g., noun and verb phrases) because there are many more plausible candidates and multiple tokens. Thus, improving PLMs' understanding of intricate named entities during the pretraining phase will possibly benefit PLMs' comprehension ability later.
(a) BERTLARGE
Figure 4: Heatmaps of two-stage transfer learning.
(b) T5LARGE
3.4 Reasoning Tests