Venue: NAACL 2022

# ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models

Junyi Li\({}^{1,5}\)

Tianyi Tang\({}^{1}\)

Zheng Gong\({}^{1}\)

Lixin Yang\({}^{2}\)

 Zhuohao Yu\({}^{2}\)

Zhipeng Chen\({}^{2}\)

Jingyuan Wang\({}^{3,4}\)

Wayne Xin Zhao\({}^{1,5}\) and Ji-Rong Wen\({}^{1,5}\)

###### Abstract

Nowadays, pretrained language models (PLMs) have dominated the majority of NLP tasks. While, little research has been conducted on systematically evaluating the language abilities of PLMs. In this paper, we present a large-scale empirical study on genFalanguage ability evaluation of PLMs **(ElitePLM)**. In our study, we design four evaluation dimensions, _i.e.,_ memory, comprehension, reasoning, and composition, to measure ten widely-used PLMs within five categories. Our empirical results demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; (3) PLMs have excellent transferability between similar tasks. Moreover, the prediction results of PLMs in our experiments are released as an open resource for more deep and detailed analysis on the language abilities of PLMs. This paper can guide the future work to select, apply, and design PLMs for specific tasks. We have made all the details of experiments publicly available at [https://github.com/RUCAIBox/ElitePLM](https://github.com/RUCAIBox/ElitePLM).

+
Footnote †: Corresponding author

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3519 – 3539 July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Recent years have featured a trend towards Transformer Vaswani et al. (2017) based pretrained language models (PLMs) in natural language processing (NLP) systems. By being pretrained on massive unlabeled text, PLMs can be directly fine-tuned on downstream tasks, entirely removing the need for task-specific architectures Radford et al. (2018). This paradigm has led to significant progress on many challenging NLP tasks such as reading comprehension Devlin et al. (2019) and text generation Brown et al. (2020).

With rising new state-of-the-art results that approach or surpass human performance on several tasks, it is a non-trivial research topic about how to systematically evaluate the language abilities of PLMs from a wide range of perspectives. Given a wide range of publicly released PLMs, it is particularly useful to derive principles or guidelines for selecting suitable PLMs for specific downstream tasks. However, existing works either target some single ability Talmor et al. (2020); Zhou et al. (2020), or consider a simple mixture of multiple (small-scale) tasks that lack a comprehensive design and test Wang et al. (2019b); Liang Xu (2020). There has been no detailed and systematic analysis of PLM's abilities in large-scale NLP tasks. To fill the gap of PLMs evaluation, we introduce the genFal language ability evaluation (**ElitePLM**) for empirically and systematically assessing the general language abilities of PLMs.

The ideal goal behind PLMs is to create a human-like machine learner where it can understand the language and then perform any specific task related to language. In cognitive science, Wechsler Adult Intelligence Scale (WAIS) Kaufman and Lichtenberger (2005) is the most commonly used intelligence quotient (IQ) test for measuring the intelligence and cognitive ability of humans. This test would assess the level of individuals on verbal comprehension, perceptual reasoning, working memory, and processing speed. Thus, by imitating the intelligence test on humans, we design four evaluation dimensions in ElitePLM for measuring the abilities of PLMs, including _memory_, _comprehension_, _reasoning_, and _composition_. Following previous works Zhou et al. (2020); Wang et al. (2019b), for each ability in ElitePLM, we elaborate and select multiple representative tasks (_e.g.,_ question answering for the comprehension ability) and commonly-used benchmarks (_e.g.,_ GLUE and SQuAD) to quantitatively evaluate the performance of PLMs. These results can serve as numerical explanations of PLMs at a specific ability.

In human intelligence tests, the background of participants (_e.g.,_ gender, race, and occupation)

should be as much as diverse. Thus, in ElitePLM, we select a diversity of PLMs to conduct generalized and meaningful comparisons. According to training objectives, PLMs can be divided into three types: bidirectional LMs (_e.g.,_ BERT Devlin et al. (2019)) for natural language understanding (NLU), unidirectional LMs (_e.g.,_ GPT Radford et al. (2019)) for natural language generation (NLG), and hybrid LMs (_e.g.,_ UniLM Dong et al. (2019)) for combining these two paradigms. Furthermore, knowledge-enhanced LMs (_e.g.,_ ERNIE Zhang et al. (2019)) and text-to-text LMs (_e.g.,_ T5 Raffel et al. (2020)) also emerge as important branches of PLMs. Considering the variety, we finally select ten widely-used PLMs within the above five categories and evaluate their abilities on four dimensions. We show the comparisons of these PLMs in Table 7 of Appendix A.

From the ability test results, we have three salient findings. First, PLMs with varying pretraining objectives and strategies are good at different kinds of downstream tasks. Specifically, we observe that bidirectional LMs like BERT and pretraining strategies like larger training batches in RoBERTa are helpful for memorizing pretraining corpora; permutation language modeling in XLNet is beneficial for modeling the bidirectional context in language comprehension; inter-sentence coherence objective in ALBERT is suitable for sentence-level reasoning tasks; text-to-text LMs using denoising objective like BART perform better in short text generation. Second, when fine-tuning PLMs in downstream tasks, their performance is typically sensitive to the data distribution in fine-tuning stage, which can be addressed by incorporating intermediate datasets or tasks to alleviate such a discrepancy. Third, PLMs have excellent transferability between similar tasks, especially reasoning tasks. This finding will inspire future researchers to leverage data-rich tasks for improving data-scarce tasks. For more clarity, we illustrate the impact level of each factor for PLMs' abilities in Table 8 of Appendix A.

Besides ElitePLM being an evaluation benchmark of PLMs' language ability, more importantly, the predicted results of ElitePLM can be used as an open resource for more depth and granularity in analyzing PLMs performance on each ability. For example, we further analyze the comprehension test results of PLMs across answer types in QA tasks. The analysis shows that PLMs are good at simple single-token answers such as dates but more challenged on intricate phrase answers. Moreover, by analyzing human test and Turing test results on composition, we observe that summaries with high accuracy are more likely to pass the Turing test while rich information is more important for story generation. Overall, ElitePLM can act as an analysis tool to gain more insight into PLMs. We show the details of our used datasets and predicted outputs of PLMs in Appendix B.

This paper is intended to help establish sound principles for choosing, applying, interpreting and designing PLMs for NLP tasks in practical settings. We have released the code and predicted results of each ability experiment, providing the research and industry community with off-the-shelf tools to evaluate and analyze their PLMs.

## 2 ElitePLM

In this section, we will detail these four kinds of language abilities, _i.e.,_ memory, comprehension, reasoning, and composition, in ElitePLM.

**Memory Ability.** Memory is the most basic ability of humanity, involved in how much information we recall throughout our lives Miyake and Shah (1999). By analogy, ElitePLM will measure how much knowledge and language patterns PLMs have memorized in pretraining, as assessed by tests of recalling words based on contexts. Based on the memorized information, PLMs can effectively adapt to downstream tasks for understanding and reasoning about the similar context in a specific text. On the other hand, efficiency is also a critical aspect of memory ability for PLMs learning from new data distribution in the fine-tuning stage. Thus, besides recalling words, we also compare the memory efficiency of PLMs in terms of memorizing the given new information.

**Comprehension Ability.** Comprehension is an intricate and multifaceted ability. It typically consists of understanding a text's vocabulary, background knowledge of a specific topic, and comprehension of its linguistic structures like grammar Cain and Oakhill (2008). In particular, background (prior) knowledge is used to comprehend a special situation, lesson, or text. For example, readers should be aware of the background knowledge of dog behavior when reading a text about dog training. In ElitePLM, we will assess PLMs' comprehension ability from three aspects, _i.e.,_ vocabulary, background knowledge, and linguistic structures 



**Reasoning Ability.** Based on the comprehension of a text, reasoning ability refers to the power of the processes and strategies used in drawing inferences, reaching conclusions, arriving at solutions, and making decisions (Kyllonen and Christal, 1990). In ElitePLM, we mainly focus on three types of reasoning abilities. In detail, commonsense reasoning requires PLMs to draw inferences using common-sense knowledge about the world, like the fact that "_matches_" plus "_logs_" usually equals "_fire_" (Sap et al., 2020); Note that subtle differences exist between commonsense knowledge and background knowledge in comprehension ability. Commonsense knowledge is broadly defined as the total accumulation of facts and information that a person has gained from previous experiences. Deductive reasoning involves PLMs drawing conclusions from a set of given premises in the form of categorical syllogisms (_e.g.,_ all \(x\) are \(y\)) or symbolic logic (_e.g.,_ if \(p\) then \(q\)) (Johnson-Laird, 1999); Abductive reasoning involves reaching the most likely explanation for a set of facts, such as a scientific theory to explain a set of empirical findings (Walton, 2014).

**Composition Ability.** In the literature (Connors, 1997), composition is a highly intelligent and synthetic process where a writer assembles words and sentences to create a coherent and meaningful work (_e.g.,_ poem, music, and novel) from scratch, which closely resembles to the text generation task in NLP (Berninger, 1999). Therefore, in ElitePLM, we introduce several text generation tasks to evaluate the composition ability of PLMs, including story generation, text summarization, and question generation. Note that, story generation is a representative composition task which needs PLMs to not only comprehend the given story background, but also reason about and create reasonable and coherent story endings (Fan et al., 2018). During the composition process, PLMs should include a good vocabulary, grammar, spelling, and punctuation knowledge, and deliberate the text structure.

## 3 Experiments

In this section, we first set up baselines, and then report the results and analysis on four ability tests.

### Models

As mentioned before, we compare the performance of ten publicly released PLMs from five categories: (1) _Bidirectional LMs_: BERT (Devlin et al., 2019), RoBERTa Liu et al. (2019), and ALBERT Lan et al. (2020); (2) _Unidirectional LMs_: GPT-2 (Radford et al., 2019); (3) _Hybrid LMs_: XLNet Yang et al. (2019) and UniLM Dong et al. (2019); (4) _Knowledge-enhanced LMs_: ERNIE Zhang et al. (2019); (5) _Text-to-Text LMs_: BART Lewis et al. (2020), T5 Raffel et al. (2020), and ProphetNet Qi et al. (2020). We implement these models and ability tests mostly on huggingface Wolf et al. (2020), fairseq Ott et al. (2019), and jiant Phang et al. (2020). To reflect the true level of language abilities, we adopt the best hyper-parameter values reported in their original papers for each PLM.

### Memory Tests

**Datasets and Metrics.** The goal of memory tests is to assess how much knowledge and language patterns PLMs have memorized during pretraining. For this purpose, we adopt two datasets for evaluation, _i.e.,_ LAMA (F. Petroni and Riedel, 2019) and English Wikipedia (2,500M words). Specifically, LAMA is a knowledge probe corpus containing a set of knowledge facts, where facts are either subject-relation-object triples or question-answer pairs. Each fact is converted into a cloze statement where the subject or object entity is masked. Wikipedia is one of the widely-used pretraining corpora for our selected PLMs (except GPT-2 and T5). Therefore, to conduct a fair comparison, we continuously train GPT-2 and T5 on Wikipedia using their pretraining objectives. Similar to LAMA, we randomly sample 100,000 texts from Wikipedia and then mask a proportion of \(15\%\) tokens following BERT. By querying PLMs with the missing tokens on Wikipedia and LAMA, we can test the language patterns and factual knowledge in PLMs' memory. For metrics, we use _Mean Precision at One (P@\(1\))_ of predicting missing tokens. For efficiency, we measure it as the performance _w.r.t._ the number of training epochs: the more efficient a model is, the fewer epochs to achieve a reference performance.

**Results and Analysis**. To evaluate how much text PLMs have recalled in pretraining, we directly test PLMs using Wikipedia and LAMA without fine-tuning, similar to zero-shot learning. The results on \(P\text{@}1\) metric are shown in Table 1. Compared with bidirectional and hybrid LMs (_e.g.,_ BERT and XLNet), GPT-2 uses auto-regressive self-attention where every token can only attend to the context to

its left. This unidirectional training objective naturally limits the performance of GPT-2 in terms of memorizing information. It has been previously reported that PLMs can remember more information by scaling up the model size Brown et al. (2020). However, in our tests, BART-large (400M) achieves worse results than RoBERTa-base (125M) with the same training corpus and similar vocabulary sizes (50,295 vs 50,265). During pretraining, RoBERTa adopts bidirectional objectives and novel strategies like larger training batches. It can be concluded that, as opposed to model size, **training objectives and strategies reflect the way that PLMs memorize information, making significant impacts on PLMs' memory ability**. Besides, we can clearly observe that all PLMs achieve their best results in T-REx (created from Wikipedia triples) among LAMA, and perform relatively well on Wikipedia. This implies that PLMs indeed remember a large proportion of knowledge and language patterns from pretraining corpora.

To test the memory efficiency, we fine-tune five models, BERT, ALBERT, GPT-2, BART, and XLNet, for several epochs. As shown in Figure 1, to achieve a reference performance, the bidirectional training objective like BERT needs fewer epochs than other kinds of objectives. This further implies that the bidirectional training objective is also helpful to facilitate the memory efficiency since bidirectional language modeling can make PLMs more quickly capture the language patterns.

Based on the memory test results, we further analyze how to effectively elicit the information from PLMs' memory. LAMA hand-crafts templates to test PLMs by filling the [MASK] token. Therefore, we conduct a pilot study on designing different templates for two relations in Google-RE. Table 2 shows that different templates can result in substantial differences in eliciting PLMs' memory. The bidirectional LMs, _e.g.,_ BERT, show relatively adaptability to varying templates, further verifying their strength in memory ability. Therefore, with large-scale knowledge stored in PLMs, how to derive an effective and appropriate method to provoke them is a key challenge.

\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{3}{c}{**Bidirectional**} & \multicolumn{1}{r}{**Uni.**} & \multicolumn{2}{c}{**Hybrid**} & \multicolumn{2}{c}{**KE**} & \multicolumn{2}{c}{**Text-to-Text**} \\ \cline{2-10}  & BERT & RoBERTa & ALBERT & GPT-2 & XLNet & UniLM & ERNIE & T5 & BART & ProphetNet \\ \hline
**Vocab Size** & 28996 & 50265 & 30000 & 50257 & 32000 & 28996 & 28996 & 32100 & 50295 & 30522 \\ \hline
**LAMA** & & & & & & & & & \\ Google-RE & **11.0** & 7.1 & 3.3 & 3.9 & 10.0 & 9.6 & 1.3 & 4.0 & 9.4 & 0.1 \\ T-REx & **29.2** & 23.9 & 21.0 & 12.0 & 28.9 & 28.4 & 13.4 & 21.7 & 15.8 & 1.1 \\ ConceptNet & 19.1 & **21.6** & 20.0 & 6.4 & 19.5 & 18.3 & 13.0 & 17.1 & 7.7 & 0.3 \\ SQuAD & 17.0 & **21.0** & 20.6 & 5.6 & 20.8 & 17.4 & 8.1 & 11.7 & 3.1 & 0.7 \\ \hline
**Wikipedia** & 70.9 & 71.1 & 63.9 & 42.7 & 68.7 & **71.5** & 45.7 & 65.0 & 47.8 & 31.3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Memory test results on LAMA and Wikipedia datasets (test set). These results are based on the Large version of each PLM and more results can be found in the Appendix C. Bold and underlined fonts denote the best and the second best performance of a PLM (the same as below).

Figure 1: Memory efficiency (\(P@1\)) of five PLMs on Google-RE and T-REx datasets.

 



### Comprehension Tests

**Datasets and Metrics.** In comprehension tests, we take into account three aspects of comprehension ability, including vocabulary, background knowledge, and linguistic structures. Therefore, we employ five datasets for comprehension tests, _i.e.,_ GLUE Wang et al. (2019), SuperGLUE Wang et al. (2019), SQuAD v1.1 Rajpurkar et al. (2016), SQuAD v2.0 Rajpurkar et al. (2018), and RACE Lai et al. (2017). Among these datasets, GLUE and SuperGLUE are two widely-used comprehension benchmarks. Several tasks, like word sense disambiguation and coreference resolution, can assess PLMs' understanding of vocabulary meaning and grammatical structure of a text. By contrast, SQuAD v1.1&v2.0, and RACE are three popular question answering datasets. To answer the natural language questions, PLMs should be aware of the background knowledge about some particular topic. For example, to answer the question "_what can be used as rewards for dog training?_", the background knowledge "_dogs like bones_" will be helpful for PLMs to answer "_bones_". For evaluation, we report the corresponding metrics results for each task, such as the _Matthews corr._ metric for CoLA.

**Results and Analysis**. Table 3 presents the results of comprehension test in GLUE dataset (results in other four datasets can be found in Appendix D). The last column in this table indicates the average overall performance across all tasks. Interestingly, the models behaving well in memory tests (_e.g.,_ RoBERTa and XLNet) also present good results in many comprehension tasks. The results indicate that **the improvement on memory ability is beneficial for the performance of comprehension ability**, which is in line with our intuition. Compared with bidirectional language modeling in

Figure 3: PLMs Performance on SQuAD v1.1&v2.0 stratified by five types of answer.

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & **WNLI** & **CoLA** & **MNLI** & **RTE** & **QNLI** & **SST-2** & **QQP** & **STS-B** & **MRPC** & **Avg.** \\  & Acc. & Matt. & M/MM. & Acc. & Acc. & Acc. & F1/Acc. & P/S Corr. & F1/Acc. & \\ \hline BERT\({}_{\text{BASE}}\) & 65.1 & 52.1 & 84.6/83.4 & 66.4 & 90.5 & 93.5 & 69.9/88.2 & 77.4/73.7 & 79.0/85.1 & 76.5 \\ BERT\({}_{\text{LARGE}}\) & 65.1 & 60.5 & 86.7/85.9 & 70.1 & 92.7 & 94.9 & 72.1/89.3 & 87.6/86.5 & 85.4/89.3 & 80.5 \\ RoBERT\({}_{\text{BASE}}\) & 65.1 & 61.4 & 87.4/87.2 & 75.1 & 92.9 & 95.7 & 72.5/89.4 & 89.2/88.5 & 87.5/90.7 & 81.8 \\ RoBERT\({}_{\text{BLADE}}\) & 89.0 & 67.8 & 90.8/90.2 & 82.8 & 98.9 & 96.7 & 74.9/20.2 & 92.2/91.9 & 89.9/24.4 & 88.5 \\ ALBERT\({}_{\text{XLADE}}\) & 65.8 & 58.2 & 35.6/36.5 & 62.5 & 94.2 & 95.1 & 77.8/89.8 & 67.8/66.6 & 69.8/80.3 & 72.7 \\ ALBERT\({}_{\text{XLADE}}\) & 64.4 & 64.7 & 89.7/89.6 & 70.4 & 95.3 & 96.0 & 70.7/88.4 & 91.3/90.6 & 68.1/80.4 & 80.6 \\ GPT-2\({}_{\text{SMALL}}\) & 54.8 & 33.8 & 81.1/81.4 & 62.1 & 86.7 & 91.2 & 69.8/87.9 & 79.0/76.5 & 76.9/83.6 & 71.9 \\ GPT-2\({}_{\text{MLMEM}}\) & 54.1 & 50.5 & 84.8/84.5 & 63.6 & 91.2 & 92.1 & 71.4/88.6 & 84.3/82.7 & 80.0/85.5 & 75.8 \\ XLNet\({}_{\text{BASE}}\) & 58.9 & 26.2 & 86.1/85.3 & 59.9 & 91.3 & 94.0 & 71.5/88.9 & 83.9/82.9 & 84.3/88.3 & 74.0 \\ XLRet\({}_{\text{LMAGE}}\) & **92.5** & **70.2** & **90.9/90.9** & **88.5** & **99.0** & **97.1** & **74.7/90.4** & **93.0/92.6** & **90.5/92.9** & **89.5** \\ UniLM\({}_{\text{BASE}}\) & 65.1 & 49.0 & 83.0/82.2 & 60.3 & 88.7 & 92.3 & 70.7/88.4 & 82.3/81.4 & 84.3/88.7 & 76.2 \\ UniLM\({}_{\text{LMAGE}}\) & 65.1 & 61.1 & 87.0/85.9 & 70.9 & 92.7 & 94.5 & 71.5/89.2 & 86.6/85.3 & 85.2/89.1 & 80.5 \\ ERNIE\({}_{\text{BASE}}\) & 65.1 & 52.3 & 84.0/83.2 & 68.8 & 91.3 & 93.5 & 70.5/88.4 & 85.1/83.8 & 80.3/85.9 & 70.7 \\ T5\({}_{\text{BASE}}\) & 78.8 & 51.1 & 87.1/86.2 & 80.1 & 93.7 & 95.2 & 72.6/89.4 & 89.4/88.6 & 87.5/90.7 & 82.7 \\ TS\({}_{\text{JLAGE}}\) & 85.6 & 61.2 & 89.9/89.6 & 87.2 & 94.8 & 96.3 & 73.9/89.9 & 89.9/89.2 & 89.8/92.4 & 86.4 \\ BART\({}_{\text{BASE}}\) & 65.1 & 52.8 & 85.1/84.3 & 69.5 & 92.6 & 94.4 & 72.5/89.7 & 87.6/86.6 & 86.1/89.5 & 79.5 \\ BART\({}_{\text{LABLE}}\) & 58.9 & 62.4 & 90.2/89.3 & 83.5 & 94.8 & 96.3 & 73.6/90.1 & 91.1/90.4 & 87.8/91.1 & 83.1 \\ ProphetNet\({}_{\text{LAGE}}\) & 52.1 & 24.2 & 81.3/80.8 & 51.3 & 93.2 & 93.6 & 70.6/88.1 & 73.5/72.3 & 69.7/80.8 & 69.2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comprehension tests results on GLUE (test set). All results are scored by the GLUE evaluation server1.

Figure 2: Few-shot results of four PLMs on CoLA and QNLI tasks.



BERT, permutation language modeling (relying on all permutations of the factorization order) used in XLNet enables PLMs to learn more context for enhancing PLMs' understanding of text, which seems to be effective for good comprehension ability.

Among these tasks, we observe a significant performance drop in the linguistic acceptability task (CoLA) since it has different data distribution from the pretraining corpora Wang et al. (2021). This kind of sensitiveness to unfamiliar tasks is also reflected in Figure 2, where the model performance on CoLA shows a more volatile fluctuation (ranging from 10 to 35) than QNLI (ranging from 15 to 20). It indicates that **the performance of PLMs is closely related to the similarity of data distributions in pretraining and fine-tuning**. To solve this challenge, it will be better to adopt intermediate fine-tuning, which involves first fine-tuning PLMs on an intermediate dataset similar to the final target dataset and then transferring tuned PLMs to the final dataset.

To gain more insights into PLMs' comprehension ability, we choose four representative PLMs (_i.e.,_ BERT, RoBERTa, ALBERT, and BART) and humans to analyze their performance across the answer types of SQuAD v1.1&v2.0. The results in Figure 3 show that PLMs perform well on simple answers such as dates and persons. For these categories of answers, there are usually only a few plausible candidates and most answers are single tokens. The models are more challenged on other intricate answer types (_e.g.,_ noun and verb phrases) because there are many more plausible candidates and multiple tokens. Thus, improving PLMs' understanding of intricate named entities during the pretraining phase will possibly benefit PLMs' comprehension ability later.

### Reasoning Tests

Datasets and Metrics.In reasoning tests, we mainly consider three forms of reasoning, _i.e.,_ commonsense reasoning, deductive reasoning, and abductive reasoning, focusing on commonsense utilization, conclusion induction, and reason derivation, respectively. For evaluation, we choose six reasoning datasets, namely CommonsenseQA Talmor et al. (2019), ROCStories Mostafazadeh et al. (2016), SWAG Zellers et al. (2018), HellaSwag Zellers et al. (2019), Sense Making Wang et al. (2019), and ARCT Habernal et al. (2018). Specifically, CommonsenseQA requires PLMs to reason about commonsense knowledge in human experience of everyday life Liu and Singh (2004). ROCStories, SWAG, HellaSwag, and Sense Making Task A are concerned with deriving the conclusions of stories and events, while Sense Making Task B and ARCT focus on identifying the reason behind a statement. For evaluation, we report the _Accuracy_ results for each dataset.

Results and Analysis.Table 4 shows the model performances in reasoning ability. It can be clearly observed that performing well in comprehension tests, ALBERT and RoBERTa also achieve stronger performance in almost all reasoning tasks. In pre

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Datasets**} & \multicolumn{3}{c}{**Bidirectional**} & \multicolumn{3}{c}{**Uni.**} & \multicolumn{3}{c}{**Hybrid**} & \multicolumn{3}{c}{**KE**} & \multicolumn{3}{c}{**Text-to-Text**} \\ \cline{2-11}  & BERT & RoBERTa & ALBERT & GPT-2 & XLNet & UniLM & ERNIE & T5 & BART & ProphetNet \\ \hline
**CQA** & 55.9 & 72.2 & **80.0** & 60.8 & 62.9 & 62.3 & 54.1 & 69.8 & 75.8 & 21.3 \\
**ROCStories** & 90.2 & **97.4** & 97.1 & 59.9 & 93.8 & 86.9 & 84.7 & 91.4 & 91.7 & 82.2 \\
**SWAG** & 86.3 & 89.9 & **90.7** & 79.7 & 86.8 & 83.1 & 80.2 & 73.7 & 87.9 & 70.1 \\
**HellaSwag** & 47.3 & 85.2 & **90.1** & 60.4 & 79.7 & 46.7 & 44.5 & 79.1 & 76.6 & 26.4 \\
**SM-A** & 89.4 & **93.0** & 92.5 & 88.7 & 83.7 & 89.3 & 88.7 & 92.7 & 82.9 & 85.5 \\
**SM-B** & 85.8 & **92.3** & **92.3** & 73.4 & 88.7 & 86.4 & 87.7 & 88.2 & 67.9 & 78.0 \\
**ARCT** & 71.2 & 57.9 & 79.5 & 66.7 & 83.1 & 72.3 & 73.7 & 69.4 & **84.2** & 65.5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Reasoning tests results on seven datasets (test set). CQA is short for CommonsenseQA. SM-A and SM-B denote the Task A and Task B of Sense Making, respectively. We report the results of Large version for each model in this table and more results can be found in the Appendix E.

Figure 4: Heatmaps of two-stage transfer learning.

 

training, ALBERT introduces an inter-sentence coherence objective to capture the relationship among sentences, which is helpful for the sentence-level reasoning ability of PLMs. It has been found that the next sentence prediction (NSP) loss in BERT might hurt the performance of PLMs in sentence-level tasks of downstream datasets Liu et al. (2019b). Interestingly, despite being the best in comprehension tests, XLNet does not perform as well as we expected in reasoning tests. We speculate that the permutation operation in XLNet disturbs the semantic relationship between sentences, thus leading to poor reasoning ability. **To improve PLMs' reasoning ability, it would be useful to design sentence-level reasoning objectives like inter-sentence coherence loss in ALBERT**. Moreover, despite incorporating knowledge, ERNIE still shows mediocre performance in knowledge-related datasets such as CQA. A possible reason might be that ERNIE only uses trained KB embeddings to enhance semantic representations but ignores the reasoning structure of KBs. This inspires us that designing appropriate and effective fusion methods to integrate knowledge is more important.

To further analyze the transferability of PLMs' reasoning ability, we conduct a two-stage study on three task datasets, _i.e.,_ ROCStories, SM-A, and ARCT. We first train PLMs on source tasks with full data and then fine-tune PLMs on target tasks with ten instances. In Figure 4, it can be observed that **PLMs have better reasoning transferability between similar tasks** such as deductive reasoning tasks (ROCStories and SM-A). This shows that model performance on data-scarce reasoning tasks can be improved by incorporating additional training on data-rich similar tasks Wang et al. (2021).

### Composition Tests

**Datasets and Metrics.** Composition is similar to the text generation task, aiming at generating new content from scratch. Therefore, we use four text generation benchmarks for composition tests, _i.e.,_ WritingPrompts Fan et al. (2018) on story generation, CNN/Daily Mail Hermann et al. (2015) and GigaWord Rush et al. (2015) on text summarization, and SQuAD v1.1 Rajpurkar et al. (2016) on question generation. According to the length of the target text, text summarization and question generation is short text generation, while story generation is long text generation. For evaluation, we adopt three automatic metrics, _i.e.,_ BLEU Papineni et al. (2002), ROUGE Lin (2004), and METEOR Banerjee and Lavie (2005). Besides, following Zou et al. (2021), we conduct human test from five aspects, _i.e., Fluency_, _Informativeness_, _Accuracy_, _Relevance_ and _Overall_. The overall score is rated from 1 to 10, while the others are rated from 1 to 5. Inspired by Turing (2009), we further de

\begin{table}
\begin{tabular}{l r|r r r r} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{5}{c}{**GigaWord**} \\ \cline{2-7}  & **TT (\%)** & **Flu.** & **Info.** & **Acc.** & **Overall** \\ \hline GPT-2 & 26.09 & 3.11 & 2.79 & 2.64 & 4.87 \\ UniLM & 50.34 & **4.02** & 3.49 & 3.45 & 6.73 \\ T5 & **53.67** & 3.95 & 3.45 & 3.46 & 6.68 \\ BART & 51.10 & 4.01 & 3.46 & **3.49** & 6.73 \\ ProphetNet & 53.02 & 3.99 & **3.52** & 3.45 & **6.74** \\ \hline Gold & 40.77 & 3.61 & 3.29 & 3.15 & 6.05 \\ \hline \multicolumn{7}{c}{**WritingPrompts**} \\ \cline{2-7}  & **TT (\%)** & **Flu.** & **Info.** & **Rel.** & **Overall** \\ \hline GPT-2 & **45.70** & **3.42** & **3.17** & 3.20 & 5.87 \\ UniLM & 1.20 & 1.32 & 1.88 & 2.03 & 2.74 \\ T5 & 34.40 & 3.01 & 2.80 & 3.09 & 5.18 \\ BART & 45.20 & 3.37 & 3.16 & **3.39** & **5.96** \\ ProphetNet & 29.60 & 2.95 & 2.91 & 3.10 & 5.18 \\ \hline Gold & 71.30 & 3.79 & 4.07 & 3.87 & 7.37 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Turing test (TT) and human scores on the test set of GigaWord and WritingPrompts. Flu.., Info., Acc. and Rel. denote fluency, informativeness, accuracy and relevance respectively. We report the result of Large version for each model in this table and more results can be found in the Appendix F.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{5}{c}{**CNN/DailyMail**} & \multicolumn{5}{c}{**GigaWord**} & \multicolumn{5}{c}{**SQuAD**} & \multicolumn{5}{c}{**WritingPrompts**} \\ \cline{2-11}  & R-1 & R-2 & R-L & R-1 & R-2 & R-L & B-4 & R-L & ME & B-4 & R-L & ME \\ \hline GPT-2 & 27.00 & 8.00 & 23.08 & 23.72 & 8.12 & 21.56 & 8.48 & 18.82 & 26.77 & 14.47 & 3.23 & 7.29 \\ UniLM & 43.44 & 20.21 & 40.51 & 38.45 & 19.45 & 35.75 & 4.42 & 17.43 & 20.13 & **26.88** & 1.84 & 5.01 \\ T5 & 42.50 & 20.68 & 39.75 & 34.75 & 16.26 & 31.49 & 11.19 & 22.35 & 30.53 & 8.61 & **4.19** & **9.51** \\ BART & 44.16 & **21.28** & 40.90 & 39.41 & 20.21 & 36.42 & **15.87** & **25.47** & **38.42** & 14.72 & 3.14 & 7.08 \\ ProphetNet & **44.20** & 21.17 & **41.30** & **39.51** & **20.42** & **36.69** & 14.20 & 23.97 & 35.99 & 19.31 & 2.59 & 7.19 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Composition tests on four datasets. R-1, R-2, R-L are short for ROUGE-1, ROUGE-2, ROUGE-L respectively. B-4 and MT denote BLEU-4 and METEOR, respectively. We report the result of Large version for each model in this table and more results can be found in the Appendix F.

 

sign a Turing test to assess the generation ability of PLMs, where a human interrogator is requested to distinguish whether the given text is generated by a human. From the generated texts of each model and gold texts, we randomly select 500 texts scored by judges. More details of human test and Turing test are shown in Appendix F.

**Results and Analysis**. Table 5 and Table 6 present the automatic evaluation and human evaluation results on composition ability, respectively. We can observe that ProphetNet and BART achieve great performance on short text generation, while GPT-2 and T5 show better results on long text generation. Specifically, BART employs denoising objectives for reconstructing the corrupted original text, and ProphetNet adopts future \(n\)-gram prediction, which is flexible for modeling the semantic relations between tokens and phrases in short texts. However, in long texts, a small ratio of masked tokens (_i.e.,_ 15%) might be not effective in capturing the complex long-range dependency. By comparison, the left-to-right prediction objective in GPT-2 can be more suitable to model the long-range semantic continuity in long texts, and T5 has the largest model size to achieve a strong composition ability. For composition ability, we conclude that **the denoising objective is helpful for short text composition, while the left-to-right objective is more powerful for long text composition**. Besides, the model size is also an important factor in improving PLMs' composition ability.

To further investigate what factors affect the pass rate of the Turing test, we deeply analyze the intermediate scoring results in the human test and Turing test. As shown in Figure 5, we calculate the pass rate of the Turing test for each human test metric across 1 to 5 scale. Moreover, we compute the Pearson correlation coefficient between the pass rate and each metric. In story generation (WritingPrompts), the coefficients for _Fluency_, _Informativeness_, and _Relevance_ are 96.63, 97.93, 96.44, respectively. While, in text summarization (GigaWord), the coefficients for _Fluency_, _Informativeness_, and _Accuracy_ are 96.08, 97.67, 98.38, respectively. From these analysis results, we can conclude that _Informativeness_ is more important for story generation, while _Accuracy_ is more influential in text summarization. Besides, we compute the text similarity between the generated texts from different PLMs, which is shown in Appendix F.

## 4 Discussion

Based on the above four ability tests, we intend to provide a guideline for helping researchers choose, apply, interpret and design PLMs for NLP tasks.

In section 3.3, we observe that the improvement in memory ability is likely to be helpful for the performance of comprehension ability. Hence, designing PLMs with special objectives like bidirectional language modeling in BERT and strategies like larger training batches in RoBERTa for larger memory capacity will further benefit PLMs in the downstream comprehension tasks. Besides, when applying PLMs to downstream tasks, the similarity of data distribution between pretraining and fine-tuning has a great impact on PLMs performance. Possible solutions such as introducing intermediate tasks or datasets can alleviate such a discrepancy. Moreover, we further find some limitations in PLMs' comprehension ability, where PLMs are good at simple single-token answer types in QA such as dates but perform worse in complex phrase answers.

Compared to comprehension, reasoning in section 3.4 is much more intricate and usually involves inferring the semantic relationships among multiple sentences. Therefore, PLMs such as ALBERT trained with sentence-level objectives can be more suitable for conducting reasoning tasks. Intuitively, incorporating sentence-level objectives during pretraining will help PLMs learn the correlation among different sentences. Note that PLMs have better reasoning transferability between similar tasks, thus data-scarce reasoning tasks can be improved by first training on data-rich tasks.

For composition ability, PLMs with denoising training objectives perform much better on short text composition, while PLMs with left-to-right objectives or larger model size are more suitable for long text composition. This might be because

Figure 5: Impact factors of Turing Test.

 

PLMs with different training objectives can finally capture different ranges of semantic dependency between tokens and phrases. Moreover, to obtain a higher pass rate of Turing test, different text generation tasks will be concerned with varying factors, such as informativeness is much more critical for story generation.

## 5 Related Work

**Pretrained Language Models**. Owing to the great achievements Transformer Vaswani et al. (2017) has made, the paradigm of pretrained language models (PLMs) is thriving Radford et al. (2019); Devlin et al. (2019); Liu et al. (2019b); Lewis et al. (2020); Raffel et al. (2020). It is widely recognized that PLMs can learn massive knowledge from corpora Li et al. (2021c), leading to significant progress in various language tasks Li et al. (2021a, b). With such encouraging results in extensive NLP tasks, it is a non-trivial topic to systematically evaluate the abilities of PLMs, which can further deepen our understanding of PLMs and facilitate their application to more fields.

**Language Model Evaluation**. Many efforts have studied the evaluation of language model performance. Liu et al. (2019a) evaluate BERT Devlin et al. (2019), GPT Radford et al. (2018), and ELMo Peters et al. (2018) on a variety of linguistics tasks. Their findings indicate that the features generated by PLMs are sufficient for good performance on a board set of tasks but fall short on tasks requiring fine-grained linguistic knowledge. Tenney et al. (2019) evaluate similar models on a range of sub-sentence linguistic analysis tasks, showing that PLMs encode both syntax and semantics into parameters. Zhou et al. (2020) also report that PLMs can learn rich knowledge but focus on evaluating the commonsense. However, these studies only look at one dimension of PLMs ability evaluation. Other work such GLUE Wang et al. (2019b) and CLUE Liang Xu (2020) just consider a simple mixture of multiple tasks lacking comprehensive evaluation. To the best of our knowledge, this is the first work to systematically evaluate PLMs by defining various kinds of language abilities and performing extensive comparison.

## 6 Conclusion

This paper investigates the general language ability evaluation of pretrained language models. We design four kinds of language abilities of PLMs, including memory, comprehension, reasoning, and composition, and measure ten widely-used PLMs within five categories. For each language ability, we select multiple representative tasks to quantitatively evaluate the performance of PLMs. Our experimental results demonstrate that PLMs with varying objectives and strategies are good at different ability tests. Note that our final predicted outputs of PLMs can also be reused as an open resource for more depth and granularity in analyzing PLMs' language abilities. As a result, it is believed that this study will benefit future work about choosing or designing suitable PLMs for the target NLP tasks based on their properties.

## Acknowledgement

This work was partially supported by Beijing Natural Science Foundation under Grant No. 4222027, National Natural Science Foundation of China under Grant No. 61872369 and 82161148011, Beijing Outstanding Young Scientist Program under Grant No. BJWZVJH012019100020098, and the Outstanding Innovative Talents Cultivation Funded Programs 2021 of Renmin University of China. We are grateful to Amazon Web Services for providing efficient GPU computing resource support and technical support for this NLP research project. Xin Zhao is the corresponding author.

## References

* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005_, pages 65-72. Association for Computational Linguistics.
* Berninger (1999) Virginia W Berninger. 1999. Coordinating transcription and text generation in working memory during composing: Automatic and constructive processes. _Learning Disability Quarterly_, 22(2):99-112.
* Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Cain and Oakhill (2008) Kate Cain and Jane Oakhill. 2008. _Children's comprehension problems in oral and written language: A cognitive perspective_. Guilford Press.

 

Robert Connors. 1997. _Composition-rhetoric: Backgrounds, theory, and pedagogy_. University of Pittsburgh Pre.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 13042-13054.
* Miller P. Lewis A. Bakhtin Y. Wu F. Petroni, T. Rocktaschel and S. Riedel. 2019. Language models as knowledge bases? In _In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019_.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_, pages 889-898. Association for Computational Linguistics.
* Habernal et al. (2018) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 1930-1940. Association for Computational Linguistics.
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_, pages 1693-1701.
* Johnson-Laird (1999) Philip N Johnson-Laird. 1999. Deductive reasoning. _Annual review of psychology_, 50(1):109-135.
* Kaufman and Lichtenberger (2005) Alan S Kaufman and Elizabeth O Lichtenberger. 2005. _Assessing adolescent and adult intelligence_. John Wiley & Sons.
* Kyllonen and Christal (1990) Patrick C Kyllonen and Raymond E Christal. 1990. Reasoning ability is (little more than) working-memory capacity?! _Intelligence_, 14(4):389-433.
* Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_, pages 785-794. Association for Computational Linguistics.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 7871-7880. Association for Computational Linguistics.
* Li et al. (2021a) Junyi Li, Tianyi Tang, Gaole He, Jinhao Jiang, Xiaouan Hu, Puzhao Xie, Zhipeng Chen, Zhuohao Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2021a. Textbox: A unified, modularized, and extensible framework for text generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_, pages 30-39.
* Li et al. (2021b) Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021b. Few-shot knowledge graph-to-text generation with pretrained language models. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_, volume ACL/IJCNLP 2021 of _Findings of ACL_, pages 1558-1568. Association for Computational Linguistics.
* Li et al. (2021c) Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021c. Pretrained language model for text generation: A survey. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event/ Montreal, Canada, 19-27 August 2021_, pages 4492-4499. ijcai.org.
* Ha et al. (2020) Lu Li Hai Hu Chenjie Cao Weitang Liu Junyi Li Yudong Li Kai Sun Yechen Xu Yiming Cui Cong Yu Qianqian Dong Yin Tian Dian Yu Bo Shi Jun Zeng Rongzhao Wang Weijijain Xie Yamting Li Yin Patterson Zduyou Tian Yiwen Zhang He Zhou Shaoweihua Liu Qipeng Zhao Cong Yue Xirnui Zhang Zhengliang Yang Zhenzhong Lan Liang Xu, Xuanwei Zhang. 2020. Clue: A chinese language understanding evaluation benchmark. _arXiv preprint arXiv:2004.05986_ 

* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Liu and Singh (2004) Hugo Liu and Push Singh. 2004. Concept--a practical commonsense reasoning tool-kit. _BT technology journal_, 22(4):211-226.
* Liu et al. (2019a) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 1073-1094. Association for Computational Linguistics.
* Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Miyake and Shah (1999) Akira Miyake and Priti Shah. 1999. _Models of working memory: Mechanisms of active maintenance and executive control_. Cambridge University Press.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. _arXiv preprint arXiv:1604.01696_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* Phang et al. (2020) Jason Phang, Phil Yeres, Jesse Swanson, Haokun Liu, Ian F. Tenney, Phu Mon Htut, Clara Vania, Alex Wang, and Samuel R. Bowman. 2020. jiant 2.0: A software toolkit for research on general-purpose text understanding models. [http://jiant.info/](http://jiant.info/).
* Qi et al. (2020) Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetten: Predicting future n-gram for sequence-to-sequence pre-training. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_, pages 2401-2410. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_, pages 784-789. Association for Computational Linguistics.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 2383-2392. The Association for Computational Linguistics.
* Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 379-389. The Association for Computational Linguistics.
* Sap et al. (2020) Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reasoning for natural language processing. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, ACL 2020, Online, July 5, 2020_, pages 27-33. Association for Computational Linguistics.
* on what language model pre-training captures. _Trans. Assoc. Comput. Linguistics_, 8:743-758.
* Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In _Proceedings of the 2019 Conference of 

the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers)_, pages 4149-4158. Association for Computational Linguistics.
* Tenney et al. [2019] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. _arXiv preprint arXiv:1905.06316_.
* Turing [2009] Alan M Turing. 2009. Computing machinery and intelligence. In _Parsing the turing test_, pages 23-65. Springer.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Walton [2014] Douglas Walton. 2014. _Abductive reasoning_. University of Alabama Press.
* Wang et al. [2019a] Alex Wang, Yada Pruksachatku, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 3261-3275.
* Wang et al. [2019b] Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wang et al. [2019c] Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019c. Does it make sense? and why? A pilot study for sense making and explanation. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 4020-4026. Association for Computational Linguistics.
* Wang et al. [2021] Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. _arXiv preprint arXiv:2104.14690_.
* Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 5754-5764.
* November 4, 2018_, pages 93-104. Association for Computational Linguistics.
* Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 4791-4800. Association for Computational Linguistics.
* Zhang et al. [2019] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 1441-1451. Association for Computational Linguistics.
* Zhou et al. [2020] Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pretrained language models. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 9733-9740. AAAI Press.
* Zou et al. [2021] Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. 2021. Controllable generation from pre-trained language models via inverse prompting. _arXiv preprint arXiv:2103.10685_ 



**Supplementary Material for _ElitePLM_**

We give some experiment-related information as supplementary materials. The appendix is organized into six sections:

* Configurations and pretraining setting comparisons for selected models are presented in Appendix A;
* Data statistics of each test are presented in Appendix B;
* Full results for memory tests are presented in Appendix C;
* Full results for comprehension tests are presented in Appendix D;
* Full results for reasoning tests are presented in Appendix E; and
* Full results for composition tests are presented in Appendix F.

## Appendix A Configurations of Pretrained Language Models

The selected ten PLMs within five categories and the comparisons of these PLMs in configuration and pretraining setting have been shown in Table 7. The effect extent of each factor for PLMs abilities in Table 8.

## Appendix B Data Statistics

**Memory Tests.** The data statistics of LAMA and Wikipedia of each model are presented in Table 9. Due to the differences of each PLM, we drop the data that are not in the vocabulary.

**Comprehension Tests.** The data statistics of GLUE, SuperGLUE, SQuAD and RACE are presented in Table 10.

**Reasoning Tests.** The data statistics for common-sense reasoning, deductive reasoning and abductive reasoning are presented in Table 11.

**Composition Tests.** The data statistics for text summarization, question generation and story generation are presented in Table 12. For the first three datasets, we truncate the source text considering the input length of PLMs during training. And for WritingPrompts, we reconstruct the original dataset and discard examples where text contains more than 512 tokens.

## Appendix C Memory Tests

Full results on LAMA and Wikipedia datasets are presented in Table 13.

## Appendix D Comprehension Tests

Full results on SuperGLUE, SQuAD and RACE are presented in Table 14 and Table 15.

## Appendix E Reasoning Tests

Full results on CommonsenseQA, ROCStories, SWAG, HellaSwag, Sense Making, and ARCT are presented in Table 16.

## Appendix F Composition Tests

For automatic metrics, BLEU-\(n\) and ROUGE-\(n\) compute the ratios of overlapping \(n\)-grams between generated and real text, while METEOR measures word-to-word matches based on WordNet between generated and real text. For the human test, _Fluency_ evaluates whether the text is well-formed and logical to read; _Informativeness_ measures whether the text contains useful information; _Accuracy_ tests whether the text describes the given content accurately; _Relevance_ measures whether the text is relevant to the given context; _Overall_ evaluates the overall quality of the text.

In the human test, we ramdomly select 500 generated texts for each PLM and 500 gold text. Therefore, there are 3000 texts totally. The judges are all PhD students which do not know about where each text comes from. Each text will be scored by two judges from the above five aspects, and the final score is the average of the two scores. In the Turing test, each text will also be distinguished by two judges. Only when two judges make the same decisions that the text is generated by human, we will consider the text is true.

Full results on CNN/Daily-Mail, GigaWord, SQuAD, and WritingPrompts are presented in Table 17. Turing test results are presented in Table 6. We also show some summaries and stories generated by different PLMs in Table 19, Table 20, and Table 21



\begin{table}
\begin{tabular}{c|c|c c|c c} \hline \hline  & **G-RE** & **T-REx** & **ConceptNet** & **SQuAD** & **Wikipedia** \\ \hline
**\#Origin** & 6,106 & 34,014 & 14,878 & 305 & 100,000 \\

- \\ \hline
**BERT / UniLM** & 5,527 & 34,014 & 11,658 & 305 & 85,836 \\
**RoBERTa** & 4,618 & 29,500 & 12,505 & 286 & 85,862 \\
**ALBERT** & 5,469 & 33,636 & 12,389 & 291 & 86,533 \\

**BART** & 4,618 & 29,500 & 12,505 & 286 & 85,862 \\
**T5** & 4,256 & 25,850 & 10,905 & 230 & 78,069 \\
** GPT-2** & 4,618 & 29,500 & 7,477 & 196 & 1,184 \\
**XLNet** & 5,202 & 32,293 & 12,080 & 279 & 85,228 \\
**ProphetNet** & 5,527 & 34,014 & 12,506 & 305 & 87,516 \\ \hline \multicolumn{2}{l}{**The Predicted Outputs**} & The predicted token of “[MASK] ” in each template. \\ \hline \hline \end{tabular}
\end{table}
Table 9: Statistics of datasets in memory tests, including LAMA and Wikipedia. #Origin and #Relation denote the number of examples and relations in original dataset, and the number of each model denotes the number of examples after selected. The predicted outputs is the intermediate result resources we provide.

\begin{table}
\begin{tabular}{c|c|c c|c c} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**Models**} & \multicolumn{2}{c|}{**Configurations**} & \multicolumn{2}{c}{**Pretraining Setting**} \\ \cline{3-6}  & & **Size** & **\#Parameter** & **Corpus** & **Size** \\ \hline \multirow{3}{*}{Bidirectional} & BERT & base/large & 110M/340M & BooksCorpus, English Wikipedia, & 16GB \\  & RoBERTa & base/large & 125M/355M & BooksCorpus, CC-News, & 160GB \\  & ALBERT & xlarge/xxlarge & 60M/235M & BERT Corpus & 16GB \\ \hline Unidirectional & GPT-2 & small/medium & 117M/345M & WebText (removing Wikipedia) & 40GB \\ \hline \multirow{3}{*}{Hybrid} & XLNet & base/large & 110M/340M & BooksCorpus, English Wikipedia, & 158GB \\  & UniLM & base/large & 110M/340M & BigaS, ClueWeb, Common Crawul & 16GB \\ \hline Knowledge-Enhanced & ERNIE & base & 114M & English Wikipedia, Wikipedia & 17GB \\ \hline Text-to-Text & T5 & base/large & 220M/770M & Colossal Clean Crawled Corpus & 745GB \\  & BART & base/large & 140M/400M & RoBERTa Corpus & 160GB \\  & ProphetNet & large & 373M & RoBERTa Corpus & 160GB \\ \hline \hline \end{tabular}
\end{table}
Table 7: Configurations and pretraining setting comparisons for our selected models.





\begin{table}
\begin{tabular}{c c r r r l} \hline \hline
**Reasoning Task** & **Corpus** & **\#Train** & **\#Valid** & **\#Test** & **The Predicted Outputs** \\ \hline
**Com.sense** & **CQA** & 9,741 & 1,221 & 1,140 & The predicted answer choice to a commonsense question. \\ \hline  & **ROCS.** & 1,257 & 314 & 1,571 & The predicted ending choice based on the context. \\
**Deductive** & **SWAG** & 73,546 & 20,006 & 20,005 & The predicted answer choice based the grounded situation. \\  & **HELAS.** & 39,905 & 10,042 & 10,003 & The predicted valid sentence between two sentences. \\  & **SM-A** & 10,000 & 1,000 & 1,000 & The predicted valid sentence between two sentences. \\ \hline
**Abductive** & **SM-B** & 10,000 & 1,000 & 1,000 & The predicted reason choice why the sentence is invalid. \\  & **ARCT** & 1,210 & 316 & 444 & The predicted warrant choice that justifies reason and claim. \\ \hline \hline \end{tabular}
\end{table}
Table 11: Statistics of datasets in reasoning tests, including commonsense reasoning, deductive reasoning and abductive reasoning. CQA is short for CommonsenseQA. SM-A and SM-B denote the Task A and Task B of Sense Making, respectively. The Predicted outputs is the intermediate result resources we provide.





\begin{table}
\begin{tabular}{l c r r r r r r} \hline \hline

- \\ T5BASE & 32100 & 5.5 & 20.0 & 13.2 & 9.6 & 60.5 & 36.3 \\ T5LAKGE & 32100 & 4.0 & 21.7 & 17.1 & 11.7 & 65.0 & 39.3 \\ BARTBASE & 50295 & 5.7 & 11.7 & 9.5 & 4.2 & 47.9 & 27.8 \\ BARTLLARC & 50295 & 9.4 & 15.8 & 7.7 & 3.1 & 47.8 & 28.4 \\ ProphetNELAGLE & 30522 & 0.1 & 1.1 & 0.3 & 0.7 & 31.3 & 15.9 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Memory tests results on LAMA and Wikipedia datasets (test set). We report accuracy score for each dataset. Average is computed by averaging the scores of LAMA and Wikipedia (the score of LAMA is averaged among four dataset first). LAMA-G, LAMA-T, LAMA-C and LAMA-S denote the LAMA corpus Google-RE, T-REx, ConceptNet and SQuAD, respectively.

\begin{table}
\begin{tabular}{c l c c c c c c} \hline \hline
**Task** & **Corpus** & **\#Train** & **\#Valid** & **\#Test** & **\#Input** & **\#Output** & **The Predicted Outputs** \\ \hline  & \multicolumn{2}{c}{**CNN/**} & \multirow{2}{*}{287,113} & \multirow{2}{*}{13,368} & \multirow{2}{*}{11,490} & \multirow{2}{*}{822.3} & \multirow{2}{*}{57.9} & The generated summary given a news. \\
**TS** & \multicolumn{2}{c}{**DailyMail**} & & & & & & \\
**Bigward** & & & & & & & & \\
**QG** & \multicolumn{2}{c}{**SQuAD**} & & & & & & \\ \cline{2-7}
**SG** & \multicolumn{2}{c}{**Writing**} & \multirow{2}{*}{67,765} & \multirow{2}{*}{3,952} & \multirow{2}{*}{3,784} & \multirow{2}{*}{30.2} & \multirow{2}{*}{281.2} & The generated story given a prompt and \\
**Prompts** & & & & & & & & corresponding Turing test and aspect scores. \\ \hline \hline \end{tabular}
\end{table}
Table 12: Statistics of datasets in composition tests, including text summarization (TG), question generation (QG) and story generation (SG). #Input and #Output denote the average number of tokens in the input text and output text. The Predicted outputs is the intermediate results and human evaluation resources we provide.





\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **SQuAD v1.1** & **SQuAD v2.0** & \multicolumn{4}{c}{**RACE**} \\

- & 67.8 &
- \\ T5BASE & 85.4 & 92.1 & 77.6 & 81.3 & 70.6 & 74.4 & 68.4 \\ TS1ARG & 86.7 & 93.8 &
- & 80.4 & 82.6 & 77.8 \\ BARTBASE & 84.6 & 91.0 & 76.0 & 79.2 & 70.1 & 72.4 & 63.2 \\ BARTLARE & 88.8 & 94.6 & 86.1 & 89.2 & 82.2 & 82.5 & 79.6 \\ ProphetNetlarge &
- & 74.1 &
- \\ \hline \hline \end{tabular}
\end{table}
Table 15: Comprehension tests results on SQuAD and RACE (test set).




\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline

- & 88.7 &
- & 73.7 \\ T5\({}_{\text{BASE}}\) & 61.9 & 88.2 & 65.8 & 55.2 & 89.2 & 82.9 & 63.3 \\ T5\({}_{\text{LARGE}}\) & 69.8 & 91.4 & 73.7 & 79.1 & 92.7 & 88.2 & 69.4 \\ BART\({}_{\text{BASE}}\) & 61.0 & 88.9 & 81.2 & 53.4 & 72.0 & 67.9 & 71.8 \\ BART\({}_{\text{LARGE}}\) & 75.8 & 91.7 & 87.9 & 76.6 & 82.9 & 67.9 & **84.2** \\ ProphetNet\({}_{\text{LARGE}}\) & 21.3 & 82.2 & 70.1 & 26.4 & 85.5 & 78.0 & 65.5 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Reasoning tests results on seven datasets (test set). We report accuracy score for each dataset. CQA is short for CommonsenseQA. SM-A and SM-B denote the Task A and Task B of Sense Making, respectively.

\begin{table}
\begin{tabular}{l c c|c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**CNN-DailyMail**} & \multicolumn{2}{c}{**GigaWord**} & \multicolumn{2}{c}{**SOuAD**} & \multicolumn{2}{c}{**WritingPrompts**} \\ \cline{2-11} \multirow{2}{*}{**Models**} & R-1 & R-2 & R-L & R-1 & R-2 & R-L & B-4 & R-L & ME & B-4 & R-L & ME \\ \hline GPT-2\({}_{\text{Small}}\) & 24.60 & 7.21 & 21.06 & 25.25 & 9.03 & 23.20 & 5.13 & 14.83 & 21.06 & 11.58 & 3.80 & 8.18 \\ GPT-2\({}_{\text{MDIUM}}\) & 22.95 & 5.99 & 22.08 & 23.72 & 8.12 & 21.56 & 8.48 & 18.82 & 26.77 & 14.47 & 3.23 & 7.29 \\ UniLM\({}_{\text{BASE}}\) & 17.83 & 0.11 & 5.50 & 16.64 & 6.11 & 15.12 & 4.47 & 17.65 & 20.30 & **27.71** & 2.35 & 5.47 \\ UniLM\({}_{\text{LARGE}}\) & 43.44 & 20.21 & 40.51 & 38.45 & 19.45 & 35.75 & 4.42 & 17.43 & 20.13 & 26.88 & 1.84 & 5.01 \\ T5\({}_{\text{BASE}}\) & 42.05 & 20.34 & 39.40 & 33.13 & 15.60 & 30.18 & 11.18 & 21.82 & 29.93 & 6.04 & **4.61** & **9.81** \\ TS\({}_{\text{LARGE}}\) & 42.50 & 20.68 & 39.75 & 34.75 & 16.26 & 31.49 & 11.19 & 22.35 & 30.53 & 8.61 & 4.19 & 9.51 \\ BAPTR\({}_{\text{BASE}}\) & 36.36 & 20.87 & 33.32 & 38.65 & 19.43 & 35.82 & 14.44 & 24.11 & 36.92 & 11.91 & 3.57 & 7.69 \\ BART\({}_{\text{LARGE}}\) & 44.16 & **21.28** & 40.90 & 39.41 & 20.21 & 36.42 & **15.87** & **25.47** & **38.42** & 14.72 & 3.14 & 7.08 \\ ProphetNet\({}_{\text{LARGE}}\) & **44.20** & 21.17 & **41.30** & **39.51** & **20.42** & **36.69** & 14.20 & 23.97 & 35.99 & 19.31 & 2.59 & 7.19 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Composition tests with four datasets. R-1, R-2, R-L are short for ROUGE-1, ROUGE-2, ROUGE-L respectively. B-4 and MT denote BLEU-4 and METEOR, respectively.





\begin{table}
\begin{tabular}{l l}
**Passage** & japan ’s nec corp. and UNK computer corp. of the united states said wednesday they had agreed to join forces in supercomputer sales . \\ \hline
**GPT-2\({}_{\text{MEDIUM}}\)** & unk computer to jointly sell supercomputers \\
**UniLM\({}_{\text{LARGE}}\)** & nec [UNK] to join forces in supercomputer sales \\
**TS\({}_{\text{LARGE}}\)** & nc and unk computer to join forces in supercomputer sales \\
**BART\({}_{\text{LARGE}}\)** & nec and unk computer to join forces in supercomputer sales \\
**ProphetNet\({}_{\text{LARGE}}\)** & nec unk computer to join forces in supercomputer sales \\ \hline
**Gold** & nec UNK in computer sales tie-up \\ \end{tabular} 
\begin{tabular}{l l}
**Passage** & the sri Lankan government on Wednesday announced the closure of government schools with immediate effect as a military campaign against tamil separatists escalated in the north of the country . \\ \end{tabular} 
\begin{tabular}{l l}
**GPT-2\({}_{\text{MEDIUM}}\)** & closed as military campaign escalates \\
**UniLM\({}_{\text{LARGE}}\)** & sri Lankan gov t shuts down schools in north \\
**TS\({}_{\text{LARGE}}\)** & sri Lanka closes schools as fighting escalates \\
**BART\({}_{\text{LARGE}}\)** & sri Lanka closes government schools \\
**ProphetNet\({}_{\text{LARGE}}\)** & sri Lanka closes schools as military escalates \\ \end{tabular}
\end{table}
Table 19: Test set examples of GigaWord dataset for each model.





\begin{table}
\begin{tabular}{l|p{284.5pt}}
**Prompts** & [ WP ] You have a very shitty type of preposition . \\ \hline
**GPT-2\({}_{\text{MEDIUM}}\)** & ¡‘m not sure what i’ve done wrong. ‘t’ rather nor bet able to do it. i don’t know what i have done wrong, i don’t know what it has done. i have been told that i havne n’ve been wrong. ¡newline ¿=new york. \\  & ¡york¿



\begin{table}
\begin{tabular}{l l} \hline
**Prompts** & [ WP] ” Before we allow humankind to live among us in the stars , tell us the faults of your kind. ” ‘ “newline>“ I don’t know, I’m not a scientist, but I do have a degree in astronomy, and I do know a thing or two about science. I know that a lot of people think that science is a good thing, that it’s a great thing. But, if you think about it, you’re a fucking shithole. You’ve got a bunch of crazies, all of them. So, what do you think? Do you know what? Does you know what? I mean, they ’ll tell you. And, of course, we ’d like to know what you think of us. \\ \hline
**BART\({}_{\texttt{LARGE}}\)** & “Before we allow humankind to live among us in the stars, tell us the faults of your kind. ” “ “newline>“We ll, first of all, they aren’t very smart. They don’t know how to read. They’re not very good at math. They haven’t learned how to write yet. They are also very lazy. They spend most of their time staring at their screens. They can't even get up to go to the bathroom. They just sit there and stare at the screen. They also have a tendency to state at their phones for hours at a time. I’m not sure why they do that, but I guess it’s because they’re bored.” " \\ \hline
**ProphetNet\({}_{\texttt{LARGE}}\)** & ` before we allow humankind to live among us in the stars, tell us the faults of our kind. ” “ “newline>“I mewlines>>“ Well, first of all, they aren’t very smart. They don’t know how to read. They’re not very good at math. They haven’t learned how to write yet. They are also very lazy. They spend most of their time staring at their screens. They can’t even get up to go to the bathroom. They just sit there and stare at the screen. They also have a tendency to state at their phones for hours at a time. I’m not sure why they do that, but I guess it’s because they’re bored. ” \\ \hline
**PophetNet\({}_{\texttt{LARGE}}\)** & `’ before we allow humankind to live among us in the stars, tell us the faults of your kind. ” “ “newline>“I mewlines>>“Well, first of all, they aren’t very smart. They don’t know how to read. They’re not very good at math. They haven’t learned how to write yet. They are also very lazy. They spend most of their time staring at their screens. They can’t even get up to go to the bathroom. They just sit there and stare at the screen. They also have a tendency to state at their phones for hours at a time. I’m not sure why they do that, but I guess it’s because they’re bored. ” \\ \hline
* the shibboletti
- that unlocked the cosmos ? !newline><newline>The Masters could have picked a scientist to answer but they feared she might mask ignorance . They could have picked from our global leaders bit they feared that they would mask deceit . They could have picked a holy man but feared he would mask violence , oppression , hate , intolerance ... the list of disqualifying sns was almost too long to enumerate . “cweinline><newline>So they picked Josh Thornton , a 45 year old MBA in human resources . ¡cweinline><newline>“Our greatest weakness ? Well , I think we work a little too hard and , as a race , we might be a bit of a perfectionist . \\ \hline \end{tabular}
\end{table}
Table 21: Test set examples of WritingPrompts dataset for each model.