SlowGuess/NAACL_2022_OCR huggingface dataset:
Datasets and Metrics. In reasoning tests, we mainly consider three forms of reasoning, i.e., commonsense reasoning, deductive reasoning, and abductive reasoning, focusing on commonsense utilization, conclusion induction, and reason derivation, respectively. For evaluation, we choose six reasoning datasets, namely CommonsenseQA (Talmor et al., 2019), ROCStories (Mostafazadeh et al., 2016), SWAG (Zellers et al., 2018), HellaSwag (Zellers et al., 2019), Sense Making (Wang et al., 2019c), and ARCT (Habernal et al., 2018). Specifically, CommonsenseQA requires PLMs to reason about commonsense knowledge in human experience of everyday life (Liu and Singh, 2004). ROCStories, SWAG, HellaSwag, and Sense Making Task A are concerned with deriving the conclusions of stories and events, while Sense Making Task B and ARCT focus on identifying the reason behind a statement. For evaluation, we report the Accuracy results for each dataset.
Results and Analysis. Table 4 shows the model performances in reasoning ability. It can be clearly observed that performing well in comprehension tests, ALBERT and RoBERTa also achieve stronger performance in almost all reasoning tasks. In pre
Models	CNN/DailyMail	GigaWord	SQuAD	WritingPrompts
R-1	R-2	R-L	R-1	R-2	R-L	B-4	R-L	ME	B-4	R-L	ME
GPT-2	27.00	8.00	23.08	23.72	8.12	21.56	8.48	18.82	26.77	14.47	3.23	7.29
UniLm	43.44	20.21	40.51	38.45	19.45	35.75	4.42	17.43	20.13	26.88	1.84	5.01
T5	42.50	20.68	39.75	34.75	16.26	31.49	11.19	22.35	30.53	8.61	4.19	9.51
BART	44.16	21.28	40.90	39.41	20.21	36.42	15.87	25.47	38.42	14.72	3.14	7.08
ProphetNet	44.20	21.17	41.30	39.51	20.42	36.69	14.20	23.97	35.99	19.31	2.59	7.19
training, ALBERT introduces an inter-sentence coherence objective to capture the relationship among sentences, which is helpful for the sentence-level reasoning ability of PLMs. It has been found that the next sentence prediction (NSP) loss in BERT might hurt the performance of PLMs in sentence-level tasks of downstream datasets (Liu et al., 2019b). Interestingly, despite being the best in comprehension tests, XLNet does not perform as well as we expected in reasoning tests. We speculate that the permutation operation in XLNet disturbs the semantic relationship between sentences, thus leading to poor reasoning ability. To improve PLMs' reasoning ability, it would be useful to design sentence-level reasoning objectives like inter-sentence coherence loss in ALBERT. Moreover, despite incorporating knowledge, ERNIE still shows mediocre performance in knowledge-related datasets such as CQA. A possible reason might be that ERNIE only uses trained KB embeddings to enhance semantic representations but ignores the reasoning structure of KBs. This inspires us that designing appropriate and effective fusion methods to integrate knowledge is more important.
To further analyze the transferability of PLMs' reasoning ability, we conduct a two-stage study on three task datasets, i.e., ROCStories, SM-A, and ARCT. We first train PLMs on source tasks with full data and then fine-tune PLMs on target tasks with ten instances. In Figure 4, it can be observed that PLMs have better reasoning transferability between similar tasks such as deductive reasoning tasks (ROCStories and SM-A). This shows that model performance on data-scarce reasoning tasks can be improved by incorporating additional training on data-rich similar tasks (Wang et al., 2021).
3.5 Composition Tests
Datasets and Metrics. Composition is similar to the text generation task, aiming at generating new
Table 5: Composition tests results on four datasets. R-1, R-2, R-L are short for ROUGE-1, ROUGE-2, ROUGE-L respectively. B-4 and MT denote BLEU-4 and METEOR, respectively. We report the result of LARGE version for each model in this table and more results can be found in the Appendix F.
Models	GigaWord
TT (%)	Flu.	Info.	Acc.	Overall
GPT-2	26.09	3.11	2.79	2.64	4.87
UniLM	50.34	4.02	3.49	3.45	6.73
T5	53.67	3.95	3.45	3.46	6.68
BART	51.10	4.01	3.46	3.49	6.73
ProphetNet	53.02	3.99	3.52	3.45	6.74
Gold	40.77	3.61	3.29	3.15	6.05
Models	WritingPrompts
TT (%)	Flu.	Info.	Rel.	Overall
GPT-2	45.70	3.42	3.17	3.20	5.87
UniLM	1.20	1.32	1.88	2.03	2.74
T5	34.40	3.01	2.80	3.09	5.18
BART	45.20	3.37	3.16	3.39	5.96
ProphetNet	29.60	2.95	2.91	3.10	5.18
Gold	71.30	3.79	4.07	3.87	7.37
Table 6: Turing test (TT) and human scores on the test set of GigaWord and WritingPrompts. Flu., Info., Acc. and Rel. denote fluency, informativeness, accuracy and relevance respectively. We report the result of LARGE version for each model in this table and more results can be found in the Appendix F.
content from scratch. Therefore, we use four text generation benchmarks for composition tests, i.e., WritingPrompts (Fan et al., 2018) on story generation, CNN/Daily Mail (Hermann et al., 2015) and GigaWord (Rush et al., 2015) on text summarization, and SQuAD v1.1 (Rajpurkar et al., 2016) on question generation. According to the length of the target text, text summarization and question generation is short text generation, while story generation is long text generation. For evaluation, we adopt three automatic metrics, i.e., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and ME-TEOR (Banerjee and Lavie, 2005). Besides, following (Zou et al., 2021), we conduct human test from five aspects, i.e., Fluency, Informativeness, Accuracy, Relevance and Overall. The overall score is rated from 1 to 10, while the others are rated from 1 to 5. Inspired by Turing (2009), we further de
(a)GigaWord
Figure 5: Impact factors of Turing Test.
(b) WritingPrompts
sign a Turing test to assess the generation ability of PLMs, where a human interrogator is requested to distinguish whether the given text is generated by a human. From the generated texts of each model and gold texts, we randomly select 500 texts scored by judges. More details of human test and Turing test are shown in Appendix F.
Results and Analysis. Table 5 and Table 6 present the automatic evaluation and human evaluation results on composition ability, respectively. We can observe that ProphetNet and BART achieve great performance on short text generation, while GPT-2 and T5 show better results on long text generation. Specifically, BART employs denoising objectives for reconstructing the corrupted original text, and ProphetNet adopts future  $n$ -gram prediction, which is flexible for modeling the semantic relations between tokens and phrases in short texts. However, in long texts, a small ratio of masked tokens (i.e.,  $15\%$ ) might be not effective in capturing the complex long-range dependency. By comparison, the left-to-right prediction objective in GPT-2 can be more suitable to model the long-range semantic continuity in long texts, and T5 has the largest model size to achieve a strong composition ability. For composition ability, we conclude that the denoising objective is helpful for short text composition, while the left-to-right objective is more powerful for long text composition. Besides, the model size is also an important factor in improving PLMs' composition ability.
To further investigate what factors affect the pass rate of the Turing test, we deeply analyze the intermediate scoring results in the human test and Turing test. As shown in Figure 5, we calculate the pass rate of the Turing test for each human test metric across 1 to 5 scale. Moreover, we compute the Pearson correlation coefficient between the pass rate and each metric. In story genera
tion (WritingPrompts), the coefficients for Fluency, Informativeness, and Relevance are 96.63, 97.93, 96.44, respectively. While, in text summarization (GigaWord), the coefficients for Fluency, Informativeness, and Accuracy are 96.08, 97.67, 98.38, respectively. From these analysis results, we can conclude that Informativeness is more important for story generation, while Accuracy is more influential in text summarization. Besides, we compute the text similarity between the generated texts from different PLMs, which is shown in Appendix F.
Based on the above four ability tests, we intend to provide a guideline for helping researchers choose, apply, interpret and design PLMs for NLP tasks.
In section 3.3, we observe that the improvement in memory ability is likely to be helpful for the performance of comprehension ability. Hence, designing PLMs with special objectives like bidirectional language modeling in BERT and strategies like larger training batches in RoBERTa for larger memory capacity will further benefit PLMs in the downstream comprehension tasks. Besides, when applying PLMs to downstream tasks, the similarity of data distribution between pretraining and finetuning has a great impact on PLMs performance. Possible solutions such as introducing intermediate tasks or datasets can alleviate such a discrepancy. Moreover, we further find some limitations in PLMs' comprehension ability, where PLMs are good at simple single-token answer types in QA such as dates but perform worse in complex phrase answers.
Compared to comprehension, reasoning in section 3.4 is much more intricate and usually involves inferring the semantic relationships among multiple sentences. Therefore, PLMs such as ALBERT trained with sentence-level objectives can be more suitable for conducting reasoning tasks. Intuitively, incorporating sentence-level objectives during pretraining will help PLMs learn the correlation among different sentences. Note that PLMs have better reasoning transferability between similar tasks, thus data-scarce reasoning tasks can be improved by first training on data-rich tasks.
For composition ability, PLMs with denoising training objectives perform much better on short text composition, while PLMs with left-to-right objectives or larger model size are more suitable for long text composition. This might be because
PLMs with different training objectives can finally capture different ranges of semantic dependency between tokens and phrases. Moreover, to obtain a higher pass rate of Turing test, different text generation tasks will be concerned with varying factors, such as informativeness is much more critical for story generation.
Pretrained Language Models. Owing to the great achievements Transformer (Vaswani et al., 2017) has made, the paradigm of pretrained language models (PLMs) is thriving (Radford et al., 2019; Devlin et al., 2019; Liu et al., 2019b; Lewis et al., 2020; Raffel et al., 2020). It is widely recognized that PLMs can learn massive knowledge from corpora (Li et al., 2021c), leading to significant progress in various language tasks (Li et al., 2021a,b). With such encouraging results in extensive NLP tasks, it is a non-trivial topic to systematically evaluate the abilities of PLMs, which can further deepen our understanding of PLMs and facilitate their application to more fields.
Language Model Evaluation. Many efforts have studied the evaluation of language model performance. Liu et al. (2019a) evaluate BERT (Devlin et al., 2019), GPT (Radford et al., 2018), and ELMo (Peters et al., 2018) on a variety of linguistics tasks. Their findings indicate that the features generated by PLMs are sufficient for good performance on a board set of tasks but fall short on tasks requiring fine-grained linguistic knowledge. Tenney et al. (2019) evaluate similar models on a range of sub-sentence linguistic analysis tasks, showing that PLMs encode both syntax and semantics into parameters. Zhou et al. (2020) also report that PLMs can learn rich knowledge but focus on evaluating the commonsense. However, these studies only look at one dimension of PLMs ability evaluation. Other work such GLUE (Wang et al., 2019b) and CLUE (Liang Xu, 2020) just consider a simple mixture of multiple tasks lacking comprehensive evaluation. To the best of our knowledge, this is the first work to systematically evaluate PLMs by defining various kinds of language abilities and performing extensive comparison.
This paper investigates the general language ability evaluation of pretrained language models. We
design four kinds of language abilities of PLMs, including memory, comprehension, reasoning, and composition, and measure ten widely-used PLMs within five categories. For each language ability, we select multiple representative tasks to quantitatively evaluate the performance of PLMs. Our experimental results demonstrate that PLMs with varying objectives and strategies are good at different ability tests. Note that our final predicted outputs of PLMs can also be reused as an open resource for more depth and granularity in analyzing PLMs' language abilities. As a result, it is believed that this study will benefit future work about choosing or designing suitable PLMs for the target NLP tasks based on their properties.
This work was partially supported by Beijing Natural Science Foundation under Grant No. 4222027, National Natural Science Foundation of China under Grant No. 61872369 and 82161148011, Beijing Outstanding Young Scientist Program under Grant No. BJWZYJH012019100020098, and the Outstanding Innovative Talents Cultivation Funded Programs 2021 of Renmin University of China. We are grateful to Amazon Web Services for providing efficient GPU computing resource support and technical support for this NLP research project. Xin Zhao is the corresponding author.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65-72. Association for Computational Linguistics.
Virginia W Berninger. 1999. Coordinating transcription and text generation in working memory during composing: Automatic and constructive processes. Learning Disability Quarterly, 22(2):99-112.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Kate Cain and Jane Oakhill. 2008. Children's comprehension problems in oral and written language: A cognitive perspective. Guilford Press.
Robert Connors. 1997. Composition-rhetoric: Backgrounds, theory, and pedagogy. University of Pittsburgh Pre.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054.
A. H. Miller P. Lewis A. Bakhtin Y. Wu F. Petroni, T. Rocktäschel and S. Riedel. 2019. Language models as knowledge bases? In In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 889-898. Association for Computational Linguistics.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1930-1940. Association for Computational Linguistics.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693-1701.
Philip N Johnson-Laird. 1999. Deductive reasoning. Annual review of psychology, 50(1):109-135.
Alan S Kaufman and Elizabeth O Lichtenberger. 2005. Assessing adolescent and adult intelligence. John Wiley & Sons.
Patrick C Kyllonen and Raymond E Christal. 1990. Reasoning ability is (little more than) working-memory capacity?! Intelligence, 14(4):389-433.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785-794. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Junyi Li, Tianyi Tang, Gaole He, Jinhao Jiang, Xiaoxuan Hu, Puzhao Xie, Zhipeng Chen, Zhuohao Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2021a. Textbox: A unified, modularized, and extensible framework for text generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 30-39.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021b. Few-shot knowledge graph-to-text generation with pretrained language models. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1558-1568. Association for Computational Linguistics.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021c. Pretrained language model for text generation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event/Montreal, Canada, 19-27 August 2021, pages 4492-4499. ijcai.org.
Lu Li Hai Hu Chenjie Cao Weitang Liu Junyi Li Yudong Li Kai Sun Yechen Xu Yiming Cui Cong Yu Qianqian Dong Yin Tian Dian Yu Bo Shi Jun Zeng Rongzhao Wang Weijian Xie Yanting Li Yina Patterson Zuoyu Tian Yiwen Zhang He Zhou Shaoweihua Liu Qipeng Zhao Cong Yue Xinrui Zhang Zhengliang Yang Zhenzhong Lan Liang Xu, Xuanwei Zhang. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1073-1094. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Akira Miyake and Priti Shah. 1999. Models of working memory: Mechanisms of active maintenance and executive control. Cambridge University Press.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of NAACL-HLT* 2019: Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237. Association for Computational Linguistics.
Jason Phang, Phil Yeres, Jesse Swanson, Haokun Liu, Ian F. Tenney, Phu Mon Htut, Clara Vania, Alex Wang, and Samuel R. Bowman. 2020. jiant 2.0: A software toolkit for research on general-purpose text understanding models. http://jiant.info/.