SlowGuess/NAACL_2022_OCR huggingface dataset:
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 2401-2410. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784-789. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 379-389. The Association for Computational Linguistics.
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reasoning for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, ACL 2020, Online, July 5, 2020, pages 27-33. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. olympics - on what language model pre-training captures. Trans. Assoc. Comput. Linguistics, 8:743-758.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4149-4158. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316.
Alan M Turing. 2009. Computing machinery and intelligence. In *Parsing the Turing test*, pages 23-65. Springer.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Douglas Walton. 2014. Abductive reasoning. University of Alabama Press.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019c. Does it make sense? and why? A pilot study for sense making and explanation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4020-4026. Association for Computational Linguistics.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le
Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 93-104. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791-4800. Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1441-1451. Association for Computational Linguistics.
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pretrained language models. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9733-9740. AAAI Press.
Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. 2021. Controllable generation from pre-trained language models via inverse prompting. arXiv preprint arXiv:2103.10685.
Supplementary Material for ElitePLM
We give some experiment-related information as supplementary materials. The appendix is organized into six sections:
- Configurations and pretraining setting comparisons for selected models are presented in Appendix A;
- Data statistics of each test are presented in Appendix B;
Full results for memory tests are presented in Appendix C;
- Full results for comprehension tests are presented in Appendix D;
Full results for reasoning tests are presented in Appendix E; and
Full results for composition tests are presented in Appendix F.
A Configurations of Pretrained Language Models
The selected ten PLMs within five categories and the comparisons of these PLMs in configuration and pretraining setting have been shown in Table 7. The effect extent of each factor for PLMs abilities in Table 8.
Memory Tests. The data statistics of LAMA and Wikipedia of each model are presented in Table 9. Due to the differences of each PLM, we drop the data that are not in the vocabulary.
Comprehension Tests. The data statistics of GLUE, SuperGLUE, SQuAD and RACE are presented in Table 10.
Reasoning Tests. The data statistics for commonsense reasoning, deductive reasoning and abductive reasoning are presented in Table 11.
Composition Tests. The data statistics for text summarization, question generation and story generation are presented in Table 12. For the first three datasets, we truncate the source text considering the input length of PLMs during training. And for WritingPrompts, we reconstruct the original
dataset and discard examples where text contains more than 512 tokens.
C Memory Tests
Full results on LAMA and Wikipedia datasets are presented in Table 13.
D Comprehension Tests
Full results on SuperGLUE, SQuAD and RACE are presented in Table 14 and Table 15.
E Reasoning Tests
Full results on CommonsenseQA, ROCStories, SWAG, HellaSwag, Sense Making, and ARCT are presented in Table 16.
F Composition Tests
For automatic metrics, BLEU-  $n$  and ROUGE-  $n$  compute the ratios of overlapping  $n$ -grams between generated and real text, while METEOR measures word-to-word matches based on WordNet between generated and real text. For the human test, Fluency evaluates whether the text is well-formed and logical to read; Informativeness measures whether the text contains useful information; Accuracy tests whether the text describes the given content accurately; Relevance measures whether the text is relevant to the given context; Overall evaluates the overall quality of the text.
In the human test, we randomly select 500 generated texts for each PLM and 500 gold text. Therefore, there are 3000 texts totally. The judges are all PhD students which do not know about where each text comes from. Each text will be scored by two judges from the above five aspects, and the final score is the average of the two scores. In the Turing test, each text will also be distinguished by two judges. Only when two judges make the same decisions that the text is generated by human, we will consider the text is true.
Full results on CNN/Daily-Mail, GigaWord, SQuAD, and WritingPrompts are presented in Table 17. Turing test results are presented in Table 6. We also show some summaries and stories generated by different PLMs in Table 19, Table 20, and Table 21.
Type	Models	Configurations	Pretraining Setting
Size	#Parameter	Corpus	Size
Bidirectional	BERT	base/large	110M/340M	BooksCorpus, English Wikipedia	16GB
RoBERTa	base/large	125M/355M	BooksCorpus, CC-News, WebText, Stories	160GB
ALBERT	xlarge/xxlarge	60M/235M	BERT Corpus	16GB
Unidirectional	GPT-2	small/medium	117M/345M	WebText (removing Wikipedia)	40GB
Hybrid	XLNet	base/large	110M/340M	BooksCorpus, English Wikipedia, Giga5, ClueWeb, Common CWEI	158GB
UniLM	base/large	110M/340M	BERT Corpus	16GB
Knowledge-Enhanced	ERNIE	base	114M	English Wikipedia, Wikipedia	17GB
Text-to-Text	T5	base/large	220M/770M	Colossal Clean Crawled Corpus	745GB
BART	base/large	140M/400M	RoBERTa Corpus	160GB
ProphetNet	large	373M	RoBERTa Corpus	160GB
Table 7: Configurations and pretraining setting comparisons for our selected models.
Ability	MA	DD	MS	PO	PS
Memory	★★	★	★	★★★	★★★
Comprehension	★★	★★	★	★★	★★★
Reasoning	★	★★★	★	★★★	★★★
Composition	★	★★★	★★★	★★★	★
Table 8: The impact extent of each factor for four language abilities of PLMs. MA, DD, MS, PO, and PS are short for model architecture, data distribution, model size, pretraining objective, and pretraining strategy, respectively
G-RE	T-REx	ConceptNet	SQuAD	Wikipedia
#Origin	6,106	34,014	14,878	305	100,000
#Relation	3	41	16	-	-
BERT / UniLM	5,527	34,014	11,658	305	85,836
RoBERTa	4,618	29,500	12,505	286	85,862
ALBERT	5,469	33,636	12,389	291	86,533
ERNIE	1,900	9,071	11,649	173	-
BART	4,618	29,500	12,505	286	85,862
T5	4,256	25,850	10,905	230	78,069
GPT-2	4,618	29,500	7,477	196	1,184
XLNet	5,202	32,293	12,080	279	85,228
ProphetNet	5,527	34,014	12,506	305	87,516
The Predicted Outputs	The predicted token of “[MASK]” in each template.
Table 9: Statistics of datasets in memory tests, including LAMA and Wikipedia. #Origin and #Relation denote the number of examples and relations in original dataset, and the number of each model denotes the number of examples after selected. The predicted outputs is the intermediate result resources we provide.
Corpus	#Train	#Valid	#Test	The Predicted Outputs
GLUE	CoLA	8,551	1,043	1,063	The predicted binary class whether a sentence is grammatical.
SST-2	67,349	872	1,821	The predicted sentiment (positive/negative) of a sentence.
MRPC	3,668	408	1,725	The predicted binary class whether two sentences are semantically equivalent.
QQP	363,846	40,430	390,965
STS-B	5,749	1,500	1,379	The predicted similarity score (1-5) of two sentences.
MNLI	392,702	9,815	9,796	The predicted relation (entailment/contradiction/neutral)
-M.
MNLI
-MM.	9,832	9,847	between two sentences.
QNLI	104,743	5,463	5,463
RTE	2,490	277	3,000	The predicted relation (entailment or not) between two sentences.
WNLI	635	71	146
Super GLUE	BoolQ	9,427	3,270	3,245	The predicted answer (yes/no) to the passage-based question.
CB	250	57	250	The predicted relation (entailment/contradiction/neutral) between two sentences.
COPA	400	100	500	The predicted cause or effect of the premise from two choices.
MultiRC	5,100	953	1,800	The predicted answer choice to the passage-based question.
Wic	6,000	638	1,400	The predicted binary class whether a word is used with the same sense in two sentences.
WNLI	635	71	146	The predicted relation (entailment or not) between two sentences.
WSC	554	104	146	The predicted noun phrase referent of the pronoun from among the provided choices.
SQuAD	v1.1	88,567	10,790	-	The predicted answer span to the passage-based question.
v2.0	131,924	12,165	-
RACE	all	25,137	1,389	1,407
87,866	4,887	4,934
middle	6,409	368	362	The predicted answer choice to the passage-based question.
25,421	1,436	1,436
high	18,728	1,021	1,045
62,445	3,451	3,498
Table 10: Statistics of datasets in comprehension tests including GLUE, SuperGLUE, SQuAD and RACE. #Train, #Valid and #Test denote the number of instances in train, valid and test set, respectively (the same as below). MNLI-M. and MNLI-MM. denote MNLI-match and MNLI-mismatch, respectively. SQuAD doesn't have test set, and we utilize the valid set as the test set. The predicted outputs is the intermediate result resources we provide.
Reasoning Task	Corpus	#Train	#Valid	#Test	The Predicted Outputs
Com.sense	CQA	9,741	1,221	1,140	The predicted answer choice to a commonsense question.
Deductive	ROCS.	1,257	314	1,571	The predicted ending choice based on the context.
SWAG	73,546	20,006	20,005	The predicted answer choice based the grounded situation.
HellaS.	39,905	10,042	10,003
SM-A	10,000	1,000	1,000	The predicted valid sentence between two sentences.
Abductive	SM-B	10,000	1,000	1,000	The predicted reason choice why the sentence is invalid.
ARCT	1,210	316	444	The predicted warrant choice that justifies reason and claim.
Table 11: Statistics of datasets in reasoning tests, including commonsense reasoning, deductive reasoning and abductive reasoning. CQA is short for CommonsenseQA. SM-A and SM-B denote the Task A and Task B of Sense Making, respectively. The Predicted outputs is the intermediate result resources we provide.
Task	Corpus	#Train	#Valid	#Test	#Input	#Output	The Predicted Outputs
TS	CNN/DailyMail	287,113	13,368	11,490	822.3	57.9	The generated summary given a news.
Gigaword	3,803,957	189,651	1,951	33.7	8.7	The generated headline given a paragraph and corresponding Turing test and aspect scores.
OG	SQuAD	75,722	10,570	11,877	149.4	11.5	The generated question given a passage and corresponding answer.
SG	Writing Prompts	67,765	3,952	3,784	30.2	281.2	The generated story given a prompt and corresponding Turing test and aspect scores.
Table 12: Statistics of datasets in composition tests, including text summarization (TG), question generation (QG) and story generation (SG). #Input and #Output denote the average number of tokens in the input text and output text. The Predicted outputs is the intermediate results and human evaluation resources we provide.
Models	Vocab Size	LAMA-G	LAMA-T	LAMA-C	LAMA-S	Wikipedia	Average
BERTBASE	28996	10.3	27.5	15.3	12.8	66.8	41.6
BERTLARGE	28996	11.0	29.2	19.1	17.0	70.9	45.0
RoBERTaBASE	50265	7.5	19.9	17.9	13.3	66.9	40.8
RoBERTaLARGE	50265	7.1	23.9	21.6	21.0	71.1	44.8
ALBERTXLARGE	30000	2.9	19.6	16.8	14.4	64.3	38.9
ALBERTXXLARGE	30000	3.3	21.0	20.0	20.6	63.9	40.1
GPT-2SMALL	50257	1.3	6.8	4.0	3.0	36.0	19.9
GPT-2MEDIUM	50257	3.9	12.0	6.4	5.6	42.7	24.8
XLNetBASE	32000	0.0	0.0	2.8	0.0	64.6	32.7
XLNetLARGE	32000	0.0	0.0	5.5	0.4	68.7	35.1
UniLMBASE	28996	8.5	27.6	15.4	11.8	66.9	41.4
UniLMLARGE	28996	9.6	28.4	18.3	17.4	71.5	46.4
ERNIEBASE	28996	1.3	13.4	13.0	8.1	-	-
T5BASE	32100	5.5	20.0	13.2	9.6	60.5	36.3
T5LARGE	32100	4.0	21.7	17.1	11.7	65.0	39.3
BARTBASE	50295	5.7	11.7	9.5	4.2	47.9	27.8
BARTLARGE	50295	9.4	15.8	7.7	3.1	47.8	28.4
ProphetNetLARGE	30522	0.1	1.1	0.3	0.7	31.3	15.9
Table 13: Memory tests results on LAMA and Wikipedia datasets (test set). We report accuracy score for each dataset. Average is computed by averaging the scores of LAMA and Wikipedia (the score of LAMA is averaged among four dataset first). LAMA-G, LAMA-T, LAMA-C and LAMA-S denote the LAMA corpus Google-RE, T-REx, ConceptNet and SQuAD, respectively.
Model	WSC Acc.	CB F1/Acc.	RTE Acc.	COPA Acc.	Wic Acc.	BoolQ Acc.	MultiRC F1/EM	Avg
BERTBASE	60.6	78.7/80.4	66.4	65.0	69.9	74.6	68.1/16.9	65.5
BERTLARGE	63.5	89.0/92.9	70.1	73.0	72.7	75.6	69.4/22.6	70.3
RoBERTaBASE	71.1	89.1/91.1	75.1	78.0	67.2	81.1	72.6/31.9	73.6
RoBERTaLARGE	75.0	95.0/96.4	88.2	84.0	72.7	85.4	81.7/47.2	80.8
ALBERTXLARGE	63.5	81.1/85.7	62.5	75.0	66.5	62.2	63.6/12.4	64.4
ALBERTXXLARGE	64.4	87.6/92.9	70.4	91.0	74.3	62.2	85.1/54.0	74.6
GPT-2SMALL	54.8	64.0/76.8	62.1	62.0	64.1	68.2	67.3/19.5	60.7
GPT-2MEDIUM	61.5	84.4/82.1	63.6	63.0	67.2	73.9	71.5/29.2	66.1
XLNetBASE	64.4	91.0/91.1	59.9	65.0	67.9	76.9	72.5/29.6	68.0
XLNetLARGE	65.3	87.6/92.9	88.5	82.0	69.7	84.7	79.0/41.6	77.3
UniLMBASE	63.5	74.7/82.1	60.3	67.0	68.5	73.3	67.9/20.5	65.0
UniLMLarge	65.4	86.5/87.5	70.9	76.0	72.3	82.3	75.7/36.3	72.8
ERNIEBASE	65.4	81.6/82.1	68.8	64.0	70.8	74.4	68.7/21.3	67.2
T5BASE	79.8	86.2/94.0	80.1	71.2	68.3	81.4	79.7/43.1	76.0
T5LARGE	84.6	91.6/94.8	87.2	83.4	69.3	85.4	83.3/50.7	81.4
BARTBASE	64.4	86.6/85.7	69.5	70.0	65.7	75.7	74.2/31.7	69.2
BARTLARGE	65.4	97.4/96.4	83.5	86.0	70.4	85.1	82.9/50.6	79.2
ProphetNetLARGE	63.5	94.7/92.9	51.3	61.0	60.7	67.4	64.7/17.2	62.7
Table 14: Comprehension tests results on SuperGLUE (valid set). Avg column is computed by averaging the scores of tasks to its left (the scores for CB and MultiRC are first averaged).
Models	SQuAD v1.1	SQuAD v2.0	RACE
EM	F1	EM	F1	RACE	RACE-M	RACE-H
BERTBASE	80.8	88.5	72.8	76.0	65.0	71.7	62.3
BERTLARGE	84.1	90.9	78.7	81.9	72.0	76.6	70.1
RoBERTaBASE	86.1	92.3	80.3	83.4	72.8	72.6	26.6
RoBERTaLARGE	88.9	94.6	86.5	89.4	83.2	86.5	81.3
ALBERTXLARGE	86.1	92.5	83.1	86.1	78.1	76.7	79.8
ALBERTXXLARGE	88.3	94.1	85.1	88.1	87.4	85.9	87.1
GPT-2small	63.6	75.1	57.1	61.5	61.2	62.9	58.2
GPT-2MEDIUM	70.3	80.8	61.5	66.0	62.2	65.0	61.4
XLNetBASE	12.8	14.7	78.5	81.3	71.3	72.8	67.5
XLNetLARGE	89.7	95.1	87.9	90.6	85.4	88.6	84.0
UniLMBASE	82.8	89.9	74.9	78.0	59.0	64.1	50.3
UniLMLARGE	86.5	92.7	80.5	83.4	70.3	70.0	66.4
ERNIEBASE	-	-	-	-	-	67.8	-
T5BASE	85.4	92.1	77.6	81.3	70.6	74.4	68.4
T5LARGE	86.7	93.8	-	-	80.4	82.6	77.8
BARTBASE	84.6	91.0	76.0	79.2	70.1	72.4	63.2
BARTLARGE	88.8	94.6	86.1	89.2	82.2	82.5	79.6
ProphetNetLARGE	-	-	-	-	-	74.1	-
Table 15: Comprehension tests results on SQuAD and RACE (test set).
Model	CQA	ROCStories	SWAG	HellaSwag	SM-A	SM-B	ARCT
BERTBASE	53.0	88.1	81.6	40.5	87.3	80.1	65.1
BERTLARGE	55.9	90.2	86.3	47.3	89.4	85.8	71.2
RoBERTaBASE	72.1	93.3	82.6	61.0	89.3	87.5	46.1
RoBERTaLARGE	72.2	97.4	89.9	85.2	93.0	92.3	57.9
ALBERTXLARGE	66.2	90.4	84.6	75.9	87.9	89.4	56.1
ALBERTXXLARGE	80.0	97.1	90.7	90.1	92.5	92.3	79.5
GPT-2small	47.8	58.8	48.1	39.9	84.2	74.7	66.0
GPT-2MEDIUM	60.8	59.9	79.7	60.4	88.7	73.4	66.7
XLNetBASE	53.8	92.0	80.4	55.1	81.6	85.4	80.2
XLNetLARGE	62.9	93.8	86.8	79.7	83.7	88.7	83.1
UniLMBASE	47.6	80.6	77.0	36.3	86.2	83.6	48.4
UniLMLarge	62.3	86.9	83.1	46.7	89.3	86.4	72.3
ERNIEBASE	54.1	84.7	-	-	88.7	-	73.7
T5BASE	61.9	88.2	65.8	55.2	89.2	82.9	63.3
T5LARGE	69.8	91.4	73.7	79.1	92.7	88.2	69.4
BARTBASE	61.0	88.9	81.2	53.4	72.0	67.9	71.8
BARTLARGE	75.8	91.7	87.9	76.6	82.9	67.9	84.2
ProphetNetLARGE	21.3	82.2	70.1	26.4	85.5	78.0	65.5
Table 16: Reasoning tests results on seven datasets (test set). We report accuracy score for each dataset. CQA is short for CommonsenseQA. SM-A and SM-B denote the Task A and Task B of Sense Making, respectively.
Models	CNN-DailyMail	GigaWord	SQuAD	WritingPrompts
R-1	R-2	R-L	R-1	R-2	R-L	B-4	R-L	ME	B-4	R-L	ME
GPT-2SMALL	24.60	7.21	21.06	25.25	9.03	23.20	5.13	14.83	21.06	11.58	3.80	8.18
GPT-2MEDIUM	22.95	5.99	22.08	23.72	8.12	21.56	8.48	18.82	26.77	14.47	3.23	7.29
UniLMBASE	17.83	0.11	5.50	16.64	6.11	15.12	4.47	17.65	20.30	27.71	2.35	5.47
UniLM LARGE	43.44	20.21	40.51	38.45	19.45	35.75	4.42	17.43	20.13	26.88	1.84	5.01
T5BASE	42.05	20.34	39.40	33.13	15.60	30.18	11.18	21.82	29.93	6.04	4.61	9.81
T5LARGE	42.50	20.68	39.75	34.75	16.26	31.49	11.19	22.35	30.53	8.61	4.19	9.51
BARTBASE	36.36	20.87	33.32	38.65	19.43	35.82	14.44	24.11	36.92	11.91	3.57	7.69
BARTLARGE	44.16	21.28	40.90	39.41	20.21	36.42	15.87	25.47	38.42	14.72	3.14	7.08
ProphetNetLARGE	44.20	21.17	41.30	39.51	20.42	36.69	14.20	23.97	35.99	19.31	2.59	7.19
Table 17: Composition tests results on four datasets. R-1, R-2, R-L are short for ROUGE-1, ROUGE-2, ROUGE-L respectively. B-4 and MT denote BLEU-4 and METEOR, respectively.
Models	TT (%)	Fluency	Informativeness	Accuracy	Coherence	Overall
GPT-2_MEDIUM	45.7	3.42	3.17	3.20	3.23	5.87
UniLM LARGE	1.2	1.32	1.88	2.03	1.71	2.74
T5 LARGE	34.4	3.01	2.80	3.09	2.87	5.18
BART LARGE	45.2	3.37	3.16	3.39	3.22	5.96
ProphetNet LARGE	29.6	2.95	2.91	3.10	2.89	5.18
Gold	71.3	3.79	4.07	3.87	3.80	7.37
Table 18: Turing test and human scores on the test set of WritingPrompts. TT is short for Turing Test and represents the percentage of makers who believe the text is written by human. Coherence evaluates how content is coherent considering both intra- and inter-sentence correlation of a paragraph. Rated from 1 to 5.
Passage	japan's nec corp. and UNK computer corp. of the united states said wednesday they had agreed to join forces in supercomputer sales .
GPT-2_MEDIUM	unk computer to jointly sell supercomputers
UniLM LARGE	nec [UNK] to join forces in supercomputer sales
T5 LARGE	nc and unk computer to join forces in supercomputer sales
BART LARGE	nec and unk computer to join forces in supercomputer sales
ProphetNet LARGE	nec unk computer to join forces in supercomputer sales
Gold	nec UNK in computer sales tie-up
Passage	the sri lankan government on wednesday announced the closure of government schools with immediate effect as a military campaign against tamil separatists escalated in the north of the country .
GPT-2_MEDIUM	closed as military campaign escalates
UniLM LARGE	sri lankan gov t shuts down schools in north
T5 LARGE	sri lanka closes schools as fighting escalates
BART LARGE	sri lanka closes government schools
ProphetNet LARGE	sri lanka closes schools as military escalates
Gold	sri lanka closes schools as war escalates
Table 19: Test set examples of GigaWord dataset for each model.
Table 20: Test set examples of WritingPrompts dataset for each model.
Table 21: Test set examples of WritingPrompts dataset for each model.