SlowGuess/NAACL_2021_OCR huggingface dataset:
(1) CFCS as classification: In this form, given a document and a corresponding summary sentence, the model needs to identify if the sentence is factually correct with respect to the document (entailed) or not. In order to perform the classification, we first obtain our entailment scorer by fine-tuning the multiple choice classification model (Figure 3) on the  $\mathrm{RACE}_{\mathrm{Converted}}$  dataset and use the dev set to calibrate a threshold\*\* (described in Section 5.1) to obtain the two-class entailment model (Figure 2).
<sup>1</sup>We use the dev and test dataset provided by Krysciński et al. (2019) for this task.
**Balanced accuracy is used to find the best threshold.
(2) CFCS as ranking: Given a source document and a set of five machine generated summaries, the model is required to rank at least one factually correct summary above all incorrect summary alternatives. Note that a variable number of these five machine generated summaries can be factually correct (Falke et al., 2019). However, there is always at least one incorrect summary in this set.
Model	Balanced Accuracy	F1-score
BERT+FactCCautogen* #	74.15	0.51
RoBERTa	54.76	0.30
RoBERTa+MNLI	51.92	0.15
RoBERTa+MNLIsegmented	69.87	0.70
RoBERTa+ANLI	62.61	0.41
RoBERTa+ANLIsegmented	57.34	0.58
RoBERTa+RACEconverted	86.55	0.73
* These results are reported from Krysciński et al. (2019).
$\# \mathrm{FactCC}_{\mathrm{autogen}}$  is the automatically generated training data used by Krysciński et al. (2019).
Table 5: Balanced accuracy and macro F1 score on the test set for the task of CFCS posed as a classification problem.
Model	% Correct
ESIM + SNLI *	60.70%
RoBERTa	50.47%
RoBERTa+MNLI	49.53%
RoBERTa+MNLIsegmented	66.36%
RoBERTa+ANLI	54.20%
RoBERTa+ANLIsegmented	66.35%
RoBERTa+RACEconverted	75.70%
* Reported from Falke et al. (2019).
Table 6: Performance of various models on the CFCS on the sentence-ranking and summary-ranking tasks. The numbers denote the fraction of highest ranked summaries which are labelled factually correct.
Table 5 and Table 6 present the results for CFCS as classification and CFCS as ranking, respectively. Similar to the MCRC task, the model trained on the long-premise  $\mathrm{RACE}_{\mathrm{converted}}$  dataset outperforms the models trained on the short-premise NLI datasets in both regular and segmented forms of evaluation on each of the CFCS task types. Moreover, it also outperforms the FactCC model which uses the automatically generated long-premise training data (Krysciński et al., 2019).
The results of evaluations on the MCRC and CFCS tasks – which inherently contain long contexts – provide strong evidence supporting our long premise conjecture.
6.3 Confounding Factors
Natural language experiments are often vulnerable to artifacts that may leak exploitable signals into the training data that the model can fit on. Such extraneous factors, if present, can prevent the empirical isolation of the premise-length as a major factor. We therefore discuss and eliminate the two most obvious potential confounding factors.
6.3.1 Vocabulary Overlap
In the zero-shot evaluation setup, a high vocabulary overlap between the training data and the target data can potentially help a model perform better. To eliminate this confounding factor from our experiments, we calculate the vocabulary overlap of RACE, MNLI and ANLI (training data) with the 3 MCRC datasets (evaluation data). We define overlap as:
words in [Vocab(train data)  $\cap$  Vocab.eval.data)]
words in Vocab.eval.data)
Table 8 shows that all the datasets have similar vocabulary overlap with the three MCRC datasets. However, from Table 4, we see that the model trained on  $\mathrm{RACE}_{\mathrm{converted}}$  considerably outperforms the models trained on the short-premise NLI datasets. This indicates that vocabulary overlap is not playing a big role in the model's performance.
To substantiate this claim, we further evaluate the two models on those subsets of the three MCRC datasets that consist only of examples where the vocabulary overlap is high ( $\geq 0.9$ ). Table 7 shows that the performance of the two models on these high vocabulary overlap subsets is similar to their overall performances on the respective datasets. We can thus conclude that vocabulary overlap is not helping either of the models in terms of predictive performance.
6.3.2 Automated Conversion
We evaluate the models trained on the short-premise NLI datasets and  $\mathrm{RACE}_{\mathrm{converted}}$  on the converted forms of the MCRC datasets. However, only the model trained on the  $\mathrm{RACE}_{\mathrm{converted}}$  dataset is exposed to the same conversion strategy during training. It is therefore possible that the conversion
MultiRC	DREAM	CosmosQA
Overall	Subset	Overall	Subset	Overall	Subset
RoBERTa+RACEconverted	77.4	77.6	83.5	85.5	73.5	74.0
RoBERTa+MNLI	60.5	61.1	67.7	68.6	38.1	37.6
RoBERTa+ANLI	67.9	68.7	74.1	73.7	49.7	49.9
Table 7: Performance of the models on high vocabulary overlap subsets of the MCRC datasets.
MultiRC	DREAM	CosmosQA
RACE	0.905	0.974	0.852
MNLI	0.928	0.950	0.839
ANLI	0.840	0.913	0.729
mechanism itself becomes a confounding factor, enabling the  $\mathrm{RACE}_{\mathrm{converted}}$  model to perform better on the MCRC task. To assess this nuance, we manually annotate a subset of the MCRC datasets using Label Studio (Tkachenko et al., 2020), with a random set of examples annotated by each of the authors. To create a setting where the difference is vivid, we design the annotation subsets such that the  $\mathrm{RACE}_{\mathrm{converted}}$  model gives an accuracy of around  $50\%$  using the hybrid conversion strategy. The independent manual annotations prevent any exploitable signal from leaking into the training data of the model through the conversion mechanism. We compare the performance of models trained on converted forms of the RACE dataset using both our hybrid strategy as well as manual annotation. $^{\dagger\dagger}$  We manually annotate 100 examples from MultiRC and 50 each from ComsosQA and DREAM. MultiRC is evaluated at an option-level with each question-answer pair considered an individual example. On the other hand, CosmosQA and DREAM are evaluated at a question-level, with each example consisting of three question-answer pairs, and one label corresponding to the correct answer option.
Table 9 shows that the  $\mathrm{RACE}_{\mathrm{converted}}$  model performs better on the manually annotated subset; this eliminates the possibility of the conversion mechanism being a confounding factor in our results.
Table 8: Vocabulary Overlap with MCRC datasets. The value in cell (i,j) is given by  $\frac{\text{SizeOf}(\text{Vocab}_{i\cap}\text{Vocab}_{j})}{\text{SizeOf}(\text{Vocab}_{j})}$
MultiRC	DREAM	CosmosQA
Automatic	52.08	50.00	50.00
Manual	55.21	54.00	72.00
Table 9: Evaluation of the  $\mathrm{RACE}_{\mathrm{converted}}$  model on the manually annotated subset of the MCRC datasets as compared to the same subsets with Hybrid conversion.
The difficulty of transferring entailment (NLI) knowledge to downstream NLP tasks can be largely attributed to the difference in data distributions, specifically the premise lengths. Models trained on short-premise NLI datasets are not very good at performing inference over longer texts, which is a central feature of important downstream tasks such as QA and text summarization.
We leverage the abundance of large and diverse MCRC datasets and the ease of conversion from MCRC into the NLI format to automatically and scalably create a long-premise NLI dataset to test this long-premise conjecture. We show that the long-premise nature of the converted dataset indeed helps achieve better performance on the downstream tasks of MCRC and CFCS when compared against models trained on traditional short-premise NLI datasets. We further discuss and eliminate possible confounding factors in our experiments to ensure the validity of our results.
Our work highlights a major shortcoming in popular NLI datasets that limits their usefulness to downstream NLP applications; and emphasizes the need for long-premise NLI datasets. Future work in this direction can take us closer to realizing the full potential of NLI as a fundamental task in natural language understanding.
$\dagger \dagger$  It is important to note that this setting is solely for the purpose of establishing the role of the hybrid conversion strategy as a potential confounding factor in the performance of the  $\mathrm{RACE}_{\mathrm{converted}}$  model. The absolute accuracy numbers are not reflective of the model performance on the overall dataset.
In this work, we use open source datasets, libraries, and services which are freely available and appropriately cited. We do not release the converted form of the MCRC dataset in respect of existing copyright; however, we provide all the information required to reproduce our experimental setup, datasets, and results in the content of the main paper as well as in the appendix.
All rules used in the conversion strategies (Section 4), as well as the manual annotations performed as part of the confounding factors analysis, were produced solely by the group of authors. Our work did not involve any external human subjects; and did not require institutional review.
Looking forward, it is certainly possible that the neural conversion strategy proposed by us in Section 4 may be applied by readers of this work in other – potentially scaled-up – contexts. Since the conversion is used as a means to an end (producing an appropriate long-premise dataset) rather than as the central contribution of the current work, we do not provide an extended analysis of the pros and cons of this strategy.
We thank Rajarshi Das, and Andrew McCallum for their constructive feedbacks in the 696DS class during Spring 2020. We also thank the anonymous reviewers for their insightful suggestions.
Xiang Lorraine Li is supported in part by IBM Research AI through the AI Horizons Network and in part under the University of Southern California subcontract no. 123875727 under Office of Naval Research prime contract no. N660011924032. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. Conference Proceedings - EMNLP 2015: Conference on Empirical Methods in Natural Language Processing, pages 632-642.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework, fracas: a framework for computational semantics. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv, abs/1809.02922.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220.
Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1481-1491, Seattle, Washington, USA. Association for Computational Linguistics.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277.
Pavan Kapanipathi, Veronika Thost, Siva Sankalp Patel, Spencer Whitehead, Ibrahim Abdelaziz, Avinash Balakrishnan, Maria Chang, Kshitij Fadnis, Chulaka Gunasekara, Bassem Makni, Nicholas Mattei, Kartik Talamadupula, and Achille Fokoue. 2020. Infusing knowledge into the textual entailment task using
graph convolutional networks. Proceedings of the AAAI Conference on Artificial Intelligence.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Conference on Artificial Intelligence.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840.
Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017a. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 100-109, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017b. Race: Large-scale reading comprehension dataset from examinations. In EMNLP.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. arXiv preprint arXiv:1804.08207.
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text
to Universal Dependencies, pages 160-170, Brussels, Belgium. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In EMNLP.
Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2019. Probing natural language inference models through semantic fragments. arXiv preprint arXiv:1909.07521.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231.
Maxim Tkachenko, Mikhail Malyuk, Nikita Shevchenko, Andrey Holmanyuk, and Nikolai Liubimov. 2020. Label Studio: Data labeling software. Open source software available from https://github.com/heartlabs/studio.
Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. 2019. Repurposing entailment for multi-hop question answering tasks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2948-2958.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228.
Shuohang Wang and Jing Jiang. 2015. Learning natural language inference with lstm. arXiv preprint arXiv:1512.08849.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In ACL.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Yuhui Zhang, Yuhao Zhang, and Christopher D. Manning. 2020. A close examination of factual correctness evaluation in abstractive summarization.
A Quality of Converted Datasets
We evaluate the quality of the converted datasets by using benchmarks that probe the trained models for semantic phenomenon (Poliak et al., 2018; Richardson et al., 2019). Tables 10 and 11 show the performance of the models on the different semantic phenomena. We show that the converted NLI datasets are at par, and sometimes better than a specially curated NLI dataset such as MNLI. For the purpose of illustration, we report results on the  $\mathrm{RACE}_{\mathrm{converted}}$  dataset and MNLI.
B Comparison of Converted MCRC Datasets
We use the hybrid conversion strategy discussed in Section 4 to generate long-premise NLI datasets from each of the MCRC datasets - RACE, MultiRC, DREAM and CosmosQA. As seen from Table 1 and 3, RACE has the longest average premise length as well as the most number of training examples. It is thus, intuitive to see from Tables 12, 13 and 14 that the  $\mathrm{RACE}_{\mathrm{converted}}$  model outperforms the other converted models in each of the tasks.
C Reproducibility Checklist
C.1 Details of the datasets used
Table 15 gives the train/dev/test splits of the various source datasets used in this work. We follow the same splits after the conversion to NLI form. Since the test datasets are not openly available for MultiRC and CosmosQA, we use the corresponding dev sets to report our results.
Table 16 shows the proportion (absolute numbers) of neural, rule-based and  $\mathrm{Q + A}$  examples in the final hybrid datasets.
C.2 Neural Conversion
We use the following training sequence to obtain the final neural conversion model:
1. Obtain the pre-trained BART model (Lewis et al., 2019) fine-tuned on CNN/Dailymail from HuggingFace library.*
2. Fine-tune the model using the hyperparameters mentioned in Table 17 on google-sentence completion dataset (Filippova and Altun, 2013) $^{\dagger}$
3. Further fine-tune the model on the QA2D dataset (Demszky et al., 2018).
C.2.1 Experiments
- The hyperparams for the models used throughout the Section 6 are shown in Table 18. These were obtained using minimal manual tuning.
- The threshold for CFCS as classification experiments (Section 6.2 (1)) we calculated by tuning for best balanced accuracy https://scikit-learn.org/stable/modules/generated/sklearn.metrics-balanced.accuracy_score.html.
D Conversion examples
Tables 19, 20 and 21 show examples of rule-based and neural conversions on RACE, MultiRC and DREAM respectively.
\*https://huggingface.co/facebook/bart-large-cnn
$\dagger$ https://github.com/
google-research-datasets/
sentence-compression
$\ddagger$ https://worksheets.
codalab.org/worksheets/
0xd4ebc52cebb84130a07cbfe81597AAF0/
Semantic Fragment	RACEconverted	MNLI
Boolean	84	93.0
Comparative	66.8	74.5
Conditional	83.4	61.1
Counting	45.3	54.0
Monotonicity (Hard)	50.9	62.8
Monotonicity (Simple)	67.6	62.0
Negation	86.8	82.5
Quantifier	71	77.6
Overall	69.48	70.94
Table 10: Results on the Semantic Fragments dataset from (Richardson et al., 2019).
Sem. Phenomenon	RACEconverted	MNLI
Factuality	61.47	67.39
NER	69.28	58.36
Pun	49.71	58.54
Sentiment	94.17	68.5
Lexicosyntactic (MV)	35.62	43.49
Lexicosyntactic (VN)	48.13	55
Lexicosyntactic (VC)	50.25	46.59
Overall	63.22	56.08
* These results are reported from Krysciński et al. (2019).
Table 11: Results on the Diverse Natural Language Inference dataset from (Poliak et al., 2018).  $\mathbf{RACE}_{\mathrm{Neural}}$  refers to the converted RACE dataset created using the BART model. MV refers to the MegaVeridicality dataset; VN to the VerbNet dataset; VC to the Verb-Corner dataset.
Model\Dataset*	RACE(271)	MultiRC(252)	DREAM(110)	CosmosQA(75)
Random Guess	25.00	50.00	33.33	33.33
MultiNLI	44.34	60.58	67.76	38.11
MultiNLI Segmented	41.01	61.71	42.28	43.28
\( RACE_{converted} \)	83.99	77.43	83.58	73.58
\( MultiRC_{converted} \)	58.02	81.22	67.12	43.65
\( DREAM_{converted} \)	65.01	71.08	83.99	61.00
\( CosmosQA_{converted} \)	49.27	48.80	72.46	83.89
* Datasets are in NLI form created using hybrid conversion method (Section ??) for the models trained on the converted datasets.
Table 12: Zero-shot evaluation accuracies achieved by models trained on converted NLI datasets and MultiNLI on other MCRC datasets (in NLI form) using the transferable model architecture described in Section 5.1. The numbers in the parenthesis of the column headers denote the average premise lengths of the datasets.
Model	Balanced Accuracy	F1-score
BERT+FactCCautogen* #	74.15	0.51
RoBERTa	54.76	0.30
RoBERTa+MultiNLI	51.92	0.15
RoBERTa+MultiNLIsegmented	69.87	0.70
RoBERTa+CosmosQAconverted	55.96	0.52
RoBERTa+DREAMconverted	75.69	0.69
RoBERTa+MultiRCconverted	82.03	0.72
RoBERTa+RACEconverted	86.55	0.73
FactCCautogen is the automatically generated training data used by Krysciński et al. (2019).
Table 13: Balanced accuracy and macro F1 score on the test set for the task of CFCS posed as a classification problem.
Model	% Correct
Sentence-pair Ranking	Summary Ranking
ESIM + SNLI *	67.60%	60.70%
BERT+FactCCautogen † #	70.00%	-
QAGS‡	72.10%	-
RoBERTa	56.03%	50.47%
RoBERTa+MultiNLI	81.76%	49.53%
RoBERTa+MultiNLIsegmented	81.23%	66.36%
RoBERTa+CosmosQAconverted	76.41%	49.53%
RoBERTa+DREAMconverted	78.28%	68.22%
RoBERTa+MultiRCconverted	72.21%	67.23%
RoBERTa+RACEconverted	86.59%	75.70%
*†‡ Reported from Falke et al. (2019), Krysciński et al. (2019) and Wang et al. (2020), respectively.
FactCCautogen is the automatically generated training data for their model.
Table 14: Performance of various models on the CFCS on the sentence-ranking and summary-ranking tasks. The numbers denote the fraction of highest ranked summaries which are labelled factually correct.
Dataset	Number of examples
Train	Dev	Test
RACE	87866	4887	4934
MultiRC	27243	4848	-
DREAM	6116	2040	2041
CosmosQA	6116	2040	-
FactCC	-	931	503
Sentence Ranking	-	746	-
Summary Ranking	-	2555	530
Table 15: Number of examples in each of the datasets.
Dataset	Split	Neural	Rule-based	Q+A
RACE	Train	314448	16808	20208
Dev	17447	912	1189
Test	18284	580	872
MultiRC	Train	23613	3630	0
Dev	4156	692	0
DREAM	Train	16708	1530	110
Dev	5531	531	58
Test	5588	495	40
CosmosQA	Train	7298	848	32
Dev	60009	10889	400