SlowGuess/NAACL_2021_OCR huggingface dataset:
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li  
College of Information and Computer Sciences,  
University of Massachusetts Amherst
Pavan Kapanipathi, Kartik Talamadupula IBM Research
Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the multiple-choice reading comprehension (MCRC) and checking factual correctness of textual summarization (CFCS) tasks to investigate potential reasons for this. Our findings show that: (1) the relatively shorter length of premises in traditional NLI datasets is the primary challenge prohibiting usage in downstream applications (which do better with longer contexts); (2) this challenge can be addressed by automatically converting resource-rich reading comprehension datasets into longer-premise NLI datasets; and (3) models trained on the converted, longer-premise datasets outperform those trained using short-premise traditional NLI datasets on downstream tasks primarily due to the difference in premise lengths.
Large-scale, open Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2018) have catalyzed the recent development of NLI models that exhibit close to human-level performance. However, the use of these NLI models for other downstream Natural Language Processing (NLP) tasks has met with limited success. Two of the most popular downstream tasks where NLI models' use has been explored are Multiple-choice Question Answering (MCRC) and Checking Factual Correctness of Summaries (CFCS) (Trivedi et al., 2019; Falke et al., 2019; Clark et al., 2018) - both of which can easily be cast into the NLI form, as shown in Figure 1. Looking closely at the composition of these datasets, it is evident that there is a stark difference in the lengths of the contexts/premises when compared to NLI datasets. As seen in Table 1, traditional NLI datasets have much
Passage/Premise/Full text
The first time my father and I ever went fishing became a family legend. ... We were hot, sticky, and mad that the fish refused to suck up our night crawlers . . . While driving out we saw a truck with a boat trailer and boat that was stuck in the mud. ... my dad helped pull the man from the mud. In return, this fellow gave dad some fish ... we agreed to take in the fish as if we had caught them. ... As we got up to do the dishes, mom cleared her throat. "I just have one question of you two great fishermen, How was it again that you two managed to not only clean your fish, but also freeze them before you got home."
Hypothesis (NLI)
The fishing became a family legend because they make themselves a fool in front of the mother.
Question (QA)
Why did fishing become a family legend?
They make themselves a fool in front of the mother
Summary (Summarization)
The father and son pretended catching a fish which was given to them making a fool in front of the mother. That is how the fishing trip became a family legend.
Figure 1: The tasks of Question Answering and Checking Factual Consistency of Text-Summaries can naturally be transformed into the Natural Language Inference problem.
shorter premises than the context texts from these downstream tasks. Prior research has shown that the capabilities required for handling local inference are very different from those required to perform inference over longer forms of text (Cooper et al., 1996; Lai et al., 2017a). In this work, we explore this conflict as a major bottleneck in the utility of NLI models (trained on traditional NLI datasets) for downstream NLP tasks. We compare the usage of long and short-premise NLI datasets on the downstream tasks of MCRC and CFCS, which have inherently long contexts.
Such a comparison has not been possible thus far because traditional NLI datasets do not exhibit long premises. We hence look towards recasting other tasks into NLI to generate datasets that can
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1322-1336 June 6-11, 2021. ©2021 Association for Computational Linguistics
Task	Dataset	Word Count (Avg)
NLI	SNLI	14
Scitail	17
MNLI	22
RTE	42
ANLI	54
MCRC	RACE	271
MultiRC	252
DREAM	110
CosmosQA	75
CFCS	FactCC	546
Summary Reranking	738
Table 1: The average premise (context) length in various datasets. The key point to notice here is the sharp increase in premise lengths from NLI datasets to MCRC and CFCS datasets.
be used to evaluate our conjecture. The Question-Answering (QA) task can easily be cast into the NLI form, and QA datasets (Rajpurkar et al., 2016; Lai et al., 2017b; Khashabi et al., 2018; Sun et al., 2019; Huang et al., 2019) encompass a variety of semantic phenomena that only occur in longer (con)texts. We leverage the resource-rich MCRC task to generate long-premise NLI datasets for our experiments via an automated conversion strategy.
We contrast the zero-shot model performance on the MCRC and CFCS tasks of a model pre-trained on our converted long-premise NLI dataset and a model trained on two short-premise NLI datasets - MNLI and ANLI. We show that the presence of longer premises is the primary factor for better performance on these two tasks. We further discuss other potential confounding factors for this performance difference - such as dataset vocabulary overlap and dataset conversion strategies - and eliminate the possibility of their contribution through targeted experiments.
Performance on the NLI task has improved significantly due to the availability of large scale datasets (Bowman et al., 2015; Williams et al., 2018) that can be used to train data-hungry deep learning models (Kapanipathi et al., 2020; Wang and Jiang, 2015), including transformer-based architectures (Devlin et al., 2018). However, there has been very limited success in translating this performance to downstream NLP tasks. Work relevant to the use of these NLI models for downstream
tasks can be categorized into two categories: (1) work focusing on using models trained on short-premise NLI datasets with fixed or learned aggregations over segmented premises to perform a target downstream task with long contexts (Falke et al., 2019; Trivedi et al., 2019); and (2) work addressing the need for task-specific NLI datasets (Krysciński et al., 2019; Demszky et al., 2018; Welleck et al., 2019).
Despite several attempts, efforts to apply models trained on available NLI datasets to downstream NLP tasks such as MCRC and CFCS have had limited success. Trivedi et al. (2019) use hand-crafted rules to first cast MCRC to NLI; and subsequently divide a long passage into smaller sentence-level premises. They use a pre-trained NLI model to evaluate per-sentence relevance scores concerning one particular hypothesis, and then combine the resulting scores using a learned representation aggregation module to assess the answer given the long passage. Falke et al. (2019) apply a similar approach for the CFCS task, and divide both the provided summary as well as the source documents into single-sentence premises and hypotheses. They use a max pooling operation over the entailment scores of all sentence-level premise-hypothesis pairs to obtain the factual correctness score for each provided summary. Both these works note that models trained on sentence-level NLI datasets do not transfer well to the MCRC and CFCS tasks. We argue that this divide and conquer approach is not ideal for the problem, and highlight the need for an NLI dataset with longer premises.
Another line of research focuses on re-casting datasets from other tasks into an NLI form to facilitate the direct use of NLI models on downstream tasks like MCRC and CFCS. Khot et al. (2018) use manual annotation to re-cast SciQ (a QA dataset) to SciTail – an NLI dataset. However, Clark et al. (2018) show that an NLI model trained on SciTail does not perform well on the task of MCRC. Similarly, Krysciński et al. (2019) create an automatically generated training dataset for CFCS. Even though the generated data has relatively long contexts, analysis in Zhang et al. (2020) demonstrated that a model trained on the aforementioned data showed performance improvement only when the token overlap with the source is high. Besides, Demszky et al. (2018) derive an NLI dataset by converting subsets of various QA datasets. They try two approaches for the conversion – rule-based
and neural. For the rule-based approach, they extract POS tags from the question-answer pair and apply hand-crafted rules on them to convert the pair to a hypothesis sentence. Their neural approach uses a trained SEQ2SEQ BiLSTM-with-copy model (Gu et al., 2016) to convert each <question, answer> pair into a hypothesis sentence (the corresponding passage being the premise). While their approach looks promising, they do not show the utility of these converted datasets by training an NLI model on them. Thus, it remains unclear whether the NLI datasets generated by the conversion are beneficial for NLP tasks. We posit that this direction of research is promising and largely unexplored. In our work, we attempt to leverage the abundance of large and diverse MCRC datasets to generate long-premise NLI datasets, and show that such datasets are useful towards addressing downstream NLP tasks such as MCRC and CFCS which have inherently long contexts.
3 NLI for Downstream Tasks
Typically, NLI is cast as a multi-class classification problem, where given a premise and a hypothesis, the model classifies the relation between them as entails, contradicts, or neutral. For the two downstream tasks under consideration: (1) MCRC: Multiple Choice Reading Comprehension, and (2) CFCS: Checking Factual Correctness of Text-Summarization; differentiating between the neutral and contradicts class is often unnecessary. The task is thus reduced to a two-class problem; where the contradicts and neutral classes are clubbed into a not-entails class.
MCRC can be cast as an NLI task by viewing the given context as the premise and the transformed question-answer combinations as different hypotheses (Trivedi et al., 2019). The multiple answeroption setting can then be approached as: (a) an individual option entailment task, where more than one answer-option can be correct; or (b) a multiclass classification task across all the answer options, when only a single correct answer exists.
CFCS can also be reduced to a two-class NLI problem. A factually correct summary should be entailed by the given source text – it should not contain hallucinated facts, and it should also not contradict facts present in the source text.
3.1 The Long Premise Conjecture
Despite being ideally suited for reduction to NLI, both MCRC and CFCS have proved to be difficult to solve using models trained on short-premise NLI datasets (Trivedi et al., 2019; Falke et al., 2019). Datasets for these tasks contain significantly longer contexts than traditional short-premise NLI datasets (Table 1). This shift in the text length brings about a fundamental change in the nature of the NLI problem. Thus, models trained on short-premise NLI datasets are incapable of performing inference over longer texts, which we posit as the main cause for their poor performance on downstream tasks like CFCS and MCRC*.
The paucity of manually-annotated long-premise NLI datasets poses a barrier to assessing this conjecture. We thus shift our focus towards leveraging the abundance of large and diverse MCRC datasets which can be easily recast into NLI form. While the CFCS task also provides a similar opportunity, the sheer lack of annotated training instances inhibits its use. Table 3 shows the abundance of training instances in MCRC datasets, and highlights the deficiency in CFCS datasets.
In the following section, we present our conversion strategy for reformating MCRC datasets into long-premise NLI datasets, which are needed to test the long premise conjecture.
4 Conversion of MCRC to NLI
As shown in Figure 1, we can convert MCRC datasets into two-class NLI datasets by reusing the passage as a premise, and paraphrasing the question along with each answer option as individual hypothesis options.
We begin by using a rule-based conversion method. A dependency parse of both the question and answer option is generated using the Stanford CoreNLP package (Qi et al., 2018). This is followed by the application of conversion rules proposed by Demszky et al. (2018) to generate a hypothesis sentence. However, due to the limited coverage of rules and errors in the dependency parse, some of the generated hypotheses sound unnatural (e.g. the first example in Table 2). In order to generate more natural and diverse hypotheses and to get broader coverage in conversion, we implement a neural conversion strategy.
*In our experiments, we broadly consider long texts, and do not differentiate between long single sentences and multiple sentences.
Rule-based	Neural	Hybrid
Q: What building were the four captives inside on Tuesday?
A: CNN headquarters	The four captives inside on Tuesday were CNN headquarters.	The four captives were inside CNN headquarters on Tuesday.	The four captives were inside CNN headquarters on Tuesday.
Q: How many people were hurt when overhanging metalwork crashed onto a stage in a Toronto park Saturday afternoon.
A: Four	Four were hurt when overhanging metalwork crashed onto a stage in a Toronto park Saturday afternoon.	Four people were hurt when overhanging metalwork crashed onto a stage in a Toronto park Saturday afternoon.. “” # # # # “.” was the number of peo	Four were hurt when overhanging metalwork crashed onto a stage in a Toronto park Saturday afternoon.
Table 2: Examples of Rule-based, Neural and Hybrid Conversions
Due to the recent success of transformer-based text generation models, we train a BART (Lewis et al., 2019) model to generate a grammatically coherent hypothesis from question + answer option (word/phrase) as input. We use a sequence of datasets as a curriculum to finetune the BART conversion model: (1) starting with CNN/Daily Mail summarization dataset (Hermann et al., 2015), which makes the generated sentences coherent; (2) followed by Google's sentence compression dataset (Filippova and Altun, 2013), which limits the generated sequence to a single sentence; and (3) finally the annotated dataset provided by Demszky et al. (2018) which has around 71,000 <question-answer, hypothesis> pairs from various QA datasets. Based on manual inspection, we find that the hypotheses generated by this method indeed sound more natural and diverse than the ones produced by the rule-based conversion†. In some cases, however, the generated hypotheses either discard crucial information, or contain hallucinated facts that do not convey the exact information in the source question-answer pair (Table 2). We thus define a hybrid conversion strategy, combining the desirable aspects of the rule-based and neural conversion strategies.
We design a heuristic to compose a hybrid dataset to overcome the caveats in the neural conversion. We use the number of words in the question-answer concatenation as a proxy for the expected length of the hypothesis. We target the problems of hallucination and missing information in the neural conversions by accepting only those neural-generated hypotheses that lie in the range of 0.8 and 1.2 times the length of the question-answer concatenation. We replace the rejected neural hypotheses with the rule-based hypothesis, if rule-based conversion is feasible; or with the
question-answer concatenation otherwise; as seen in Table 2. The selection policy is driven by the need to get more natural and coherent conversions without compromising on the accuracy and preservation of factual information in the question and answer option. The choice of the specific range is purely empirical in nature. We use this hybrid conversion strategy to generate long-premise NLI datasets from MCRC datasets for our experiments and evaluate them in contrast to short-premise NLI datasets.
Our experiments involve zero-shot evaluations of pre-trained NLI models on downstream NLP tasks. In this section, we describe the transfer learning setup and the datasets used in our experiments.
5.1 A Transferable NLI model
$\ddagger$  In order to use a pretrained NLI model for MCRC and CFCS, we need that model to be agnostic to the peculiarities of the downstream task. We use a standard transfer learning setting where the model architecture is divided into two parts: (1) a transferable entailment scorer; and (2) a weight-free comparator on top of the scorer. Each premise-hypothesis pair is encoded as a single sequence, and passed through the transferable entailment scorer to produce an entailment score. Depending on the problem setup, the comparator can either be a sigmoid function (for a two-class entailment problem) as shown in Figure 2; or a softmax function (for multiple choice classification) as shown in Figure 3. This segmentation of the model makes it easy to transfer the model weights across different tasks. For the entailment scorer, we use a 2-layer feedforward network on top of the [CLS] token of
$\dagger$  More examples of conversion results are presented in Appendix D.
$^{\ddagger}$ Code available here: https://github.com/nli-for-qa/transformers-nli
pre-trained RoBERTa  $\S$
To evaluate the transferability of the entailment model, we perform various zero-shot evaluations. This requires interpreting the entailment scores a bit differently for each task. To transfer the weights from a multiple choice classification model (Figure 3) to a two class entailment model (Figure 2), we copy the weights of the transferable entailment scorer as-is, and calibrate a threshold using a dev set to interpret the outputs from the sigmoid comparator for binary classification. Since the softmax comparator does not need any calibration, the transfer in the other direction, i.e., from a two class entailment model to a multiple choice classification model is more straightforward – we simply copy the weights of the transferable entailment scorer.
For our experiments, we use the NLI form of 4 MCRC datasets (created using the conversion method described in Section 4); 2 CFCS datasets; and 2 traditional short-premise NLI datasets. These datasets are described below:
MCRC Datasets:
RACE (Lai et al., 2017b) broadly covers detail reasoning, whole-picture reasoning, passage summarization, and attitude analysis.
MultiRC (Khashabi et al., 2018) mainly contains questions which require multi-hop reasoning and co-reference resolution.
DREAM (Sun et al., 2019) is a dialogue-based MCRC dataset, where the context is a multi-turn, multi-party dialogue.
CosmosQA (Huang et al., 2019) focuses on commonsense and inductive reasoning, which require reading between the lines.
CFCS Datasets:
FactCC (Krysciński et al., 2019) consists of tuples of the form <article, sentence>, where the articles are taken from the CNN/DailyMail corpus, and sentences come from the summaries for these articles generated using several state-of-the-art abstractive summarization models.
Ranking Summaries for Correctness (evaluation set) (Falke et al., 2019) consists of articles and a set of summary alternatives for each article, where
Task	Dataset	Dataset Size
MCRC	RACE	87866
MultiRC	27243
DREAM	6116
CosmosQA	23766
CFCS	FactCC	931
Summary Reranking	1000
Table 3: The number of annotated instances in MCRC and CFCS datasets. MCRC is an extremely resource-rich task whereas CFCS is considerably resource-deficient.
some of the provided summaries are factually inconsistent with respect to the article.
Short-Premise NLI Datasets:
MNLI (Williams et al., 2018) is a large-scale general domain NLI dataset that is widely used to learn and evaluate short-premise NLI models.
ANLI (Nie et al., 2019) is a large-scale NLI dataset generated through an adversarial human-in-the-loop process; where the annotations are constrained such that models trained on MNLI and SNLI predict incorrect answers. This dataset also has the longest premise lengths amongst the traditional NLI datasets compared in Table 1.
Long-Premise NLI Datasets:
We convert the following MCRC datasets to generate long-premise NLI datasets using the hybrid conversion strategy described in Section D. We refer to these datasets with a subscript converted attached to the source MCRC dataset.
As seen from Table 1 and Table 3, RACE is the largest dataset amongst the MCRC datasets, and also has the longest average premise length. In line with this intuition, the model trained on the  $\mathrm{RACE}_{\mathrm{converted}}$  dataset outperforms the converted forms of other MCRC datasets (Appendix B) on all the evaluation tasks. Due to this, in the following section, we only discuss and report results on the  $\mathrm{RACE}_{\mathrm{converted}}$  dataset for brevity and clarity of comparison. Amongst the traditional NLI datasets, we use MNLI and ANLI for a good mix of average premise lengths along with a large number of training samples.
Our experiments aim to answer the following questions: (1) Are long premise NLI datasets more use
$^{\S}$ The RoBERTa model is pre-trained on the masked language modeling objective as described in Liu et al. (2019). We obtain it from the HuggingFace library (Wolf et al., 2019).  $^{\text{II}}$ Questions where the answer is "None of the above" are removed from the CosmosQA dataset.
Figure 2: Two class entaiment model.
ful for downstream tasks compared to short premise NLI datasets? (Section 6.1 & 6.2); (2) How much do possible confounding factors affect our empirical evaluations? (Section 6.3). To answer these, we perform zero-shot evaluation on the MCRC and CFCS tasks.
We contrast the performance of NLI models trained on the short-premise NLI datasets (MNLI, ANLI) with one that is trained on a long-premise NLI dataset  $(\mathrm{RACE}_{\mathrm{converted}})$ . The models trained on short-premise NLI datasets are evaluated in two ways: (1) by treating the entire premise as input; and (2) by segmenting the premise into shorter segments and using a max aggregation over the entailment scores of all the segments (Falke et al., 2019). Since the model architecture remains the same, we use the name of the training dataset to refer to the model trained on it.
6.1 Evaluation on MCRC
For evaluating NLI models on the MCRC task, we use the hybrid conversion (Section 4) to create evaluation datasets. The MultiRC dataset contains multiple correct answer options and hence is evaluated with each question-answer option posed as a separate example. DREAM and CosmosQA datasets have only a single correct answer-option (out of 3 answer-options). Hence, for these datasets, a multiclass classification problem is posed as described in Section 3, using the model architecture described in Figure 3.
As seen in Table 4, the model trained on the long-premise  $\mathrm{RACE}_{\mathrm{converted}}$  dataset outperforms the model trained on the short-premise NLI datasets in both regular and segmented forms of evaluation. We assert that this difference in performance can
Figure 3: Multiple choice classification model.
Model\Dataset*	MultiRC	DREAM	CosmosQA
Random Guess	50.00	33.33	33.33
MNLI	60.58	67.76	38.11
\( MNLI_{segmented} \)	61.71	42.28	43.28
ANLI	67.95	74.12	49.71
\( ANLI_{segmented} \)	63.45	61.42	49.60
\( RACE_{converted} \)	77.43	83.58	73.58
* Datasets are in NLI form created using hybrid conversion method (Section 4).
Table 4: Zero-shot evaluation accuracies on MCRC datasets (in NLI form) using the transferable model architecture described in Section 5.1.
be attributed to the difference in premise lengths of the datasets. However, we allow for the possibility that using the same conversion strategy for the evaluation datasets could potentially benefit the model trained on  $\mathrm{RACE}_{\mathrm{converted}}$ . We discuss such confounding factors in Section 6.3.2.
6.2 Evaluation on CFCS
Evaluations on CFCS are set up in two ways: