# Evaluating Deep Learning Techniques for Natural Language Inference
## 4 Datasets
The hypotheses were created by combining the question with its correct answer. To collect the premises, the authors used an IR method to obtain relative sentences from the web. The premises and hypotheses were then given to AMT crowd workers for labeling. If the premise supported the answer, the label \"entails\" was given, or else the label \"neutral\" was given. Thus, SciTail is a 2-way classification dataset. SciTail is split into training, development, and test sets, with each having 23.596/2.126/1.304 examples, respectively. In the whole set, there are 17,000 'neutral' examples and 10,000 'entail' examples. The Recognizing Textual Entailment Challenges (RTE) [3] were a series of datasets for the task of NLI. The first RTE challenge was created to provide a framework for capturing semantic inferences across applications. RTE-1 was the first formulation of the NLI task, and in total, there are 7 RTE challenges. The first RTE challenge was comprised of pairs obtained with different NLP tasks, such as IE and IR, and labeled as true or false. RTE-2 provided more realistic examples. RTE-3 included some longer premises (up to a paragraph). RTE-4 introduced three-way classification. RTE-5 introduced even longer premises (up to 100 words). In our experiments, we used the RTE dataset included in the GLUE benchmark, which is a combination of RTE-1, 2, 3 and 5. RTE-5 was converted to a two-way classification task by merging 'contradiction' and 'neutral' into 'not entailment'. RTE is split into training and development sets, with each having 2500 and 300 examples, respectively. The Question-Answering NLI dataset (QNLI) was created from SQuAD [34], a question-answering dataset of over 100,000 questions on 500+ articles. To convert SQuAD into an NLI dataset, each question in SQuAD was matched with every sentence in its article. The task of QNLI is to find if the sentence contains the answer to the question. In total, QNLI is comprised of 105,000 examples in the training set and 5500 in the development set. There are two labels, 'entailment' and 'not entailment', and they are distributed equally in the dataset. The Winograd NLI dataset (WNLI) was created from the Winograd Schema Challenge (WSC) [35]. WSC is a reading comprehension dataset that was published as an alternative to the Turing test. An example of WSC consists of a sentence and a question. The sentence in WSC contains two noun phrases or parties, and later in the sentence, a pronoun is used to refer to one of the parties. The question involves determining which party the pronoun is referring to. The problem is recasted to NLI by replacing the pronoun with each of the parties both in the sentence and the question. WNLI is a small dataset of 634 examples in the training set and 71 in the development set. The two labels, 'entailment' and 'not entailment', are equally distributed. The BreakingNLI dataset [14] is a single test set of over 8000 examples. The pairs were taken from the SNLI dataset but edited to differ by one word at maximum. The goal of this is to make NLI models fail. The test set contains mostly contradictions (7164), few entailments (982), and very few neutrals (47).