# Evaluating Deep Learning Techniques for Natural Language Inference
###### Abstract
Since then, a lot of progress has been made both in terms of available data, as well as in the development of models that try to face the NLI problem. In 2015, the first large-scale NLI dataset was collected and provided neural models with 550,000 examples of crowdsource-labeled data [4]. These assisted in gaining more attention to the field of NLI, which brought many advances but also many criticisms. In 2017, the GLUE benchmark for natural language understanding [5] was published, which includes four NLI task datasets (MNLI, QNLI, RTE, WNLI). The collection method of each dataset is different, and so each dataset evaluates a different type of inference ability. This benchmark later evolved into SuperGLUE [6], which includes even harder and more challenging tasks. As already stated, the main type of inference models today are deep neural network models. Up to 2017, the dominant type of inference model consisted of LSTMs (a deep learning model) encoding the premise and hypothesis, then applying an attention mechanism before passing the final vector to a SoftMax function for classification. A model that fits in this description is ESIM [7], which at that time demonstrated a state-of-the-art 88.0% accuracy on the SNLI dataset. Later adaptations of such models included additional learning from a knowledge graph to incorporate external world knowledge into the model [8]. Recently, the invention of transformers [9] and, more specifically, BERT [10] has switched research to using pre-trained models on large corpora of text that are then fine-tuned on specific data. Such models (BERT, XLNet, RoBERTa, ALBERT) quickly became the state-of-the-art models for NLI, performing with over 90% accuracy on some of the most challenging datasets. However, many researchers questioned the ability of these models' inference abilities as they argued that the models take advantage of various annotation artifacts within the \begin{table}
\begin{tabular}{l l l} \hline \hline
**Premise** & **Hypothesis** & **Label** \\ \hline A soccer game with multiple males playing & Some men are playing a sport & Entailment \\ The girl went out wearing a costume & The girl went out wearing a fairy costume & Neutral \\ A boat sank in the Pacific Ocean & A boat sank in the Atlantic Ocean & Contradiction \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Examples of NLI sentence pairs and their label. datasets to achieve their results. One of the first papers that addressed this issue was [PERSON] et al. 's hypothesis-only approach [PERSON] et al. (2018). In their paper, they showed that only using the hypothesis sentence for some NLI datasets performed well above random chance, suggesting that specific words in the hypotheses are related to a specific label. Furthermore, some studies [PERSON] et al. (2018) found instances of social bias in the examples of the SNLI dataset, while others [PERSON] et al. (2018) found wrong labels in the SICK dataset as well. To continue testing that idea, some researchers developed NLI stress tests to break top-performing NLI models, such as the BreakingNLI dataset [PERSON] et al. (2018). Furthermore, additional adversarial datasets were developed with a focus on collecting examples that top-performing models predict wrong [PERSON] et al. (2018). The protocol of collecting annotations for sentence pairs has been criticized by researchers in the field as not satisfactory [PERSON] et al. (2018); [PERSON] et al. (2018). The idea behind the criticisms is that one should not take the label most voted on by humans as the gold label and should not ignore other opinions. [PERSON] et al. (2018) believe that NLI needs a revision because the vague task of \"do as a human would\" is not in agreement with the fact that different humans can extract different conclusions from the same sentence pair. Therefore, they suggested a new type of measurement that takes into account the entire spectrum of opinions. One such dataset is ChaosNLI [PERSON] et al. (2018), which instead of gold labels, uses a distribution over a collective of human opinions.