# Evaluating Deep Learning Techniques for Natural Language Inference
###### Abstract
Another way of improving NLI is by creating new and more diverse datasets that cover a quite broad range of linguistic phenomena. An example in this direction is the IMPLI dataset [PERSON] et al. (2018), which uses sentence pairs of figurative language and tests models on idiomatic expressions and metaphors. Another dataset is ANLI, which uses adversarial data to trick NLI models. However, some researchers argue that models could be implemented to purposely exploit such data for better scores [PERSON] et al. (2018). One recent approach to creating datasets for NLI is WANLI WANLI (2018), which takes advantage of the progress of natural language generation models to include them in the process of data creation together with human annotators. Additionally, human explanations of data have been used in the training process, with the aim of improving NLI, with good results [PERSON] et al. (2018). We are not interested in a model that performs exceptionally well in a specific task; rather, we are looking for a model that genuinely learns to infer information and can do so using different datasets. One way to test this ability is by generalizing knowledge from one task to another, which is the main objective of this work. In this paper, we experiment with five (5) modern deep learning models plus their variations on eight (8) popular benchmarks for NLI. The models can be distinguished into two categories: _traditional deep learning models with attention mechanisms_ and _modern transformer-based models_. In the first category, we deal with the decomposable attention model (DAM) [PERSON] et al. (2018), which is a model that uses attention without LSTMs, and ESIM, which is a model that uses chain LSTMs with an attention mechanism. The second category of models comprises three BERT implementations: BERT, [PERSON] et al. (2018), and [PERSON] et al. (2018). These models are pre-trained on large corpora of text data. We also present and use eight popular NLI datasets: MNLI [PERSON] et al. (2018), QNLI, RTE, WNLI (all four belonging to the GLUE Benchmark), SNLI, [PERSON] et al. (2018), SICK Sticki et al. (2018), and DNLI [PERSON] et al. (2018). For each model and dataset combination, we explore a wide range of hyperparameters. We compare the best accuracies on the test or development set. Next, we test our models on BreakingNLI, a dataset that evaluates models' inference ability, which requires basic lexical and world knowledge. For this experiment, we train the models on SNLI and MNLI training data. Finally, we conduct generalization ability tests such as [PERSON]'s and [PERSON]'s tests Talman (2018), where we train each model on a dataset but test its predictions on test data from the other datasets. This experiment tests whether the inference ability of models can be transferred across NLI datasets. We show that BERT models very clearly outperform the old-generation models on all NLI benchmarks. Furthermore, between the BERT models, we see an improvement in accuracy when using the larger version, although ALBERT shows a comparative performance with 18 times fewer parameters. In the attempt to \"break\" our models with the BreakingNLI test set, we see that the pre-trained transformers keep their high scores in contrast to the other models. However, the BERT models fail on the generalization test as the accuracy drops significantly when the datasets are not similar enough, which is in line with findings in the literature [30]. The main contributions of this paper are:  
* Configuration of new state-of-the-art deep learning models for the NLI problem. * Extensive evaluation of the models on the most widely used datasets. * Extensive evaluation of the generalization ability of the models. The rest of the paper is organized as follows. Section 2 presents a literature review of the deep learning methods for solving the NLI problem. In Section 3, our methodology is described, whereas in Section 4, the datasets used are described.