# Evaluating Deep Learning Techniques for Natural Language Inference
## 3 Methodology  
We configured and trained 5 DL models, DAM, ESIM, BERT, RoBERTa, and ALBERT, along with their variations, on the 8 most popular datasets (presented in the next section). We tuned the hyperparameters for each model and task to achieve the best accuracies. Next, we tested our model's inference ability on the Breaking NLI dataset when trained on SNLI or MNLI. Finally, we carried out a generalization experiment with the 5 state-of-the-art models and the 8 datasets (4 three-way classifications and 4 two-way classifications).  
For DAM, we used both the vanilla version as well as the version with the intra-sentence attention. For the DAM implementations, we chose our hyperparameters starting from the recommended ones in the original paper. The values we tried were a batch size of {16, 32, 64}, hidden layer size of {100, 200, 300}, the AdaGrad optimizer with a learning rate of {0.05, 0.025, 0.01}, a dropout rate of {0.2, 0.5}, and a weight decay of {10\({}^{-5}\), 10\({}^{-4}\)}. We used 300-dimensional GloVe embeddings, which remained fixed during training. We trained the model with an early stopping patience value depending on each dataset. The GluonNLP implementations demonstrated 85.0% accuracy on SNLI, which is 1.3% lower than the original paper.  
We trained ESIM starting from the hyperparameters of the original paper. In our experiments, we tried the following values: a batch size of {16, 32, 64}, a hidden layer size of {100, 200, 300}, the Adam optimizer with a learning rate of {0.0001, 0.0004, 0.0005}, a dropout rate of {0.5, 0.8}, and a weight decay of {10\({}^{-5}\), 10\({}^{-4}\)}. Similarly, with DAM, we used 300d GloVe embeddings and early stopping for training.  
We trained BERT (Base/Large) and RoBERTa (Base/Large) models on all datasets. On the GLUE datasets (MNLI, QNLI, RTE, WNLI), we used the GluonNLP toolkit and tried the following hyperparameters for BERT-Base: a batch size of {16, 32, 64}, a learning rate of {10\({}^{-5}\), 10\({}^{-4}\), 2 \(\times\) 10\({}^{-5}\), 3 \(\times\) 10\({}^{-5}\), 4 \(\times\) 10\({}^{-5}\), 5 \(\times\) 10\({}^{-5}\), 5 \(\times\) 10\({}^{-6}\)}, a warmup ratio of {0.05, 0.1, 0.2}, and {4} epochs. For the Large versions, we were restricted to using a batch size of 16 due to GPU memory limitations on Kaggle. Especially for the MNLI dataset, we trained it for 3 epochs due to Kaggle time restrictions. For the other 4 datasets, we used a different toolkit: JiantNLP. We still trained the models with the same batch sizes and learning rates; however, we performed validation tests more often on a subset (500 samples) of the valuation set. Furthermore, we performed early stopping on the accuracies of these subsets. Therefore, the training process was quicker without losing on performance. In the gluonNLP experiments, the max input sequence was set to 128, and on JiantNLP, it was set to 256. In all the experiments, we used the AdamW optimizer with an epsilon of 10\({}^{-6}\).  
We used JiantNLP for all the experiments with ALBERT (max input sequence was set to 256). We used the base and the large versions of ALBERT, which are significantly smaller than the BERT and Roberta equivalents. We tried the same hyperparameters with the addition of using a bigger batch size with the large version, as its smaller size allows it. The rest of the details are the same as the previous experiments with Jiant.