# Evaluating Deep Learning Techniques for Natural Language Inference
## 6 Testing Generalization
### Talman and Chatzikyriakidis' Test  
The second method is presented in [30]. According to that, a model was developed based on a dataset and was tested on a different dataset. Therefore, to evaluate the generalization capabilities of the above models in conjunction with the used datasets, we conducted several experiments. First, we configured and evaluated the eight deep learning models based on each of seven (out of eight) datasets, i.e., using a training and a test set from the same dataset, which are the baseline models. Then, we configured and evaluated the eight deep learning models based on pairs of the seven datasets; that is, the training set was from one dataset, and the test set wasfrom the other dataset. In this way, we tested how well a trained model based on a dataset generalizes when it is used on a different dataset. The results in terms of the largest and smallest drops in accuracy in relation to baseline models are presented in Table 5 (for the datasets that use three labels) and in Table 6 (for the datasets that use two labels). The full results for both cases are presented in Appendix A. Testing generalization from MNLI to SNLI (Table 5), we find that the largest drop from baseline is that of DAM (11.43%) and the smallest is that of RoBERTa-large (2.49). The transformers generalize well to SNLI, having the largest drop in accuracy of 6.31% from baselines. The opposite, however, brings some interesting results. Generalizing from SNLI to MNLI, we observe a much larger drop in accuracies from the baselines. The best-performing model is still RoBERTa-large, with a drop of 8.57% in accuracy, while the drop in performance of the rest of the transformers ranges from 12% to 18%. The non-transformer \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **SNLI** & **MNLI** \\ \hline DAM & 31.60 & 30.11 \\ ESIM & 55.68 & 64.05 \\ BERT-Base & 87.28 & 90.89 \\ BERT-Large & 92.68 & 94.54 \\ RoBERTa-Base & 93.21 & 94.50 \\ RoBERTa-Large & 96.39 & **96.47** \\ ALBERT-Base & 93.94 & 92.89 \\ ALBERT-Large & **96.92** & 95.69 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracies (%) of the models on Breaking NLI test set trained on SNLI and MNLI. The best results are in bold. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Training Data**} & \multirow{2}{*}{**Test Data**} & \multicolumn{2}{c}{**Largest Drop in Acc.**} & \multicolumn{2}{c}{**Smallest Drop in Acc.**} \\ \cline{3-5}  & & **Value** & **Model** & **Value** & **Model** \\ \hline MNLI & SNLI & 11.43 & DAM & 2.49 & RoBERTa-Large \\ MNLI & SICK & 31.90 & ALBERT-Large & 19.05 & ALBERT-Base \\ MNLI & DNLI & 22.01 & RoBERTa-Large & 14.28 & DAM \\ SNLI & MNLI \({}^{*}\) & 25.34 & ESIM & 8.57 & RoBERTa-Large \\ SNLI & SICK & 33.70 & DAM & 25..07 & ALBERT-Base \\ SNLI & DNLI & 42.93 & DAM & 24.95 & ROBERTa-Base \\ SICK & MNLI \({}^{*}\) & 48.24 & ALBERT-Large & 36.75 & ESIM \\ SICK & SNLI & 48.79 & ALBERT-Large & 37.57 & BERT-Base \\ SICK & DNLI & 50.42 & ALBERT-Base & 37.24 & DAM \\ DNLI & MNLI \({}^{*}\) & 55.57 & ALBERT-Base & 46.42 & DAM \\ DNLI & SNLI & 54.17 & ALBERT-Large & 35.42 & BERT-Base \\ DNLI & SICK & 59.43 & BERT-Large & 48.18 & ESIM \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The largest and smallest drops in accuracy (%) of models trained on data of three labels belonging to a different dataset from that of test data, with reference to the baseline models (trained on data from the same dataset as the test set). Data with (*) are validation data. models perform even worse, with a drop of around 25%. The conclusion for this dataset pair is that MNLI is harder than SNLI and creates models with better generalization. Generalizing from MNLI to SICK, we find that performance drops at a high rate regardless of the model. In this case, the non-transformers are not the worst-performing models, as they suffered a drop of 23% from the baselines.