# Evaluating Deep Learning Techniques for Natural Language Inference
## 6 Testing Generalization
### Talman and Chatzikyriakidis' Test
The worst performance came from the large models, with BERT-large having a drop of 31.32%, RoBERTa-large a 30.9% drop in accuracy, and ALBERT-large having a 31.9% drop in accuracy. The base models had the best generalization score, with ALBERT-base having the best of those (60.84%) with a drop of 19.05%. Generalizing from SICK to MNLI, however, doubles the drop in performance difference from baseline. All the models fail on MNLI when trained on the SICK training set. The best-performing model is BERT-base, with an accuracy of 48.29% and a drop of 39.34%. We see a much larger difference between these two datasets, as MNLI is a newer and more difficult set. Therefore, models trained on MNLI generalize better than those trained on SICK. Next, we compared MNLI and DNLI as far as their generalization capability is concerned. The best generalization performance from MNLI to DNLI came from RoBERTa-base, demonstrated an accuracy of 69.2%. The largest drop from baseline came from RoBERTa-large. Both DAM and ESIM dropped their accuracy by 14% from their baseline, with ESIM achieving 62.12% accuracy, which was a higher score than some of the transformers. The opposite generalization failed across all the models. Allmodels performed just above random chance, with the best being RoBERTa-large with a 41.38% accuracy, representing a 50% drop from the baseline. This decrease notably points out the simplicity, and hence, the weak generalization, of DNLI. Testing generalization from SNLI to SICK brought results that are comparable to those of MNLI to SICK. The best-performing model was ALBERT-base, with an accuracy of 61.42%, which means a drop of 25.07% from baseline. Non-transformers stayed at around 50%, with a drop of 33%. The opposite generalization testing gave far worse results, as expected. The models cannot learn the task of SNLI, and performance across all models was below 50%, except for BERT-base, which scored a 50.06% accuracy, resulting in a drop of 37.57%. The non-transformer models performed just above random chance, both at around 36% accuracy. Comparing SNLI and DNLI, we found that the best-performing model was both versions of RoBERTa, with the best being RoBERTa-base with a 62.00% accuracy and a drop of 24.95%. The rest of the transformers performed just above 50%, and the non-transformers were even lower than that. The best model to generalize from DNLI to SNLI was BERT-base, with a score of 54.53% and a drop of 35.42%. The large transformers performed worse than the base versions in this experiment, scoring accuracies comparable to the non-transformers. Finally, we tested generalization between the two lowest-scoring models, SICK and DNLI. Only one model passed the 50% mark, and that was BERT-base; the rest of the \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Training Data**} & \multirow{2}{*}{**Test Data**} & \multicolumn{2}{c}{**Largest Drop in Acc.**} & \multicolumn{2}{c}{**Smallest Drop in Acc.**} \\ \cline{3-6}  & & **Value** & **Model** & **Value** & **Model** \\ \hline QNLI & SCTAIL * & 25.99 & ALBERT-Base & 16.49 & DAM \\ QNLI & RTE & 44.15 & ALBERT-Base & 30.60 & DAM \\ SCTAIL & QNLI & 41.96 & RoBERTa-Large & 18.00 & DAM \\ SCTAIL & RTE & 30.97 & BERT-Base & 18.84 & ESIM \\ RTE & QNLI & 27.93 & RoBERTa-Large & 10.04 & ESIM \\ RTE & SCTAIL * & 24.04 & DAM & \(-\)14.19 & BERT-Large \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The largest and smallest drops in accuracy (%) of models trained on data of two labels belonging to a different dataset from that of test data, with reference to the baseline models (trained on data from the same dataset as the test set). Data with (*) are validation data. models exhibited lower scores. Generalizing from DNLI to SICK resulted in the lowest scores out of all the experiments, with the best being the 42.38% accuracy of RoBERTa-base. As far as two-label classification is concerned (Table 6), testing generalization from QNLI to SciTail, we observed the smallest drop from RoBERTa-large and the largest from ALBERT-base.