Venue: NAACL 2022

# Embarrassingly Simple Performance Prediction for

Abductive Natural Language Inference

Emils Kadikis, Vaibhav Srivastava, and Roman Klinger

Institut fur Maschinelle Sprachverarbeitung

University of Stuttgart

Pfaffenwaldring 5b, 70569 Stuttgart

{emils.kadikis,vaibhav.srivastav,klinger@ims.uni-stuttgart.de}

###### Abstract

The task of abductive natural language inference (\(\alpha\)NLI), to decide which hypothesis is the more likely explanation for a set of observations, is a particularly difficult type of NLI. Instead of just determining a causal relationship, it requires common sense to also evaluate how reasonable an explanation is. All recent competitive systems build on top of contextualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particular NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this practical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the \(\alpha\)NLI task when just comparing sentence embeddings with cosine similarity to what the performance that is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Since the similarity computation is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 6031 - 6037

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Abduction is a type of reasoning that infers an explanation for some observations Peirce (1931). It is a particularly challenging type of inference; as opposed to deduction and induction, which derive the conclusion from only the information present in the observations, abduction requires making assumptions about an implicit context beyond just the given observations. Abductive reasoning is therefore at the core of the way humans understand the world and how world knowledge is involved.

Abductive reasoning in the natural language domain has been introduced by Bhagavatula et al. (2020) who defined the abductive natural language inference (\(\alpha\)NLI) task. In it, we are given four sentences - two observations \(o_{1}\) and \(o_{2}\) and two hypotheses \(h_{1}\) and \(h_{2}\), where we know that the sequence of events was \(o_{1}\rightarrow(h_{1}|h_{2})\to o_{2}\). The task then is to decide which of the two hypotheses is the more plausible one.

An example from Bhagavatula et al. (2020) is the following:

* [noitemsep]
* It was lunchtime and Kat was hungry.
* Kat and her coworkers enjoyed a nice lunch outside.
* Kat went to get a salad.
* Kat decided to take a nap instead of eating.

While it is not inconceivable that someone would decide to take a nap on their lunch break (\(h_{2}\)), given \(o_{2}\) the first hypothesis becomes more likely.

Currently, transformer-based architectures Vaswani et al. (2017) are state of the art in a wide variety of natural language processing (NLP) tasks Devlin et al. (2019); He et al. (2021); Li et al. (2021), including \(\alpha\)NLI. However, with an ever-changing landscape of transformer models and pre-training techniques (with over 100001 different fine-tuned models available on the HuggingFace hub Wolf et al. (2020)), finding the best model for a given task has become a time-consuming process since, in order to compare multiple models, they each need to be separately fine-tuned on the task.

This model selection process might lead to a prohibitive runtime, which has led to research on performance prediction, namely to predict the expected performance out of parameters of the model configuration, without actually training the model. This procedure has been evaluated for a set of NLP tasks, including span prediction Papay et al. (2020) and language modelling Chen (2009). However,

we are not aware of any previous work that performed performance prediction for \(\alpha\)NLI.

In this paper, we introduce a fast performance prediction method for the \(\alpha\)NLI task that allows a more guided way of choosing which models to fine-tune. We use various pre-trained transformer models to embed the observations and hypotheses with the approach introduced in Reimers and Gurevych (2019), then compare which hypothesis is closer to the observations with cosine similarity. We find that the performance of the similarity-based approach is correlated to results obtained via fine-tuning. Therefore, the similarity-based approach can serve as a performance prediction method.

## 2 Related work

There are three research topics that need to be mentioned. Approaches to abductive reasoning, pretrained language models, and performance prediction. In this section, we explore them in detail.

Abductive natural language inference.NLI has been proposed as the task of recognizing textual entailment by MacCartney and Manning (2008) and now constitutes a major challenge in NLP which has found application for other downstream tasks, including question answering or zero-shot classification Yin et al. (2019); Mishra et al. (2020). Based on the initial goal of establishing inference relations between two short texts, a myriad of variants have been proposed Yin et al. (2021); Williams et al. (2018); Bowman et al. (2015). One such variant is abductive NLI (\(\alpha\)NLI, Bhagavatula et al. (2020).

Transformer-based architectures dominate the \(\alpha\)NLI leaderboard,2 including RoBERTa-based models Liu et al. (2020); Mitra et al. (2020) which explore how additional knowledge can improve performance on reasoning tasks, and Zhu et al. (2020) who approach \(\alpha\)NLI as a ranking task. The task authors improved upon their result in Lourie et al. (2021) by using a T5 model Raffel et al. (2020) and experimenting with multi-task training and fine-tuning over multiple reasoning tasks. Both the multi-task criteria and the Text-to-Text framework of T5 help the model generalize and understand the context better.

Footnote 2: [https://leaderboard.allenai.org/anli/submissions/public](https://leaderboard.allenai.org/anli/submissions/public)

The second-best model on the leaderboard is a DeBERta model He et al. (2021). The model replaces the masked language modelling task with a replaced-token detection task. It also uses a disentangled attention mechanism to encode the position and content information.

The current state of the art shows an accuracy of 91.18% using a new unified-modal pre-training method to leverage multi-modal data for single-modal tasks Li et al. (2021). This result approaches the human baseline of 92.9%.

Pre-trained language models.The \(\alpha\)NLI task requires the model to successfully "understand" the context of both the observations and use that understanding to identify the more likely hypothesis entailing it. Most semantic representations in practical applications rely on distributional semantics. Such methods include the word-level embedding methods Word2Vec Mikolov et al. (2013) and GloVe Pennington et al. (2014) and language model-based word representation like ELMo Peters et al. (2018), ULMFit Howard and Ruder (2018), and GPT Radford et al. (2018).

The current state of the art are pre-trained transformer architectures Vaswani et al. (2017) like BERT Devlin et al. (2019), which use a masked language modelling and next sentence prediction objective. This not only helps the model understand the context within a sentence but also intabetween consecutive sentences. There is however a trade-off in terms of the time it takes to train transformer models. For example, a from-scratch training of BERT takes 6.4 days on an 8 GPU Nvidia V100 server3Devlin et al. (2019) recommend finetuning the language model between 2-4 epochs for a given task. However, in practice, multiple trials are required to find the optimal hyperparameters. These long training times and multiple fine-tuning runs make model selection a time-intensive process Liu and Wang (2021).

Footnote 3: [https://aws.amazon.com/blogs/machine-learning/amazon-web-services-achieves-fastestraining-times-for-bert-and-mask-r-cnn/](https://aws.amazon.com/blogs/machine-learning/amazon-web-services-achieves-fastestraining-times-for-bert-and-mask-r-cnn/)

Performance prediction.The task of performance prediction is to estimate the performance of a specific system without explicitly running it. It helps in setting hyperparameters, finding feature sets, or identifying candidate language models for a downstream NLP task. Chen (2009) develop, for instance, a generalized method for predicting the performance of exponential language models. They analyze the backoff features in an exponential \(n\)-gram model. Papay 

meta-learning to identify candidate model performance on the task of span identification. They train a linear regressor as a meta-model to predict span ID performance based on model features and task metrics for an unseen task. Ye et al. (2021) propose performance prediction methods particularly suited for fine-grained evaluation metrics. They also develop methods for estimating the reliability of these performance predictions. Contrary to the previously mentioned papers, Xia et al. (2020) build regression models to predict the performance across a variety of NLP tasks, however, they do not consider NLI as one of them.

In contrast to our work, all these previous methods build on top of the idea to train a surrogate model for performance prediction and depend on the information about past runs of these models. Our approach focuses solely on the embeddings provided by the language model and leverages those as a predictor of performance. This particular setup is also motivated by the \(\alpha\)NLI task itself, in which a sentence needs to be chosen for a given set of other sentences.

## 3 Methods

Our paper investigates how well a fine-tuned transformer model's performance on the \(\alpha\)NLI task Bhagavatula et al. (2020) is approximated by the cosine similarity of embeddings of the input sentences which we obtain from the pre-trained models before fine-tuning them.

Intuitively, if a model embeds the correct hypothesis close to the observations in some latent space (not necessarily a semantic similarity space), then a classification model built on top of that latent space should have an easier time discerning which is the correct hypothesis, because apparently that latent space captured some features that were salient for the \(\alpha\)NLI task.

### Sentence Embedding

For both the similarity baseline and the fine-tuned classification model, the starting point is the pretrained transformer model itself. We add a mean pooling layer to convert the token-by-token output of the model into a single sentence embedding Reimers and Gurevych (2019). Given some tokenized input \({\bf x}=[x_{1},x_{2},\dots,x_{n}]\) and a pre-trained transformer model \(E\) which encodes each token \(E(x_{i})\), we calculate the sentence embedding \(\mathrm{emb}({\bf x})=\frac{1}{n}\sum_{i=1}^{n}E(x_{i})\).

Al alternative to mean pooling would have been to use the embedding of the CLS token. We opted against that for three reasons. Firstly, Reimers and Gurevych (2019) show that mean pooling slightly outperforms using the CLS token in their semantic similarity models. Secondly, for some models, the CLS token does not have any particular significance before fine-tuning on the downstream task due to the training objective they use (such as RoBERTa Liu et al. (2020), which uses masked language modelling). Thirdly, pooling is a general approach that can be adopted for any model, even if it does not output a CLS token. Since our goal was to avoid any model-specific enhancements, a universal blanket approach like this was preferable.

### Similarity-based \(\alpha\)Nli

To perform \(\alpha\)NLI on a validation instance, we obtain three sentence embeddings - one for the combined observations \(o_{1}+o_{2}\) and one for each of the hypotheses \(h_{1}\), \(h_{2}\). To predict the more plausible hypothesis, we calculate which of them is closer to the observations with cosine similarity:

\[\hat{h}=\operatorname*{arg\,max}_{h^{\prime}\in\{h_{1},h_{2}\}}\cos(\mathrm{ emb}(o_{1}+o_{2}),\mathrm{emb}(h^{\prime}))\]

### Classification-based \(\alpha\)Nli

For the classification model, we add a classification head on top of the pre-trained model, which consists of a mean pooling layer to get sentence embeddings, then a fully-connected layer and a softmax output layer. For each instance of (\(o_{1},o_{2},h_{1},h_{2}\)), the model takes two different inputs which consist of both observations with each of the hypotheses, namely \(\mathrm{emb}(o_{1}+o_{2}+h_{1})\) and \(\mathrm{emb}(o_{1}+o_{2}+h_{2})\).

Both of these input representations are then used in a fully connected layer \(f\) with a softmax output layer to get the probability score for each input. The hypothesis that is assigned the largest probability constitutes the prediction:

\[\hat{h}^{\prime}=\operatorname*{arg\,max}_{h\in\{h_{1},h_{2}\}}\mathrm{softmax }(f(\mathrm{emb}(o_{1}+o_{2}+h)))\]

In our experiments, we fine-tune the classification head on the \(\alpha\)NLI training set without updating weights in the underlying language model. This is mostly due to time and resource constraints, however, we believe that while fine-tuning would improve classification performance across the board, it would not affect the ranking as such. Since we are comparing models amongst themselves, the ranking is more important 



## 4 Experiments

We compare the similarity-prediction-based \(\alpha\)NLI approach and the classification-based \(\alpha\)NLI approach to evaluate if the first can act as an approximation for the performance expected by the second. We use the pre-trained transformer models which are available on the HuggingFace Wolf et al. (2020) hub. The full list of models we use is listed in Table 1. The code for the experiments is available online.4.

Footnote 4: [https://github.com/Vaibhavs1O/anli-performance-prediction](https://github.com/Vaibhavs1O/anli-performance-prediction)

### Dataset

All of our experiments were run on the train and validation split in the ART dataset provided for the \(\alpha\)NLI challenge Bhagavatula et al. (2020). It consists of 169,654 training and 1,532 validation samples, each consisting of two observations and two hypotheses obtained from a narrative short story corpus and augmented with wrong hypotheses written by crowd-sourced workers.

The training data contains repetitions of the same \((o_{1},o_{2})\) pairs with different sets of hypotheses, ranging from one plausible and one implausible hypothesis to two plausible hypotheses where one of them is more plausible. The validation set was constructed using adversarial filtering, which selects one plausible and one implausible hypothesis for each set of observations that are hard to distinguish. This increases the probability that the instances are free of annotation artifacts, which the authors defined as "unintentional patterns in the data that leak information about the target label" Bhagavatula et al. (2020).

### Experimental Setup

All of our classification and similarity experiments were run on an Nvidia RTX 2080 GPU. For training the classifier we used the maximum batch size that fit on the GPU (which is different for different sized models, ranging between 12 and 128). For similarity experiments, we only infer the embeddings from pre-trained models. For hyperparameter selection, to keep the comparison fair, we tuned the batch size and learning rate and considered the same set of possible combinations across all the models. The specific values used for each model are available in Table 2 in the appendix. We train for 3 epochs with learning rates ranging \([10^{-5};9\cdot 10^{-5}]\) and a weight decay of 0.01. For each pre-trained model, we pick the one that achieved the highest accuracy on the validation set.

### Evaluation and Results

Table 1 shows the results as accuracy scores, obtained with each pre-trained model when using cosine similarity and when using a classifier built on top of it. We also show the training runtimes.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & & Accuracy & Run time \\ \cline{2-5} Model & Citation & Sim. & Class. & Sim. & Class. \\ \hline Albert-base-v2 & Lan Lan (2020) & 50.78\% & 65.34\% & 5.68 & 0:55:58 \\ Albert-large-v2 & Lan Lan (2020) & 50.13\% & 69.71\% & 7.93 & 2:51:01 \\ bert-base-uncased & Devlin (2019) & 51.69\% & 65.99\% & 2.69 & 1:13:28 \\bert-large-uncased & Devlin (2019) & 52.67\% & 67.04\% & 7.22 & 3:13:54 \\ distilber-base-uncased & Sanh (2019) & 51.63\% & 62.60\% & 1.63 & 0:26:36 \\ squeezerbViewer-uncased & Iandola (2020) & 50.52\% & 61.95\% & 2.23 & 0:37:00 \\ google/mobilbert-uncased & Sun (2020) & 48.75\% & 61.68\% & 4.29 & 0:36:20 \\ google/canine-s & Clark (2021) & 49.21\% & 58.09\% & 3.82 & 1:30:40 \\ google/electrora-small-discriminator & Clark (2020) & 51.17\% & 63.51\% & 1.69 & 0:12:27 \\ google/electora-base-discriminator & Clark (2020) & 52.28\% & 78.07\% & 2.77 & 0:51:51 \\ google/electra-large-discriminator & Clark (2020) & 52.74\% & 88.51\% & 7.23 & 3:09:35 \\ microsoft/tmpnet-base & Song (2020) & 51.50\% & 74.87\% & 2.77 & 0:52:36 \\ roberta-base & Liu (2020) & 51.50\% & 74.15\% & 2.70 & 0:52:28 \\ roberta-large & Liu (2020) & 51.63\% & 84.14\% & 6.87 & 3:32:47 \\ google/bigbird-roberta-base & Zaheer (2020) & 51.50\% & 71.02\% & 5.74 & 1:02:08 \\ ksteveu/isbetr-roberta-base & Kim (2021) & 51.50\% & 73.63\% & 2.78 & 1:00:39 \\ distiliroberta-base & Sanh (2019) & 51.43\% & 65.99\% & 1.67 & 0:29:08 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracy on the \(\alpha\)NLI validation set using similarity and a classification model. The similarity runtime (how long it takes to evaluate the model since no training is required) is shown in seconds, the classification runtime (how long it takes to fine-tune and evaluate the model) in hours, minutes, and seconds. Note that the given training time is for a single set of hyperparameters. Identifying the best hyperparameters involved training each model multiple times.



The primary observation is that the accuracy scores of classification and similarity are correlated. This can be seen in Figure 1. The Pearson correlation coefficient is \(r=.65\) (\(p=0.005\)). The Spearman's correlation coefficient is \(\rho=.67\) (\(p=0.003\)), indicating that the ranking obtained with the similarity-based prediction is a reliable indicator that is helpful for model selection. Additionally, model fine-tuning takes on average 620 times longer than the similarity-based estimate. Tuning the hyperparameters involved training each model multiple times.

## 5 Conclusions & Future Work

In this paper, we have shown that similarity measures based on the distributional semantic representation in pre-trained transformer models serve as an effective proxy for fine-tuned transformer-based classification in \(\alpha\)NLI. Since fine-tuning a transformer model takes notably more time than performing similarity comparisons, our approach supports efficient model selection procedures. Future work should investigate the suitability of similarity-based performance prediction for other similar tasks, like next sentence prediction, question answering, summarization.

## Acknowledgements

We thank the anonymous reviewers and the action editor at ACL Rolling Review for their helpful comments. This work has been conducted in the context of projects funded by the German Research Council (DFG), project "Computational Event Analysis based on Appraisal Theories for Emotion Analysis" (CEAT, project number KL 2869/1-2) and project "Automatic Fact Checking for Biomedical Information in Social Media and Scientific Literature" (FIBISS, KL 2869/5-1).

## References

* D. Bahagavatula, R. L. Bras, C. Malaviya, K. Sakaguchi, A. Holtzman, H. Rashkin, D. Downey, W. tai Yih, and Y. Choi (2020)Abductive commonsense reasoning. In International Conference on Learning Representations, Cited by: SS4.
* S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 632-642. External Links: Link Cited by: SS4.
* S. Chen (2009)Performance prediction for exponential language models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Colorado, USA, pp. 450-458. External Links: ISBN 978-1-467-386-3, Document, Link Cited by: SS4.
* J. H. Clark, D. Garrette, I. Turc, and J. Wieting (2021)CANINE: pre-training an efficient tokenization-free encoder for language representation. CoRRabs/2103.06874. External Links: Link, 2103.06874 Cited by: SS4.
* K. Clark, M. Luong, Q. V. Le, and C. D. Manning (2020)Electra: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, Vol. 1, pp. 4171-4186. External Links: Link, Document Cited by: SS4.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link, Document Cited by: SS4.
* P. He, X. Liu, J. Gao, and W. Chen (2021)Deberta: decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, Cited by: SS4.
* J. Howard and S. Ruder (2018)Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 328-339. External Links: Link, Document Cited by: SS4.

Figure 1: Relation between similarity and classification accuracy scores. Note that the similarities are closer to each other than the classification values, therefore we chose a different scale.

 

Forrest Iandola, Albert Shaw, Ravi Krishna, and Kurt Keutzer. 2020. SqueezeBERT: What can computer vision teach NLP about efficient neural networks? In _Proceedings of SustainNLP: Workshop on Simple and Efficient Natural Language Processing_, pages 124-135, Online. Association for Computational Linguistics.
* Kim et al. (2021) Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. 1-bert: Integer-only bert quantization. In _Proceedings of the 38th International Conference on Machine Learning_, volume 139 of _Proceedings of Machine Learning Research_, pages 5506-5518. PMLR.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In _Proceedings of the International Conference on Learning Representations_.
* Li et al. (2021) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2592-2607, Online. Association for Computational Linguistics.
* Liu and Wang (2021) Xueqing Liu and Chi Wang. 2021. An empirical study on hyperparameter optimization for fine-tuning pre-trained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2286-2300, Online. Association for Computational Linguistics.
* Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A robustly optimized BERT pretraining approach.
* Lourie et al. (2021) Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. In _Proceedings of the AAAI Conference on Artificial Intelligence_, pages 13480-13488.
* MacCartney and Manning (2008) Bill MacCartney and Christopher D. Manning. 2008. Modeling semantic containment and exclusion in natural language inference. In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_, pages 521-528, Manchester, UK. Coling 2008 Organizing Committee.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
* Mishra et al. (2020) Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Li, Pavan Kapanipathi, and Kartik Talamadupula. 2020. Reading comprehension as natural language inference:a semantic analysis. In _Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics_, pages 12-19, Barcelona, Spain (Online). Association for Computational Linguistics.
* Mitra et al. (2020) Arindam Mitra, Pratyay Banerjee, Kuntal Kumar Pal, Swaroop Mishra, and Chitta Baral. 2020. How additional knowledge can improve natural language commonsense question answering?
* Papay et al. (2020) Sean Papay, Roman Klinger, and Sebastian Pado. 2020. Dissecting span identification tasks with performance prediction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4881-4895, Online. Association for Computational Linguistics.
* Peirce (1931) Charles Sanders Peirce. 1931. _Collected papers of Charles Sanders Peirce_. Harvard University Press.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Preprint.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics 

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In _NeurIPS EMC2 Workshop_.
* Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. In _Advances in Neural Information Processing Systems_, volume 33, pages 16857-16867. Curran Associates, Inc.
* Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2158-2170, Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Huggingface's transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xia et al. (2020) Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020. Predicting performance for natural language processing tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8625-8646, Online. Association for Computational Linguistics.
* Ye et al. (2021) Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. 2021. Towards more fine-grained and reliable NLP performance prediction. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 3703-3714, Online. Association for Computational Linguistics.
* Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
* Yin et al. (2021) Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document-level natural language inference. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 4913-4922, Online. Association for Computational Linguistics.
* Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. _Advances in Neural Information Processing Systems_, 33.
* Zhu et al. (2020) Yunchang Zhu, Liang Pang, Yanyan Lan, and Xueqi Cheng. 2020. L2r\({}^{2}\): Leveraging ranking for abductive reasoning. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '20.

## Appendix A Model Hyperparameters

All models were trained for 3 epochs with a weight decay of 0.01. The batch size and learning rate used for each model can be seen in Table 2

\begin{table}
\begin{tabular}{l r r} \hline \hline Model & Learning rate & Batch size \\ \cline{2-3}  & Albert-base-v2 & \(10^{-5}\) & 60 \\ \cline{2-3}  & alert-large-v2 & \(10^{-5}\) & 20 \\ \cline{2-3}  & bert-base-uncased & \(5\cdot 10^{-5}\) & 32 \\ \cline{2-3}  & bert-large-uncased & \(10^{-5}\) & 16 \\ \cline{2-3}  & distilbent-base-unc. & \(9\cdot 10^{-5}\) & 128 \\ \cline{2-3}  & squeezerbert/squeezeber-unc. & \(7\cdot 10^{-5}\) & 64 \\ \cline{2-3}  & google/mobilepet-unc. & \(7\cdot 10^{-5}\) & 100 \\ \cline{2-3}  & google/canines & \(10^{-5}\) & 24 \\ \cline{2-3}  & google/electra-small-discr. & \(7\cdot 10^{-5}\) & 128 \\ \cline{2-3}  & google/electra-base-discr. & \(3\cdot 10^{-5}\) & 64 \\ \cline{2-3}  & google/electrora-large-discr. & \(10^{-5}\) & 16 \\ \cline{2-3}  & microsoft/tmpnet-base & \(3\cdot 10^{-5}\) & 64 \\ \cline{2-3}  & roberta-base & \(3\cdot 10^{-5}\) & 60 \\ \cline{2-3}  & roberta-large & \(10^{-5}\) & 12 \\ \cline{2-3}  & google/bigbird-roberta-base & \(10^{-5}\) & 40 \\ \cline{2-3}  & ksksteven/ibert-roberta-base & \(3\cdot 10^{-5}\) & 64 \\ \cline{2-3}  & distilbrotetra-base & \(5\cdot 10^{-5}\) & 80 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The learning rate and batch size that resulted in the best classification accuracy for each model.