SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ence in the slope before and after the turning point. Specifically, OOD performance demonstrates slow growth until the turning point, after which a minor increase in ID performance yields a substantial improvement in OOD performance. The observed trend may be attributed to the findings of [91], which indicates that models initially capture spurious correlations in ID datasets before acquiring comprehensive task knowledge. Consequently, models prioritize learning these spurious correlations to address the ID task, resulting in minimal improvements on OOD datasets. However, in the later stages of training, models progressively acquire greater task knowledge, leading to improved OOD performance.
Type III. The V-shaped fitted lines shown in Figure 2c mainly occurs on ID-OOD pairs of NLI tasks. This pattern is divided into two stages by a turning point in ID performance. In the first stage, OOD performance experiences worsening performance degradation during the distribution shift. However, in the second stage, the ID-OOD correlation becomes positive. This trend resembles the U-shaped scaling law of LLMs observed by [100], thus suggesting a shared explanation. [100] attributes this phenomenon to the "distractor task" in the dataset, apart from the "true task". Medium-capacity models may perform better than low-capacity models on the distractor task, which may harm performance on the "true task". As the model capability increases, it can ignore the distractor task and focus on improving performance on the true task. Here, we identify the distractor task in NLI datasets as detecting the word overlap or other spurious correlations between the premise and hypothesis.
Outlier. There is an exceptional case regarding the distribution shift from Civil Comments to AdvCivil (see Figure 3). The figure depicts two distinct lines, both exhibiting a monotonic linear negative correlation. This may stem from the model's increased reliance on spurious correlations and the adversarial nature of AdvCivil. Prior research suggests that models can learn non-robust features, such as spurious correlations, to enhance ID accuracy [44]. However, adversarial samples preserve the semantics of the original texts while introducing perturbations that eliminate spurious correlations. Hence, when the ID model becomes more dependent on spurious correlations during training, its performance degradation on adversarial samples intensifies.
Figure 3: The OOD performance exhibits a negative correlation with ID performance. Refer to Figure 2 for legends.
4 Evaluation of OOD Robustness
4.1 Robustness-enhanced Methods
After analyzing the learning behavior of PLMs under vanilla fine-tuning, we examine their performance when trained with other methods. Although massive methods have been proposed to improve the robustness of PLMs, their evaluations rely on non-standard benchmarks, which may result in inaccurate evaluations and impede progress clarity. Therefore, in this section, we first conduct extensive experiments to re-evaluate the effectiveness of diverse robustness-enhanced methods.
Setting. We consider the categories of robustness-enhanced methods summarized by [99]: data-driven, model and training-based, inductive-prior-based, and causal intervention methods. We select the most representative one from each category for evaluation. Specifically, we choose EDA [101] and FreeLB [118] for data-driven methods, label smoothing [84] and focal loss [58] for model and training-based methods, and model ensemble [16] for inductive-prior-based methods. We do not consider causal intervention methods since they are typically applied to low-resource scenarios. As explained in section 2.3, we apply the above methods to DeBERTa-base models for the NER task and to T5-base models for the other tasks.
Results. The results are shown in Table 5, where the mark ' -' indicates that a certain method is not applicable to the task. We summarize the findings in the following takeaways:
Takeaway 1: The vanilla fine-tuning (empirical risk minimization) remains a strong baseline. Despite existing methods outperforming vanilla fine-tuning on certain datasets like E-NER, WNUT, and NewsQA, they show limited superiority or can potentially harm model performance. Specifically, only FreeLB demonstrates beneficial effects over half of the datasets, standing out as the most
Table 5: The evaluation of robustness-enhanced methods. The results that surpass the vanilla baseline are underlined. We use abbreviations representative of the datasets to conserve space. Please refer to Table 4 for their corresponding full dataset names.
Task	SA	TD	NLI	NER	EQA
Dataset	AZ	DS	SE	SST	CC	AC	IH	TG	MN	AN	CN	WN	FN	CoNLL	ENER	WNET	SQuAD	AQA	NQA	SQA
Vanilla	90.94	47.38	49.90	75.16	87.15	57.47	63.77	68.83	89.40	36.19	37.06	63.32	79.89	69.10	48.01	45.45	93.14	51.19	63.77	37.47
EDA	91.66	46.39	48.02	75.82	87.15	57.47	63.77	68.83	65.57	34.50	46.25	46.40	-	-	-	-	-	-	-	-
FreeLB	91.39	47.94	47.88	76.66	85.63	63.55	62.22	67.98	89.83	36.13	40.94	63.58	80.08	66.66	50.84	47.77	93.51	51.07	65.03	39.57
FL	91.02	46.20	50.11	76.76	87.10	57.72	62.29	67.66	89.17	36.53	39.26	63.32	79.30	61.04	50.49	45.51	92.97	50.64	63.96	36.03
LS	90.19	47.31	46.35	76.19	86.65	57.84	62.66	67.98	89.68	36.50	39.36	62.86	79.66	68.81	48.04	47.21	93.32	50.93	63.97	34.51
ES	50.72	41.83	54.98	63.36	82.99	47.02	61.05	65.32	77.67	35.16	17.79	17.79	-	-	-	-	-	-	-	-
Table 6: Evaluations of LLMs on BOSS. Small Ref represents the results of supervised fine-tuned small models in Table 5 (Vanilla). We observe that given enough ID data, fine-tuning domain-specific models is predominant when testing on ID examples. In contrast, LLMs with In-context learning should be given priority on OOD instances.
Task	SA	TD	NLI	NER	EQA
Dataset	AZ	DS	SE	SST	CC	AC	IH	TG	MN	AN	CN	WN	FN	CoNLL	ENER	WNUT	SQuAD	AQA	NQA	SQA
Small Ref	Full-data	90.94	47.38	49.90	75.16	88.63	50.67	62.29	65.74	89.40	36.19	37.06	63.32	79.89	69.10	48.01	45.45	93.14	51.19	63.77	37.47
T0-3B	0-shot	88.33	43.80	41.08	58.76	10.60	80.92	38.48	44.15	44.50	35.00	46.29	39.82	0	0	0	0	80.84	41.89	54.23	39.58
T5-3B	0-shot	84.55	33.63	34.27	37.68	21.03	75.94	40.72	44.04	35.44	33.53	46.29	37.16	0	0	0	0	46.64	18.88	21.37	13.42
ICL	84.55	33.63	34.27	37.68	21.03	75.94	40.72	44.04	35.44	33.53	46.29	37.16	0	0	0	0	35.50	17.16	9.13	7.75
5-shot	66.73	42.73	44.12	63.92	66.08	43.01	57.58	54.47	33.19	34.94	11.24	47.60	0	0	0	0	57.67	23.74	26.02	15.10
Full-data	90.63	51.11	50.93	74.13	86.32	60.75	63.12	70.64	90.64	45.97	44.57	65.16	46.80	48.28	56.63	34.97	94.38	57.48	66.40	34.75
LLaMA-7B	0-shot	75.66	54.05	37.60	46.43	67.72	43.70	57.33	59.98	32.81	26.83	68.18	44.44	0.49	0.38	0.07	0	58.98	30.22	40.78	45.80
ICL	84.30	55.19	42.66	59.14	89.70	20.17	63.62	59.68	39.81	33.50	19.30	38.17	0.63	0.70	0.81	0.16	67.57	37.35	44.15	43.78
ICL*	-	52.26	47.25	53.80	-	-	-	60.74	-	33.47	-	37.98	-	0.21	-	0	-	37.09	48.32	-
LLaMA-13B	0-shot	81.35	56.48	42.73	59.59	89.87	19.60	62.65	58.80	32.07	24.43	47.06	38.14	0.16	1.00	0.47	0.00	66.90	37.34	45.15	55.60
ICL	82.72	54.71	40.63	57.69	83.91	38.88	66.96	67.87	38.58	34.28	20.42	38.26	0.37	1.11	0.85	0.00	69.71	41.90	45.32	58.67
ICL*	-	46.56	38.46	53.05	-	-	-	68.09	-	36.00	-	37.31	-	0.50	-	0.00	-	40.99	49.30	-
Davinci3	0-shot	82.92	70.09	66.86	77.13	74.20	69.14	60.56	69.04	69.74	48.44	37.40	52.24	38.96	47.03	35.52	32.54	83.68	55.50	54.78	63.74
ICL	84.60	74.88	68.88	77.23	78.18	65.98	61.96	77.00	71.21	50.88	55.33	53.32	48.56	57.08	46.27	42.10	82.33	57.58	54.79	69.55
ICL*	-	75.74	65.14	73.95	-	-	-	75.29	-	49.91	-	53.06	-	55.83	-	40.71	-	57.90	56.15	-
Turbo	0-shot	85.63	74.46	68.33	77.04	76.10	80.57	52.69	72.76	68.78	46.84	50.82	48.95	36.53	51.36	30.29	33.35	81.21	50.49	47.68	64.38
ICL	87.75	77.18	65.26	75.07	60.16	80.56	54.67	76.07	68.12	49.47	49.59	49.47	40.21	57.02	38.77	35.25	82.82	50.78	50.37	64.10
ICL*	-	79.10	66.79	74.95	-	-	-	73.90	-	49.72	-	49.46	-	56.12	-	36.38	-	54.29	56.63	-
effective approach. Conversely, the inductive-prior-based ensemble is the worst, consistently leading to performance degradation except for the SemEval dataset.
Takeaway 2: Method effectiveness is consistent, yet limited to specific datasets. Across multiple datasets, various methods consistently demonstrate (in)effectiveness, as indicated in Table 5. However, no method consistently performs well on all datasets within the same task.
To summarize, current approaches fall short of meeting the expectations in enhancing the OOD robustness of models, highlighting the urgent demand for more advanced improvement techniques.
4.2 Large Language Models
LLMs are receiving increasing attention from NLP researchers. Considering the impressive zero/few-shot ability of LLMs, and the large difference in their fine-tuning and in-context learning paradigms, it is also intriguing to benchmark their generalizability on various downstream tasks and explore the best paradigm to leverage their power.
Setting. We consider three prominent state-of-the-art LLMs, LLaMA-7B and LLaMA-13B (i.e., LLaMA-series) [89], OpenAI text-davinci-003 [9] and gpt-3.5-turbo (denoted as Davinci3 and Turbo respectively). For comparison, we include two relatively smaller (yet still large) models, T0-3B [81] and T5-3B. We perform zero-shot inference based on task instructions on all these models since this paradigm is the most general. Then, we adopt other paradigms in a model-specific way. For T5-3B, we include 5-shot and full-data fine-tuning, and we also select exemplars from the ID dataset for in-context learning. For the other three LLMs, we apply in-context learning with two kinds of contexts, one from the ID dataset and another from the original training split of the evaluated OOD dataset, denoted as ICL and  $\mathrm{ICL}^*$  respectively. The implementation details are in Appendix E.2.
Results. We present the results in Table 6, where the mark  $\cdot$  means that ICL\* paradigm is not applicable for those datasets due to the absence of a training split or the limit of context window size. Our findings can be summarized as follows:
Takeaway 1: Fine-tuning small domain-specific models is superior when enough training data is available, while LLMs may be favored in low-resource scenarios. To be specific, supervised fine-tuned small models and T5-3B with the entire dataset consistently achieves the best performance on the ID dataset, especially for the structure prediction task (e.g., NER). In contrast, LLMs exhibit
better performance on most OOD datasets. This observation reinforces the view that large pre-trained models possess strong generalization capabilities, whereas, with sufficient training data, an accurate estimate of data distribution can be achieved even without a large number of parameters [76].
Takeaway 2: In-context learning always brings no gains to the generalization ability of small models, while it generally helps Turbo and significantly improves LLaMA-series and Davinci3. For small models like T5-3B, the performance of in-context learning is the same with or even worse than the zero-shot inference. For Turbo, providing ID examples for in-context learning presents advantages on nearly two-thirds of the datasets, with the NER task benefiting the most. For LLaMA-series and Davinci3, the superiority of in-context learning is prominent as it enhances performances on most of the datasets.
Takeaway 3: Examples from ID datasets are generally more effective for in-context learning than those from the original training split of the testing OOD dataset. Specifically, when considering samples from our OOD datasets as contexts, the performance of Turbo is comparable to using ID samples, whereas the LLaMA-series and Davinci3 models consistently exhibit inferior performance compared to using ID examples as contexts. However, all models show improved performance on the EQA task when contexts from our OOD datasets are utilized. This may be attributed to the variations in sample length or question styles across EQA datasets, hence models acquire more precise instructions from the original training samples. The overall ineffectiveness of ICL* can be explained by the findings of [69]. According to [69], in-context demonstrations aim to guide the models to learn the target label space, instead of the feature-label mapping. The ID examples contain more diverse information due to the construction process of our benchmark. Thus, ID examples can better prompt the language models to locate the target label space, compared to the OOD examples that may target a specific domain.
Discussion. Two paradigms are prevalent in developing downstream NLP systems: leveraging general-purpose LLMs or gathering domain-specific data for fine-tuning smaller models. For the first paradigm, the overarching objective of general-purpose LLM development is to employ a single model for solving various downstream tasks [9]. Consequently, LLMs are anticipated to exhibit high performance on both ID and OOD datasets. However, our study exposes the shortcomings of LLMs on ID datasets when compared to fine-tuned domain-specific models. Considering the higher inference and deployment costs associated with LLMs, substantial progress is still needed to effectively improve LLMs in developing downstream applications, particularly for challenging tasks like EQA. For the second paradigm, our study reveals the limitations of fine-tuning models on ID datasets for OOD performance in comparison to LLMs. Thus, further research is required to develop advanced techniques that enhance the robustness of fine-tuned domain-specific models. Overall, the existing two prevalent paradigms still fall short in addressing the OOD problem in NLP, necessitating further advancements and effective approaches.
However, we also note that there can exist confounders in our evaluations. It is still ambiguous which datasets are indeed OOD to LLMs, given that LLMs have been pre-trained on massive public corpora. The potential data contamination issue can result in overinflated performance on our OOD datasets, tangling the credit of the memory and generalizability of LLMs. The only confirmed distribution shift for LLMs is the temporal shift, necessitating the evaluation based on data released subsequent to their pre-training data collection cut-off. Therefore, the NLP community demands new downstream datasets independent of the pre-training corpus to meet the evaluation requirements for LLMs.
Distribution shifts in NLP has been widely studied in various forms. We examine several representative cases as outlined below. Domain shift refers to the challenge of testing data originating from diverse domains, often due to data collection from various sources [63, 40, 53, 79]. Temporal shift examines the degradation of models' performance over time [42, 1]. Spurious correlation examines the issue of models acquiring dataset-specific knowledge on ID data, which may not generalize effectively to OOD data [66, 73, 91, 32, 37, 16, 17]. Additionally, a requirement is for models to exhibit robustness when confronted with artificially constructed OOD samples. One typical type is malicious adversarial attacks, which involve assessing the resilience of models against inputs crafted by malevolent adversaries [56, 51, 112]. These inputs, distinct from ID samples, have the potential to induce model failures [12]. Adversarial attacks can also be effectively utilized to simulate
diverse user inputs to examine models' robustness in the real world [98, 13, 33]. Another category is backdoor attacks, characterized by intentionally introduced spurious correlations that can be exploited by attackers for their advantage [18, 55].
OOD Evaluation in NLP can be broadly classified into automatic and static evaluation approaches. Automatic evaluation utilizes diverse textual transformation techniques, such as introducing typos, to conduct a rigorous evaluation of OOD robustness. Three essential elements in the automatic OOD evaluation encompass the establishment of suitable transformation methods, evaluation metrics, and effective techniques to ensure sample validity [34, 98]. Static evaluation, in contrast to automated methods, offers the advantage of constructing benchmarks with higher quality, resulting in an improved estimation of OOD robustness. Numerous OOD benchmarks have been introduced, focusing on adversarial attacks [96] or spurious correlations [117, 66]. A relevant study to ours is GLUE-X [107], which establishes an OOD benchmark derived from the GLUE benchmark [93]. Nevertheless, they do not establish a coherent benchmark construction protocol and primarily rely on dataset selection driven by popularity, incorporating datasets into the benchmark without comprehensive explanation and seemingly opting for a somewhat arbitrary selection, thus lacking a systematic approach.
We revisit OOD robustness research in NLP, identifying deficiencies in benchmarks and evaluation. Correspondingly, a benchmark construction protocol and an OOD robustness evaluation suite are proposed to facilitate future research. The correlation between OOD and ID performance, the effectiveness of existing methods, and the challenges faced by LLMs are investigated.
We identify two limitations in this work. First, as discussed in section 4.2, due to the lack of new datasets in the community, there is a possibility that some datasets have been included in the pretraining corpus of LLMs, so they may not be suitable to test the generalizability of recent LLMs. However, we note that with our benchmark construction protocol, we can easily update the benchmark as new datasets come out. Second, we only consider five tasks in this benchmark, which is not a comprehensive collection of current NLP literature. We explain the reason for the current task selection in Appendix A.1.
This work is sponsored by the Tsinghua-Toyota Joint Research Fund.
Lifan Yuan and Yangyi Chen initiated the project. Lifan Yuan, Yangyi Chen, and Ganqu Cui designed the experiments. Lifan Yuan, Yangyi Chen, and Hongcheng Gao constructed the AdvCivil dataset. Lifan Yuan conducted experiments and wrote the paper. Yangyi Chen and Ganqu Cui revised the paper. Everyone participated in the discussion. Heng Ji, Zhiyuan Liu, and Maosong Sun advised the project.
[1] Oshin Agarwal and Ani Nenkova. Temporal effects on pre-trained models for language processing tasks. arXiv preprint arXiv:2111.12790, 2021.
[2] Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. In Proceedings of EMNLP, 2021.
[3] Ting Wai Terence Au, Vasileios Lampos, and Ingemar Cox. E-NER — an annotated named entity recognition corpus of legal text. In Proceedings of the Natural Legal Language Processing Workshop, 2022.
[4] Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the AI: Investigating adversarial human annotation for reading comprehension. Transactions of ACL, 2020.
[107] Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. In Findings of ACL, 2023.
[108] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of EMNLP, 2018.
[109] Hai Ye, Yuyang Ding, Juntao Li, and Hwee Tou Ng. Robust question answering against distribution shifts with test-time adaption: An empirical study. In *Findings of EMNLP*, 2022.
[110] Wenpeng Yin, Dragomir Radev, and Caiming Xiong. Docnli: A large-scale dataset for document-level natural language inference. In Findings of ACL, 2021.
[111] Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. Learning and evaluating general linguistic intelligence, 2019.
[112] Lifan Yuan, Yichi Zhang, Yangyi Chen, and Wei Wei. Bridge the gap between CV and nlp! A gradient-based textual adversarial attack framework. In Findings of ACL, 2023.
[113] Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. In Proceedings of NAACL, 2019.
[114] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL, 2019.
[115] Aston Zhang, Zachary Chase Lipton, Mu Li, and Alex Smola. Dive into deep learning. CoRR, 2021.
[116] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of NeurIPS, 2015.
[117] Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130, 2019.
[118] Chen Zhu, Yu Cheng, Zhe Gan, S. Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In Proceedings of ICLR, 2019.
Table 7: The comparison among different semantic representation methods, from which we can observe that despite different methods varying to some extent, the trend of similarity almost remains unchanged.
Method	SimCSE	ESimCSE	Sentence-Transformer
Src / Tgt	AZ	DSC	DS	IMDb	SE	SST	Yelp	AZ	DSC	DS	IMDb	SE	SST	Yelp	AZ	DSC	DS	IMDb	SE	SST	Yelp
AZ	100	86.02	57.30	36.67	24.74	33.70	49.22	100	95.27	88.46	76.07	76.50	76.19	81.75	100	90.25	46.57	48.62	30.65	48.92	49.23
DSC	86.02	100.00	59.15	54.55	31.70	44.40	55.45	95.27	100	84.22	80.06	76.68	80.97	79.26	90.25	100	37.27	61.54	25.28	54.79	49.49
DS	57.30	59.15	100	32.69	28.17	19.68	88.99	88.46	84.22	100	70.64	79.28	74.41	90.52	46.57	37.27	100	24.76	36.94	32.65	78.37
IMDb	36.67	54.55	32.69	100	46.95	84.62	39.88	76.07	80.06	70.64	100	72.16	83.73	72.82	48.62	61.54	24.76	100	25.22	84.26	34.05
SE	24.74	31.70	28.17	46.95	100	40.45	24.03	76.50	76.68	79.28	72.16	100	76.75	68.64	30.65	25.28	36.94	25.22	100	40.66	19.26
SST	33.70	44.40	19.68	84.62	40.45	100	19.43	76.19	80.97	74.41	83.73	76.75	100	62.44	48.92	54.79	32.65	84.26	40.66	100	24.58
Yelp	49.22	55.45	88.99	39.88	24.03	19.43	100	81.75	79.26	90.52	72.82	68.64	62.44	100	49.23	49.49	78.37	34.05	19.26	24.58	100
A.1 What is the rationale for current task selection and why not include more difficult tasks?
The chosen tasks within BOSS evaluate models across natural language understanding, structured data prediction, and question answering, which are core language model competencies. In constructing our benchmark, we did consider extending other tasks such as NLG and commonsense reasoning, but in practice, we found a lot of difficulties.
For NLG tasks, the primary concern lies in evaluation. Since different datasets exhibit distinct text styles, an in-domain (ID) model might generate responses with styles diverging from the reference answers when tested on out-of-domain (OOD) datasets. However, current NLG metrics evaluate predictions based on their resemblance to reference answers, which might not accurately reflect text quality in scenarios with a vast output space [35]. An emerging alternative involves employing Language Models like GPT-4 for scoring predictions. However, this method is cost-intensive and lacks reproducibility due to unpredictable updates. Considering these evaluation challenges, including generation tasks within this OOD benchmark might not be appropriate.
For commonsense reasoning, multiple datasets exist from various sources, each demanding distinct knowledge. These differences are substantial enough that knowledge gained from an ID dataset might not be applicable to OOD datasets. For instance, HellaSwag [114] necessitates basic world knowledge and logical reasoning to complete a sentence, while StepGame [82] relies on spatial imagination without requiring world knowledge. The dissimilarity in required abilities suggests that models should acquire knowledge through pre-training rather than fine-tuning on an ID dataset and then transferring it to OOD tasks. This approach aligns more reasonably with the task's nature.
A.2 Why is SimCSE chosen to measure distribution shift?