SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We refer to [105] as a representative study measuring text distribution shift via semantic vector distance. While [105] employs CLIP's encoder for multimodal contexts, our NLP-specific context steers us towards SimCSE for semantic representation.
To the best of our knowledge, SimCSE and sentence-BERT [80] are the most widely used models and SimCSE is more advanced according to the SimCSE paper, hence we choose SimCSE as the semantic representation model. In this discussion period, we also try the other variants of SimCSE, ESimCSE [104], and the best model in the Sentence Transformers leaderboard  $^{6}$  ('all-mpnet-base-v2'). The results are shown in Table 7. From the results, despite different methods varying to some extent, the trend of similarity almost remains unchanged. Therefore, we consider that the essential is to adopt an advanced semantic representation model, and that which particular model we choose will not lead to differences in our selection.
B Survey
Our survey draws from two NLP taxonomy papers, one focusing on robustness [99] and the other on generalization [43]. We meticulously examine all the references cited in these papers, presenting
$^{6}$ https://www.sbert.net/docs/pretrained_models.html#model-overview
Table 8: Survey of papers targeting to address OOD robustness across the five tasks as this paper. Column "Paper" signifies individual papers sorted by date of publication, column "Tasks" details the evaluated tasks, and column "Datasets" lists the encompassed datasets, with each row representing ID dataset  $\rightarrow$  OOD datasets.
Paper	Tasks	Datasets
[46]	SA	IMDb → Amazon, SemEval, Yelp
[85]	NLI	MNLI, SNLI, SICK → MNLI, SNLI, SICK
[30]	EQA	SQuAD, NewsQA, TtiviaQA, SearchQA, HotpotQA, Natural Questions → BioASQ, DROP, DuoRC, RACE, RelationExtraction, TextbookQA
[86]	EQA	SQuAD, NewsQA, SearchQA, TriviaQA, HotpotQA → CQ, CWQ, ComQA, WikiHop, DROP
[111]	EQA	SQuAD → TriviaQA, QuAC, QA-SRL, QA-ZRE
[40]	SA	SST2, IMDb, Yelp, Amazon → SST2, IMDb, Yelp, Amazon
[72]	SA	Amazon, Yelp, IMDb, SST2 → Amazon, Yelp, IMDb, SST2
[71]	NLI	MNLI, ANLI → MNLI, ANLI
[91]	TD	Wiki → Founta, Waseem
[95]	NLI	MNLI → HANS
[67]	EQA	SQuAD → Squashift
[103]	SA	SST-2 → Senti140, SemEval, Amzbook, Yelp, IMDb, IMDb-Cont., IMDb-CAD
[11]	NLI	SNLI → MNLI, SNLI-CAD, break, DNC, stress, diagnostic
[15]	SA	SST2$ → IMDb-Cont, IMDb-CAD
[15]	NLI	MNLI → ANLI, HANS
[27]	SA	SST2 → IMDB
[107]	NLI	MNLI → ANLI
[62]	EQA	SQuAD → Adv SQuAD
[57]	NLI	FEVER, MNLI → Symmetric, HANS
[52]	EQA	SQuAD → BioASQ, New Wikipedia, New York Times, Reddit posts, Amazon Reviews
[97]	SA	SST2, Amazon kitchen, Amazon electronics → SST, Amazon kitchen, Amazon electronics
[74]	SA	SST2 → Senti140, SemEval, Yelp, IMDB, Contrast, CAD
[107]	NLI	MNLI → PI, LI, HANS, WaNLI, SNLI, ANLI
[62]	TD	HateXplain
[47]	NLI	MNLI,SNLI → HANS
[106]	NER	CoNLL → CrossNER
[19]	NER	OntoNotes → 12B2'14, CONLL'03, WNUT'17, GUM, Few-NERD
[109]	EQA	SQuAD → NewsQA, SearchQA, TriviaQA, HotpotQA, Natural Questions
a subset of references that specifically address OOD robustness in sentiment analysis (SA), toxic detection (TD), natural language inference (NLI), name entity recognition (NER), and extractive question answering (EQA) tasks. The summarized findings are outlined in Table 8. Through the survey, we can observe that there is no single uniform set of datasets for evaluation in NLP, namely no "standardized OOD benchmark suites tailored for NLP". And we can find that existing work includes evaluation datasets without particular consideration while mainly directly using popular datasets, i.e. adopting "popularity-based dataset selection strategies".
In section 2.4, we outlined the datasets utilized in the BOSS benchmark. In this section, we provide additional descriptions for the datasets not covered in the main body of this paper, and elaborate on the data processing procedures for all candidate datasets.
C.1 Description
Sentiment Analysis. DSC [48] consists of product reviews of 24 different categories in Amazon, with binary sentiments. IMDb [64] is another binary classification dataset with movie reviews from the IMDb website. Yelp [116] dataset is constructed by extracting product reviews from the Yelp website.
Toxic Detection. AbuseAnalyzer [10] collects abusive contents from Grab, with labels of Hateful and non-Hateful. AdvCivil is a new adversarial dataset for toxic detection introduced in this paper. We employ the adversarial attack technique proposed by [12] to simulate real-world ma
licious attacks by introducing typos and distracting sentences into the original sentence. Additionally, a post-human inspection process is undertaken to validate the constructed samples. Three human annotators are employed to filter out label-inconsistent and semantically broken samples. Hate Speech [21] covers posts on various topics and nationalities from several sub-forums on Stormfront. HSOL [20] extracts tweets containing pre-defined toxic words or phrases and categorizes them into three classes: hate speech, offensive but not hate speech, or neither offensive nor hate speech. OLID [113] collects tweets and annotates each of them according to whether the text is offensive or not.
Natural Language Inference. BioNLI [5] is created from abstracts of biomedical publications on PubMed, with each sample labeled as entailed or not-entailed and regarding the experimental evidence as the premises. The entailed samples directly adopt the conclusions of mechanistic information in the abstracts as the hypothesis, while the not-entailed samples manipulate the information to generate pseudo conclusions with nine different strategies. CB [22] consists of clause-embedded texts from Wall Street Journal, British National Corpus, and Switchboard, each premise containing an embedded clause and the hypothesis being the extraction of the clause. DocNLI [110] processes five existing datasets of different tasks into NLI formats, all from news or Wikipedia. The premises are all at the document level, while the hypotheses have a variety of different granularities. SNLI [7] extracts image captions from Flickr30k as premises, and generates hypotheses by human annotation.
Name Entity Recognition. CrossNER [61] collect sentences from Wikipedia on topics of artificial intelligence, literature, music, natural science, and politics, covering seven coarse-grained entity types in Few-NERD except for the Building.
Extractive Question Answering. HotpotQA [108] is based on Wikipedia, with each question requiring information from two articles to answer. NaturalQuestions [50] takes real queries from Google search engine as questions, corresponding search results as contexts, and human-annotated texts as answers. Trivia-QA [45] collects long documents from Wikipedia and constructs questions that can not be directly answered by span extraction. SQuAD Shifts [67] inherits the dataset construction pipeline from SQuAD, but collects passages from more diverse sources, including Wikipedia, New York Times, Reddit, and Amazon.
C.2 Processing
C.2.1 Sentiment Analysis
Generally, datasets for this task are label-imbalanced, which is detrimental to model performance. Therefore, If not specified, we tackle this issue by aligning  $n_i$  with  $\min \{n_1, \dots, n_{|\text{classes}|}\}$  by discarding redundant samples in each class, where  $|\text{classes}|$  is the number of classes and  $n_i$  is the number of samples for label  $i$  in each dataset,  $i \in \{1, \dots, |\text{classes}|\}$ .
Amazon has 29 subsets of different products. To encourage diversity of the review texts, we aim to merge all the subsets together. However, the scale of the entire dataset is prodigious and the sizes of the subsets vary greatly, making model training time-consuming and potentially ignoring the effect of certain types of reviews. Therefore, we first reduce large subsets to 20k samples, and then drop samples of stars 2 or 4 and maintain samples of starts 1, 3, and 5. After that, we get a ternary classification of the entire dataset, split into training and test datasets by 9:1. Finally, for label balance, we sample 10k reviews for each class in the training dataset and discard the rest.
DSC is a binary classification dataset, thus samples of both labels will be retained. The final training and test dataset are formed by combining all types of product reviews.
Dynaset has two rounds of training and test datasets, which are incorporated together to constitute a unified training and test dataset.
IMDb and SemEval only require label balancing processing.
SST samples are annotated with a float score indicating the sentiment, ranging from 0 to 1. We follow the common practice of SST-5 to equally divide the score into 5 bins, i.e. mapping score 0.02 to
label 1, and so on. Then we drop samples of labels 1 and 3, similar to Amazon, to adapt for ternary classification.
Yelp reviews are also rated from 1 to 5 and retained only samples of stars 1, 3, and 5 following Amazon and SST. Due to the large amount of data, we sample 10k reviews from the training dataset for each label.
C.2.2 Toxic Detection
Datasets for this task are collected from online forums and social media, therefore the texts are commonly dirty, containing some meaningless strings such as @username in the beginning. We clean the texts by removing @username, emoji, tags (with a symbol #), and URLs. Moreover, the label balancing processing is the same as for Sentiment Analysis datasets.
AbuseAnalyzer does not have a train/test split, thus we divide the original dataset into a training dataset and a test dataset by 8:2.
Civil Comments samples with the toxicity score  $\geq 0.5$  are considered toxic samples while others whose toxicity score  $< 0.5$  are in the benign class.
Hate Speech is divided into training and test datasets by 8:2.
HSOL has three types of label: hate speech, offensive but not hate speech, or neither offensive nor hate speech. We adapt it to binary classification, treating samples with the first two toxic and samples with the label neither offensive nor hate speech as benign.
Implicit Hate is annotated into three classes, i.e., not hate, implicit hate and explicit hate, with implicit hate and explicit hate being considered toxic. In probing experiments, we split the training and test dataset by 8:2. After selecting the Implicit Hate dataset into our benchmark, we treat the entire dataset as the test dataset.
AdvCivil, OLID and ToxiGen do not require specific processing.
C.2.3 Natural Language Inference
Datasets for NLI do not have a serious label-imbalance problem, hence no corresponding processing is applied.
ANLI has three rounds in total, we merge them together to form the final training and test datasets respectively.
DocNLI contains some ANLI examples, so we filter those samples out to avoid repeating ANLI.
Other datasets, i.e., BioNLI, CB, ContractNLI, MNLI, SNLI, and WANLI, do not require any other specific processing. But note that the test dataset of MNLI is the matched validation dataset, and we do not use the unmatched one since the two subsets are too similar.
C.2.4 Name Entity Recognition
Datasets for NER usually have different name entity tags and thus require label mapping for alignment. For the cross-evaluation experiment, we align the labels of each dataset to CoNLL; while for other experiments, we take the scheme of Few-NERD as the standard. Also, we adapt the datasets to the prevalent BIO schema, which indicates whether a token is the Beginning, the Inside, or the Outside of an entity.
CoNLL contains the four most common entity types which are covered in Few-NERD, thus no label mapping is needed.
CrossNER constitutes various entity types from diverse domains. Therefore, we design the following mapping for label alignment:
labelMapping  $=$  {"academicjournal": "product", "album": "product", "algorithm": "miscellaneous",
"astronomicalobject": "miscellaneous",   
"award": "miscellaneous",   
"band": "organization",   
"book": "art",   
"chemicalcompound": "miscellaneous",   
"chemicalelement": "miscellaneous",   
"conference": "event",   
"country": "location",   
"discipline": "miscellaneous",   
"election": "event",   
"enzyme": "miscellaneous",   
"event": "event",   
"field": "miscellaneous",   
"literarygenre": "art",   
"location": "location",   
"magazine": "product",   
"metrics": "miscellaneous",   
"misc": "miscellaneous",   
"musicalartist": "person",   
"musicalinstrument": "product",   
"musicgenre": "art",   
"organisation": "organization",   
"person": "person",   
"poem": "art",   
"politicalparty": "organization",   
"politician": "person",   
"product": "product",   
"programlang": "miscellaneous",   
"protein": "miscellaneous",   
"researcher": "person",   
"scientist": "person",   
"song": "art",   
"task": "miscellaneous",   
"theory": "miscellaneous",   
"university": "organization",   
"writer": "person"
Few-NERD requires processing to adapt for the scheme of CoNLL in the probing experiments. Since it has covered all the entity types annotated in CoNLL, the only process is to set other tags, i.e. building, art, product, and event, to be miscellaneous. Note that this process is also required for CrossNER and WNUT in the cross-evaluation experiment, since these two datasets have already been aligned with Few-NERD.
WNUT conduct the following operation to align labels with Few-NERD:
labelMapping  $=$  { "corporation": "organization", "creative-work": "art", "group": "organization", "location": "location", "person": "person", "product": "product" }
C.2.5 Extractive Question Answering
Datasets are all normalized to a unified extractive setting with the same format as SQuAD, following MRQA [30].
AdvQA has several subsets generated by fooling different models. We adopt the combination of all these subsets for our experiments.
Table 9: Statistics of toxic detection candidate datasets.
Dataset	Source	# Classes	# Samples	Avg. Length
Train	Test	Train	Test
AbuseAnalyzer	Grab	2	6,080	1,520	14.24	14.38
AdvCivil	Adversarial	2	-	823	-	70.73
Civil Comments	Civil Comments	2	60,000	97,320	50.09	51.15
Hate Speech	Stormfront	2	8,562	2,141	18.07	18.1
HSOL	Twitter	2	5,823	2,485	13.37	13.1
Implicit Hate	Twitter	2	-	21,480	-	16.81
OLID	Twitter	2	13,240	860	19.62	23.16
ToxiGen	Synthetic	2	8,960	940	18.14	18.63
Table 10: SimCSE scores between each pair of datasets regarding the toxic detection task. AA: AbuseAnalyzer; AC: AdvCivil; CC: Civil Comments; HS: Hate Speech; IH: Implicit Hate; TG: ToxiGen.
Train	Test	AA	AC	CC	HS	HSOL	IH	OLID	TG
AbuseAnalyzer	100	80.05	79.60	78.89	73.97	75.78	85.33	63.06
AdvCivil	80.05	100	95.17	59.07	51.35	69.54	74.49	58.63
Civil Comments	79.60	95.17	100	66.89	50.96	69.05	76.20	62.09
Hate Speech	78.89	59.07	66.89	100	64.96	76.02	70.03	74.00
HSOL	73.97	51.35	50.96	64.96	100	41.73	69.80	43.32
Implicit Hate	75.78	69.54	69.05	76.02	41.73	100	64.60	79.79
OLID	85.33	74.49	76.20	70.03	69.80	64.60	100	51.76
ToxiGen	63.06	58.63	62.09	74.00	43.32	79.79	51.76	100
Table 11: The OOD performance of the T5-large when trained on the Civil Comments dataset. AA: AbuseAnalyzer; AC: AdvCivil; CC: Civil Comments; HS: Hate Speech; IH: Implicit Hate; TG: ToxiGen.
Train	Test	AA	AC	CC	HS	HSOL	IH	OLID	TG
Civil Comments	74.41	57.47	87.15	80.43	75.98	63.77	85.00	68.83
SQuAD can be used in the original format, while other datasets i.e. HotpotQA, NaturalQuestions, NewsQA, SearchQA, and Trivia-QA, are adopted from MRQA to fit the extractive setting.
D Dataset Selection for Other Tasks
In section 2.3, we have taken the task of sentiment analysis as an example to demonstrate how to select ID and OOD datasets for our benchmark. Next, we will explain how we choose datasets for other tasks.
D.1 Toxic Detection
Candidate Datasets. We use the same approach for searching toxic detection datasets as we do for sentiment analysis. We gather several dataset candidates, including Civil Comments, Hate Speech, HSOL, Implicit Hate, OLID, and ToxiGen. We aim to adopt an adversarial dataset for each NLU task, which is lacking in the current literature. Hence, we later construct a new dataset through adversarial attacks on the chosen ID dataset.
Probing Experiments. The setup is the same as sentiment analysis.
Results. The dataset information can be found in Tables 9. The sources are very diverse, with each source, except Twitter, corresponding to only one dataset. For the ID dataset, Civil Comments is significantly larger than all the other datasets, and it is the only one containing more than 10k samples for each class. In addition, Civil Comments contain 6 subtypes of toxicity and 24 types of targeted identity, meeting the requirement for dataset diversity. Considering the dataset size and the text source diversity, we choose Civil Comments as our ID dataset for toxic detection.
Table 12: Statistics of natural language inference candidate datasets.
Dataset	Source	# Classes	# Samples	Avg. Length (p)	Avg. Length (h)
Train	Test	Train	Test	Train	Test
ANLI	Adversarial	3	162,865	3,200	54.13	54.42	9.6	10.22
BioNLI	Biomedical	2	5,544	68,243	215.93	217.93	26.91	29.65
Wall Street Journal & Switchboard
CB	British National Corpus & Switchboard	3	-	306	-	56.4	-	7.55
ContractNLI	Legal	3	7,191	2,091	1673.63	1708.82	12.82	12.82
DocNLI	News & Wikipedia	2	942,314	263,886	318.89	385.13	47.5	71.88
MNLI	Open American National Corpus	3	392,662	9,815	19.81	19.27	9.97	9.92
SNLI	Flickr	3	549,361	9,824	12.85	13.91	7.42	7.48
WANLI	Synthetic	3	102,884	5,000	17.49	17.48	9.93	9.83
Table 13: SimCSE scores between each pair of datasets regarding the natural language inference task.
Train	Test	ANLI	BioNLI	CB	ContractNLI	DocNLI	MNLI	SNLI	WANLI
ANLI	100	31.32	29.37	21.99	53.79	16.27	21.99	22.41
BioNLI	31.32	100	5.51	10.39	28.07	2.28	-4.78	-0.12
CB	29.37	5.51	100	17.81	40.95	68.11	41.88	59.17
ContractNLI	21.99	10.39	17.81	100	19.96	3.62	-7.17	6.12
DocNLI	53.79	28.07	40.95	19.96	100	34.91	21.23	28.00
MNLI	16.27	2.28	68.11	3.62	34.91	100	47.42	85.23
SNLI	21.99	-4.78	41.88	-7.17	21.23	47.42	100	46.80
WANLI	22.41	-0.12	59.17	6.12	28.00	85.23	46.80	100
Table 14: The OOD performance of the T5-large when trained on the MNLI dataset.
Train	Test	ANLI	BioNLI	CB	ContractNLI	DocNLI	MNLI	SNLI	WANLI
MNLI	36.19	77.63	63.40	37.06	75.67	89.40	87.32	63.32
Next, we consider OOD datasets from other text sources. As aforementioned, we first supplement an adversarial dataset for toxic detection, i.e. AdvCivil, based on Civil Comments. The construction details can be found in Appendix C.1. Then we consider the semantic similarity among the dataset candidates. From the results in Table 10, we can filter out AbuseAnalyzer and OLID because of their high similarity with Civil Comments. We also observe a high similarity between AdvCivil and Civil Comments, but we do not disregard the former dataset at this stage since the high similarity has an acceptable attribution considering the construction process. Moreover, both HSOL and Implicit Hate come from tweets, thus they can not coexist in the benchmark. We are leaning to select Implicit Hate since it contains more challenging implicit toxicity.
Thereafter, we have four datasets left, i.e. AdvCivil, Hate Speech, Implicit Hate, and ToxiGen. We still need to drop one dataset based on performance degradation. According to results in Table 11, AdvCivil, Implicit Hate, and ToxiGen can provoke a performance drop of over 20 points. By contrast, the ID model can still achieve accuracy over 80 on Hate Speech, indicating that this shift may be the least challenging. Therefore, we discard Hate Speech and adopt the other three as the OOD datasets. It is also worth noting that Implicit Hate leads to a much more severe performance drop than HSOL, supporting our previous claim. Finally, the distribution shift for toxic detection is Civil Comments  $\rightarrow$  (AdvCivil, Implicit Hate, ToxiGen).
D.2 Natural Language Inference
Candidate Datasets. We investigate datasets on Paperswithcode and finally include ANLI, BioNLI, CB, ContractNLI, DocNLI, MNLI, SNLI, and WANLI as the candidates.
Probing Experiments. For semantic similarity evaluation, we only feed premises in each dataset to the unsupervised SimCSE model<sup>7</sup>, instead of concatenating the premises and hypotheses together. Note that we do not adopt the supervised SimCSE model here, because it was contrastively trained
<sup>7</sup>https://huggingface.co/princeton-nlp/unsup-simcse-roberta-large
Table 15: Statistics of named entity recognition candidate datasets.
Dataset	Source	# Classes	# Samples	Avg. Length
Train	Test	Train	Test
CoNLL	Reuters	4	14,042	3,454	14.50	13.44
CrossNER	Wikipedia	7	701	2,507	38.46	38.22
E-NER	Legal	4	-	11,692	-	34.52
Few-NERD	Wikipedia	8	131,768	37,649	24.49	24.47
WNUT	YouTube & Twitter & StackExchange & Reddit	6	3,395	1,288	18.48	18.16
Table 16: SimCSE scores between each pair of datasets regarding the name entity recognition task.
Train | Test	CoNLL	CrossNER	E-NER	Few-NERD	WNUT
CoNLL	100	62.86	43.48	73.76	45.60
CrossNER	62.86	100	37.19	87.47	44.84
E-NER	43.48	37.19	100	42.29	25.85
Few-NERD	73.76	87.47	42.29	100	45.74
WNUT	45.60	44.84	25.85	45.74	100
on NLI datasets, thus may lead to a bias in evaluating the semantic similarity between NLI datasets. For cross-evaluation experiments, the setup is the same as semantic analysis.
Results. Dataset information is shown in Tables 12. These NLI datasets are commonly large and their text sources do not overlap. Based on these results, we start to pick the ID dataset. First of all, BioNLI and DocNLI should be excluded from the discussion of ID dataset selection, because they only have two classes but all the other candidates have three categories. Otherwise, the trained ID model may fail to distinguish the class Neutral and Contradiction, namely suffering from a label shift [43]. Then, consider the dataset size. The majority of candidates contain over 100k training samples, while CB only has about 300 samples in total and ContractNLI has less than 10k samples, much smaller than others. Therefore, these two datasets should be excluded too. Among the remaining datasets, ANLI and WANLI mainly contain adversarial and challenging features, instead of general ones; MNLI claims to be more diverse than SNLI since it consists of both written and spoken English and contains 10 sub-genres, thus more abundant in text styles and topics. Hence, we pick MNLI as the ID dataset.
For OOD datasets, since there is no dataset drawn from the same source with MNLI, only semantic similarity and performance drop should be involved. From Table 13, the similarities between NLI datasets and MNLI are all relatively low, except for WANLI, CB, and SNLI. We will ignore CB and SNLI in the later procedure but still remain WANLI because WANLI is based on MNLI and thus it is reasonable to be somehow similar to MNLI. Thereafter, we examine the performance drop caused by each dataset. As shown in Table 14, the ID model presents little performance degradation on BioNLI and DocNLI, while suffering significant degradation on the other three datasets. Hence, the distribution shifts for NLI will be  $\mathrm{MNLI} \rightarrow (\mathrm{ANLI}, \mathrm{ContractNLI}, \mathrm{WANLI})$ .
D.3 Name Entity Recognition
Candidate Datasets. Since NER datasets typically have different sets of entity type labels that require specific domain knowledge, we loosen the search standard to include the datasets that have partially overlapping entity type labels, rather than requiring an exact match. We find five suitable candidate datasets, i.e. CoNLL, CrossNER, E-NER, Few-NERD, and WNUT. To align their label sets, we process them to be consistent with Few-NERD since we consider the label set of Few-NERD to be the most general.
Probing Experiments. The setup for semantic similarity evaluation follows the way we do for sentiment analysis, but the backbone model used in the performance evaluation here is a DeBERTa-base model, instead of a T5-base. The reason is that T5 requires inputs to be organized with prompts while standard prompt-based tuning methods for NER are still lacking. Hence, T5 does not perform well on NER task, so we resort to fine-tuning the encoder-only model, DeBERTa, as the alternative.
Table 17: The OOD performance of the DeBERTa-large when trained on the Few-NERD dataset.
Train | Test | CoNLL	CrossNER	E-NER	Few-NERD	WNUT
Few-NERD	69.10	66.63	48.01	79.89	45.45