SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 18: Statistics of extractive question answering candidate datasets. AQA: AdvQA; HQA: HotpotQA; NQ: NaturalQuestion; SQA: SearchQA; SS: SQuAD Shifts; TQA: Trivia-QA.
Dataset	Source	# Samples	Avg. Length
Train	Test	Train	Test
AdvQA	Adversarial	30,000	3,000	117.36	114.83
HotpotQA	Wikipedia	72,928	5,901	153.56	193.21
NaturalQuestion	Google search	104,071	128,36	152.03	158.8
NewsQA	CNN News	74,160	42,12	495.8	492.59
SearchQA	Google search & J!Archive	117,384	16,980	646.87	642.87
SQuAD	Wikipedia	87,599	10,570	119.76	123.95
SQuAD Shifts	Wikipedia & Amazon & Reddit & New York Times	-	29,753	-	139.1
Trivia-QA	Wikipedia	61,688	7,785	673.75	672.64
Table 19: SimCSE scores between each pair of datasets regarding the extractive question answering task. AQA: AdvQA; HQA: HotpotQA; NQ: NaturalQuestion; SQA: SearchQA; SS: SQuAD Shifts; TQA: Trivia-QA.
Train	Test	AQA	HQA	NQ	NQA	SQA	SQuAD	SS	TQA
AdvQA	100	66.85	67.66	46.74	62.72	96.29	43.05	69.62
HotpotQA	66.85	100	81.25	65.53	83.99	72.99	39.89	87.81
NaturalQuestions	67.66	81.25	100	54.67	79.26	73.5	37.06	80.67
NewsQA	46.74	65.53	54.67	100	66.94	51.98	35.94	64.14
SearchQA	62.72	83.99	79.26	66.94	100	69.22	48.19	94.17
SQuAD	96.29	72.99	73.5	51.98	69.22	100	42.83	75.72
SQuAD Shifts	43.05	39.89	37.06	35.94	48.19	42.83	100	48.12
Trivia-QA	69.62	87.81	80.67	64.14	94.17	75.72	48.12	100
Table 20: The OOD performance of the T5-large when trained on the SQuAD dataset.
Train	Test	AQA	HQA	NQ	NQA	SQA	SQuAD	SS	TQA
SQuAD	51.19	74.08	64.54	63.77	37.47	93.14	89.02	73.15
Results. The dataset statistics are shown in Table 15. Few-NERD is much larger than other datasets and contains the largest variety of samples, with 8 coarse-grained and 66 fine-grained types. Therefore, we select it as the ID dataset for the name entity recognition task.
For OOD datasets, since CrossNER shares the same source with the ID dataset Few-NERD, it should not be adopted to our benchmark. The results in Tables 16 and 17 supports this claim from the perspectives of semantic similarity and performance drop, respectively. Then, the remaining three datasets will be used to construct the distribution shifts, i.e., Few-NERD  $\rightarrow$  (CoNLL, E-NER, WNUT).
D.4 Extractive Question Answering
Candidate Datasets. We consider AdvQA, HotpotQA, NaturalQuestions, NewsQA, SearchQA, SQuAD, SQuAD Shifts, and Trivia-QA as the candidates.
Probing Experiments. In the experiment of semantics similarity, we take the passages as inputs for the supervised SimCSE model. In the experiment of performance evaluation, we choose T5-base as the backbone model and measure F1 scores.
Results. Table 18 shows the dataset statistics. Except for SQuAD Shifts, which does not have a training split, all the other datasets are large in size. However, considering the diversity, we assume that Wikipedia provides the most extensive knowledge base. Hence, we restrict the scope of the ID dataset to those created from Wikipedia. Among the three datasets, HotpotQA, SQuAD, and Trivia-QA, SQuAD is the largest one. and therefore, we adopt SQuAD as the ID dataset.
Then, since SQuAD has been selected to be the ID dataset, HotpotQA and Trivia-QA are considered in the OOD dataset selection. And besides these two datasets, NaturalQuestions also have a high semantic similarity with SQuAD, as shown in Table 19. Therefore, for the distinction of the distributions, NaturalQuestions will not be taken into account either. In contrast, despite a high semantic similarity with SQuAD, we do not exclude AdvQA for this reason. This is because AdvQA is constructed by human adversaries based on SQuAD, aiming to reveal the weakness of models trained on SQuAD. Therefore, we consider the high similarity between AdvQA and SQuAD reasonable and should not disregard AdvQA for this sake. Next, we evaluate the ID model on each candidate dataset to examine the challenge of each distribution shift. From Tabel 20, we can observe that the ID model performs poorly on AdvQA, NewsQA, and SearchQA while maintaining a high F1 score on SQuAD Shifts, indicating that the shift to SQuAD Shifts is the least challenging. Eventually, we establish the distribution shifts for extractive question answering as SQuAD  $\rightarrow$  (AdvQA, NewsQA, SearchQA).
E Additional Descriptions and Results of the Analysis and LLMs Evaluation
E.1 Correlation between ID and OOD performance
E.1.1 Experimental Setting
To measure the correlation between ID and OOD performance, we train models with various ID performances by manipulating model scales, training steps, available training samples, and tunable parameters following [14]. The details are introduced below.
Model scale. We adopt the small, base, and large versions of the T5 and DeBERTa models.
Available training samples. For Sentiment Analysis, Toxic Detection, and Natural Language Inference, we randomly sample K samples for each class. For Name Entity Recognition, we adopt the N-way K-shot sampling strategy proposed by [26], ensuring that entities of all N types occur  $\mathrm{K}\sim 2\mathrm{K}$  times in the sampled dataset. For Extractive Question Answering, we randomly sample K questions. In all experiments, we run five repeated experiments for each K.
Number of tunable parameters. We employ parameter-efficient tuning methods to regulate the tuned parameter for adapting to downstream tasks. We prioritize adopting the strong approach Adapter [41]. However, in experiments, we find that Extractive Question Answering models can achieve a high ID performance even with a very insignificant amount of parameters being tuned, thus presenting no performance change when the number of tunable parameters increases. Therefore, we turn to an alternative, Soft-prompt [54], for a clearer observation.
Training steps. We evaluate model performance at epoch  $\{0, 0.1, \dots, 0.9\} \cup \{1.0, \dots, \mathrm{EPOCH}\}$ , where EPOCH is the total number of training epochs.
E.1.2 Full Results
We present the complete results of ID-OOD correlation in Figure 4. Note that due to the variance of data points, we manually check and remove outlier points to make the fitting results clearer. Next, we will illustrate the results task by task.
Sentiment Analysis. All figures of SA comprise two straight lines. The smoother line, which mainly fits the results of T5-small models with poor ID performances, stretches from around ID accuracy  $= 60$ . Meanwhile, the steeper line accommodates T5-base and T5-large model results and begins around ID accuracy  $= 85$ . Despite the different slopes, both lines represent a linear positive correlation between ID performance and OOD generalization, thus we still categorize this kind of relation as Type I.
Toxic Detection. This task encompasses two types of ID-OOD relations. AdvCivil stands as an outlier as aforementioned, whereas Implicit Hate and ToxiGen fall under the category of Type I relation, characterized by a straight line that intersects the diagonal line  $y = x$  with a flatter slope. This trend indicates that although achieving initially higher performance on OOD examples than on ID examples, the model's performance gap between ID and OOD decreases and eventually reverses as training proceeds
Natural Language Inference. All relations are classified as Type III, a non-monotonic correlation. Each graph has a V-shaped curve, consisting of two straight lines with different slopes. Specifically, for ANLI, the first straight line has a shallower slope than the second one, while ContractNLI and WANLI exhibit the opposite trend.
Named Entity Recognition. The ID-OOD relations in this task are all typical Type I, with all fitted lines lying below the diagonal line. The CoNLL dataset exhibits the highest correlation to a perfectly linear relationship  $(y = x)$ , while the WNUT dataset has the lowest correlation. This phenomenon indicates that improving OOD performance on CoNLL is the easiest while on WNUT is the hardest.
Extractive Question Answering. Type II relation is observed in this task, with varying degrees of curve changes. Specifically, in the SearchQA dataset, the angle of the curve change is the most pronounced, while in the NewsQA dataset it is the smoothest.
E.2 LLMs Evaluation
Experiment Setting. The templates and instructions we used are shown in Tables 21 and 22. We take Dynasant as an example to demonstrate the in-context learning settings for LLMs. The in-context (ID) setting includes exemplars drawn from the ID training dataset of the sentiment analysis task, i.e. training dataset of Amazon. However, the in-context (OOD) setting samples the in-context exemplars from the training split of the OOD dataset, i.e. training dataset of Dynasant. Note, however, that the latter paradigm may not be applicable to all datasets. All ID datasets are excluded, as well as OOD datasets without a training split. Besides, those datasets whose text length is too long on average would not be taken into account either, as the contexts could exceed the context window of LLMs.
Table 21: Templates for T0-3B and T5-series models.
Task	Template	Verbalizer
SA	{input} All in all, the sentiment was {"mask"}.	negative / positive / neutral
TD	{input} All in all, the toxicity was {"mask"}.	benign / toxic
NLI	Given the two sentences:
(1) {{input_1}}.
(2) {{input_2}}.
Does the first sentence entail the second? {"mask"}.	Yes / Maybe / No
NER	ner text: {input} label: {"mask"}	-
EQA	Extract the answer to the question from the following context.
Question: {{input_1}}
Context: {{input_2}}|l|
Answer: {"mask"}	-
Table 22: Instructions for LLMs, including LLaMA-7B, Davinci3, and Turbo.
Task	Instruction
SA	##### Instruction ###
Solve the sentiment analysis task. Options for sentiment: negative, positive, neutral. ###
##Format ###
Text: {{Text}} // Prediction: {{Prediction}} ###
##Input ###
Text: {{input}} // Prediction:
TD	####### Instruction ###
Solve the toxic detection task. Options for toxicity: benign, toxic. ###
##Format ###
Text: {{Text}} // Prediction: {{Prediction}} ###
##Input ###
Text: {{input}} // Prediction:
NLI	####### Instruction ###
Solve the NLI task. Options for entailment relationship: entailment, neutral, contradic-tion. ###
##Format ###
Premise: {{Premise}} // Hypothesis: {{Hypothesis}} // Prediction: {{Prediction}} ###
##Input ###
Premise: {{input_1}} // Hypothesis: {{input_2}} // Prediction:
NER	####### Instruction ###
Solve the NER task, identifying the Organization, Person, Location, Miscellaneous, Building, Art, Product, and Event entities from given text. ###
##Format ###
Text: {{Text}} // Entity: Organization: None || Person: Word1 || Location: Word6, Word7 || Miscellaneous: None || Building: None || Art: Word 3 || Product: None || Event: None.
##Input ###
Text: {{input}} // Entity:
EQA	####### Instruction ###
Solve the extractive question answering task. Referring to the passage below and extract answer for the question. The answer should be the shortest phrase as it can be. ###
##Format ###
Passage: {{Passage}} // Question: {{Question}} // Answer: {{Answer}}.
##Input ###
Passage: {{input_1}} // Question: {{input_2}} // Answer:
(a) Sentiment Analysis
(b) Toxic Detection
(c) Natural Language Inference
(d) Name Entity Recognition
(e) Extractive Question Answering
Figure 4: Complete results of the relation between ID and OOD performance.