How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study (2024)

Abstract: The advent of pre-trained Language Models (LMs) has markedly advanced natural language processing, but their efficacy in out-ofdistribution (OOD) scenarios remains a significant challenge (Hupkes et al., 2023). The field of computational argumentation (CA), modeling human argumentation processes, is notably impacted by these challenges because complex annotation schemes and high annotation costs naturally lead to resources barely covering the multiplicity of available text sources and topics. Due to this data scarcity, generalization to data from uncovered covariant distributions is a common challenge for CA tasks like stance detection or argument classification. This work systematically assesses LMs’ capabilities for such OOD scenarios. While previous work targets specific OOD types like topic shifts (Stab et al., 2018) or OOD uniformly (Yuan et al., 2023), we address three prevalent OOD scenarios in CA: topic shift, domain shift, and language shift. Our findings challenge the previously asserted general superiority of in-context learning (ICL) for OOD. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts with heavy label divergences between train and test data, prompt-based finetuning surpasses for shifts when semantic differences prevail, like topic shifts. Navigating the heterogeneity of OOD scenarios in CA, our work empirically underscores the potential of base-sized LMs to overcome these challenges. 1

heading: 1 Introduction
text: Argumentation as a communication tool for human reasoning has engaged researchers over millennia (Aristotle and Kennedy, ca. 350 B.C.E., translated 2007; Toulmin, 1960; Van Eemeren et al., 2019)
∗* Corresponding author andreas.waldis@live.com 1We provide data and code at online.
and has become an important research area in natural language processing under the umbrella of computational argumentation (Lippi and Torroni, 2016; Lauscher et al., 2022). Specifically, computational argumentation (CA) models human argumentative processes and leads to complex tasks such as stance detection (Mohammad et al., 2016) and argument quality evaluation (Toledo et al., 2019). However, developing resources for such CA tasks requires significant annotation efforts (Habernal and Gurevych, 2017; Schiller et al., 2022), which often inadequately capture the wide range of heterogeneity in available text sources and topics. This situation makes OOD scenarios, especially those involving significant covariant distribution shifts, a common challenge for CA tasks since LMs are anticipated to generalize across such shifts in current and future applications (Slonim et al., 2021). These shifts occur when input data distribution changes between the training and testing phrases and can be viewed as a specific aspect of out-of-distribution (OOD) scenarios (Zhang et al., 2020).
This work focuses on three types of covariant distribution shifts frequently encountered in CA tasks: topic shift, domain shift, and language shift.
14878
Figure 1 illustrates these three types of OOD scenarios, in which researchers aimed at developing systems for CA tasks that generalize across unseen topics (Shnarch et al., 2018; Toledo et al., 2019), text domains (Lauscher et al., 2020; Hardalov et al., 2021), or languages (Eger et al., 2018; Vamvas and Sennrich, 2020). These studies have observed that CA systems often fail to handle OOD scenarios.
Given this variety of OOD scenarios in CA and the need for data efficiency, this paper aims to answer the following research question: “how to handle different types of OOD scenarios in computational argumentation using LMs”? Most previous work on evaluating the generalization and robustness of NLP models has either predominantly focused on a single type of OOD scenario, such as domain shift (Blitzer et al., 2007; Hardalov et al., 2021), or on general OOD that does not distinguish among various types of OOD scenarios (Yuan et al., 2023). However, these studies overlook the heterogeneous nature of OOD and, thereby, limit the transferability of the corresponding findings to the spectrum of shift types in CA tasks. This study introduces a detailed evaluation framework encompassing a list of holistic performance measures (§ 3) to pinpoint crucial generalization flaws such as misalignment between performance and training loss in models. In addition, we feature a heterogeneous collection of eleven CA tasks (§ 4) covering three types of OOD scenarios. We evaluate these tasks with an extensive experimental setup (§ 5) covering twelve LMs of various sizes and eight learning paradigms, including gradient-based learning like vanilla fine-tuning (FT) and prompt-based finetuning (P+FT), as well as in-context learning (ICL). From the observed results (§ 6), we conduct an in-depth analysis (§ 7) to gain a better understanding of how learning paradigms and LMs differ for different types of OOD for computational argumentation.
In contrast to Yuan et al. (2023), suggesting in-context learning (ICL) surpasses fine-tuning LMs for addressing general OOD, we find different learning paradigms excel in different types of OOD for CA tasks. In particular, ICL outperforms gradient-based learning for domain shifts where train and test label distributions heavily differ. However, gradient-based learning surpasses ICL for topic shifts characterized by a clear semantic divergence in the covered topics between the training and testing datasets.
In summary, our main contributions are:
1. We propose an evaluation framework covering eleven CA tasks across three types of OOD scenarios. Along with a comprehensive assessment of LM’s OOD capabilities, it provides a clear picture of the generalization challenges in CA and offers guidance to practitioners in tackling these challenges.
2. Results of extensive experiments offer valuable insights and show that different learning paradigms effectively manage OOD scenarios for CA under different conditions. Particularly, in-context learning should be preferred for domain shifts, while gradient-based learning is the first choice for generalization across semantic differences (topic shifts).
3. We shed light on the unused potential of basesized LMs for OOD scenarios. We demonstrate that training a fraction of the parameters of base-sized LMs with LoRA achieves performance comparable to full LM tuning, and such parameter-efficient training offers better stability than larger LMs.
4. This work emphasizes the critical role of OOD heterogeneity in tackling generalization challenges within CA tasks. This paves the way for future research to conduct detailed and targeted examinations of OOD scenarios in other research areas.

heading: 2 Related Work
text: Out-of-Distribution Generalization Studies in NLP target OOD generalization from different perspectives, focusing on the robustness of LMs (Hendrycks et al., 2019; Jin et al., 2020; Zhou et al., 2020; Wang et al., 2021) or OOD detection (Koner et al., 2021; Cho et al., 2023). Similar to computer vision (Tseng et al., 2020), NLP studies primarily focus on considering covariant distribution shifts (Zhang et al., 2020) and analyze single types of them in isolation, such as domain across datasets (Hardalov et al., 2021; Yang et al., 2023; Yuan et al., 2023), language (K et al., 2020; Conneau et al., 2020a), topic (Stab et al., 2018; Allaway and McKeown, 2020). This shortage of comprehensively analyzing OOD hinders analytical or methodological advancements in a challenging field such as computational argumentation since generalization of methods is limited when relying on shift-specific
features (Liang et al., 2022; Xu et al., 2018; Peng et al., 2018; Rietzler et al., 2020).
Prompt-based Fine-tuning Commonly, pretrained LMs are fine-tuned by providing a natural language input and optimizing regarding an arbitrary label (Devlin et al., 2019). Instead, promptbased fine-tuning (Liu et al., 2021a) (P+FT) allows relying upon acquired competencies during pretraining, both for encoding the input and predicting the label by formulating the task as a cloze test. This procedure allows LMs to reach comparable performance to their large-sized counterparts (Schick and Schütze, 2021a,b) with the same limited data as in few-shot settings. Despite their success for few-shot scenarios, little work analyzed how P+FT generalizes differently than FT or how it performs considering complete datasets, exceptionally Raman et al. (2023) showed the robustness of P+FT against adversarial attacks.
With this work (Figure 2), we address the need to comprehensively evaluate OOD abilities of LMs with a particular focus on computational argumentation. In particular, we assess a variety of LMs using in-context and gradient-based learning paradigms, considering three types of OOD scenarios covering eleven different tasks.

heading: 3 Methodology

heading: 3.1 OOD Types
text: We distinguish between two generalization scenarios: in-distribution (ID) and out-of-distribution (OOD) generalization. While ID assumes train and test data being independent and identically distributed, OOD accounts for practical challenges where we expect apparent distribution shifts between the training and testing instances (Shen et al.,
2021). To capture the success of a classifier f(y|x) in such scenarios, we measure its ability to transfer learning from train instances Xtrain to test instances Xtest. However, OOD potentially introduces covariant, label, and concept shifts between train and test data (Zhang et al., 2020). In this work, we focus on three types of covariant shifts (topic shift, domain shift, and language shift), as illustrated in Figure 1, due to their frequent prevalence in computational argumentation tasks.

heading: 3.2 OOD Evaluation Protocol
text: Generalization success is typically measured with single task metrics, like the F1 macro score. However, solely relying on one metric ignores known stability issues, such as apparent deviations regarding randomness (Mosbach et al., 2021). Thus, we compose a set of three requirements that a superior learning model should fulfill: good task performance (Applicability), better alignment between optimization and evaluation (Reliability), and Stability regarding data and randomness. We ground this evaluation for a specific task on a given set of runs (r ∈ R), trained for one distinct fold and seed over a number of epochs (e ∈ E). Note that we formalize these requirements with OOD classification in mind and, therefore, rely on F1 macro score as the reference metric. However, these requirements can be generalized to other types of OOD tasks using the corresponding reference metrics, such as ROUGE for text generation.
Applicability captures the task-specific performance. Specifically, we measure the average taskspecific metric, here F1 macro score (µF1), across all runs r covering different folds and seeds.
Reliability requires that the learning process (optimization objective) is reflected in the obtained task performance. We evaluate the model using the development dataset that embodies the same OOD type as the training dataset. Specifically, we approximate, after each epoch e of a run r, learning as the loss β = {∀e ∈ E(r)|fdevloss(e)} and performance using task metric (F1 macro) γ = {∀e ∈ E(r)|fdevF1 (e)}. Then, we calculate the Kendall correlation between β and γ and average it for every r as µτ . Ideally, we expect a negative correlation (τ = −1), indicating that improvements in learning are reflected in better performance and vice-versa. However, since we determine final labels using the argmax operation,
dev loss and performance can increase simultaneously. For example, while predicting the same class (ŷ = c0) the class probabilities can change from (95%, 5%) to (90%, 10%). At the same time, the cross-entropy changes from 0.074 to 0.15. Therefore, we assume the model is becoming less sure about the prediction. This aspect is particularly relevant for OOD generalization, where overfitting to distributional properties of training data, such as unique vocabulary, likely introduces uncertainty during inference.
Stability demands a low impact from varying data and randomness on both Applicability and Reliability. As recommended by Reimers and Gurevych (2017), we measure the standard deviation of σF1 and στ across R runs covering different data folds and seeds.

heading: 4 CA Tasks Across OOD Types
text: In this section, we present the selection of computational argumentation tasks (§ 4.1) and subsequently show their heterogeneous distribution shifts, focusing on covariant and label properties (§ 4.2).

heading: 4.1 Task Selection
text: We choose eleven tasks from computational argumentation and related fields (Stede, 2020) that inherent OOD as a fundamental challenge. We broadly categorize them according to their targeted covariant distribution shift, either topic, domain, or language. For example, domain for stance detection across datasets (Hardalov et al., 2021), sentiment analysis across languages (Prettenhofer and Stein, 2010), or argument quality across topics Toledo et al. (2019). Figure 2 compares our study with previous research in terms of the number of tasks and the range of OOD types covered. Below we briefly describe each task:
Argument Quality (arg-qua) Toledo et al. (2019) analyzed 9,100 argument pairs across 22 topics to determine which one has higher quality.
Argument Similarity (arg-sim) Reimers et al. (2019) annotated 3,595 arguments pairs of 28 topics to decide whether they are similar or not.
Argument Classification (arg-cls) Stab et al. (2018) annotated the stance of arguments (pro, con, neutral) regarding one of eight topics.
Evidence Classification (evi-cls) Shnarch et al. (2018) presented 5,785 sentences annotated as relevant or not for one out of 118 topics.
Sentiment Classification (review) Blitzer et al. (2007) annotated 8,000 reviews as positive or negative for four domains (Amazon product groups).
Multi-Dataset Stance Detection (stance) Following Hardalov et al. (2021), we use the semeval (Mohammad et al., 2016), emergent (Ferreira and Vlachos, 2016), and iac dataset (Walker et al., 2012) to evaluate stance detection across three domains (social media, news, and debating). All of them provide the same labels (pro, con, neutral).
Multi-Dataset Entailment (entail) Following Yang et al. (2023), we consider three medium-sized datasets (rte (Wang et al., 2018), SciTail (Khot et al., 2018), hans (McCoy et al., 2019)) to evaluate textual-entailment across three domains.
Multi-Lingual Stance Detection (x-stance) This dataset (Vamvas and Sennrich, 2020) includes 63,000 multilingual comments (de, fr, it) annotated as favor or against regarding 12 topics.
Multi-Lingual Sentiment Classification (xreview) Prettenhofer and Stein (2010) presents a set of 43,000 positive or negative reviews covering four languages (de, en, fr, jp) and three domains (Amazon product groups).
While the first seven English-only datasets mentioned above, include annotations for one considered shift (topic or domain), the selected multilingual datasets come with multiple such annotations. This enables formulating four OOD tasks from two datasets addressing two shift types each: language and domain shifts for x-review and topic and language shifts for x-stance.

heading: 4.2 Tasks Characteristics
text: In this subsection, we delve into the nature of the distribution shifts embodied by the selected tasks.
Shift Characteristics We focus on covariant properties of the input x (such as semantics) and the label y to describe the characteristic of distribution shifts between training and testing instances. Table 1 show these properties, with higher values denoting increased challenge levels. First, we assess the separability of train and test instances based on their semantic representation. Following Sun
et al. (2022), we embed2 all instances and apply kmeans clustering (Lloyd, 1982; MacQueen, 1967) to form two clusters. The alignment of these clusters with the train/test split is measured using the adjusted Rand index (Hubert and Arabie, 1985). A higher score suggests a more pronounced semantic shift between train and test sets. Subsequently, we examine biases in surface-level text features introduced during training by calculating differences in average readability (Flesch, 1948) and word count (∆ Flesch, ∆ Words) between training and testing instances. Furthermore, we evaluate distributional disparities in class labels using Kullback-Leibler (KL) divergence (Boyd and Vandenberghe, 2004). Higher KL values indicate more pronounced imbalances, complicating the task, as LMs often develop biases towards the training label distribution.
Task Difficulties Drawing from the above analyses, we categorize tasks into distinct groups. argqua, arg-sim, and stance demonstrate high semantic differences, with separability scores ranging between 75.8 and 86.7. Tasks like review, stance, entail, and x-review present surface-level challenges due to varying readability (∆ Flesch) and text lengths (∆ Word Count). Additionally, tasks such as evi-cls, stance, and entail show notable label distribution imbalances, reflected in high KL divergence values, thereby adding further complexity. Notably, stance emerges as particularly challenging, exhibiting distinct semantic, surface form, and label differences between training and testing instances, coupled with significant divergence from the LMs’ pre-trained text understanding.
2Following Reimers and Gurevych (2019), we use paraphrase-multilingual-mpnet-base-v2 for embedding.

heading: 5 Experimental Setup
text: This section outlines the experimental setup covering the models, learning paradigms, and evaluation.
Models We primarily experiment with basesized LMs, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and DeBERTa-v3 (He et al., 2021b) and their multilingual counterparts (Devlin et al., 2019; Conneau et al., 2020b; He et al., 2021b). For additional experiments, we consider base-sized version of ALBERT (Lan et al., 2020), DeBERTa (He et al., 2021a), ELECTRA (Clark et al., 2020), and 3B version of T5 (Raffel et al., 2020) and FLAN-T5 (Chung et al., 2022). Further we consider GPT-3.5 (Ouyang et al., 2022), Llama-2-Chat (70B) (Touvron et al., 2023), and Orca-2 (13B) (Mitra et al., 2023).
Learning Paradigms We assess the generalization capabilities of LMs under various learning paradigms. This includes vanilla fine-tuning (FT), prompt-based fine-tuning (P+FT), and in-context learning (ICL). Further, we consider linear probing (LP) and cloze prompting (P) as lower bounds to capture the LM’s pre-trained capabilities. In LP and FT, we train task-specific classification heads, in which the LM remains either frozen (LP) or trainable (FT)3. For P and P+FT, we embed the input into a cloze and let the pre-trained MLM head to predict the masking token and keep the LM frozen (P) or trainable (P+FT). In addition, we verify scaling gradient-based methods to bigger LMs using parameter efficient methods, including LoRA (Hu et al., 2022), P-Tuning (Liu et al., 2021b), and Prompt-Tuning (Lester et al., 2021). Finally, using ICL, we verify the capabilities of large LMs with task-specific instructions and demonstrations. Appendix § A.5 and § A.6 provide more details about these learning paradigms.
Evaluation We enforce distribution shifts for OOD evaluation by composing train/dev/test splits, including instances with distinct distributional properties, such as unique topics or text domains (Figure 1). We utilize multi-fold cross-validation (CV) to account for data variability and ensure each distinct distributional property (like a unique topic) is tested precisely once 4. We evaluate all tasks on all learning paradigms, taking LP and P as a
3We use [SEP] to concatenate the input with its topic, if available
4See Appendix § A.3 for more details.
lower bound and ID fine-tuning (FT-ID) as an upper bound. We assess every task using three random seeds to account for randomness. Using these runs, we employ comprehensive performance measurement including average Stability (µF1), Reliability (µτ ), and the Stability (σF1 ,στ ) - as previously defined in § 3.2.

heading: 6 Results
text: This section reports results on a detailed (Table 2) and aggregated level (Figure 3) and discusses six key findings.
i) Generalization flaws and the efficacy of prompt-based fine-tuning. The aggregation of the comprehensive evaluation (Figure 3) reveals crucial generalization flaws of OOD fine-tuning (blue). Compared to ID fine-tuning (red), it provides a lower Applicability (F1 score), optimization (loss) and performance (F1 score) are less aligned (lower Reliability), and measurements are less stable across different seeds and folds (Stability). In particular, we see this misalignment of loss and performance - a violation of a fundamental generalization assumption - crucially affects vanilla fine-tuning’s (FT) degraded OOD generalization capabilities. Turning to prompt-based finetuning (P+FT, green), it partially overcomes these flaws. Paired with DeBERTa-v3 and RoBERTa, it achieves higher absolute performance (Applicability), a better Reliability, and fewer deviations regarding data and randomness (Stability).
ii) Superiority of DeBERTa-v3. Next, we focus on the detailed results (Table 2) to compare the different LMs. Overall, we note the superior performance of DeBERTa-v3 compared to RoBERTa and BERT for all learning paradigms across all tasks.5 In particular, when paired with promptbased fine-tuning (P+FT), DeBERTa-v3 provides 3.3 better Applicability (µF1), 2.9 better Reliability (µτ ), and similar Stability with −0.4 (σF1) and +0.3 (στ ) than RoBERTa with P+FT. Moreover, we see DeBERTa-v3 with P+FT outperforms FT in ten out of eleven tasks and reaches ID performance for two tasks (arg-sim and review).
iii) Label differences cause significant generalization gaps. Table 2 reveals significant generalization gaps between OOD (FT and P+FT) and ID
5These findings extend to ID scenarios — see Appendix § B.1.
results (FT-ID) for stance and entail. These difficulties correlate with their previously identified label differences between train and test instances based on their high KL divergences (Table 1). These generalization issues are also visible when we compare linear probing (LP) and cloze prompting (P) for stance and entail with other tasks. Since these two paradigms largely evaluate the pre-trained capabilities of LMs, we expect a big gap between them and full LM tuning paradigms (FT and P+FT) when they are done without significant generalization flaws. However, this gap is smaller for stance and entail than for other tasks, indicating that PF and P+FT exhibit higher generalization problems for these two tasks. Still, we see again that P+FT partially overcomes such generalization flaws and provides, paired with DeBERTa-v3, improvements of 3.4 (stance) and 4.6 (entail) compared to FT.
iv) Pre-training influences the success of promptbased fine-tuning. Next, we compare the gap between vanilla fine-tuning (FT) and prompt-based fine-tuning (P+FT) for three additional base-sized LMs to better understand the efficacy of DeBERTav3 paired with P+FT. In particular, we focus on its design properties like token-only pre-training objective, disentangled attention (DA), ELECTRAstyle training, and extensive vocabulary. To determine which design choice of DeBERTa-v3 has the greatest impact on its superior OOD performance, we evaluate additional LMs on the English-
only tasks involving topic and domain shifts to test these properties (Figure 4). First, we found that DeBERTa(-v3), RoBERTa, and ELECTRA benefit more from prompt-based fine-tuning when pretrained with token-only objectives (like masked language modeling or replaced token detection). In contrast, LMs such as BERT or ALBERT, trained with additional sentence objectives like next sentence prediction or sentence order prediction, exhibit minor gains or perform worse with P+FT than FT. Second, we do not find DeBERTa-v3 gains from ELECTRA-style pre-training, as the FT vs. P+FT gap is more pronounced for DeBERTa than ELECTRA itself. Third, DeBERTa (with DA) performs better than RoBERTa (without DA) on both FT and P+FT. DeBERTa’s disentangled attention (DA) mechanism impacts its superior OOD performance since both models are pre-trained on the same datasets with masked language modeling. Finally, we see DeBERTa-v3’s extensive vocabulary (120k tokens) as another factor in its success, as it outperforms its ancestor (DeBERTa) with 50k tokens. These results show how pre-training crucially shapes LMs differently beyond their performance on downstream applications (Wang et al., 2018). We see these insights to be well aligned with other work, particularly in the examination of how the internal representations of LMs vary among different pre-training setups (Waldis et al., 2024).
v) No free lunch for in-context learning or gradient-based methods. We show in Figure 5 results of evaluating English-only tasks using
in-context learning (ICL) with GPT-3.5 (turbo), Llama-2-chat (70B), and Orca-2 (13B).6 Comparing these LMs for average Applicability, notably Orca-2 (66.2) outperforms GPT-3.5 (64.9) and Llama-2 (60.4). We see this strongly related to the reasoning-oriented pre-training of Orca-2. Moreover, ICL does not reach the average performance level of the best gradient-based (FT and P+FT) approach based on DeBERTa-v3. However, we note the superiority of ICL in scenarios involving heavy domain shifts, particularly in cross-dataset tasks, such as stance and entail, where heavy differences between train and test label distribution (high KL divergence) cause substantial generalization flaws for gradient-based learning methods. These flaws are visible when comparing the gap between P and P+FT. Since we tune the LM for P+FT, we expect a significant gap for successful generalization. However, these gaps are relatively small for stance and entail - from Table 2 +6.5 for stance and +5.6 for entail with DeBERTa-v3 compared to +34.3 for arg-cls. In addition, there is no clear gain of using P+FT for the relatively easy and popular sentiment analysis task (review). Due to its popularity, we assume this task is well covered in the enormous pre-training corpus of large LMs - such as GPT-3.5. In contrast, we note the superiority of P+FT for topic shift scenarios, which predominantly involve challenges of a semantic nature and moderate label distribution differences.
6Please find details in the Appendix (§ A.6).
vi) Few parameters are enough for competitive performance and allow to scale to larger LMs. Next, we compare the performance of different parameter-efficient tuning strategies with full model tuning. As shown in rows two to four in Table 3, we see LoRA with r = 4 outperforms P-Tuning and Prompt-Tuning on most tasks. Further, it performs on par with full-parameter tuning (first row) regarding Applicability, provides better Reliability, but degraded Stability. Further experiments considering bigger LMs show that LoRA allows their efficient use for OOD scenarios. Precisely, the large version of DeBERTa-v3 with 300 million provides 1.7 higher Applicability and 2.6 better Reliability than the base version (86m). Simultaneously, this scaling effect does not continue. T5 or Flan-T5, with three billion parameters, seem to be generally more affected by random seeds and different folds (Stability) without apparent Applicability. From these observations, OOD fine-tuning still leaves a large potential of LMs unexploited, while larger LMs improve the performance but introduce new Stability flaws.

heading: 7 Analysis
text: Next, we discuss four aspects which differentiate learning paradigms based on arg-cls where we observe prominent differences.
i) The bias regarding surface features. We show in Figure 7 average word counts and input complexities of test instances for ID and OOD vanilla fine-tuning (FT-ID and FT) and promptbased fine-tuning with (P+FT) using DeBERTav3 and in-context learning (ICL) with Orca-2 and GPT-3.5. LMs predict shorter and more complex instances (higher Flesch score) more likely correct, and vice versa for wrong ones - compared to the dataset average (dashed line). However, P+FT exhibits less bias on surface correlations than FT and shows similar patterns as FT-ID, hinting at the superior abilities of P+FT. In contrast, ICL predictions, in particular of Orca-2, are less biased for both surface features. Still, deviations from the dataset average suggest fundamental bias in such features.
ii) P+FT provides more prediction confidence and relies less on surface features. From Table 4, P+FT provides higher average confidence (defined as the logit of the predicted label) than FT and a similar one as ID fine-tuning (FT-ID). Thus, we assume P+FT is less confused by the distribution shift, which is also visible in the lower correlation between confidence and surface features (Flesch score or word counts) than FT. For example, FT seems less confident when the input is longer and more complex.
iii) Prompt-based fine-tuning considers input differently. Next, we analyze how LMs attribute to the input tokens. We follow Kobayashi et al. (2020) and calculate the attribution of a token using the attention and the norm of the token embeddings. FT-ID and FT have higher average attributions than P+FT, indicating that token attributions are more evenly distributed since they are normalized using the Euclidean norm within a given input sentence. This is already visible when comparing the token attribution before fine-tuning (raw vs. P+raw). Apparent differences are also visible when we compare how attributions of correct or wrong predicted instances differ. While P+FT shows maximum 0.4 differences (P+FT), this rises to 1.0 for FT-ID. With these results, we assume LMs applied in promptbased or vanilla fine-tuning fundamentally differ in how inputs are processed.
iv) P+FT retains more semantic information. Figure 6 visualizes the layer-wise embeddings of the classification proxy tokens - CLS for FT and MASK for P+FT. It shows that P+FT retains more semantic information (about topics) until the last layers, while FT eliminates them across all layers during training. Hinting, again, at substantial differences between FT and P+FT.

heading: 8 Conclusion
text: This work marks the most extensive study to date addressing the heterogeneous types of OOD scenarios in CA by systematically evaluating different OOD types. We evaluate a multiplicity of LMs and learning paradigms on eleven CA tasks. With this extensive evaluation, we shed light on the challenges of having diverse covariant distribution shifts in CA. In addition, we provide takeaways of general relevance, such as the superiority of ICL for domain shifts, where gradient-based learning fails to generalize effectively due to significant label discrepancies between the training and testing data. In contrast, gradient-based learning surpasses ICL when generalization across significant semantic differences is required, like in cases of topic shifts. With the rise of larger LMs, systematic evaluation of distribution shifts becomes even more important, necessitating the consideration of additional factors such as computational efficiency, task complexity, and data contamination. Finally, our findings highlight the untapped potential of base-sized models, which points towards a need for further advancements in gradient-based learning paradigms.
Ethical Considerations and Limitations

heading: 8.1 Higher Input Length
text: By embedding the input into a prompt, we sacrifice potential input tokens. Since the used tasks have relatively short inputs, this is not crucial for this work. However, this can be an essential limitation for other tasks when inputs get longer.

heading: 8.2 Efficiency
text: We always refer to efficient fine-tuning when discussing efficient methods in this work. Therefore, we did not consider efficient methods to make inferences on larger LMs more feasible. We see this as another crucial and essential aspect of realworld applications. Simultaneously, we think performance and efficiency will alternate in the future. Therefore, we keep that for future work.

heading: 8.3 Large Language Models
text: We show the competitive performance of ChatGPT compared to gradient-based approaches by only relying on four demonstrations and without any tuning. Simultaneously, we need to assume that the pre-training corpus of ChatGPT leaks crucial aspects - like broadly covers controversially discussed topics like Nuclear Energy or includes instances of popular datasets (like RTE (Wang et al., 2018) or SemEval2016 (Mohammad et al., 2016)) word-by-word. When we have in mind that we use OOD to verify generalization capabilities required for upcoming scenarios, we need to examine the performance of ChatGPT carefully and whether it was able to learn the task or just remembered some semantic aspects of the pre-training.

heading: Acknowledgements
text: We thank Cecilia Liu, Thy Thy Tran, and Kexin Wang for their valuable feedback. Andreas Waldis has been funded by the Hasler Foundation Grant No. 21024. Yufang Hou is supported by the Visiting Female Professor Programme from TU Darmstadt.

heading: A Additional Details of the Experiments
text: A.1 Training Setup For all our experiments, we use NVIDIA RTX A6000 GPUs with CUDA (11.7), python (3.8.10), transformers (4.28.0), PyTorch (1.13.1), and openprompt (1.0.1).
A.2 Hyperparameters We use for the experiments fixed hyperparameters; AdamW (Loshchilov and Hutter, 2019) as optimizer; a batch size of 16; a learning rate of 0.00002; a dropout rate of 0.1; a warmup rate of 10% of the steps; random seeds: [0, 1, 2]. In the case of parameter-efficient tuning, we use a learning rate of a learning rate of 0.0002. Moreover, we use the following tags from the huggingface model hub:
• albert-base-v2
• bert-base-uncased
• aajrami/bert-mlm-base
• microsoft/deberta-base
• microsoft/deberta-v3-base
• roberta-base
• google/electra-basediscriminator
• t5-3b
• google/flan-t5-xl
• TheBloke/Llama-2-70B-Chat-AWQ
• TheBloke/Orca-2-13B-AWQ
A.3 Fold Composition With our evaluation, we want to cover a given dataset fully. Therefore, we conduct a multi-folded evaluation that covers every instance of the dataset once in one of the tests splits Xtest. For a fair comparison, we use the same number of folds for OOD and ID and synchronize their dimension, i.e., train, dev, and test split of the first fold have the same number of instance for OOD and ID.
We show with Figure 8 an example of a dataset with a topic shift. We colorize topics and indicate train, dev, and test splits with solid, dashed, and dotted lines. First, we sort all dataset instances according to their assigned topic for OOD while we
randomly shuffle them for ID. Then we compose the test splits X (OOD)test and X (ID) test in a way to cover every instance exactly once. Next, we form the train and dev splits by randomly distribution the left-over topics (OOD) or instances (ID) for all folds. When composing these splits, we compose the ID splits to match the respective OOD splits of the same fold. For example considering the first fold, the splits X (OOD)train-1 , X (OOD) dev-1 , and X (OOD) test-1 have the same number of instances as the splits X (ID)train-1, X (ID)dev-1, and X (ID) test-1.
Based on the number of unique distribution shift properties (topics, domains, or languages), we use the different number of folds to distribute these properties as even as possible across the different test splits X (OOD)test . Therefore, we use whenever possible a three-folded setup. However, when the number of distribution properties is equal to four (i.e., four domains), we conduct a four-folded evaluation. Please find this concrete number of folds per task in the source code.
A.4 Dataset Details
As a part of this work, we propose eleven different OOD classification tasks based on 13 different datasets. In the following, we provide additional details. Table 5 shows an overview of these tasks and examples for every task. Furthermore, we show in Figure 9 how these task examples diverge from the LMs’ pre-trained textual understanding based on Wikipedia, which is a major pre-training dataset for BERT, RoBERTa, and DeBERTa. Specifically, we compute the pseudo perplexity (Salazar et al., 2020), determined as the cross-entropy of each token, for 500 randomly chosen instances per task. For English tasks, bert-base-uncased is used, while bert-base-multilingual-uncased is
deployed for multilingual tasks. We then compare the averaged cross-entropy of these tasks (refer to Figure 9) against 500 randomly selected samples from the Wikipedia pre-training corpus (Devlin et al., 2019). The comparison indicates a notable divergence in the chosen tasks from a common pre-training dataset Wikipedia, except for evi-cls, which leverages Wikipedia data, and x-stance, which aligns closely with Wikipedia’s text genre.
Finally, Table 6 lists the prompt templates used for all tasks and languages.
A.5 Prompt-Based Fine-Tuning In this paper, we adopt prompt-based fine-tuning (P+FT) (Liu et al., 2021a) as an alternative approach to vanilla fine-tuning (FT). Unlike FT, P+FT relies on the pre-trained masked-language modeling (MLM) head and avoids using new classification heads.
Forward Pass We show in Figure 10 an exemplary overview of P+FT for sentiment analysis
given two classes K = {positive, negative}. In detail, we wrap the review with a cloze template and add a masking token as the prediction proxy. Next, the LM processes this prompt and outputs the most probable tokens T along with their log probabilities L using the MLM head. Then, the verbalizer selects the relevant tokens A within L and assigns them a class mapping - like positive (green) or negative (red). In contrast to other probing work (Schick and Schütze, 2021a; Raman et al., 2023), we automatically select indicative tokens Schick et al. (2020) using the likelihood ratio regarding every class k in K based on the train instances. With these token-class mappings, we sum up the log-probabilities for every class k in K as wk = ∑ a∈A(k) L(a) and apply the softmax (Equation 1) to find ŷ.
ŷ = argmax k∈K exp(wk)∑ k′∈K exp(wk′)
(1)
Backward Pass While the forward pass represents the prompting paradigm (P), we analyze and evaluate LMs without parameter optimization. However, to fine-tune the LM and MLM head, we calculate the cross-entropy loss LCE and update the weights through back-propagation. Note that we initialized the automatic verbalizer before the training and did not update it anymore afterward.
A.6 In-Context Learning Setup
As reported in § 6, we evaluated ChatGPT using incontext learning (ICL). In detail, we provide four demonstration samples from the training instances for every test instance. We use the templates reported in Table 7 for every training demonstration instance and the test instance, where we exclude the LABEL for the test one. For a fair comparison with gradient-based approaches (FT, P+FT), we allow to sample these demonstration instances from the entire training set. We use BM25 to calculate the similarity between the test and train instances. Afterward, we use the top-4 most similar train instances as a demonstration for a given test instance.

heading: B Additional Results
text: In addition to results shown in the main paper (§ 6), we show in the following the effectiveness of P+FT for ID scenarios (§ B.1) and that other methods to prevent freshly initialized classification heads underperforms prompt-based fine-tuning (§ B.2).
B.1 In-Distribution Results Table 8 shows the superior performance of promptbased fine-tuning transfers to ID scenarios.
B.2 Classification Head Pre-Initialisation In addition to prompt-based fine-tuning (P+FT), we experimented with pre-initializing the classification head using a linear probe (LP+FT) following Kumar et al. (2022). As reported in Table 9, we did not find a positive effect of using LP+FT.
Dataset Description Distribution Shift
arg-qua Argument Quality (Toledo et al., 2019)
Choose which argument out of two has the higher quality: TOPIC : we should ban fossil fuels ARG-1 : fossil fuels pollute and cause a lot of diseases ARG-2 : fossil fuel companies often have incredibly bad and dangerous working conditions LABEL : ARG-1
Topical (22 topics)
arg-sim Argument Similarity (Reimers et al., 2019)
Decide whether two arguments are similar or not-similar : TOPIC : organ donating ARG-1 : One organ and tissue donation can save or enhance the lives of nearly 100 people ARG-2 : By donating your organs after you die, you can save or improve as many as 50 lives LABEL : similar
Topical (28 topics)

heading: Future Work
text: Simultaneously, we think performance and efficiency will alternate in the future. Therefore, we keep that for future work