Venue: NAACL 2024

# Rectifying Demonstration Shortcut in In-Context Learning

Joonwon Jang Sanghwan Jang Wonbin Kweon Minjin Jeon Hwanjo Yu\({}^{1,2,*}\)

Graduate School of AI, POSTECH\({}^{1}\)

Department of Computer Science and Engineering, POSTECH\({}^{2}\)

Institute of Artificial Intelligence, POSTECH\({}^{3}\)

{kaoara, s.jang, kwb4453, minjinj, hwanjoyu}@postech.ac.kr

###### Abstract

Large language models (LLMs) are able to solve various tasks with only a few demonstrations utilizing their in-context learning (ICL) abilities. However, LLMs often rely on their pre-trained semantic priors of demonstrations rather than on the input-label relationships to proceed with ICL prediction. In this work, we term this phenomenon as the 'Demonstration Shortcut'. While previous works have primarily focused on improving ICLR prediction results for predefined tasks, we aim to rectify the Demonstration Shortcut, thereby enabling the LLM to effectively learn new input-label relationships from demonstrations. To achieve this, we introduce In-Context Calibration, a demonstration-aware calibration method. We evaluate the effectiveness of the proposed method in two settings: (1) the Original ICL Task using the standard label space and (2) the Task Learning setting, where the label space is replaced with semantically unrelated tokens. In both settings, In-Context Calibration demonstrates substantial improvements, with results generalized across three LLM families (OPT, GPT, and Llama2) under various configurations. 1

Footnote 1: [https://github.com/Lainshower/In-Context-Calibration.git](https://github.com/Lainshower/In-Context-Calibration.git)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4294–4321

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Large language models (LLMs) have demonstrated their effectiveness on a wide range of tasks through in-context learning (ICL), where models learn to perform a task from demonstrations Brown et al. (2020). Leveraging their pre-trained knowledge, LLMs can associate various words in the demonstration with specific semantics (e.g., associating 'extremely painful' with 'negative'), thereby performing new tasks using only a small set of input-label examples, without requiring parameter updates Dong et al. (2022); Wei et al. (2023).

However, LLMs often rely on the semantics from their pre-trained knowledge of given demonstrations, resulting in insufficient task learning for the patterns of the provided input-label pairs. Reynolds and McDonell (2021); Min et al. (2022); Wei et al. (2023); Pan et al. (2023). This issue intensifies as the model size decreases Wei et al. (2023). Kossen et al. (2023) suggest that smaller LLMs show promise in learning new mappings from demonstrations in some tasks, yet they still struggle to override semantic priors acquired during pre-training. Therefore, it is necessary to develop a method that enables LLMs of various sizes to effectively mitigate semantic priors preferences and learn to perform unseen tasks from demonstrations.

Prior works have primarily focused on the instabilities of LLMs in ICL prediction Holtzman et al. (2021); Fei et al. (2023). To mitigate these instabilities, these studies introduced content-free tokens or utilized the entire test set to calibrate prediction probabilities Holtzman et al. (2021); Fei et al. (2023); Zhou et al. (2023). However, they lack consideration of the semantic priors of LLMs on the demonstration and do not verify whether their approach enhances LLMs to learn new tasks from the demonstrations.

In this work, we investigate how the LLMs' pre-trained knowledge on the demonstrations affects ICL. We define the following phenomenon as a _Demonstration Shortcut_: the reliance of LLMs on their pre-trained semantic priors of demonstrations in ICL prediction, rather than learning from the input-label relationships presented in these demonstrations. Due to the Demonstration Shortcut, LLMs' ICL predictions may be overly dependent on the semantics of the given demonstrations even when the label distribution is uniform and the order is identical (Figure 1).

To tackle this problem, we propose In-Context Calibration, a method designed to rectify the

Demonstration Shortcut in ICL. In-Context Calibration estimates the semantic prior of LLMs on each demonstration sample with the in-context examples. Formally, for each example in the demonstration, we estimate its semantic prior relative to the remaining examples and calculate the expected semantic priors of the demonstrations. At test time, we use this term to rectify LLMs' dependency on semantic priors and enable the model to learn the intended input-label relationships from the demonstrations.

We evaluate the effectiveness of In-Context Calibration on 27 classification datasets from two perspectives: (1) Original ICL Task and (2) Task Learning settings. In the Original ICL Task, we use the standard label space. In the Task Learning setting, the label space is replaced with semantically unrelated tokens. This requires LLMs to learn the novel input-label relationships to achieve high performance, as these relationships are never seen in pre-training. Our proposed method not only demonstrated enhanced performance across various tasks but also showed improvement in task learning abilities. Specifically, In-Context Calibration outperforms other ICL methods in Natural Language Inference (NLI) tasks, which demand high task learning ability. We also demonstrate that In-Context Calibration enhances ICL performance across various model types and sizes, effectively rectifying the 'Demonstration Shortcut' problem.

## 2 Backgrounds

How LLMs utilize demonstrations in ICLFollowing ICL's accomplishments, extensive prior works have sought to understand how LLMs use demonstrations, yet there is still no consensus on the following two contradictory perspectives. One line of research claims that LLMs do not learn new input-label relationships from the demonstrations, with the evidence that ICL performance only marginally drops when labels in the demonstrations are replaced with random labels [35]. Instead, LLMs independently recognize the semantics of input and label of in-context demonstrations using their pre-trained knowledge and perform ICL prediction with its language modeling objective [46, 39]. On the other hand, while some studies suggest that LLMs can learn novel tasks through demonstrations, there is a notable lack of concrete experimental proof in real-world LLM applications [80, 81]. Addressing this gap, Wei et al. wei2023learning provides evidence that larger LLMs can learn the new input-label mappings from demonstrations. Summarizing these perspectives, Pan et al. pan2023learning show that applying pre-trained knowledge to demonstrations for task recognizing is a broad capability across scales while learning new input-label mappings becomes more feasible as the scale increases. This indicates that as LLMs decrease in size, they rely more on pre-trained knowledge of demonstrations in ICL prediction. Furthermore, when labels in the demonstrations are flipped with different labels (e.g., labeling 'positive' as 'negative' and vice versa), these models struggle to override semantic priors obtained during pre-training. [45, 46].

Improving ICL through CalibrationVarious studies have focused on the instability of ICL prediction in LLMs. [81, 82] reveal the instability of ICL prediction arises from the majority label bias and recency label bias, and Fei et al. fei2023learning identifies domain as a factor contributing to label bias behind this instability. These studies have attempted to estimate the instability of ICL prediction by introducing content-free tokens [81] or using the entire test set to calibrate ICL prediction probabilities [44, 81]. Although their approaches show an improvement in ICL prediction, they do not address the reliance of LLMs on the semantic priors of the demonstration. In other words, the primary objective of these methods is to enhance ICL prediction performance for pre-defined tasks, rather than enabling the model to learn input-label mappings from the demonstrations. Furthermore, they fail to demonstrate whether their calibration method allows LLMs to learn new input-label mappings through demonstrations. Moreover, it is unreasonable to assume that the entire test set is available when learning new input-label relationships. Based on the discussions above, our analysis focuses on the reliance of LLMs on their semantic priors of demonstrations, offering a novel perspective on the calibration method.

## 3 Demonstration Shortcut

In this section, we first introduce a new typology termed _Demonstration Shortcut_. This concept 

refers to the reliance of LLMs on their pre-trained semantic priors of demonstrations for making ICL predictions, rather than learning the input-label relationships presented in the demonstrations. Figure 1 illustrates the underlying concept of the Demonstration Shortcut. In a zero-shot setting, the LLM predicts the test article to the world label (the ground-truth label is technology). Next, we constructed two demonstration sets from the same training dataset, each with a uniform label distribution. While these sets have identical label orders, they differ in the semantics of the examples. The first demonstration set mostly features business-related semantics across all examples, while the second set leans more toward sports-related semantics. After ICL with the first demonstration set, the LLM predicted the article to the business label. However, LLM predicted the article to the sports label with the second demonstration set. Despite both sets having uniform label distributions and identical orders, LLM relies on the semantic prior from each demonstration set, exhibiting a Demonstration Shortcut and failing to predict the correct answer. This indicates that an overdependence on semantic prior interrupts the ability of LLMs to learn the new input-label mapping relationships from the demonstrations.

To deeply describe how semantic priors acquired from pre-training affect ICL prediction, consider the first example in Demo #1 of Figure 1, titled _'Vietnam Hosts Investment Conference - Hopping to Boost Business Ties with Singapore \(\sim\)'_ (see Appendix A for the full example). The overall context of the article allows the LLM's pre-training knowledge to associate its semantics with

Figure 1: The overall illustration of the _Demonstration Shortcut_. In a zero-shot setting, an LLM predicts the test article to the world label. With the first demonstration set, the LLM predicts the business label through ICL. However, with the second demonstration set which has the same label order but different semantics in the examples the LLM predicts the sports label. GPT-J is used for these experiments. See Appendix A for a full description of the demonstrations.

 

the business label. Meanwhile, word-by-word examination (e.g., Vietnam or Singapore) also reveals potential associations of its pre-trained semantics with the world label Tang et al. (2023). By iterating this process for all examples, regardless of the assigned ground-truth label, the LLM may proceed with ICL prediction based on pre-trained semantic distributions of the demonstrations, leading to the Demonstration Shortcut.

Substantiating our intuition, we conducted additional experiments based on Figure 1. We constructed Demo #3 to be characterized by world semantics across all examples, while Demo #4 features across technology-related semantics. All demonstration sets were designed to have a uniform label distribution and an identical sequence, as depicted in Figure 1. We ensured the test set also followed a uniform label distribution (25 samples for each label) and reported the average label prediction probabilities for these examples in Table 1. Despite having uniform and identical label distributions in the demonstrations, the LLM predictions for all demonstration sets exhibit different behaviors; this variance aligns with the overall semantics of each demonstration.

## 4 In-Context Calibration

This section introduces a novel calibration method to rectify the Demonstration Shortcut. We first revisit existing calibration methods designed to improve ICL predictions and analyze their limitations, particularly regarding Demonstration Shortcut. We then propose In-Context Calibration, our approach to rectifying the Demonstration Shortcut in ICL.

Revisiting Previous MethodsPrior works on calibrating LLMs attempt to adjust ICL predictions by estimating the prompt prior with a content-free token 'N/A' Zhao et al. (2021) or by estimating the task prior by utilizing the entire test distribution Fei et al. (2023); Zhou et al. (2023). Specifically, Contextual Calibration (CC) estimates the content-free prediction prior as \(P_{LM}(y|^{\cdot}N/A^{\cdot},[(x_{i},y_{i})]_{i\in[K]})\), where \(x_{i}\) represents the text input, \(y_{i}\) corresponds to a verbalized label name, and \(K\) denotes the total number of examples. Meanwhile, Domain Calibration (DC) estimates task prior with \(\frac{1}{M}\sum_{r=1}^{M}P_{LM}(y|[r/w]_{r},[(x_{i},y_{i})]_{i\in[K]})\), where \([r/w]\) represents random words drawn from the entire test set. However, the introduced terms are not entirely content-free; their neutrality depends on the task type and the demonstrations Fei et al. (2023); Zhou et al. (2023). Therefore, if these terms' pre-trained semantics mismatch with the semantic priors of demonstrations, they are limited in rectifying the Demonstration Shortcut. Moreover, relying on the entire test distribution is impractical in real-world settings.

In-Context CalibrationTo overcome these limitations, we propose In-Context Calibration, which rectifies the Demonstration Shortcut of the model in the ICL setting.

For each \(x_{i}\) in the demonstration, we estimate the semantic prior of LLMs on each demonstration sample with remaining in-context demonstrations:

\[P_{t}=P_{LM}(y\mid x_{i},[(x_{j},y_{j})]_{j\in[K]\setminus\{i\}}),\forall i, \tag{1}\]

where \(P_{i}\) represents the semantic distribution of \(x_{i}\) given the remaining \(K-1\) demonstrations. This allows us to estimate the contextual semantic prior of each demonstration sample while preserving the remaining in-context demonstrations order and conditions present in the original ICL setting.

Additionally, we estimate each demonstration sample's word-by-word semantic distribution by applying the random shuffling function \(R()\) to \(x_{i}\), which shuffles the order of the words as follows:

\[P_{R(i)}=P_{LM}(y\mid R(x_{i}),[(x_{j},y_{j})]_{j\in[K]\setminus\{i\}}),\forall i. \tag{2}\]

The resulting random order of \(R(x_{i})\) is not grammatically meaningful, yet it enables context-agnostic estimation of the LLMs' pre-trained semantics for each demonstration sample. In other words, \(R(x_{i})\) is stripped of context and retains only the words with their meanings, thereby preventing the LLMs from making predictions based solely on the semantics of individual words Fei et al. (2023); Tang et al. (2023)2

Footnote 2: Please refer to Appendix E for additional details for the random shuffling function.

We iterate this process across all \(K\) demonstrations and compute the average to estimate the semantic priors of the demonstrations:

\[\frac{1}{K}\sum_{i=1}^{K}(\lambda\cdot P_{i}+(1-\lambda)\cdot P_{R(i)}), \tag{3}\]

where \(\lambda\) is the hyperparameter that controls the balance between the two terms. Considering every dependency from the \(K\) demonstrations and taking 

the average, we can estimate the expected semantic priors of demonstrations, enabling more effective _demonstration-aware calibration_. The model then makes ICL predictions based on the following estimates:

\[\hat{y}_{pred}=\arg\max_{y\in\mathcal{L}}\frac{P_{LM}(y~{}|~{}x_{\text{test}},[( x_{i},y_{i})]_{i\in[K]})}{\frac{1}{K}\sum_{i=1}^{K}(\lambda\cdot P_{i}+(1-\lambda) \cdot P_{R(i)})}, \tag{4}\]

where \(P_{LM}(y~{}|~{}x_{\text{test}},[(x_{i},y_{i})]_{i\in[K]})\) is the original ICL prediction.

## 5 Experimental Setups

We investigate the effectiveness of In-Context Calibration in two aspects: (1) the model's performance on the Original ICL Task (using standard label space from the dataset) and (2) the Task Learning setting (label space is randomly mapped to semantically unrelated tokens), following the experimental settings of Pan et al. (2023). Since the newly introduced input-label mappings have never been parameterized during pre-training, the model must utilize its task learning abilities to handle the problem. In our main experiment, unless stated otherwise, we conduct task learning experiments using string numbers3, and to avoid any pre-trained bias, each label is randomly assigned to a unique string number for every seed.

Footnote 3: String numbers demonstrate better task learning ability than other tokens in Pan et al. (2023).

### Datasets

We conducted experiments on 27 classification datasets across three types of tasks: Sentiment, NLI, and Detection classification task. Our dataset selection and prompts largely follow the methodologies of prior ICL works Zhao et al. (2021); Min et al. (2022); Fei et al. (2023), and more details are described in Appendix B.

### Base Models

To validate our method across a diverse set of models, we use three state-of-the-art LLM families: GPT (2.7B, J (6B), 20B) Brown et al. (2020), OPT (2.7B, 6.7B, 13B) Zhang et al. (2022), and Llama2 (7B, 13B) Touvron et al. (2023). For the GPT models, we use the open-sourced versions provided by EleutherAI Gao et al. (2020); Wang (2021); Black et al. (2022) as Fei et al. (2023). Consequently, we utilize checkpoints from the Transformers library Wolf et al. (2020) for all the aforementioned models.

Figure 2: Averaged Macro F1 scores for OPT (Top), GPT (Medium), and Llama2 (Bottom) across Sentiment, NLI, and Detection Tasks. The left the left three columns depict the performance on the Original ICL Task. The right three columns plot the Task Learning scores. In both graphs, the x-axis represents the model size.

 



### Implementation Details

Adopting a sampling-based evaluation approach, we sample different sets of demonstrations from the training set for each seed and report the mean and standard deviation of the results. We use \(K=8\) examples and conduct five evaluations using different random seeds, per the methodology described by Fei et al. (2023). Unless stated otherwise, we set \(\lambda\) to \(0.5\). For the baselines, we selected Contextual Calibration (CC) Zhao et al. (2021) and Domain Calibration (DC) Fei et al. (2023) to assess the performance on the Original ICL Task and Task Learning setting. For Domain Calibration, the original method involves constructing a bag of words from the entire test set, which is impractical for real-world inference. To facilitate a fair comparison, we adapted this method. The adapted version samples random words from the demonstration and is labeled as "w/ Demo" (with Demo), while we refer to the original method as "w/ Test" (with Test).

## 6 Experimental Results

### Main Results

Figure 2 shows the Macro F1-scores of OPT, GPT, and Llama2 on three categories of datasets in the Original ICL Task and Task Learning setting. The results demonstrate that In-Context Calibration effectively rectifies the Demonstration Shortcut, enhancing performance in the Original ICL Task and improving learning capabilities for new input-label tasks. Specifically, for Llama2, In-Context Calibration resulted in an average F1 score improvement of 23% compared to the original inference in the Original ICL Task. Regarding CC using 'N/A' token and DC w/ Test sampling random words from the test set, if the newly introduced tokens fail to accurately estimate the neutrality of the demonstration's semantic distribution, the model remains constrained by the Demonstration Shortcut, limiting performance improvements. This is particularly pronounced in NLI tasks, where previous works Pan et al. (2023); Kossen et al. (2023) have shown to be challenging for LLMs to learn input-label pairs due to strong reliance on semantic priors in the ICL setting, while In-Context Calibration effectively rectifies the Demonstration Shortcut for all LLMs, resulting in notable performance increases.

This improvement is also evident in the Task Learning setting, where the label space is replaced with string numbers. For GPT, In-Context Calibration achieved an average F1 Score increase of 27% over the original inference. Particularly in the NLI task, other methodologies struggled to mitigate the dependency of the semantic priors of the models on demonstration, leading to decreased performance compared to original inference in some cases. Across various dataset categories and model types, In-Context Calibration consistently outperformed baseline methods on tasks with novel input-label pairs by effectively reducing the model's reliance on the demonstration's semantic prior (see Appendix G for comprehensive results).

### Ablation Study

We conducted an ablation study of the proposed method. First, we replaced the \(R(x_{i})\) term with either a 'N/A' token or words randomly sampled from the bag of words constructed from the entire test set. The replacement with the 'N/A' token resulted in lower performance compared to the original inference in the Task Learning setting, likely due to the instability of the 'N/A' token in estimating prompt neutrality Fei et al. (2023). Furthermore, words randomly sampled from the test set underperformed in both tasks compared to In-Context Calibration with \(\lambda=0.5\). This underperformance could stem from the semantic distribution mismatch between the sampled words and the demonstrations, limiting the model's ability to rectify the Demonstration Shortcut. However, the demonstration-aware calibrating of In-Context Calibration leads to performance improvements.

Ablation studies on \(\lambda\) demonstrate that using only \(R(x_{i})\) for calibration (\(\lambda=0\)), which shuffles the token order for each demonstration, leads to limited performance gains in both the Original ICL

\begin{table}
\begin{tabular}{l c l} \hline Method & Original & TL \\ \hline Original Inference & 0.48 & 0.40 \\ \(R(x_{i})\rightarrow\text{`$N/A$''}\) & 0.49 & 0.38 \\ \(R(x_{i})\rightarrow\text{random words}\) & 0.57 & 0.46 \\ In Context Calibration (\(\lambda=0.5\)) & 0.60 & 0.49 \\ In Context Calibration (\(\lambda=0\)) & 0.56 & 0.46 \\ In Context Calibration (\(\lambda=1\)) & 0.58 & 0.50 \\ \hline \end{tabular}
\end{table}
Table 2: Analysis of In-Context Calibration: Performance comparing \(R(x_{i})\) replacement with ‘N/A’ (CC) or randomly sampled test set words (DC), and effects of varying \(\lambda\) values, shown through average Macro F1 scores across 27 datasets in Original ICL Task (Original) and Task Learning (TL) settings using GPT-J.



Task and Task Learning setting, attributing to the loss of contextual information. On the other hand, not utilizing \(R(x_{i})\) for calibration (\(\lambda=1\)) results in better learning of new input-label relationships. However, this approach is less effective in mitigating the model's dependency on word-wise pre-trained semantics of demonstration when considering labels in the Original ICL Task. We conduct a detailed task-wise analysis of the \(\lambda\) value's effect in Section 6.5 and Appendix F. Therefore, after comprehensive analysis for \(\lambda\) values, we set \(\lambda\) to \(0.5\) in the main experiment.4

Footnote 4: Please refer to the Appendix F for the comprehensive analysis for \(\lambda\) value.

### Analysis with Enhanced Models

Instruction-Tuned ModelIn previous studies Min et al. (2022); Wei et al. (2023), instruction tuning has been demonstrated to strengthen the usage of semantic priors in the demonstration. Therefore, we conducted experiments to determine whether In-Context Calibration consistently rectifies the Demonstration Shortcut and improves task learning ability in instruction-tuned LLMs. As depicted in Figure 3, In-Context Calibration increases the model's F1 score across all Original ICL Tasks and particularly shows substantial improvement over other calibration methods in the Task Learning setting. Especially in the Task Learning setting, other calibration methods often underperformed the original inference. These experiments demonstrate that In-Context Calibration remains effective and enhances the model's task learning abilities, even after instruction tuning has strengthened the LLM's reliance on semantic priors.

Over 50B Scale ModelsWe conducted experiments to verify that the proposed method consistently improves performance in larger models. As demonstrated by Pan et al. (2023), in the Task Learning setting, the performance of some models (e.g., OPT) starts to match their performance in the Original ICL Task for models larger than 50B. These models can utilize the mapping information provided in the demonstrations. Therefore, we conducted experiments on OPT 66B, Llama2 70B, and Llama2-chat 70B, reporting average F1 scores for 27 datasets in both the Original ICL Task and Task Learning settings, as shown in Figure 4. Particularly for Llama2 70B, In-Context Calibration improved F1-score performance by 15% over the original inference in the Original ICL Task and by 9% in the Task Learning setting, while some methods hurt the models performance. These findings suggest that In-Context Calibration not only boosts performance in smaller-scale models but also consistently improves the learning ability of larger-scale models, by effectively rectifying the Demonstration Shortcut through its demonstration-aware calibration.

Analysis with Other Input-Label Mappings Overridding Semantic Priors (Permutating Label Space)Wei et al. (2023) and Kossen et al. (2023) reveal that LLMs struggle to override semantic priors preference with input-label mappings in ICL setting. To test whether our proposed method helps the models override semantic priors preference by using its task learning abilities, we evaluate performance by randomly permuting the label space. For instance, in the AGNews dataset, the original label 'sports' is permuted to 'world,' 'business' data to 'technology,' 'technology' data to 'sports,' and 'world' data to 'business' to construct a demonstration set. The test labels are similarly permuted for evaluation. Due to semantic priors preferences, the models tend to rely more

Figure 3: Averaged Macro F1 scores for Llama2-Chat across Sentiment, NLI, and Detection Tasks. The left three columns depict the performance of the Original ICL Task. The right three columns plot the Task Learning scores. In both graphs, the x-axis represents the model size.

 

on their pre-trained knowledge than on learning new relationships from input-label pairs, resulting in lower performance on permuted datasets. We report the average F1 score across 27 datasets. Figure 5 illustrates that the GPT model outperforms other calibration methods using In-Context Calibration. For results related to other models, please refer to Figure 7 in Appendix G. This indicates that In-Context Calibration's demonstration-aware calibrating is needed to diminish the model's semantic priors preferences and let it learn new tasks from demonstrations, especially those that contradict pre-trained knowledge.

Task Learning with different label mapping (Symbol Token)Pan et al. (2023) demonstrate that replacing the label space with symbols in the Task Learning setting leads to underperformance, attributing to their unnaturalness in pre-training stages. To verify whether our method enhances learning ability in a more general task learning environment, we conducted experiments by mapping the label space to symbols. In other words, we randomly replace the label space with one of [@, #, $, ...] at every seed5. We report the average F1-score across 27 datasets as experimental results. Figure 6 shows that our method outperforms other calibration methods, particularly demonstrating a significant improvement over the original inference. This indicates that our method enhances LLM's task learning ability in a broader context.

Footnote 5: Please refer to the Appendix D for the detailed implementation.

### Analysis of \(\lambda\) across Different Task Categories

We investigated the utility of \(R()\) across the different task categories, by calculating the task-wise average F1 score as shown in Table 3. For the Sentiment and Detection task, where the semantics of each word are crucial in associating a specific label in pre-trained knowledge, applying \(R()\)

Figure 4: Averaged Macro F1 scores across 27 classification tasks for over 50B scale LLMs. The left graphs depict performance in the Original ICL Task, while the right graphs plot task learning scores. In both sets of graphs, the x-axis denotes the model type. Full details of the data-type scores are provided in Appendix G.

Figure 5: Averaged Macro F1 scores for the GPT model are presented across 27 classification tasks, each featuring a permuted label space. The x-axis represents the model size. Results for the OPT and Llama2 models are provided in Appendix G.

Figure 6: Averaged Macro F1 scores for the 6-7B scale model families are presented across 27 datasets with each label space replaced by symbol tokens. The x-axis represents the model type. Results for the 13-20B scale models are available in Figure 8.

 

for calibration proves more effective. In contrast, for NLI tasks, where the LLMs must discern the logical relationship between two input sentences, In-Context Calibration demonstrates its effectiveness over the baseline in mitigating the reliance of LLMs on semantic priors of the demonstration (as show in Figure 2). However, applying \(R()\) in NLI tasks disrupts the order of the sentences, necessitating contextual awareness calibrating for better performance. We reserved for additional results in Appendix F.

## 7 Conclusion

In this paper, we introduced the term 'Demonstration Shortcut', which refers to the reliance of LLMs on their pre-trained semantic priors of demonstrations for making ICL predictions. To rectify this Demonstration Shortcut and enable the model to learn the input-label relationships from the demonstrations, we proposed a novel method, In-Context Calibration, based on the provided demonstrations. This demonstration-aware calibration consistently yields the improved performance, regardless of model sizes or types, across various settings. With the introduction of In-Context Calibration, we anticipate more reliable applications of Large Language Models.

### Limitations

In this work, we investigate the reliance of Large Language Models on their pre-trained semantic priors on demonstrations in in-context learning prediction. While In-Context Calibration demonstrates its effectiveness across various tasks and enhances the LLMs' task learning abilities, our experiments primarily focus on classification tasks. However, the effect of the Demonstration Shortcut might manifest differently in generation tasks. Further analysis and adaptation of our In-Context Calibration method for these tasks are left for future research. Due to budget constraints, experiments with larger models (e.g., GPT4 API) and in multilingual settings were not feasible. Future studies with diverse settings and sufficient resources could provide a more comprehensive understanding.

Due to computational constraints, it was impractical to explore every possible \(\lambda\) value for each model. While there may be variations across different models that remain unexplored, we believe that the comprehensive analysis provided in the appendix will offer practical guidelines for selecting the \(\lambda\) value.

### Ethical Considerations

Our work focuses on how Large Language Models utilize demonstrations in in-context learning. To enhance the ability of LLMs to learn input-label relationships from demonstrations, we conducted several additional inferences, requiring only minimal computational resources compared to updating model parameters. Additionally, we used only open-source LLMs and publicly available text classification datasets. Therefore, we do not concern about significant ethical issues arising from our work. On the contrary, we anticipate that future works could utilize our analysis to rectify harmful biases inherent in pre-trained model knowledge through demonstration-based methods.

## Acknowledgements

We appreciate Keonwoo Kim, Jaehee Kim, Yukyung Lee, and Hyowon Cho for their invaluable comments. We also thank POSTECH DI LAB members and anonymous reviewers for their comments on the paper. This work was supported by Institute of Information communications Technology Planning Evaluation (IITP) grant funded

\begin{table}
\begin{tabular}{l c c c} \hline Task & Orig. & ICC (\(\lambda=1\)) & ICC (\(\lambda=0.5\)) \\ \hline Sentiment & 0.69 & 0.75 & 0.78 \\ NLI & 0.33 & 0.51 & 0.49 \\ Detection & 0.41 & 0.52 & 0.54 \\ \hline \end{tabular}
\end{table}
Table 3: GPT-J’s averaged F1 scores across different task categories with and without applying \(R()\). Orig. denotes original inference, while ICC stands for In-Context Calibration.

 

by the Korea government (MSIT) (No.2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH)), Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2018-0-00584, (SW starlab) Development of Decision Support System Software based on Next-Generation Machine Learning) , National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00217286), Digital Innovation Hub project supervised by the Daegu Digital Innovation Promotion Agency (DIP) grant funded by the Korea government (MSIT and Daegu Metropolitan City) in 2024 (No. DBSD1-07). MSIT: Ministry of Science and ICT, and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2022R1A6A1A03052954).

## References

* Basile et al. [2019] Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, Minneapolis, Minnesota, USA, pp. 54-63. External Links: Document Cited by: SS1.
* Black et al. [2022]Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. Cited by: SS1.
* Brown et al. [2020]Torn Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. Cited by: SS1.
* Ido Dagan, Oren Glickman, and Bernardo Magnini [2005]The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Cited by: SS1.
* On de Gibert et al. [2018]O. de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros [2018]H. Peech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), Brussels, Belgium, pp. 11-20. External Links: Link Cited by: SS1.
* Marie-Catherine De Marreffe, Mandy Simons, and Judith Tonhauser [2019]M. De Marreffe, M. Simons, and Judith Tonhauser [2019]The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Simu und Bedetung, Vol. 23, pp. 107-124. Cited by: SS1.
* Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyao Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui [2022]A survey for in-context learning. arXiv preprint arXiv:2301.00234. Cited by: SS1.
* Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosseltut [2023]Y. Fei, Yifan Hou, Zeming Chen, and Antoine Bosseltut [2023]Mitigating label biases for in-context learning. arXiv preprint arXiv:2305.19148. Cited by: SS1.
* E. Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Hlorace He, Anish Thite, Noa Nabeshima, et al. [2020] The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Cited by: SS1.
* A. Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer [2021]Surface form competition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 7038-7051. External Links: Link Cited by: SS1.
* E. Hovy, L. Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran [2001]Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, Cited by: SS1.
* M. Hu and B. Liu [2004]Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168-177. Cited by: SS1.
* Z. Jiang, Yuanze Zhang, Cao Liu, Jun Zhao, and Kang Liu [2023]Generative calibration for context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2312-2333. Cited by: SS1.
* J. Kossen, T. Rainforth, and Y. Gal [2023]In-context learning in large language models learns label relationships but is not conventional learning. arXiv preprint arXiv:2307.12375. Cited by: SS1.
* H. Levesque, Ernest Davis, and L. Morgenstern [2012]The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, Cited by: SS1.
* P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala [2014]Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology65. Cited by: SS1.
* M. Marcelli, S. Menini, M. Baroni, L. Bentivolgi, Raffaella Bernardi, and Roberto Zamparelli [2014]A. SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), 

216-223, Reykjavik, Iceland. European Language Resources Association (ELRA).
* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Mohammad et al. (2016) Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 31-41.
* Mollas et al. (2020) Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. _arXiv preprint arXiv:2006.08328_.
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Pan et al. (2023) Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 8298-8319, Toronto, Canada. Association for Computational Linguistics.
* Pang and Lee (2004) Bo Pang and Lillian Lee. 2004. A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_, pages 271-es.
* PaNgB (2005) L PaNgB. 2005. Exploitingclassrelationshipfsorstiernntcate gorizationwithrespectratingsales. _IN: ProceedingsofACL r05_.
* Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_, pages 1-7.
* Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5477-5490, Online. Association for Computational Linguistics.
* Sheng and Uthus (2020) Emily Sheng and David Uthus. 2020. Investigating societal biases in a poetry composition system. In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_, pages 93-106, Barcelona, Spain (Online). Association for Computational Linguistics.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Tang et al. (2023) Ruixiang Tang, Dehan Kong, Lo li Huang, and Hui Xue. 2023. Large language models can be lazy learners: Analyze shortcuts in in-context learning. In _Annual Meeting of the Association for Computational Linguistics_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Van Hee et al. (2018) Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In _Proceedings of The 12th International Workshop on Semantic Evaluation_, pages 39-50.
* Wang (2021) Ben Wang. 2021. Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX. [https://github.com/kingofloliz/mesh-transformer-jax](https://github.com/kingofloliz/mesh-transformer-jax).
* Wei et al. (2023) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently, 2023. _URL https://arxiv. org/abs/2303.03846_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvian Gugger, Marianna Darme, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface's transformers: State-of-the-art natural language processing.
* Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. In _International Conference on Learning Representations_.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 75-86.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_ 

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _NIPS_.
* Zhang et al. (2023) Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. 2023. What and how does incontext learning learn? bayesian model averaging, parameterization, and generalization. _arXiv preprint arXiv:2305.19420_.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _Proceedings of the 38th International Conference on Machine Learning_, volume 139 of _Proceedings of Machine Learning Research_, pages 12697-12706. PMLR.
* Zhou et al. (2023) Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2023. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. _arXiv preprint arXiv:2309.17249_.

## Appendix A Full Description of the Articles

A full description of the articles in Figure 1 is provided in Figure 10. We used the AGNews dataset [22] for this experiment. In a zero-shot setting, the LLM (GPT-J) predicts the test article the world label, although the ground-truth label is technology. We then sampled two demonstration sets from the same training set, each having a uniform label distribution and the same label order. The first demonstration set predominantly features business-related semantics across all examples, while the second set leans more toward sports-related semantics. Despite both demonstration sets having the same uniform label distribution and identical order, the LLM fails to learn the input-label relationships, instead relying on semantic priors of demonstration to make ICL predictions.

## Appendix B Datasets

We use 27 text classification datasets for our experiments, most of which are widely used in existing ICL works [23; 24; 25]. **Sentiment** task datasets include AGNews [22], CR [26], financial_phrasebank [27], poem_sentiment [28], MR [29], sst2 [29], Subj [28], and TREC [10]; **Natural Language Inference** task datasets include ANLI [26], WNLI [26], RTE [25], CB [28], and SICK [20] For the **Detection** task we use social_bias_frames [29], tweet_eval_stance_athesim, tweet_eval_stance_feminist [27], tweet_eval_hate [24], tweet_eval_irony [28], tweet_eval_offensive [29], hate_speech18 [2], ethos_binary, ethos_disability, ethos_gender, ethos_national_origin, ethos_race, ethos_religion, and ethos_violence [27].

We constructed demonstrations by sampling from the training set and using the validation set for evaluation. In cases where a validation set does not exist, we utilized the test set. For evaluation, we sampled either a maximum of 500 examples or the entire dataset size, whichever is larger 

Prompt Templates

Our natural language prompts are largely based on Zhao et al. (2021) and Fei et al. (2023). We have adjusted the templates as needed to better align with each dataset's specific intent. The complete list of prompts is provided in Table 7.

## Appendix D Implementation Details

We set \(M=20\) for the Domain Calibration implementation, following the original settings used by Fei et al. (2023). We mainly followed the Pan et al. (2023)'s setting for symbol token selection and incorporated additional symbol tokens to enhance generalization. Consequently, in each seed, every label was randomly mapped to one of the following symbols: ["@", "#", "$", "%", "*", " \(\land\) ", "##", "$$", "%%", "**"].

## Appendix E Random Shuffling Function

In our main experiment, we applied the random shuffling function only once to each \(x_{i}\) to calculate \(P_{R(i)}\). To validate whether a single shuffling process introduces randomness, we conducted additional experiments, varying the number of shuffles to 1, 5, and 10. We present supplementary ablation studies using GPT-J with default settings in our main experiments.

The results illustrate that a single shuffle does not incur significant randomness in the estimations. These observations are similar to those in Fei et al. (2023), which used different random seeds to demonstrate the rapid stabilization and convergence of the sampling process.

## Appendix F Comprehensive analysis for \(\lambda\) Values

We present the complete results for various \(\lambda\) values of GPT-J in Table 5. GPT-J performs best with an Original ICL Task at 0.5 and the higher \(\lambda\) values lead to higher task learning ability.

To further analyze the comprehensive impact of \(\lambda\), we conducted supplementary studies using OPT-6.7B and Llama2-7B, which have similar sizes to GPT-J, exploring broader \(\lambda\) values (Table 6). As with GPT-Js \(\lambda\) value, OPT-6.7B and Llama2-7B also perform best with an Original ICL Task at 0.5 (OPT-6.7B also performs best at the value of 1.0). In line with our primary findings (Table 3), a higher \(\lambda\) value enhances task learning capabilities, leading to less grammatical information corruption in the original input sentences. We also analyze the task-wise value as in Table 8. Consistent across all models, a higher \(\lambda\) improves performance in the NLI task, corroborating our initial findings. Similar to the results seen in Fei et al. (2023), the semantics of words play a critical role in Sentiment and Detection tasks (where a lower \(\lambda\) still shows comparative performance, despite the lost of grammatical information), as the language model's dependence on the label's semantics is significant in these tasks. Therefore, (1) we recommend starting with a \(\lambda\) value of 0.5 in In-Context

\begin{table}
\begin{tabular}{l c c} \hline \hline Method & Original ICL & TL \\ \hline Original Inference & 0.48 & 0.40 \\ Context Calibration & 0.47 & 0.34 \\ Domain Calibration (w/ Test) & 0.55 & 0.43 \\ ICC (1) & 0.60 & 0.49 \\ ICC (5) & 0.60 & 0.48 \\ ICC (10) & 0.60 & 0.49 \\ \hline \hline \end{tabular}
\end{table}
Table 4: ICC stands for In-Context Calibration, and the number in (N) means shuffling number. We calculated the expectation of the randomly shuffled term when the shuffling number exceeded 1.

\begin{table}
\begin{tabular}{l r r} \hline \hline Method & Original ICL & TL \\ \hline Original Inference & 0.48 & 0.40 \\ Context Calibration & 0.47 & 0.34 \\ Localization (w/ Test) & 0.55 & 0.43 \\ ICC (1) & 0.60 & 0.49 \\ ICC (5) & 0.60 & 0.48 \\ ICC (10) & 0.60 & 0.49 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Average Macro F1 scores across 27 datasets for both Original ICL Task (Original) and Task LearningTML 

Calibration and adjusting it based on the task-wise experimental results obtained from the validation set in the Original ICL Task and (2) search for a value with 0.5 or higher in the Task Learning setting. However, due to computational constraints, performing a grid search across all models for every \(\lambda\) value was impractical, so we opted for \(\lambda\) of 0.5 in the main experiment for all models for efficiency.

## Appendix G Detailed Results

We demonstrate the results of the OPT, GPT, Llama2, and Llama2-Chat models in both the Original ICL Task and the Task Learning setting across Table 9 to 16. For models above 50B scale, please refer to Table 17 and 18. We average the performance across a total of 5 seeds, recording mean values and standard deviations. 'Orig.' denotes original inference, 'CC' refers to Context Calibration, 'DC' stands for Domain Calibration, and 'ICC' represents In-Context Calibration. Furthermore, our results indicate that In-Context Calibration outperforms the baselines in most tasks.

Figure 7 presents experimental results for OPT and Llama2 under the same setting as figure 5. We demonstrate that our method is effective across various model types and sizes. Specifically, OPT shows a 19% performance increase compared to the original inference, while Llama2 exhibits a 15% improvement.

In Figure 8, we present the performance of LLMs when labels are mapped to symbols. Context Calibration significantly improves performance across different model types and sizes, highlighting the consistent enhancement of task learning ability.

## Appendix H Adding More In-Context Examples (K=8/12/16)

We study the effect of adding more in-context examples by evaluating the GPT-J. As shown in Figure 9, In-Context Calibration achieves approximately a 6% higher performance than other calibration methods in the Original ICL Task. Similar to Pan et al. (2023), Task Learning performance further improves as the number of demonstrations increases. In-Context Calibration consistently outperforms the baselines, whereas Context Calibration lags behind the original inference. In conclusion, In-Context Calibration consistently demonstrates superior performance in both the Original ICL Task and Task Learning settings, irrespective of the number of demonstrations.

Figure 8: Averaged Macro F1 scores for the 13-20B scale model families are presented across 27 datasets with each label space replaced by symbol tokens. The x-axis represents the model type.

Figure 7: Averaged Macro F1 scores for the OPT (Top Graph) and Llama2 (Bottom graph) model are presented across 27 classification tasks, each featuring a permuted label space. The x-axis represents the model size.

 

Figure 9: The top graph depicts the average Macro F1 score for the Original ICL Task across 27 datasets for GPT-J. The bottom graph plots the average Macro F1 score for the Task Learning setting. In both graphs, the x-axis represents the number of demonstrations.

 


Figure 10: Full description of the _Demonstration Shortcut_ illustrated in Figure 1. All articles and labels for the demonstrations and the test article were selected from the AGNews (Zhang et al., 2015) dataset. GPT-J was used in this experiment.

 



\begin{table}
\begin{tabular}{l l c}
**Dataset** & **Natural Language Template** & **Label Space** \\ \hline
**Sentiment Task** & & \\ \hline sst2, MR, CR & Review: [INPUT] & positive, negative \\  & Sentiment: [LABEL] & \\ \hline financial_phrasebank & Sentence: [INPUT] & positive, negative, \\  & Sentiment: [LABEL] & neutral \\ \hline poem\_sentiment & Verse text: [INPUT] & positive, negative, \\  & Sentiment: [LABEL] & neutral \\ \hline Subj & Input: [INPUT] & objective \\  & Label: [LABEL] & subjective/personal \\ \hline AG News & Classify the news articles into the categories of **Label Space**. & world, technology \\  & Article: [INPUT] & sports, business \\  & Answer: [LABEL] & \\ \hline TREC & Classify the questions based on whether their answer type is a **Label Space**. & number, location, \\  & Question: [INPUT] & description, entity, \\  & Answer: [LABEL] & abbre, person \\ \hline
**NLI Task** & & \\ \hline ANLI, SICK & [PREMISE] question: [HYPOTHESIS] & true, neutral, false \\  & true, neutral, or false? answer: [LABEL] & \\ \hline RTE, WNLI & [PREMISE] question: [HYPOTHESIS] True or False? & True, False \\  & answer: [LABEL] & \\ \hline CB & [PREMISE] question: [HYPOTHESIS] true, false, or neither? & true, false, neither \\  & answer: [LABEL] & \\ \hline
* 1) for each seed.




\begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{2}{c}{Sentiment} & \multicolumn{2}{c}{NLI} & \multicolumn{2}{c}{Detection} \\ \cline{2-7}  & OPT-6.7B & Llama2-7B & OPT-6.7B & Llama2-7B & OPT-6.7B & Llama2-7B \\ \hline Original Inference & 0.674 & 0.720 & 0.342 & 0.478 & 0.457 & 0.371 \\ Contextual Calibration & 0.685 & 0.735 & 0.326 & 0.515 & 0.465 & 0.423 \\ Domain Calibration w/ Test & 0.734 & 0.752 & 0.316 & 0.496 & 0.524 & 0.523 \\ ICC (\(\lambda=0)\) & 0.764 & 0.774 & 0.338 & 0.454 & 0.543 & 0.523 \\ ICC (\(\lambda=0.25\)) & 0.764 & 0.796 & 0.337 & 0.511 & 0.522 & 0.555 \\ ICC (\(\lambda=0.5\)) & 0.749 & 0.792 & 0.398 & 0.551 & 0.552 & 0.558 \\ ICC (\(\lambda=0.75\)) & 0.751 & 0.782 & 0.428 & 0.557 & 0.532 & 0.564 \\ ICC (\(\lambda=1.0\)) & 0.749 & 0.775 & 0.445 & 0.547 & 0.530 & 0.532 \\ \hline \end{tabular}
\end{table}
Table 8: OPT-6.7B and Llama2-7B’s averaged F1 scores across different task categories using various \(\lambda\) value. ICC stands for In Context Calibration.





\begin{table}
\begin{tabular}{c c l l l l l} \hline Dataset & Size & Method & & & \\ \hline Sentiment Task & & Orig. & Occ. & CC & Dev/name & DC w/test & ICC \\  & 2.78 & 0.295(-0.072) & 0.669(-0.117) & 0.656(-0.676) & 0.803(-0.048) & 0.811(-0.020) \\ AGNews & & 6.78 & 0.521(-0.140) & 0.716(-0.103) & 0.677(-0.677) & 0.766(-0.018) & 0.811(-0.023) \\  & 1.38 & 0.722(-0.077) & 0.716(-0.104) & 0.673(-0.735) & 0.796(-0.045) & 0.828(-0.011) \\  & 2.78 & 0.883(-0.031) & 0.892(-0.021) & 0.911(-0.911) & 0.918(-0.008) & 0.908(-0.038) \\ CR & & 6.78 & 0.860(-0.028) & 0.823(-0.024) & 0.941(-0.911) & 0.911(-0.007) & 0.921(-0.005) \\  & 1.38 & 0.800(-0.025) & 0.917(-0.013) & 0.921(-0.922) & 0.920(-0.013) & 0.925(-0.006) \\  & 2.78 & 0.621(-0.191) & 0.945(-0.026) & 0.667(-0.677) & 0.719(-0.115) & 0.710(-0.008) \\ financial_pltreasbrunk & & 6.78 & 0.693(-0.075) & 0.556(-0.106) & 0.522(-0.524) & 0.411(-0.017) & 0.723(-0.099) \\  & 1.38 & 0.471(-0.116) & 0.683(-0.037) & 0.673(-0.678) & 0.676(-0.099) & 0.690(-0.008) \\  & 2.78 & 0.410(-0.077) & 0.367(-0.124) & 0.414(-0.444) & 0.524(-0.009) & 0.545(-0.003) \\ peen,seintment & 6.78 & 0.516(-0.166) & 0.423(-0.050) & 0.501(-0.000) & 0.618(-0.000) \\  & 1.38 & 0.321(-0.057) & 0.485(-0.097) & 0.414(-0.447) & 0.427(-0.004) & 0.492(-0.004) \\ \hline MR & 2.78 & 0.755(-0.097) & 0.875(-0.034) & 0.801(-0.084) & 0.900(-0.008) & 0.900(-0.022) \\  & 6.78 & 0.895(-0.097) & 0.891(-0.011) & 0.913(-0.917) & 0.917(-0.002) & 0.918(-0.004) \\  & 1.38 & 0.895(-0.099) & 0.921(-0.011) & 0.936(-0.039) & 0.917(-0.004) & 0.923(-0.035) \\ set2 & & 2.78 & 0.846(-0.136) & 0.919(-0.048) & 0.910(-0.019) & 0.917(-0.004) & 0.923(-0.035) \\  & 1.38 & 0.900(-0.048) & 0.921(-0.011) & 0.943(-0.049) & 0.945(-0.007) & 0.945(-0.002) \\ \hline Subj & & 2.78 & 0.364(-0.034) & 0.545(-0.106) & 0.545(-0.106) & 0.545(-0.008) & 0.644(-0.007) \\  & 6.78 & 0.581(-0.182) & 0.658(-0.161) & 0.642(-0.642) & 0.744(-0.001) & 0.702(-0.10) \\  & 1.38 & 0.794(-0.079) & 0.772(-0.116) & 0.760(-0.106) & 0.762(-0.191) & 0.825(-0.046) \\  & 2.78 & 0.278(-0.27) & 0.272(-0.405) & 0.435(-0.399) & 0.338(-0.038) & 0.386(-0.039) \\ TRRC & & 6.78 & 0.356(-0.049) & 0.287(-0.044) & 0.308(-0.386) & 0.026(-0.036) & 0.361(-0.020) \\  & 1.38 & 0.382(-0.112) & 0.324(-0.041) & 0.324(-0.324) & 0.324(-0.041) & 0.493(-0.051) \\ \hline NLI Task & & Orig. & CC & Dev/name & DC w/Test & ICC \\  & 2.78 & 0.224(-0.099) & 0.269(-0.049) & 0.229(-0.229) & 0.222(-0.047) & 0.240(-0.042) \\ ANLJ & & 6.78 & 0.215(-0.055) & 0.253(-0.212) & 0.222(-0.047) & 0.258(-0.034) & 0.318(-0.020) \\  & 1.38 & 0.261(-0.046) & 0.289(-0.029) & 0.243(-0.230) & 0.281(-0.07) & 0.307(-0.016) \\  & 2.78 & 0.317(-0.027) & 0.269(-0.030) & 0.300(-0.330) & 0.300(-0.049) & 0.410(-0.022) \\ WNLI & & 6.78 & 0.337(-0.056) & 0.377(-0.033) & 0.380(-0.030) & 0.300(-0.046) & 0.461(-0.057) \\  & 1.38 & 0.460(-0.012) & 0.374(-0.21) & 0.389(-0.386) & 0.375(-0.003) & 0.414(-0.099) \\ \hline RTR & & 2.78 & 0.499(-0.044) & 0.368(-0.044) & 0.372(-0.372) & 0.515(-0.000) & 0.529(-0.055) \\ RITE & & 6.78 & 0.530(-0.055) & 0.465(-0.025) & 0.355(-0.153) & 0.341(-0.057) & 0.480(-0.040) \\  & 1.38 & 0.766(-0.046) & 0.499(-0.056) & 0.455(-0.455) & 0.443(-0.055) & 0.547(-0.037) \\ \hline CD & & 2.78 & 0.356(-0.033) & 0.333(-0.132) & 0.242(-0.22) & 0.496(-0.049) & 0.327(-0.065) \\ CB & & 6.78 & 0.251(-0.047) & 0.426(-0.029) & 0.131(-0.115) & 0.163(-0.057) & 0.367(-0.064) \\  & 1.38 & 0.260(-0.076) & 0.217(-0.076) & 0.218(-0.218) & 0.226(-0.008) & 0.317(-0.067) \\ SICK & & 2.78 & 0.285(-0.058) & 0.279(-0.042) & 0.317(-0.371) & 0.299(-0.076) & 0.328(-0.072) \\ SICK & & 6.78 & 0.267(-0.088) & 0.617(-0.039) & 0.393(-0.395) & 0.395(-0.047) & 0.477(-0.073) \\  & 138 & 0.225(-0.066) & 0.225(-0.046) & 0.





\begin{table}
\begin{tabular}{c c c c c c c} Dataset & Size & Method & Method & & & ICC \\ \hline Sentiment Rate & & Outg. & CC & DC/Umean & DC/Uint & ICC \\ \hline  & -2.78 & 0.53 (0.91) & 0.74 (0.91) & 0.56 (0.95) & 0.71 (0.93) & 0.72 (0.03) \\ ACNs & -1.60 & 0.755 (0.12) & 0.737 (0.07) & 0.726 (0.776) & 0.755 (0.046) & 0.867 (0.012) \\  & 208 & 0.640 (0.066) & 0.826 (0.01) & 0.798 (0.798) & 0.831 (0.014) & 0.818 (0.030) \\ \hline  & -2.78 & 0.74 (0.106) & 0.377 (0.095) & 0.537 (0.837) & 0.873 (0.822) & 0.827 (0.039) \\ CR & -1.60 & 0.689 (0.053) & 0.826 (0.052) & 0.804 (0.904) & 0.802 (0.032) & 0.912 (0.034) \\  & 208 & 0.599 (0.031) & 0.919 (0.019) & 0.914 (0.914) & 0.918 (0.003) & 0.919 (0.035) \\ \hline  & -2.78 & 0.561 (0.021) & 0.841 (0.067) & 0.575 (0.673) & 0.688 (0.016) & 0.696 (0.026) \\ financial_gltrextbank & -1.60 & 0.599 (0.100) & 0.418 (0.115) & 0.540 (0.540) & 0.617 (0.073) & 0.681 (0.056) \\  & 208 & 0.649 (0.194) & 0.470 (0.052) & 0.522 (0.520) & 0.590 (0.077) & 0.684 (0.073) \\ \hline  & -2.78 & 0.420 (0.077) & 0.367 (0.121) & 0.411 (0.414) & 0.542 (0.090) & 0.546 (0.083) \\ pcomt,setminet & -1.60 & 0.512 (0.157) & 0.328 (0.148) & 0.359 (0.359) & 0.524 (0.098) & 0.639 (0.023) \\  & -2.00 & 0.401 (0.164) & 0.433 (0.165) & 0.668 (0.044) & 0.649 (0.054) & 0.571 (0.064) \\ \hline  & -2.78 & 0.647 (0.067) & 0.856 (0.027) & 0.424 (0.947) & 0.997 (0.005) & 0.897 (0.003) \\ MRI & -1.60 & 0.897 (0.025) & 0.506 (0.007) & 0.893 (0.905) & 0.892 (0.019) & 0.915 (0.006) \\  & 208 & 0.857 (0.037) & 0.949 (0.019) & 0.917 (0.917) & 0.915 (0.009) & 0.917 (0.003) \\ \hline  & -2.78 & 0.602 (0.269) & 0.791 (0.010) & 0.867 (0.027) & 0.920 (0.017) & 0.844 (0.029) \\ ncl2 & -1.760 & 0.692 (0.046) & 0.641 (0.042) & 0.942 (0.945) & 0.944 (0.056) & 0.950 (0.007) \\  & 208 & 0.944 (0.015) & 0.956 (0.011) & 0.958 (0.935) & 0.956 (0.012) & 0.960 (0.009) \\ \hline  & -2.78 & 0.327 (0.060) & 0.619 (0.171) & 0.721 (0.172) & 0.778 (0.018) & 0.601 (0.022) \\ Subj & -1.60 & 0.669 (0.091) & 0.566 (0.141) & 0.988 (0.958) & 0.758 (0.116) & 0.606 (0.100) \\  & 208 & 0.578 (0.119) & 0.578 (0.147) & 0.721 (0.712) & 0.778 (0.025) & 0.740 (0.025) \\ \hline  & -2.78 & 0.431 (0.057) & 0.387 (0.035) & 0.666 (0.546) & 0.535 (0.077) & 0.547 (0.075) \\ TREC & -1.60 & 0.416 (0.078) & 0.638 (0.066) & 0.531 (0.353) & 0.564 (0.010) & 0.564 (0.013) \\  & 208 & 0.418 (0.055) & 0.578 (0.572) & 0.572 (0.032) & 0.604 (0.047) & 0.668 (0.057) \\ \hline NLT Task & - & Outg. & CC & DC/Umean & DC/Uint & ICC \\ \hline  & -2.78 & 0.221 (0.035) & 0.288 (0.035) & 0.267 (0.260) & 0.255 (0.166) & 0.240 (0.050) \\ ANLJ & -1.60 & -2.566 (0.12) & 0.236 (0.260) & 0.267 (0.267) & 0.252 (0.044) & -0.394 (0.109) \\  & 208 & 0.237 (0.039) & 0.299 (0.058) & 0.196 (0.199) & 0.164 (0.023) & 0.266 (0.017) \\ \hline  & -3.04 (0.024) & 0.253 (0.253) & 0.253 (0.173) & 0.460 (0.066) & 0.417 (0.079) \\ WNLJ & -1.760 & -0.304 (0.005) & 0.408 (0.055) & 0.416 (0.414) & 0.444 (0.055) & 0.497 (0.044) \\  & 208 & 0.344 (0.023) & 0.353 (0.358) & 0.358 (0.359) & 0.391 (0.046) & 0.465 (0.079) \\ \hline  & -2.78 & 0.385 (0.110) & 0.392 (0.174) & 0.497 (0.047) & 0.384 (0.079) & 0.492 (0.060) \\ RTE & -1.60 & -0.495 (0.058) & 0.429 (0.023) & 0.535 (0.353) & 0.488 (0.099) & 0.580 (0.049) \\  & 208 & 0.505 (0.050) & 0.438 (0.034) & 0.477 (0.477) & 0.478 (0.0355) & 0.566 (0.021) \\ \hline  & -2.78 & 0.269 (0.059) & 0.189 (0.116) & 0.265 (0.256) & 0.265 (0.153) & 0.348 (0.107) \\ CN & -1.60 & -0.687 (0.071) & 0.458 (0.116) & 0.271 (0.271) & 0.256 (0.079) & 0.484 (0.095) \\  & 208 & 0.312 (0.030) & 0.188 (0.12) & 0.291 (0.291) & 0.229 (0.159) & 0.539 (0.076) \\ SICK & -2.78 & 0.526 (0.125) & 0.313 (0.024) & 0.292 (0.32) & 0.356 (0.107) & 0.425 (0.102) \\ \hline  & -1.60 & -0.366 (0.056)



\begin{table}
\begin{tabular}{c c r r r r r r} \hline Dataset & Size & \multicolumn{3}{c}{Method} \\ \hline Sentiment Task & \multicolumn{1}{r}{Orig.} & CC & DC w/Temn & DC w/Test & ICC \\ \hline \multirow{2}{*}{AGNews} & 7B & 0.803 (0.010) & 0.790 (0.013) & 0.753 (0.753) & 0.759 (0.016) & 0.822 (0.018) \\ \cline{2-7}  & 7B & 0.740 (0.064) & 0.733 (0.035) & 0.730 (0.730) & 0.730 (0.026) & 0.764 (0.047) \\ \hline \multirow{2}{*}{CR} & 7B & 0.892 (0.039) & 0.926 (0.010) & 0.929 (0.929) & 0.927 (0.007) & 0.938 (0.013) \\ \cline{2-7}  & 1B & 0.928 (0.023) & 0.934 (0.013) & 0.932 (0.923) & 0.931 (0.015) & 0.936 (0.015) \\ \hline \multirow{2}{*}{financial_phrasebank} & 7B & 0.665 (0.192) & 0.426 (0.066) & 0.515 (0.515) & 0.588 (0.102) & 0.720 (0.082) \\ \cline{2-7}  & 1B & 0.531 (0.141) & 0.598 (0.066) & 0.744 (0.744) & 0.752 (0.037) & 0.772 (0.029) \\ \hline \multirow{2}{*}{pcom\_entiment} & 7B & 0.408 (0.089) & 0.936 (0.049) & 0.368 (0.083) & 0.509 (0.040) & 0.544 (0.061) \\ \hline \multirow{2}{*}{MR} & 7B & 0.431 (0.138) & 0.415 (0.073) & 0.404 (0.040) & 0.531 (0.084) & 0.575 (0.066) \\ \cline{2-7}  & 7B & 0.928 (0.013) & 0.942 (0.009) & 0.959 (0.939) & 0.940 (0.010) & 0.943 (0.086) \\ \hline \multirow{2}{*}{st2} & 7B & 0.950 (0.018) & 0.971 (0.009) & 0.960 (0.960) & 0.967 (0.009) & 0.965 (0.007) \\ \cline{2-7}  & 1B & 0.964 (0.007) & 0.970 (0.005) & 0.957 (0.967) & 0.968 (0.004) & 0.971 (0.001) \\ \hline \multirow{2}{*}{Subj} & 7B & 0.608 (0.146) & 0.699 (0.159) & 0.628 (0.628) & 0.674 (0.065) & 0.800 (0.086) \\ \hline \multirow{2}{*}{TREC} & 7B & 0.457 (0.066) & 0.471 (0.056) & 0.401 (0.401) & 0.449 (0.052) & 0.537 (0.029) \\ \hline \multirow{2}{*}{NLI Tack} & 7B & 0.521 (0.074) & 0.573 (0.755) & 0.539 (0.539) & 0.573 (0.074) & 0.592 (0.053) \\ \hline \multirow{2}{*}{ANLI} & Orig. & CC & DC w/Temn & DC w/Total & ICC \\ \hline ANLI & 7B & 0.291 (0.053) & 0.344 (0.057) & 0.324 (0.324) & 0.337 (0.026) & 0.350 (0.059) \\ \hline \multirow{2}{*}{WNLI} & 7B & 0.379 (0.032) & 0.385 (0.017) & 0.324 (0.324) & 0.327 (0.033) & 0.419 (0.040) \\ \hline \multirow{2}{*}{WEL} & 7B & 0.456 (0.081) & 0.543 (0.112) & 0.450 (0.450) & 0.481 (0.099) & 0.570 (0.046) \\ \hline \multirow{2}{*}{RTE} & 7B & 0.574 (0.063) & 0.642 (0.056) & 0.659 (0.659) & 0.721 (0.037) & 0.729 (0.015) \\ \hline \multirow{2}{*}{CB} & 7B & 0.454 (0.023) & 0.531 (0.051) & 0.392 (0.392) & 0.448 (0.115) & 0.557 (0.020) \\ \hline \multirow{2}{*}{CBS} & 1B & 0.580 (0.025) & 0.624 (0.049) & 0.614 (0.641) & 0.598 (0.012) & 0.652 (0.073) \\ \hline \multirow{2}{*}{SICK} & 7B & 0.315 (0.138) & 0.485 (0.113) & 0.349 (0.349) & 0.430 (0.11) & 0.537 (0.068) \\ \hline \multirow{2}{*}{Detection Task} & 1B & 0.534 (0.091) & 0.587 (0.118) & 0.436 (0.436) & 0.463 (0.07) & 0.669 (0.052) \\ \hline \multirow{2}{*}{social_base,frames} & Orig. & CC & DC w/Temn & DC w/Total & ICC \\ \hline \multirow{2}{*}{tworet\_eval\_stance\_shreisen} & 7B & 0.365 (0.004) & 0.734 (0.029) & 0.687 (0.687) & 0.756 (0.018) & 0.741 (0.018) \\ \hline \multirow{2}{*}{tworet\_eval\_stance\_fminist} & 1B & 0.359 (0.008) & 0.620 (0.093) & 0.753 (0.753) & 0.751 (0.011) & 0.753 (0.012) \\ \hline \multirow{2}{*}{tworet\_eval\_stance\_sfmentists} & 7B & 0.414 (0.000) & 0.178 (0.072) & 0.140 (0.140) & 0.146 (0.070) & 0.226 (0.032) \\ \hline \multirow{2}{*}{tworet\_eval\_stance\_feminist} & 1B & 0.204 (0.056) & 0.194 (0.089) & 0.195 (0.195) & 0.186 (0.018) & 0.201 (0.031) \\ \hline \multirow{2}{*}{tworet\_eval\_stance\_feminist} & 7B & 0.374 (0.059) & 0.401 (0.011) & 0.395 (0.395) & 0.411 (0.039) & 0.499 (0.035) \\ \hline \multirow{2}{*}{tworet\_eval\_stance\_fminist} & 1B & 0.600 (0.040) & 0.412 (0.028) & 0.217 (0.417) & 0.413 (0.035) & 0.560 (0.037) \\ \hline \multirow{2}{*}{tworet\_eval\_tave} & 7B & 0.412 (0.113) & 0.330 (0.033) & 0.515 (0.515) & 0.601 (0.046) & 0.575 (0.034) \\ \hline \multirow{2}{*}{tworet\_eval\_tovey} & 1B & 0.455 (0.078) & 0.446 (0.105) & 0.517 (0.517) & 0.557 (0.026) & 0.536 (0.068) \\ \hline \multirow{2}{*}{tworet\_eval\_trovey} & 7B & 0.238 (0.038) & 0.440 (0.046) & 0.526 (0.526) & 0.631 (0.072) & 0.667 (0.04) \\ \hline \multirow{2}{*}{tworet\_eval\_tovey} & 1B & 0.238 (0.085) & 0.





\begin{table}
\begin{tabular}{c c r r r r r r} \hline Dataset & Size & Method & & & \\ \hline Sentiment Task & & Orig. & CC & DC/Werno & DC w/Test & ICC \\ \hline \multirow{2}{*}{AGNews} & 7B & 0.734 (0.059) & 0.774 (0.014) & 0.773 (0.773) & 0.767 (0.040) & 0.784 (0.042) \\  & 7B & 0.765 (0.018) & 0.745 (0.019) & 0.778 (0.778) & 0.772 (0.029) & 0.774 (0.030) \\ \hline \multirow{2}{*}{CR} & 7B & 0.926 (0.009) & 0.929 (0.007) & 0.924 (0.924) & 0.927 (0.011) & 0.930 (0.008) \\  & 7B & 0.919 (0.021) & 0.928 (0.012) & 0.928 (0.029) & 0.928 (0.012) & 0.937 (0.010) \\ \hline \multirow{2}{*}{financial_phrasebank} & 7B & 0.869 (0.052) & 0.574 (0.058) & 0.759 (0.759) & 0.774 (0.049) & 0.877 (0.040) \\  & 7B & 0.720 (0.119) & 0.708 (0.075) & 0.860 (0.866) & 0.790 (0.047) & 0.829 (0.041) \\ \hline \multirow{2}{*}{pcom, sentiment} & 7B & 0.562 (0.010) & 0.534 (0.017) & 0.527 (0.527) & 0.526 (0.060) & 0.575 (0.079) \\  & 7B & 0.464 (0.054) & 0.534 (0.019) & 0.472 (0.472) & 0.521 (0.021) & 0.521 (0.042) \\ \hline \multirow{2}{*}{MR} & 7B & 0.923 (0.024) & 0.930 (0.099) & 0.931 (0.931) & 0.928 (0.014) & 0.951 (0.099) \\  & 7B & 0.926 (0.011) & 0.931 (0.009) & 0.928 (0.025) & 0.928 (0.011) & 0.929 (0.005) \\ \hline \multirow{2}{*}{stl} & 7B & 0.951 (0.004) & 0.954 (0.015) & 0.957 (0.957) & 0.962 (0.005) & 0.965 (0.007) \\  & 7B & 0.955 (0.006) & 0.954 (0.006) & 0.954 (0.054) & 0.955 (0.006) & 0.965 (0.007) \\ \hline \multirow{2}{*}{Subj} & 7B & 0.334 (0.011) & 0.926 (0.174) & 0.671 (0.671) & 0.673 (0.163) & 0.689 (0.134) \\ \cline{2-6}  & 7B & 0.425 (0.037) & 0.431 (0.036) & 0.486 (0.486) & 0.494 (0.037) & 0.460 (0.022) \\ \hline \multirow{2}{*}{TREC} & 13B & 0.669 (0.052) & 0.741 (0.036) & 0.789 (0.079) & 0.788 (0.042) & 0.712 (0.040) \\ \hline NLI Tack & & Orig. & CC & DC w/Dermo & DC w/Test & ICC \\ \hline ANLJ & 7B & 0.429 (0.036) & 0.446 (0.022) & 0.439 (0.439) & 0.440 (0.035) & 0.469 (0.013) \\ \hline \multirow{2}{*}{WNLI} & 7B & 0.416 (0.033) & 0.385 (0.385) & 0.385 (0.11) & 0.413 (0.012) \\ \hline WNLJ & 7B & 0.652 (0.046) & 0.716 (0.039) & 0.675 (0.675) & 0.686 (0.043) & 0.721 (0.027) \\ \hline \multirow{2}{*}{RTE} & 7B & 0.594 (0.096) & 0.630 (0.085) & 0.630 (0.630) & 0.594 (0.076) & 0.671 (0.033) \\ \hline \multirow{2}{*}{RITE} & 7B & 0.758 (0.096) & 0.756 (0.012) & 0.751 (0.751) & 0.758 (0.099) & 0.770 (0.007) \\  & 7B & 0.750 (0.097) & 0.756 (0.011) & 0.752 (0.752) & 0.723 (0.018) & 0.757 (0.007) \\ \hline \multirow{2}{*}{CB} & 7B & 0.585 (0.074) & 0.610 (0.095) & 0.597 (0.597) & 0.586 (0.091) & 0.691 (0.063) \\ \hline \multirow{2}{*}{SICK} & 13B & 0.704 (0.032) & 0.669 (0.040) & 0.714 (0.741) & 0.671 (0.065) & 0.715 (0.044) \\ \hline \multirow{2}{*}{Detection Task} & 7B & 0.460 (0.053) & 0.487 (0.044) & 0.491 (0.41) & 0.481 (0.036) & 0.533 (0.03) \\ \hline \multirow{2}{*}{social_bias_frames} & 7B & 0.510 (0.065) & 0.445 (0.039) & 0.397 (0.397) & 0.408 (0.014) & 0.537 (0.041) \\ \hline \multirow{2}{*}{tweet\_eval\_stunc\_shreinsit} & 7B & 0.464 (0.011) & 0.743 (0.024) & 0.777 (0.777) & 0.776 (0.014) & 0.777 (0.017) \\ \hline \multirow{2}{*}{tweet\_eval\_stunc\_shreinsit} & 7B & 0.369 (0.013) & 0.713 (0.038) & 0.758 (0.758) & 0.765 (0.026) & 0.771 (0.021) \\ \hline tweet\_eval\_stunc\_shreinsit} & 7B & 0.496 (0.019) & 0.169 (0.021) & 0.192 (0.192) & 0.194 (0.048) & 0.233 (0.080) \\ \hline \multirow{2}{*}{tweet\_eval\_stunc\_feminist} & 13B & 0.224 (0.030) & 0.190 (0.028) & 0.197 (0.197) & 0.138 (0.097) & 0.260 (0.068) \\ \hline tweet\_eval\_stune\_feminist} & 7B & 0.496 (0.072) & 0.402 (0.022) & 0.412 (0.418) & 0.418 (0.031) & 0.551 (0.021) \\ \hline \multirow{2}{*}{tweet\_eval\_stune\_freqiry} & 13B & 0.578 (0.033) & 0.479 (0.040) & 0.524 (0.524) & 0.496 (0.046) & 0.596 (0.032) \\ \hline tweet\_eval\_stune\_freqiry} & 7B & 0.481 (0.077) & 0.476 (0.093) & 0.521 (0.611) & 0.611 (0.077) & 0.561 (0.079) \\ \hline \multirow{2}{*}{retweet\_eval\_stune\_freqiry} & 13B & 0.431 (0.081) & 0.368 (0.062) & 0.419 (0.419) & 0.513 (0.060) & 0.504 (0.023) \\ \hline tweet\_eval\_proxy} & 7B & 0.283 (0.080) & 0.524 (0.065) & 0.607 (0



\begin{table}
\begin{tabular}{c c c c c c c} \hline Dataset & Size & Method & Method & \\ \hline Sentiment Task & & Orig. & CC & DC/Wense & DC/West & ICC \\ \hline  & 2.78 & 0.193(-0.058) & 0.167(-0.037) & 0.187(-0.187) & 0.195(-0.043) & 0.261(-0.002) \\ AGNews & 4.78 & 0.184(-0.063) & 0.143(-0.043) & 0.228(-0.228) & 0.226(-0.045) & 0.305(-0.009) \\  & 1.38 & 0.057(-0.017) & 0.158(-0.048) & 0.193(-0.195) & 0.240(-0.029) & 0.311(-0.047) \\ \hline  & 2.78 & 0.038(-0.035) & 0.037(-0.000) & 0.065(-0.036) & 0.062(-0.046) & 0.305(-0.009) \\ CR & **6.78** & 0.578(-0.255) & 0.078(-0.000) & 0.048(-0.044) & 0.848(-0.087) & 0.084(-0.069) \\  & 1.18 & 0.087(-0.180) & 0.041(-0.097) & 0.875(-0.087) & 0.900(-0.019) & 0.884(-0.077) \\  & 2.78 & 0.339(-0.046) & 0.307(-0.077) & 0.298(-0.028) & 0.302(-0.055) & 0.377(-0.069) \\ financial_pltrextbank & **6.78** & 0.216(-0.058) & 0.305(-0.058) & 0.212(-0.122) & 0.403(-0.009) & 0.393(-0.091) \\  & 1.38 & 0.396(-0.067) & 0.376(-0.15) & 0.263(-0.025) & 0.405(-0.037) & 0.433(-0.066) \\  & 2.78 & 0.295(-0.066) & 0.294(-0.025) & 0.242(-0.024) & 0.301(-0.077) & 0.309(-0.046) \\ poem, sentiment & **6.78** & 0.224(-0.049) & 0.297(-0.039) & 0.279(-0.029) & 0.300(-0.022) & 0.362(-0.037) \\  & 1.78 & 0.175(-0.025) & 0.260(-0.015) & 0.259(-0.259) & 0.371(-0.062) & 0.319(-0.062) \\ \hline  & 2.78 & 0.325(-0.122) & 0.330(-0.049) & 0.858(-0.085) & 0.090(-0.019) & 0.893(-0.033) \\ MR & **6.78** & 0.039(-0.143) & 0.949(-0.091) & 0.838(-0.038) & 0.918(-0.011) & 0.912(-0.066) \\  & 1.38 & 0.215(-0.232) & 0.347(-0.112) & 0.280(-0.080) & 0.902(-0.081) & 0.880(-0.088) \\  & 2.78 & 0.555(-0.275) & 0.533(-0.326) & 0.650(-0.060) & 0.624(-0.037) & 0.609(-0.410) \\ set2 & **7.88** & 0.503(-0.241) & 0.428(-0.056) & 0.596(-0.596) & 0.590(-0.197) & 0.624(-0.174) \\  & 1.18 & 0.578(-0.265) & 0.659(-0.252) & 0.676(-0.596) & 0.695(-0.296) & 0.577(-0.258) \\ \hline  & 2.78 & 0.201(-0.241) & 0.427(-0.273) & 0.273(-0.728) & 0.765(-0.123) & 0.801(-0.031) \\ Subj & 6.78 & 0.520(-0.176) & 0.696(+0.128) & 0.694(-0.049) & 0.699(-0.092) & 0.789(-0.098) \\  & 1.18 & 0.425(-0.146) & 0.612(-0.164) & 0.761(-0.051) & 0.754(-0.106) & 0.677(-0.138) \\  & 2.78 & 0.114(-0.075) & 0.538(-0.156) & 0.260(-0.266) & 0.244(-0.064) \\ TRRC & **6.78** & 0.188(-0.038) & 0.088(-0.015) & 0.270(-0.27) & 0.268(-0.048) & 0.240(-0.041) \\  & 1.18 & 0.09 (-0.052) & 0.157(-0.081) & 0.176(-0.17) & 0.185(-0.125) & 0.226(-0.101) \\ \hline NLI Task & & Orig. & CC & DC/Wense & DC/West & ICC \\ \hline  & 2.78 & 0.226(-0.046) & 0.247(-0.046) & 0.247(-0.247) & 0.247(-0.037) & 0.219(-0.039) & 0.303(-0.039) \\ ANLJ & **6.78** & 0.199(-0.033) & 0.294(-0.031) & 0.267(-0.267) & 0.242(-0.045) & 0.296(-0.026) \\  & 1.18 & 0.203(-0.038) & 0.240(-0.030) & 0.259(-0.295) & 0.245(-0.028) & 0.265(-0.041) \\  & 2.78 & 0.377(-0.072) & 0.299(-0.035) & 0.331(-0.33) & 0.399(-0.036) & 0.362(-0.059) \\ WNLI & **6.78** & 0.318(-0.035) & 0.305(-0.030) & 0.341(-0.341) & 0.366(-0.009) & 0.433(-0.055) \\  & 1.18 & 0.038(-0.043) & 0.392(-0.073) & 0.317(-0.373) & 0.376(-0.044) & 0.434(-0.034) \\ \hline  & 2.78 & 0.415(-0.078) & 0.411(-0.075) & 0.413(-0.413) & 0.389(-0.047) & 0.447(-0.033) \\ RITE & **6.78** & 0.435(-0.078) & 0.306(-0.082) & 0.427(-0.427) & 0.427(-0.042) & 0.403(-0.032) \\  & 1.18 & 0.397(-0.088) & 0.492(-0.046) & 0.469(-0.069) & 0.469(-0.07) & 0.099(-0.035) \\ \hline  & 2.78 & 0.257(-0.103) & 0.211(-0.074) & 0.246(-0.266) & 0.279(-0.148) & 0.273(-0.099) \\ CB & **6.78** & 0.257(-0.066) & 0.477(-0.065) & 0.513(-0.32) & 0.322(-0.075) & 0.418(-0.077) \\  & 1.18 & 0.270(-0.046) & 0.365(-0.115) & 0.294(-0.294) & 0.366(-0.097) & 0.428(-0.058) \\ SICK & **6.78** & 0.255(-0.075) & 0.290(-0.031) & 0.296(-0.296) & 0.314(-0.090) & 0.343(-0.03) \\  & 1.18 & 0.281(-0.037) & 0.266(-0.069)



\begin{table}
\begin{tabular}{c c c c c c c c} Dataset & Size & Method & Method & & & \\ \hline Sentiment Rate & & Orig. & CC & DC/W-mean & DC at/Test & ICC \\ \hline  & 2.78 & 0.15 (0.043) & 0.161 (0.039) & 0.299 (0.299) & 0.349 (0.193) & 0.366 (0.055) \\ AGNus & 1.61 (0.295) & 0.25 (0.023) & 0.197 (0.048) & 0.218 (0.239) & 0.237 (0.046) & 0.314 (0.046) \\  & 2.08 & 0.29 (0.030) & 0.206 (0.041) & 0.239 (0.239) & 0.281 (0.048) & 0.341 (0.034) \\ \multirow{2}{*}{CR} & 2.78 & 0.63 (0.176) & 0.77 (0.077) & 0.727 (0.275) & 0.649 (0.053) & 0.538 (0.043) \\  & 1.68 (0.096) & 0.163 (0.183) & 0.125 (0.183) & 0.379 (0.389) & 0.589 (0.057) & 0.593 (0.033) \\  & 2.08 & 0.17 (0.159) & 0.689 (0.044) & 0.360 (0.360) & 0.866 (0.022) & 0.899 (0.052) \\ \multirow{2}{*}{functual_plarestbank} & 2.78 & 0.91 (0.117) & 0.566 (0.021) & 0.466 (0.046) & 0.476 (0.023) & 0.432 (0.120) \\  & 1.68 (0.358) & 0.097 (0.423) & 0.071 (0.405) & 0.405 (0.405) & 0.498 (0.039) & 0.566 (0.033) \\ \multirow{2}{*}{functual_plarestbank} & 2.08 & 0.19 (0.042) & 0.350 (0.030) & 0.361 (0.341) & 0.366 (0.074) & 0.410 (0.041) \\  & 2.78 & 0.22 (0.075) & 0.247 (0.283) & 0.228 (0.250) & 0.200 (0.043) \\  & 2.68 (0.23) & 0.061 (0.066) & 0.228 (0.285) & 0.246 (0.036) & 0.266 (0.024) \\ \multirow{2}{*}{gcom,setminet} & 2.08 & 0.29 (0.052) & 0.266 (0.012) & 0.302 (0.354) & 0.364 (0.037) & 0.367 (0.017) \\ \multirow{2}{*}{MR} & 2.78 & 0.50 (0.180) & 0.084 (0.034) & 0.230 (0.034) & 0.520 (0.034) & 0.865 (0.044) \\  & 1.76 (0.011) & 0.683 (0.037) & 0.689 (0.038) & 0.689 (0.035) & 0.506 (0.039) \\ \multirow{2}{*}{srl} & 2.08 & 0.439 (0.247) & 0.680 (0.035) & 0.518 (0.034) & 0.687 (0.035) & 0.506 (0.039) \\  & 2.78 & 0.39 (0.016) & 0.537 (0.17) & 0.640 (0.046) & 0.663 (0.212) & 0.618 (0.101) \\ \multirow{2}{*}{srl2} & 2.76 & 0.638 (0.264) & 0.562 (0.644) & 0.625 (0.036) & 0.686 (0.271) \\  & 2.08 & 0.68 (0.233) & 0.681 (0.031) & 0.593 (0.039) & 0.508 (0.032) & 0.515 (0.077) \\ \multirow{2}{*}{Sulbj} & 2.78 & 0.37 (0.016) & 0.577 (0.196) & 0.475 (0.174) & 0.576 (0.039) & 0.576 (0.079) \\  & 1.68 (0.417) & 0.417 (0.137) & 0.369 (0.140) & 0.579 (0.570) & 0.501 (0.137) & 0.653 (0.074) \\ \multirow{2}{*}{TREC} & 2.08 & 0.522 (0.017) & 0.366 (0.127) & 0.572 (0.127) & 0.572 (0.037) & 0.613 (0.115) \\ \multirow{2}{*}{CIN} & 2.78 & 0.166 (0.091) & 0.236 (0.034) & 0.239 (0.239) & 0.288 (0.037) & 0.318 (0.068) \\  & 2.08 & 0.174 (0.048) & 0.170 (0.045) & 0.145 (0.145) & 0.169 (0.029) & 0.226 (0.040) \\  & 2.08 & 0.21 (0.077) & 0.215 (0.122) & 0.212 (0.038) & 0.300 (0.079) & 0.200 (0.079) \\ \multirow{2}{*}{ANLJ} & 2.08 & 0.04 (0.152) & 0.04 (0.043) & 0.220 (0.251) & 0.218 (0.042) & 0.217 (0.033) \\  & 2.08 & 0.213 (0.036) & 0.196 (0.034) & 0.284 (0.247) & 0.247 (0.030) & 0.248 (0.019) \\ \multirow{2}{*}{WNVL} & 2.78 & 0.31 (0.026) & 0.183 (0.026) & 0.267 (0.327) & 0.235 (0.038) & 0.462 (0.031) \\  & 2.08 & 0.33 (0.035) & 0.386 (0.039) & 0.322 (0.392) & 0.309 (0.016) & 0.366 (0.035) \\  & 2.08 & 0.34 (0.000) & 0.304 (0.032) & 0.333 (0.333) & 0.315 (0.015) & 0.415 (0.034) \\ \multirow{2}{*}{RTE} & 2.78 & 0.41 (0.076) & 0.663 (0.036) & 0.370 (0.370) & 0.366 (0.013) & 0.467 (0.046) \\  & 1.61 (0.045) & 0.055 (0.035) & 0.452 (0.038) & 0.355 (0.037) & 0.373 (0.046) & 0.466 (0.077) \\  & 208 & 0.456 (0.091) & 0.381 (0.037) & 0.473 (0.473) & 0.411 (0.0465) & 0.577 (0.069) \\ \multirow{2}{*}{CN} & 2.78 & 0.26 (0.31) & 0.275 (0.036) & 0.226 (0.226) & 0.251 (0.035) & 0.256 (0.066) \\  & 1.68 (0.318) & 0.143 (0.143) & 0.199 (0.014) & 0.267 (0.260) & 0.250 (0.130) & 0.310 (0.113) \\ \multirow{2}{*}{SNCK} & 20.08 & 0.274 (0.00) & 0.401 (0.04) & 0.266 (0.246) & 0.238 (0.037) & 0.315 (0.073) \\  & 2.78 & 0.31 (0.038) & 0.307 (0.038) & 0.396 (0.03) & 0.322 (0.359) & 0.349 (0.070) \\ \multirow{2}{*}{SCK} & 1.68



\begin{table}
\begin{tabular}{c c r r r r r r} \hline Dataset & Size & Method & & & \\ \hline Sentiment Task & & Orig. & CC & Dev/Memo & DC w/Test & ICC \\ \hline \multirow{2}{*}{AGNews} & 7B & 0.377 (0.045) & 0.317 (0.068) & 0.344 (0.344) & 0.334 (0.088) & 0.462 (0.019) \\  & 7B & 0.381 (0.053) & 0.315 (0.056) & 0.320 (0.320) & 0.297 (0.064) & 0.488 (0.062) \\ \hline \multirow{2}{*}{CR} & 7B & 0.920 (0.016) & 0.602 (0.116) & 0.966 (0.906) & 0.928 (0.010) & 0.932 (0.050) \\  & 13B & 0.623 (0.163) & 0.569 (0.204) & 0.774 (0.74) & 0.821 (0.041) & 0.823 (0.061) \\ \hline \multirow{2}{*}{financial_phrasebank} & 7B & 0.346 (0.199) & 0.408 (0.038) & 0.398 (0.398) & 0.383 (0.062) & 0.418 (0.093) \\  & 13B & 0.367 (0.095) & 0.354 (0.071) & 0.395 (0.395) & 0.368 (0.073) & 0.434 (0.101) \\ \hline \multirow{2}{*}{pem,setminent} & 7B & 0.230 (0.066) & 0.231 (0.068) & 0.221 (0.21) & 0.263 (0.039) & 0.304 (0.044) \\  & 13B & 0.268 (0.065) & 0.312 (0.068) & 0.249 (0.249) & 0.272 (0.039) & 0.324 (0.021) \\ \hline \multirow{2}{*}{MR} & 7B & 0.893 (0.067) & 0.919 (0.028) & 0.919 (0.919) & 0.933 (0.019) & 0.933 (0.013) \\  & 13B & 0.610 (0.257) & 0.471 (0.212) & 0.633 (0.633) & 0.678 (0.236) & 0.663 (0.268) \\ \hline \multirow{2}{*}{st2} & 7B & 0.839 (0.234) & 0.917 (0.057) & 0.876 (0.876) & 0.900 (0.101) & 0.916 (0.069) \\  & 13B & 0.657 (0.267) & 0.784 (0.111) & 0.893 (0.893) & 0.930 (0.034) & 0.938 (0.052) \\ \hline \multirow{2}{*}{Subj} & 7B & 0.577 (0.178) & 0.433 (0.172) & 0.498 (0.498) & 0.482 (0.110) & 0.706 (0.072) \\ \cline{2-6}  & 13B & 0.549 (0.186) & 0.414 (0.118) & 0.604 (0.064) & 0.522 (0.181) & 0.754 (0.085) \\ \hline \multirow{2}{*}{TREC} & **7B** & 0.242 (0.083) & 0.200 (0.099) & 0.262 (0.262) & 0.263 (0.075) & 0.286 (0.046) \\ \hline \multirow{2}{*}{NLI Tack} & 13B & 0.329 (0.093) & 0.328 (0.081) & 0.314 (0.344) & 0.331 (0.077) & 0.345 (0.114) \\ \hline NLI Tack & & Orig. & CC & DC w/Memo & DC w/Test & ICC \\ \hline ANLJ & 7B & 0.196 (0.065) & 0.255 (0.042) & 0.225 (0.225) & 0.223 (0.093) & 0.269 (0.041) \\ \hline \multirow{2}{*}{WNLI} & 13B & 0.196 (0.093) & 0.255 (0.055) & 0.229 (0.229) & 0.250 (0.053) & 0.260 (0.029) \\ \hline WNLJ & 7B & 0.352 (0.096) & 0.302 (0.003) & 0.303 (0.303) & 0.302 (0.003) & 0.440 (0.106) \\ \hline \multirow{2}{*}{RTE} & 7B & 0.408 (0.111) & 0.419 (0.125) & 0.438 (0.438) & 0.430 (0.120) & 0.667 (0.118) \\ \hline RITE & 7B & 0.571 (0.150) & 0.558 (0.066) & 0.556 (0.565) & 0.585 (0.079) & 0.645 (0.070) \\ \hline \multirow{2}{*}{CB} & 7B & 0.250 (0.111) & 0.286 (0.066) & 0.226 (0.226) & 0.231 (0.062) & 0.362 (0.027) \\ \hline \multirow{2}{*}{CBS} & 13B & 0.477 (0.064) & 0.526 (0.055) & 0.522 (0.522) & 0.521 (0.079) & 0.558 (0.028) \\ \hline SICK & 7B & 0.337 (0.114) & 0.284 (0.073) & 0.270 (0.270) & 0.269 (0.080) & 0.348 (0.086) \\ \hline Detection Task & 13B & 0.336 (0.127) & 0.350 (0.088) & 0.354 (0.354) & 0.348 (0.092) & 0.361 (0.084) \\ \hline \multirow{2}{*}{social_bias_frames} & 7B & 0.419 (0.085) & 0.367 (0.019) & 0.479 (0.479) & 0.498 (0.103) & 0.537 (0.077) \\ \hline \multirow{2}{*}{tweet\_eval\_stmore\_shreinsit} & 13B & 0.532 (0.134) & 0.568 (0.046) & 0.623 (0.623) & 0.640 (0.064) & 0.670 (0.070) \\ \cline{2-6}  & 7B & 0.366 (0.081) & 0.269 (0.074) & 0.411 (0.411) & 0.315 (0.088) & 0.449 (0.032) \\ \hline \multirow{2}{*}{tweet\_eval\_stmore\_feminist} & 13B & 0.403 (0.110) & 0.336 (0.078) & 0.387 (0.387) & 0.385 (0.089) & 0.614 (0.067) \\ \hline tweet\_eval\_stmore\_feminist & 7B & 0.302 (0.049) & 0.289 (0.087) & 0.354 (0.354) & 0.334 (0.060) & 0.425 (0.030) \\ \hline \multirow{2}{*}{tweet\_eval\_stmore\_feminist} & 13B & 0.421 (0.089) & 0.356 (0.052) & 0.404 (0.404) & 0.344 (0.125) & 0.481 (0.081) \\ \hline \multirow{2}{*}{tweet\_eval\_stmore\_freqiry} & 7B & 0.457 (0.111) & 0.302 (0.017) & 0.446 (0.446) & 0.438 (0.099) & 0.494 (0.067) \\ \hline \multirow{2}{*}{tweet\_eval\_stmore\_freqiry} & 13B & 0.461 (0.085) & 0.313 (0.032) & 0.404 (0.044) & 0.475 (0.071) & 0.499 (0.078) \\ \hline \multirow{2}{*}{tweet\_eval\_stmore\_freqiry} & 7B & 0.478 (0.078) & 0.285



\begin{table}
\begin{tabular}{c c r r r r r r} Dataset & Size & Method & & & \\ \hline Sentiment Task & & Onig. & CC & Dev/Memo & DC w/Test & ICC \\ \hline \multirow{2}{*}{AGNews} & 7B & 0.315 (0.108) & 0.330 (0.044) & 0.263 (0.263) & 0.320 (0.062) & 0.391 (0.042) \\ \cline{2-7}  & MSnews & 13B (0.4960, 1.341) & 0.413 (0.055) & 0.479 (0.479) & 0.574 (0.087) & 0.528 (0.138) \\ \hline \multirow{2}{*}{CR} & TR & 0.696 (0.160) & 0.525 (0.190) & 0.609 (0.609) & 0.609 (0.144) & 0.725 (0.114) \\ \cline{2-7}  & MSnews & 0.841 (0.080) & 0.713 (0.202) & 0.848 (0.548) & 0.540 (0.093) & 0.853 (0.033) \\ \hline \multirow{2}{*}{financial_phrasebank} & 7B & 0.218 (0.072) & 0.241 (0.112) & 0.344 (0.344) & 0.380 (0.103) & 0.388 (0.064) \\ \cline{2-7}  & MSnews & 13B (0.467 (0.110) & 0.462 (0.080) & 0.744 (0.474) & 0.531 (0.085) & 0.488 (0.068) \\ \hline \multirow{2}{*}{pcom\_sentiment} & 7B & 0.282 (0.073) & 0.245 (0.097) & 0.299 (0.299) & 0.341 (0.053) & 0.376 (0.070) \\ \hline \multirow{2}{*}{MR} & TR & 0.310 (0.036) & 0.212 (0.090) & 0.307 (0.307) & 0.351 (0.057) & 0.373 (0.045) \\ \hline \multirow{2}{*}{MR} & TR & 0.960 (0.032) & 0.867 (0.085) & 0.895 (0.893) & 0.892 (0.060) & 0.927 (0.055) \\ \hline \multirow{2}{*}{st2} & TR & 13B (0.800) & 0.022 (0.066) & 0.136 (0.136) & 0.779 (0.779) & 0.788 (0.176) & 0.833 (0.109) \\ \hline \multirow{2}{*}{st2} & TR & 0.951 (0.079) & 0.910 (0.050) & 0.952 (0.952) & 0.952 (0.013) & 0.959 (0.010) \\ \hline \multirow{2}{*}{Stbj} & TR & 0.744 (0.109) & 0.626 (0.127) & 0.828 (0.838) & 0.868 (0.035) & 0.844 (0.062) \\ \hline \multirow{2}{*}{Stbj} & TR & 0.644 (0.120) & 0.497 (0.194) & 0.586 (0.586) & 0.552 (0.184) & 0.655 (0.105) \\ \hline \multirow{2}{*}{TREC} & TR & 13B (0.555 (0.103)) & 0.527 (0.092) & 0.702 (0.02) & 0.522 (0.114) & 0.698 (0.142) \\ \hline \multirow{2}{*}{TREC} & TR & 0.191 (0.040) & 0.151 (0.046) & 0.173 (0.173) & 0.207 (0.054) & 0.237 (0.070) \\ \hline \multirow{2}{*}{NLT Tack} & 13B (0.380 (0.117)) & 0.389 (0.143) & 0.360 (0.360) & 0.417 (0.116) & 0.442 (0.104) \\ \hline \multicolumn{2}{l}{NLL Track} & Orig. & CC & DC w/Memo & DC w/Test & ICC \\ \hline ANLJ & 7B & 0.268 (0.026) & 0.268 (0.020) & 0.268 (0.268) & 0.266 (0.025) & 0.271 (0.022) \\ \hline \multirow{2}{*}{WNLI} & TR & 0.231 (0.048) & 0.462 (0.026) & 0.280 (0.280) & 0.280 (0.037) & 0.281 (0.048) \\ \hline \multirow{2}{*}{WNLI} & TR & 0.528 (0.153) & 0.477 (0.078) & 0.531 (0.531) & 0.494 (0.166) & 0.563 (0.133) \\ \hline \multirow{2}{*}{RTE} & TR & 0.513 (0.133) & 0.548 (0.140) & 0.562 (0.562) & 0.536 (0.128) & 0.596 (0.120) \\ \hline \multirow{2}{*}{RITE} & TR & 0.629 (0.082) & 0.633 (0.062) & 0.653 (0.635) & 0.666 (0.073) & 0.668 (0.040) \\ \hline \multirow{2}{*}{CB} & TR & 0.729 (0.051) & 0.731 (0.050) & 0.722 (0.722) & 0.726 (0.083) & 0.731 (0.050) \\ \hline \multirow{2}{*}{CB} & TR & 0.419 (0.058) & 0.337 (0.032) & 0.429 (0.429) & 0.368 (0.042) & 0.462 (0.052) \\ \hline \multirow{2}{*}{SICK} & TR & 0.256 (0.059) & 0.274 (0.048) & 0.255 (0.25) & 0.515 (0.051) & 0.597 (0.011) \\ \hline \multirow{2}{*}{Detection Task} & TR & 0.256 (0.089) & 0.272 (0.048) & 0.255 (0.25) & 0.266 (0.045) & 0.280 (0.076) \\ \hline \multirow{2}{*}{social_bias_frames} & TR & 0.255 (0.082) & 0.234 (0.055) & 0.237 (0.237) & 0.236 (0.072) & 0.264 (0.07) \\ \hline \multirow{2}{*}{bweet\_eval\_stance\_shtein} & One & CC & DC w/Memo & DC w/test & ICC \\ \hline \multirow{2}{*}{social_bias\_frames} & TR & 0.437 (0.099) & 0.471 (0.037) & 0.448 (0.448) & 0.451 (0.085) & 0.485 (0.091) \\ \hline \multirow{2}{*}{twet\_eval\_stance\_zhtein} & TR & 0.706 (0.091) & 0.677 (0.106) & 0.694 (0.064) & 0.665 (0.078) & 0.719 (0.048) \\ \hline \multirow{2}{*}{tweet\_eval\_stance\_feminist} & TR & 0.390 (0.050) & 0.489 (0.069) & 0.484 (0.448) & 0.465 (0.083) & 0.503 (0.072) \\ \hline \multirow{2}{*}{tweet\_eval\_stance\_feminist} & TR & 0.371 (0.103) & 0.393 (0.083) & 0.483 (0.483) & 0.417 (0.070) & 0.501 (0.055) \\ \hline \multirow{2}{*}{tweet\_eval\_stance\_feminist} & TR & 0.480 (0.037) & 0.443 (0.031) & 0.506 (0.050) & 0.435 (0.090) & 0.505 (0.063) \\ \hline \multirow{2}{*}{tweet\_eval\_hote} & TR & 0.468 (0.091) & 0.469 (0.076) & 0.434 (0.434) & 0.475 (0.088) & 0.496 (0.087)



\begin{table}
\begin{tabular}{c c c c c c c c} \hline Dataset & Size & \multicolumn{3}{c}{Method} \\ \hline Sentiment Task & & Orig. & CC & DC-Volume & DC+Unit & KC \\ \cline{2-7}  & CPT 468 (0.791) & 0.580(0.003) & 0.580(0.003) & 0.761(0.01) & 0.794(0.005) & 0.827(0.0173) \\ \multirow{2}{*}{AGNnews} & Lumin 130 (0.805) & 0.580(0.003) & 0.580(0.004) & 0.587(0.007) & 0.589(0.005) & 0.591(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.806) & 0.580(0.003) & 0.580(0.004) & 0.580(0.004) & 0.580(0.004) & 0.582(0.004) \\ \hline \multirow{2}{*}{CR} & CHT 468 (0.821) & 0.496(0.012) & 0.591(0.012) & 0.591(0.017) & 0.592(0.006) & 0.592(0.009) \\ \cline{2-7}  & Linear-ch 300 (0.803) & 0.593(0.002) & 0.592(0.002) & 0.592(0.003) & 0.591(0.011) & 0.593(0.007) \\ \hline \multirow{2}{*}{financial_plansearch} & CHT 468 (0.801) & 0.560(0.003) & 0.560(0.003) & 0.426(0.004) & 0.560(0.003) & 0.562(0.003) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.003) & 0.592(0.002) & 0.592(0.003) & 0.591(0.007) & 0.592(0.007) \\ \hline \multirow{2}{*}{financial_plansearch} & CHT 468 (0.801) & 0.560(0.003) & 0.560(0.003) & 0.426(0.004) & 0.560(0.003) & 0.562(0.003) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.003) & 0.590(0.004) & 0.590(0.004) & 0.592(0.003) & 0.591(0.004) \\ \hline \multirow{2}{*}{preon_gentimeters} & CHT 468 (0.802) & 0.590(0.003) & 0.590(0.004) & 0.590(0.004) & 0.590(0.003) & 0.591(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.802) & 0.590(0.007) & 0.592(0.007) & 0.592(0.003) & 0.591(0.004) & 0.591(0.004) \\ \hline \multirow{2}{*}{MR} & CHT 468 (0.802) & 0.590(0.003) & 0.591(0.009) & 0.591(0.004) & 0.591(0.004) & 0.591(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.802) & 0.590(0.007) & 0.592(0.007) & 0.592(0.003) & 0.593(0.004) & 0.593(0.004) \\ \hline \multirow{2}{*}{wd2} & CHT 468 (0.801) & 0.590(0.003) & 0.591(0.003) & 0.596(0.003) & 0.592(0.003) & 0.596(0.002) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.003) & 0.591(0.003) & 0.596(0.004) & 0.596(0.004) & 0.596(0.002) \\ \hline \multirow{2}{*}{Shij} & CHT 468 (0.803) & 0.560(0.003) & 0.560(0.003) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.003) & 0.591(0.003) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \hline \multirow{2}{*}{TRICE} & CHT 468 (0.804) & 0.590(0.003) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \hline \multirow{2}{*}{NRI Task} & CHT 468 (0.802) & 0.590(0.005) & 0.591(0.009) & 0.591(0.004) & 0.591(0.004) & 0.591(0.004) \\ \hline \multirow{2}{*}{ANLI} & CHT 468 (0.803) & 0.590(0.002) & 0.591(0.009) & 0.592(0.003) & 0.593(0.004) & 0.593(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.002) & 0.599(0.007) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \hline \multirow{2}{*}{WNLI} & CHT 468 (0.804) & 0.590(0.004) & 0.591(0.003) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.004) & 0.596(0.007) & 0.596(0.005) & 0.596(0.004) & 0.596(0.004) \\ \hline \multirow{2}{*}{RTIE} & CHT 468 (0.804) & 0.590(0.004) & 0.596(0.007) & 0.596(0.007) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.003) & 0.597(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \hline \multirow{2}{*}{CR} & CHT 468 (0.804) & 0.590(0.004) & 0.597(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.004) & 0.597(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \hline \multirow{2}{*}{SCK} & CHT 468 (0.804) & 0.590(0.004) & 0.595(0.004) & 0.596(0.004) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.004) & 0.595(0.004) & 0.596(0.004) & 0.596(0.004) & 0.593(0.005) \\ \hline \multirow{2}{*}{secid_bus_frames} & CHT 468 (0.803) & 0.590(0.004) & 0.596(0.007) & 0.597(0.007) & 0.596(0.007) & 0.596(0.004) & 0.596(0.004) \\ \cline{2-7}  & Linear-ch 300 (0.804) & 0.590(0.007) & 0.596(0.007) & 0.596(0.007) & 0.5



\begin{table}
\begin{tabular}{c c c c c c c c} \hline Dataset & Size & Method & Method & & & \\ \hline Sentiment Task & & Orig. & CC & De-Wine & DC w/Test & KC \\ \cline{2-7}  & Off 6048 & 1,727 (0.065), 918 (0.077), 0.157 (1.057), 1.159 (0.033), 0.336 (0.077) & \\ AONsnews & Lumi 180 & 0.749 (1.11), 0.744 (0.077), 0.151 (0.035), 0.795 (0.079), 0.795 (0.065) & \\  & Limited-n 700 & 0.666 (0.005), 0.645 (0.000), 0.648 (0.000), 0.648 (0.000), 0.649 (0.000) & 0.640 (0.007) \\ \hline CR & CUT 608 & 0.768 (0.156), 0.038 (0.047), 0.019 (0.019), 0.037 (0.017), 0.013 (0.019) & \\  & Limited-n 700 & 0.267 (0.267), 0.722 (0.238), 0.030 (0.031), 0.031 (0.015), 0.036 (0.017) & \\  & Limited-n 700 & 0.912 (0.108), 0.046 (0.004), 0.077 (0.017), 0.319 (0.014), 0.016 (0.016) & \\ \hline \multirow{3}{*}{financial_glmecht} & GPT 604 & 0.819 (0.009), 0.049 (0.116), 0.542 (0.042), 0.549 (0.018), 0.546 (0.022) & \\  & Limited-n 700 & 0.919 (0.123), 0.066 (0.004), 0.546 (0.005), 0.546 (0.005) & 0.546 (0.012) & \\ \hline \multirow{3}{*}{financial_glmecht} & CUT 604 & 0.809 (0.113), 0.522 (0.066), 0.546 (0.005), 0.546 (0.005), 0.546 (0.017) & \\  & Limited-n 700 & 0.947 (0.137), 0.522 (0.125), 0.427 (0.042), 0.462 (0.015), 0.546 (0.021) & \\ \hline \multirow{3}{*}{preon_genstunter} & CUT 604 & 0.248 (0.046), 0.211 (0.005), 0.288 (0.028), 0.227 (0.066), 0.329 (0.067) & \\  & Limited-n 700 & 0.975 (0.038), 0.236 (0.014), 0.329 (0.037), 0.339 (0.004), 0.337 (0.004), 0.337 (0.004) & \\ \hline \multirow{3}{*}{MR} & CUT 604 & 0.88 (0.120), 0.061 (0.235), 0.546 (0.043), 0.546 (0.017), 0.546 (0.016) & 0.546 (0.016) \\  & Limited-n 700 & 0.900 (0.035), 0.031 (0.009), 0.096 (0.009), 0.319 (0.011), 0.018 (0.028) & \\ \hline \multirow{3}{*}{sw2} & CUT 604 & 0.68 (0.232), 0.798 (0.121), 0.866 (0.006), 0.860 (0.016), 0.861 (0.016), 0.862 (0.001) & \\  & Limited-n 700 & 0.900 (0.039), 0.743 (0.111), 0.864 (0.004), 0.922 (0.005), 0.916 (0.013) & \\ \hline \multirow{3}{*}{Subj} & CUT 604 & 0.55 (0.131), 0.662 (0.036), 0.659 (0.036), 0.657 (0.036), 0.654 (0.005), 0.654 (0.005) & \\  & Limited-n 700 & 0.699 (0.019), 0.699 (0.117), 0.722 (0.022), 0.646 (0.016), 0.725 (0.117) & \\ \hline \multirow{3}{*}{TRICE} & CUT 604 & 0.102 (0.004), 0.179 (0.005), 0.146 (0.014), 0.148 (0.015), 0.260 (0.007) & \\  & Limited-n 700 & 0.38 (0.037), 0.429 (0.010), 0.477 (0.43), 0.425 (0.015), 0.426 (0.014) & \\ \hline NLI Task & CUT 604 & 0.39 (0.006), 0.035 (0.006), 0.362 (0.02), 0.367 (0.049), 0.475 (0.025) & \\ \hline NLI Task & Oog. & Dec +