Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding
Authors: Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, Yancheng Yuan, Dacheng Tao
Accepted at: acl25

Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding

Keqin Peng1, Liang Ding2*, Yuanxin Ouyang1, Meng Fang3, Yancheng Yuan4, Dacheng Tao5
1Beihang University 2The University of Sydney 3University of Liverpool 4The Hong Kong Polytechnic University 5Nanyang Technological University
keqin.peng@buaa.edu.cn, liangding.liam@gmail.com

Abstract

Large language models (LLMs) excel at a range of tasks through in-context learning (ICL), where only a few task examples guide their predictions. However, prior research highlights that LLMs often overlook input-label mapping information in ICL, relying more on their pre-trained knowledge. To address this issue, we introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples. Experiments on 7 natural language understanding (NLU) tasks show that our ICCD method brings consistent and significant improvement (up to +1.8 improvement on average) upon 6 different scales of LLMs without requiring additional training. Our approach is versatile, enhancing performance with various demonstration selection methods, demonstrating its broad applicability and effectiveness. The code and scripts are released at https://github.com/Romainpkq/CD_ICL.

1 Introduction

In-context learning (ICL, Brown et al., 2020) is one of the most remarkable emergent capabilities of large language models (LLMs, Achiam et al., 2023; Dubey et al., 2024). By leveraging just a few carefully selected input-output examples, ICL enables models to adapt to new tasks without parameter updating (Dong et al., 2022; Peng et al., 2024). This approach has proven highly effective in unlocking the advanced capabilities of LLMs and has become a standard technique for tackling a spectrum of tasks, like translation, coding, and reasoning (Peng et al., 2023; Wang et al., 2025; Wibisono and Wang, 2024).

Previous studies (Pan et al., 2023; Wei et al., 2023) have identified two critical factors for successful ICL: task recognition (TR), which involves identifying the task from the demonstrations and utilizing prior knowledge to make predictions, and task learning (TL), which focuses on directly learning the input-label mappings from the demonstrations. However, ICL faces challenges in overcoming the biases introduced by pretraining (Kossen et al., 2024), and LLMs tend to underutilize input-label mapping information (Min et al., 2022). For example, in tasks like SST-2 (Socher et al., 2013b), the model may default to using its internal knowledge rather than learning the specific input-label mappings provided in the context.

To address this issue, we propose a simple yet effective method called in-context contrastive decoding (ICCD). Our method is inspired by the contrastive decoding technique (Li et al., 2023; Senrich et al., 2024; Kim et al., 2024; Zhong et al., 2024; Wang et al., 2024), which increases the probability of the desired output by suppressing undesired outputs, and our ICCD enhances the model’s attention to input-label mapping during generation. Specifically, we construct negative in-context examples by altering the inputs of the demonstrations, creating incorrect input-label mappings while keeping the labels unchanged. By comparing the output distributions between positive and negative examples, ICCD effectively emphasizes the correct input-label mappings, integrating this information into the original ICL process. Notably, our method works with any pretrained LLMs without requiring additional training.

Experimental results across seven natural language understanding tasks demonstrate that our ICCD strategy consistently and significantly improves performance upon several advanced LLMs, e.g., Llama-3.1, Llama-3.2, and Qwen2, across various datasets and model scales. Moreover, we show that ICCD can be seamlessly integrated with different demonstration selection methods, showcasing its robustness and universal applicability.
2 Methodology

2.1 Background
Given an input query \( x \), the probability of generating the target \( y \) using a casual LLM \( M \) parameterized by \( \theta \) can be formulated as follows:

\[
y \sim p_\theta(y \mid c, \mathcal{T}(x)),
\]

where \( \mathcal{T}(\cdot) \) is the template used to wrap up inputs and \( c = \mathcal{T}(x_1), \cdots, \mathcal{T}(x_k) \) is the context string concatenating \( k \) in-context examples, \( p_\theta(y \mid c, \mathcal{T}(x)) = \operatorname{softmax}[\logit_\theta(y \mid c, \mathcal{T}(x))] \) is the probability for the predicted token. For obtaining the desired \( y \), the regular decoding method is to choose the token with the highest probability (i.e., greedy decoding) or sampling from its distribution (e.g., top-k decoding).

Here, we can observe that there are two kinds of knowledge contributing to model prediction, models’ prior knowledge and input-label mapping information in in-context learning. However, LLMs usually prioritize prior knowledge over input-label mapping information (Kossen et al., 2024), leading to ICL’s struggle to fully overcome prediction preferences acquired from pre-training.

2.2 In-Context Contrastive Decoding
To mitigate the issue above, we construct negative in-context examples to factor out the input-label mapping from the models’ original output distribution contrastively. Specifically, in addition to the origin in-context examples \( c \), we construct negative in-context examples \( c^- \) with incorrect input-label mapping. We then subtract the negative output \( z_t^- \) from the positive output \( z_t \) to isolate the knowledge of input-label mapping. Finally, we integrate this knowledge with the original in-context learning to reinforce the importance of input-label mapping:

\[
y_t \sim \operatorname{softmax}(z_t + \alpha(z_t - z_t^-)),
\]

where \( \alpha \) is a hyperparameter that governs the importance of input-label mapping information. Equivalently,

\[
y_t \sim \tilde{p}_\theta(y|c, c^-, \mathcal{T}(x)) \\
\propto p_\theta(y|c, \mathcal{T}(x)) \left( \frac{p_\theta(y|c, \mathcal{T}(x))}{p_\theta(y|c^-, \mathcal{T}(x))} \right)^\alpha.
\]

Construction of \( c^- \). The negative in-context examples \( c^- \) is the key to the success of the in-context contrastive decoding method (ICCD). Considering the label bias (Zhao et al., 2021) of in-context learning, directly altering the labels of demonstrations may introduce a completely different label bias, potentially distorting the input-label mapping information. Hence, we adjust the inputs instead of the labels to change input-label mapping information. Specifically, for each demonstration \( (x_i, y_i) \), we first randomly select a different label \( y_j (y_j \neq y_i) \) from the label space. Then we randomly choose an input \( x_j \) whose label is \( y_j \) from the demonstrations pool to construct the negative demonstration \( (x_j, y_i) \). We compare the effect of different \( c^- \) in Section 5.

3 Experimental Setup

Models and Baselines. We perform experiments across different sizes of models, including Llama-series: Llama3.2-1B (1B), Llama3.2-3B (3B) and Llama3.1-8B (8B) (Dubey et al., 2024) and Qwen2 series: Qwen2-0.5B (0.5B), Qwen2-1.5B (1.5B) and Qwen2-7B (7B) (Yang et al., 2024), which are all widely-used decoder-only dense LMs. We also conduct experiments on extensive alignment models, e.g., Llama3.2-1B-Instruct, Llama3.2-3B-Instruct, and Llama3.1-8B-Instruct (Dubey et al., 2024) to verify the generalizability of our approach. For the baseline, we use the regular decoding methods following prior work (Shi et al., 2024; Zhao et al., 2024).

Demonstration Selection methods. To verify that our method is complementary to different demonstration selection methods, we mainly consider three different demonstration selection methods that do not require additional training.

• Random baseline randomly select in context examples for each testing sample.

• BM25 (Robertson et al., 2009) baseline uses BM25 to calculate the word-overlap similarity between samples and test input and select the high-similarity samples as demonstrations.

• TopK (Liu et al., 2022) baseline uses the nearest neighbors of a given test sample as the corresponding in-context examples.

Datasets and Metrics. We conduct a systematic study across 7 NLU tasks, including binary, multi-class classification tasks (SST-2, SST-5 (Socher et al., 2013a), CR, Subj (Wang et al., 2018)) and natural language inference tasks: MNLI (Williams
et al., 2018) and QNLI (Wang et al., 2018). We will report the accuracy to show the performance.

Experimental Details. Our method introduces a hyperparameter \( \alpha \) to control the input-label mapping information. For simplicity, we set \( \alpha = 1 \) for all models and settings. We ran all experiments 3 times with different random seeds and reported the average accuracies. We use 16-shot ICL for all models. Without a special statement, we report the results of the random selection method.

4 Main Results

We demonstrate the effectiveness of our method in 7 NLU tasks described in the Datasets and Metrics section. We summarize the results in Table 1, Table 2, Tabel 3, and Figure 1. Based on the results, we can find that:

Our method brings gain across different tasks and model scales. Results on Table 2 show that our method can achieve consistently better performance across the majority of tasks under different model scales than the regular decoding method. Specifically, our method brings over 1.0 improvements (in accuracy) in all Llama-series models and Qwen2-series models. It’s worth highlighting that ICCD brings +2.3 gains on average in the Qwen2-1.5B model. Furthermore, it is noteworthy that our approach can achieve more significant improvements in challenging tasks with the increase of model scale, such as QNLI and MNLI tasks, respectively bringing 5.1% (1.4%) and 1.8% (1.2%) gains compared to regular decoding in Llama3.1-8B (Qwen2-7B), demonstrating the effectiveness and universality of our method.

Our method consistently improves the performance with different in-context examples selection methods. Table 1 lists the average performance and standard deviation of different models with different demonstration selection methods. Clearly, our method can achieve better and stable performance with different demonstration selection methods. When the model scale increases, our method can achieve more improvement gains compared to the regular decoding method, +0.5 and +1.1 with BM25 method under Llama3.2-3B and Llama3.1-8B, respectively. These results prove that ICCD can be complementary with different demonstration selection methods.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Decoding</th>
    <th colspan="2">Random</th>
    <th colspan="2">BM25</th>
    <th colspan="2">TopK</th>
  </tr>
  <tr>
    <th>avg.</th>
    <th>std.</th>
    <th>avg.</th>
    <th>std.</th>
    <th>avg.</th>
    <th>std.</th>
  </tr>
  <tr>
    <td>Llama3.2-1B</td>
    <td>Regular</td>
    <td>66.1</td>
    <td>-</td>
    <td>72.5</td>
    <td>-</td>
    <td>73.6</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>68.3</b></td>
    <td>0.19</td>
    <td><b>72.9</b></td>
    <td>0.11</td>
    <td><b>73.4</b></td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Llama3.2-3B</td>
    <td>Regular</td>
    <td>72.9</td>
    <td>-</td>
    <td>76.6</td>
    <td>-</td>
    <td>76.7</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>74.6</b></td>
    <td>0.47</td>
    <td><b>77.1</b></td>
    <td>0.28</td>
    <td><b>76.9</b></td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>Regular</td>
    <td>77.6</td>
    <td>-</td>
    <td>79.7</td>
    <td>-</td>
    <td>80.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>79.4</b></td>
    <td>0.19</td>
    <td><b>80.8</b></td>
    <td>0.15</td>
    <td><b>80.9</b></td>
    <td>0.05</td>
  </tr>
</table>

Table 1: Average performance and standard deviation of 7 Natural Language Understanding (NLU) tasks with different in-context example selection methods. Red results indicate that our method brings improvement over the regular decoding, while Green results denote no improvement.

Our method works for aligned chat models. To verify the effectiveness of our method for the chat LLMs, we conducted experiments on different instruction-tuned and RLHF-tuned LLMs. Figure 1 show that our method can achieve consistent improvement in different chat models, demonstrating that our method also works for instruction-tuned and safety-enhanced models.

Our method works for a larger number of target classes. To verify the effectiveness of our method for a larger number of target classes, we conducted experiments on datasets TREC (6 target classes) and Dbpedia (14 target classes) with the random selection method. Results on Table 3 show that our method can achieve remarkable improvement, demonstrating the effectiveness of our method in larger target classes.

5 Analysis

To further explore the impact of different factors on the effectiveness of our method, we conduct further analysis with the Llama3.2-8B models.

Effects of Different Negative In-context Examples. As mentioned in Section 2.2, the choice of negative in-context examples is important to the performance of our methods. Here, we conduct contrastive experiments to analyze the impact of different negative examples. Specifically, we refer to the selected negative examples as Input, if the input-label mapping is altered by modifying the inputs of the demonstrations. Additionally, we construct another variant, Label, in which the labels of the demonstrations are changed. For comparison, we also include NULL, which does not use any negative demonstrations, similar to Shi et al. (2024). The results in Table 4 show that Input out-
<table>
  <tr>
    <th>Model</th>
    <th>Decoding</th>
    <th>SST2</th>
    <th>CR</th>
    <th>SST5</th>
    <th>Subj</th>
    <th>QNLI</th>
    <th>MNLI</th>
    <th>AG_NEWS</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Llama3.2-1B</td>
    <td>Regular</td>
    <td>89.8</td>
    <td>83.0</td>
    <td>43.7</td>
    <td>72.8</td>
    <td>53.5</td>
    <td>36.6</td>
    <td>83.3</td>
    <td>66.1</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>91.1</b></td>
    <td><b>83.7</b></td>
    <td><b>43.3</b></td>
    <td><b>83.0</b></td>
    <td><b>53.8</b></td>
    <td><b>39.2</b></td>
    <td><b>84.1</b></td>
    <td><b>68.3 (+2.1)</b></td>
  </tr>
  <tr>
    <td>Llama3.2-3B</td>
    <td>Regular</td>
    <td>93.7</td>
    <td>87.2</td>
    <td>46.2</td>
    <td>86.0</td>
    <td>54.2</td>
    <td>56.9</td>
    <td>86.4</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>94.0</b></td>
    <td><b>88.1</b></td>
    <td><b>46.5</b></td>
    <td><b>92.1</b></td>
    <td><b>57.2</b></td>
    <td><b>57.0</b></td>
    <td><b>86.9</b></td>
    <td><b>74.6 (+1.7)</b></td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>Regular</td>
    <td>96.7</td>
    <td>92.3</td>
    <td>48.0</td>
    <td>94.0</td>
    <td>60.3</td>
    <td>65.3</td>
    <td>86.7</td>
    <td>77.6</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>96.5</b></td>
    <td><b>93.2</b></td>
    <td><b>49.3</b></td>
    <td><b>96.1</b></td>
    <td><b>65.4</b></td>
    <td><b>67.5</b></td>
    <td><b>87.6</b></td>
    <td><b>79.4 (+1.8)</b></td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>Regular</td>
    <td>87.9</td>
    <td>89.4</td>
    <td>34.5</td>
    <td>62.2</td>
    <td>52.5</td>
    <td>47.6</td>
    <td>78.1</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>89.2</b></td>
    <td><b>89.6</b></td>
    <td><b>33.9</b></td>
    <td><b>68.1</b></td>
    <td><b>53.2</b></td>
    <td><b>47.6</b></td>
    <td><b>78.7</b></td>
    <td><b>65.8 (+1.2)</b></td>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td>Regular</td>
    <td>95.2</td>
    <td>91.0</td>
    <td>49.0</td>
    <td>72.3</td>
    <td>60.2</td>
    <td>61.8</td>
    <td>76.7</td>
    <td>72.3</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>95.1</b></td>
    <td><b>91.3</b></td>
    <td><b>48.3</b></td>
    <td><b>81.5</b></td>
    <td><b>61.8</b></td>
    <td><b>65.2</b></td>
    <td><b>79.1</b></td>
    <td><b>74.6 (+2.3)</b></td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>Regular</td>
    <td>96.0</td>
    <td>91.5</td>
    <td>51.9</td>
    <td>82.3</td>
    <td>71.4</td>
    <td>78.7</td>
    <td>83.8</td>
    <td>79.4</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>96.3</b></td>
    <td><b>91.7</b></td>
    <td><b>52.9</b></td>
    <td><b>90.4</b></td>
    <td><b>72.8</b></td>
    <td><b>79.9</b></td>
    <td><b>85.0</b></td>
    <td><b>81.3 (+1.9)</b></td>
  </tr>
</table>

Table 2: Performance of different models across 7 Natural Language Understanding (NLU) tasks. Red results indicate our method brings improvement over the regular decoding, while Green denote no improvement.

<table>
  <tr>
    <th>Model</th>
    <th>Decoding</th>
    <th>TREC</th>
    <th>Dbpedia</th>
  </tr>
  <tr>
    <td>Llama3.2-1B</td>
    <td>Regular</td>
    <td>40.0</td>
    <td>85.6</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>46.2 (+6.2)</b></td>
    <td><b>90.5 (+4.9)</b></td>
  </tr>
  <tr>
    <td>Llama3.2-3B</td>
    <td>Regular</td>
    <td>44.4</td>
    <td>83.1</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>49.6 (+5.2)</b></td>
    <td><b>91.4 (+8.3)</b></td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>Regular</td>
    <td>41.0</td>
    <td>87.5</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td><b>46.6 (+5.6)</b></td>
    <td><b>93.8 (6.3)</b></td>
  </tr>
</table>

Table 3: Average performance of two datasets with larger target classes. Red results indicate that our method brings improvement over the regular decoding.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Selection Method</th>
  </tr>
  <tr>
    <th>Random</th>
    <th>BM25</th>
    <th>TopK</th>
  </tr>
  <tr>
    <td>Regular Decoding</td>
    <td>77.6</td>
    <td>79.7</td>
    <td>80.2</td>
  </tr>
  <tr>
    <td colspan="4"><i>Equipped with our method</i></td>
  </tr>
  <tr>
    <td>+NULL</td>
    <td>73.0</td>
    <td>75.8</td>
    <td>76.5</td>
  </tr>
  <tr>
    <td>+Label</td>
    <td>77.3</td>
    <td>79.5</td>
    <td>80.0</td>
  </tr>
  <tr>
    <td>+Input</td>
    <td>79.4</td>
    <td>80.8</td>
    <td>80.9</td>
  </tr>
</table>

Table 4: Average performance with different negative in-context examples. Red results indicate that our method brings improvement over the regular decoding, while Green results denote no improvement.

<table>
  <tr>
    <th>SST2</th>
    <th>CR</th>
    <th>SST5</th>
    <th>Subj</th>
    <th>QNLI</th>
    <th>MNLI</th>
    <th>AGNEWS</th>
  </tr>
  <tr>
    <td>KL divergence</td>
    <td>0.64</td>
    <td>0.48</td>
    <td>0.43</td>
    <td>0.49</td>
    <td>0.04</td>
    <td>0.27</td>
    <td>0.79</td>
  </tr>
</table>

Table 5: The average KL divergence between the normalized output distributions with positive and negative in-context examples with Llama3.2-8B.

datasets, which means that the output distributions of positive and negative in-context examples are notably different. This demonstrates that our method can truly lead to models to contrast the positive and negative in-context examples.

Effects of Different number of shots. We gradually increase the number of in-context examples (denoted as N) from 1 to 16 to verify the influence of the number of shots in our method. Figure 2 reports the average performance of 7 NLU tasks and the different task QNLI. We see that our method can consistently outperform the regular decoding method with a different number of shots on aver-
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="5">\( \alpha \)</th>
  </tr>
  <tr>
    <th>0.0</th>
    <th>0.5</th>
    <th>1.0</th>
    <th>1.5</th>
    <th>2.0</th>
  </tr>
  <tr>
    <td rowspan="3">SST5</td>
    <td><b>Random</b></td>
    <td>48.0</td>
    <td>49.1</td>
    <td>49.3</td>
    <td>49.3</td>
    <td><b>49.5</b></td>
  </tr>
  <tr>
    <td><b>BM25</b></td>
    <td>53.0</td>
    <td><b>53.6</b></td>
    <td>53.3</td>
    <td>53.2</td>
    <td>53.1</td>
  </tr>
  <tr>
    <td><b>TopK</b></td>
    <td>53.0</td>
    <td>53.2</td>
    <td><b>53.2</b></td>
    <td>52.9</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td rowspan="3">MNLI</td>
    <td><b>Random</b></td>
    <td>65.3</td>
    <td>66.8</td>
    <td>67.5</td>
    <td>67.8</td>
    <td><b>68.1</b></td>
  </tr>
  <tr>
    <td><b>BM25</b></td>
    <td>65.8</td>
    <td>66.6</td>
    <td>67.1</td>
    <td>67.4</td>
    <td><b>67.6</b></td>
  </tr>
  <tr>
    <td><b>TopK</b></td>
    <td>65.9</td>
    <td>67.0</td>
    <td>67.4</td>
    <td><b>67.7</b></td>
    <td>67.7</td>
  </tr>
</table>

Table 6: The SST5 and MNLI performance with different \( \alpha \).

![Line graph showing performance with different shots for regular and ours methods](page_324_670_484_246.png)

Figure 2: The performance with different shots.

age. For the task QNLI, as the number of shots increases, the performance gains of our method also improve. We attribute this to the model acquiring more input-label mapping information from the demonstrations, which aligns with previous findings (Pan et al., 2023).

Effects of \( \alpha \). The factor \( \alpha \) in Eq. 2, which controls the importance of input-label mapping information, is an important hyper-parameter. In this part, we analyze its influence by evaluating the performance on SST5 and MNLI varying \( \alpha \) from 0 to 2. The results on Table 6 show that: 1) the performance improves with the increase of \( \alpha \), and it becomes stable when \( \alpha \geq 1.0 \), we set \( \alpha = 1 \) as default; 2) For advanced demonstration selection methods(e.g. TopK), too large positive \( \alpha \) values lead to performance degradation.

6 Conclusion

Large language models suffer from insufficient attention to the input-label mapping compared to their prior knowledge in in-context learning, leading to an unfaithful generation of the input query. In this work, we present a simple yet effective in-context contrastive decoding method that highlights input-label mapping by contrasting positive and negative in-context examples. Our experiments across various datasets and model architectures demonstrate the effectiveness and broad applicability of our approach, confirming its potential to enhance in-context learning.

Limitations

While the results presented in this paper demonstrate the effectiveness of our In-Context Contrastive Decoding (ICCD) method, there are a few limitations that warrant future exploration. First, our experiments were conducted on models up to 8B parameters, primarily due to computational limitations. Extending our method to even larger models (e.g., 70B parameters) could provide further insights into its scalability and effectiveness. Second, while our method shows promise across various Natural Language Understanding (NLU) tasks, its performance in specialized domains, such as legal or medical texts, has yet to be thoroughly examined. Third, our method requires additional forward passes to compute contrastive distributions. Although these passes are executed in parallel, the overall inference time may still increase. Future work will explore the generalizability of ICCD to these domains, as well as investigate its interaction with domain-specific datasets. Additionally, while we focused on classification tasks, other NLP tasks like text generation, machine translation, and summarization remain unexplored.

Acknowledges

We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. This work is supported by the National Natural Science Foundation of China (No. 62377002). This project is supported by the National Research Foundation, Singapore, under its NRF Professorship Award No. NRF-P2024-001.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint.

Taehyeon Kim, Joonkee Kim, Gihun Lee, and Se-Young Yun. 2024. Instructive decoding: Instruction-tuned large language models are self-refiner from noisy instructions. In ICLR.

Jannik Kossen, Yarin Gal, and Tom Rainforth. 2024. In-context learning learns label relationships but is not conventional learning. In ICLR.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In ACL.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In DeeLIO.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP.

Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning “learns” in-context: Disentangling task recognition and task learning. In ACL Findings.

Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. Revisiting demonstration selection strategies in in-context learning. In ACL.

Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. In Findings of EMNLP.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval.

Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. 2024. Mitigating hallucinations and off-target machine translation with source-contrastive and language-contrastive decoding. In EACL.

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In NAACL.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013a. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP.

Shuai Wang, Liang Ding, Li Shen, Yong Luo, Zheng He, Wei Yu, and Dacheng Tao. 2024. Uscd: Improving code generation of llms by uncertainty-aware selective contrastive decoding. arXiv preprint.

Shuai Wang, Liang Ding, Yibing Zhan, Yong Luo, Zheng He, and Dapeng Tao. 2025. Leveraging metamemory mechanisms for enhanced data-free code generation in llms. arXiv preprint.

Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint.

Kevin Christian Wibisono and Yixin Wang. 2024. In-context learning from training on unstructured data: The role of co-occurrence, positional information, and training data structure. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models.

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing
Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report.

Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024. Enhancing contextual understanding in large language models through contrastive decoding. In NAACL.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In ICML.

Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2024. ROSE doesn’t do that: Boosting the safety of instruction-tuned large language models with reverse prompt contrastive decoding. In Findings of ACL.

A  Datasets

Natural Language Understanding (NLU) Dataset information is detailed in Table 7. All NLU datasets are loaded from the HuggingFace Hub. For most NLU datasets, we report the results on the test set; while for the datasets MNLI and QNLI, we report the results on the validation set due to restricted access to their test sets.

B  Templates

The templates of NLU tasks used in this paper are detailed in Table 8.
<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th># of Classes</th>
    <th>Data Split</th>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>Sentiment Classification</td>
    <td>2</td>
    <td>6920/872/1821</td>
  </tr>
  <tr>
    <td>SST-5</td>
    <td>Sentiment Classification</td>
    <td>5</td>
    <td>8544/1101/2210</td>
  </tr>
  <tr>
    <td>CR</td>
    <td>Sentiment Classification</td>
    <td>2</td>
    <td>3394/0/376</td>
  </tr>
  <tr>
    <td>Subj</td>
    <td>Subjectivity Analysis</td>
    <td>2</td>
    <td>8000/0/2000</td>
  </tr>
  <tr>
    <td>AgNews</td>
    <td>Topic Classification</td>
    <td>4</td>
    <td>120000/0/7600</td>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>Natural Language Inference</td>
    <td>3</td>
    <td>392702/19647/19643</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>Natural Language Inference</td>
    <td>2</td>
    <td>104743/5463/5463</td>
  </tr>
</table>

Table 7: Details of NLU datasets.

<table>
  <tr>
    <th>Task</th>
    <th>Prompt</th>
    <th>Class</th>
  </tr>
  <tr>
    <td rowspan="2">SST-2</td>
    <td>Review: "&lt;X&gt;" Sentiment: positive</td>
    <td>positive</td>
  </tr>
  <tr>
    <td>Review: "&lt;X&gt;" Sentiment: negative</td>
    <td>negative</td>
  </tr>
  <tr>
    <td rowspan="4">SST-5</td>
    <td>Review: "&lt;X&gt;" Sentiment: terrible</td>
    <td>terrible</td>
  </tr>
  <tr>
    <td>Review: "&lt;X&gt;" Sentiment: bad</td>
    <td>bad</td>
  </tr>
  <tr>
    <td>Review: "&lt;X&gt;" Sentiment: okay</td>
    <td>okay</td>
  </tr>
  <tr>
    <td>Review: "&lt;X&gt;" Sentiment: good</td>
    <td>good</td>
  </tr>
  <tr>
    <td rowspan="2">SST-5</td>
    <td>Review: "&lt;X&gt;" Sentiment: great</td>
    <td>great</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Subj</td>
    <td>Input: "&lt;X&gt;" Type: objective</td>
    <td>objective</td>
  </tr>
  <tr>
    <td>Input: "&lt;X&gt;" Type: subjective</td>
    <td>subjective</td>
  </tr>
  <tr>
    <td rowspan="2">CR</td>
    <td>Review: "&lt;X&gt;" Sentiment: positive</td>
    <td>positive</td>
  </tr>
  <tr>
    <td>Review: "&lt;X&gt;" Sentiment: negative</td>
    <td>negative</td>
  </tr>
  <tr>
    <td rowspan="4">AgNews</td>
    <td>Input: "&lt;X&gt;" Type: world</td>
    <td>World</td>
  </tr>
  <tr>
    <td>Input: "&lt;X&gt;" Type: sports</td>
    <td>Sports</td>
  </tr>
  <tr>
    <td>Input: "&lt;X&gt;" Type: business</td>
    <td>Business</td>
  </tr>
  <tr>
    <td>Input: "&lt;X&gt;" Type: technology</td>
    <td>Sci/Tech</td>
  </tr>
  <tr>
    <td rowspan="3">MNLI</td>
    <td>Premise: &lt;C&gt; Hypothesis: &lt;X&gt; Prediction: entailment</td>
    <td>Entailment</td>
  </tr>
  <tr>
    <td>Premise: &lt;C&gt; Hypothesis: &lt;X&gt; Prediction: neutral</td>
    <td>Neutral</td>
  </tr>
  <tr>
    <td>Premise: &lt;C&gt; Hypothesis: &lt;X&gt;? Prediction: contradiction</td>
    <td>Contradiction</td>
  </tr>
  <tr>
    <td rowspan="2">QNLI</td>
    <td>&lt;C&gt; Can we know &lt;X&gt;? Yes.</td>
    <td>Entailment</td>
  </tr>
  <tr>
    <td>&lt;C&gt; Can we know &lt;X&gt;? No.</td>
    <td>Contradiction</td>
  </tr>
</table>

Table 8: Templates of NLU tasks. Placeholders (e.g., &lt;X&gt; and &lt;C&gt;) will be replaced by real inputs.

Reviewer paper summaries: The authors aim to tackle the problem of language models in in-context learning, which use their prior knowledge instead of learning from input-label mapping. To tackle this challenge, the authors then propose a simple yet effective strategy motivated by contrastive decoding strategies, which are designed to highlight the input-label mapping by contrasting positive and negative examples, where the negative examples are constructed by altering the in-context examples. The authors then perform experiments with multiple tasks and multiple language models, showing the effectiveness of the proposed approach over relevant baselines.
Summary of strengths: * The idea to utilize the contrastive decoding technique to tackle the challenge of learning from input-output mapping in ICL is sound and reasonable.
* The authors experimentally validate the effectiveness of the proposed approach over the one with regular decoding from existing studies, with multiple datasets and models, which are sufficient.
Summary of weaknesses: In general, I think the depth of this study is appropriate to the short paper. However, it would be great if the authors could address my major concerns below:
* The performance improvement of the proposed approach against the regular decoding approach seems marginal, especially the one with the top-k demonstration selection strategy. As the authors perform multiple runs, it would be great if the authors could perform the statistical test. 
* While the authors present results on multiple datasets with multiple language models, it is questionable whether the proposed approach can truly lead the models to contrast the positive and negative in-context examples. Some analyses on this aspect are needed to support the main claim of the authors, besides the performance improvement. 
* The number of target classes for the ICL datasets and scenarios that the authors evaluate might be small, which are around 2 or 3. It would be interesting to see whether the proposed approach can be further beneficial to a large number of classes, as its core design is to contrast the positive and negative samples and it can potentially have a positive effect.
Comments, suggestions and typos: Please see my comments in the Weaknesses section. One additional (but minor) comment is that it would be great if the authors could visualize the overall process. Also, for the numbers in the tables, while the authors use the colors green and red to highlight the superior (or inferior) performance of the proposed approach, the contrast is too strong and somewhat distracting.
Overall assessment scores: 3