ALW: Adaptive Layer-Wise contrastive decoding enhancing reasoning ability in Large Language Models
Authors: Yuechi Zhou, Chuyue Zhou, Jianxin Zhang, Juntao Li, Min Zhang
Accepted at: acl25

ALW: Adaptive Layer-Wise contrastive decoding enhancing reasoning ability in Large Language Models

Yuechi Zhou, Chuyue Zhou, Jianxin Zhang, Juntao Li*, Min Zhang
School of Computer Science and Technology, Soochow University
Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
{yczhouu,cyzhou,20204027008}@stu.suda.edu.cn
{ljt,minzhang}@suda.edu.cn

Abstract
Large language models (LLMs) have achieved remarkable performance across various reasoning tasks. However, many LLMs still encounter challenges in reasoning, especially for LLMs with fewer parameters or insufficient pre-training data. Through our experiments, we identify that noise accumulation across layers often leads to unstable token predictions during reasoning. We find that contrasting the probability distributions across layers effectively mitigates this interference. Building on this insight, we propose Adaptive Layer-Wise contrastive decoding (ALW), a novel framework that enhances reasoning ability by dynamically disentangling noise in shallow layers from critical signals in deep layers. Extensive experiments on several reasoning benchmarks demonstrate that ALW consistently improves answer accuracy across multiple LLMs while maintaining inference efficiency. For example, we achieve a 48% improvement on the Gsm8k using the LLaMA-7B model and an absolute accuracy increase of 5.2 points on the BBH evaluation benchmark with the LLaMA-65B model*.

1 Introduction
Recent years have witnessed significant advancements in reasoning tasks (Qiao et al., 2023; Hong et al., 2024) due to the development and deployment of large language models (LLMs) (Yang et al., 2024; Zhao et al., 2024; Guo et al., 2025). However, not all language models (LMs) possess strong reasoning abilities. They encounter a variety of reasoning issues, such as arithmetic mistakes and semantic errors (Wang et al., 2023).

Existing research enhances reasoning capabilities by linking to external knowledge and retrieving reasoning evidence (Zhu et al., 2023; Wang et al., 2024). Recent works find that contrast decoding of two different models can improve text quality in open-ended text generation (Li et al., 2023) and have great potential in reasoning tasks (O’Brien and Lewis, 2023). Chuang et al. (2024) further achieve performance improvement on open-ended generation tasks through self-contrast decoding but with limited gains on reasoning tasks. In view of this, we start from recent observations on noise signals within LLMs (Ye et al., 2024), in which LLMs assign disproportionately high scores to non-critical tokens, and directly subtracting the attention distributions of different attention heads can effectively diminish the influence of noisy irrelevant tokens to the generation results. This insight leads us to question whether the noise from overly-attended irrelevant tokens accumulates across different layers and results in diverse reasoning errors observed from the final layer, e.g., repetitive generation. Figure 1 preliminarily confirms that subtracting the prediction distribution of the shallow layer from that of the final layer can weaken noisy context and produce the correct answer “No”.

Therefore, we mitigate noise at the layer level and propose adaptive layer-wise contrastive decoding, a framework designed to enhance reasoning capabilities by explicitly disentangling shallow noise from deep reasoning. To achieve this, we develop a quantification strategy to determine the optimal early exit layer for noise suppression. Furthermore, we introduce a lightweight LM that learns which layer of the LLM requires noise suppression in an end-to-end manner and dynamically adjusts the contrastive strategy during the inference stage. Additionally, we extend ALW by developing a tuning-free method based on semantic similarity, reducing resource consumption.

Specifically, we first identify the optimal noise layer that requires the most suppression for different inputs and tasks. Then, a predictor, either LM-based or similarity-based, is employed to capture the mapping between the prompt and the optimal contrastive layer. During inference, the prompt is
Figure 1: Illustration of the probability distributions during the decoding of the final answer token. The model prompt is: “Q: Would a pear sink in water? A: . . . CoT. . . . The answer is”. The model used is LLaMA-7B.

processed by both the LLM and the predictor. The LLM contrasts the final layer with the early exit layer selected by the predictor, effectively filtering noise from shallow layers while preserving key signals in the final layer. Extensive experiments demonstrate that ALW achieves stable performance improvements and significantly enhances the reasoning abilities of LLMs.

Our contributions are as follows:

• We introduce ALW, a novel decoding framework designed to enhance reasoning capabilities by dynamically disentangling shallow noise from deep signals.

• We evaluate ALW on several LLM benchmark datasets across multiple choice tasks and open-ended generation tasks, demonstrating its effectiveness in enhancing reasoning abilities.

• We conduct a comprehensive analysis of ALW’s generalization ability, computational cost, effectiveness, and limitations.

2 Preliminary

2.1 Noise across Different Layers of LLMs
Inspired by noise suppression across attention heads (Ye et al., 2024), we explore the issue of vertical noise propagation (across different layers). Noise can be understood as task-irrelevant interference, such as incorrect attention allocation, which may cause the model to hesitate between multiple unrelated tokens, manifesting as high entropy in the probability distribution. In contrast, lower entropy indicates greater confidence in the next prediction and a reduction in noise.

We use entropy to quantify the noise at each layer. Figure 2 presents the results of a randomly sampled case (more cases are given in Appendix A). We can observe that shallow layers consistently exhibit higher entropy (~10), indicating greater noise and uncertainty, while deeper layers show significantly lower entropy (~1), reflecting more stable and refined representations. After contrasting the probability distributions of the final layer and the shallow layer, we obtain a new distribution (“Diff” row in Figure 2) and observe that the entropy at each step is consistently lower than that of the final layer. We attribute this to the information redundancy and noise differences between shallow and deep layers, where the contrastive operation further suppresses the noise, making the probability distribution more focused on the target token.

This is also consistent with the observation in Figure 1, where we examine the distribution just before decoding the final answer. In the final layer, the model struggles to choose between “yes” and “no”, with their probabilities being very close. In fact, this decision is influenced by noise from the shallow layers where the probability of “yes” is much higher than “no”, leading to uncertainty in the final layer. After denoising, the model is able to confidently select the correct token.

2.2 Noise Suppression
From cases to general tasks, we initially attempt to randomly select contrastive layers for noise suppression. However, this approach clearly failed, as different inputs exhibit varying sensitivities to noise distributions. Therefore, we quantify the optimal layer for each input (Section 3.2), where the optimal layer is the one that maximizes the probability of the gold token after denoising. Our goal is not only to generate the correct final answer but also to produce a reasonable reasoning process.

Next, we focus on the effective utilization of the generated data pairs. We explored rule-based methods, such as denoising with the most frequent layer, but these proved unsuccessful (Appendix B). We believe the task of layer selection is complex and requires capturing the implicit relationships between prompts and layers, leading us to utilize
Question: Who was the first Nigerian to win the Nobel Prize, in which year?
Answer: Wole Soyinka was the first Nigerian to win the Nobel Prize, in 1986.

Figure 2: The entropy of the probability distribution for the final layer and several early exit layers in LLaMA-7B under teacher forcing at each step. The horizontal axis represents the decoding step, while the vertical axis (ranging from 0 to 32 in intervals of 2) denotes the considered early exit layers and the final layer.

LMs (Sec 3.3). In our experiments, we confirm the optimal configuration for the LM predictor. Additionally, we devised a tuning-free method to select the optimal layer based on semantic similarity for lower resource consumption (ALW-sim).

3 Method

3.1 Self-Contrast Decoding

Typically, an LLM comprises an embedding layer, followed by \( M \) decoder layers indexed from 1 to \( M \) (embedding layer as 0), and ends with an output layer \( W_o \). Given a prompt of length \( n \), denoted by \( X = \{ x_0, x_1, \ldots, x_{n-1} \} \), \( h_n^e \) represents the hidden states produced by the \( e \)-th layer when decoding \( x_n \). For each early exit layer \( e \in [0, M] \), the probability distribution is then obtained as follows:

\[
p_e(x_n|X) = \operatorname{softmax}(W_o(h_n^e)).
\]

To cancel noise, we contrast the prediction of an early exit layer \( e \) with that of the final layer \( M \). Specifically, we subtract the log probabilities of the two layers, yielding the following distribution:

\[
\psi_e(x_n|X) = \log p_M(x_n|X) - \log p_e(x_n|X).
\]

For the actual selection of the early exit layer \( e \), we first apply a quantification strategy on the training set. This allows us to identify the optimal contrastive layer and construct the corresponding prompt-layer data pairs (\S 3.2). Subsequently, we train a lightweight LM end-to-end (\S 3.3), which assists the LLM in determining the most suitable contrastive layer during inference (\S 3.4).

3.2 Quantification Strategy

In the generation tasks, given a prompt \( X \) and a reference answer \( A = \{ a_1, a_2, \ldots, a_m \} \) of length \( m \), we analyze each token \( a_i \in A \) individually. For each \( a_i \), we determine the optimal contrastive layer \( l \) that maximizes its probability, conditioned on the concatenation of \( X \) and its preceding tokens \( a_{<i} \). This process is formulated as follows:

\[
l = \arg\max_e \psi_e(a_i|X + a_{<i}), e \in [0, M/2].
\]

In multiple choice tasks, given a prompt \( X \), a correct option T, and K incorrect options \( \{ F_i \}_{i=0}^{K-1} \), we calculate the log-likelihood score \( s \) for each option in each early exit layer \( e \). For the correct option T at layer \( e \), the score is defined as:

\[
s_e(T|X) = \sum_k \log \operatorname{softmax}(\psi_e(x_k|X + x_{<k})),
\]

where \( x_k \) denotes each token in the option T. The score \( s_e(F_i|X) \) for each incorrect option \( F_i \) is calculated in the same way. If the score for the correct option is not lower than that of all possible incorrect options, then layer \( e \) is considered to have produced a correct prediction. Consequently, we define the optimal layer \( l \) as the one that maximizes the log-likelihood of the correct option among all
such contrastive layers, formulated as follows:

\[
l = \arg\max_e s_e(T|X),\ e \in [0, M/2],
\]
st. \( s_e(T|X) \geq \max \{ s_e(F_i|X) \}_{i=0}^{K-1} . \tag{5} \)

However, if the score of the option T computed by the final layer matches that of the optimal layer l, the contrastive operation is deemed unnecessary, and l is assigned a value of −1. Similarly, in generation tasks, if \( p_M \geq \text{softmax}(\psi_l) \), l is also set to −1. Furthermore, to reduce the search space, we focus on the first half of the model’s layers. For example, in the LLaMA-7B model, which consists of 33 early exit layers, the possible values for l range from −1 to 16. Figure 3 illustrates the process of quantifying the optimal layer. In generation tasks, given an answer of length m, we can generate m pairs, whereas in multiple choice tasks, only a single data pair can be generated.

3.3 Layer Prediction

As described above, for each task, we obtain multiple \( (X, l) \) data pairs. These pairs are stored in a database \( \mathcal{D}_b = \{ (X_i, l_i) \}_{i=0}^{N-1} \), where N is the total number of pairs. We formulate layer prediction as a classification task and train a lightweight model, such as RoBERTa, end-to-end.

Furthermore, to balance accuracy and recall, we assign appropriate weights to each category. Let \( p_\theta \) represent the prediction probability of the LM. The model is optimized as follows:

\[
\mathcal{L} = -\frac{1}{b} \sum_{i=1}^b \sum_j w_j \cdot l_{ij} \log p_\theta(y_{ij} | X_i),
\]
\[
w_j = \frac{1/n_j}{\sum_j 1/n_j} \propto \frac{1}{n_j},
\] \tag{6}

where b is the batch size, i indexes the samples, j indexes the categories, and \( n_j \) represents the frequency of category j in \( \mathcal{D}_b \). An example illustrating the categories is provided in Appendix C.

In addition to the LM-based layer prediction strategy, we introduce a tuning-free method that selects the optimal layer based on the cosine similarity between the input embeddings and the data pairs in \( \mathcal{D}_b \). We refer to this approach as similarity-based layer prediction, denoted as ALW-sim. In contrast, the method that requires LM training to fully activate ALW is termed ALW-lm.

3.4 Decoding Strategy

Drawing on prior research (Li et al., 2023) that used adaptive plausibility constraints to address false positive and false negative errors in contrastive objectives, we extend this to our inference stage. If certain tokens are characterized by high probabilities, they will be grouped into \( V_{\text{head}}(x_n|x_{<n}) \):

\[
\{ x_t \in V : p_M(x_n|x_{<n}) \geq \alpha \max p_M(x_n|x_{<n}) \}, \tag{7}
\]

where \( \alpha \) is a hyperparameter ranging from 0 to 1, which truncates the distribution of low-probability tokens. During the inference stage, we leverage the final layer to ensure token reliability. If a token
achieves a high probability after contrast but is not recognized by the final layer (not in \( \mathcal{V}_{\text{head}} \)), we exclude it during the decoding process. Finally, the LLM decodes with ALW-lm or ALW-sim:

\[
\psi(x_n|X) = \begin{cases}
\log \frac{p_M(x_n|X)}{p_l(x_n|X)} & \text{if } x_n \in \mathcal{V}_{\text{head}}; \\
-\infty & \text{Otherwise},
\end{cases}
\]
\[
l = \arg\max_k p_\theta(y_k|X) \text{ or } l = \operatorname{Sim}(X, \mathcal{D}_b),
\]

where the function \( \operatorname{Sim}(\cdot) \) computes the cosine similarity between \( X \) and all prompts in \( \mathcal{D}_b \), selecting the prompt with the highest similarity and assigning its corresponding \( l \) as the contrastive layer. A detailed procedure is provided in Algorithm 1.

4 Experiments

4.1 Experimental Setup

Tasks. Our framework is evaluated across various tasks. We assess arithmetic reasoning performance on Gsm8k (Cobbe et al., 2021) MathQA (Aida et al., 2019). For commonsense reasoning evaluation, we employ StrategyQA (Geva et al., 2021), and PiQA (Bisk et al., 2020). Additionally, we utilize the logically complex and demanding dataset Folio (Han et al., 2022) for natural language reasoning. Finally, we adopt three widely recognized testing benchmarks, MMLU (Hendrycks et al., 2021), BBH (Suzgun et al., 2022), and ARC-Challenge (Clark et al., 2018). These benchmarks are respected in the field for their diversity and breadth of tasks (details in Appendix D), allowing us to comprehensively analyze the model’s abilities to manage intricate reasoning across varied domains. Among these tasks, Gsm8k, StrategyQA, and Folio are designed as open-ended generation tasks, and the others are multiple-choice tasks.

Models and Baselines. We employ the LLaMA family (Touvron et al., 2023) as our fundamental models, covering sizes of 7B, 13B, 30B, and 65B. In addition, we also consider the updated and more powerful model LLaMA3-8B-Instruct (Dubey et al., 2024). We test our framework against three baselines: Greedy Decoding, Decoding by Contrasting Layers (DoLa) (Chuang et al., 2024), and Contrastive Decoding (CD) (Li et al., 2023). Greedy decoding selects the token with the highest probability at each step, DoLa leverages the Jensen-Shannon divergence between transformer layers within Language Models to enhance factuality, and CD ensures plausible outputs by making a comparison between an LLM and a smaller one.

Algorithm 1 ALW Decoding Strategy
Input: Prompt \( X = \{x_0, x_1, \ldots, x_{n-1}\} \), LLM, LM or Sim(\cdot), Output Layer \( W_o \), database \( \mathcal{D}_b \)
Output: Next token \( x_n \)
1: Hidden states \( \{h^t\}_{t=0}^{M} \leftarrow \) Forward LLM(X)
2: \( l \leftarrow \) Forward LM(X) or Sim(X, \( \mathcal{D}_b \))
3: if \( l = -1 \) then
4: \hspace{1em} \( \psi(x) = W_o(h^M) \)
5: else
6: \hspace{1em} Final prob \( p_M = \operatorname{softmax}(W_o(h^M)) \)
7: \hspace{1em} Early prob \( p_l = \operatorname{softmax}(W_o(h^l)) \)
8: \hspace{1em} \( \psi(x) = \log(p_M) - \log(p_l) \)
9: \hspace{1em} \( Mask \leftarrow \) Token indices not in \( \mathcal{V}_{\text{head}} \)
10: \hspace{1em} \( \psi(Mask) \leftarrow -\infty \)
11: end if
12: return \( x_n = \arg\max \operatorname{softmax}(\psi(x)) \)

Details. In our experiments, the variable \( \alpha \) in Equation 7 is set to 0.1, a value that has shown robustness across various domains (Li et al., 2023). Our predictor utilizes a RoBERTa-base model (Liu et al., 2019) or a sentence-transformer (Reimers and Gurevych, 2019) to compute semantic similarity. For open-ended generation tasks, we implement Chain-of-Thought (Wei et al., 2022). Additional details can be found in Appendix E.

4.2 Results

4.2.1 Arithmetic Reasoning

Results are illustrated in Table 1, where the open-end generation task is indicated by (*). Notably, we maintain the layer search space for DoLa within the range [0, \( M/2 \)], matching the strategy used in ALW. For CD, we employ LLaMA-7B as the amateur model, and larger models (13B, 30B, 65B) are utilized as expert models. However, due to the absence of LLaMA models smaller than LLaMA-7B, the baseline CD is not included in the experiments conducted at the 7B scale, and the same as LLaMA3-8B-Instruct. ALW significantly enhances the model’s arithmetic reasoning capabilities. Our approach leads the other three baselines across four model scales on Gsm8k and MathQA. Compared to greedy decoding, ALW achieves a substantial increase in performance, which is more pronounced in smaller models. For example, there is a 48% increase on Gsm8k for LLaMA-7B (28% for LLaMA-30B and 9% for 65B).
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">Arithmetic</th>
    <th colspan="2">Commonsense</th>
    <th colspan="4">General</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Gsm8k*</th>
    <th>MathQ</th>
    <th>StrQA*</th>
    <th>PiQA</th>
    <th>Folio*</th>
    <th>MMLU</th>
    <th>BBH</th>
    <th>ARC-C</th>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-7B-HF</td>
    <td>Greedy</td>
    <td>13.6</td>
    <td>22.3</td>
    <td>66.4</td>
    <td><b>78.5</b></td>
    <td>16.5</td>
    <td>34.8</td>
    <td>34.1</td>
    <td>37.0</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>12.9</td>
    <td>24.8</td>
    <td>65.1</td>
    <td>76.8</td>
    <td>29.8</td>
    <td>35.8</td>
    <td>38.7</td>
    <td>37.9</td>
    <td>40.2</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>20.1</b></td>
    <td><b>36.3</b></td>
    <td><b>70.3</b></td>
    <td><b>78.1</b></td>
    <td><b>29.8</b></td>
    <td><b>40.6</b></td>
    <td><b>41.7</b></td>
    <td><b>41.3</b></td>
    <td><b>44.7</b></td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA3-8B-Inst</td>
    <td>Greedy</td>
    <td>76.5</td>
    <td>38.1</td>
    <td>71.6</td>
    <td>79.8</td>
    <td>52.5</td>
    <td>38.4</td>
    <td>43.7</td>
    <td>41.7</td>
    <td>55.3</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>73.2</td>
    <td>50.2</td>
    <td>69.9</td>
    <td>77.7</td>
    <td>52.1</td>
    <td>38.2</td>
    <td>49.5</td>
    <td>43.5</td>
    <td>56.8</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>78.8</b></td>
    <td><b>56.3</b></td>
    <td><b>71.6</b></td>
    <td><b>80.1</b></td>
    <td><b>52.9</b></td>
    <td><b>44.2</b></td>
    <td><b>50.8</b></td>
    <td><b>47.8</b></td>
    <td><b>60.3</b></td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-13B-HF</td>
    <td>Greedy</td>
    <td>18.9</td>
    <td>24.9</td>
    <td>67.2</td>
    <td><b>80.1</b></td>
    <td>34.7</td>
    <td>36.2</td>
    <td>37.2</td>
    <td>36.5</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>CD</td>
    <td>8.3</td>
    <td>22.3</td>
    <td>66.4</td>
    <td>69.9</td>
    <td>37.2</td>
    <td>29.5</td>
    <td>36.6</td>
    <td>34.2</td>
    <td>38.1</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>22.0</td>
    <td>30.2</td>
    <td>69.9</td>
    <td>78.1</td>
    <td>39.7</td>
    <td>38.0</td>
    <td>43.1</td>
    <td>40.6</td>
    <td>45.2</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>22.7</b></td>
    <td><b>37.8</b></td>
    <td><b>72.5</b></td>
    <td><b>79.6</b></td>
    <td><b>50.4</b></td>
    <td><b>38.2</b></td>
    <td><b>44.5</b></td>
    <td><b>42.1</b></td>
    <td><b>48.5</b></td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-30B-HF</td>
    <td>Greedy</td>
    <td>32.6</td>
    <td>26.3</td>
    <td>72.9</td>
    <td>81.2</td>
    <td>44.6</td>
    <td>37.8</td>
    <td>39.5</td>
    <td>37.2</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>CD</td>
    <td>32.6</td>
    <td>24.8</td>
    <td>70.7</td>
    <td>72.2</td>
    <td>20.5</td>
    <td>31.5</td>
    <td>39.2</td>
    <td>33.4</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>28.0</td>
    <td>29.0</td>
    <td>73.4</td>
    <td>77.0</td>
    <td>41.3</td>
    <td>37.5</td>
    <td>43.9</td>
    <td>40.6</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>41.7</b></td>
    <td><b>35.7</b></td>
    <td><b>76.0</b></td>
    <td><b>81.3</b></td>
    <td><b>44.6</b></td>
    <td><b>40.4</b></td>
    <td><b>45.0</b></td>
    <td><b>41.0</b></td>
    <td><b>50.7</b></td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-65B-HF</td>
    <td>Greedy</td>
    <td>50.8</td>
    <td>27.2</td>
    <td>76.0</td>
    <td>81.3</td>
    <td>42.1</td>
    <td>40.4</td>
    <td>37.9</td>
    <td>39.6</td>
    <td>49.4</td>
  </tr>
  <tr>
    <td>CD</td>
    <td>43.9</td>
    <td>25.1</td>
    <td>74.2</td>
    <td>72.2</td>
    <td>21.5</td>
    <td>32.0</td>
    <td>35.1</td>
    <td>34.9</td>
    <td>42.4</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>54.0</td>
    <td>30.4</td>
    <td>72.9</td>
    <td>66.8</td>
    <td>41.3</td>
    <td>37.1</td>
    <td>42.9</td>
    <td>45.1</td>
    <td>48.8</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>55.3</b></td>
    <td><b>38.5</b></td>
    <td><b>77.7</b></td>
    <td><b>81.3</b></td>
    <td><b>43.8</b></td>
    <td><b>41.4</b></td>
    <td><b>43.1</b></td>
    <td><b>47.8</b></td>
    <td><b>53.6</b></td>
  </tr>
</table>

Table 1: The overall performance of ALW for three types of reasoning tasks across LLaMA-7B, 13B, 30B, 65B and LLaMA3-8B-Instruct. Bold indicates the best results. ALW outperforms three baselines consistently.

4.2.2 Commonsense Reasoning

For commonsense reasoning tasks, LLMs effectively suppress noises from shallow layers, thereby enhancing their commonsense reasoning capabilities. Results show that ALW outperforms other baselines on the StrategyQA benchmark. However, the improvement on PiQA is less pronounced. A closer analysis reveals that in over 90% of instances, the contrast operation was not applied, suggesting that decoding in the final layer is more fruitful than employing the contrast operation. We hypothesize that this could be due to the inherent simplicity of the PiQA dataset, which enables the task to be effectively handled by the shallow layers. As a result, less noise will present in the shallow layers. Further details can be found in Appendix F.

4.2.3 Comprehensive Evaluation

We verify the reasoning capabilities of LLMs on Folio, MMLU, BBH, and ARC-C. In each case, ALW consistently surpasses all baselines. ALW proves particularly efficient in Folio, which involves first-order logic. This efficacy arises from the inadequacies of the shallow layers in grasping deep logical relationships, thereby enabling the deeper layers to circumvent various reasoning errors. Notably, after using ALW, LLaMA-13B saw a significant increase in reasoning abilities on Folio, with a result of 50.4 surpassing all other model settings. Moreover, we present the results of the MMLU and BBH sub-

<table>
  <tr>
    <th>BBH</th>
    <th>MathQ</th>
    <th>PiQA</th>
    <th>ARC</th>
    <th>MMLU</th>
    <th>Gsm8k</th>
    <th>Folio</th>
    <th>StrQA</th>
  </tr>
  <tr>
    <th>7B</th>
    <td>59.6</td>
    <td>79.8</td>
    <td>93.5</td>
    <td>80.5</td>
    <td>85.5</td>
    <td>80.2</td>
    <td>82.1</td>
    <td>71.2</td>
  </tr>
  <tr>
    <th>8B</th>
    <td>73.8</td>
    <td>91.6</td>
    <td>95.4</td>
    <td>81.6</td>
    <td>90.1</td>
    <td>81.3</td>
    <td>84.9</td>
    <td>75.2</td>
  </tr>
  <tr>
    <th>13B</th>
    <td>51.0</td>
    <td>69.7</td>
    <td>93.1</td>
    <td>77.2</td>
    <td>81.1</td>
    <td>77.1</td>
    <td>81.6</td>
    <td>70.6</td>
  </tr>
  <tr>
    <th>30B</th>
    <td>46.1</td>
    <td>50.1</td>
    <td>87.2</td>
    <td>62.3</td>
    <td>80.3</td>
    <td>79.1</td>
    <td>84.2</td>
    <td>71.3</td>
  </tr>
  <tr>
    <th>65B</th>
    <td>40.8</td>
    <td>43.6</td>
    <td>80.2</td>
    <td>50.0</td>
    <td>75.4</td>
    <td>77.0</td>
    <td>83.1</td>
    <td>71.0</td>
  </tr>
</table>

Table 2: Accuracy of predictor in multiple-choice tasks (left); in open-ended generation tasks (right).

tasks in Appendix K and analyze the differences across various sub-tasks and models.

4.2.4 Accuracy of Predictor

For multiple-choice tasks, we report the accuracy of the predictor (lm-based) in the test set, as shown in Table 2 (left). For open-ended generation tasks, we cannot definitively determine whether a generated token is correct or incorrect during generation. However, during the training process, we can maximize the probability of golden tokens after contrasting. Thus, we report the results on the validation set, which is shown in Table 2 (right). It can be observed that in all settings, the accuracy rates are almost always greater than 50%, indicating that the predictor can provide relatively accurate predictions of the optimal layer for a given prompt. In addition, we present the distribution of the predictor output in Appendix G.
<table>
  <tr>
    <th>Datasets</th>
    <th>BERT-base</th>
    <th>RoBERTa-base</th>
    <th>RoBERTa-large</th>
  </tr>
  <tr>
    <td>MathQA</td>
    <td>36.1 (-0.2)</td>
    <td>36.3</td>
    <td>27.4(-8.9)</td>
  </tr>
  <tr>
    <td>Folio*</td>
    <td>31.4 (+1.6)</td>
    <td>29.8</td>
    <td>16.5(-13.3)</td>
  </tr>
  <tr>
    <td>BBH</td>
    <td>40.5 (-1.2)</td>
    <td>41.7</td>
    <td>40.7(-1.0)</td>
  </tr>
</table>

Table 3: Ablation study of predictors. LLaMA-7B with RoBERTa-base achieves the best performance.

4.3 Ablation

Impact of Predictor. We carry out experiments with different pre-trained LMs as predictors to probe the configuration of the ideal model. Table 3 suggests that the application of BERT-base (Devlin et al., 2019) as a predictor produces slightly subpar results in contrast to the use of RoBERTa-base. This is substantiated by the relatively diminished accuracy on MathQA and BBH. However, larger models do not guarantee superior performance. In fact, the use of RoBERTa-large leads to a decline in the overall reasoning competency of LLMs. This may arise from a mismatch between the volume of training data and the size of the model.

Generalization. In Table 4, we present ALW’s generalization performance in-domain and out-of-domain in LLaMA-7B. The mathematical reasoning domain (Domain 1) includes Gsm8k and MathQA; the comprehensive reasoning domain (Domain 2) includes MMLU and BBH; and StrategyQA serves as the commonsense reasoning domain (Domain 3). The arrows indicate values that are lower than those of greedy decoding. Observing the position where the arrow appears, we conclude as follows: whether within the domain or across domains, transferring from multiple-choice tasks to generation tasks is not ideal; conversely, the effect can exceed greedy decoding but does not reach the performance specific to the dataset. Besides this, ALW has demonstrated exceptional generalization performance. For example, in Domain 2, when trained solely on MMLU, ALW achieved an accuracy of 40.4 when reasoning on BBH, reaching 96% of the performance (41.7) in a specific dataset.

<table>
  <tr>
    <th>Domain</th>
    <th colspan="2">Domain1</th>
    <th colspan="2">Domain2</th>
    <th>Domain3</th>
  </tr>
  <tr>
    <th></th>
    <th>Gsm8k*</th>
    <th>MathQA</th>
    <th>MMLU</th>
    <th>BBH</th>
    <th>StrQA*</th>
  </tr>
  <tr>
    <td>Gsm8k*</td>
    <td>-</td>
    <td>28.4</td>
    <td>37.2</td>
    <td>40.2</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>MathQA</td>
    <td>12.1↓</td>
    <td>-</td>
    <td>37.2</td>
    <td>40.7</td>
    <td>65.9↓</td>
  </tr>
  <tr>
    <td>MMLU</td>
    <td>9.85↓</td>
    <td>27.2</td>
    <td>-</td>
    <td>40.4</td>
    <td>66.8</td>
  </tr>
  <tr>
    <td>BBH</td>
    <td>15.9</td>
    <td>27.2</td>
    <td>37.0</td>
    <td>-</td>
    <td>65.9↓</td>
  </tr>
  <tr>
    <td>StrQA*</td>
    <td>13.6</td>
    <td>27.5</td>
    <td>35.3</td>
    <td>40.9</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Base</td>
    <td>13.6</td>
    <td>22.3</td>
    <td>34.8</td>
    <td>34.1</td>
    <td>66.4</td>
  </tr>
</table>

Table 4: In-domain and out-domain generalization.

<table>
  <tr>
    <th>Prompt</th>
    <th>Would a pear sink in water?</th>
  </tr>
  <tr>
    <td>Greedy<br>(×)</td>
    <td>A pear is a fruit. Fruits float in water. Thus, a pear would sink in water. So the answer is yes.</td>
  </tr>
  <tr>
    <td>Case 1<br>(×)</td>
    <td>... Thus, a pear would <b>float</b> in water. So the answer is yes.</td>
  </tr>
  <tr>
    <td>Case 2<br>(×)</td>
    <td>... Thus, a pear would <b>float</b> in water. So the answer is <b>no</b>.</td>
  </tr>
  <tr>
    <td>ALW<br>(✓)</td>
    <td>A pear would float in water because its density is lower than that of water. So the answer is no.</td>
  </tr>
</table>

Table 5: A case demonstrating the effectiveness of ALW.

still recommend using the more advanced ALW, but if training resources are extremely limited, ALW-sim can serve as a substitute.

Random layer. We verify the importance of the predictor by randomly selecting the contrastive layer. The results in Table 8 show that random selection led to a significant decrease in the reasoning accuracy across all settings, with only 25% of the cases outperforming the results of using greedy decoding (highlighted in bold). Therefore, a predictor that learns the patterns of optimal layer occurrence with a quantification strategy is necessary.

The effect of ALW. We explore why ALW is effective, attributing its success to the dynamic correction of reasoning paths. As shown in Table 5, greedy decoding makes an error when reasoning through “pear is fruit” → “fruit float” → “pear sink”. Applying ALW only at the erroneous step corrects the prediction to “would float” (Case 1). However, continuing with greedy decoding afterward still fails to yield the correct answer, while suppressing noise consistently results in a correct answer (Case 2). By analyzing the probability distribution when predicting “yes” or “no” (Figure 1), we observe that the model cannot confidently select an answer, and denoising significantly enhances the confidence while steering the model toward the correct choice.
Figure 4: Averaged decoding latency per token in milliseconds (ms/token) for LLaMA-30B.

Figure 5: ALW-sim vs ALW across five different datasets in LLaMA-7B.

This suggests that ALW performs probability recalibration at key decision points, amplifying the signal of the correct answer. Notably, if ALW is applied throughout the entire reasoning process, it suppresses the shallow statistical association of “fruit → float” while reinforcing the deeper physical rule of “density < water → float”. Consequently, ALW guides the reasoning path in a more principled direction from the outset by mitigating noise.

Latency. We further examine the decoding latency. As illustrated in Figure 4, ALW registers only a 6% increase compared to greedy decoding. Compared to DoLa (+10%) and CD (+37%), ALW results in almost no additional latency. Please refer to Appendix H for further analysis and explanation.

Elimination of Errors. Building on the research by Wang et al. (2023), we quantitatively investigate whether reasoning errors have been eliminated in the final layer and, if so, what types of errors are addressed. We analyze on Gsm8k before and after denoising, as shown in Table 9. Aside from “miss step” errors, the noise suppression significantly reduces various errors, particularly those related to model degeneration and arithmetic errors.

In addition, we conduct further analyses, including the rationale for selecting the first half of the layers (Appendix I) and case studies (Appendix J).

5 Related work

5.1 Early-Exiting Approaches

Panda et al. (2016) pioneered the exploration of early exiting in neural networks to accelerate the decoding process. Recently, early exit methods have become integral to efficient language models, as summarized by Khoshnoodi et al. (2024). Layer-Skip (Elhoushi et al., 2024) employs early exit layers as a speculative decoding model, while SkipDecode (Corro et al., 2023) establishes a single exit point for each token in a batch at every sequence position. Contemporaneous work includes EE-LLM (Chen et al., 2024) and ConsistentEE (Zeng et al., 2024), as well as studies summarized by Sajjad et al. (2023). However, these studies emphasize extreme efficiency, such as pruning 40% of the layers while retaining up to 98% of the original performance (Sajjad et al., 2023). In our study, early exiting serves as an intermediate step. We use early exit alongside the final layer to capture exploitable patterns for enhancing reasoning, focusing more on performance than efficiency.

5.2 Contrastive Decoding

Su et al. (2022) put forward the notion of Contrastive Search, a method dedicated to accentuating semantic distinctions among tokens within the representational space. Furthermore, Li et al. (2023) introduced Contrastive Decoding, exploiting the disparities between expert and amateur models. In addition, Gera et al. (2023); Chuang et al. (2024) involved experimental contrastive measures between different layers of a single backbone model, producing high-quality text by capitalizing on pre-existing language models. Recently, researchers Arias et al. (2024); Waldendorf et al. (2024); Sennrich et al. (2024) conducted numerous novel studies, primarily focusing on the open-text generation or contrast decoding to mitigate hallucinations. However, we focus on downstream reasoning tasks and offer a broader framework with novel explanations for effectiveness. Additionally, we leverage layer-wise noise patterns to enhance LLMs internally.

6 Conclusion

In this paper, we start from the assumptions of attention noise and explore the noise across different layers of LLMs. We observe that noise accumula-
tion across layers often leads to unstable token predictions during reasoning. To address this issue, we propose adaptive layer-wise contrastive decoding, a search-based decoding approach that enhances reasoning ability by dynamically disentangling noise in shallow layers from critical signals in deep layers. We design a quantification strategy and optimal layer selection methods based on LMs or similarity. The experimental results demonstrate that ALW significantly enhances the reasoning capabilities of LLMs, and further analysis reveals that ALW improves reasoning by focusing on the correct reasoning path through noise suppression.

Limitation

While the adaptive layer-wise contrastive decoding method has significantly enhanced the reasoning capabilities of Large Language Models, it still has some limitations:

• ALW is well-suited for challenging tasks. When the task is relatively simple, the shallow layers are highly confident in the prediction, leading to less noise present in the shallow layers, which can render ALW ineffective.

• We recommend using the more advanced ALW other than ALW-sim, while ALW necessitates training preparation before inference. Although the predictor is small-scale, it still adds additional computational overhead.

• Although ALW enhances reasoning ability by focusing on the correct reasoning path, it cannot fully resolve the issue of correct answers with incorrect paths. While ALW alleviates this, a small residue effect still remains.

7 Acknowledgments

We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), and the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001).

References

Amini Aida, Gabriel Saadia, Lin Peter, Koncel-Kedziorski Rik, Choi Yejin, and Hajishirzi Hannaneh. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319.

Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, and Matthias Aßennacher. 2024. Adaptive contrastive search: Uncertainty-guided decoding for open-ended text generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 15060–15080. Association for Computational Linguistics.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. EE-LLM: large-scale training and inference of early-exit large language models with 3d parallelism. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. 2023. Skipdecode: Autoregressive skip decoding with batching and caching for efficient LLM inference. CoRR, abs/2307.02628.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, and Carole-Jean Wu. 2024. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 12622–12642. Association for Computational Linguistics.

Ariel Gera, Roni Friedman, Ofir Ariviv, Chulaka Gunasekara, Benjamin Sznaider, Noam Slonim, and Eyal Shnarch. 2023. The benefits of bad advice: Autocontrastive decoding across model layers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 10406–10420. Association for Computational Linguistics.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL).

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yuqie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. 2022. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Ruixin Hong, Xinyu Pang, and Changshui Zhang. 2024. Advances in reasoning by prompting large language models: A survey. Cybernetics and Intelligence, pages 1–15.

Mahsa Khoshnoodi, Vinija Jain, Mingye Gao, Malavika Srikanth, and Aman Chadha. 2024. A comprehensive survey of accelerated generation techniques in large language models. CoRR, abs/2405.13019.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 12286–12312. Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Sean O’Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. CoRR, abs/2309.09117.

Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. 2016. Conditional deep learning for energy-efficient and enhanced pattern recognition. In 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 475–480. IEEE.

Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023. Reasoning with language model prompting: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5368–5393. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping layers of pretrained transformer models. Comput. Speech Lang., 77:101429.

Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. 2024. Mitigating hallucinations and off-target machine translation with source-contrastive and language-contrastive decoding. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 2: Short Papers, St. Julian’s, Malta, March 17-22, 2024, pages 21–33. Association for Computational Linguistics.

Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2024. Dynamollm: Designing LLM inference clusters for performance and energy efficiency. CoRR, abs/2408.00741.

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35:21548–21561.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Jonas Waldendorf, Barry Haddow, and Alexandra Birch. 2024. Contrastive decoding reduces hallucinations in large multilingual machine translation models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 1: Long Papers, St. Julian’s, Malta, March 17-22, 2024, pages 2526–2539. Association for Computational Linguistics.

Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2717–2739. Association for Computational Linguistics.

Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2024. Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 17716–17736. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao-ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, and Na Ni. 2024. Qwen2 technical report. CoRR, abs/2407.10671.

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. 2024. Differential transformer. arXiv preprint arXiv:2410.05258.

Ziqian Zeng, Yihuai Hong, Hongliang Dai, Huiping Zhuang, and Cen Chen. 2024. Consistentee: A consistent and hardness-guided early exiting method for accelerating language models inference. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 19506–19514. AAAI Press.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2024. A survey of large language models. Preprint, arXiv:2303.18223.

Minjun Zhu, Yixuan Weng, Shizhu He, Cunguang Wang, Kang Liu, Li Cai, and Jun Zhao. 2023. Learning to build reasoning chains by reliable path retrieval. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pages 1–5. IEEE.
A  More Preliminary Cases

We provide more preliminary cases in Figure 6 and 7. The observed phenomenon is consistent with the description in Figure 2. Additionally, the contrast operation helps alleviate prediction uncertainty in the final layer, enhancing the model’s confidence in the correct token. This effect arises from removing noise related to shallow layers in the final layer. The token-level representation, rather than the distribution-level representation, is shown in Figure 1.

B  Most Frequent Layer

Based on the frequency of layer occurrence in the training set, we selected a common layer to be contrasted during testing. Taking the use of LLaMA-65B for testing on BBH as an example, we considered layers 0 to 40 and the case without contrasting (−1), and tallied the occurrences of the optimal layer. Details are shown in Table 10. We observe that the layer requiring no contrast operation appears most frequently, while other layers are also involved, indicating that denoising is dynamically related to the input. Additionally, shallow layers appear more frequently than higher layers.

As layer 1 appeared 147 times, we fixed the contrast layer to the layer labeled as 1 during testing. We conduct experiments on Gsm8k, StraQA, and PiQA and the results are shown in Table 6. The numbers in parentheses indicate the change in accuracy when using common layer selection compared to using the predictor. The results in bold represent outcomes that exceed the greedy decoding results.

In fact, this idea is similar to the static mode in DoLa. DoLa employs a static method, consistently using the 0th layer to enhance the factuality of the LLMs. They believe that the static method is inferior to the dynamic method (baseline in our paper). However, our main experiments show that ALW outperforms DoLa’s dynamic method. The results just given also prove this point.

<table>
  <tr>
    <th></th>
    <th>7B</th>
    <th>13B</th>
    <th>30B</th>
    <th>65B</th>
  </tr>
  <tr>
    <td>Gsm8k</td>
    <td>0/12.1(-8.0)</td>
    <td>0/15.9(-6.8)</td>
    <td>0/<b>40.9</b>(-0.8)</td>
    <td>0/<b>52.3</b>(-3.0)</td>
  </tr>
  <tr>
    <td>PiQA</td>
    <td>1/78.0(-0.1)</td>
    <td>1/79.2(-0.4)</td>
    <td>1/<b>81.2</b>(-0.1)</td>
    <td>2/80.5(-0.8)</td>
  </tr>
  <tr>
    <td>StraQA</td>
    <td>0/65.1(-5.2)</td>
    <td>0/<b>70.3</b>(-2.2)</td>
    <td>0/<b>73.4</b>(-2.6)</td>
    <td>0/75.5(-2.2)</td>
  </tr>
</table>

Table 6: The result of using the most frequent layer as contrast layer. On the left of / is the layer number that appears the most, and on the right is the result.

C  A Description about Category

The LLM has an embedding layer and \( M \) transformer decode layers, labeled 1 to \( M \). We employ a quantification strategy to find the most important layer \( l \) among the first half of the early exit layers. Taking LLaMA-7B as an example, there are 32 decode layers labeled 1 to 32, and we consider layers 1 to 16 and the embedding layer, so the range of \( l \) is [0, 16]. However, we recognize that there are cases where the final layer itself has sufficient confidence to decode the gold token. In such cases, \( l \) is labeled −1. This is indeed equivalent to the final layer, but we want to highlight that no contrast operation was used, to avoid any confusion among researchers about how layer \( M \) could be contrasted with layer \( M \), since \( l \) refers to the contrasting layer. The notation \( l = -1 \) clearly indicates that this is a special case worth noting.

Based on the explanation above, the range of \( l \) is [−1, 16], and these 18 categories are the number of classes that the predictor needs to learn to classify.

D  Concerns about Evaluation Scope

In Section 4.1, we introduced that in addition to arithmetic and commonsense reasoning datasets, we also tested MMLU and BBH. They have been selected as standard evaluation benchmarks by the Open LLM Leaderboard for assessing large language models, which also proves their versatility and extensive applicability. In addition to MMLU and BBH, we evaluated Folio to assess the logical reasoning capabilities of LLMs, not limiting the scope to just arithmetic and commonsense reasoning.

E  More Experiment Details

Alignment of LM and LLM. We believe that a perfect match between RoBERTa tokenizer and the LLM tokenizer is not necessary. When RoBERTa predicts the optimal contrast layer, its input is the full, raw text, rather than the ID sequence tokenized by LLM. The task for RoBERTa is to predict which layer’s noise needs to be suppressed most for the current input, which is a sentence-level or paragraph-level semantic decision, rather than a fine-grained subword match. In fact, this approach is similar to the length prediction method used in LLM scheduling (Stojkovic et al., 2024), where BERT is employed to predict the possible output length for an LLM given the input to plan mem-
Question: When was the Internet first made available to the public?
Answer: The Internet was first made available to the public in the early 1980s.

Figure 6: More preliminary cases (case 1).

Question: When did the first human land on the Moon?
Answer: The first human landed on the Moon in July 20, 1969, during NASA’s Apollo 11 mission.

Figure 7: More preliminary cases (case 2).

ory usage, and such methods often do not require alignment between PLM and LLM tokenizers.

α for multiple choice tasks. When handling multiple choice questions, we compute the log probability sum of a given sentence. However, if any token in the sentence is not present in \( \mathcal{V}_{head} \), its score would be \( -\infty \) according to Equation 8, which would make the log probability sum of the entire sentence \( -\infty \), which is unreasonable. To address this, for tokens not found in \( \mathcal{V}_{head} \), we apply a penalty of \( -1000 \) instead of \( -\infty \), ensuring a more stable and reasonable computation.

GPUs. The main experiments were conducted on an NVIDIA GeForce RTX 3090. When running ALW with LLaMA-7B as the LLM and RoBERTa-base as the predictor, one RTX 3090 is required. The predictor remains the same. For the 13B model, two 3090 GPUs are required; for the 30B model, four GPUs; and for the 65B model, eight GPUs.

Since the predictor’s inference speed is very fast and does not require a lot of GPU communication, the budget is essentially consistent with simply inferring the LLaMA model.

Others. The number of shots assigned for the three open-ended generation datasets is six shots for StrategyQA, eight for Gsm8k, and three for Folio. Moreover, we present the CoT prompts for three open-ended generation datasets: Gsm8k, StrategyQA, and Folio. The CoT prompts for Gsm8k and StrategyQA are constructed according to the method outlined in (Wei et al., 2022). For the Folio dataset, prompts are constructed by a random selection and construction process, which is described in Table 13. In addition, when using ALW-sim, in order to balance the data distribution, we discard some data labeled –1 to maintain balance. Relatively, we added learning weight when using ALW-lm (Sec 3.3).
F  Concerns about Performance Variability

For concerns about variability in performance gains across different tasks and model sizes, we believe it is normal for some variation in gains across different settings. What we have developed is a universal reasoning framework that is applicable to various situations and having the added benefit is more important. We have achieved this goal.

In Section 4, we demonstrate the robust capabilities of ALW across eight datasets and five model sizes, ensuring that it can enhance reasoning abilities in most cases. It shows strong performance on datasets such as BBH, ARC-C, Gsm8k, and MathQA. For PiQA, where the gain in performance is smaller, we attempt to analyze in Section 4.3 that this may be because, for easier tasks, fewer noises are presented in shallow layers.

G  The Distribution of Predictor Outputs

We present the distribution of layers predicted by the predictor when testing Gsm8k with LLaMA-7B in Table 11. On 5,621 occasions, the predictor chose to skip the contrast operation. When a contrast is required, the predictor does not have an extreme tendency to predict a specific layer, and the overall distribution shows a trend of decreasing predicted frequency as the layer number increases. This shows that modeling the relationship between prompt and layer is not a simple task.

H  Concerns about Decoding Latency

We test the average time needed to decode each token using decoding methods on four NVIDIA GeForce RTX 3090 GPUs for the LLaMA-30B. As shown in Figure 4, ALW incurs the least time expenditure, being only 1.06 times that of the base, with virtually no increase in time cost.

Moreover, we consider additional operations from a mathematical perspective. For ALW-sim, we need to additionally calculate the cosine similarity of each row in the semantic representation vector [1, d] and the precomputed representation vector [n, d], which can be approximated as matrix multiplication of two [n, d] vectors, with a computational cost of only \( dn \) (d=384 in our experiment).

For ALW-lm, the additional operation is the forward process of LM and contrast operation. The RoBERTa-base model in our experiment has 110 million parameters, and if we use LLaMA-7B, its parameter count is 64 times that of RoBERTa-base.

Thus, for LLM inference, the inference time of the LM can be considered negligible. Furthermore, since LM inference runs in parallel with LLM, as long as the forward pass of the LM is shorter than the inference time of the LLM, the time impact introduced by the LM can be completely disregarded. In this case, the additional computations involve only **the subtraction of two hidden states and the softmax operation**.

I  Why First Half of All Layers?

As shown in Figure 2, 6 and 7, after surpassing half of the layers, the layers contain less noise and the predictions become more focused. That is, deeper layers tend to retain clearer signals, distinguishing them from the noise in the shallow layers. Therefore, theoretically, subtracting the deeper layers in the final stage is unlikely to yield better results. Moreover, if we consider only the latter half of the layers, the search space is consistent with the first half but the occurrence is less frequent, covering a narrower range of scenarios. On the other hand, using all layers from 0 to 32 undoubtedly greatly increases the search space, and the increased number of classifications makes it more difficult for the predictor to train. We conducted supplementary experiments in Table 12. We searched within the latter half of the layers, and the results indicated a significant decrease in performance. This is consistent with the phenomena observed in Dola.

J  Qualitative Study

This section presents generation cases from ALW and baselines across three open-ended generation datasets: Gsm8k (Table 16), StrategyQA (Table 17), and Folio (Table 18). Using Folio as an example, in the case shown in Table 18, ALW and greedy decoding produced entirely opposite generations when decoding \( \forall x \) (Student(x) \( \rightarrow \) Teacher(x)). The erroneous reasoning in the greedy decoding led to \( \neg \)Student(leon) \( \rightarrow \neg \)Teacher(leon), resulting in the conclusion “unknown”. In contrast, ALW accurately expressed \( \forall x \) (Teacher(x) \( \rightarrow \neg \)Student(x)) according to the given description, correcting the erroneous reasoning path of the greedy decoder and leading to the correct result “True”.

However, there are issues where the result is correct but the reasoning path is incorrect, and we analyze this situation in Table 19. In fact, we have explained in Section 4.3 that the reason ALW improves reasoning is that LLMs often struggle with
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Arithmetic</th>
    <th colspan="3">Commonsense</th>
    <th colspan="4">General</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Gsm8k*</th>
    <th>MathQ</th>
    <th>StrQA*</th>
    <th>PiQA</th>
    <th>Folio*</th>
    <th>MMLU</th>
    <th>BBH</th>
    <th>ARC-C</th>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-7B-HF</td>
    <td>Greedy</td>
    <td>13.6</td>
    <td>22.3</td>
    <td>66.4</td>
    <td>78.5</td>
    <td>16.5</td>
    <td>34.8</td>
    <td>34.1</td>
    <td>37.0</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>Ours-sim</td>
    <td><b>15.2</b></td>
    <td><b>28.6</b></td>
    <td><b>66.4</b></td>
    <td>78.0</td>
    <td><b>19.8</b></td>
    <td><b>36.9</b></td>
    <td><b>39.5</b></td>
    <td><b>40.1</b></td>
    <td><b>41.0</b></td>
  </tr>
  <tr>
    <td>Ours-lm</td>
    <td>20.1</td>
    <td>36.3</td>
    <td>70.3</td>
    <td>78.1</td>
    <td>29.8</td>
    <td>40.6</td>
    <td>41.7</td>
    <td>41.3</td>
    <td>45.5</td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-8B-HF</td>
    <td>Greedy</td>
    <td>76.5</td>
    <td>38.1</td>
    <td>71.6</td>
    <td>79.8</td>
    <td>52.5</td>
    <td>38.4</td>
    <td>43.7</td>
    <td>41.7</td>
    <td>55.3</td>
  </tr>
  <tr>
    <td>Ours-sim</td>
    <td><b>76.5</b></td>
    <td><b>44.7</b></td>
    <td>70.2</td>
    <td>78.3</td>
    <td><b>52.5</b></td>
    <td><b>40.4</b></td>
    <td><b>46.6</b></td>
    <td><b>42.3</b></td>
    <td><b>56.4</b></td>
  </tr>
  <tr>
    <td>Ours-lm</td>
    <td>78.8</td>
    <td>56.3</td>
    <td>71.6</td>
    <td>80.1</td>
    <td>52.9</td>
    <td>44.2</td>
    <td>50.8</td>
    <td>47.8</td>
    <td>60.3</td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-13B-HF</td>
    <td>Greedy</td>
    <td>18.9</td>
    <td>24.9</td>
    <td>67.2</td>
    <td>80.1</td>
    <td>34.7</td>
    <td>36.2</td>
    <td>37.2</td>
    <td>36.5</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>Ours-sim</td>
    <td>14.4</td>
    <td><b>30.2</b></td>
    <td><b>70.2</b></td>
    <td>79.7</td>
    <td><b>38.8</b></td>
    <td><b>37.2</b></td>
    <td><b>40.1</b></td>
    <td><b>37.9</b></td>
    <td><b>43.9</b></td>
  </tr>
  <tr>
    <td>Ours-lm</td>
    <td>22.7</td>
    <td>37.8</td>
    <td>72.5</td>
    <td>79.6</td>
    <td>50.4</td>
    <td>38.2</td>
    <td>44.8</td>
    <td>42.1</td>
    <td>48.5</td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-30B-HF</td>
    <td>Greedy</td>
    <td>32.6</td>
    <td>26.3</td>
    <td>72.9</td>
    <td>81.2</td>
    <td>44.6</td>
    <td>37.8</td>
    <td>39.5</td>
    <td>37.2</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>Ours-sim</td>
    <td><b>32.6</b></td>
    <td><b>28.3</b></td>
    <td>70.2</td>
    <td><b>81.2</b></td>
    <td><b>44.6</b></td>
    <td><b>39.6</b></td>
    <td><b>41.7</b></td>
    <td><b>38.5</b></td>
    <td><b>47.4</b></td>
  </tr>
  <tr>
    <td>Ours-lm</td>
    <td>41.7</td>
    <td>35.7</td>
    <td>76.0</td>
    <td>81.3</td>
    <td>44.6</td>
    <td>40.4</td>
    <td>45.0</td>
    <td>41.0</td>
    <td>50.7</td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA-65B-HF</td>
    <td>Greedy</td>
    <td>50.8</td>
    <td>27.2</td>
    <td>76.0</td>
    <td>81.3</td>
    <td>42.1</td>
    <td>40.4</td>
    <td>37.9</td>
    <td>39.6</td>
    <td>49.4</td>
  </tr>
  <tr>
    <td>Ours-sim</td>
    <td>46.2</td>
    <td><b>31.5</b></td>
    <td>74.2</td>
    <td>80.1</td>
    <td><b>42.1</b></td>
    <td><b>41.0</b></td>
    <td><b>39.2</b></td>
    <td><b>43.1</b></td>
    <td><b>50.1</b></td>
  </tr>
  <tr>
    <td>Ours-lm</td>
    <td>55.3</td>
    <td>38.5</td>
    <td>77.7</td>
    <td>81.3</td>
    <td>43.8</td>
    <td>41.4</td>
    <td>43.1</td>
    <td>47.8</td>
    <td>53.6</td>
  </tr>
</table>

Table 7: The performance of ALW-sim and ALW-lm across all settings. The non-bold entries indicate that ALW-sim is weaker than greedy decoding. The performance of ALW-sim lies between the base and ALW. We recommend using the more advanced ALW, but if training resources are extremely limited, ALW-sim can serve as a substitute.

<table>
  <tr>
    <th>Datasets</th>
    <th>LLaMA-7B-HF</th>
    <th>LLaMA-13B-HF</th>
    <th>LLaMA-30B-HF</th>
    <th>LLaMA-65B-HF</th>
  </tr>
  <tr>
    <td>Gsm8k*</td>
    <td>11.4 (-8.7)</td>
    <td>18.9 (-3.8)</td>
    <td>32.6 (-9.1)</td>
    <td>50.0 (-5.3)</td>
  </tr>
  <tr>
    <td>PiQA</td>
    <td>74.4 (-3.7)</td>
    <td>72.8 (-6.8)</td>
    <td>74.9 (-6.4)</td>
    <td>73.9 (-7.4)</td>
  </tr>
  <tr>
    <td>Folio*</td>
    <td><b>26.4</b> (-3.4)</td>
    <td><b>38.8</b> (-11.6)</td>
    <td>33.1 (-11.5)</td>
    <td>32.2 (-11.6)</td>
  </tr>
  <tr>
    <td>MMLU</td>
    <td><b>35.9</b> (-4.7)</td>
    <td>34.4 (-3.8)</td>
    <td><b>37.9</b> (-2.5)</td>
    <td>36.0 (-5.4)</td>
  </tr>
</table>

Table 8: The importance of layer predictor. The numbers in parentheses indicate the change in accuracy when using random layer selection compared to using the predictor. The results in bold represent outcomes that exceed the greedy decoding results.

<table>
  <tr>
    <th>Contrast</th>
    <th>Arithmetic</th>
    <th>Missing Step</th>
    <th>Semantic</th>
    <th>Degeneration</th>
    <th>Total Errors</th>
  </tr>
  <tr>
    <td>×</td>
    <td>6.1%</td>
    <td>30.3%</td>
    <td>42.4%</td>
    <td>7.6%</td>
    <td>86.4%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td><b>5.3%</b></td>
    <td>30.3%</td>
    <td><b>40.9%</b></td>
    <td><b>3.0%</b></td>
    <td><b>79.5%</b></td>
  </tr>
</table>

Table 9: Proportion of errors in of a set of 132 Gsm8k questions. Contrast significantly mitigates various errors, especially the model’s degeneration and arithmetic errors.

<table>
  <tr>
    <th>layer</th>
    <th>-1</th>
    <th>0</th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
    <th>4</th>
    <th>5</th>
    <th>6</th>
    <th>7</th>
    <th>8</th>
    <th>9</th>
    <th>10</th>
    <th>11</th>
    <th>12</th>
    <th>13</th>
    <th>14</th>
    <th>15</th>
    <th>16</th>
    <th>17</th>
    <th>18</th>
    <th>19</th>
  </tr>
  <tr>
    <td>num</td>
    <td>1742</td>
    <td>30</td>
    <td><b>147</b></td>
    <td>131</td>
    <td>29</td>
    <td>36</td>
    <td>29</td>
    <td>20</td>
    <td>27</td>
    <td>48</td>
    <td>28</td>
    <td>31</td>
    <td>46</td>
    <td>19</td>
    <td>82</td>
    <td>51</td>
    <td>47</td>
    <td>40</td>
    <td>37</td>
    <td>33</td>
    <td>67</td>
  </tr>
  <tr>
    <td>layer</td>
    <td>20</td>
    <td>21</td>
    <td>22</td>
    <td>23</td>
    <td>24</td>
    <td>25</td>
    <td>26</td>
    <td>27</td>
    <td>28</td>
    <td>29</td>
    <td>30</td>
    <td>31</td>
    <td>32</td>
    <td>33</td>
    <td>34</td>
    <td>35</td>
    <td>36</td>
    <td>37</td>
    <td>38</td>
    <td>39</td>
    <td>40</td>
  </tr>
  <tr>
    <td>num</td>
    <td>58</td>
    <td>25</td>
    <td>23</td>
    <td>9</td>
    <td>21</td>
    <td>16</td>
    <td>23</td>
    <td>29</td>
    <td>22</td>
    <td>28</td>
    <td>35</td>
    <td>25</td>
    <td>18</td>
    <td>50</td>
    <td>30</td>
    <td>33</td>
    <td>27</td>
    <td>16</td>
    <td>21</td>
    <td>9</td>
    <td>16</td>
  </tr>
</table>

Table 10: The statistics of the optimal layer. Using LLaMA-65B and BBH datasets. The layer requiring no contrast operation appears most frequently, while other layers are also involved. Moreover, shallow layers appear more frequently than higher layers.
<table>
  <tr>
    <th>layer</th>
    <th>n</th>
    <th>layer</th>
    <th>n</th>
  </tr>
  <tr>
    <td>1</td>
    <td>5621</td>
    <td>8</td>
    <td>236</td>
  </tr>
  <tr>
    <td>0</td>
    <td>1358</td>
    <td>9</td>
    <td>130</td>
  </tr>
  <tr>
    <td>1</td>
    <td>1114</td>
    <td>10</td>
    <td>256</td>
  </tr>
  <tr>
    <td>2</td>
    <td>928</td>
    <td>11</td>
    <td>161</td>
  </tr>
  <tr>
    <td>3</td>
    <td>964</td>
    <td>12</td>
    <td>154</td>
  </tr>
  <tr>
    <td>4</td>
    <td>825</td>
    <td>13</td>
    <td>512</td>
  </tr>
  <tr>
    <td>5</td>
    <td>335</td>
    <td>14</td>
    <td>186</td>
  </tr>
  <tr>
    <td>6</td>
    <td>448</td>
    <td>15</td>
    <td>86</td>
  </tr>
  <tr>
    <td>7</td>
    <td>741</td>
    <td>16</td>
    <td>252</td>
  </tr>
</table>

Table 11: The distribution of predictor outputs

<table>
  <tr>
    <th></th>
    <th>7B</th>
    <th>13B</th>
    <th>30B</th>
  </tr>
  <tr>
    <td>[0, 16]</td>
    <td>20.1</td>
    <td>22.7</td>
    <td>41.7</td>
  </tr>
  <tr>
    <td>[16, 32]</td>
    <td>12.1</td>
    <td>15.2</td>
    <td>30.3</td>
  </tr>
  <tr>
    <td>Vanilla</td>
    <td>13.6</td>
    <td>18.9</td>
    <td>32.6</td>
  </tr>
</table>

Table 12: The comparison between results of using different model sizes in Gsm8k.

path decisions, and ALW enables the model to focus confidently on deep reasoning signals, thereby making the correct next-step choice.

Observing an example from Gsm8k in Table 19, ALW incorrectly multiplies the task time by the total quantity. However, the erroneous calculations for small and large holes happen to cancel each other out (30×3 + 15×10 = 240 minutes = 4 hours), which coincidentally matches the correct calculation (30×3 = 90 minutes, 15×10 = 150 minutes, total = 240 minutes). Contrastive decoding works when the model struggles to choose between the correct or incorrect token. That is, if the model strongly favors an incorrect token (e.g., 95% probability), it is difficult to change the relative probabilities between the incorrect and correct tokens through contrastive decoding. We believe the correctness of the reasoning path depends on the model’s understanding of the problem. If, for a given model, there is no correct path in the path space or if its probability is very low, it is indeed difficult to correct it using ALW. In fact, when comparing the outputs from ALW and greedy decoding, the vanilla output is repetitive, indicating that the model was already confused about the problem. In this case, ALW can only improve the quality of the content in these cases. Observing an example from Folio, all reasoning paths are correct except for “∀x (BasketballPlayer(x) → American(x))”. In fact, because ALW has already inferred that Yuri is not a basketball player, it can already answer that Yuri is not an American basketball player. The statement “∀x (BasketballPlayer(x) → American(x))” should have been correctly modeled as “∃”, which might be due to the very limited presence of data including “∃” in the training set.

K Sub-task Results

BBH Sub-tasks. As shown in Table 14, all models on the “penguins_in_a_table” task show significant improvement (+40.0 to +67.0), with an average score of 82.6 (+53.4). This task involves table-based data reasoning, where ALW can more effectively extract structured information by denoising, thereby reducing redundant interference. Additionally, all models exhibit significant improvement on “snarks”, with an average increase of 13.1 points. The “snarks” typically involves implicit meaning in complex contexts, which heavily relies on the model’s understanding of deep semantic and contextual relationships. ALW enhances the model’s ability to capture deep semantic features by suppressing shallow noise (e.g., literal ambiguity, surface feature interference).

However, performance is suboptimal on some tasks, such as “movie_recommendation”, where the average score drops to 68.0 (−7.2) after using ALW, significantly lower than other semantic tasks. Movie recommendation requires integrating multi-dimensional information, such as user preferences (e.g., actors, directors, genres) and context (e.g., time, ratings). Shallow layers may capture local high-frequency features (e.g., “sci-fi”, “high rating”), but ALW mistakenly treats them as noise to be suppressed, leading to the loss of key signals.

MMLU Sub-tasks. As shown in Table 15, there are differences in the performance of various sub-tasks across different models, particularly in more complex domains (such as economics and law), where performance fluctuations are more pronounced. In certain sub-tasks, ALW significantly improves reasoning performance through shallow layer adaptive selection and final layer denoising strategies. For example, more noticeable improvements can be observed in tasks related to mathematics and medicine. However, in areas related to philosophy and management, while some improvement is seen, the overall enhancement may be smaller, indicating that the reasoning process in these tasks may inherently be more straightforward or simple, requiring less denoising.
<table>
  <tr>
    <th>shot-1</th>
    <td><b>Context:</b> All kids love animals. All toddlers are kids. If someone loves animals, then they are not bad. All pirates are bad. If Nancy is not a pirate, then Nancy loves animals. If Nancy is not a toddler, then Nancy is bad.<br>
<b>Question:</b> The statement "Nancy is a pirate" is True, False or Unknown?<br>
A: \( \forall x\ (Kid(x) \rightarrow LoveAnimals(x)),\ \forall x\ (Toddler(x) \rightarrow Kid(x)),\ \forall x\ (LoveAnimals(x) \rightarrow \neg Bad(x)),\ \forall x\ (Pirate(x) \rightarrow Bad(x)),\ \neg Pirate(nancy) \rightarrow LoveAnimals(nancy),\ \neg Toddler(nancy) \rightarrow Bad(nancy)\). The statement is Unknown.</td>
  </tr>
  <tr>
    <th>shot-2</th>
    <td><b>Context:</b> Oxford Circus is a road junction connecting Oxford Street and Regent Street. Oxford Street is in London. John Nash designed the construction on Regent Street. John Nash designed Oxford Circus. John Nash is an architect in British. Oxford Circus is the entrance to Oxford Circus tube station which is a part of the Central line in 1900.<br>
<b>Question:</b> The statement "Oxford Circus is in London" is True, False or Unknown?<br>
A: OxfordStreet(oxfordCircus) \land RegentStreet(oxfordCircus), \( \forall x\ (OxfordStreet(x) \rightarrow London(x)),\ \exists x\ (\text{RegentStreet}(x) \land \text{Design}(johnNash,\ x)),\ \text{Design}(johnNash,\ oxfordCircus),\ \text{British}(johnNash) \land \text{Architect}(johnNash),\ \text{CentralLine}(oxfordCircus)\). The statement is True.</td>
  </tr>
  <tr>
    <th>shot-3</th>
    <td><b>Context:</b> Businesses are either sanctioned or unsanctioned. Sanctioned businesses are limited. Unsanctioned businesses are free. The Crude Oil Data Exchange is a business that isn’t free.<br>
<b>Question:</b> The statement "Crude Oil Data Exchange is unsanctioned" is True, False or Unknown?<br>
A: \( \forall x\ (\text{Business}(x) \rightarrow \text{Sanctioned}(x) \land \neg \text{Sanctioned}(x)),\ \forall x\ (\text{Business}(x) \land \text{Sanctioned}(x) \rightarrow \text{Limited}(x)),\ \forall x\ (\text{Business}(x) \land \neg \text{Sanctioned}(x) \rightarrow \text{Free}(x)),\ \text{Business(crudeoildataexchange)} \land \neg \text{Free(crudeoildataexchange)}\). The statement is False.</td>
  </tr>
</table>

Table 13: CoT prompt for Folio.

<table>
  <tr>
    <th>Subtasks</th>
    <th>LLaMA-7B</th>
    <th>LLaMA3-8B</th>
    <th>LLaMA-13B</th>
    <th>LLaMA-30B</th>
    <th>LLaMA-65B</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>temporal_sequences</td>
    <td>12.0<sub>4.0</sub></td>
    <td>40.0<sub>0.0</sub></td>
    <td>36.0<sub>24.0</sub></td>
    <td>20.0<sub>4.0</sub></td>
    <td>20.0<sub>4.0</sub></td>
    <td>25.6<sub>7.2</sub></td>
  </tr>
  <tr>
    <td>disambiguation_qa</td>
    <td>28.0<sub>–8.0</sub></td>
    <td>24.0<sub>–4.0</sub></td>
    <td>28.0<sub>–8.0</sub></td>
    <td>32.0<sub>4.0</sub></td>
    <td>28.0<sub>0.0</sub></td>
    <td>28.0<sub>–3.2</sub></td>
  </tr>
  <tr>
    <td>date_understanding</td>
    <td>88.0<sub>20.0</sub></td>
    <td>76.0<sub>12.0</sub></td>
    <td>80.0<sub>8.0</sub></td>
    <td>72.0<sub>–8.0</sub></td>
    <td>76.0<sub>–4.0</sub></td>
    <td>78.4<sub>5.6</sub></td>
  </tr>
  <tr>
    <td>tracking_shuffled_objects_three_objects</td>
    <td>24.0<sub>4.0</sub></td>
    <td>40.0<sub>0.0</sub></td>
    <td>32.0<sub>4.0</sub></td>
    <td>40.0<sub>8.0</sub></td>
    <td>24.0<sub>0.0</sub></td>
    <td>32.0<sub>3.2</sub></td>
  </tr>
  <tr>
    <td>penguins_in_a_table</td>
    <td>73.3<sub>40.0</sub></td>
    <td>100.0<sub>67.0</sub></td>
    <td>86.7<sub>46.7</sub></td>
    <td>80.0<sub>60.0</sub></td>
    <td>73.3<sub>53.3</sub></td>
    <td>82.6<sub>53.4</sub></td>
  </tr>
  <tr>
    <td>geometric_shapes</td>
    <td>8.0<sub>0.0</sub></td>
    <td>12.0<sub>4.0</sub></td>
    <td>8.0<sub>0.0</sub></td>
    <td>8.0<sub>0.0</sub></td>
    <td>12.0<sub>4.0</sub></td>
    <td>9.6<sub>1.6</sub></td>
  </tr>
  <tr>
    <td>snarks</td>
    <td>72.2<sub>16.9</sub></td>
    <td>72.2<sub>11.0</sub></td>
    <td>77.8<sub>10.8</sub></td>
    <td>72.2<sub>11.2</sub></td>
    <td>77.8<sub>16.7</sub></td>
    <td>74.4<sub>13.1</sub></td>
  </tr>
  <tr>
    <td>ruin_names</td>
    <td>60.0<sub>4.0</sub></td>
    <td>76.0<sub>12.0</sub></td>
    <td>64.0<sub>12.0</sub></td>
    <td>80.0<sub>16.0</sub></td>
    <td>64.0<sub>0.0</sub></td>
    <td>68.8<sub>8.8</sub></td>
  </tr>
  <tr>
    <td>tracking_shuffled_objects_seven_objects</td>
    <td>16.0<sub>4.0</sub></td>
    <td>16.0<sub>0.0</sub></td>
    <td>20.0<sub>–8.0</sub></td>
    <td>20.0<sub>0.0</sub></td>
    <td>20.0<sub>–4.0</sub></td>
    <td>18.4<sub>–1.6</sub></td>
  </tr>
  <tr>
    <td>tracking_shuffled_objects_five_objects</td>
    <td>24.0<sub>20.0</sub></td>
    <td>16.0<sub>–4.0</sub></td>
    <td>8.0<sub>0.0</sub></td>
    <td>12.0<sub>4.0</sub></td>
    <td>14.0<sub>–14.0</sub></td>
    <td>14.8<sub>8.1</sub></td>
  </tr>
  <tr>
    <td>logical_deduction_three_objects</td>
    <td>60.0<sub>20.0</sub></td>
    <td>64.0<sub>0.0</sub></td>
    <td>48.0<sub>12.0</sub></td>
    <td>52.0<sub>0.0</sub></td>
    <td>48.0<sub>4.0</sub></td>
    <td>54.4<sub>7.2</sub></td>
  </tr>
  <tr>
    <td>hyperbaton</td>
    <td>72.0<sub>–16.0</sub></td>
    <td>96.0<sub>24.0</sub></td>
    <td>84.0<sub>0.0</sub></td>
    <td>80.0<sub>–12.0</sub></td>
    <td>84.0<sub>0.0</sub></td>
    <td>83.2<sub>–0.8</sub></td>
  </tr>
  <tr>
    <td>logical_deduction_five_objects</td>
    <td>24.0<sub>4.0</sub></td>
    <td>32.0<sub>4.0</sub></td>
    <td>28.0<sub>12.0</sub></td>
    <td>36.0<sub>12.0</sub></td>
    <td>28.0<sub>8.0</sub></td>
    <td>29.6<sub>8.0</sub></td>
  </tr>
  <tr>
    <td>logical_deduction_seven_objects</td>
    <td>8.0<sub>0.0</sub></td>
    <td>40.0<sub>–4.0</sub></td>
    <td>24.0<sub>4.0</sub></td>
    <td>44.0<sub>8.0</sub></td>
    <td>24.0<sub>8.0</sub></td>
    <td>28.0<sub>3.2</sub></td>
  </tr>
  <tr>
    <td>movie_recommendation</td>
    <td>64.0<sub>–8.0</sub></td>
    <td>76.0<sub>–8.0</sub></td>
    <td>72.0<sub>0.0</sub></td>
    <td>56.0<sub>–20.0</sub></td>
    <td>72.0<sub>0.0</sub></td>
    <td>68.0<sub>–7.2</sub></td>
  </tr>
  <tr>
    <td>saliency_translation_error_detection:</td>
    <td>24.0<sub>8.0</sub></td>
    <td>32.0<sub>0.0</sub></td>
    <td>16.0<sub>0.0</sub></td>
    <td>16.0<sub>0.0</sub></td>
    <td>24.0<sub>8.0</sub></td>
    <td>22.4<sub>4.2</sub></td>
  </tr>
  <tr>
    <td>all</td>
    <td>41.1<sub>7.0</sub></td>
    <td>50.8<sub>7.1</sub></td>
    <td>44.5<sub>7.3</sub></td>
    <td>45.0<sub>5.5</sub></td>
    <td>43.1<sub>5.2</sub></td>
    <td>44.9<sub>6.4</sub></td>
  </tr>
</table>

Table 14: The results of BBH subtasks. The subscript indicates the difference from the greedy decoding, where red represents a lower value than greedy decoding, and blue indicates a value that is not lower than greedy decoding.
<table>
  <tr>
    <th>Subtasks</th>
    <th>LLaMA-7B</th>
    <th>LLaMA3-8B</th>
    <th>LLaMA-13B</th>
    <th>LLaMA-30B</th>
    <th>LLaMA-65B</th>
    <th>Average</th>
  </tr>
  <tr><td>abstract_algebra</td><td>60.0<sub>0.0</sub></td><td>50.0<sub>10.0</sub></td><td>50.0<sub>10.0</sub></td><td>80.0<sub>30.0</sub></td><td>70.0<sub>20.0</sub></td><td>62.0<sub>6.0</sub></td></tr>
  <tr><td>anatomy</td><td>42.9<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>57.1<sub>0.0</sub></td><td>57.1<sub>0.0</sub></td><td>50.0<sub>7.1</sub></td><td>51.4<sub>-1.4</sub></td></tr>
  <tr><td>astronomy</td><td>25.0<sub>6.2</sub></td><td>37.5<sub>0.0</sub></td><td>31.2<sub>0.0</sub></td><td>37.5<sub>6.2</sub></td><td>37.5<sub>6.2</sub></td><td>33.8<sub>-3.8</sub></td></tr>
  <tr><td>business_ethics</td><td>90.0<sub>20.0</sub></td><td>60.0<sub>0.0</sub></td><td>60.0<sub>10.0</sub></td><td>60.0<sub>10.0</sub></td><td>80.0<sub>10.0</sub></td><td>70.0<sub>20.0</sub></td></tr>
  <tr><td>clinical_knowledge</td><td>51.9<sub>1.1</sub></td><td>44.4<sub>1.7</sub></td><td>48.1<sub>1.4</sub></td><td>40.7<sub>1.4</sub></td><td>48.1<sub>1.7</sub></td><td>46.7<sub>6.7</sub></td></tr>
  <tr><td>college_biology</td><td>26.7<sub>13.3</sub></td><td>40.0<sub>6.7</sub></td><td>40.0<sub>13.3</sub></td><td>33.3<sub>0.0</sub></td><td>33.3<sub>0.0</sub></td><td>34.7<sub>6.7</sub></td></tr>
  <tr><td>college_chemistry</td><td>40.0<sub>0.0</sub></td><td>50.0<sub>10.0</sub></td><td>20.0<sub>-20.0</sub></td><td>40.0<sub>0.0</sub></td><td>10.0<sub>-20.0</sub></td><td>32.0<sub>-10.0</sub></td></tr>
  <tr><td>college_computer_science</td><td>50.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>60.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>52.0<sub>2.0</sub></td></tr>
  <tr><td>college_mathematics</td><td>30.0<sub>20.0</sub></td><td>20.0<sub>10.0</sub></td><td>40.0<sub>20.0</sub></td><td>40.0<sub>30.0</sub></td><td>60.0<sub>20.0</sub></td><td>38.0<sub>22.0</sub></td></tr>
  <tr><td>college_medicine</td><td>33.3<sub>0.0</sub></td><td>50.0<sub>16.7</sub></td><td>50.0<sub>11.1</sub></td><td>44.4<sub>22.2</sub></td><td>38.9<sub>16.7</sub></td><td>43.3<sub>3.3</sub></td></tr>
  <tr><td>college_physics</td><td>27.3<sub>0.0</sub></td><td>36.4<sub>9.1</sub></td><td>27.3<sub>0.0</sub></td><td>36.4<sub>9.1</sub></td><td>36.4<sub>9.1</sub></td><td>32.7<sub>5.5</sub></td></tr>
  <tr><td>computer_security</td><td>40.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>40.0<sub>10.0</sub></td><td>50.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>46.0<sub>-2.0</sub></td></tr>
  <tr><td>conceptual_physics</td><td>62.5<sub>1.2</sub></td><td>70.8<sub>8.3</sub></td><td>41.7<sub>-4.2</sub></td><td>54.2<sub>-12.5</sub></td><td>66.7<sub>4.2</sub></td><td>59.2<sub>-1.7</sub></td></tr>
  <tr><td>econometrics</td><td>33.3<sub>0.0</sub></td><td>33.3<sub>-16.7</sub></td><td>25.0<sub>0.0</sub></td><td>25.0<sub>0.0</sub></td><td>25.0<sub>0.0</sub></td><td>28.3<sub>-3.3</sub></td></tr>
  <tr><td>electrical_engineering</td><td>20.0<sub>-13.3</sub></td><td>46.7<sub>0.0</sub></td><td>33.3<sub>13.3</sub></td><td>20.0<sub>6.7</sub></td><td>20.0<sub>6.7</sub></td><td>28.0<sub>-2.7</sub></td></tr>
  <tr><td>elementary_mathematics</td><td>44.7<sub>15.8</sub></td><td>55.2<sub>21.1</sub></td><td>28.9<sub>-26.6</sub></td><td>34.2<sub>0.0</sub></td><td>39.5<sub>2.6</sub></td><td>40.5<sub>7.4</sub></td></tr>
  <tr><td>formal_logic</td><td>38.5<sub>7.7</sub></td><td>46.1<sub>7.7</sub></td><td>38.5<sub>7.7</sub></td><td>61.5<sub>15.4</sub></td><td>46.1<sub>7.7</sub></td><td>46.2<sub>6.2</sub></td></tr>
  <tr><td>global_facts</td><td>40.0<sub>0.0</sub></td><td>70.0<sub>20.0</sub></td><td>60.0<sub>30.0</sub></td><td>60.0<sub>20.0</sub></td><td>50.0<sub>10.0</sub></td><td>56.0<sub>8.0</sub></td></tr>
  <tr><td>high_school_biology</td><td>32.3<sub>19.4</sub></td><td>58.1<sub>32.2</sub></td><td>32.3<sub>19.2</sub></td><td>41.9<sub>15.5</sub></td><td>35.5<sub>-3.2</sub></td><td>40.0<sub>11.6</sub></td></tr>
  <tr><td>high_school_chemistry</td><td>42.9<sub>19.1</sub></td><td>57.1<sub>19.0</sub></td><td>38.1<sub>10.0</sub></td><td>38.1<sub>14.8</sub></td><td>47.6<sub>19.1</sub></td><td>44.8<sub>10.5</sub></td></tr>
  <tr><td>high_school_computer_science</td><td>60.0<sub>30.0</sub></td><td>40.0<sub>20.0</sub></td><td>50.0<sub>30.0</sub></td><td>60.0<sub>40.0</sub></td><td>60.0<sub>40.0</sub></td><td>54.0<sub>40.0</sub></td></tr>
  <tr><td>high_school_european_history</td><td>41.2<sub>-11.8</sub></td><td>29.4<sub>-11.8</sub></td><td>29.4<sub>-23.5</sub></td><td>29.4<sub>0.0</sub></td><td>29.4<sub>-17.6</sub></td><td>31.8<sub>-12.9</sub></td></tr>
  <tr><td>high_school_geography</td><td>60.0<sub>0.0</sub></td><td>55.0<sub>0.0</sub></td><td>55.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>54.0<sub>0.0</sub></td></tr>
  <tr><td>high_school_government_and_politics</td><td>65.0<sub>0.0</sub></td><td>50.0<sub>-5.0</sub></td><td>60.0<sub>-10.0</sub></td><td>65.0<sub>-10.0</sub></td><td>45.0<sub>-35.0</sub></td><td>57.0<sub>-12.0</sub></td></tr>
  <tr><td>high_school_macroeconomics</td><td>30.8<sub>-5.1</sub></td><td>33.3<sub>-2.6</sub></td><td>20.5<sub>-12.8</sub></td><td>30.8<sub>-2.6</sub></td><td>35.9<sub>-2.6</sub></td><td>30.3<sub>-4.1</sub></td></tr>
  <tr><td>high_school_mathematics</td><td>40.7<sub>9.5</sub></td><td>66.7<sub>55.6</sub></td><td>40.7<sub>4.7</sub></td><td>37.0<sub>7.4</sub></td><td>40.7<sub>11.1</sub></td><td>45.2<sub>21.5</sub></td></tr>
  <tr><td>high_school_microeconomics</td><td>45.8<sub>0.8</sub></td><td>50.0<sub>3.8</sub></td><td>45.8<sub>1.2</sub></td><td>37.5<sub>4.2</sub></td><td>50.0<sub>3.8</sub></td><td>45.8<sub>7.5</sub></td></tr>
  <tr><td>high_school_physics</td><td>56.2<sub>6.2</sub></td><td>50.0<sub>-12.5</sub></td><td>56.2<sub>6.2</sub></td><td>68.8<sub>18.8</sub></td><td>62.5<sub>18.8</sub></td><td>58.8<sub>5.0</sub></td></tr>
  <tr><td>high_school_psychology</td><td>54.5<sub>0.0</sub></td><td>67.5<sub>10.9</sub></td><td>54.5<sub>0.0</sub></td><td>54.5<sub>1.8</sub></td><td>54.5<sub>-10.9</sub></td><td>57.1<sub>7.0</sub></td></tr>
  <tr><td>high_school_statistics</td><td>27.3<sub>1.8</sub></td><td>40.9<sub>18.2</sub></td><td>27.3<sub>1.8</sub></td><td>40.9<sub>22.7</sub></td><td>27.7<sub>4.5</sub></td><td>31.8<sub>2.7</sub></td></tr>
  <tr><td>high_school_us_history</td><td>33.3<sub>1.8</sub></td><td>42.9<sub>28.6</sub></td><td>33.3<sub>1.8</sub></td><td>38.1<sub>19.1</sub></td><td>33.3<sub>1.8</sub></td><td>36.2<sub>0.5</sub></td></tr>
  <tr><td>high_school_world_history</td><td>20.8<sub>2.2</sub></td><td>33.3<sub>3.3</sub></td><td>20.8<sub>0.0</sub></td><td>25.0<sub>8.3</sub></td><td>33.3<sub>16.7</sub></td><td>26.7<sub>7.5</sub></td></tr>
  <tr><td>human_aging</td><td>43.5<sub>0.0</sub></td><td>47.8<sub>13.1</sub></td><td>52.2<sub>8.7</sub></td><td>65.2<sub>13.1</sub></td><td>39.1<sub>-13.0</sub></td><td>49.6<sub>4.3</sub></td></tr>
  <tr><td>human_sexuality</td><td>42.9<sub>0.0</sub></td><td>42.9<sub>7.2</sub></td><td>50.0<sub>0.0</sub></td><td>35.7<sub>0.0</sub></td><td>50.0<sub>14.3</sub></td><td>44.3<sub>5.7</sub></td></tr>
  <tr><td>international_law</td><td>30.8<sub>14.5</sub></td><td>23.1<sub>1.7</sub></td><td>23.1<sub>1.7</sub></td><td>23.1<sub>1.7</sub></td><td>38.5<sub>14.5</sub></td><td>27.7<sub>7.7</sub></td></tr>
  <tr><td>jurisprudence</td><td>18.2<sub>1.1</sub></td><td>45.5<sub>0.1</sub></td><td>18.2<sub>0.1</sub></td><td>27.3<sub>0.1</sub></td><td>36.4<sub>18.2</sub></td><td>29.1<sub>-3.6</sub></td></tr>
  <tr><td>logical_fallacies</td><td>41.2<sub>0.0</sub></td><td>35.3<sub>-17.6</sub></td><td>35.3<sub>-17.6</sub></td><td>23.5<sub>-11.8</sub></td><td>41.2<sub>5.9</sub></td><td>35.3<sub>-10.6</sub></td></tr>
  <tr><td>machine_learning</td><td>41.7<sub>0.0</sub></td><td>25.0<sub>-33.3</sub></td><td>41.7<sub>-8.3</sub></td><td>25.0<sub>-25.0</sub></td><td>41.7<sub>-8.3</sub></td><td>35.0<sub>-15.0</sub></td></tr>
  <tr><td>management</td><td>45.5<sub>0.1</sub></td><td>63.0<sub>27.3</sub></td><td>36.4<sub>0.0</sub></td><td>45.5<sub>0.0</sub></td><td>45.5<sub>0.0</sub></td><td>47.3<sub>7.3</sub></td></tr>
  <tr><td>marketing</td><td>58.3<sub>-16.7</sub></td><td>58.3<sub>-12.5</sub></td><td>50.0<sub>-12.5</sub></td><td>66.7<sub>8.3</sub></td><td>58.3<sub>0.0</sub></td><td>58.3<sub>-6.7</sub></td></tr>
  <tr><td>medical_genetics</td><td>80.0<sub>0.0</sub></td><td>70.0<sub>0.0</sub></td><td>90.0<sub>0.0</sub></td><td>70.0<sub>0.0</sub></td><td>90.0<sub>20.0</sub></td><td>80.0<sub>20.0</sub></td></tr>
  <tr><td>miscellaneous</td><td>48.1<sub>-7.6</sub></td><td>77.2<sub>21.4</sub></td><td>46.8<sub>-7.6</sub></td><td>54.4<sub>-5.1</sub></td><td>57.0<sub>-3.8</sub></td><td>56.7<sub>0.0</sub></td></tr>
  <tr><td>moral_disputes</td><td>34.3<sub>0.0</sub></td><td>28.6<sub>-5.7</sub></td><td>34.3<sub>-2.9</sub></td><td>31.4<sub>0.0</sub></td><td>42.9<sub>7.0</sub></td><td>34.3<sub>-0.6</sub></td></tr>
  <tr><td>moral_scenarios</td><td>60.0<sub>6.7</sub></td><td>23.3<sub>0.0</sub></td><td>32.2<sub>9.0</sub></td><td>34.4<sub>11.1</sub></td><td>28.9<sub>6.7</sub></td><td>35.8<sub>2.4</sub></td></tr>
  <tr><td>nutrition</td><td>22.6<sub>5.5</sub></td><td>22.6<sub>5.5</sub></td><td>22.6<sub>5.5</sub></td><td>22.6<sub>0.0</sub></td><td>22.6<sub>0.0</sub></td><td>22.6<sub>3.2</sub></td></tr>
  <tr><td>philosophy</td><td>31.2<sub>15.6</sub></td><td>25.0<sub>0.0</sub></td><td>25.0<sub>6.2</sub></td><td>31.2<sub>5.1</sub></td><td>40.6<sub>18.7</sub></td><td>30.6<sub>8.8</sub></td></tr>
  <tr><td>prehistory</td><td>45.5<sub>0.0</sub></td><td>54.5<sub>0.0</sub></td><td>54.5<sub>0.0</sub></td><td>54.5<sub>0.0</sub></td><td>54.5<sub>0.0</sub></td><td>52.7<sub>1.2</sub></td></tr>
  <tr><td>professional_accounting</td><td>20.7<sub>-6.9</sub></td><td>37.9<sub>0.0</sub></td><td>34.5<sub>6.9</sub></td><td>20.7<sub>-10.3</sub></td><td>31.0<sub>-17.2</sub></td><td>29.0<sub>-5.5</sub></td></tr>
  <tr><td>professional_law</td><td>22.7<sub>0.7</sub></td><td>29.2<sub>4.2</sub></td><td>25.3<sub>1.9</sub></td><td>23.4<sub>2.6</sub></td><td>26.0<sub>3.3</sub></td><td>25.3<sub>0.6</sub></td></tr>
  <tr><td>professional_medicine</td><td>39.3<sub>3.6</sub></td><td>42.9<sub>7.2</sub></td><td>32.1<sub>-3.6</sub></td><td>32.1<sub>2.6</sub></td><td>35.7<sub>-3.6</sub></td><td>36.4<sub>1.4</sub></td></tr>
  <tr><td>professional_psychology</td><td>45.2<sub>1.6</sub></td><td>48.4<sub>5.5</sub></td><td>46.8<sub>9.7</sub></td><td>50.0<sub>0.7</sub></td><td>51.6<sub>1.2</sub></td><td>48.4<sub>6.1</sub></td></tr>
  <tr><td>public_relations</td><td>81.8<sub>27.3</sub></td><td>77.2<sub>9.1</sub></td><td>81.8<sub>0.1</sub></td><td>81.8<sub>0.1</sub></td><td>81.8<sub>18.2</sub></td><td>80.0<sub>4.5</sub></td></tr>
  <tr><td>security_studies</td><td>32.0<sub>0.0</sub></td><td>28.0<sub>-1.0</sub></td><td>36.0<sub>0.0</sub></td><td>36.0<sub>0.0</sub></td><td>36.0<sub>0.0</sub></td><td>33.6<sub>2.4</sub></td></tr>
  <tr><td>sociology</td><td>28.6<sub>4.8</sub></td><td>47.6<sub>21.8</sub></td><td>33.3<sub>1.8</sub></td><td>33.3<sub>4.8</sub></td><td>23.8<sub>-9.5</sub></td><td>33.3<sub>5.7</sub></td></tr>
  <tr><td>us_foreign_policy</td><td>40.0<sub>30.0</sub></td><td>20.0<sub>0.0</sub></td><td>20.0<sub>0.0</sub></td><td>30.0<sub>0.0</sub></td><td>40.0<sub>0.0</sub></td><td>30.0<sub>4.0</sub></td></tr>
  <tr><td>virology</td><td>29.4<sub>0.0</sub></td><td>23.5<sub>0.0</sub></td><td>17.6<sub>-5.9</sub></td><td>41.2<sub>0.0</sub></td><td>29.4<sub>0.0</sub></td><td>28.2<sub>-1.2</sub></td></tr>
  <tr><td>world_religions</td><td>44.4<sub>0.0</sub></td><td>44.4<sub>0.0</sub></td><td>50.0<sub>0.0</sub></td><td>55.6<sub>0.0</sub></td><td>55.6<sub>-2.8</sub></td><td>50.0<sub>-4.4</sub></td></tr>
  <tr><td>all</td><td>40.6<sub>5.8</sub></td><td>44.2<sub>6.9</sub></td><td>38.2<sub>2.0</sub></td><td>40.4<sub>2.6</sub></td><td>41.4<sub>1.0</sub></td><td>41.0<sub>3.7</sub></td></tr>
</table>

Table 15: The results of MMLU subtasks.
<table>
  <tr>
    <th>Question</th>
    <td>James runs 12 miles a day for 5 days a week. If he runs 10 miles an hour how many hours does he run a week?</td>
  </tr>
  <tr>
    <th>ALW (√)</th>
    <td>James runs 12 miles a day for 5 days a week. That means he runs 12 * 5 = 60 miles a week. If he runs 10 miles an hour, it takes him 6 hours to run 60 miles. The answer is 6 hours.</td>
  </tr>
  <tr>
    <th>Baseline (×)</th>
    <td>James runs 12 miles a day for 5 days a week. 12 miles is 12 x 5 = 60 miles. 10 miles an hour is 10 x 60 = 600 miles. So he runs 600 miles a week. 600 / 60 = 10 hours. The answer is 10 hours.</td>
  </tr>
</table>

Table 16: Qualitative Study for LLaMA-7B with ALW and baselines on Gsm8k.

<table>
  <tr>
    <th>Question</th>
    <td>If your skin was turning the color of a zombie, could it be because of nickel?</td>
  </tr>
  <tr>
    <th>ALW (√)</th>
    <td>Nickel is a metal that causes allergic reactions in some people. A common symptom of nickel allergy is a greenish discoloration of the skin. Zombies are often depicted with green skin. Thus, if your skin was turning the color of a zombie, it could be because of nickel. So the answer is yes.</td>
  </tr>
  <tr>
    <th>Baseline (×)</th>
    <td>Nickel is a metal. Metals are not known to cause skin to turn green. Thus, your skin turning green is not likely to be caused by nickel. So the answer is no.</td>
  </tr>
</table>

Table 17: Qualitative Study for LLaMA-30B with ALW and baselines on StrategyQA.

<table>
  <tr>
    <th>Question</th>
    <td>Context: All professors are teachers. All researchers are professors. No teachers are students. All people who study are students. Leon is not studying or a researcher. If Leon is not studying, then Leon is a teacher.<br>Question: The statement "Leon is neither a student nor a researcher." is True, False or Unknown?</td>
  </tr>
  <tr>
    <th>ALW (√)</th>
    <td>∀x (Professor(x) → Teacher(x)), ∀x (Researcher(x) → Professor(x)), ∀x (Teacher(x) → Not(Student(x))), ~Studying(leon) → Teacher(leon), ~Researcher(leon) → Teacher(leon), Teacher(leon) → Not(Student(leon)). The statement is True.</td>
  </tr>
  <tr>
    <th>Baseline (×)</th>
    <td>∀x (Professor(x) → Teacher(x)), ∀x (Researcher(x) → Professor(x)), ∀x (Student(x) → Teacher(x)), ∀x (Teacher(x) → Not(Student(x))), ~Student(leon) → ~Teacher(leon), ~Researcher(leon) → Teacher(leon). The statement is Unknown.</td>
  </tr>
</table>

Table 18: Qualitative Study for LLaMA-13B with ALW and baselines on Folio.

<table>
  <tr>
    <th>Question</th>
    <td>It takes Matthew 3 minutes to dig a small hole for shrubs and 10 minutes to dig a large hole for trees. How many hours will it take him to dig 30 small holes and 15 large holes?</td>
  </tr>
  <tr>
    <th>ALW (√)</th>
    <td>Matthew can dig 30 small holes in 3 minutes. That means he can dig 30 x 3 = 90 small holes in 1 hour. He can also dig 15 large holes in 10 minutes. That means he can dig 15 x 10 = 150 large holes in 1 hour. So he can dig 90 + 150 = 240 holes in 1 hour. 240 / 60 = 4 hours. The answer is 4.</td>
  </tr>
  <tr>
    <th>Baseline (×)</th>
    <td>Matthew will dig 30 small holes in 30 minutes. 10 minutes to dig a large hole. 30 small holes + 10 large holes = 40 holes. 40 holes in 30 minutes + 10 holes in 10 minutes = 50 minutes. 50 minutes / 60 minutes = 0.833. 0.833 hours. The answer is 0.833 hours.</td>
  </tr>
</table>

<table>
  <tr>
    <th>Question</th>
    <td>Context: Some basketball players are not American. All basketball players are tall. If someone is tall, they wear large shoes. If someone wears large shoes, they wear large-sized clothes. Yuri does not wear large-size clothes.<br>Question: The statement "Yuri is an American basketball player." is True, False or Unknown?</td>
  </tr>
  <tr>
    <th>ALW (√)</th>
    <td>∀x (BasketballPlayer(x) → American(x)), ∀x (BasketballPlayer(x) → Tall(x)), ∀x (Tall(x) → WearLargeShoes(x)), ∀x (WearLargeShoes(x) → WearLargeSizeClothes(x)), ~WearLargeSizeClothes(yuri). The statement is False.</td>
  </tr>
  <tr>
    <th>Baseline (×)</th>
    <td>∀x (BasketballPlayer(x) → American(x)), ∀x (Tall(x) → BasketballPlayer(x)), ∀x (LargeShoes(x) → LargeClothes(x)), ∀x (LargeClothes(x) → BasketballPlayer(x)), ~LargeClothes(yuri) → Yuri. The statement is True.</td>
  </tr>
</table>

Table 19: Discussion on cases where the reasoning path is incorrect but the answer is correct.

Reviewer paper summaries: The authors propose Adaptive Layer-Wise Contrastive Decoding (ALW), a novel framework that dynamically disentangles noise in shallow layers from critical signals in deep layers. By contrasting probability distributions across layers, ALW improves reasoning accuracy while maintaining inference efficiency. Experiments on benchmarks like GSM8K and BBH demonstrate significant performance gains.

This paper considers noise accumulation across layers in LLMs. The authors propose ​Adaptive Layer-Wise Contrastive Decoding (ALW), a framework that dynamically suppresses shallow-layer noise by contrasting probability distributions between layers. Experiments on reasoning benchmarks demonstrate the effectiveness of ALW.

The paper tests the impact of an early exit strategy on a variety of reasoning benchmarks. In particular, an encoder transformer LM is used alongside different versions of LLama. The encoder is trained to predict the early exit layer of the LLama model. The experiments show that this methods outperforms three baselines relatively consistently. They also show the impact of using different encoders as layer predictors, and generalization between training and testing on different benchmarks, as well as random exit layers and other baselines.

I have reviewed the previous submission of this paper. I very much appreciate the detailed response letter indicating changes related to the previous version.
Summary of strengths: 1. The proposed ALW framework is innovative, addressing the underexplored issue of noise accumulation across layers in LLMs. The idea of contrasting probability distributions across layers to mitigate noise is both novel and well-motivated.

2.	The paper demonstrates significant improvements across multiple benchmarks (e.g., GSM8K, BBH) and model sizes (e.g., LLaMA-7B, LLaMA-65B), showcasing the generalizability and effectiveness of ALW.

3.	ALW maintains inference efficiency while improving accuracy, making it a practical solution for real-world applications.

1. ALW dynamically mitigates ​cross-layer noise accumulation by contrasting shallow and deep layer distributions, significantly reducing reasoning errors
2. Experiments across multiple choice tasks and open- ended generation tasks shows the effectiveness of ALW.
3. ALW requires no additional training or heavy computation, preserving inference speed while improving performance

The paper suggests an interesting method for improving the decoding strategy of language models. 
The relatively consistent positive results on a wide range of datasets clearly show the potential and relevance of this method. 
Furthermore, the extensive experiments, ablations and focused evaluations provide helpful insight into the detailed workings of the method.
Summary of weaknesses: 1. Unclear Definition of Noise Based on Differential Transformer: The paper does not clearly explain how the definition of noise across layers is derived from the idea of avoiding common errors in differential transformers. The approach seems more aligned with methods like DoLa, which highlight tokens with significant logit changes between shallow and deep layers. A clearer connection to the differential transformer's error-avoidance mechanism would strengthen the theoretical foundation of ALW.

2. Early Exit Layer Selection: The paper selects the early exit layer from the first half of the model layers. The rationale for limiting the early exit layer to the first half but not entire layers is not well-justified.

1. When the shallow layers are highly confident in the prediction, ALW is probably hard to distinguish the difference between layers, which renders the improvement of ALW
2. ALW could not resolve the problems of incorrect reasoning paths.
3. The mechanism behind noise accumulation and suppression is primarily empirically validated but lacks ​rigorous theoretical analysis (e.g., formal definitions of "noise" in transformer layers).
4. The connections between the proposed Layer-Wise method and skip connection structure of Transformer model need more investigation.

I appreciate that the readability of the paper has really improved since the last submission, such as the rewriting of Sec. 2 in terms of noise and the general motivation. An issue that remains is that the masked language model for layer prediction, and the generative model receive a wide variety of names ("LM", "lightweight model", "LLM", "predictor", "large LM and a smaller LM", ...) throughout the paper. Here it would really help to introduce designated notation for these two components. 

In l58-61 and Fig.1: What does Fig. 1 display? From the description, I understand that the values for "Yes" or "No" in Fig.1c are (the values for "Yes" and "No" from Fig.1b) - (values for "Yes" and "No" from Fig. 1a). But this does not add up when considering the actual numbers, e.g. Yes has roughly .5 in Fig1b and roughly .00011 in Fig.1a, so the value for Yes in Fig.1c should be around .5-.00011=0,49989, which is far from the displayed value. Is this computed wrt. the cutoff described in Eq.7? 

Personally, I find l48 of the abstract weird. Llama 7B increases from 13.6 to 20.1. The score is increased by 48% and this sounds like a lot. But the overall result is really low - almost all other settings score higher than the 20.1 on Gsm8k in Table 1. Other settings show much bigger decrease in error rate as well as similar or bigger absolute accuracy gains.
Comments, suggestions and typos: 1. Provide a more detailed explanation of how the definition of noise is inspired by the differential transformer's approach to avoiding common errors. If the method is more similar to DoLa, explicitly discuss the similarities and differences, and justify why ALW's approach is more effective.

2. Justify the choice of selecting the early exit layer from the first half of the model layers. Why not identify critical layers across the entire model. The identification time cost for entire layers is also acceptable. 

3. To ensure a fair comparison, it would be helpful to clarify whether contrastive decoding is also applied to baselines like DoLa. Additionally, including ablation studies to isolate the contribution of contrastive decoding to ALW's performance would provide a clearer understanding of its impact.

4. A theoretical analysis explaining why noise accumulates in certain layers and how ALW's contrastive decoding mitigates this issue would strengthen the paper's foundation. 

5. An analysis of how the optimal early exit layer varies across different datasets and models would be valuable. For instance, do certain tasks or model sizes consistently favor earlier or later exit layers? This would provide practical guidance for applying ALW in diverse scenarios.

6. It would be interesting to include an analysis of why easier tasks tend to present fewer noises in shallow layers. Understanding this relationship could offer deeper insights into the behavior of noise accumulation and its impact on reasoning tasks of varying complexity.

1. Table 2 needs more explanations, such as why 8B achieves the best results. 
2. Results on PiQA dataset are not good and why is that?
3. The selection of optimal contrastive layer $l$ and early exit layer $e$ needs more demonstrations and case study. This would be valuable for analyzing how wrong answers are influenced by different layer selection.
line 196, the last $a$ should be consistent with $A$ in line 194

- Algorithm 1, line 9 and l266,926,930: the "V"s are not the same notation
- l367 "Foilo"
Overall assessment scores: 2.5

3

4
Metareview score: 3.5
Metareview: The paper hypothesizes that models accumulate noise across layers which leads to unstable predictions during reasoning. It presents a strategy to mitigate unstable or inconsistent behaviour of model's reasoning capabilities by disentangling noise in shallow layers. The experiments are performed on a number of reasoning datasets with promising results.
Summary of reasons to publish: The work is relevant to the ACL community and presents an interesting idea.