Large Language Models Can Be More Robust Multiple Choice Selectors Through Attention Intervention

Authors: Lihui Gu, Wenxiao Wang, Chen Shen, Liang Xie, Penghui Shang, Binbin Lin, Xiaofei He, Jieping Ye
Primary area: interpretability and explainable AI
Keywords: Large Language Models; Reasoning; In-context learning

Abstract:
Multiple-choice question (MCQ) is a common task for evaluating large language models (LLMs). LLMs' performance on MCQ is often affected by various biases. Previous research has extensively examined the impact of inherent option bias on MCQ predictions, where this bias refers to a preference for a specific option ID token introduced during the model's training. However, in an in-context learning scenario, few-shot prompting can also introduce a form of bias, known as context option bias. This occurs, for instance, in extreme cases where all demonstration answers are consistently option A, in which case LLMs may predict A for the given question whatever the question is. Context option bias can significantly degrade LLMs' performance. To better observe the LLMs' behavior when affected by the context option bias, we deliberately use demonstrations with obvious context option bias for MCQ to amplify the effect. The results indicate that certain attention heads in LLMs are particularly sensitive to context option bias. Motivated by this observation, we propose our approach, CoLo, to address this issue. First, using samples with ordinary and biased demonstrations as input, CoLo compares the outputs of two types of inputs and localizes attention heads sensitive to context option bias through sequential interventions. Then, we propose an attention scaling-based method to intervene in the identified attention heads during the inference stage, thereby mitigating the impact of context option bias on the LLMs’ predictions. Experimental results demonstrate that CoLo effectively alleviates the impact of context option bias and improves the LLM's robustness on MCQ tasks.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730532939851

thread_type:
official_review

<thread_object>
creator: reviewer_hfgx

<content>
summary:
This paper presents an approach, termed CoLo, to address the bias in multiple-choice question (MCQ) selection that arises from context option bias in large language models (LLMs). CoLo mitigates context option bias by intervening on specific attention heads that are highly sensitive to biased in-context demonstrations. By selectively scaling attention in these heads, CoLo reduces the model's predisposition to biased options, showing improvements in MCQ accuracy across several datasets. The authors also conduct extensive experiments to validate the cross-domain robustness of their method, indicating that CoLo's effect generalizes well across various MCQ scenarios.

strengths:
* I appreciate the focus on context option bias, a previously-overlooked form of bias that can impact LLM performance in MCQ tasks. The design of CoLo is also well-motivated.
* The paper provides extensive empirical evidence that shows improvements in MCQ accuracy and recall standard deviation. The authors also consider both zero-shot and few-shot scenarios and the cross-dataset setting, enhancing the generalizability of their findings.

weaknesses:
I would like to see evaluation with more larger models, as the current experiments are conducted with at most 7B/8B-level models. So I am not fully convinced about CoLo's scalability. For example, how CoLo would perform with 30B/70B-level, or MoE models (these experiments may be conducted with Qwen2/Qwen2.5 models, as LLaMA-3 does not have too many variants)? Also, what the specific GPU hardware and hours will be needed for various-sized models? The lack of experiments and analysis on larger models may limit the broader impact and implications of the proposed method.

questions:
1. Identifying the attention heads most associated with context option bias requires multiple rounds of sampling, which could add computational complexity in real-world applications (and higher than the PriDe baseline). While this overhead is justified within the paper’s scope, discussing potential optimizations or alternative approaches might help broaden CoLo’s applicability, especially for larger-scale implementations.
2. While the concept of CoLo is sound, the process for selecting and intervening on attention heads may seem somewhat convoluted to readers not already familiar with the technical nuances of attention heads and scaling interventions. Additional clarifications or illustrative examples could enhance the accessibility of the proposed approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693314875

thread_type:
official_review

<thread_object>
creator: reviewer_g8pb

<content>
summary:
Paper proposes a new approach to identify and intervene with attention heads in order to minimize biases from in-context examples

strengths:
_ Proposed approach is novel and the problem of biased in-context examples in important
- Paper is clearly written and easy to understand
- Experiments are thorough, including different models and tasks
- Results show that the approach leads to improved performance and is robust to the style of the input and the dataset

weaknesses:
- Results mostly show 1-3% improvement, which is nice, but not a huge improvement
- Experiments are focused on multiple choice. It would be great to show that the proposed method is able to generalize to more open ended question answering tasks, which are more representative of realistic tasks that LLMs are used for
- The proposed method requires both original and biased versions of in-context examples. However, in more open ended question, LLMs may still suffer from biased in-context examples, but it may not be clear what the bias is. Is there a way to not required both biased and unbiased versions of in-context examples in the method?

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730740424031

thread_type:
official_review

<thread_object>
creator: reviewer_cigt

<content>
summary:
The paper proposes  a method  aiming at mitigating context option bias in multiple-choice questions taks, by comparing the LLMs’ outputs for biased and ordinary prompts in order to localize attention heads associated with the context option bias. Attention scaling interventions is then employed. Experiments on several dataset are described in order to validate the method, although the standard deviations are not reported.

strengths:
-Paper addresses the weaknesses in large language models related with content bias in multiple-choice questions task (MCQ).

-The paper claims that the proposed method is demonstrated empirically to reduce the content bias in MCQ task on MMLU, although the standard deviations in the experiments are not  reported.

weaknesses:
1)The paper provides no theoretical explanation for why only specific attention heads contribute to bias or why scaling the specific attention heads mitigates the bias effectively. This lack of theoretical grounding may affect the reproducibility and understanding of the method, making it difficult to predict its behavior across other models.

2)The standard deviations are not reported in the experiments. Especially in the view of the small differences in several important experiments, this makes impossible to evaluate the method empirically.

2)Narrow focus on attention scaling as the intervention technique. The paper uses exclusively the attention scaling to address bias without exploring other possible intervention methods, such as layer-based adjustments or targeted fine-tuning. This narrow approach may miss alternative solutions that could yield more robust or efficient debiasing effects.

3)Recent large language models, such as  for example Llama 3.2, Mistral, Qwen 2 were not used in experiments, do they suffer from similar biases? Is the proposed method effective in these cases?

4)Reliance on artificially biased prompts. The study constructs heavily biased demonstrations to amplify context option bias and identify attention heads associated with this bias. This contrived setup may not accurately reflect the subtler biases encountered in real-world applications, potentially limiting the generalizability of the findings. It is not clear why reducing the bias in the artificially biased scenario will lead to increased performance in real-world applications. 

5)Sensitivity to hyperparameter choice. The method has hyperparameters, such as the scaling factor T for attention adjustments, but the paper  analysis of how sensitive the model's performance is to these choices is limited. Without detailed guidance for these hyperparameters, reproducing the results and applying the method to new datasets or models would require extensive tuning, which can be  resource-intensive.

6)Limited evaluation of cross-domain generalizability.  The evaluation of cross-dataset robustness is limited to certain datasets and tasks, where the reported small change is impossible to validate in the absence of  standard deviations reported. A more comprehensive assessment across a broader range of domains would be necessary to confirm the method’s generalizability, especially given the diverse biases in data sources.

7)The readability and clarity of the paper can be improved. The paper often employs long sentences which can obscure the main points. Sentences like 'To better observe the LLMs’ behavior when affected by the context option bias, we deliberately use demonstrations with obvious context option bias for MCQ to amplify the effect' are long and somewhat repetitive. Table 2 is discussed briefly without highlighting the most important findings or explaining how they support the claims. Redundant phrases and wordiness occasionally affect readability. For example, phrases like 'To better excavate the reasons behind this problem and resolve it...' could be made more concise. 'One of effective methods' requires 'the' article here; '...significantly degrade LLMs’ performance' as  'context option bias' is singular, so 'degrade' should be 'degrades.'  Vague pronoun references, in a few instances, pronouns (like 'it' or 'this') refer ambiguously to previous sentences, making the meaning less clear. 'The experiments find that... This inspires us to test...' — here, "This" could refer to several prior ideas, so specifying 'This result' or 'This observation' would improve clarity.

8)The captions under Figure 8 are somewhat misleading, how to read the frequency of the localized field from picture (c)?

questions:
Did you experiment with other recent llms, such as Llama 3.2, Mistral, Qwen 2, do they suffer from similar biases? Is the proposed method effective in these cases? 

Why reducing the bias in such artificially biased scenario will lead to better performance in real-world applications?

Did you try other options of reducing the bias, like targeted fine-tuning?

Why the standard deviations in the experiments were not reported?

How to tune the hyperparameters of the method such as T? the performance seems to depend heavily on its value (see table 1)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730875271149

thread_type:
official_review

<thread_object>
creator: reviewer_dxnu

<content>
summary:
This paper proposes a method to interpret which attention cells are responsible for “context option bias” — a form of bias that occurs when an LLM inclines to generating one option more than others in an in-context learning scenario. The authors propose a difference metric that measures the change in attention values when biased or ordinary examples are used with in-context learning. This metric is also used to detect top-layers that are the most important for context option bias. They also propose a solution to alleviate the impact of bias: scaling attention values of the detected cells using a custom temperature. Experimental results suggest that the proposed method improves MMLU performance and detected attention heads transfer to other domains such as CEVAL. The detected heads are robust to different number of choice or demonstrations but sensitive to the particular biased option.

strengths:
Update: Authors provided detailed explanations with many additional experiments which addressed my concerns. I increased my score accordingly.

1. The custom temperature per attention head is a simple but affective approach for detecting context option bias.

2. Experimental results indicate success with better detection and improved performance.

3. Transferability to other tasks indicate that these heads are feature of the model and pre-training rather than downstream domain. This is an interesting finding.

weaknesses:
There are a few concerns that I detailed below.

1. I think a better motivation on why context option bias for MCQ is a practical problem is needed. Given that options in in-context examples can be shuffled freely, this needs a better motivation.

2. What is the minimal number of examples for successful detection of bias? More specifically, how does the performance scale with more examples?

3. What if you add a custom prompt message that explains that “there are multiple options, the answer can be any one of them, and no particular option such as A or B is more likely.” Would this be an affective strategy to steer the attention heads?

4. How does the problem change with increasing model size? Are larger models more or less susceptible?

5. Equation 2 should be $y=argmax$.

6. How did you choose $K_l$ and $K_h$? How does the performance change as $K_l$ and $K_h$ increases?

7. Previous work has used different temperatures for different domains (see [1] for one such work) that are not necessarily 1.0. Can you run a search over different temperatures for the baseline and report the best performing model?

8. Can you explain why the performance of Llama-3 and Gemma models are very different from official numbers from respective papers?

9. Can you explain what do you mean by confidence in Section 4.5?

10. Caption of the Figure-5 is not grammatically sound. Please rewrite.

[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. Bradley Brown and Jordan Juravsky and Ryan Ehrlich and Ronald Clark and Quoc V. Le and Christopher Ré and Azalia Mirhoseini.

questions:
1. Why does context option bias for MCQ a practical problem?

2.  How does the performance scale with more examples?

3. What happens if you use a custom prompt message?

4. What is the impact of model size on the problem?

5. How does the performance change w.r.t. different $K_l$ and $K_h$? How did you choose these parameters?

6. Can you improve the baseline by finding the best temperature?

7. Why does the performance of Llama-3 and Gemma different from official numbers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733241558515

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal (Many Thanks to All Reviewers and AC)

comment:
Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank all four reviewers for their constructive comments and insightful auestions, which helped us refine our work.

*We would like to express our sincere gratitude to the reviewers for their appreciation of our work*.

**[Impactful and well motivated]**

- **Reviewer hFgX**: "The design of CoLo is also well-motivated."
- **Reviewer G8PB**: "The approach leads to improved performance and is robust to the style of the input and the dataset."
- **Reviewer DxNU**: "Transferability to other tasks indicate that these heads are feature of the model. This is an interesting finding."

**[Experiment Analysis]**

- **Reviewer hFgX**: "The paper provides extensive empirical evidence."
- **Reviewer G8PB**: "Experiments are thorough, including different models and tasks."
- **Reviewer DxNU**:  “Experimental results indicate success with better detection and improved performance.”

**[Writing]**

- **Reviewer DxNU**: "The custom temperature per attention head is a simple but affective approach."
- **Reviewer G8PB**: "Paper is clearly written and easy to understand."

During the response period, we carefully try our best to provide feedback and conduct supplementary experiments to all comments from reviewers, *We concisely summarize our responses here*:

**[Presentation Issue]**

- We revised a sentence in the abstract to make it more concise.
- We expanded the title of Table 2，and highlighted "using the attention heads obtained from MMLU" in bold in the Table 3's caption.
- A detailed explanation of this concept has been added in Section 4.5.
- We included the workflow diagram of CoLo (Figure 10) in Appendix F.
- We adjusted the order of (a), (b), and (c) in Figure 4.

**[Experiment Issue]**

- We have added experiments involving the CoLo algorithm on Qwen2.5.
- We have included experiments using prompts to mitigate option bias, further strengthening the motivation for intervention strategies.
- We have introduced experiments on global attention scaling interventions, which support the motivation for targeting specific heads.
- We have conducted experiments on sample size and performance, further determining the sample size required for CoLo.
- We have incorporated an ablation study on the impact of hyperparameters $K_l$ and $K_h$ on performance, providing recommended values.
- We have added experiments examining the improvement in the standard deviation of the model’s option debiasing performance with CoLo.
- We have added experiments that use prompts to increase option bias and construct sample pairs for CoLo.

**[More Explanations]**

> **The studied bias is somewhat narrow and extend CoLo to include a broader range of biases in open-domain settings.**

We fully agree with the suggestion to expand the scope to include a broader range of biases. This is why we are committed to employing inference-based interventions rather than relying on prior-based debiasing methods[1].

However, **option bias remains a crucial issue, and a significant body of research has focused on this topic** [2,3,4,5]. 
We discovered that **using a prompt-based method can also effectively generate sample pairs instead of swapping in the order of options**. We have conducted preliminary experiments in this direction. This simplified approach is highly conducive to extending the study to a wider variety of biases, particularly in open-domain settings. We are confident that our approach has significant potential for scaling to incorporate additional types of biases.

We hope our detailed responses can address the concerns from reviewers. And your valuable comments have helped us to refine our work.

Best regards and thanks,

Authors of #9076

**Reference**

[1] Zheng C, Zhou H, Meng F, et al. Large language models are not robust multiple choice selectors.

[2] Pezeshkpour P, Hruschka E. Large language models sensitivity to the order of options in multiple-choice questions

[3] Wang H, Zhao S, Qiang Z, et al. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models

[4] Balepur N, Ravichander A, Rudinger R. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?

[5] Xue et al. Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors, ACL 2024
</content>
</thread_object>

</thread>


Metareview:
metareview:
Reviewers of this submission positively commented on a simple and effective method, leading to consistent gains in multiple-choice selection problems for LLMs, both in terms of accuracy (modest improvements) and recall standard deviation. Reviewers also commented positively on the cross-domain generalization insight, suggesting an expanded version of that analysis.

Unfortunately, concern remains regarding the impact of this work due to a lack of theoretical explanation of the motivation of the proposed method, the lack of experiments on larger and more diverse models (despite the authors adding small Qwen 2.5 models during the rebuttal) and the rather narrow problem of option bias multiple choice question, which the authors admit "our method is currently effective solely in mitigating option bias". In addition, a reviewer criticised that not enough consideration was given to alternative ideas for option biasing, which the authors could have answered with another set of experiments. 

As a result, at this stage, I do recommend this submission for publication at ICLR.