Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models

Authors: Rishabh Adiga, Besmira Nushi, Varun Chandrasekaran
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: bias, attention, llms

Abstract:
We explore the internal mechanisms of how bias emerges in large language models (LLMs) when provided with ambiguous comparative prompts: inputs that compare or enforce choosing between two or more entities without providing clear context for preference. Most approaches for bias mitigation focus on either post-hoc analysis or data augmentation. However, these are transient solutions, without addressing the root cause: the model itself. Numerous prior works show the influence of the attention module towards steering generations. We believe that analyzing attention is also crucial for understanding bias, as it provides insight into how the LLM distributes its focus across different entities and how this contributes to biased decisions. To this end, we first introduce a metric to quantify the LLM's preference for one entity over another. We then propose $ATLAS$ (Attention-based Targeted Layer Analysis and Scaling), a technique to localize bias to specific layers of the LLM by analyzing attention scores and then reduce bias by scaling attention in these biased layers. To evaluate our method, we conduct experiments across 3 datasets (BBQ, Crows-Pairs, and WinoGender) using $GPT$-$2$ $XL$ (1.5B), $GPT$-$J$ (6B), $LLaMA$-$2$ (7B) and $LLaMA$-$3$ (8B). Our experiments demonstrate that bias is concentrated in the later layers, typically around the last third. We also show how $ATLAS$ effectively mitigates bias through targeted interventions without compromising downstream performance and an average increase of only 0.34\% in perplexity when the intervention is applied. We see an average improvement of 0.28 points in the bias score across all the datasets.

Decision: withdrawn
Reviewer scores: 5, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730350213520

thread_type:
official_review

<thread_object>
creator: reviewer_fnsk

<content>
summary:
The paper addresses the critical issue of bias mitigation within LLMs, particularly examining how biases can emerge internally when processing comparative prompts. The proposed method, Attention-based Targeted Layer Analysis and Scaling (ATLAS), is designed to localize and reduce bias by adjusting attention scores in specific layers. Focusing on model-internal mechanisms rather than data-centric methods, ATLAS offers a model-agnostic approach. The authors present experimental results demonstrating that ATLAS performs effectively according to the newly introduced "bias ratio" metric and maintains fluency as measured by perplexity.

strengths:
- This paper tackles a critical issue by examining bias at the model level and introduces a new method focusing on attention layers to localize bias, providing insights into where bias might occur within the model.
- ATLAS operates model-agnostic, making it broadly applicable across different LLMs.

weaknesses:
- The approach relies mainly on averaged attention weights, which may miss the complex interactions across tokens and layers, contributing to bias formation. This simplification could limit the method's ability to fully capture nuanced biases embedded in the model.
- The evaluation centers on a newly introduced "bias ratio" metric, which could favor the specific method over more established metrics. The absence of comparisons to state-of-the-art (SOTA) bias mitigation methods and limited ablation studies further restricts a comprehensive evaluation of ATLAS’s effectiveness.
- The binary selection setup constrains the method's applicability to broader, real-world scenarios where decision-making may involve multiple options or more nuanced judgments. This limitation could affect the generalizability of the results to other types of bias.
- Using perplexity to assess response quality may be less effective in this context, as the setting requires only words as an answer rather than full sentences. A different metric that better captures the quality and appropriateness of these shorter outputs could provide more relevant insights into ATLAS’s impact on fluency.

questions:
- In terms of top/bottom-k layers, does "k" refer to specific layers (e.g., exactly the k-th layer) or all layers within the range of 1 to k?
- If tokenization affects initial entity tokens (e.g., “grandfather” becomes "grand" + "father"), how is attention managed for variations in token splits like “grandfather” vs. “grandson”?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730447645033

thread_type:
official_review

<thread_object>
creator: reviewer_k5bz

<content>
summary:
The paper presents a novel approach for identifying and reducing bias in LLMs when faced with ambiguous comparative prompts. The authors argue that bias often emerges due to how the attention mechanism distributes focus among entities, particularly in the later layers of the model. By analyzing attention scores, they localize biased behavior to specific layers and apply targeted interventions to mitigate bias without significantly affecting model performance. Experiments conducted across multiple datasets demonstrate that ATLAS effectively reduces bias while maintaining fluency in the model's responses.

strengths:
1. This paper focuses on the internal bias in LLMs, which is a significant area of research.
2. This paper identifies bias by analyzing attention weights, which is reasonable.
3. Experiments demonstrate the effectiveness of the proposed methods.

weaknesses:
1. The paper proposes two approaches: using the difference and using the most probable candidate. While the authors select Approach 2, they do not provide a detailed analysis of why it is superior to Approach 1.
2. The paper focuses solely on comparative biases between two entities, which limits the research scope considerably. Focusing on comparative biases may overlook the complex interactions and potential biases that arise among multiple entities. For instance, many real-world scenarios involve multiple entities (such as social groups, genders, ethnicities, etc.), where the manifestations of bias can be more intricate and diverse. Simply comparing two entities may fail to capture this complexity.
3. There is a writing error: "Table 4" in line 485 should be corrected to "Table 3."

questions:
1. Could the authors provide a more detailed analysis of their choice of Approach 2? For instance, it would be helpful to compare the two approaches in terms of bias reduction, computational efficiency, and impact on model performance.
2. I would like the authors to elaborate on whether their method has the potential for extension to other bias scenarios, such as biases involving multiple entities. If so, how could this method be adapted to address these additional scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697375217

thread_type:
official_review

<thread_object>
creator: reviewer_sln8

<content>
summary:
This paper presents Atlas, a two-step approach to identifying and mitigating biases in LLMs when responding to ambiguous comparative prompts (i.e. asking the LLM to select one from two entities). The proposed approach first localizes the layers of transformers that contribute the most to biases, then reduces biases by scaling attention scores in these layers corresponding to tokens of entities. The paper shows that the proposed approach can effectively reduces biases while maintaining performance.

strengths:
1. The paper studies one important problem, the findings presented might be of interest to a large number of audiences.

2. Analyzing the internal mechanisms of how biases emerges in LLMs seem novel.

3. The proposed approach seems to be effective according to the experimental results.

weaknesses:
1. The motivation shall be stronger. Figure 1 only shows which layers obtain higher attention scores for entities instead of which entity will be biased. For example, some layers would assign more weights to "grandson" while others would assign more weights to "grandfather".

2. For one entities, the paper only considers the first token of the entity for attention score computation, which might be problematic when the first token of two entities are the same (e.g., grand for grandson and grandfather).

3. While the intervention approach of scaling attention is straightforward, the sum of all attention weights will become less than 1. 

4. The method needs to determine the scaling factor \lambda for each prompt, which makes it not quite usable in practice. Overall, it is hard to guarantee the LLM to generate unbiased output in general use cases. 

5. The experiments should include larger models such as Llama-70B.

6. The paper only considers two baselines, given that there are so many papers on mitigating llms' biases, more baselines results should be reported to better show the significance of the proposed approach.

questions:
1. The introduction of the attention mechanism in section 2 should be polished, there are some errors that should be carefully addressed. 

2. Since this work focuses on comparative prompts, one factor that shall not be ignored is the oder of the two entities, the paper should add results when the two entities are swapped. Moreover, I think it would be more useful to report the average results of the two different orders.

3. For approach 1 in line 269, would not the absolute value be adopted?

4. In table 3, you reported the change ratio of the entity with the higher prob. I believe it would be better to report the ratio of each entity that is preferred to gain better insights on to what extend the bias issue is resolved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716840609

thread_type:
official_review

<thread_object>
creator: reviewer_pvbe

<content>
summary:
This paper introduces a simple and straightforward method for localizing and mitigating bias in large language models. The approach focuses on the attention scores associated with specific target words, allowing us to identify the layers where intervention is most effective. By scaling down the attention on these target words in the identified layers, the method can effectively reduce bias. Experimental results demonstrate that the proposed method significantly improves bias scores compared to baseline approaches.

strengths:
This paper tackles the important challenge of mitigating bias and stereotypes in large language models. The proposed approach is simple, lightweight, and intuitive, yet it proves to be highly effective in the evaluated setting.

weaknesses:
The setting explored in this paper is overly simplistic and may not reflect real-world scenarios. It is uncertain how the proposed method would generalize to more practical settings. For instance: (1) candidates may not be explicitly annotated or may even be absent in the text; (2) biased or stereotypical outputs may be more implicit than those examined; and (3) the model’s outputs can depend on candidates in more nuanced ways so suppressing attention may hurts this dependency modeling.

questions:
1. Can the proposed method generalize to the settings where tokens corresponding to candidates are unknown or absent in the text? If not, is there any implication on the more general settings?
2. When the model's output requires the information of candidates in more nuanced ways, would the model still work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732554258520

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Concise consolidated information on major feedback and responses

comment:
**1) Scope and Generalizability of the Approach (detailed responses under reviewer pvbe, reviewer K5bz, reviewer fnSK)**
  
**Concern**: ATLAS focuses solely on comparative biases with two entities, which limits its applicability to real-world scenarios and may overlook the complex interactions and potential biases that arise among multiple entities.  

**Response:**  
While we do acknowledge that our comparative prompt framework—by focusing on explicitly mentioned candidates—may not encompass the full complexity of real-world scenarios where candidates are implicit or even absent, we emphasize that our work is intended as a first step. 

We stress that bias is not a simple phenomenon; it is deeply multi-faceted and context-dependent. As such, it is unlikely that there exists a single solution – a "one-stop shop"—capable of addressing all forms of bias comprehensively. By focusing on attention-based bias reduction in explicitly defined comparative settings, we provide an interpretable and effective tool for mitigating one particular dimension of bias while acknowledging the need for additional methods to address other facets of this complex issue. We believe this foundational approach lays the groundwork for future extensions to explore more nuanced and implicit settings. Notably, there is currently no work that investigates the role of attention for bias reduction in cases involving implicit or inferred entities.

- **A nuanced experiment analyzing more complex interactions has been conducted, demonstrating that ATLAS remains effective.**  

- **ATLAS can be modified to address scenarios involving multiple entities by balancing attention across all candidates iteratively.**  
---  
**2) Baselines (detailed responses under reviewer fnSK)**  

**Concern**: The current baselines are not competitive and better bias mitigation baseline can be used.  

**Response**:  
A new baseline, PASTA (Post-hoc Attention Steering Approach), was evaluated and included. Results show ATLAS **consistently outperforms** PASTA across bias categories due to its more refined localization and intervention mechanism even though PASTA already performs well in this framework.  

**Details**:
We consider PASTA (Post-hoc Attention STeering Approach) (Published at ICLR 2024) as an exemplar activation steering approach that is devoid of the aforementioned shortcomings. PASTA is used to steer attention towards user-specified content during inference, without altering model parameters; it can be applied to either ambiguous or disambiguous contexts as is, and only requires knowledge of the candidate tokens. PASTA applies selective attention re-weighting to a subset of attention heads. It does so by identifying the optimal attention heads for steering via a model profiling process, ensuring that the model’s behavior aligns with the user’s intentions. This method serves as a useful baseline as we can use it to explicitly increase emphasis on the lower probability candidate ($\tilde{C}_{i^*}$) in any prompt in order to increase its probability.  

**Results**:  We observe that while PASTA  results in improvements, ATLAS still achieves better performance. This is likely because of PASTA’s reliance on pre-determined attention heads which do not fully account for prompt-specific nuances in the attention distribution. In contrast, ATLAS’s targeted approach to bias localization across layers allows for more refined interventions, specifically addressing the layers most responsible for biased behavior for each prompt. On average, ATLAS performs 0.10 points better than PASTA across categories.

---  
**3) Sensitivity to Entity Order (reviewer SLn8)**

**Concern**:   
Results may vary depending on the order of entities in the comparative prompts.

**Response**:  
 Additional experiment was performed to show minimal sensitivity to entity order, confirming the robustness of ATLAS. Swapping the entities has negligible impact on the EBS scores, both for the default model and after ATLAS has been applied.

---  
**4) Choice of Scaling Approach (reviewer K5bz)**

**Concern**: Results when  Approach 2 is used over Approach 1 for localization are not present.

**Response**:   
Additional experiment performed to show that Approach 1 results in a larger increase in EBS than Approach 2. Our analysis shows that Approach 2’s focus on the most probable candidate allows for more targeted scaling, as it pinpoints the specific layers where the higher probability entity has the largest focus rather than looking at layers with large difference in attention scores between the entities.

---  
**5) Additional  larger model (reviewer SLn8)**

  We have additionally added results on bias mitigation on a larger model (LLaMA 2-13B). The results show that ATLAS’ influence on bias mitigation is agnostic of scale.
Due to GPU memory constraints and incompatibilities with pre-trained quantized models, testing on LLaMA-70B was not feasible during the rebuttal period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733270216744

thread_type:
official_comment

<thread_object>
creator: area_chair_ikcx

<content>
title:
End of reviewer-author discussion phase

comment:
Dear reviewers,

As we near the conclusion of the reviewer-author discussion phase, I wanted to kindly follow up to see if you’ve had a chance to review the author responses on your comments. Could you confirm that you’ve read it and, if needed, update your review and scores accordingly?

Thank you for your time and effort!

Your AC
</content>
</thread_object>

</thread>


Metareview: