Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models

Authors: Nivedha Sivakumar, Natalie Mackraz, Samira Khorshidi, Krishna Patel, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: fairness, bias transfer hypothesis, prompt adaptation, large language models, coreference resolution

Abstract:
Large language models (LLMs) are increasingly being adapted to new tasks and deployed in real-world decision systems. Several previous works have investigated the bias transfer hypothesis (BTH) and find that fairness of pre-trained masked language models has limited effect on the fairness of these models when adapted using fine-tuning. In this work, we expand the study of BTH to causal models under prompt adaptations, as prompting is an accessible, and compute-efficient way to deploy models in real-world systems. In contrast to previous work,  we establish that intrinsic biases in pre-trained Mistral, Falcon and Llama models are strongly correlated (rho >= 0.94) with biases when the same models are zero- and few-shot prompted, using a pronoun co-reference resolution task. Further, we find that biases remain strongly correlated even when LLMs are specifically pre-prompted to exhibit fair or biased behavior (rho >= 0.92), and also when varying few shot composition parameters such as sample size, stereotypical content, occupational distribution and representational balance (rho >= 0.90). Our findings highlight the importance of ensuring fairness in pre-trained LLMs, especially when they are later used to perform downstream tasks via prompt adaptation.

Decision: reject
Reviewer scores: 6, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730670474022

thread_type:
official_review

<thread_object>
creator: reviewer_xdwp

<content>
summary:
The paper investigates how gender biases in pre-trained large language models (LLMs) transfer to their adapted forms, particularly when using prompting methods like zero- and few-shot prompting instead of full fine-tuning. The study assesses the bias transfer hypothesis (BTH), which suggests that social biases embedded in pre-trained models persist and influence behavior in adapted tasks. The study underscores the necessity of addressing biases during the pre-training of language models to minimize downstream impacts in real-world applications.

strengths:
1. This paper presents a substantive and meaningful contribution in examining bias transfer in prompt-adapted language models, filling a notable gap in current literature by focusing on causal models and prompting rather than only on fine-tuned or masked language models. 

2. The quality of the paper’s methodology and experiments is strong.

weaknesses:
1. While the study employs zero-shot, few-shot, and pre-prompting methods to analyze bias transfer, it does not explore other potentially impactful prompting techniques, such as chain-of-thought prompting, which has shown to affect LLM responses in reasoning tasks. Additionally, the prompts used are static, but dynamic or adaptive prompting—where prompts evolve based on intermediate model responses—could offer additional insight into the biases that emerge as models adapt to changing inputs. Including a broader variety of prompting techniques would increase the study's comprehensiveness and potentially reveal nuanced aspects of bias transfer.

2. A key weakness lies in the exclusive focus on gender bias within a binary framework. Although gender bias is a pressing issue, expanding bias analysis beyond binary gender categories would add substantial depth to the study. Including gender-neutral or non-binary pronouns, as well as biases related to other demographic dimensions (e.g., race, age, socioeconomic status), would reflect a more comprehensive understanding of model fairness.

3. The paper highlights how fairness-inducing and bias-inducing pre-prompts impact bias levels but does not deeply examine why certain prompts (e.g., negative or bias-inducing prompts) are more effective at altering bias than positive prompts. This leaves an important gap in understanding how pre-prompts function on a mechanistic level. A qualitative analysis of model outputs in response to different types of prompts, or a breakdown of how pre-prompts influence intermediate layers, could help clarify why some prompts succeed in reducing bias more than others.

questions:
1. Have you considered testing for additional types of biases beyond binary gender (e.g., racial or intersectional biases)? If so, what challenges did you encounter that led you to focus solely on gender?

2. Why did you choose zero-shot, few-shot, and pre-prompting as the primary adaptation methods? Did you consider exploring other prompt strategies like chain-of-thought or dynamic prompting, which might have unique effects on bias transfer?

3. What specific barriers do you anticipate in implementing fairer pre-training, and how do you suggest practitioners address them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696088175

thread_type:
official_review

<thread_object>
creator: reviewer_1aiy

<content>
summary:
The paper studies the gender bias of autoregressive LLMs across several experimental settings while performing a coreference resolution task, focusing on *bias transfer*, which is the correlation between models' "intrinsic" bias (next-token prediction) versus their bias in zero- or few-shot prompting. The results indicate that standard prompt engineering techniques, such as modifying task instruction prompts or including few-shot task examples, have only a small effect on bias transfer.

strengths:
The paper is generally well-written, with detailed and well-documented experiments and analysis that all appear to be sound. Comprehensive experiments are performed across a range of LLM prompting settings and three key open-source models.

weaknesses:
The primary weaknesses of the paper are (1) its failure to acknowledge closely related prior work, and (2) its lack of novelty when the missing prior work is taken into account. There is a large family of work studying the same questions as in this work [1-5] -- i.e., zero- and few-shot evaluation of causal (autoregressive) LLM bias; see [5] for a helpful survey -- but none of it is cited. Instead, the paper is written as if the only prior studies of LLM bias had been conducted in the context of masked language models or fine-tuned autoregressive models, artificially inflating the paper's contribution.

[1] Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., Chen, A., ... & Kaplan, J. (2023). The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459.

[2] Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2024). Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402.04105.

[3] Lin, L., Wang, L., Guo, J., & Wong, K. F. (2024). Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception. arXiv preprint arXiv:2403.14896.

[4] Huang, D., Bu, Q., Zhang, J., Xie, X., Chen, J., & Cui, H. (2023). Bias assessment and mitigation in llm-based code generation. arXiv preprint arXiv:2309.14345.

[5] Ranjan, R., Gupta, S., & Singh, S. N. (2024). A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions. arXiv preprint arXiv:2409.16430.

One interesting fact is that one of the primary arguments made in this paper -- that prompt engineering techniques such as optimizing task instruction prompts or including few-shot examples are not broadly effective tools for mitigating LLM bias -- seems to contradict the findings of several other works that have studied the same question [3, 4] (neither of which are cited in the paper). This disconnect is the result of the paper's overwhelming focus on *bias transfer* (the Pearson correlation coefficient between bias in next-token prediction and bias in zero- or few-shot prompting) rather than absolute bias numbers: i.e., if bias is significantly reduced using prompt engineering but remains directionally the same as in the "intrinsic" setting, then bias transfer remains high despite the reduction of bias
- E.g., in Table 2, for unambiguous (Type 2) task instances, the LLM bias (A-SB) goes from 27.7% in the "intrinsic" setting to 9.47% using a zero-shot prompt under the "fair" instruction prompt, and down to 4.16% using the same task instruction prompt in a few-shot setting; but the bias transfer (Pearson correlation) is $\rho \geq 0.92$.

Instead of presenting the paper as if it is the first work to consider research questions that have already been extensively studied, it would be interesting to examine whether the bias transfer metric remains useful in understanding LLM bias by comparing and contrasting with the findings presented by prior work. Such a paper might retain some novelty in what has become a crowded research area -- though there does appear to be at least one other paper considering this question as well [6], it only considers bias for a summarization task in the context of fine-tuning approaches, meaning that the specific experimental findings of this work would remain novel. 

[6] Ladhak, F., Durmus, E., Suzgun, M., Zhang, T., Jurafsky, D., McKeown, K., & Hashimoto, T. B. (2023, May). When do pre-training biases propagate to downstream tasks? a case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3206-3219).

Finally, there are a few confusing or misleading statements that should be revised:
- "MLMs fare better [than causal/autoregressive LLMs] at text classification and sentence analysis" (line 49-50) -- what is the evidence to support this statement?
- "causal models are much larger than MLMs" (line 50-51) -- while this is true of the leading available open-source models of each architecture, this is simply because causal models have become much more popular, so they have recently been scaled much larger; but scale is not an inherent feature of either architecture.
    - The sentence continues "[...] and therefore possess a greater capacity to perpetuate societal biases (Bender et al., 2021)." However, Bender et al. (2021) does not argue that LLMs' capacity to perpetuate societal bias is correlated with model scale (or the causal/autoregressive architecture).

questions:
Argumentation:
- In line 42-45, it is claimed that "the conclusion that bias does not transfer [...] has potentially dire implications for the fairness of task-specific models beyond MLMs, as it implies that the fairness of the pre-trained model does not matter." -— Why is this considered "dire for fairness"? If bias does not transfer, this would mean that it is easier to mitigate the inherent bias of LLMs as compared to a scenario where bias does transfer.

Experimental design:
- What samples is the Pearson correlation in Figure 3 and sections 4.1 and 4.2 computed over? Is it the O-SB values for each sample, each occupation, or each occupation-ambiguity combination?
- What are each of the points in the scatterplot visualized in Figure 3? Is it O-SB for each occupation individually, with each point as a different occupation (and all using the same prompt)?
- How many few-shot examples are included for the results in Table 2?

(UPDATE: score increased from 3 to 5 following rebuttal.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745423954

thread_type:
official_review

<thread_object>
creator: reviewer_wic1

<content>
summary:
This paper shows how biases existing in pre-trained language models transfer to downstream tasks through prompt adaptation. It shows that there is a strong correlation between biases in pre-trained models and models with zero- and few-shot prompts. Even when models are specifically pre-prompted to promote fairness or bias, such a correlation is still very significant.

strengths:
1. This paper explores bias transfer under various prompt settings, including zero-shot, few-shot, and pre-prompted configurations. It offers a thorough analysis of bias transfer for different prompting methods.

2. The strong correlation between intrinsic biases in pre-trained models and biases in prompt-adapted tasks emphasizes the need for fairness-aware pretraining or fairness-aware tuning.

weaknesses:
1. The study primarily uses the WinoBias dataset for gender bias evaluation, and it ignores the biases related to other demographic factors, such as race.

2. The observations are interesting, but the explanations for the observations are not sufficient.

3. The novelty is quite limited. It neither does not provide deep explanations for the observations nor any methods to mitigate the bias.

questions:
1. The observations are interesting, but they are kind of superficial for me. Could the authors provide any analysis to explain why the correlations exist? For example, some analysis is based on representations or logit lens.

2. If in-context learning (prompting) cannot be the promising solution for bias transfer. What is the next step? Could the authors talk more about the future work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730926287956

thread_type:
official_review

<thread_object>
creator: reviewer_clvc

<content>
summary:
### Summary

- The paper studies the "Bias Transfer Hypothesis" for modern LLMs.
- The bias transfer hypothesis states that biases during pre-training (intrinsic biases) translate to downstream task specific harms (extrinsic biases) and harms.
- Several previous works have studied this in the context of MLMs but not in the context of modern LLMs.
- The previous works did not find strong evidence in support of the Bias Transfer Hypothesis i.e. a strong causal effect of intrinsic biases on extrinsic biases
- In this work, the authors show that there is indeed a positive correlation between intrinsic and extrinsic biases for modern autoregressive LLMs.
- Furthermore, they experiment with various fairness and bias inducing prompts and tinker with the occupation and bias profile of in-context-examples and find that these have relatively small to no effect on changing the bias direction of the model.
- They use the following metrics in their experiments
	-- Referrent Prediction Accuracy
	-- Occupational Selection Bias
	-- Aggregate Selection Bias
- These metrics are consistent with standard metrics in the literature and are easy to follow. The visualization and plots in the paper are immensely informative and I commend the authors for the simplicity in presentation of their results
- In summary all the different experiments, indicate that Bias Transfer Hypothesis is indeed true for modern LLMs. However, I am a bit unsure about the setting in which this has been evaluated. (See my comments Weaknesses below)

Please consider citing the following additional work in the field of LM fairness and evaluation of bias in LMs
- https://arxiv.org/pdf/2208.01448#page=14.61
- https://aclanthology.org/2022.findings-acl.55.pdf
- https://aclanthology.org/2022.acl-long.401.pdf
- https://ojs.aaai.org/index.php/AAAI/article/view/26279
- https://link.springer.com/chapter/10.1007/978-3-030-62077-6_14

strengths:
- Well written , easy to follow, results mostly anticipated.
- Great plots and visualizations and good ablations.

weaknesses:
- I would like to suggest a point of discussion regarding the paper's conceptual framework. The characterization of prompting as an adaptation technique, while innovative, may benefit from further elaboration. Previous studies primarily examined bias transfer in the context of fine-tuning, which involves direct parameter modification. Given that prompting operates through input manipulation rather than model parameter changes, it represents a distinctly different approach to model behavior modification.
- Second, the results would benefit from additional empirical analysis. While the current findings are valuable, incorporating a broader range of intrinsic and extrinsic bias metrics from the literature could provide more comprehensive insights. This expansion could help establish stronger connections between this work and existing research in the field, and potentially reveal additional patterns in bias transfer mechanisms.

questions:
See Weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733170468832

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request to consider our supplementary results and clarifications in final paper assessment

comment:
Thank you for your thoughtful feedback and suggestions during the rebuttal period. We appreciate your engagement and have worked to address your comments.

Over the past two weeks, we have conducted **supplementary experiments to further support the presence of bias transfer under prompting** as already demonstrated in our paper. Our new findings include:

* We expanded our analysis beyond binary gender categories to the BBQ-lite dataset, revealing statistically significant bias transfer effects for age, nationality, physical appearance, and socio-economic status.
* We expanded our analysis beyond zero- and few-shot prompting to Chain-of-Thought prompting and again found statistically significant bias transfer.
* **Our investigation into attention activations revealed significantly different behavior of certain attention heads towards different demographics. Notably, attention activations were remarkably similar in intrinsic and prompted states, suggesting the attention mechanism plays a role in model biases.**

We have also updated our PDF to provide clearer motivation for our research and explicitly outline our additions (highlighted in red and blue).

As you finalize your evaluation of our paper, we respectfully request that you consider the additional experiments and clarifications we have provided in response to your feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates the bias transfer hypothesis (BTH) in large language models (LLMs), specifically focusing on causal models under prompt adaptation. The authors aim to determine whether intrinsic biases in pre-trained models persist and influence downstream tasks when adapted using zero-shot and few-shot prompting. Through various experiments, the paper shows that intrinsic biases in popular models like Mistral, Falcon, and Llama are strongly correlated with biases observed during prompt adaptation, even when models are pre-prompted to exhibit fair or biased behavior. 


Strengths:
* Generally well-written paper, easy to follow
* **Timely and relevant**. The findings are highly relevant for practitioners who use LLMs in real-world applications.
* Thorough and detailed experiments with interesting findings, convincingly presented in good visualizations
* Various informative ablations
* The introduction of new metrics (RPA, O-SB, and A-SB) to measure both intrinsic and extrinsic biases consistently is a notable contribution, addressing inconsistencies in previous studies.

Weaknesses:
* **Limited Evaluation Scope**: The paper primarily focuses on binary gender bias using the WinoBias dataset, which limits the generalizability of the findings to other demographic biases (e.g., race, age, socioeconomic status). Additional experiments were provided in-comments during the rebuttal phase, but not in the final revision (see next point).
* **Some rebuttal clarifications and additional results excluded from final revision**. Multiple reviewers pointed the narrowness of the WinoBias-only evaluation, which prompted the authors to run additional experiments on an additional dataset (BBQ-lite). However, these results do not seem to have been included in the revised version for some reason. For this reason, and the fact that crucial additional results were provided *only* (late) in the rebuttal phase after requests from reviewers and therefore have not gone through the same level of scrutiny, I am downweighing these additional results (but still taking them into account).
* **Very narrow research question**. An informative back-and-forth between the authors and reviewer 1AiY revealed that the novelty of this paper lies on assuming a very narrowly-defined research question: "how prompting affects the transformation of intrinsic biases into downstream biases (for autoregressive models, under a specific bias metric)". On these grounds, the authors decide to limit their comparison and had originally excluded very related work. While most of that work has been incorporated into the Related Work section, the specificity of the research question is weakness of this paper compared to the ICLR standard.
* **Limited exploration of causes and consequences**: the paper does not investigate in detail why certain prompts are more effective at altering bias than others, leaving an important gap in understanding the mechanism by which prompts mitigate or perpetuate biases.
* **Potential Confounders**: As pointed out by reviewer 1AiY, the pretrained open-source models used in this study have all likely been fine-tuned to mitigate bias, which likely introduces confounders that should be controlled for when studying prompt-based debiasing. This is an important question that is not sufficiently discussed/addressed in this work.

Decisions to accept/reject:
Given that this paper focuses on such a narrow research question, the bar for execution is much higher to rise to the level of ICLR. In particular, one would expect: 
1. a more-thorough-than-usual discussion of closely related work, to meticulously delineate the contribution
2. going beyond demonstration of the phenomenon, additionally studying the *causes* and *mechanisms* for the phenomenon
3. discussing and controlling for confounders of the (single) phenomenon studied

Thus, although I quite like this paper and enjoy reading it, I must side with the reviewers who do not find it to be above the bar for ICLR.