Prompting Fairness: Integrating Causality to Debias Large Language Models

Authors: Jingling Li, Zeyu Tang, Xiaoyu Liu, Peter Spirtes, Kun Zhang, Liu Leqi, Yang Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Model, Prompting, Social Bias, Causality, Debias, Selection Mechanism

Abstract:
Large language models (LLMs), despite their remarkable capabilities, are susceptible to generating biased and discriminatory responses. As LLMs increasingly influence high-stakes decision-making (e.g., hiring and healthcare), mitigating these biases becomes critical. In this work, we propose a causality-guided debiasing framework to tackle social biases, aiming to reduce the objectionable dependence between LLMs' decisions and the social information in the input. Our framework introduces a novel perspective to identify how social information can affect an LLM's decision through different causal pathways. Leveraging these causal insights, we outline principled prompting strategies that regulate these pathways through selection mechanisms. This framework not only unifies existing prompting-based debiasing techniques, but also opens up new directions for reducing bias by encouraging the model to prioritize fact-based reasoning over reliance on biased social cues. We validate our framework through extensive experiments on real-world datasets across multiple domains, demonstrating its effectiveness in debiasing LLM decisions, even with only black-box access to the model.

Decision: accept (poster)
Reviewer scores: 6, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730174757792

thread_type:
official_review

<thread_object>
creator: reviewer_lwq1

<content>
summary:
This article conducted a causality analysis for bias in LLM's decision and provided a prompting-based solution for bias mitigation. The solution included three strategy pathways and demonstrated that the combining strategy can achieve comprehensive debiasing.

strengths:
The given causality-analysis clarified the origins of bias in well-trained LLMs, making the proposed bias mitigation strategies more explainable. 

In addition to examining the influence of training data corpus and prompts, the authors also considered bias caused by data selection, offering a more systematic and comprehensive bias-mitigation solution.

weaknesses:
1. The assumption in A.3 CAUSALITY AND LLMS, 
"We adopt a rather mild assumption that a well-trained and well-aligned LLM captures the dependence pattern in the training data and that such a pattern is internalized and utilized during reasoning" 
needs to be put in the main text. If LLMs are not well-trained and well-aligned, then the author's prompting-based de-dependency method will no longer be so effective, so I think this is an assumption worth stating in the main text.

2. There is a lack of discussion related to other prompting-based debiasing methods in LLMs, such as those in [1][2][3].

[1] Zhang, C., Zhang, L., Zhou, D., & Xu, G. (2024). Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment. arXiv preprint arXiv:2403.02738.

[2] Li, J., Tang, Z., Liu, X., Spirtes, P., Zhang, K., Leqi, L., & Liu, Y. (2024). Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework. arXiv preprint arXiv:2403.08743.

[3] Furniturewala, S., Jandial, S., Java, A., Banerjee, P., Shahid, S., Bhatia, S., & Jaidka, K. (2024). Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models. arXiv preprint arXiv:2405.10431.

questions:
1. When the three strategies are combined, both social-salient text representation and social-agnostic fact
representation are independent of social category
representation. Does this mean that it is equivalent to using only prompts Q:{Base Question}, a prompt that is completely unrelated to sensitive features, to let LLM complete the answer to Original Question? If not, when both social-salient text representation and social-agnostic fact
representation are independent of social category
representation, what is the performance difference of LLM decision brought by prompts Q:{Base Question} and Q:{Given that your answer to {Base Question} is {Answer to Base Question}, Assume male and female are equally represented in drivers and in teachers. Do not answer the question using gender information. {Original Question}}? Can the author give experimental results on this point?
2. Should there be a solid line in Figure 3 connecting prompt to LLM potential decision?
3. In addition to the fact that prompting-based techniques are suitable for dealing with black-box scenarios, can the authors add a discussion on the advantages and limitations of prompting-based techniques compared to direct fine-tuning of model parameters and prompting-based techniques, for example, a limitation of  prompting-based technique ——the need for human users to be proactive and knowledgeable to complete debiasing?
4. Can the authors provide a comparison of this work with other prompting-based debiasing methods mentioned in weakness, either experimentally or analytically?
5. In the Table 2, why the sum of the true-rate (42.33%, 32.02%) and false-rate (8.68%, 8.82%) of the base question under GPT-3.5/Counteract Only/Anti case is not 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730344964615

thread_type:
official_review

<thread_object>
creator: reviewer_qvwj

<content>
summary:
This paper proposes a causality-guided framework for debiasing large language models through prompting strategies. The authors introduce a novel perspective to identify how social information influences LLM decisions through different causal pathways and develop principled prompting strategies to regulate these pathways through selection mechanisms. The framework encompasses two main approaches: encouraging fact-based reasoning and discouraging biased reasoning. The authors validate their framework through extensive experiments on multiple benchmark datasets (WinoBias, BBQ, and Discrim-Eval) across various social dimensions, demonstrating its effectiveness in debiasing LLM decisions while maintaining strong performance.

strengths:
This work presents a novel theoretical framework that bridges causality and LLM debiasing, providing clear intuitions and principled strategies for addressing bias. The causal modeling of both training data generation and LLM reasoning processes offers valuable insights into bias sources and mitigation approaches.

weaknesses:
Although the authors demonstrate improved performance across multiple models, there's limited discussion of how the effectiveness of their approach might vary with model scale or architecture. Additionally, while the authors show reduced bias metrics, there could be more analysis of potential trade-offs between bias reduction and task performance.

questions:
How does the effectiveness of the proposed debiasing strategies vary with model size and architecture?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634882493

thread_type:
official_review

<thread_object>
creator: reviewer_qljv

<content>
summary:
This paper addresses responsive bias in both open-source and proprietary LLMs by formulating a theoretical debiasing framework that analyzes the impact of social information on an LLM's decisions from a novel causal perspective. The framework identifies causal pathways through which social information influences model outputs and develops the integrated inference-time prompting strategies to accordingly regulate information flow across different pathways, thereby suppressing potential social bias. Extensive experiments on real-world datasets across multiple domains validate the framework, demonstrating its effectiveness in debiasing LLM decisions.

strengths:
1. This paper provides a detailed analysis of the internal mechanisms behind biased decision-making in LLMs from a causal perspective, offering a theoretical understanding for the proposed prompting-based debiasing methods.
2. The Introduction section is well-written and effectively conveys the existing challenges.
3. Extensive experiments demonstrate that the proposed DDP significantly outperforms other baseline methods.

weaknesses:
1. The proposed DDP suffers from poor applicability for general-purpose generation: DDP is a prompting-based technique that relies on adding extra textual guidance into  the prompt context to achieve debiasing. For evaluation, DDP needs dataset-specific design to obtain its corresponding textual guidance, thereby adapting to different evaluation datasets. So DDP can not be applied to free-form generation, especially when social attributes of interest are not directly given in input prompt but emerge as the intermediate generated results of LLMs, limiting its further applicability in more critical scenarios.
2. The paper organization is unclear and redundant: the pre-trained data generating process is not necessary for understanding the main contributions of this paper. The theoretical insights (strategies) are coupled together with the technical details, making it hard to fully and clearly assess the technical contributions after reading Section 3.
3. Prompting-based debiasing approaches have limitations and may not be particularly meaningful. Traditionally, the responsibility for model alignment and debiasing lies with model deployers. Implementing prompting-based debiasing can be unsafe and inevitably introduces additional time costs.
4. The textual guidance introduced by DDP may introduce noise that hinders the reasoning abilities of LLMs. As shown in Table 1-Type II, while DDP reduces the bias gap, it significantly degrades performance in terms of Anti and Pro (on GPT-3/3.5 and Claude 2). Therefore, their robustness remains an unsolved problem.

questions:
1. Does the 'default' in Table 2 refer to the same meaning as the 'default' in Table 1, which solely receives the original question as input? If so, why does this method also include the metrics (TT, TF, FT, FF)?
2. See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730778770363

thread_type:
official_review

<thread_object>
creator: reviewer_qvwj

<content>
summary:
This paper introduces a prompt-based method to remove biases in a language model's output. It motivates the prompts using the idea of selection bias from causal inference literature. Experiments show a significant reduction in measured bias on two datasets.

strengths:
* Theoretical justification of prompting strategies using a stylized selection bias argument
* Significant reduction of bias in demonstrated experiments.

weaknesses:
* The connection of selection bias, causal graph and the proposed prompt is weak. In one dataset, strategy I and II is used. In the other dataset, strategy I and III. So it is not clear that the set of selection biases listed are exhaustive. Rather, it appears that the list of such selection biases may increase as the datasets increase, and so the current selection biases presented are the union of those that were useful for prompts for these two data sets. 
* Ablations are not provided. What happens when all strategies are used for first dataset, and what happens when strategy I and II is used for second dataset? 
* Other methods of removing bias are not compared. Since the final solution is just a prompting change, there can be other ways of prompting the LLM that are simple to implement. For example, the text "avoid any gender bias while answering the question" can be added to the prompt.  
* Strategy I seems the most useful. However, I have two concerns. First, it may be computationally intensive. Can the authors clarify how many calls would it need to answer a single question? If I understand correctly, there will be one call to create the two possible scenarios for the base question, and then there will be one call to decide which of them is more plausible, and then a final call to actually answer the question? So there are going to be three calls per question? If that's the case, I can think of other methods, for example, the React multi-agent framework, where the LLM is asked to respond to the question first, then there is a critique agent that checks whether there is any bias in the answer, and if yes, it asks the LLM to regenerate the answer giving the feedback from the critique as a part of the prompt. Second concern is that the creation of the base question, while could be templated for simple examples like the one shown in the paper, but eventually will also become a task that an LLM will need to do so, are there ways to use a smaller model or something more efficient for this?

questions:
I have mixed opinions about this paper. On the one hand, I appreciate the selection bias analogy and the abstraction of the problem to a causal graph and the conclusions that come from it. On the other hand, the final solution proposed is just an ensemble of intuitive prompts, and it is not clear to me that these prompts are meaningfully guided by the selection bias theory. And it is not clear to me that other better prompts could not have been obtained without any selection bias theorization. So my questions to the authors are :
1. Can you show that avoiding bias based on selection bias is better than simply asking the LLM to avoid bias? See one suggestion of a modified prompt in the weaknesses above. I guess the literature on debiasing LLMs may have more simple prompt additions or system prompts that can be added.  In your comparison, also consider the computational cost of your proposed method.
2. How does the selection bias theory help in choosing the correct strategy, and later, the prompt? What was the justification for using strategy II in one case and strategy III in another? 
3. How would Strategy I generalize to more complex bias scenarios? Do you need an LLM for creating the base prompt in that case? Many cases of biases may not be simple template pronoun substitutions.
4. Strategy I seems closest to a variant of in-context-learning where the closest example is chosen so to help the LLM answer the question correctly. Can you compare to a baseline that selects the K nearest in context examples to add to the prompt, rather than a fixed set of in context examples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732990930509

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
To all reviewers,

We sincerely appreciate your time and effort in reviewing our work! We have carefully addressed all your comments and provided detailed responses in the rebuttal phase. Additionally, we have updated our manuscript to reflect these improvements. You can access the updated manuscript through this [anonymous link](https://file.io/x4qcO5DjnpGe).

We look forward to any further insights you might have, and we would love to continue further discussions.

Best regards,

Authors of Submission 12960
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel framework that leverages causal analysis to mitigate biases in large language models (LLMs) through prompting strategies. The paper’s strengths include its theoretical foundation, extensive experimental results demonstrating bias reduction, and a comprehensive bias-mitigation solution.

However, the paper has some weaknesses, such as a weak connection between selection bias, causal graphs, and proposed prompts, limited discussion on the effectiveness of the approach with different model scales and architectures, and concerns about computational intensity and general applicability. The authors addressed these concerns by providing additional experiments, clarifying theoretical foundations, and emphasizing the scalability and adaptability of their approach.

The authors’ responses to the reviewers’ questions were thorough and addressed many of the raised concerns, leading to an improved understanding of the paper’s contributions. One reviewer raised some concerns about the applicability of the method. The authors provided reasonable attempts to respond to the questions; however, the reviewer was not responsive.

Given these considerations, I suggest accepting this paper,