Prompting Fairness: Integrating Causality to Debias Large Language Models

## Abstract

Abstract Large language models (LLMs) can easily generate biased and discriminative responses.
As LLMs tap into consequential decision-making (e.g., hiring and healthcare), it is of crucial importance to develop strategies to mitigate these biases.
This paper focuses on social bias, tackling the association between demographic information and LLM outputs.
We propose a causality-guided debiasing framework that utilizes causal understandings of (1) the data-generating process of the training corpus fed to LLMs, and (2) the internal reasoning process of LLM inference, to guide the design of prompts for debiasing LLM outputs through selection mechanisms.
Our framework unifies existing de-biasing prompting approaches such as inhibitive instructions and in-context contrastive examples, and sheds light on new ways of debiasing by encouraging bias-free reasoning.
Our strong empirical performance on real-world datasets demonstrates that our framework provides principled guidelines on debiasing LLM outputs even with only the black-box access.

## 1 Introduction

Large Language Models (LLMs), which are trained on massive text corpora, have been found to exhibit concerning levels of social biases (Sheng et al., ; Gonen & Goldberg, ; Schick et al., ; Bender et al., ; Dodge et al., ).
The unchecked biases can potentially perpetuate and amplify societal inequities, leading to unfair or even unethical outcomes.
This issue is particularly significant as LLMs become more capable and start to serve as foundational components in decision-making systems across various sectors such as healthcare and education.
Many debiasing approaches have been proposed to tackle this issue, for instance, direct fine-tuning of model parameters (Kaneko & Bollegala, ; Garimella et al., ; Lauscher et al., ; Guo et al., ), modifying the decoding steps (Schick et al., ), and prompting-based techniques (Si et al., ; Tamkin et al., ; Oba et al., ; Ganguli et al., ).
For various reasons such as security and business interests, many of the most capable LLMs are closed-sourced, for instance, GPT-4 (Achiam et al., ), Gemini (Anil et al., ), Claude 2 (Anthropic, ), where the general public do not have access to models’ internal structures or parameters.
Thus, prompting-based techniques largely become the only viable option to mitigate bias when using closed-sourced LLMs.

Figure: Figure 1: Different reasoning to answer a coreference question. A biased answer may be due to the use of a gender shortcut, while a bias-free answer is made by considering proper world knowledge given the circumstances.(^2^22The use of demographic information does not necessarily indicate the reasoning is biased: sometimes certain demographic information (e.g., gender) should be considered for situations such as making medical decisions.)
Refer to caption: figures/motivate_figure.png

In this work, we focus on prompting techniques to steer LLMs towards unbiased responses.
To study this, we notice that obtaining unbiased responses essentially boils down to the process of *selecting* proper pieces from the model’s internal representations and knowledge to make the decision.
Let us consider a simple task of resolving the coreference of a gender pronoun in a given sentence.
A model may output a biased answer by using a gender shortcut potentially learned from the imbalanced representations in the training data.
For example, in Figure 2(a), the model may associate the pronoun “he” with “physician” rather than “secretary” due to social biases reflected in the training data – the gender distributions are disproportional for different occupations.
We denote such reasoning process as biased reasoning, where the model *selects* internal representations in an improper way as this specific demographic information (gender) is unrelated to solving the coreference task.
We provide more detailed discussions on selection mechanisms in Section 3.

Figure: Figure 2: Different prompting-based methods guided by one or both intuitions: discouraging ([-]) biased reasoning, and encouraging ([+]) bias-free reasoning.
Refer to caption: figures/motivate_figure_exp.png

Most of the existing prompting-based debiasing methods focus on *discouraging biased reasoning*, for instance, using explicit prompts to avoid using biased associations or prohibiting the utilization of gender information when making the decision.
While these approaches help to a certain extent, another overlooked prompting-based strategy is to *encourage bias-free reasoning* by only *selecting* proper pieces related to the nature of the question.
For example, Figure 2(b) demonstrates one possible way of conducting bias-free reasoning by comparing the likelihood of two situations occurring in real life and drawing on the more plausible situation to infer the coreference resolution.

We reveal the essential role of *selection mechanisms* in the interplay between LLM’s internal reasoning process and different strategies for designing the external prompts.
We first construct a detailed causal model on the data generation process of the training corpus to identify ways biases may be smuggled in the pretraining phase.
Then we construct a causal model of the LLM’s potential reasoning process and connect the two causal models by analyzing how different input prompts could modulate the LLM’s output through *selection mechanisms*.
Building upon the above causal understandings, we introduce a causality-guided framework to debias LLMs, underpinned by the two intuitions: (1) discouraging biased reasoning  and (2) encouraging bias-free reasoning.
Current prompting-based debiasing methods can also be viewed as instantiating one or both of the intuitions, as illustrated in Figure 2.

Using the causality-guided debiasing framework, we propose prompt-design strategies and further conduct a variety of empirical studies to analyze their effectiveness.
We find that prompts employing both intuitions significantly outperform existing prompting-based debiasing methods.
The strong empirical results clearly demonstrate the effectiveness of the causality-guided debiasing framework, even when we only have black-box accesses.

Our contributions can be summarized as follows:

- •
We construct detailed causal modelings for both the underlying data-generating process of the training corpus and the LLM reasoning process. We note that *selection mechanisms* play an essential role in identifying how the LLM’s output could be modulated by different prompts.
- •
We formulate a causality-guided debiasing framework. Through causal understandings of data-generating processes, we identify conditions that strategies for designing prompts should satisfy to debias in a principled and effective way.
The framework also unifies existing prompting-based debias methods.
- •
Using these strategies, we show strong empirical results on debiasing various social biases across different LLMs, demonstrating the clear benefit of our framework.

## 2 Preliminaries

In this section, we provide a brief introduction to causal modeling and causal reasoning (Section 2.1).
We also provide a motivating example to illustrate how the selection mechanism can reshape dependence patterns within the different data (Section 2.2).(^3^33We discuss related works in detail in Appendix A.)

### 2.1 A Brief Introduction to Causality

For two random variables $X$ and $Y$, $X$ is a cause of $Y$ if there is a change in the distribution of $Y$ when we apply an intervention on $X$ while holding all other variables fixed (Spirtes et al., ; Pearl, ).
We can represent causal relations among variables with a directed acyclic graph (DAG), where nodes represent variables, and edges represent direct causal relations between variables.
We denote the direct causal relation between the ordered pair $(X,Y)$ by a directed edge $X\rightarrow Y$.

Local causal modules in a DAG, which characterize the causal relations between the corresponding variable and its direct causes, do not interfere with each other because of causal modularity.
This property is also known as the exogeneity (Engle et al., ), or the independence of causal mechanism (Peters et al., ), resulting directly from the causal Markov condition for the DAG (Spirtes et al., ; Pearl, ).(^4^44There are additional classes of graphs considered in causality literature, for example, directed cyclic graphs (DCGs) (Spirtes, ), ancestral graphs (Richardson & Spirtes, ), and so on. In this paper, we consider causal processes that can be modeled by a DAG. Other graph classes are beyond the scope of our work.)
In the context of language processing, the definition of a variable representing text or tokens is relatively abstract compared to the statistical notion of a random variable in tabular data.
Within the scope of this work, we use the terms “variable” and “node” interchangeably when the context permits clear understandings.

### 2.2 A Motivating Example of Selection Mechanisms

Ideally, we would like samples to be drawn uniformly from the underlying population of interest.
However, in practice, it is very common that the probability of including certain data points in the corpus depends on the characteristics of the data points themselves.

Previous literature has investigated selection mechanisms from different perspectives, for instance,
the influence of selection bias on statistical inference in economic and sociological studies (Heckman, , ; Winship & Mare, ),
causal discovery when there are selection variables and latent common causes (Spirtes et al., ; Zhang, ),
the identification and estimation of functional causal models when selection exists (Zhang et al., ),
the identifiability of causal effect in the presence of selection bias from the graphical condition perspective (Bareinboim & Pearl, ; Bareinboim & Tian, ; Correa et al., ) and from the potential outcome perspective (Hernán & Robins, ),
and the identification of the existence of selection bias from observational data under certain functional assumptions (Kaltenpoth & Vreeken, ).

Figure: Figure 3: An illustrative example of selection mechanisms.
Refer to caption: x1.png

Let us consider an example in the context of medical care.
As illustrated in Figure 3, observed variables $(X_{1},X_{2})$ denote diseases, and $(Y_{1},Y_{2})$ denote corresponding symptoms.
Apart from them, there exist potential binary selection variables $S_{i}$’s ($i\in\{1,2,\ldots,5\}$), where $S_{i}=1$ denotes being selected.
$X_{1}$ ($X_{2}$) is the direct and only cause of $Y_{1}$ ($Y_{2}$), and the two diseases are unrelated in the general population, i.e., when none of the $S_{i}$’s exists.
We use solid edges to represent causal relations among observed variables and dashed edges for those pertaining to selection mechanisms.

For instance, $S_{2}$ is an example of outcome-dependent selection (Zhang et al., ), which can be selecting individuals with symptom $Y_{1}$ from the general population.
The conditional probability $P(Y_{1}\mid X_{1},S_{2}=1)$ in the selected data typically differs from its counterpart $P(Y_{1}\mid X_{1})$ in general population (Zhang et al., ).
As another example, $S_{4}$ can denote the selection mechanism of only considering the hospital in-patient data.
This signifies the setting of the well-known Berkson’s Paradox (Berkson, ), where two unrelated diseases $(X_{1},X_{2})$ appear to be correlated in the hospital data, simply because the data only contains selected patients who have at least one symptoms in $(Y_{1},Y_{2})$.

Figure 3 shows that the selection can be based solely on the cause (e.g., $S_{1}$), solely on the effect (e.g., $S_{2}$), or on both (e.g., $S_{3}$).
Meanwhile, the selection can be based on variables that are not causally related in the general population (e.g., $S_{4}$ and $S_{5}$).
Selection mechanisms can reshape dependence patterns among involved variables, and as a consequence, potentially change downstream outputs as well.
We will see in Section 3 that such property of selection also applies to natural language processing (NLP) contexts.

Figure: (a) Training data corpus
Refer to caption: x2.png

## 3 Causality-Guided Debiasing Framework

We propose our debiasing framework guided by causal understandings of involved data-generating processes.
In Section 3.1, we present detailed causal models of both the underlying data-generating process for the training corpus and the LLMs’ reasoning process.
We reveal that the LLM’s reasoning process is essentially an interplay between its internal representations and conditions specified by the external prompt designs, in which selection mechanisms play a key role.
In Section 3.2, we identify different sets of conditions for prompt designs, which lead to prompting strategies that are not only intuitive but also theoretically grounded.

### 3.1 Detailed Causal Modeling of Processes

We present the causal modeling of the underlying data-generating process of the training data corpus (Section 3.1.1) and the reasoning process of LLMs (Section 3.1.2).

#### 3.1.1 Underlying Data Generating Process of Text in Training Data Corpus

The training data corpus often reflects historical discrimination, and selection mechanisms are commonly involved.
For instance, gender stereotype in occupations arises not due to the existence of a direct causal relation or a common cause between gender and occupation, but due to an underlying selection mechanism.
Specifically, in the training data, among all possible combinations between gender (e.g., male and female) and occupation (e.g., CEO and secretary), there is a tendency to associate CEO more often with male, and secretary with female.
This phenomenon occurs because the training data is a subset selected from an imaginary data corpus that is ideally diverse and comprehensive.

More generally, in Figure 4(a) we model causal relations in the underlying data-generating process of training data corpus (e.g., text scraped from the internet).
Other than demographic information (e.g., race, gender, age), the causal graph contains additional variables of interest.
We use “scenario” to represent the practical situation or context that sets the background of the text (e.g., medical care, hiring).
We use “entity” to denote participants or stakeholders involved in the scenario, for instance, a patient may be an entity in the medical care scenario and a secretary may be an entity in the hiring scenario.
The selection variable $S$ explicitly models the association between demographic information and entity, which is recognized as a major type of stereotype in NLP (Sweeney, ; Bolukbasi et al., ; Zhao et al., ; Tamkin et al., ).
There are different types of texts in Figure 4(a).
“Demographic-agnostic fact” denotes the text that does not explicitly contain demographic information.
“Derivational text based on fact” denotes the text derived from fact, e.g., inference according to definition, fact-check Q&A, and restatement without altering factual contents.
“Demographic-aware text” denotes the text where demographic information appears explicitly.

#### 3.1.2 Reasoning Process of LLMs

Section 3.1.1 introduces causal modeling of the underlying data-generating process to characterize how discrimination is instantiated in LLMs’ training data corpus.
The goal and expectation of the training is for LLM models to capture dependence patterns in the data corpus, and to extract and internalize the related knowledge.
Therefore, we assume that the internal reasoning process of LLMs shares similarities with the underlying data-generating process of data corpus.
Under this mild assumption, we formulate the reasoning process of how LLMs generate outputs based on input prompts in Figure 4(b).
In particular, we reveal the interplay between internal representations and conditions specified by external inputs, and more importantly, how LLM outputs are shaped and modulated by different prompt designs through selection mechanisms.

In Figure 4(b), we use dotted contours to distinguish LLMs’ internal representations from the actual external information in the data corpus, e.g., the contrast between “demographic representation” in Figure 4(b) and “demographic information” in Figure 4(a).
Notice that the internal nodes are not directly observable or accessible.
We use double-stroke contours to indicate selection mechanisms,
directed edges to represent direct causal relations, and dashed edges with hollow arrowheads to denote selection mechanisms.

A “prompt” serves as an external input to LLMs but not as a direct cause of internal representations.
This is because the internal knowledge and representations exist beforehand, making them irrelevant to whether a specific prompt is provided.
The prompt also does not act as an indicator for causal interventions.
Because internal nodes are not directly observable or accessible, one cannot set them to certain values via hard interventions (Spirtes et al., ; Pearl, ; Peters et al., ; Hernán & Robins, ), or change the functional behavior of causal modules via soft interventions (Eberhardt & Scheines, ; Huang et al., ; Correa & Bareinboim, ).
However, the prompt directly changes the selection variable “prompt properly considered” (PPC) through its designs.
If the LLM model is well-trained and well-aligned, we can expect that PPC is always conditioned upon when the model produces the output, and the “LLM potential output” will become the actual output we obtain from the model.
Just like the motivating example where selection can reshape dependence patterns among involved variables (Section 2.2), the prompt can influence the LLM reasoning process in significant ways by specifying conditions in selection mechanisms.

#### 3.1.3 A Remark on the Modeled Processes

Figure 4(a) and Figure 4(b) each have their generating process of interest and distinct emphases.
Although not pertaining to LLM reasoning itself, the underlying generating process modeled in Figure 4(a) provides hints on local causal modules of interest that prompt designs can specifically attend to for debiasing purposes.
When we consider these hints with reference to the LLM reasoning process modeled in Figure 4(b), we can identify debiasing strategies that are both intuitive and theoretically grounded, as we will see in more detail in Section 3.2.
The two detailed causal models complement each other, both of which are essential for understanding the source of bias, and furthermore, effectively debiasing LLMs.

### 3.2 Debiasing Guided by Causal Understandings

We present our prompt-based LLM debiasing framework guided by causal understandings of the related data-generating processes.
Our core idea is to formulate conditions that should be specified in the prompt design, such that through the influence of selection mechanisms in the LLM reasoning process, one can effectively debias LLM outputs.

Figure 4(c) presents an annotated version of the LLM reasoning process (Section 3.1.2).
Based on the understanding of the underlying generating process of training data corpus, and the mild assumption that the trained LLM captures the dependence patterns in training data, we highlight certain edges in light coral (accompanied by annotations marked with circled red numerals) in Figure 4(c) to denote unregulated information flow from demographic representations to LLM outputs in the internal reasoning process.
We use circled blue Roman numerals to denote selection mechanisms that can be specified by the external input prompt (Section 3.1.2), and circled red Roman numerals to represent those corresponding to historical discrimination (Section 3.1.1).

According to the LLM reasoning process presented in Figure 4(c), we consider the information flows from the demographic representation (upstream node) to the LLM potential output (downstream node).
We present additional conditions and constraints the prompt designs should specify for debiasing purposes.
To clearly present the intuitions and theoretical groundings of our causality-guided framework, we identify three prompting strategies for debiasing LLMs, each of which serves as a solid starting point.

###### Strategy I ( Nudge Towards Demographic-Agnostic Fact ) .

The intuition behind is to nudge LLMs towards utilizing demographic-agnostic fact when generating the output:

- Condition I
The internal representations for demographic-agnostic fact and demographic information should be conditionally independent in the presence of PPC and existing selection $S$, i.e.,
$\hskip 0.0pt\mathclap{\makebox[55.00008pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic \\
\scriptsize representation}}}}\makebox[30.00005pt][r]{}\mathbin{\mathchoice{\hbox to0.0pt{\hbox{\set@color$\displaystyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\displaystyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\textstyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\textstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptstyle\perp$}\hss}\kern 2.48856pt{}\kern 2.48856pt\hbox{\set@color$\scriptstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptscriptstyle\perp$}\hss}\kern 1.78203pt{}\kern 1.78203pt\hbox{\set@color$\scriptscriptstyle\perp$}}}\mathclap{\makebox[127.50018pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic-agnostic \\
\scriptsize fact representation}}}}\makebox[65.00009pt][r]{}\mid S=1,\mathrm{PPC}=1.$
(1)

Strategy I introduces Condition I to specify the selection mechanism over internal representations of demographic information and demographic-agnostic fact, denoted by edges i and v in Figure 4(c).

An example prompt employing Strategy I can be:
“Considering the fact that the sentence ‘The physician hired the secretary because the secretary is highly recommended’ is practically more viable than the sentence ‘The physician hired the secretary because the physician is highly recommended’, who does ‘he’ refer to in ‘The physician hired the secretary because he is highly recommended’?”

###### Strategy II ( Counteract Existing Selection Bias ) .

The intuition behind this strategy is to directly counteract the effect of existing historical discriminations:

- Condition II.1
The internal representations for demographic information and entity should be conditionally independent in the presence of PPC and existing selection $S$, i.e.,
$\hskip 0.0pt\mathclap{\makebox[55.00008pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic \\
\scriptsize representation}}}}\makebox[30.00005pt][r]{}\mathbin{\mathchoice{\hbox to0.0pt{\hbox{\set@color$\displaystyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\displaystyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\textstyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\textstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptstyle\perp$}\hss}\kern 2.48856pt{}\kern 2.48856pt\hbox{\set@color$\scriptstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptscriptstyle\perp$}\hss}\kern 1.78203pt{}\kern 1.78203pt\hbox{\set@color$\scriptscriptstyle\perp$}}}\mathclap{\makebox[85.00012pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize entity \\
\scriptsize representation}}}}\makebox[45.00006pt][r]{}\mid S=1,\mathrm{PPC}=1.$
(2)
- Condition II.2
No new association between internal representations of demographic information and demographic-agnostic fact is introduced,
$\hskip 0.0pt\mathclap{\makebox[52.50008pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic \\
\scriptsize representation}}}}\makebox[27.50003pt][r]{}\mathbin{\mathchoice{\hbox to0.0pt{\hbox{\set@color$\displaystyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\displaystyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\textstyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\textstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptstyle\perp$}\hss}\kern 2.48856pt{}\kern 2.48856pt\hbox{\set@color$\scriptstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptscriptstyle\perp$}\hss}\kern 1.78203pt{}\kern 1.78203pt\hbox{\set@color$\scriptscriptstyle\perp$}}}\mathclap{\makebox[122.50018pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic-agnostic \\
\scriptsize fact representation}}}}\makebox[62.50009pt][r]{}\mid\mathclap{\makebox[35.00005pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize entity \\
\scriptsize repr.}}}}\makebox[17.50002pt][r]{},\mathclap{\makebox[47.50006pt][r]{\raisebox{-1.99997pt}{\shortstack{\scriptsize scenario \\
\scriptsize repr.}}}}\makebox[23.99997pt][r]{},\raisebox{-3.00003pt}{\shortstack{
\scriptsize$S=1$ \small,
\\
\scriptsize$\mathrm{PPC}=1$
}}.$
(3)

In Strategy II, Condition II.1 and Condition II.2 serve different purposes.
Condition II.1 aims to counteract existing bias instantiated by selection $S$ (edges vi and vii) by constraining marginal dependence between representations for demographic information and entity (edges i and ii).

Condition II.2 acts as a safeguard, making sure no new bias is introduced in the causal downstream of the entity so that the above counteraction effectively proceeds to the final output.

An example prompt employing Strategy II can be:
“The physician can be either male or female, and the secretary can also be either male or female. Who does ‘he’ refer to in ‘The physician hired the secretary because he is highly recommended’?”

###### Strategy III ( Nudge Away from Demographic-Aware Text ) .

The intuition behind this strategy is to nudge LLMs away from utilizing demographic-aware text to generate outputs:

- Condition III
The internal representations for demographic-aware text and demographic information should be conditionally independent in the presence of PPC and existing selection $S$,
$\hskip 0.0pt\mathclap{\makebox[55.00008pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic \\
\scriptsize representation}}}}\makebox[30.00005pt][r]{}\mathbin{\mathchoice{\hbox to0.0pt{\hbox{\set@color$\displaystyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\displaystyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\textstyle\perp$}\hss}\kern 3.46875pt{}\kern 3.46875pt\hbox{\set@color$\textstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptstyle\perp$}\hss}\kern 2.48856pt{}\kern 2.48856pt\hbox{\set@color$\scriptstyle\perp$}}{\hbox to0.0pt{\hbox{\set@color$\scriptscriptstyle\perp$}\hss}\kern 1.78203pt{}\kern 1.78203pt\hbox{\set@color$\scriptscriptstyle\perp$}}}\mathclap{\makebox[117.50017pt][r]{\raisebox{-2.5pt}{\shortstack{\scriptsize demographic-aware \\
\scriptsize text representation}}}}\makebox[60.00009pt][r]{}\mid S=1,\mathrm{PPC}=1.$
(4)

Strategy III utilizes Condition III to specify the selection mechanism over internal representations of demographic information and demographic-aware text (edges i and iv) to regulate the information flow along the edge 1.

An example prompt employing Strategy III can be:
“Do not use gender information to answer the question. Who does ‘he’ refer to in ‘The physician hired the secretary because he is highly recommended’?”

###### Strategy I ( Nudge Towards Demographic-Agnostic Fact ) .

###### Strategy II ( Counteract Existing Selection Bias ) .

###### Strategy III ( Nudge Away from Demographic-Aware Text ) .

##### Remarks on Three Strategies

The three strategies offer certain effectiveness individually but are not perfect on their own.(^5^55We provide additional illustrations in Appendix B.)
While Strategy I nudges LLMs towards utilizing demographic-agnostic facts, it does *not* explicitly prevent LLMs from using demographic-aware text representations to generate the output, and the demographic information can potentially be associated to the output through an unregulated path containing edges 1 and 5.
Similarly, while Strategy II aims to regulate information flows along edges { 2, 3, 4, vi, vii}, there is *no* explicit constraint involving edges 1 and 5, which leaves space for bias to sneak in during the reasoning process.

Compared to Strategy I that pushes LLMs to focus on demographic-agnostic facts, Strategy III prevents LLMs from referring to demographic-aware text during reasoning.
While edge 1 is explicitly regulated by the selection mechanism specified by Condition III, the information flow along edge 5 can still result in the association between demographic information and the output.
This is because Strategy III does *not* involve conditions or constraints over the dependence patterns among variables including demographic information and demographic-agnostic fact.
As we can see from the conditions specified in Strategy II, no explicit appearance of demographic information in a demographic-agnostic fact does not guarantee the non-existence of dependence between the two.
We would like to note that debiasing is better realized when the strategies are combined as they can address social bias in LLMs more comprehensively.

**Table 1: Performance comparison of various debiasing methods on WinoBias. We show that combining *encouraging bias-free reasoning* and *discouraging biased reasoning* together (Reduce + Fact) significantly alleviates the gender bias for both coreference tasks: Type I coreference resolution requires world knowledge, and Type II coreference resolution can be resolved using only syntactic information. Pro in the table stands for coreference with pro-stereotypical pronouns, and Anti stands for coreference with anti-stereotypical pronouns. A smaller gap indicates less bias.**
| Accuracy (%) | GPT 3 | GPT 3.5 | Claude 2 | GPT 4 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Anti | Pro | $\text{Gap}_{\downarrow}$ | Anti | Pro | $\text{Gap}_{\downarrow}$ | Anti | Pro | $\text{Gap}_{\downarrow}$ | Anti | Pro | $\text{Gap}_{\downarrow}$ |  |
| Type I |  |  |  |  |  |  |  |  |  |  |  |  |
| Default | 43.01 | 79.24 | 36.23 | 62.96 | 94.03 | 31.07 | 67.57 | 92.13 | 24.56 | 82.50 | 97.96 | 15.47 |
| COT (zero shot) | 41.79 | 75.85 | 34.06 | 62.14 | 90.64 | 28.49 | 70.56 | 91.59 | 21.03 | 84.40 | 95.66 | 11.26 |
| ICL | 46.81 | 94.57 | 47.76 | 45.18 | 92.81 | 47.63 | 73.68 | 92.27 | 18.59 | 88.87 | 98.10 | 9.23 |
| Reduce + Fact | 73.27 | 73.95 | 0.68 | 72.73 | 84.67 | 11.94 | 74.08 | 75.17 | 1.09 | 94.57 | 96.74 |  |

## 4 Results and Analyses

In this section, we demonstrate how the two types of approaches, i.e., (1) encouraging bias-free reasoning (Strategy I), and (2) discouraging biased reasoning (Strategy II and Strategy III), effectively steer LLMs towards unbiased response on various aspects of social bias (e.g., gender, race, and age).
We present our experimental settings and results on the WinoBias data set (Zhao et al., ) in Section 4.1, and these on the Discrim-Eval data set (Tamkin et al., ) in Section 4.2.

### 4.1 Gender Bias: WinoBias

The WinoBias data set (Zhao et al., ) evaluates how likely models will assign stereotypical gender pronouns to occupations under coreference resolution tasks.
The sentences in WinoBias are designed to be structurally parallel but differ in gender pronouns. It contains two sets of sentences: pro sentences with the pro-stereotypical gender pronouns (e.g., nurses as she, engineers as he), and anti sentences with anti-stereotypical gender pronouns (e.g., nurses as he, engineers as she).
The data set also has two types of tasks with different levels of difficulties: coreference decisions in Type I task are challenging and must be made using world knowledge about given circumstances, whereas Type II task can be resolved using only syntactic information.

#### 4.1.1 Experimental Settings (WinoBias)

For each sentence in WinoBias, we define the original question to be “Who does [gender pronoun] refer to in the sentence ‘[original sentence]’?”, and we measure the performance of four large language models: GPT-3, GPT-3.5, GPT-4, and Claude 2) across the above two types of coreference tasks (Type I and Type II).

##### Baseline Approaches

There are a few existing works employing prompting techniques to mitigate the bias in LLMs. We consider these three baselines:

- •
Default: asks the original question directly without any additional prompts.
- •
ICL with contrastive examples: provides contrastive examples followed by original question (Figure 2 middle). The examples are considered to be contrastive as the answer remains unchanged with different pronouns. In WinoBias, we use the same 16 ICL examples as in Si et al. ().
- •
Zero-shot COT (Kojima et al., ): asks the model to also “think step by step” after the original question. Shaikh et al. (); Ganguli et al. () design similar COT approaches to mitigate bias in LLMs.

##### Evaluation Metrics

We measure the performances of LLMs on pro and anti sentences in terms of accuracy, and the gap between the two indicates the level of gender bias exhibited by the models (a smaller gap is better).

##### Our Method: Reduce + Fact

We propose Reduce + Fact as one way of encouraging bias-free reasoning collectively with discouraging biased reasoning.
As shown in Figure 2 right panel, we first create two gender-agnostic sentences by replacing the gender pronoun with the two occupations that appeared in the original sentence. Then we ask the model a factual question: which sentence is more likely to happen in real life, as this question does not contain any gender-related information. We then prepend its answer (to the factual question) before asking the original question. This allows us to distill LLM’s non-gender-related world knowledge (Strategy III) and nudge it to explicitly reason with this fact during coreference resolution (Strategy I).
On top of encouraging bias-free reasoning (Strategies I and III), we also prompt that both occupations are equally likely to be male or female to counteract existing selection bias (Strategy II), discouraging biased reasoning.

One caveat of utilizing LLM’s world knowledge is that the performance of Reduce + Fact will increase as the capabilities of the LLM grow, since better world knowledge will further help LLM answer both factual and original questions.

#### 4.1.2 Our Results (WinoBias)

We compare Reduce + Fact with the three baseline approaches on both coreference tasks and summarize our results in Table 3.2.
We expect all LLMs to achieve higher accuracies on sentences containing pro-stereotypical pronouns (pro sentences) compared to these with anti-stereotypical pronouns (anti sentences) due to historical biases in the training data.

##### Coreference Resolution Requiring World Knowledge

For Type I tasks, world knowledge is required to perform coreference resolution.
The Default prompting shows significant biases, with large gaps in accuracy between pro sentences and anti sentences for all models.
The method Zero-shot COT marginally reduces the bias, as seen in the smaller gaps, but its performance is lower on both pro and anti scenarios for GPT-3 and GPT-3.5 when compared with Default, and it only marginally improves the general performance when applied to more capable ones (Claude 2 and GPT-4).
For ICL with counterfactual examples, although the gap becomes larger for less capable models (GPT-3 and GPT-3.5), it can further reduce biases with more capable LLMs (Claude 2 and GPT-4), especially for GPT-4 (with a gap of 9.23%).

Remarkably, Reduce + Fact, which encourages bias-free reasoning while discouraging biased reasoning, substantially decreases the bias across all LLMs. The achieved minimal gaps, with GPT-4 exhibiting a mere 2.17% gap and 94.57% accuracy on anti sentences, demonstrate significantly greater effectiveness compared to other approaches.

##### Coreference Resolution Only with Syntactic Information

For Type II tasks, coreference can be resolved only with syntactic cues.
All models generally show higher accuracies with smaller gaps. However, since there exists a shortcut in the Type II task – the correct answer is always the second entity in the sentence – a smaller gap may not necessarily indicate less bias. Still, Reduce + Fact outperforms the three baselines, particularly with GPT-4, achieving a near-negligible gap of 0.13%, suggesting that our method is highly effective in encouraging models to utilize non-gendered cues for coreference resolution.

##### Summary

These experimental results suggest prompt designs that encourage bias-free reasoning and discourage biased reasoning are effective at mitigating gender biases in LLMs by directing them to rely more on non-gender-related world knowledge (Strategies I and III) and less on gender shortcuts (Strategy II), thus promoting fairer and less biased responses.
Also, the performance gap between pro sentences and anti sentences decreases as the LLMs become more capable, which may indicate that LLMs are less prone to assign occupations with stereotypical gender pronouns as their general (reasoning) capabilities grow.

Figure: Figure 5: Performance comparison on Discrim-Eval across three demographics. The bar denotes the degree of discrimination by comparing the least privileged group with the most privileged group in a given demographic category (the higher the bar, the deeper the discrimination). Different methods (prompt designs) are colored differently (lighter colors denote the ones that amplify bias-free reasoning). Encouraging bias-free reasoning universally decreases the relative gap when added with methods that reduce biased reasoning.
Refer to caption: figures/gender/model_comparison.png

#### 4.1.3 Ablation studies

We further conduct ablation studies to individually examine the effectiveness of the two types of approaches, i.e., encouraging bias-free reasoning and discouraging biased reasoning. We divide the prompt Reduce + Fact into two parts: the Reduce Only part prompting both men and women can perform either occupation equally, and the Fact Only part telling the model which sentence(s) it regards to be more viable in real life.

Moreover, we categorize each LLM’s response into four groups: (i) TT: where the LLM answers both the factual question and the original question correctly, (ii) TF: where the LLM only answers the factual question correctly, (iii) FT: where the LLM only answers the original question correctly, and (iv) FF: where the LLM answers wrongly on both questions.

The above categorization allows us to attribute the model’s coreference mistakes into two categories: the mistake caused by its non-gender-related world knowledge (FF), and the mistakes caused by its gender bias (TF).
Moreover, we can have a better understanding of the model’s correctness as well by looking at FT, where the correctness may be due to potentially biased shortcuts.

Table: Table 2: Error Analysis on WinoBias Type I coreference task. We divide the models’ responses into 4 categories to better understand the cause of their errors and success. FF denotes the (%) of examples where both questions are answered incorrectly, indicating the coreference errors caused by the model’s world knowledge. TF denotes the coreference errors caused by gender bias as only factual questions are correctly answered, and FT indicates coreference success that may be due to gender shortcut since the factual questions get wrong but original questions are correctly answered.

GPT-4
Reduce + Fact 94.44 96.07 1.76 0.27 0.14 0.68 3.66 2.99
Fact Only 91.45 95.93 5.29 0.14 0.54 0.81 2.71 3.12
Reduce Only 70.69 88.20 25.78 8.28 0.95 1.76 2.58 1.76
Default 69.47 87.25 26.87 8.68 1.36 2.44 2.31 1.63

As shown in Table 4.1.3, when we separate the prompting strategies into Fact Only and Reduce Only, we observe that Fact Only still performs quite well, but Reduce Only leads to a significant decrease in performance. This suggests that while discouraging biased reasoning is important, providing explicit instructions that promote bias-free reasoning is crucial for mitigating bias in LLMs.

Moreover, comparing the performance of GPT-4 with GPT-3.5, we can see that the GPT-4’s improvements mostly stem from better (non-gender-related) world knowledge (making way less mistakes on factual questions) and self-consistency (less mistakes in the TF group).

More experimental results can be found in Appendix D.1, where we design prompts to control the degree to which we steer LLMs away from biased reasoning.

### 4.2 Demographic Bias: Discrim-Eval

The Discrim-Eval data set (Tamkin et al., ) encompasses a collection of scenarios, each depicting a hypothetical case where a decision is required, such as approving a loan or issuing press credentials.
The task is to make a Yes-or-No binary decision (affirmative or negative) on the individual involved in each scenario.
The individuals can be characterized by three demographic factors: age (within a range of 20 to 100, in 10-year intervals), gender (male, female, or non-binary), and race (White, Black, Asian, Hispanic, or Native American).

#### 4.2.1 Experimental Settings (Discrim-Eval)

The scenarios in the Discrim-Eval data set are designed such that an affirmative decision (Yes) is always favorable.
The bias is measured using the probability of the Yes decision across a set of demographic variables (Tamkin et al., ).

##### Evaluation Metrics

For our evaluation of demographic bias, we use a relative gap to measure the adverse impact of different demographic variables (Hort & Sarro, ; Pessach & Shmueli, ).
To be more specific, given a bias category that we want to evaluate (e.g., ethnicity), we obtain the probabilities of the ‘Yes’ token for all five different races, and we measure the gap between the largest probability and the smallest probability and divide the gap by the largest one.
This way allows us to understand how substantial the discrimination is when comparing the most privileged group with the least privileged within the given bias category.

##### Baseline Approaches

The prompt-based approaches introduced in Tamkin et al. () can all be viewed as ways to discourage biased reasoning, instantiating Strategy III.
For our baseline, we choose the top four of them that reduce the discrimination the most (i.e., Illegal, Ignore, Really (4x), and Illegal + Ignore).
The exact statements of each prompt can be found in Tamkin et al. ().

To amplify bias-free reasoning (Fact), we first remove all demographic information in each scenario and ask the LLM to make a Yes or No decision based on this base scenario. Then we explicitly include its answer to the base scenario in the prompt and ask the model to decide on the original scenario which contains specific demographic information (e.g., age 20).

#### 4.2.2 Our Results (Discrim-Eval)

Since the evaluation metric requires accessing the logit of each token, our experiments on Discrim-Eval are conducted on these four LLMs: Mistral 7B, GPT-3, GPT-3.5, and GPT-4.
Figure 5 shows our experimental results on three demographic categories: gender, ethnicity, and age.

Figure: Figure 6: GPT has an inherent bias towards aging. GPT-3’s default response (black) is more biased as age increases.
Refer to caption: figures/age/gpt_3_no_legend.png

Each bar in the figure represents the relative gap, which measures the disparity in the probability of making an affirmative decision (‘Yes’) for the most and least privileged groups within each demographic category, characterizing the relative extent of discrimination.
The bars are color-coded according to the methods used, where lighter shades correspond to the methods that explicitly encourage bias-free reasoning (e.g. purple bars denote Illegal and light purple bars denote Illegal + Fact). Default (black bar) denotes the prompt in the original scenario (with demographic information), without any modifications.

While different prompts of discouraging biased reasoning show various degrees of effectiveness in reducing demographic bias, no single prompt stands out as the universally most effective option across all LLMs and demographic categories.
However, encouraging bias-free reasoning (Fact) collectively with discouraging biased reasoning can further decrease the relative gap, compared to employing the approaches on their alone.
This observation is consistent across 4 LLMs and 3 demographic categories.

It is also noteworthy that none of the approaches achieves a zero gap.
This indicates that while the debiasing prompt designs can reduce the discrimination in LLM outputs, they do not completely eliminate it.

##### Adverse Impact on Specific Demographic Groups

We further examine the relative gaps for specific demographic features.
For example, in Figure 6, focusing on GPT-3 (which has the smallest relative gaps among the 4 chosen LLMs), we observe a growing trend of relative gaps on the default response (black bars) as the age increases.
This suggests that GPT-3’s default responses may be more biased towards more senior individuals compared to junior ones.
It is worth noting that the trend may potentially differ across LLMs, and we include additional plots for other LLMs on different demographic groups in Appendix D.2.

## 5 Conclusion

This paper presents a causality-guided and prompting-based LLM debiasing framework.
In particular, we highlight the critical role of *selection mechanisms* in modeling data corpus bias and, more importantly, formulating how prompt designs can influence LLM outputs by specifying different selection conditions on its internal representations.
Guided by causal understandings of such interplay, we identify principled prompting strategies for more effective debiasing.
Our strong empirical results demonstrate the benefits of our framework, offering clear intuitions and theoretical foundations for effective debiasing approaches.
Future work will naturally extend to acquiring bias-free knowledge and representations for LLMs.

### Broader Impact

In this paper, we present a causality-based LLM output debiasing framework.
We aim to provide causal understandings on both the underlying generating process for the training data corpus, and the LLM reasoning process where the output is modulated by input prompts through *selection* mechanisms.
Our proposed framework serves as a general solution to promote fairness in pretrained LLMs without accesssing their model parameters.
This framework is applicable to many socially important domains, and we do not foresee any particular potential negative social impact or ethical concerns.

### Acknowledgement

This material is based upon work supported by the AI Research Institutes Program funded by the National Science Foundation (NSF) under AI Institute for Societal Decision Making (AI-SDM), Award No. 2229881.
This project is also partially supported by an Amazon Research Award (Fall 2022 CFP), the National Institutes of Health (NIH) under Contract R01HL159805, and grants from Apple Inc., KDDI Research Inc., Quris AI, and Infinite Brain Technology.

## Appendix A Related Works

In this section, we provide detailed discussions on related works.
We consider the combinations of very related topics including causality, algorithmic fairness, and LLM reasoning.
In Section A.1, we consider causal notions of fairness that do not specifically pertain to the LLM context.
In Section A.2, we consider existing efforts to draw connections between causality and LLM reasoning.
In Section A.3, we consider previous works on LLM debiasing.
In Section A.4, we consider previous works that involve all three topics.

### A.1 Causality and Fairness

It has been recognized in the algorithmic fairness literature that causality provides a unique tool to facilitate a better understanding of the data-generating process, and therefore, more effective bias quantifications and mitigations (Kilbertus et al., ; Kusner et al., ; Nabi & Shpitser, ; Nabi et al., ; Chiappa, ; Nabi et al., ; von Kügelgen et al., ; Tang et al., ).
Previous causal fairness literature has considered notions based on estimating or bounding various kinds of causal effects (Kilbertus et al., ; Kusner et al., ; Nabi & Shpitser, ; Nabi et al., ; Chiappa, ),
and also the causal modeling of the dynamics as well as the long-term implications of bias mitigation strategies (Creager et al., ; Zhang et al., ; Tang et al., , ).

Because of the relatively abstract definition of the variable or node in the language context, previous approaches for characterizing and enforcing causal fairness are not directly applicable in LLM debiasing tasks.
That being said, as we have demonstrated in our causality-guided debiasing framework, causal understandings of the involved data-generating processes help identify effective debiasing strategies that are both intuitively clear and theoretically grounded.

### A.2 Causality and LLMs

The intersection between causality and LLMs has drawn increasing attention.
Zhang et al. () considers three types of causal questions and aims to evaluate LLMs’ abilities to identify causal relations, discover new knowledge from data, and quantitatively estimate the consequences of actions.
Kıcıman et al. () investigate LLMs’ abilities to perform causal reasoning and solve covariance-/logic- based causal questions.
They also study the failure modes of LLMs and provide techniques to interpret the model robustness.
Jin et al. () propose a benchmark data set for evaluating LLMs’ causal inference capabilities via the task of determining causal relationships from a set of correlational statements.

This line of research focuses on complex causal reasoning abilities in general settings, without specific attention to potential fairness violations.
In comparison, our causality-guided debiasing framework does not involve assumptions/requirements on LLMs’ general-purpose causal reasoning capabilities.
We adopt a rather mild assumption that a well-trained and well-aligned LLM captures the dependence pattern in the training data and that such a pattern is internalized and utilized during reasoning.

### A.3 Debiasing Language Models

There is a large amount of work discussing bias and fairness in the context of language models (LMs) (Bordia & Bowman, ; Liang et al., ; Abid et al., ; Wang et al., ; Liu et al., ; Ray, ; Rozado, ), and our investigation lies on debiasing techniques with causal understandings of the sources of biases.
For debiasing approaches in the context of LMs, there are proposals involving direct fine-tuning of model parameters (Kaneko & Bollegala, ; Garimella et al., ; Lauscher et al., ; Guo et al., ), modifying the decoding steps (Schick et al., ), incorporating Reinforcement Learning with Human Feedback (RLHF) to better align the models with human values (Ouyang et al., ; Bai et al., ; Yao et al., ), and prompting-based techniques (Si et al., ; Tamkin et al., ; Oba et al., ; Ganguli et al., ).
We focus on prompting-based techniques and identify principles for prompt designs to steer LLMs toward unbiased responses by a). reducing biased reasoning and b). encouraging bias-free reasoning.
We provide demonstrations of how we can employ the above two principles.
Works on LLMs reasoning such as Wei et al. (); Zheng et al. () can also be incorporated into with our framework to encourage bias-free reasoning.

### A.4 Debiasing LLMs from Causal Perspectives

Most closely related literature considers LLM debiasing strategies from causal perspectives.
Vig et al. () utilize causal mediation analysis and consider neuron-level intervention to investigate the instantiation of gender bias in Transformer-based language models (Vaswani et al., ).
Zhou et al. () propose Causal-Debias to mitigate the unwanted stereotypical association by fine-tuning pretrained language models.
The causal model they consider involves four variables: label-relevant factor, bias-relevant factor, raw sentence, and ground-truth label.
Wang et al. () pay special attention to entity bias and propose a specific structural causal model (SCM) for easier parameter estimations, such that the intervention-based mitigation strategy can be carried out.
Their causal model involves four variables: entity, raw text, LLM input, and LLM output.

In comparison, we provide detailed causal modelings at a sub-sentence level, considering both the training corpus generating process and the LLM reasoning process.
Furthermore, our framework explicitly models the interplay between internal representations and external inputs through selection mechanisms, providing a clearer picture regarding possible strategies to debias LLM outputs more effectively.
Our approach does not require white-box access or the ability to perform interventions, making our prompting-based framework applicable to a variety of practical scenarios.

Figure: (a) LLM reasoning process (annotated with selection mechanisms specified by prompts)
Refer to caption: x5.png

## Appendix B Further Illustrations and Discussions of Our Framework

In this section, we provide further illustrations on the debiasing strategies identified in Section 3.2.

In Figure 7, we use light coral (blue) to highlight unregulated (regulated) information flows from demographic representations to LLM outputs in the internal reasoning process.
We use mix-colored highlight to denote partially regulated information flow, e.g., the edge from demographic-aware text representation to LLM potential output in Figure 6(d).
Comparing Figure 6(e) (all strategies combined) with Figures 6(b) – 6(d) (strategies applied individually), we can see that collectively, the strategies address the social bias in language more comprehensively.

Here by “regulated”, we are referring to the constraint over the association between demographic information representation and the potential output.
We would like to note that our causal models can naturally handle situations where certain kinds of neutral dependence patterns involving demographic information are not necessarily considered problematic (Tang et al., ).
We present relevant experimental results in Section D.1.
We would also like to note that our causality-guided framework is not limited to situations where demographic information is explicitly involved.
One can adapt our framework to the specific practical setting by incorporating relevant nodes into causal graphs, thereby identifying the most suitable debiasing strategies therein.

## Appendix C Experimental Details

##### Pretrained LLMs

For the GPT models used in our experiments, we consider snapshots from June 13th, 2023 where the knowledge cut-off time is Sep 2021. Since the legacy GPT-3 model (a.k.a., text-davinci-003) is no longer supported when we conduct the experiments, we use the model “gpt-3.5-turbo-instruct” instead as it has similar capabilities as GPT-3 era models. The Mistral-7B model we use in our experiments is the improved instruction fine-tuned version (a.k.a., “Mistral-7B-Instruct-v0.2”). We are unable to use Claude 2 for our experiments on the Discrim-Eval dataset because we cannot access the probabilities of generated tokens. All LLMs responses are obtained with a temperature of 0.

##### Iterative Prompting

The responses of LLMs may not follow a given format even with specific instructions, which makes answer extraction challenging. Moreover, we observe that adding instructions to restrict the format of responses may lower the quality of the generated outputs. Therefore, we apply 2-round iterative prompting in our experiments where we let the models generate freely and then ask them to summarize their answers in one or two words. This way allows us to obtain high answer qualities with an easy extraction process at the same time.

### C.1 Gender Bias: WinoBias

For experiments on the WinoBias dataset, we combined both the training and test data for evaluation as there is no need to separate them when using prompting-based debiasing techniques.
We also noticed that the original dataset contains 400 grammar mistakes across 3000+ sentences in total. We corrected these grammar mistakes as they may hinder the co-reference resolution tasks. We have provided the cleaned version in the supplementary materials.

Since there are 571 examples where two or more models make the same mistakes on the factual questions. After looking into these mistakes, we find that some of the examples are indeed ambiguous in nature (e.g., ”The construction worker stared at the teacher because [who] was angry”).
Therefore, we collected human evaluations to understand the inherent ambiguity in the WinoBias dataset. For each of the 571 examples, we collect 5 human responses on the factual questions (i.e., which sentence is more likely to happen in real life), and we ask the human annotator to choose from {Sentence A, Sentence B, and Equally likely}. Based on the human responses, we identified 60 examples (55 from the Type I task and 5 from the Type II task) where three or more annotators disagree with the ground truth answers or think both sentences in the factual questions are equally likely to happen in real life.
We removed these 60 examples during our evaluation, and we will include the human evaluation results in our codebase.

### C.2 Demographic Bias: Discrim-Eval

The Discrim-Eval dataset contains 70 diverse decision scenarios and $9\times 3\times 5\times 70=9450$ individual decision questions which includes all combinations of [AGE] $\in$ [20, 30, 40, 50, 60, 70, 80, 90, 100], [GENDER] $\in$ [male, female, non-binary] and [RACE] $\in$ [white, Black, Asian, Hispanic, Native American].
To measure the corresponding bias in each demographic category, we reconstruct the dataset by extracting the base scenario which does not contain any demographic information (e.g., we replace all pronouns with the anaphoric reference to avoid leaking the gender information). We then ask the model to decide on each of the 70 base scenarios. There are (1/11/1/2) scenarios where (Mistral 7B/GPT-3/GPT-3.5/GPT-4) refuses to answer or does not output a Yes answer, and we removed these scenarios correspondingly when evaluating these LLMs.

Table: Table 3: Error Analysis on WinoBias Type II coreference task. We divide the models’ responses into 4 categories to better understand their success and failure cases. TT denotes the (%) of examples where LLM answers both the factual question and the original question correctly, FF denotes the (%) of examples where both questions are answered incorrectly, indicating the coreference errors caused by the model’s world knowledge. TF denotes the coreference errors caused by gender bias as only factual questions are correctly answered, and FT indicates coreference success that may be due to gender shortcut since the factual questions get wrong but original questions are correctly answered.

GPT-4
Reduce + Fact 99.62 99.62 0.13 0.00 0.00 0.13 0.25 0.25
Fact Only 99.36 99.87 0.25 0.00 0.13 0.00 0.25 0.13
Reduce Only 98.98 99.49 0.76 0.13 0.00 0.38 0.25 0.00
Default 98.73 99.75 0.89 0.13 0.25 0.00 0.13 0.13

Table: Table 4: Error analysis of Reduce + Fact on Type I and Type II questions in WinoBias.

82.47 82.34 1.27 1.40 0.76 1.91 15.50 14.36
GPT-3.5
Type I 70.15 78.70 10.58 2.04 2.58 5.97 16.55 13.16
Type II 88.69 89.71 1.14 0.13 3.43 4.70 6.73 5.46
Claude
Type I 73.00 73.00 2.99 2.99 1.09 2.17 22.93 21.85
Type II 79.80 81.19 6.61 5.21 2.54 4.07 11.05 9.53
GPT-4
Type I 94.44 96.07 1.76 0.27 0.14 0.68 3.66 2.99
Type II 99.62 99.62 0.13 0.00 0.00 0.13 0.25 0.25

Table: Table 5: Adjusting the levels of counteracting existing selection bias on GPT-3.5 with Reduce + Fact. We investigated how the level of counteracting stereotype impacts the model performance by adjusting the Fact part in the input prompt. For example, a counteract level of 100% indicates the following prompt: Assume that the physician can be male $0\%$ of the time and female $100\%$ of the time, and assume that the secretary can be male $0\%$ of the time and female $100\%$ of the time.

## Appendix D Additional Experimental Results

### D.1 Gender Bias: WinoBias

We include additional ablation studies on the WinoBias dataset, which include a detailed error analysis on WinoBias Type II coreference task (Table C.2), a detailed error analysis of Reduce + Fact with 4 LLMs on Type I and Type II tasks (Table C.2), and an ablation study on adjusting the levels of biased reasoning (Table C.2) in the prompt design.

In Section C.2, we show the detailed results on both Type I and Type II coreference tasks across 4 LLMs. As we can see, our method has bigger improvements on models with better world knowledge as.
models with worse world knowledge could limit our method to reaching its full capacity.

##### Ablation study on adjusting the levels of counteracting existing selection bias

In Section C.2, we investigated how different levels of enforcing Strategy II (counteract existing selection bias) impacts the model debiasing performance by adjusting the Fact part in the input prompt. For example, a counteract level of 100% indicates the following prompt:
Assume that the physician can be male $0\%$ of the time and female $100\%$ of the time, and assume that the secretary can be male $0\%$ of the time and female $100\%$ of the time;
while a level of 50% indicates the following prompt:
Assume that the physician can be male $50\%$ of the time and female $50\%$ of the time, and assume that the secretary can be male $50\%$ of the time and female $50\%$ of the time.
We observe that as the level of anti-stereotype goes down, the errors caused by the use of the gender shortcut increase (TF increases). In addition, by soft adjustment of reducing biased reasoning, we provide not only flexible tuning strategies for the best model performance but also a chance to dive into the underlying reasons for the error.

### D.2 Demographic Bias: Discrim-Eval

In Figure 8, Figure 9, Figure 10, Figure 11, we show in details the performance comparison on Discrim-Eval across three demographic categories across four LLMs (Mistral 7B, GPT 3, GPT 3.5, and GPT 4). The height of the bar denotes the degree of discrimination by comparing the least privileged group with the most privileged group in a given demographic category (the higher the bar, the deeper the discrimination). Different methods (prompt
designs) are colored differently (lighter colors denote the ones that amplify bias-free reasoning). The baseline setting is colored black.

Figure: Figure 8: Performance comparison on Discrim-Eval across three demographic categories in details on Mistral (7B). The height of the bar denotes the degree of discrimination by comparing the least privileged group with the most privileged group in a given demographic category (the higher the bar, the deeper the discrimination). Different methods (prompt designs) are colored differently (lighter colors denote the ones that amplify bias-free reasoning). Amplifying bias-free reasoning universally reduces the relative gap when added with methods that reduce biased reasoning.
Refer to caption: figures/age/mistral_2_no_legend.png

Figure: Figure 9: Performance comparison on Discrim-Eval across three demographic categories in details on GPT-3. The height of the bar denotes the degree of discrimination by comparing the least privileged group with the most privileged group in a given demographic category (the higher the bar, the deeper the discrimination). Different methods (prompt designs) are colored differently (lighter colors denote the ones that amplify bias-free reasoning). Amplifying bias-free reasoning universally reduces the relative gap when added with methods that reduce biased reasoning.
Refer to caption: figures/age/gpt_3_no_legend.png

Figure: Figure 10: Performance comparison on Discrim-Eval across three demographic categories in details on GPT-3.5. The height of the bar denotes the degree of discrimination by comparing the least privileged group with the most privileged group in a given demographic category (the higher the bar, the deeper the discrimination). Different methods (prompt designs) are colored differently (lighter colors denote the ones that amplify bias-free reasoning). Amplifying bias-free reasoning universally reduces the relative gap when added with methods that reduce biased reasoning.
Refer to caption: figures/age/gpt_35_no_legend.png

Figure: Figure 11: Performance comparison on Discrim-Eval across three demographic categories in details on GPT-4. The height of the bar denotes the degree of discrimination by comparing the least privileged group with the most privileged group in a given demographic category (the higher the bar, the deeper the discrimination). Different methods (prompt designs) are colored differently (lighter colors denote the ones that amplify bias-free reasoning). Amplifying bias-free reasoning universally reduces the relative gap when added with methods that reduce biased reasoning.
Refer to caption: figures/age/gpt_4_no_legend.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: I have mixed opinions about this paper. On the one hand, I appreciate the selection bias analogy and the abstraction of the problem to a causal graph and the conclusions that come from it. On the other hand, the final solution proposed is just an ensemble of intuitive prompts, and it is not clear to me that these prompts are meaningfully guided by the selection bias theory. And it is not clear to me that other better prompts could not have been obtained without any selection bias theorization. So my questions to the authors are :
1. Can you show that avoiding bias based on selection bias is better than simply asking the LLM to avoid bias? See one suggestion of a modified prompt in the weaknesses above. I guess the literature on debiasing LLMs may have more simple prompt additions or system prompts that can be added.  In your comparison, also consider the computational cost of your proposed method.
2. How does the selection bias theory help in choosing the correct strategy, and later, the prompt? What was the justification for using strategy II in one case and strategy III in another? 
3. How would Strategy I generalize to more complex bias scenarios? Do you need an LLM for creating the base prompt in that case? Many cases of biases may not be simple template pronoun substitutions.
4. Strategy I seems closest to a variant of in-context-learning where the closest example is chosen so to help the LLM answer the question correctly. Can you compare to a baseline that selects the K nearest in context examples to add to the prompt, rather than a fixed set of in context examples?
soundness: 2
strengths: * Theoretical justification of prompting strategies using a stylized selection bias argument
* Significant reduction of bias in demonstrated experiments.
summary: This paper introduces a prompt-based method to remove biases in a language model's output. It motivates the prompts using the idea of selection bias from causal inference literature. Experiments show a significant reduction in measured bias on two datasets.
weaknesses: * The connection of selection bias, causal graph and the proposed prompt is weak. In one dataset, strategy I and II is used. In the other dataset, strategy I and III. So it is not clear that the set of selection biases listed are exhaustive. Rather, it appears that the list of such selection biases may increase as the datasets increase, and so the current selection biases presented are the union of those that were useful for prompts for these two data sets. 
* Ablations are not provided. What happens when all strategies are used for first dataset, and what happens when strategy I and II is used for second dataset? 
* Other methods of removing bias are not compared. Since the final solution is just a prompting change, there can be other ways of prompting the LLM that are simple to implement. For example, the text "avoid any gender bias while answering the question" can be added to the prompt.  
* Strategy I seems the most useful. However, I have two concerns. First, it may be computationally intensive. Can the authors clarify how many calls would it need to answer a single question? If I understand correctly, there will be one call to create the two possible scenarios for the base question, and then there will be one call to decide which of them is more plausible, and then a final call to actually answer the question? So there are going to be three calls per question? If that's the case, I can think of other methods, for example, the React multi-agent framework, where the LLM is asked to respond to the question first, then there is a critique agent that checks whether there is any bias in the answer, and if yes, it asks the LLM to regenerate the answer giving the feedback from the critique as a part of the prompt. Second concern is that the creation of the base question, while could be templated for simple examples like the one shown in the paper, but eventually will also become a task that an LLM will need to do so, are there ways to use a smaller model or something more efficient for this?

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Does the 'default' in Table 2 refer to the same meaning as the 'default' in Table 1, which solely receives the original question as input? If so, why does this method also include the metrics (TT, TF, FT, FF)?
2. See Weaknesses.
soundness: 2
strengths: 1. This paper provides a detailed analysis of the internal mechanisms behind biased decision-making in LLMs from a causal perspective, offering a theoretical understanding for the proposed prompting-based debiasing methods.
2. The Introduction section is well-written and effectively conveys the existing challenges.
3. Extensive experiments demonstrate that the proposed DDP significantly outperforms other baseline methods.
summary: This paper addresses responsive bias in both open-source and proprietary LLMs by formulating a theoretical debiasing framework that analyzes the impact of social information on an LLM's decisions from a novel causal perspective. The framework identifies causal pathways through which social information influences model outputs and develops the integrated inference-time prompting strategies to accordingly regulate information flow across different pathways, thereby suppressing potential social bias. Extensive experiments on real-world datasets across multiple domains validate the framework, demonstrating its effectiveness in debiasing LLM decisions.
weaknesses: 1. The proposed DDP suffers from poor applicability for general-purpose generation: DDP is a prompting-based technique that relies on adding extra textual guidance into  the prompt context to achieve debiasing. For evaluation, DDP needs dataset-specific design to obtain its corresponding textual guidance, thereby adapting to different evaluation datasets. So DDP can not be applied to free-form generation, especially when social attributes of interest are not directly given in input prompt but emerge as the intermediate generated results of LLMs, limiting its further applicability in more critical scenarios.
2. The paper organization is unclear and redundant: the pre-trained data generating process is not necessary for understanding the main contributions of this paper. The theoretical insights (strategies) are coupled together with the technical details, making it hard to fully and clearly assess the technical contributions after reading Section 3.
3. Prompting-based debiasing approaches have limitations and may not be particularly meaningful. Traditionally, the responsibility for model alignment and debiasing lies with model deployers. Implementing prompting-based debiasing can be unsafe and inevitably introduces additional time costs.
4. The textual guidance introduced by DDP may introduce noise that hinders the reasoning abilities of LLMs. As shown in Table 1-Type II, while DDP reduces the bias gap, it significantly degrades performance in terms of Anti and Pro (on GPT-3/3.5 and Claude 2). Therefore, their robustness remains an unsolved problem.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: How does the effectiveness of the proposed debiasing strategies vary with model size and architecture?
soundness: 3
strengths: This work presents a novel theoretical framework that bridges causality and LLM debiasing, providing clear intuitions and principled strategies for addressing bias. The causal modeling of both training data generation and LLM reasoning processes offers valuable insights into bias sources and mitigation approaches.
summary: This paper proposes a causality-guided framework for debiasing large language models through prompting strategies. The authors introduce a novel perspective to identify how social information influences LLM decisions through different causal pathways and develop principled prompting strategies to regulate these pathways through selection mechanisms. The framework encompasses two main approaches: encouraging fact-based reasoning and discouraging biased reasoning. The authors validate their framework through extensive experiments on multiple benchmark datasets (WinoBias, BBQ, and Discrim-Eval) across various social dimensions, demonstrating its effectiveness in debiasing LLM decisions while maintaining strong performance.
weaknesses: Although the authors demonstrate improved performance across multiple models, there's limited discussion of how the effectiveness of their approach might vary with model scale or architecture. Additionally, while the authors show reduced bias metrics, there could be more analysis of potential trade-offs between bias reduction and task performance.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. When the three strategies are combined, both social-salient text representation and social-agnostic fact
representation are independent of social category
representation. Does this mean that it is equivalent to using only prompts Q:{Base Question}, a prompt that is completely unrelated to sensitive features, to let LLM complete the answer to Original Question? If not, when both social-salient text representation and social-agnostic fact
representation are independent of social category
representation, what is the performance difference of LLM decision brought by prompts Q:{Base Question} and Q:{Given that your answer to {Base Question} is {Answer to Base Question}, Assume male and female are equally represented in drivers and in teachers. Do not answer the question using gender information. {Original Question}}? Can the author give experimental results on this point?
2. Should there be a solid line in Figure 3 connecting prompt to LLM potential decision?
3. In addition to the fact that prompting-based techniques are suitable for dealing with black-box scenarios, can the authors add a discussion on the advantages and limitations of prompting-based techniques compared to direct fine-tuning of model parameters and prompting-based techniques, for example, a limitation of  prompting-based technique ——the need for human users to be proactive and knowledgeable to complete debiasing?
4. Can the authors provide a comparison of this work with other prompting-based debiasing methods mentioned in weakness, either experimentally or analytically?
5. In the Table 2, why the sum of the true-rate (42.33%, 32.02%) and false-rate (8.68%, 8.82%) of the base question under GPT-3.5/Counteract Only/Anti case is not 1?
soundness: 3
strengths: The given causality-analysis clarified the origins of bias in well-trained LLMs, making the proposed bias mitigation strategies more explainable. 

In addition to examining the influence of training data corpus and prompts, the authors also considered bias caused by data selection, offering a more systematic and comprehensive bias-mitigation solution.
summary: This article conducted a causality analysis for bias in LLM's decision and provided a prompting-based solution for bias mitigation. The solution included three strategy pathways and demonstrated that the combining strategy can achieve comprehensive debiasing.
weaknesses: 1. The assumption in A.3 CAUSALITY AND LLMS, 
"We adopt a rather mild assumption that a well-trained and well-aligned LLM captures the dependence pattern in the training data and that such a pattern is internalized and utilized during reasoning" 
needs to be put in the main text. If LLMs are not well-trained and well-aligned, then the author's prompting-based de-dependency method will no longer be so effective, so I think this is an assumption worth stating in the main text.

2. There is a lack of discussion related to other prompting-based debiasing methods in LLMs, such as those in [1][2][3].

[1] Zhang, C., Zhang, L., Zhou, D., & Xu, G. (2024). Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment. arXiv preprint arXiv:2403.02738.

[2] Li, J., Tang, Z., Liu, X., Spirtes, P., Zhang, K., Leqi, L., & Liu, Y. (2024). Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework. arXiv preprint arXiv:2403.08743.

[3] Furniturewala, S., Jandial, S., Java, A., Banerjee, P., Shahid, S., Bhatia, S., & Jaidka, K. (2024). Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models. arXiv preprint arXiv:2405.10431.

Meta-review: The paper presents a novel framework that leverages causal analysis to mitigate biases in large language models (LLMs) through prompting strategies. The paper’s strengths include its theoretical foundation, extensive experimental results demonstrating bias reduction, and a comprehensive bias-mitigation solution.

However, the paper has some weaknesses, such as a weak connection between selection bias, causal graphs, and proposed prompts, limited discussion on the effectiveness of the approach with different model scales and architectures, and concerns about computational intensity and general applicability. The authors addressed these concerns by providing additional experiments, clarifying theoretical foundations, and emphasizing the scalability and adaptability of their approach.

The authors’ responses to the reviewers’ questions were thorough and addressed many of the raised concerns, leading to an improved understanding of the paper’s contributions. One reviewer raised some concerns about the applicability of the method. The authors provided reasonable attempts to respond to the questions; however, the reviewer was not responsive.

Given these considerations, I suggest accepting this paper,

Decision: Accept (Poster)