ChatGPT Based Data Augmentation for Improved Parameter-Efficient Debiasing of LLMs
Authors: Pengrui Han
Rafal Dariusz Kocielnik
Adhithya Prakash Saravanan
Roy Luoyao Jiang
Or Sharir
Anima Anandkumar
Venue: COLM (2024)
Keywords: large language models, social bias, synthetic data generation, debiasing
TL;DR: This work proposes novel strategies using ChatGPT to create synthetic training data aimed at efficiently debiasing Large Language Models (LLMs)

Abstract: Large Language models (LLMs), while powerful, exhibit harmful social biases. Debiasing is often challenging due to computational costs, data constraints, and potential degradation of multi-task language capabilities. This work introduces a novel approach utilizing ChatGPT to generate synthetic training data, aiming to enhance the debiasing of LLMs. We propose two strategies: Targeted Prompting, which provides effective debiasing for known biases but necessitates prior specification of bias in question; and General Prompting, which, while slightly less effective, offers debiasing across various categories. We leverage resource-efficient LLM debiasing using adapter tuning and compare the effectiveness of our synthetic data to existing debiasing datasets. Our results reveal that: (1) ChatGPT can efficiently produce high-quality training data for debiasing other LLMs; (2) data produced via our approach surpasses existing datasets in debiasing performance while also preserving internal knowledge of a pre-trained LLM; and (3) synthetic data exhibits generalizability across categories, effectively mitigating various biases, including intersectional ones. These findings underscore the potential of synthetic data in advancing the fairness of LLMs with minimal retraining cost.

Published as a conference paper at COLM 2024
ChatGPT Based Data Augmentation for Improved Parameter- Efficient Debiasing of LLMs
Pengrui Ha,, Rafal Kocielnik Adhithya Saravanan Roy Jiang', Or Sharir!
Anima Anandkumar'
ICalifornia Institute of Technology, CCalleton College, University of Cambridge (barryhan@carleton.. edu, rafalko@caltech.cem
Abstract
Large Language models (LLMs), while powerful, exhibit harmful social biases. Debiasing is often challenging due to computational costs, data constraints, and potential degradation of multi-task language capabilities. This work introduces a novel approach utilizing ChatGPT to generate syn- thetic training data, aiming enhance the debiasing of LLMs. We propose two strategies: Targeted Prompting, which provides effective debiasing for known biases but necessitates prior specification of bias in question; and General Prompting, which, while slightly less effective, offers debiasing across various categories. We leverage resource-efficien LLM debiasing using adapter tuning and compare the effectiveness of our synthetic data existing debiasing datasets. Our results reveal that: (1) ChatGIT can efficiently produce high-quality training data for debiasing other LLMs; (2) data produced via our approach surpasses existing datasets debiasing performance while also preserving internal knowledge of pre-trained LLM; and (3) synthetic data exhibits generalizability across categories, ef- fectively mitigating various biases, including intersectional ones. These findings underscore the potential of synthetic data in advancing the fairness of LLMs with minimal retraining cost.
1 Introduction
Recent advancements in Large Language Models (LLMs) have significantly improved Natu- ral ral Language Processing (NLP) capabilities. However, concerns about fairness have emerged (Bender et al. 20211), highlighting that LLMs may inherit and amplify real-world biases like racial and gender biases (Kirk et al. 2021), toxicity (Gehman et al., 2020), and misinformation (Weidinger et al. 2022) due to their training on vast human human-generated text data. This issue is critical when LLMs are applied in sensitive areas like healthcare, job recruitment, or criminal prediction, where such biases could lead to widespread discriminatory outcomes.
Considerable efforts have been made in recent research to debias LLMs. However, with the large size of these models, social bias mitigation appears to be particularly challenging (Xie & Lukasiewicz [203] Brown et al., 2020 Hoffmann et al.] (2022). Traditional methods are computationally expensive as they often require model retraining (Tokpo al. 2023).
On top of that, retraining on limited data can lead to lowering LLM's general language capabilities due to catastrophic forgetting (Fatemi et al., 2023). On the contrary, recent parameter-efficient methods (He et al.. 2022. Ding et (022 Xie & Lukasiewicz) 2023 offer a good alternative as they only require minor and targeted parameter adjustments. While more efficient, these approaches heavily rely on the quality of training data (Delobelle et al.) and may offer limited generalization, posing a challenge for comprehensive bias reduction (Li et al. 2022).
Our Approach In this work, to bolster the robustness of light-weight debiasing, we propose a method to systematically prompt ChatGPT (Ouyang et al. 2022) to generate
Both authors contributed equally to this research

---

Published as a conference paper at COLM 2024
Deblasing Performance on GPT2 and BERT (Lower Values cate Less Bias)
Debiasing Performance GPT2 Family Models (Lower Values
Bias)
Sacelre Declodig
gial Modz) Score
70.0
Origna Model Score
Syntheic Target Dubrasing
Syrthetc Gepera bebisinn
Balk Stores
62.5
57.5
55.0
56
54
GFT2
BERT
96.0
Usma_To_v
GPI-350m
GPI-Tee- 125m
Figure Debiasing performance of different strategies on GPT-2 and BERT averaged across three bias categories and two datsets (StereoSet models before and after synthetic debiasing. and CrowS-Pairs).
Figure 2: Average bias score across three bias cate- gories and two metries for different GPT2 family
synthetic training data for LLM debiasing (Fig. 3)_ This achieved using two distinct prompting strategies: targeted prompting and general prompting, complemented by an auxiliary method, loss-guided prompting. The first one is meant to debias models for concrete category, which requires prior knowledge about the social bias to target, whereas the latter one has the potential to offer comprehensive debiasing and helps assess the generalizability of synthetic data to unknown social bias categories.
In particular, targeted prompting creates data specifically address particular category of bias, while general prompting generates data intended to be useful for mitigating bias across a range of categories. We conducted extensive evaluations on the impact of bias mitigation using synthetic dataset through the parameter-efficient method of adapter tuning (Houlsby et al. 2019) across racial, gender, and religious bias. We also show promis- ing results in debising models for challenging intersectional categories based recent BiasTestGPT dataset (Kocielnik et al. 2023b).
Findings Our primary findings include:
Our synthetic data effectively mitigates bias in popular LLMs (Fig. 1. On GPT2 and BERT, our approach surpassed the performance of the recent Wikipedia-basse from Xie & Lukasiewicz 2023). Specifically, our best method enhances bias mitigation by an average of 6.4% on GPT-2 and 1.7% on BERT. Detailed results are in (Tables 3
dataset
Our method generalizes broadly reducing bias across GPT-2 family models by: 8.2% in LLaMA-3B, 5.8% in both OPT-350m and GPT--Neo-125m (Fig, 2
We also show promising results for challenging intersectional category related to Mexican Females from Kocielnik et al. (2023b) where we lower bias on GPT-2 by 12.9% (Table
As result of our debiasing strategies, the general language model capabilities (LMS) in GPT-2 models are either slightly improved or minimally diminished (less than 1.3%). For
BERT models, the variation in LMS is within 2.5%.
Debiasing performance is improved with much less data, speeding up training by up to 60 times compared to Wikipedia-base based baselines (Xie & Lukasiewicz 2023).
Contributions
We contribute by:
Introducing novel approach to bolster the robustness of parameter-efffiient debiasing by prompting ChatGPT to generate high-quality synthetic debiasing data.
Proposing two methods for synthetic data generation for debiasing: targeted providing superior debiasing but requiring prior knowledge of social bias definition, and general mitigating range of social biases without prior knowledge but at the cost of reduced overall effectiveness.
We further experiment with a variation of targeted prompting, loss-guided prompting, that yields promising initial results on the BERT model.
We share the code our GitHub repository
2

---

Published as a conference paper at COLM 2024
Debiosing Prompting Strategles Synthetic Data Generation
Generated Terms
Anti- sterssttrrrppeeeeoo Sentences
Targeted
woman ROONGER
remarkatly
Generation
I-woman
Debissing
Prompts
"Stokk""
Targstad Detissing- (Gender)
sttl-stte Sentenoes
Training Adapter
S: The male nurse was praised for his
Adapter Based Synthelic Data
Debiasing
ChatoPT
esceptional care and sensltvity"
I-mmal"
General
t'tonsiivity"
Targeted prompting chosen category General prompting for general debiasing
Symthetic data generation through ChatGPT
Deblasing model using parameter effilent Adapter Tuning
Our debiasing framework using ChatGPT-based synthetic dataset generation and Adapter- Tuning. The upper part the process for targeted prompting and the bottom is for general prompting.
Figure 3:
2 Related Work
2.1 Social Bias in Large Language Models (LLMs)
Biases can lead to harmful outcomes and may emerge at different stages of constructing and deploying an LLM (Suresh & Guttag 2019). The primary causes include inherent biases
in the training data and those introduced during the model construction stage, where the
chosen architecture and algorithms may propagate certain biases (Ferrara 2023 Bender let al. 2021a; Baumann et al. 2023).
Since LLMs like ChatGPT are trained on vast amounts of real-world data, limitations in current data preprocessing techniques can result in the retention of various biases within the pre-training dataset. These biases may include human stereotypes, underrepresentaaion of certain social groups, racism, hate speech, and other forms of bias (Ferrara 2023 Zhao et al. 20196 Bender et al. 2021a He et al. 2023). Despite significant efforts to cleanse and curate datasets, the massive volume and complexity of the data often mean that some undesirable elements and imbalances in the representation persist. These data issues can potentially introduce biases to the language models during the training stage, and thus reproduce or even reinforce existing societal prejudices and inequalities (Li et al.) 2023).
The model architecture and training algorithm themselves can introduce additional bias. Modern LLMs are designed to generalize from their training data to new, unseen inputs. This generalization is double-edged sword. While it enables the models to understand text and sentences they have not previously encountered and to generate seemingly relevant responses to a wide range of queries, it also carries the risk of capturing only the central mass of the observed distribution, while failing to correctly model the tails. This can lead to perpetuating stereotypes and biased inferences (Ferrara 2023). Bias mitigation in LLMS often involves a compromise in general performance and language capabilities. This trade-off represents fundamental challenge in LLM bias mitigation efforts (Li et al. 2023).
2.2 Bias Mitigation Techniques
Studying and mitigating biases in LLMs has become increasingly important. Substantial effort has been devoted to addressing both algorithmic and training data bias. For algo- rithmic bias, prior work focused on developing aadesssg human-understandale methods for detecting and mitigating biases (Richardson et al.) 2023 DiCiccio et al. 2023). However, a significant challenge lies in the scarcity of suitable datasets, otten relying on limited number of benchmark datasets (Baumann et al. 2023). Additionally, these efforts do not extend to addressing biases inherent in the training data.
On the other hand, more general debiasing methods have been developed. These include novel algorithms (Yu al.; Ma et al., 2020), leveraging pre-trained language models for generating gender variants of given texts (Jain et al. 2022), using unsupervised pipelines to curate and refine instances mentioning stereotypes Gaci al. (2023), increasing the amount of data used for training (Liang et al., 2020b Schick et al., 2021a Wang et al. 2022), and employing extra prompting to suppress social biases (Oba et al. 2023). However, these debiasing techniques often still rely manual templates and are limited by the availability of the data (Delobelle et al.). Additionally, existing datasets have been shown to exhibit issues related to data quality and reliability (Blodgett et al. 2021). Furthermore, curated
3

---

Published as a conference paper at COLM 2024
datasets are often difficult to expand and generalize. Consequently, using them for debiasing could lead to overfitting specific social bias categories.
More importantly, even in the presence of proper data, the large-scale training methods that are not parameter-efficient or using post-hoc method (Liang et al. 2020a Schick et al. 2021b)
can be costly and potentially compromise the general laneuage capabilities of LLM (Xie
& Luuasieewcz 2023: Fatemi et al. 2023). This makes the process of debiasing challenging
endeavor. In this context, recent work of Xie & Lukasiewicz (2023) is noteworthy. They evaluated parameter-efficient debiasing methods on two popular language models, BERT (Devlin et ) and GPT-2 (Radford et al., 2019,, against gender, racial, and religious biases using existing datasets. Although the methods are more computationally efficient, they rely on a limited subset of the data from Wikipedia which can be used for counterfactual augmentation. The evaluation is also limited in scope to small LLMs. We contrast the effectiveness of the debiasing approach in this work to our own approach which utilizes dynamically generated synthetic data for bias mitigation. We further expand the debiasing to larger modern LLMs.
2.3 Synthetic Data Generation with Generative AI
As our debasing approach utilizes synthetic data generation, we provide a brief background of related methods in this area. Synthetic data plays a crucial role in scientific research. Commonly, it serves two primary puposes: 1) to emulate real-world data, enabling the creation of large datasets while preserving privacy (Angwin et al. 2022: Baumann et al., 2023 and 2) to generate test cases for evaluation purposes (Le Quy et al. 2022).
Traditionally, synthetic data generation has mainly relied on learning probability distribu- tions from real data [Baumann et al. 2023). However, this approach often faces limitations, particularly when real data is scarce (Liu et al., 2024). Learning from limited data can also be costly and inflexible, requiring complex modeling techniques or human-designed frameworks (Guo & Chen 2024 CRACS-INESC TEC & Faculty of Sciences 2022). In con- trast, generative AI, particularly LLMs trained vast real-world data, synthesize new data that closely resembles real data in both structure and content (Veselovsky et al. 2023).
Previous studies have shown that LLMs can be used effective training data generators for many applications (Borisov et al.) 2022 Chintagunta et al.) 2021 Sun et al. 2024).. Recent research (Yu et al., 2024 Wang al. 2023 further demonstrates that it is possible con- trol the quality and dimensions of the data during generation to achieve better outcomes. However, few studies have systematically developed strategies to generate efficient data specifically targeted for debiasing. In this work we leverage ChatGPT's knowledge of social bias to generate highly effective and generalizable data for mitigating bias in LLMs.
3 Methodology
Figure 3 introduuss our debiasing framework and prompting strategies for synthetic data generation used for parameter-efficient LLM debiasing.
Targeted Prompting: In the targeted prompting approach, we prompt ChatGPT to produce sentences that aim to debias a specific category. Our first step is to identify the category of bias we aim to mitigate. The generation process consists of two primary components: term generation and sentence generation. Initially, welprompt ChatGPT to produce social group terms related to the chosen bias category by providing a few sample terms. Subsequently, we prompt ChatGPT to create anti-stereotyped sentences using the generated terms. We instruct ChatGPT to generate sentences that counter prevailing stereotypes associated with particular social group (e.g,, race-related terms). The desired output format is communicated by asking ChatGPT produce sentences that connect social group term with an anti- stereotyped attribute. Each generated sentence, "S", should also indicate the corresponding social group term, "T", and attribute term, "A" following the format: S, T, A. The previously generated terms serve as references for ChatGPT during this process.
To ensure the quality of the produced data, we include additional specific instructions. For instance, we ask ChatGPT to diversify the terms used and to produce sentences with varying
4

---

Published as a conference paper at COLM 2024
levels of complexity, All relevant terms can be found in Appendix 21 and 23 Examples of sentences and visualizations of terms are presented in Table and Fig. respectively. The prompts used for ChatGPT are detailed in Appendix 17
General Prompting: The General Prompting approach aims to produce data that mitigates biases across various categories. Consequently, during the generation process, we afford ChatGPT greater freedom. We neither select specific bias categories nor generate social group terms. Instead, we directly|prompl ChatGPT P to create antiistereotypical sentences that counteract stereotypes, adhering the """"""""""""" format previously detailed. All terms are located in Appendix 22 and 24 Meanwhile, examples of sentences and visualization of
terms are in Table | and Fig. The ChatGPT prompts are in Appendix 17
We formalize
Targeted and General Prompting in Algorithm 1
Loss-Guided Prompting: We observed diminished effectiveness and more pronounced trade-off between debiasing performance and language ability in models outside the GPT family, such as BERT, when using synthetic data generated from ChatGPT. This could be due to out-of-distribution generations from ChatGPT that harm the pre-trained knowledge of BERT in the course of further pre-training. A phenomenon known as catastrophic forgetting (Luo et al. 2023). To address this, we aim to guide ChatGIT to generate more in-distribution sentences for the given LLM. We select 50 samples exhibiting the highest and lowest loss, respectively under given LLM, from the generated data for each category. These samples, along with their corresponding loss scores, were then provided back to ChatGPT This approach guides ChatGPT to generate data that is more in-distribution.
Loss Loss-Guided Prompting is auxiliary method used to generate more in-distribution data for targeted and general prompting, its format follows these two strategies, and we do not present it separately in the Table |
We formalize Loss-guided Prompting in Algorithm 2
Training Methodology: We train LLMs using synthetic data generated by GPT4 through adapter tuning (Houlsby et al. 2019). Adapter tuning freezes all the original parameters of an LLM and introduces, additional adapter layers into the original model architecture.
Only these are trained for downstream applications. For GPT-2 and other GPT2 family models, we modify the sentence to position the attribute word at the end, employing the Causal Language Model loss as our training objective. For the BERT model, we mask the attribute word within the sentence and use the Masked Language Modeling (MLM) loss as the objective.
4 Experiment
Metrics and Datasets: To align with Xie & Lukasiewicz (2023), we use the CrowS-Pairs (Nangia et al., 2020) and the StereoSet (Nadeem et al. (021) intrasentence datasets. The CrowS-Pairs dataset comprises pairs of contrasting sentences, one of which is more stereo- typed than the other. The StereoSet intrasentence dataset contains entries each composed of a stereotyped sentence, an anti-stereotyped sentence, and an unrelated sentence. The differ- ences among these sentences are solely the attribute word. The CrowS-Pairs dataset contains 262, 105, and 516 entries for gender, religion, and race, respectively. For the StereoSet in- trasentence set, there are 1026, 623, and 3996 examples respectively. For bias evaluations, we adopt the "stereotype score (SS)" from Meade et al. (2022). It quantifies the preference of an LLM for stereotypical association over an anti-stereotypical one, with an ideal score being 50% for an unbiased model. To assess model's 's general language capability, we use the "language modeling score (LMS)" from Nadeem et al. (202)) measures the model's preference for meaningful associations over unrelated ones, with an ideal score of 100%.
For the targeted prompting experiment, we generated three datasets for every category of bias, each containing 500 targeted sentences. For the general prompting experiment, we produced three datasets, each with 500 general sentences. We also tested various data sizes and selected 500 based on the optimal balance between debiasing per- formance and language ability impact. The performance graph for different data sizes is included in the Appendix To represent masked and autoregressive language models, and to align with Xie & Lukasiewicz 2023), we chose to debias BERT (Devlin et al.) and GPT-2
Training Details:
5

---

Published as a conference paper at COLM 2024
Race Subject Words
Gender Subject Words
Religion Subject Words zen
General Subject Words
eth persiar
woman
VOOd00 boha WICCa
Jan
Yost Tady
bodybui de
Re shiennz 1
bOy
cutch a
Tather
Peos bar boxer o a accountant
most he. ma
su SEFSSman Smanalects
Race Attribute Words
Gender Attribute Words
Religion
Attribute Words
General Attribute Words
cultural
thin wise
chin- n. cariigg
mind mindfulnee fulnnes
inp
Licity peace
poetr y coding smoithology
ba Het
pet tery
aueniia Intric aetistic diverse varratt profann
angey
compassions
as Astrophysii CS
innovative
aroggtnt Rech
unit
Sa parmony
balance
opera Cer aeeerature ature
sur
Figure 4: The most frequent words generated through each prompting are visualized via word clouds.
The larger the word, the more frequently it has been generated.
De-biasing Strategy
Bias Type
Sentence
Subject Word
Attribute Word
Targeted Prompting
Gender
"Love the Godfather not for his power, but for his genuine kindness."
"Godfather"
"Kindness"
Race
"The Scottish community has been "Scottish"
"Innovation"
a beacon of innovation in various fields."
Religion
"Many students find the Quran to be "Quran" enlightening."
"Enlightening"
General Promptiig General
"Just because she's a cheerleader "Cheerleader" doesn't mean she isn't the top student in her calculus class." "
"Calculus"
"She found that skateboarder was "Skateboarder" also connoisseur classical music."
"Classical"
Table 1: Example data of Targeted and General Prompting, including sentence, subject word, and attribute word for each example. Additional examples can be found in Appendix T6
(Radford et al. 2019). To show the generalizability of our method, we also experimented with other GPT2 family models: Llama_3b. v2 (Touvron et al., 2023) (the latest version of
LLaMA-3B model), OPT-350m (Zhang et al., 2022), and GPI-Neo-1255
(Gao et al. 2020)
We use Adapter Hub (Pfeiffer et al., 2020) and the code from Xie & Lukasiewicz 2023
We trained Llama 3b v2 model on a Google Colab A100 GPU. All other experiments were conducted a Google Colab V100 GPU. Based on empirical findings and the ratio between SS and LMS, we set the learning rate for the GPT-2 model to 5 x 10-5 6 For BERT, the learning rate was set to 1 x 10-5, For Llama 3b. v2 and OPT-350m, we used 5 10-5, and for GPT-Neo-125m: x 10-4, For each bias category or for general debiasing, we conducted the experiments for the three datasets separately and reported the average outcomes as well as the standard deviations.
Baseline: For GI'T-2 and BERT, we compare our debiasing approach, which uses synthetic datasets via adapter tuning, with other parameter-efficient methods and existing datasets, focusing particularly on the work of Xie & Lukasiewicz (2023). In their study, the authors down-sample 20% of the English Wikipedia as the debiasing corpus and augment it coun- terfactually for training (Zhao et al.) 2019a Zmigrod et al. 2019 Webster et al. 2020). The debiased corpus is then used with three distinct parameter-efiict
methods: prefix funing (Li & Liang ] prompt tuning (Lester et al., 2021), and adapter tuning (Houlsby et al. 2019). For other models in the GPT-2 family, due to the lack of relevant prior work for comparison, we assessed the effectiveness of debiasing by comparing the debiased versions our
of the models to their original versions with weights before our debiasing.
Results
Mitigating Racial Bias: Table 2 shows that our synthetic data surpasses all other parameter-efficient methods that utilize English Wikipedia for GPT-2 models in mitigating racial bias. For BERT, our results are in line with the baselines. Our general debiasing achieves the best SS for StereoSet and yields results comparable to others for the SS on CrowS-Pairs, with the difference being less than 3%. In terms of language capability, our
6

---

Published as a conference paper at COLM 2024
Racial Bias
CrowS-Pairs
Changel
StercoSet
Change
LMS
Change 1
GPT-2 Model
59.69
58,90
91.01
Wikiidebiased Prefix
59.61g.51
10.13%
57.53,22
12.33%
89.48g.08
01.68%
Wikiidebbased + Prompt
58.76p.92
11.56%
57.720.33
22.00%
89.18p.10
%
Wiki-debiased Adapter
61. 61.28
12.66%
57.770.4
11.92%
89.01p.68
22.20%
-yynthetic-arggeted Adapter
55.04a43
17.79%
47.35p
11.661%
89.93
1.19%
-Syntheti--eneeall + Adapter"
58.791.58
11.51%
53,41096
99.32%
88.74p.03
02.49%
BERT Model
62.33
57.03
84.17
Wiiiidebbassdd Prefix
57.44,.90
17.85%
56.950.39
00.14%
84.350.12
%0.21%
Wiki-debiased + Prompt
58.253.90
66.55%
58.17
12.00%
83.41p.80
00.90%
+ Wikiidebbasedd Adapter
17.204.16
18.23%
49.10
13663%
84.34p.20
70.20%
+Synthetic-targeted + Adapter
61.75p.58
10.93%
54.962
(3.63%
81.48p,38
33200%
+ -Loss-quide--trrgeeed + Adapter*
60.95p.64
22.21%
55.02
33.52%
82.27p.5v
22.66%
+Synthetic-generall + Adapter
59.22p.89
14.99%
54.84p.44
33388%
82.28p.17
2225%
LLaMA-3B Model
64.92
65.11
97.22
Syyntheti-targeete + Adapter
61.431.91
15.37%
60.761.00 oz
16.68 %
97.65o.25
70.44%
+Synthetic-general Adapter
60.21o.11
17.26%
60 60.262.97
1 17.44%
96.32p.64
00.93%
OPT-350m Model
62.98
63.24
96.81
*Synthetictargetted + Adapter*
58.91 96
66.46%
56 56.262.39
111.04%
97.370.16
0.558%
SSyntheti-generral Adapter
60.72p.91
13.59%
60.430.34
44.44%
96.95a.01
00.14%
GPI-Neo-125m Model
Model
52.13
56.32
89.7
+Syntheti-targgeted + Adapter
51.74 44.0
10.75%
54.281,49 1.99
43.62%
22.62
1.32%
SSnntheticgenerral Adapter
49.031.3
15.95%
54.35
33.50%
88.840.37
00.96%
Table 2: Results on mitigating racial bias. "** next to the method indicates our proposed approach. We present the average bias score with standard deviations from runs paired with the % change compared to the model prior to debiasing. The first column lists the dataset and the parameter-efficient method employed. "Wiki-debiased" is baseline dataset from recent work (Xie & Lukasiewicz 2023 "Synthetic- targeeeet"teeedd" and "Synthetic-general" refer our synthetic data generated via targeted and general prompting. "Prefix", "Prompt", and "Adapter" " denote the three parameter-efficient methods. For instance, --Syntheti-ttaggeeed + Adapter" " means debiasing with synthetic data from targeted prompting using the adapter tune method. For both CrowS-Pairs and StereoSet datasets, a score closer to 50 (SS) optimal, reflecting less bias. For the Language Model Score (LMS), a higher score is indicative of enhanced language capabilities. The positive direction of change is denoted in blue, while the negative in red. The best score under each metric marked in bold and underscored.
synthetic targeted approach secures the highest score on the GPT-2 model. For BERT, while our approach is outperformed by the baselines, the difference remains within 3.5%.
For the other models in the GPT-2 family, both targeted and general prompting strategies mitigate bias across both metrics, achieving an average reduction of 5.7% for targeted debiasing and 5.4% for general debiasing. Meanwhile, language ability is well-preserved: it is either slightly improved (approximately 0.5%) minimally diminished (less than 1.3%).
Mitigating Religious Bias: As seen in Table 3 our synthetic data outperforms all other methods the baseline for the GPT-2 model. While it slightly underperforms in LMS, the difference is marginal, at under 2%. For the BERT model, with the incorporation of loss-guided prompting, our synthetic data achieves the best results compared all other methods in the baseline. In terms of LMS, the discrepancy is less than 2.5%.
For the other models in the GPT-2 family, general debiasing proves highly effective, yielding an average bias reduction of 7.2%. However, targeted debiasing is less effective, achieving an average reduction of only 0.7%. In terms of LMS, is well preserved, exhibiting variation of only 1.0% compared to the original LMS.
Mitigating Gender Bias:
Our approach effectively reduces gender bias (Table 5) On GPT- 2, our targeted data achieves the best SS on Stereoset, and our general data outperforms the baseline in the average SS score. In the case of BERT, although we did not surpass the baseline, with the implementation of loss-guided prompting, we still achieved an average bias reduction of 3.9% in loss-guided targeted debiasing and 2.6% in general debiasing. For the LMS, the difference is around 2.5%.

---

Published as a conference paper at COLM 2024
Religious Bias
CrowS-Pairs
Change
StereoSet
Change
LMS
Change
GPT-2 Model
62.86
63.26
91.01
Wikiidebiased Prefix
60.95p.60
33.03%
65.160.56
13.00%
90.95p.03
%
+ Wiki-debiased Prompt
58.29
17.27%
64.89
12.57%
90.68a11
1..36%
Wikiidebiased Adapter
62.
11.21%
62.05
11.92%
90.3
J0.77%
Synttheti-targeeeed Adapter
57.78
18.09%
59.72
45.58%
89.35
01.83%
-Synthetic-geners + Adapter
58.731.98
46.35%
62.440.24
11.29%%
88.740.43
42.49%
BERT Model
62.86
59.77
84.17
Wiikideebiseed Prefix
72.761.55
115.66%
60.6
1..0%
85.42p.09
1.48%
Wiki-debiased Prompt
83.05
132.08%
60.07 1.12
10.50%
83.80a.58
10.44%
Wikiidebiased Adapter
68.004,33
18.18%
58. 18.9
11.40%
84,45
11.33%
taaneetti-targeete + Adapter
62.86u.96
10.00%
61 4933
12.87%
82. 82.480.44
12.01%
-Loss-guided-aan targeted Adapter"
59.05 14
44.63%
58.782.93
11.66%
82.34o
42.17%
+Synthetic-genera + Adapter"
59.361.29
15.57%
59.4 59.440.75
10.55%
82.280.17
42.24%
LLaMA-3B Model
75.24
63.69
97.22
*Syntheti-tarrgeeed Adapter
73.02
2295%%
61.640.99
33222%
97.811.32
10.61%
+Synthetic-general + Adapter 63.81
115.19%
60.521.39
44.98%
96.32
10.93%
OPT-350M Model
59.05
64.62
96.81
+Synthetic-targetee Adapter
62.22, 98
15.37%
63.80
11.27%
97. 97.39p.2 0.26
10.60%
-Synthetic-generall + Adapter
57.78g.55
12.15%
62.151.41
33.82%
96.95a.0
10.14%
GPT-Neo-125M Model
55.24
62.72
89.7
-Syntheticttargeted Adapter
55.561,4
10.57%
60.97
22.99%
89.19n.3
10.57%
+Synthetic-general + Adapter
48, 28.899220
111.5%
$9.188.39
55.64%
88.84b.19
%
Table 3: Results on mitigating bias around Religion. "*" next to the method indicates our proposed approach. The terminologies and definitions follow those in Table
Our method is also highly effective in reducing gender bias in other models in the GPT-2 family. We achieve an average reduction of 7.5% with targeted debiasing and 5.8% with general debiasing. The LMS varies within margin of 1.0% compared to the original model.
5.1 General Conclusion for Results
Debiasing is Effective: Across all three categories of bias, our synthetic data, generated through both targeted, general, and loss-guided prompting, has demonstrated effective- ness under different metrics. On GPT-2, our targeted debiasing approach reduced social bias by an average of 10.2% on StereoSet and 7.9% on CrowS-Pairs, while general debiasing achieved reductions of 5.3% and 5.1%. These figures surpass baseline, which achieved average reductions of 2.5% on StereoSet and 2.2% on CrowS-Pairs. For BERT, our general debiasing approach reduced biases by 1.8% and 4.9%, exceeding existing methods with 1.6% and 3.2% reductions. However, targeted debiasing was less effective for BERT, showing no improvement on StereoSet and a 1.6% reduction on CrowS-Pairs. We addressed this by introducing a loss-guided targeted approach for BERT, enhancing results to 2.1% on StereoSet and 4.5% on CrowS-Pairs, thereby surpassing the baseline.
We demonstrated broad generalizability in bias reduc- tion across various GPT family models, including LLaMA-3B, OPT-350m, and GPT--ee- 125m, across three bias categories. For LLaMA-3B, bias was reduced by 7.1% and 6.8% using targeted and general strategies, respectively, on StereoSet, and by 6.2% and 9.7% CrowS-Pairs. On OPT-350m, reductions were 5.3% and 4.6% on StereoSet, and 3.3% and 4.1% CrowS-Pairs. GPT-Neo-125m showed decreases of 4.9% and 6.0% on StereoSet, and 1.0% and 5.7% on CrowS-Pairs.
Results Generalize Across LLMs:
Targeted Prompting Usually More Effective: Targeted prompting is more effective than general prompting in most cases. This is in line with our expectations that more prior knowl- edge leads to more robust debiasing. On the other hand, general debiasing compromises bit effectiveness in exchange for a broader range of bias mitigation.
Debiasing & Language Capability Trade-off: A noticeable trade-off emerges between language proficiency and bias mitigation when working with the BERT model. Although this trade-off was reduced through loss-guided prompting, it still presents an important focus of future exploration.
8

---

Published as a conference paper at COLM 2024
Evaluator Model
Synthetic Data
Mean Score (SD)
Top 95%
General
0.0084 0.0538
0.0219
ToxicBERT
Gender
0,0289 = 0.1125
0.1302
(toxicity)
Race
0,0085 = 0.0370
0.0314
Religion
0.0115 + 0.0608
0.0242
General
0.0081 = 0.0404
0.0000
VaderSentiment
Gender
0,0318 = 0,0862
0.2421
(negatioe component)
Race
0.0039 = 0.0277
0.0000
Religion
0.0049 + 0.0350
0.0000
Table 4: Toxicity and sentiment analysis of our synthetic data. The mean score represents the average toxicity or negative sentiment for all generated sentences across 3 runs in a given category with standard deviation (SD). The top 95% is the top percentile of the highest-scoring generations. The scores are from 0.0 to 1.0, where 1.0 is the highest toxicity negative sentiment.
Debiasing is Efficient: Training costs both in terms of time and memory are substan- tially reduced. With smaller dataset than the baselines, expedite the training process by approximately a factor of 60. We frequently secure results that match or surpass the baselines and original models in terms of bias mitigation and language ability.
6 Synthetic Dataset Analysis
Dataset Similarity: natural concern arises that ChatGPT may know and merely repro- duce the original test sets. We analyzed the similarity between the generated synthetic data and the test set. We compared the original StereoSet test set, the StereoSet development set, different dataset,
synthetic dataset, and another StereoSet development set for various bias categories to check the uniqueness of our synthetic data. Table 7 in the Appendix reveals that for our synthetic dataset, the similarity matches that of a different dataset. For the targeted synthetic dataset, there is a pronounced similarity in terms of social group terms. This is anticipated because generating an extensive list of corresponding social group terms inevitably results in numerous overlaps and analogous terms. The authors of StereoSet manually ensured that the development and test sets did not share the same social group terms. We refrained from doing this to avoid referring to the test set.
To further ensure our synthetic data is not overfitting to the existing Unseen Biases:
datasets, we use BiasTestGPT (Kocielnik al. 2023b), which generates varied test sentences for different social categories and attributes through ChatGPT. While this dataset uses ChatGPT for sentence generation, the crucial social group and attribute terms defining bias categories are taken from psychology-backer We examine these biases for GPT-2 and BERT respectively (Table 6 Appendix 14). For GPT-2, our debiasing effectively mitigates bias in variety of categories including similar, intersectional, and less related categories. In the case of BERT, we observe a clear trade-off between language ability and bias mitigation, which aligns with our previous experiments.
studies from Guo & Caliskan (0021).
Sentiment Toxicity Analysis: We performed additional analysis of the potential toxicity and negative sentiment using unbiased ToxicBERT (Hanu & Unitary team 2020) and VaderSentiment (Hutto & Gilbert 2014). Table showws that the scores for our synthetic data are very low, far below the 0.5 threshold for considering text toxic (on a scale from 0.0 to 1.0). This is even for the top 95% percentile of highest-scoring generations. A manual inspection of the 55 sentences (out of all 6k generated) that scored above 0.5 any of the dimensions revealed that toxicity detectors may be overly sensitive to certain keywords and contexts that, while potentially sensitive, are not inherently negative. Terms like "arrogant" or "angry," " or references to specific nationalities or religious texts (e.g., "Serbian," "African," "the Analects of Confucianism"), seem to trigger automated flags, despite being used in neutral or positive context. Similarly, unique combinations of roles and attributes meant to counter stereotypes seem misinterpreted (e.g., "garbage collector and a specialist in hero-dynamics") Further details and scores for other dimensions of disruptive contents are presented in Table in Appendix 20

---

Published as a conference paper at COLM 2024
7 Discussion
We introduced synthetic data generation via targeted and general prompting to debias Large Language Models (LLMs). Our findings offer several avenues for deeper exploration.
Efficacy of Prompting Strategies: Our methodologies targeee versus general prompt- -ngy in their approach and effectiveness across models. Targeted Prompting provides specificity in debiasing certain categories, while General Prompting offers a broader spectrum of bias mitigation. Notably, the effectiveness of these strategies demonstrated variation across models, such as GPT-2 and BERT, and different bias categories. One potential expla- nation is the difference in model architectures, affecting how each processes training data. Another reason could be the variance training data, where different datasets or prepro- cessing methods influence the model's behavior. Finally, the specificity of bias categories might play a role, with targeted prompting being more effective for well-defined biases and general prompting for more complex or subtle biases. Further investigation is needed here.
Understanding Trade-offs: We observe a trade-off between language capability and bias mitigation, particularly pronounced in the BERT model (see graph in Appendix 15). This might be because the synthetic data is generated by ChatGPT, which significantly differs from BERT. We generate more in-distribution data through loss-guided prompting, which mitigates the issue, supporting the hypothesis. Nevertheless, the trade-off between the debiasing performance and the language ability is a fundamental problem (French 1999). Understanding this trade-off becomes pivotal in different use cases and applications.
Evaluating Synthetic Data's Universality: Our similarity analysis underscores the unique- ness of our synthetic data, ensuring it isn't merely reproduction of known datasets. Some robustness against different biases in another dataset BiasTestGri suggests broader appli- cability. This is particularly relevant in an ever-evolving societal landscape with shifting norms and biases (Linegar et al. 2023 Kocielnik et al. 2023a). We note that further explo- ration of prompting strategies should investigate a good trade-off between prior knowledge of bias and the universality of the generated debiasing data.
Limitations
Our evaluation primarily relies on benchmarks and datasets with North American English focus, which may not fully represent global biases. The effectiveness of our debiasing might vary in tasks outside our testing scenarios. There's s also concern that ChatGPTTs exposure to test sets could have impacted our synthetic datasets (Prabhumoye et al. 2021). We investigated this possibility by checking if the synthetic data merely replicates known datasets and by experimenting with a newer dataset BiasTestGPT. Nevertheless, alignment with test sets may still exist. Moreover, the dynamic nature of societal biases, which continually evolve, may require updates of our datasets. Our focus on explicit biases may overlook subtler ones (Goethals et al., 2024).. This emphasizes the need for careful interpretation of our results and continuous improvement in debiasing approaches.
Conclusion
This paper presents two new methods for generating synthetic data to reduce social bias in LLMs more efficiently: general and targeted prompting. These methods outperform the recent work using parameter-efficient debiasing in bias mitigation and training efficiency. They also preserve language model capabilities. Our work highlights the potential of synthetic data in making LLMss fairer and suggests future research directions, including improving synthetic data generation, applying our approach to other domains such as vision, and exploring its broader applications beyond fairness.
10 Acknowledgements
We thank Caltech SURF program and Carleton's Wiebolt Endowed Internship Fund for contributing to the funding of this project. Anima Anandkumar is Bren Professor at Caltech. This material is based upon work supported by the National Science Foundation under Grant # 2030859 to the Computing Research Association for the CIFellows Project
10

---

Published as a conference paper at COLM 2024
References
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pp. 254-264. Auerbach Publications, 2022.
Joachim Baumann, Alessandro Castelnovo, Riccardo Crupi, Nicole Inverardi, and Daniele Regoli. Bias on demand: A modelling framework that generates synthetic data with bias. In Proceedings of the 2023 ACM Conference Fairness, Accountability, and Trans- parency, FAceT '23, pp. 1002-1013, New York, NY, USA, 2023. Association for Com- URL https:
puting Machinery. ISBN 9798400701924. doi: 04114453599003..0
//doi.orgh org/10.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT T '21, pp. 610-623, New York, NY, USA, 2021a. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/34421885 01115//342218.4445922 38.34222... URL http::t://....rrg/////// 144404411114444222844444452222
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021b. doi: 10.1145/ 10.1145/34421m0 ////334411884459977
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join! Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ /v1 122/222.ccllooon..... URL https: http:///aaaaaanoooo.ooooooo org/ 2021 acl acl-long........... 81
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2200.06200, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and al. Language models are few-shot learners, Jul 2020. URL https://arxiv htg.//aaxxvv...../////// arggabs/2005 14165
Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. Medically aware gpt-3 as data generator for medical dialogue summarization. In Machine Learning for Healthcare Conference, pp- 354-372. PMLR, 2021.
University of Porto CRACS-INESC TEC and University of Porto Faculty of Sciences. Survey on synthetic data generation, evaluation methods and gans. Mathematiccs 10(15):2733, 2022.
Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. URL URR_https://acclaahholloo
org/2022. naacl-main 122/
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. URL https
//aclanthology.oom org./1991//11.,
Cyrus DiCiccio, Brian Hsu, Yinyin Yu, Preetam Nandy, and Kinjal Basu. Detection and mitigation of algorithmic bias via predictive parity. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAceT '23, pp. 1801-1816, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701924. doi: 10.1145 01445//553013.3544111
URL htt.lttppp///6....0/000111////// 3594117
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, and et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, Mar 2022. URL https://arxivs wwtpr/wwwor//bb//220.3...
11

---

Published as a conference paper at COLM 2024
Zahra Fatemi, Chen Xing, Wenhao Liu, and Caiming Xiong. Improving gender fairness of pre-trained language models without catastrophic forgetting, Jun 2023. URL https //arxivv org/ absa 21100.53667
Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. First Monday, November 2023. ISSN 1396-0466. doi: 10.5210 10.5210/fm.v28i1 ftt.//....211111 www.v12.13..... 1.13346.
URL http: doi. wttp:/dii...........///.. httg:/10.5210//100 5210/fm. v28111 13346.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive
sciences, 3(4) 1999.
Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. Targeting the source: Selective data curation for debiasing nlp models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 206-276222 294.. Springer, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of 2020.
diverse text for language modeling. arxiv preprint arXi::2101.0000
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. doi: 10.18653/v1/2020. findings-emnlp.300
Sofie Goethals, Toon Calders, and David Martens. Beyond accuracy-fairness: Stop evaluating bias mitigation methods solely on between-group metrics. arXiv preprint arXiv:2401.13391 2024.
Wei Guo and Aylin Caliskan. Detecting emergent intersectional biases: Contextualized word embeddings contain distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 122-133, 2021.
Xu Guo and Yiqiang Chen. Generative ai for synthetic data generation: Methods, challenges and the future, 2024.
org/ httpg//bs/2440..... httpp////aarai...rr////////....0 04190
Laura Hanu and Unitary team. Detoxify. Github. htttp://github.... hetpm//unuaaaaeetttooooo 2020.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor
and Graham Neubig, Towards a unified view of parameter-efficient transfer learning, Feb 2022. URL https
//arxiv. wwwg//org/aaaoo abpp//0/..11.... 04366
Zexue He, Marco Tulio Ribeiro, and Fereshte Khani. Targeted data generation: Finding and fixing model weaknesses, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, and et al. Training compute-optimal large language models, Mar 2022. URL httts::/erxxvv.org/ absgg//bs//// wwwprg//bs/22.....ee 25552 15556
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Paraieetr-efffcceent transfer learning for nlp. In International Conference on Machine Learning, pp. 1790-2799 PMLR, 2019.
Clayton Hutto and Eric Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pp. 216-225, 2014.
Nishtha Jain, Maja Popović, Declan Groves, and Lucia Specia. Leveraging pre-trained language models for gender debiasing. 2022.
12

---

Published as a conference paper at COLM 2024
Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Proceedings of Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021.
Rafal Kocielnik, Sara Kangaslahti, Shrimai Prabhumoye, Meena Hari, Michael Alvarez, and Anima Anandkumar. Can you label less by using out-of-domain data? active & transfer learning with few-shot instructions. In Transfer Learning for Natural Language Processing Workshop, PP. 22-32. PMLR, 2023a.
Rafal Kocielnik, Shrimai Prabhumoye, Vivian Zhang, Roy Jiang, R. Michael Alvarez, and Anima Anandkumar. Biastestgpt: Using chatgpt for social bias testing of language models, 2023b.
Tai Le Quy, Arjun Roy, Vasileios losifidis, Wenbin Zhang, and Eirini Ntoutsi. A survey datasets for fairness-aware machine learning. WIREs Data Mining and Knowledge Discovery, 12(3), March 2022. ISSN 1942-4795. doi: 1..000/wwim.1i. URL http
//dx www.doi..rgg/ae 111/111...000/// 1002/widm. 1452
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 20.1863/vv1/202/ 21263///1222...nmmpmmam 243. https://
org/ 2021. emnne-maaaee maip.mmmaii.... 243
org/2021.
Jonathan Li, Rohan Bhambhoria, and Xiaodan Zhu. Parameter-efficient legal domain adap- tation. In Proceedings of the Natural Legal Language Processing Workshop 2022, pp. 119-129, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computa- tional Linguistics. doi: 10.18653/ (118653vv1/20222 v1/2022.nlll-11.. 2022. m11p-1.10
URL https:://aclaattholllgg org/
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 15582-4997 Online, August 2021. Association for Computational Linguistics. doi: 10.18653 /v/055//1//00 http:///121222lllooog....o/
RRLLtttss//aclattoollle
wtt/ mocc-llon.... 353
Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on faimess in large language models, 2023.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. Towards debiasing sentence representations, 2020a. URL
https://arxiv. wttp:/ivw..gggaaa////o//// 08100
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. Towards debiasing sentence representations. arXiv preprint arXiv:2007.08100, 2020b.
Mitchell Linegar, Rafal Kocielnik, and Michael Alvarez. Large language models and political science. Frontiers in Political Science, 5:1257092, 2023.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data for language models, 2024. URL jtttpss/////xxx org/ abs/
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint ar xrxi:22308.087 2023.
Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. Powertransformer: Unsuper- vised controllable revision for biased language correction. arviv preprint arXiv:2010.13816, 2020.
13

---

Published as a conference paper at COLM 2024
Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. An empirical survey of the effective- ness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annua Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1898,1198, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 00t6663/v1/2222acllloo// 132. URL httphhtts::/ leatts:://aclaathollgy) hwwphhhoolgooooog2/000 wttrg/2022.acl1100 132
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Pp. 5356 5371, Online, August 2021. Association for Computational Linguistics. doi: (0.11663/1122222.ccc-l 416.
URL https://
aclattiology.orr//001.... ael-1acc-loogg 416
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. Crows-pairs: A
challenge dataset for measuring social biases in masked language models, Sep 2020. URL
httpa///araav.ooraaab///////// 00133
Daisuke Oba, Masahiro Kaneko, and Danushka Bollegala, In-contextual bias suppression for large language models. Xiv preprint arXin:2309.07251, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and et al. Training language models to follow instructions with human feedback, Mar 2022. URL h/txii//arxiv org/abs/2203 22111
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. AdapterHub: A framework for adapt- ing transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 46-54, Online, October 2020. Associ- ation for Computational Linguistics. doi: (1116////112220emm///// httpp::/aclannhoooogy wwwp:ggg..rrr///// org/2020.0000. emnle-demos .7
URL
Shrimai Prabhumoye, Rafal Kocielnik, Mohammad Shoeybi, Anima Anandkumar, and Bryan Catanzaro. Few-shot instruction prompts for pretrained language models to detect social biases. arXiv preprint arXiv:2112.07868
17866, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical Report, 2019.
Brianna Richardson, Prasanna Sattigeri, Dennis Wei, Karthikeyan Natesan Ramamurthy, Kush Varshney, Amit Dhurandhar, and Juan E. Gilbert. Add-remove-or-rellee
Practitioner-frieddly bias mitigation via influential fairness. In Proceedings of the 2023 ACM Conference Fairness, Accountability, and Transparency, FAccT '23, pp. 736-752, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701924, doi: 10.145//599013.599499. URL.hbtpee//oo...rg///...11 05111455/55550033355499999999999
Sahana Timo Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A pro- posal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408-1424, 2021a.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp, 2021b. URL 2103.00453
/bs//
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Harini Suresh and John V. Guttag. A framework for understanding unintended conse- quences of machine learning. CoRR, abs/ 1900. 2019. URL 19002
14

---

Published as a conference paper at COLM 2024
Ewoenam Tokpo, Pieter Delobelle, Bettina Berendt, and Toon Calders. How far can it go?: On intrinsic gender bias mitigation for text classification, Jan 2023. URL https //arxiv.
12855
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar ar11::2302. 2023.
Veniamin Veselovsky, Manoel Horta Ribeiro, Akhil Arora, Martin Josifoski, Ashton Ander- son, and Robert West. Generating faithful synthetic data with large language models: A case study in computational social science. ar Xiv preprint ar Xrxx:22300.11 15041,, 2023.
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. Exploring the limits of domain- adaptive training for detoxifying large-scale languuu models. Advances in Neural Informa- fion Processing Systems, 35:35811-35824, 2022.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: survey, 2023.
org wttpor/g/ab////////e///////// 2307. 12966
Kellie Webster, Xuezhi Wang, lan Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. Xiv preprint arXiv:2010.06032, 2020.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, and al. Taxonomy of risks posed by language models. 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022. doi: 10.1145/3531146.353333
Zhongbin Xie and Thomas Lukasiewicz. An empirical analysis of parameter-efficien methods for debiasing pre-trained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15730- 15745, 2023.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 629-634, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistcss. doi: 10.18653/v1/ wtw.aaatttooootooooo/oo-/-111111
N19-1064. URL
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings, 2019b.
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data
augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp.
1651-1661, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 00.18633vv1/P19 U http htt.tggg///cc.laoooooooor///////
15

---

Published as a conference paper at COLM 2024
11 Appendix- Gender Bias Mitigation Table
Gender Bias
CrowS-Pairs
Change
StereoSet
Change
LMS
Change
GPT-2 Model
56.87
62.65
91.01
+Wiki-deblased Prefix
54.73
13.76%
61,35u.0
42.08%
91.24%..r
10.25%
Wiiki-debiased Prompt
54.12
J4.84%
61.300.3
42.15%
91.37p.08
10.40%
Wikiidebbased Adapter
52.29
18.05%
60.33
13.71%
90.876.11
10.15%
Synthetic-targeted + Adapter
53.31
16.24%
59.28 59.28p.75
(5.37%
90.82..39
%
Synthetie-general Adapter
52.42
47.79%
59.77086
44.58%
88.74o.43
42.49%
BERT Model
57.25
60.28
84.17
Wikiidebiased Prefix
53.59p.19
16.39%
57 82..66
44.09%
84.750.15
10.69%
+ Wiki-debiased Prompt
57.561.41
10.54%
58.070.00
13.61%
84.710.16
10.64%
Wiiki-debassdd Adapter
51.680.52
19.70%
56 p..0
17.03%
84.97p.14
10.95%
+Synthetictarggeeed + Adapter*
54.96u.38
14.01%
60.72..50
f0.73%
79.20,.zz 2.27
15.89%
+ +Loss LLos-guidec--argeeee Adapter"
53.44
J6.66%
59. 55..6
11.21%%
82.00
12.58%
-Synthetic-generall Adapter
54.83p.44
14.17%
59.70n.4
10.96%
82.28
42.24%
LLaMA-3B Model
65.27
68.62
97.22
-Sygteeti-targeeee + Adapter
58 58.523.82
110.44%
60.883.29
11.27%%
97.27p.38
10.05%
-Synthetic-eeneral Adapter"
60.944.52
J6.63%
63.22
47.87%
96.32
10.93%
OPT-350M Model
60.69
67.35
96.81
Syntheeti-targgted + Adapter
55.34 15
18.2%%
62.90,.61
33.6%
97.37p.us
10.58%
+Synthetic-general + Adapter*
56.74g.44
66.51%
63.641.3
55.51%
96.95cun
10.14%
GPT-Neo-125M Model
54.96
63.74
89.7
tSyntheti-targgeedd + Adapter
53.44o.77
12.77%
58 58.499.98
18.24%
89.1 89.18p.04
10.60%
-Synthetic-generte + Adapter*
55.22
10.47%
58.00 58.04p.16
(8.94%
88.840.19
%
Table 5: Results mitigating gender bias. The terminologies and definitions follow those in Table
12
Appendix Targeted and General Prompting Algorithm
Algorithm
Debiasing Data Generation for Targeted and General Prompting
Input: Bias category N (optional for General Prromping), Generator Model Mg. Term generation instruction i Targeted Prompting instruction up General Prompting instruction
gP
Output: Debiasing Sentences
1: if Targeted Prompting (itp) then
2: <- GENERATE'TRMSS Mgrit)
3: 5 GENERATESENTENCES// Mg. ipp)
4: else if General Prompting (igp) then
5: S GENERATESSENEENCIS(MM,
6: end
7: Reformat S: Sentence (S), Term (T), Attribute (A)
8: return S
16

---

Published as a conference paper at COLM 2024
13
Appendix - Loss Guided Prompting Algorithm
Algorithm 2 Loss-Guided Debiasing Data Generation
Input: Debiasing
sentences from Targeted Prompting Stp, Generator Model Mg, Tested Model M,, Loss-Guided Prompting instruction igp Number loss-guided examples k Output: In-Distribution Debiasing Sentences Sip
Ltp f- { 2: for E Sip do
3:
It EVALUATELOSSS(s, M,)
4: Append tuple (s,1) to Ltp
5: end for
6: Ltp f- SELECCTHCHHLOLLLSS(pppp k)
7: SIp f- GENERAEESENTENCISS (Ltp, Mg iggp)
8: Reformat Sip: Sentence (S), Term (T), Attribute (A)
9: return Sip
17

---

Published as a conference paper at COLM 2024
14
Appendix Result Table for Testing on BiasTestGPT
Model
Debiasing Category
Original SS
Gen. Debias
Tgt. Debias
Profession
Gender
73.75
66.143.09
64.460.69
Profession Math/
57.14
63.89
64.181.37
GPT-2
Mex.Fem<>>> < Eur.Malure /emergent
60.42
53.060.64
52.640.24
Young <> Old
55.94
52.920.48
52.500.62
Profession Gender
66.76
68.88p.14
68.60g.28
Profession <> Math/ Arts
53.51
50.440.44
51.610.50
BERT
Gender<>>cience> Arrs Arts
63.39
65.030.68
64.440.68
Gender<>Carrer//BBa
55.03
55.551.01
55.130.18
Table De-biased Model Test Results Using BiasTestGPT Data. In this table, "<>" denotes bias between the chosen social categories. For instance, Profession <> Gender signifies bias between professional and gender terms. Meex.Fem<<Eur.Maan (ameee/tm represents an intersectional category, indicating bias related to both race and gender. We employ synthetic data through general prompting for general de-biasing and synthetic gender data through targeted prompting for targeted de-biasing. The scores the table SS, with being the ideal score.
15 Appendix- Dataset Comparison Table
Shared Terms (%)
Cos Similarity (%)
Dataset
Target
Attribute
Pairs
Sentence
Target
Attribute
Gender
StereoSet Gender Dev
0.0
18.6
0.0
99.5
82.7
98.0
CrowS-Pairs Gender
76.7
23.3
4.5
96.1
Synthetic-targeled*
68.91.6
10.51.0
0.8p.4
96.10.7
92.8p.1
85.22.
Synthetic-general"
8.95.7
3.80,2
0.1o.1
90.70.6 91.8
56.44.2 26.4
71.9
StereoSet Religion Dev
0.0
3.1%
0.0
90.3
Race
StereoSet Race Dev
0.0
22.4
0.0
99.9
93.9
99.3
CrowS-Pairs Race
2.7
21.4
0.7
94.2
Synthetic-targetedd
19.50.0
4.5p.5
0.0p.0
94.106
88.302
89.203
Synthetic-general
2.42.7
3.90.
0.0g.0
93.1 98.1
54.93.0 48.1
84.912
StereoSet Religion Dev
0.0
3.1
0.0
92.9
Religion
StereoSet Religion Dev
0.0
11.6
0.0
99.2
85.9
95.9
CrowS-Pairs Religion
22.2
17.4
0.9
93.9
Synthetic-targeted* Synthetic-generae
55.60.0
8.80.7
0.60.2
94.30.4
87.90.1
84.61.3
3.75.2
2.60.7
0.0p.0
94.2
40.50
72.0
StereoSet Profession Dev
0.0
21.6
0.0
97.1
32.2
93.0
Table 7: Dataset Similarity Analysis. In this table, compare both our synthetic data and other datasets with the test set. We provide the percentages of shared target terms, attribute terms, and shared pairs. Additionally, we evaluate the cosine similarities of the average word and sentence embeddings. The largest overlap between debising set and test set is underlined." denotes our generated dynthetic data.
18

---

Published as a conference paper at COLM 2024
16
Appendix- Generated Training Prompts Examples
Debiasing
Bias Type
Sentence
Subject Word
Attribute Word
Strategy
Targeted
Gender Gender 2
"The woman was remarkably stoic." " "People love the Godfather not for his power, but for his genuine kindness." "Contrary to belief, the Groomsman isn't careless; he's quite meticulous." " "The Scottish community has been a beacon of innovation various fields."
"Woman" "Godfather"
"Stoic" "Kindness"
Prompting
Gender
"Groomsman"
"Meticulous"
Race
"Scottish"
"Innovation"
Race 2
" "African architects are renowned for "African" blending traditional and contempo- rary designs."
"Renowned"
Race 3
"In literature, Chinese philosophers have always explored the enigmatic nature existence."
"Chinese"
"Enigmatic"
Religion
"Many students find the Quran to be "Quran" enlightening."
"Enlightening"
Religion
"Many Torah scholars have tributed immensely the fields ence."
"Torah"
"Science"
Religion
"Confucianism places great impor- "Confucianism" tance on family ties and respecting el- ders." "
"family"
General Prompting
"The male nurse was praised for his exceptional care and sensitivity."
"Male"
"Sensitivity"
General 1
"The football player wrote an award- winning poetry book."
"Football Player"
"Poetry"
"Her father took the day off care for his child, showcasing his nurturing side."
"Father"
"Nurturing"
General 2
"Just because she's cheerleader doesn't mean she isn't top student her calculus class."
"Cheerleader"
"Calculus"
"In many communities, it's the men who are the primary gossipers." "She found that the skateboarder was also connoisseur of classical music."
"Men"
"Gossipers"
General 3
"Skateboarder"
"Classical"
Table 8: This table showcases example prompts. For Targeted Prompting, we provide example for each generation of every category. For General Prompting, we provide two examples for each generation. Each example includes the sentence, subject word, and attribute word.
17 Appendix- ChatGPT prompts
Prompts for Targeted Term Generation:
The following link is the conversation with
ChatGPT used for targeted terms generation:
Prompts for Targeted Sentence Generation: We include the sample conversations with
ChatGPT for Targeted Sentence Generation listed below:
1. Converattin for Racial Bias:
2. Sample Conversaton for Gender Bias:
19

---

Published as a conference paper at COLM 2024
3. Sample Conversation for Religious Bias:
chatt httpa////..//aeeeeee2e6226-2-660
Prompts for General Sentence Generation:
The following link is sample conversation
with ChatGPT we used for generating general de-biasing sentences:
https://chat. wwtp:iccoo/oeaa/eeeeeeeeeaee
18
Appendix- - Trade-off Graph
Model Language Capability
Debiasing Effectiveness
LMS
SS on StereoSet
SS on CrowSPairs
35
05
34
10
Exche
Spacks
Figure 5: This graph illustrates clear trade-off between the model's language capabilities and debiasing performance during training. Lowering bias language model is likely impact its general language proficiency. This represents a fundamental challenge the field of language model fairness.
20

---

Published as a conference paper at COLM 2024
19
Appendix Performance Graph for Different Data Sizes
LMS
StereoSet
CrowSPairs
Data Size
200
F
500
1000
13
Figure 6: Performance across different data sizes. The 200 data size yields minimal debiasing perfor- mance, while the 1000 data size significantly impairs the model's language capability. Thus, to achieve a balance between debiasing performance and language capability, data size of 500 is selected.
20 Appendix- - Additional Details of Toxicity and Sentiment Analysis
VaderSentiment (Hutto & Gilbert 2014 produces individual positive, negative, and neutral sentiment component scores and a compound score. We report just the values for the negative sentiment component as these the most relevant here. Aside from general Toxicity presented in Table 4. we also analyzed all the dimensions provided by ToxicBert (Hanu& & Unitary team, 2020)- Severe Toxicity, Identity Attack, Insult, Sexually Explicit, Obscene, and Threat. We present the results for these other dimensions in Table
21

---

Published as a conference paper at COLM 2024
Evaluated Dimension
Synthetic Data
Mean Score (SD)
Top 95%
General
0.0084 0.0538
0.0219
Gender
0.0289 0.1125
0,1302
Toxicity
Race
0.0085 = 0.0370
0.0314
Religion
0.0125 + 0.0608
0.0242
General
0.0002 = 0.0010
0.0005
Gender
0.0010 + 0.0071
0.0020
Obscene
Race
0.0002 + 0.0007
0.0027
Religion
0.0006 + 0.0045
0.0006
General
0.0012 - 0.0124
0.0015
Gender
0.0010 0.0049
0.0028
Identity attack
Race
0.0056 - 0.0350
0.0151
Religion
0.0023 + 0.0087
0.0108
General
0.0053 0.0498
0.0066
Gender
0.0235 0.1105
0.0855
Insult
Race
0.0018 + 0.0105
0.0049
Religion
0.0011 + 0.0060
0.0031
General
0.0001 = 0.0005
0.0002
Gender
0.0001 0.0006
0.0003
Threat
Race
0.0001 + 0.0002
0.0002
Religion
0.0001 + 0.0001
0.0003
General
0.0001 = 0.0004
0.0003
Gender
0.0002 0.0006
0.0009
Sexually Explicit
Race
0.0003 + 0.0042
0.0002
Religion
0.0090 + 0.0764
0.0003
Table 9: Analysis using additional dimensions of offensive content provided by ToxicBert (Hanu
Unitary team 2020
21 Appendix- - Subject words for Targeted Prompting Data
22

---

Published as a conference paper at COLM 2024
Table 10: Subject Words for Gender Bias Data through largeted Prompting
Gender Generation
woman(36]
male(27)
b2y(26)
girl(21)
temale(20)
father(19)
man(18)
mother(18)
sister(17)
brother(17)
grandfather(15)
grandmother(11)
lady(12)
gentleman(11
wife(9)
son(7)
uncle(6)
lord(5)
empress(5)
daughter(7)
mister(4)
sir(5)
mrs-(4)
miss(4)
patriarch(4)
knight(4)
baron(á)
queen(6)
madlame(4)
king(7)
prince(3)
actress(3)
husband
young lady(3)
guy(6)
lad(6)
emperor(3)
dame(3)
nephew(4)
dukke(3
bride(4)
maiden(3)
matron(3)
(3)
mom(3)
dad(()
gal(d)
mr:(3)
duchess(3)
businesswoman(2)
businessman(22
granddaughter(2)
sister-in-law(3)
lass(2)
aunt(3)
matriarch(2)
maid(2)
grandson(2)
papa(2)
niece(3)
missus(2)
madam(1)
mum(1)
gent(l)
young man(1)
groom(2)
boother-in-laww2.
soldier(1)
ms.(1)
masculine(1)
boyfriend(1)
daughter-in-law(]]
count(l)
chap(1)
youth(I)
sire(1)
heir(1)
junior(1)
mother-in-law(1)
she(1)
princess(1)
heroine(1)
hostese(1)
bachelorette(1)
belle(1)
mummy(1)
bridesmaid(1)
mama(1)
bestie(l)
hero(l)
vixen(()
squire(1)
damsel(1)
bachelor(1)
countess(1)
maternal(1)
elder(1)
groomsman(1)
host(T)
heiress(1)
protector(
buddy(1)
baroness(1)
godfather(1)
ma(1)
Gender Generation
youth(12)
(1)
knight(12)
king(10)
uncle(10)
lad(10)
duchess(9)
baron(9)
bride(9)
nephew(9 (9)
protector(9)
belle(8)
chap(8)
lady(8)
gentleman(8)
aunt(7)
countess(7)
groom(7)
empress(7)
mother(7)
prince(7)
mister(7)
godfather(6)
sir(6)
heroine(6)
duke(6)
boy(6)
queen(6)
maid(6)
sire(6)
buddy(6)
maternal(6)
bachelorette(6)
maiden(5)
groomsman(5)
som(5)
gal(5)
heir(5)
patriarch(5)
missus(S)
bachelor(5)
matron(5)
damsel(5)
count(5)
princess(5)
hero(5)
junior(5)
mummy(5)
best m5)(5)
daughter(5)
niece(5)
sister-in-lawit
dame(3)
hostess(5)
son-in-law(4)
madame(4)
mother-in-law(4)
bridesmaid(4)
squire(4)
stag(4)
vixen(4)
daughter-in-law(f)
baroness(4)
lass(4)
male(4)
host(4)
matriarch(4)
father(4)
brother-in-law(3
Mr(3)
master(3)
Miss(3)
elder(3)
girlfriend(3)
b3yfriend
bestie(3)
wife(3)
sister(3)
man(3)
brother(3)
guddess(3)
motherhood(()
grandson(3)
girl(3)
(3)
mademoiselle(2)
mom(2)
Pa(2)
granddaughter(2)
husband(2)
madam(2)
grandfather(2)
grandmother((2)
godmother(2)
mistress(1)
Ma(1)
Marn((1
dad(1)
femalel(1
Mrs.(1
father-in-law(1)
feminine(1)
guy(l)
papa(1)
he(1)
she(1)
Gender Generation
baron(10)
lad(10)
uncle(10)
nephew(10)
lord
aunt(9)
king(9)
knight(9)
protector(9)
belle(9)
lady(8)
bride(8)
matron(8)
gentleman(8)
bachelor(8)
godfather(8)
duchess(8)
princess(8)
chap(8)
youth(8)
queen(7)
hero(7)
groomsman(7)
matriarch(7)
empress(7)
hostess(7)
squire(7)
heroine(7)
mother(6)
sister(6)
buddy(6)
dame(6)
duke(6)
daughter-in-law66
countess(6)
prince(6)
boy(5)
brother(S)
madame(5))
(iece(5)
maid(5)
groom(5)
motherhood(3)
elder(5)
master(3)
sister-in-law(5)
mother-in-law((5)
damsel(5)
vixen(5)
man(5)
father(4)
daughter(4)
grandfather(4)
junior(4)
(4ag(4)
bachelorette(4)
bestie(4)
sir(4)
son(4)
buy (rieend()))
count(4)
heir(4)
host(4)
Pa(4)
gal(4)
mummy(4)
bridesmaid(4)
Miss(4)
maternal(4)
he(3)
mister(3)
girlfriend(3)
granddaughter(3)
beother-in-law(33)
sire(3)
goddess(3)
son-in-law(()
patriarch(3)
lass(3)
mom(3)
mama(3)
grandmother(3)
baroness(3)
missus(3)
grandson(3)
girl(2)
female(2)
husband(2)
Papa(2)
wife(2)
maiden(2)
g2y(2)
male(2)
man(1)
she(1)
Mrs(1)
dad(1)
feminine(1)
woman(1)
emperor(1)
godmother(1)
gentlewoman(1)
Ma(1)
(I.
22 Appendix- Subject words for General Prompting Data
23

---

Published as a conference paper at COLM 2024
Table 11: Subject Words for Racial Bias Data through largeted Prompting
Generation
arab(6)
melanesian(()
cthiopian(6)
flipino(6)
malay(6)
basquec(b)
icelander66)
dutch(6)
serbian(6)
bengali(6)
scottish(5)
turkish(5)
pppanes(55)
korcan(5)
persian(5)
stalian(5)
french(5)
native american(5))
maori(5)
ashkenazi(())
slavic(5)
thai(5)
vietnamese(55)
kurd(5)
yoruba(5)
zulu(5)
hausa(5)
somali(5)
romani(3)
catalan(5)
greek(5)
norwegian({5
finuish(3)
polish(()
hungarian(5))
kesovar(5)
armenian(5)
uzbek(5)
kyrgy2(5)
(ajik(5)
sinhalles((S)
Khmen(5)
bantu(5)
guarani(S)
quechua(S)
aymara(5)
latino(4)
latina(4)
african(4)
curopcan(4)
chincser44)
indian(4)
russian(4)
german(()
irish(4)
australian aboriginal(4)
polynesiam(4)
jewish(4)
pacific islanden(4)
berber(4)
pashtun(4)
igbu(4)
danish(4)
swiss(4)
portuguese(4)
bulgarian(4)
ukrainian(44
belarusian(4)
croatian(4)
bosmiak(4)
macedonian(4)
albaniann44)
georgian(4)
azerbaljani(E)
kazakh(4)
punjabi(4)
burmese(4)
javanese(())
sundanese(4)
malagasy(4)
maltese(4)
sami(4)
inuit(4)
sherpa(4)
yazidi(4)
hispanic(3)
sephardi(3)
baltic(3)
xhosa(3)
swedish(3)
belgian(3)
romanian(3)
moldovan(3)
tamil(3)
lao(3)
creole(3)
tatar(3)
tibetan(3)
druze(3)
sunmi(3)
ainu(3)
cromo(3)
bedouin(()
samean(3)
kikuyu(3)
white(2)
asian(2)
tuareg(2)
czsch(2)
slovak(2)
montenegrin(()
turkmen(2)
black(3)
aleut(2)
nighur(2)
maronite(2)
alawite(2)
maasai(2)
welsh(2)
chamorro(2)
mestizo(1)
bashkie(1)
nepali(1)
micronesian(1)
fijian(1)
longan(1)
hawatian(1)
latvian(1)
nenets(1)
mexican()
maldivian(1)
bosnian(1)
estonian(1)
Race Generation
ashkenazi(6)
scuttish(5) 5]
turkish(S)
latimo(5)
african(S)
european(()
jppanese(5)
(orean(5)
arab(5)
persian(3)
italian(5)
french(5)
native american(5)
maori(5)
polynesian(3)
melanesian(5)
ethiopian(5)
slavic(5)
filipin(5))
(hai
vietnamese(5)
malay(5)
berber(5)
pashtun(5)
igbo(5)
yoruba(5)
zulu(5)
somali(5)
romani())
greek(5)
norweglan(5)
finnish(5)
dutch(5)
swiss(5)
portuguese(5)
bulgarian(5)
bosniak(5)
macedonian(S))
geurgian(5)
kazakh(())
punjabi(5)
malagasy(5
bantu(5)
aymara(5)
yazidi
hispanic(44
chinese(4)
german(4)
irish(4)
sephardi(4)
pacific islander(4)
tuareg(() 4)
catalan(4)
damish(4)
icelander(4)
bolgian(4)
slovak(4)
hungarian(4)
kosovar(4)
armenian(4)
azerbaijani(4)
uzbek(4)
kyngyz(4)
tajik(4)
bengali(4)
sinhalese(4)
burmeso(4)
khmerr(4)
lao(4)
javancse(4)
sundancso(4)
maltese(4)
guarani(4)
quechua(4)
inuit(4)
bedouin(4)
chamorro(4)
ainu(4)
indlan(3)
russian(3)
australian aboriginal(3)
jewish(3)
kurd(3)
hausa(3)
swedish(3)
czech(3)
ukraiinian33
belarusian(3)
croatian(3)
serbian(3)
montenegrin(3)
albanian(3)
moldovan(3)
lamil(3)
sami(3)
hawailann(3)
tongan(3)
druze(3)
sherpa(3)
mestizo(3)
chukchi(3)
micronesian(3)
bashkin(3)
khoisann3)
fijian(3)
samoan(3)
black(2)
white(2)
asian(2)
latina())
baltie(2)
xhosa
basque(2)
polish())
romanian(2)
turkmen(2)
mansai(2)
kikuyu(2)
oroma(2)
maronite(2)
kurdish
creole(2)
tatar(2)
uighur(2)
tibetan(2)
n2pali
slawite(2)
tuvaluan(2)
welsh(1)
aleut())
mulatto(1)
chuvash(!)
shia(1)
sunni(1)
shona(1)
mandinka(1)
fulani(I)
nenets(1)
yakut(1)
icelandic(1)
mexican(1)
bosnian(1)
Generation
persiant/)
vietnamesc(()
armenian(6)
french(5)
japanese(5
scuttish(5)
african(5)
kurd(5)
italian(5)
bantu(5)
turkish(5)
ashkenazi(5)
hispanic(5)
yoruba(5)
korean((5
arab(5)
quechua(5)
romani(3)
chinese(5)
indian(5)
kazakh(5)
macedomian
bedouin(5)
azerbaijani(5)
ukrainian(5)
slavic(5)
german(5)
sherpa(5)
greek(5)
pashtun(5)
sephardi(5)
khmeer(5
swedish(5)
belarusian(5)
serbian(5)
javanese(5)
lao(5)
bosniak(()
maltese(5)
kyngyz(5)
latino(S)
ethiopian(5)
bengali(5)
thai(5)
georgian(S)
latina(5)
dutch(())
finnish(5)
sinhalesu(5))
mavri(4)
native american(4)
inuit(4)
jewish(4)
polynesian(4)
icelander(4)
bulgarian(4)
somali(4)
european(4)
pacific islander(4)
basque(4)
norweglan(4)
zulu(4)
calalan(4)
lajik(4)
maasai(4)
hawaiian())
yazidi(4)
irish(4)
chamorro(4)
kikuyu(4)
samoan(4)
polish (4)
burmese(4)
igbo(4)
belgian(4)
kosovar(4)
portuguese(4)
moldovan(4)
guarani(4)
melanesian(4)
filipino(4)
russian(4)
albanian(4)
malagasy(4)
longan(3)
aymara(3)
oromo(3)
nenets(3)
croatian(3)
malay(3)
micronesian(3)
ainu(3)
punjabi(3)
sami(3)
hausa(3)
australian aboriginal(3)
danish(3)
czecht()
khwisan(3)
wxbek(3)
sundanese(3)
druze(2)
fijian(2)
bashkir(2)
uighur(2)
tuvaluan(2)
baltic(2)
brazilian(2)
estonian(2)
creole(2)
swiss(2)
aleut(2)
montenegrin((2
black(3)
slovak(2)
turkmen(2)
tamil(2)
mestizo(2)
fulani(1)
berber(1)
chukchi(1)
tibetan(1)
icelandic())
cuban(1)
maldivian(1)
palestinian(1)
mongolian(1)
tuareg(I)
bolivian(1)
kurdish(1)
slovakian(1)
bosnian(!)
xhosa(1)
hungarian(1)
romanian(1)
mulatto(1)
chuvash(1)
white(1)
asian(1)
welsh(1)
23 Appendix- Attribute words for Targeted Prompting Data
24

---

Published as a conference paper at COLM 2024
Table 12: Subject Words for Religious Bias Data through Targeted Prompting
Keligion Generalion
analects(9)
druidry(9)
voodoo(S)
torah1")
guru granth sahib??)
shamanism/?)
sen(()
sufism(()
gospel(7)
talmud(6)
(acism(6)
baha
book of moemon(()
rastafarianism((e)
wiccaf6)
sanieria(6)
mahayanad5)
kabbalah(6)
hasicismite
yazidismi()
deismi6)
panthelsm(()
unitarianism(s)
encomocutelé)
moequel6)
church(f)
chingt6)
(cttib-i-qqdeste)
aleviem(6)
avesta(6)
shinlo(5)
candomble(s)
vajrayana(()
druzelf)
quran(5)
buddhismm()
chesstiang))
[atnismm3)
slichisss[3)
badidhds)
cathollo(5))
oethodax(5)
paganism(5)
native american (hurch(5)
falun gong(S)
dianetica(5)
thenvadd
coptic(5)
groeliciim(()
monotheienn(S)
pesshylerianiom(5
amish(5)
jehovah' wiinesses
synaggguaa()
lemple(3)
monastery(5)
ritual(5)
bekcashi(s)
agnoelicism(5)
atheism(3)
animism(S)
nichinen(5)
wahisabiem(5)
ahmadisya(5)
calvinism(())
seventh-day adventise(()
society (riends(())
universalism(?)
dualism(())
baptism(()
hindu(5)
protestant(4)
zoroasitianism())
kopki14)
lutherann(4)
pilgrimago(()
umbanda(4)
samarilnnism(4)
poly polytheism(())
manichneism(4)
amglicanism(4)
church satan(4)
tenrikyo(4)
bible(4)
mandaonnssm(4)
(slam(3)
shis(3)
quskerism(()
scientologyy))
sunnit3)
mormoniams(3
confucianismm())
upanishads(()
lutheranism(1)
paganç1)
conters(])
purarass()
tentriem(() ()
bhagavad gim(()
hore krisknall)
shaktism())
vaishnavismi
shaivismm())
sankhya()) ()
vedanta(1)
advaita(1)
rigveda(1)
samavede(l)
((harvaveda()
brahmanas((1
aranyakaw[1)
Keligion Generation
baha'l99)
candomblet9)
wiccall)
sufism(8)
jainisen())
talmudr7]
protestanti
zorvastrimisml(?)
kojikät?)
chingt?)
analects(7)
kitäb (agda=-aadass7)
voodoot?)
animiism(7)
paganismi77)
druidryc71
shamanissm(?)
church salan(?)
mosquet7))
buble(s)
hadith(6)
orthodaxié)
avestate)
taodsm(6)
(astafarianism(5)
nichiren(6)
2cc1/6)
kabbulab(6)
gospelt))
symaggggue(5
hindu(5)
quran[S)
buddhism(S)
(orah(S)
christian(5)
<ikhism([)
granth sahib(3)
(slam(5)
sunna(3)
shials5
catholie(t)
shinlo(5)
corfocianism(55)
mormonisan(S)
book mormon((5)
santeria(5)
umbandaj5)
native american church(5)
samaritanisn((5)
(enrikyo(5)
(hosvadaa(5)
mahayana([))
(ajrayana(5))
wabihuhisen(5)
shmadiyya55
coptic(5)
groeticiiaxi(5
druze(B)
aleviss((5)
beklashuls)
deisml()
polytheism(5)
universalisea())
quakerism(3)
calvinisen(S)
mecmmoclte())
severib-day adventist(5)
jehovah's (itresses(5))
schriology(()
temple(5)
church((5)
monastery(5)
(ilgrimage(S)
diuull()
mandacinism(4)
falun gong(4)
dianelics(()
basidiam(4)
athelsmt(4
pantheism(4)
monotheismm()
dualissm(4)
manicharism(4)
unitarizaisn((4)
society friends(4)
antheran(4)
anglicanisen(4)
pesbyterianise(())
amish(4)
baplism(4)
albeisi())
agrostik(1)
churdhos(1)
Keligion Generation
11oca(10)
voodoo(8)
2er//)
sulssm/8)
jainiseni))
granth sahib(?)
kabbalahi7)
gospel(?)
churchi?7)
torah(6)
talmudió)
haditht6)
laoismi6)
chinglé)
confacianism((s)
aralecisté)
book of mormon|6)
castafariznism(()
animism(6)
paganismié)
church satand6)
vojrayanaie)
hasidism(é)
coptic(ó)
gnostticismm6]
druze(6)
yazidismm(()
alevismi6)
athelsm(65
deessm())
pantheism(())
manichacism(5)
unilarianism((6)
universalism(s)
calvinism(6)
Tutherants)
presbytetanismms))
mennonite(é)
scientology(6)
temple(s)
pilgrimagee()
rttual(6)
55
budddissm(()
christian(5)
catholict
arthedou(5)
avesta(5)
Kojiki(s)
bahaa)(5)
druidry(5)
shamanism(())
candomble(3)
umbandatS)
mandacinism(5)
falun gong(5)
dianetics(3)
tennikyo
michiren(S)
theravada(5)
mahayana(()
wahhabismi33
shmadlyyal5)
bektasiui
sgnosticiom(5)
polytheism((5)
(Snotheism(()
dualism
society of Iriends(5)
anglicanism(5)
amish(()
baplism(S)
seventh-daj day adventia
synagogux(5)
mosque(.3)
monastery(3)
hindu(4)
sikhismi4) 4)
protestant(
shinto(4)
mormonisan(()
nalive american church(4)
samarilanism(4)
jehovah's witresses(4)
bible(4)
islam(3)
shia(3)
2onossitianism(3)
quakerisnu())
presbylerian(1)
24 Appendix- Attribute Depridix-Attribute words for General Prompting Data
25

---

Published as a conference paper at COLM 2024
Table 13: Subject Words for General Prompting Data
Bbbarian(5)
mun(4)
cco(4]
rapper(4)
bikeri4y
accountant(4)
lawyer(4)
bartender(4)
bodyburlder(()
punk(4)
skoteboarder(4)
desert(4)
boxer(d)
politician(()
muddel(d)
fattoos(3)
foutball playen(3)
africa(()
hijab(3)
truck driver(3)
petitet3)
monk(3)
jamitor(3)
soldiert
comedian(3)
mechanlc(3)
butcher(3)
software engineer(3)
wrestler(3)
carpenten(3)
physicist(3)
m)thematiciaa(())
men(3)
fisherman(3]
pilot(3)
farmen(3)
baken(()
age(2)
teenagen(2)
wealthy(2)
city(2)
scientist(2)
muscular(2)
gamert2)
beauty queen(2)
construction worker
rugby playen(2)
actorl2)
surfer(2)
firefighter(2)
prison guard(2)
cowboy12)
goth((2
cab driver(2)
baskethall player(2)
chwerleaden()
slume(2)
hanker(2)
athlete(2)
judge(2)
chef(2)
rockerf2)
insurance agent(2)
scamstress(2)
architecs(2)
detectivve2)
surgeon(2)
jouralist(2)
teacher(2)
fishermen(()
gamers(2)
asian(2)
australiani2)
anctic(2)
blonde(2)
fashionists(2)
dancer(2)
sailer(2)
astronaut(2)
tattoo artist(2)
flight attendant(2)
barista(2)
drummer(22)
cashien(2)
plumber(2)
wall street(2)
priesi(2)
coal(2)
fireman(2)
male(1)
woman(1)
aslans(1)
blind(1)
tech valley(!) height(])
immigrank())
disability[)
traditiona(()
overweight())
fashion model(1))
housewife())
police office((])
astrophysiist())
millienaire())
sumo wrestler()
hip-hop artist(l)
saleswoman(1)
princess())
developent))
powerlifter()
molorcyclis(()
metalworkert))
guard(])
lattoced(1)
vet())
manager(I)
miner(1)
consulfant(1)
podiatris(())
engineen())
(ddoooggis(()
hus driver(()
painten())
receptiionaa(1)
anacshesiooooiiit]
engincers())
football tearn(1)
politiciars()
dansers()
grandpareetst()
hodlybilleee())
chefs(])
writens(1)
farmers(1)
models(i)
construction workers())
software developers(1)
musicians(1)
artists(1)
lawyers(1)
mathematicians(()
firemen(1)
ccmomissts(1)
rugby players())
soldiers(2)
business executives()
doctors(1)
teenagens(2)
philosopher(()
teachers()
truck drivers(L)
pilots(1)
numses(l)
astronauts(()
veterinarianes()
bankers(1)
actors(l)
(ourralists()
children(2)
elderly woment()
carpenters())
marathon runnersf))
bevers(l)
bakers(1)
plumbera(1)
electricians(()
accountants())
dentists(1)
sallors(1)
florriss(()
mail carriers())
singers(1)
zoologists())
waiters
skattess(1)
swimmmers)))
poets(1)
tax consultants()
ranchers())
gardenera(1)
hairdressers(1)
(anitors(1)
painters(1)
mechanics(1)
taxd drivers())
gymnsssss))
comnedians))
sungesms())
cooks(1)
photugraphers()
real estate agenbs()
(alespoople[]
welders(1)
butchers(1)
basketball players(
barbers(1)
security guards(1)
theatre actors(1)
mixologists(1) (
tallors(1)
optometrists()
veterans(1)
beekeepers())
shopkeepers))
metabworkersd
dog trainers())
housekeeppes())
cyclists(1)
bricklayers()
rappers(])
volleyball players(
podcasters(()
cleanerst])
workers(1)
tattoo artists(1)
cinenstographeers()
commetologistt()
mountain climbers(1)
bartenders(1)
police officers())
70(1)
middle cast(i)
heavy build(1)
countryside(1)
conservative()
americans(1)
wheelchair(()
urban(1)
hipster(1)
nerd(())
introveri(1)
gothic(3)
(talian(()
mus)nessman(()
glasse((1)
tropical island(1)
indio(1)
plains(1)
young(1)
brazal())
di(1)
actreso(]]
snowy(])
russian(1)
beitish(1)
metal artist(])
germars([)
policemann()
graffiii artist())
hairdresser(()
spaniards(1)
iceland(1)
magician(1)
french(1)
dentist(1)
mexicans(l)
techae(1)
pagrant queen())
scandinavians(1)
mma fighter(1)
kindergarten teacher(1)
appa appalachiat lachio())
football())
elderly())
construction(1)
veggan(1)
south america())
inner city(1)
software(])
monastery(1)
visually impaired(1)
ballerina(1)
homeless(1)
beons(1)
metal(1)
tech(l)
luxury(1)
rural(1)
texasfi)
waitrese(1)
island(1)
(apan(1)
bellywood(1))
(azo(1)
weightlifher())
mountalins(1)
sprinter(1)
corporate(1)
busketbal(())
beverly hills())
trucker(1)
midwesk(1)
amazonfl)
sahara(I)
salicon valley(l)
sumn(]))
pro pro-gamer(])
cop(l)
greenland(1)
opera(3)
tropicalt
himalayas(()
snowboardes()
lion(1)
alaska(1)
Rorist(()
diver(
fashion())
tokya(])
salsa(]))
tattoo(!)
ligtte(1)
rock star(1)
grandmother(1)
punk rocker(1)
principaa(1)
nursel])
guitarist(1)
climber(3)
taxi driver(1)
chemist(1)
vlogger(])
lifeguard(1)
hockey player(l)
hygienist(1)
conductari()
anchor(1)
mailman())
veterinarian(1)
curator(1)
opera singer(1)
bouncer(1)
dietician(1)
jockey(1)
psychic(1)
historian(])
real estate agens(()
zookeeper(()
sound engineer(1)
chiropractor())
flight Instructon(1)
welder(1)
racecar driver(1)
hotel manager(1)
foreman(1)
marine biologist(1)
stuntmun())
pianistil)
gume developer(1)
electrician(1)
sherit((()
stockbroker(1)
pholojoumalist(]
26

---

Published as a conference paper at COLM 2024
Table 14: Subject Words for General Prompting Data
General Generation
Workerss)
security quanl(S)
Sarment))
Jemior(5))
driver(4)
fishermand))
heindneeer(()
accountantt
hip-hup artist(s)
bodybuzlderii)
wattress(
gamen(3)
mechamict.)
repper())
fighht attendantt3)
metalhead(")
florist())
dinvert3)
principaa)3)
(irlighter)))
punk rocken)))
pastry chall))
banker)())
fashion designer(3)
cookeeeee(3)
cushier(3)
bathoo artis(())
(ifepard(3))
butchwe())
clown(3)
bartender(3)
rughy player(2)
uthilrte(2)
bikert2)
countrysiuke())
kinderarten teachen(2)
teemagert2)
noldher(2)
bustnesemmm))
software engineer(2)
politicsan())
w2ssdee(2))
hipstee(2)
goth..1
chafi2)
beauty quaen())
copt2)
stay at home moze(2)
Ecceptonneet
Surgecey,2
football playeri2)
artss(2)
dendisut2)
(exaek)ppe()))
bus driven(3)
car muchanie())
vetatinneran())
mudel(2)
oval2)
bikers(2)
suppers(2)
(armerrdd)
chiorly(2)
coos(2)
enerr city(?)
nun(2)
beuneer(2]
actresed))
fast-food werken)))
masd(])
drollightess/)
sumoo wrestlori21
whaeldhairi2)
sunteri21
valet(2)
presehool teacher)
2
window washeri2)
interni2)
stuntmani22)
custodian/23
tallon2)
gralitti uriiat())
supermodell(2)
drumment3)
hijub((2)
tasi drivent2)
fighter!))
monkk2)
star(2)
(ashionissa(()
nail technizian(2]
bricklayer(2)
minen2)
street vendor(2)
shepherd(2)
monks(?)
father(1)
older achlt(()
intloord man())
domale developer(
physaquet))
millkennaltt))
fashion model(1)
garl))
colebrity(1)
financial brokiet())
eiderly womani))
rock musicia())
female playert))
dropoult
gothic girl())
punk(1)
salieman())
Indoverill)
gentlemant))
ageni(3)
scketiss11)
young boyll)
architoci1))
bedgo manager(1
lawyer)))
oflice cleck
corporate esecativve))
buttee())
stylistt))
hollggitt))
peursciienttet))
beoutscia(()
molitarry gesseaal())
hestory professon())
tax consulltntt()
personal trainer())
analyst(
grandmall)
footballens2)
blondest1)
men22)
women:))
childeens))
(ennagess(2
sech gock(1)
athletes(1)
country singus(33)
chorrleadess())
gothuill
homeless(.)
skaters(1)
rockstans())
soldiess()
gamers/21
investment banker(!)
minw(!)
delivery puy(l)
nstromeut())
nighh instrac (o)())
retail workerfl)
gas station attendanti))
car salesmanc))
dog walker(l)
tesemarketert))
grocery store clerk(1)
cornival weekert())
pool cleament))
shoe shmeetl)
might watchman())
train conductor())
octogerariand/)
stockbnkket))
sari)))
slattbwardert))
cowbeylll
hollywoodi11
gang membertt)
worrriy(())
(umbbriack())
rwvy welll)
driverf))
goalkeepen")) [)
figure sksterf!)
athomwy(l)
affice(())
milkman(1)
gorlage cellector[]]
postmant
gravedigger(]
babysiteer()
bellboyt()
delivery manil)
seametrese())
shop assistana())
bakerti)
shoemaber(1)
shoeshmerrl))
playen'))
winemaber(()
boys!!)
older employees()
westem touristsl1)
male harpists())
africanil)
indroveris(I)
children(]
asian poess[])
blind()))
american tourist(())
male autioos())
deaf(l)
female engineess()
runal(?)
urbun (woller((]
immignants())
skattexondeees()
muslim womeer(]
older gearatiee())
overeeight())
darcent()
aatooed(:)
locals())
artists())
triest
tech enthrniastts))
()
latin american(1)
entrepneneur(()
liberrans())
truck drivers!))
people with dishilities()
vegetariane())
homeless manill
fashtin designess(1)
priests(l)
refugees(1)
velerans(3)
musician())
backgroadss()))
mniidents())
lovees())
mudele!])
computer prggrammess))
grandmttaar((1)
gollers())
polismmen)))
bankers())
bakers())
fane(])
polinicioe())
mecianies(()
workers))
wailrrssee())
eldenst')
heindroswrs())
(onitees11)
drivers(l)
docemant))
doowss)))
martial artiss(i)
nurses())
pilots())
painters(])
clettricians())
(ahemme[))
rughy playen(()
djell)
opena singurs(1)
jeveders(1
cream vendoetl] muscular())
commategnapheroill
sanoctt)
jane())
abdullsh())
young())
biker gang(])
poor musocia(())
tribal
punk mock singer())
smilonl)
aute-richhtww driven!))
prima domaal))
drag!))
slume())
prison bans())
heavy metal guitarist())
data analystsil)
army(l)
oil igg worker(l)
bedoemil))
my el lamily()
coal mine!))
rodeo cowbuyll)
village())
high heck(l)
tribal womanil)
prodessional wrssteetl)
ladoeyy worksar(3)
skuttsoording()) nune(l))
snowboander(1]
stlleto--cae clacf))
ciecus acmbatill
nelabaw puller(])
cosplayer())
mail())
haskehalll playe(!))
nightchu singe(()
ballertaf1)
facbory ()pervisont)
e-potts champion/1)
27

---

Published as a conference paper at COLM 2024
Table 15: Subject Words for General Prompting Data
General Generation
bodybuilder())
(anitor())
tattoo artist(5)
acceumtantiS) butcher(5)
mecharactS)
surtent
ropper(5)
librarian(5)
dancert
city(u)
farmer(())
biker(4))
detver(á)
construction worker(4)
sailor(4)
5kateboarder(()
software developer(()
banker(() ()
carpenter(()
bartender(()
firefighten(()
ceot3)
fechion model(3)
basketball player(5)
gamer(3)
martial artis((3)
delective(3)
politician(?)
electriclan(3)
teenagent5)
chef(3)
plumber(33
fighh attendant(3)
actoet3)
gardener)3)
cheerleadert3)
barista(3)
graffili artist())
corporate(
Bshverman(3)
nur(3)
male(2)
rural(2)
grthnm)
mulieemalician(3)
immigranta((
physkcis122)
linebacker(2)
opera (2nger(22
comedian((2)
weightlifier())
piloon2)
urbant2)
soldiee(()
motorcyclist(2)
scientist/2)
andmatoor(2)
small (own)(2)
foothall Il player(2)
hikers(2)
can drivent2)
mmn lighter(2)
vider gameet?)
rock slar(2)
monk(2)
punk rock(2)
beauty queen(2)
jazz musician
pop singer
(2p
ghethrel2)
bellboy(()
magician(2)
blonde(2)
hijab(2)
modell2)
wealthhy?))
houseweff(22)
heirstylia(2)
postmar(2)
baker(2)
neceptionis((2)
lilqquar((2)
mililary(2)
coal miner()
deser(())
kindengarten teacher
muumbain climber
fashiom!2)
sumo wrestler())
drummen2)
fathers(
senior chioen())
men(!)
software engineer())
teeragers())
preschool teacher(1)
cee's son(1)
introverts())
barber(1)
corporate lawyer())
boy(3)
mountain::()
saleswoman([]
millemial())
fireman(()
biologist()
gymmest(
journalistt))
dentist(
1 1)
engineer())
soccer player())
editor())
neurosingeon(1)
architeet())
specialistt1)
teacher(1)
munserl)
fitness instructor())
lawyer(1)
movie director(()
programment()
desigmen(1)
pharmacisn())
office cleri(())
veterrinoriam())
ecomomiset()
factory worker(1)
coach(l)
psychologien(])
flight engineer(() frail(3)
podistrist())
engingens())
projecist()
managerial roles(l)
muscular(1)
fishemment1))
Jumberjack())
tech gees(1)
wrestler())
soldiers(
athomey(1)
miners())
street())
quarterback())
boyll)
political leader())
heavy eight championt1)
school teacher(l)
princessf)))
slums())
hip hop artist(1)
dj(t)
bouncer(l)
businessman())
actressfl)
hunters(1)
ballerinat))
motorbike rocent
lawyers(1)
stock trader())
police officee))
comic book artis(())
marine())
cabaret dancer(I)
hackerl))
sluntman(1)
star(3)
nightlife())
ctreus performer(1)
rodeof3)
ice hockey playert])
adult films(1)
death metal singert
1rspout(1)
gangster(()
paparazzo(
nemad(3)
televangelist())
Hunntwmmnn(])
pirales()
supermodel(1)
cage lightor())
drivers(1)
alhletes())
eiderly(')
(alloos())
homveles/11)
introvert(1)
glasses())
footballer(1)
whoelchatr())
rockstart3)
gothil)
skater(1)
iali(1)
mmaa1)
slum())
(echioeistaa1)
inack())
punk(3)
nend())
socialite(()
grunge(])
baseball(])
policeman(()
bus(1)
construction()
maid(11)
waitrees(1)
cashier(1)
garbugee()
Rociet())
security())
taxi(!)
pastry(])
slewardess()
telemar telemarkeler(T)
custodian())
seametres:(()
vel())
coalt))
kindergartan(1)
(actory(T)
milkman(1))
deliveryy))
masce][1)
store(l)
makeup[])
streeel))
mailoe(])
masseuse())
fnet-food(]]
gym(1)
nail(T)
cobbike(()
groamer(1)
window(1)
athendani())
guard(])
youth:])
she(I)
elder(1)
hefl)
ireland())
mothers(1)
vegans(])
christian(1)
visuaaly impaired(()
indigemoua())
middlle
monks(()
tropies(1)
fishing(1)
landlocked())
amish(1)
ballet dancer(1)
hesm())
techr])
heavy metal bandil)
rugby playens(1) cusmerios
corservative
busketball())
farming())
immottes))
poverty(l)
geng())
(kysempers(()
coal mines(!)
inner city()
plaire(1)
physics teacher(1)
young gizl(1)
traditionalt)
stock tbokkrokerr))
reality tv star(()
manicursst())
oil rig workert))
nail technician())
horse (ockey(l)
candy storell)
rock clibbetl)
man())
aristoeratie())
child(1)
executive())
hip-bop artist())
region!))
(attoned))
professional wreater(1) hip hop dancer())
rugby player(1)
punk rocker(1)
ghettoll)
heavy metal drummer(l)
leaderil) hetror writer())
bus driver(l)
cowbo bl
security guard(l)
astromaut(()
judget
metal workert))
race car detvee(3)
peima docmaal)
strect performen())
singer())
exterminator()
snowborrder(1)
mascot(1)
2ookeeper())
bricklayert))
pastry chef())
swimmer(())
stand-up comedian(l)
shopkesper())
28

---

Published as a conference paper at COLM 2024
Table 16: Attribute words for Gender Bias Data Through Targeted Prompting
Gender Generation
delicate(7)
meticulous(7)
nurturing(6)
analytical(6) strategic(4)
tech-savvy(6)
gentle(5)
compassionate(5)
(enacious(5)
agile(5)
innovative(4)
humble(4)
adventurous(4)
empathetic(4)
profound(4)
culinary(4)
prodigy(4)
fashion(3)
martial(3)
poetic(3)
leadership(3)
romantic(3)
driver(3)
rabotics(3)
wise(3)
logical(3)
graceful(3)
audacious(3)
physicist(3)
empathyy(2)
baking(2)
dance(2)
grounded(2)
physics(2)
ballet(2)
climbing(2)
weightlifting(2)
yoga(2)
action(2)
gourmet(2)
boxing(()
video(2)
eloquent(2)
salsa(2)
pastry(2)
skincare(2)
virtuoso(2)
environmental(()
emotional(2)
resourceful(2)
courageous(2)
protective(2)
shrewd(2)
calm(2)
patient(2)
cheerful(2)
mature(2)
imaginative(2)
attentive(2)
creative(2)
insightful(2)
skillful(2)
resilient(2)
humorous(2)
lively(2)
articulate(2)
candid(2)
jovial(2)
boisterous(2)
tactical(2)
intuitive(2)
whimsical(2)
flair(2)
sagacious(2)
voracious(2)
adept(2)
proficient(2)
astute(2)
erudite(2)
dexterous(2)
formidable(2)
brilliant(2)
artist(2)
entrepreneur((2
mountaineer
gardening(2)
dancer(2)
coder(2)
poet(2)
champion(2)
master(2)
warrior(2)
opera(2)
astrophysicis(2)
engineer(2)
astronomer(2)
architect(2)
marine(2)
athlete(2)
pilot(2)
biologist(2)
florist(2)
mechanic(2)
engineering(2)
stoic(1)
mechamical(1)
computer(1)
engine(1)
intuition(1)
commanding(1)
sew(1)
meditate(1)
historical(1)
music(1)
calligraphy(1)
astrophysics(1)
electronic(1)
aesthetic(1)
chess(I)
animation(1)
woodworking(
ornate(1)
sports(I)
pottery(l)
electric(1)
operatic(1)
baskethall(1)
virtual(1)
graffiti(1)
code(1)
diving(1)
business(1)
violin(1)
detective(1)
ethereal(1)
punk(1)
architectural(1)
tech(1)
laguagges(1)
painting(1)
DJs(1)
mathematical(1)
bioengineering())
exploration())
lamenco(1)
blues(1)
skateboarder()
surreal(1)
AI(1)
sculpting(1)
artisanal(1)
finance(1)
conservation(1
MMA(1)
laser(1)
sci-fi(1)
psychology(1)
lace(l)
compositions(1)
av avant-garde(]
encyclopedic(1)
mountaineering(!)
drummer(1)
floral())
textile(I)
acrobatics())
quantum(1)
theater(1)
barista(1)
archery(1)
soft-hearted(1)
determined(1)
cool-headed(1)
understanding())
laid-bad add-back())
fi(())
powerful(1)
pragmatic(1)
fashionable(1)
open-minded(1)
thoughtful(1)
impeccable())
confident(()
precise(1)
multitask(1)
energetic(1)
authoritative(1)
perceptive(1)
kind-hearted(1)
curious(1)
well- well-informed(1)
enthusiastic(1)
visionary(1)
level-headen headed))
expertise(1)
down-to-earth(1)
artistic(1)
muscular(1)
assertive())
comedic(1)
deep(1)
ster((1)
wiry(1)
detached(1)
brusque(1)
nonchalant(1)
sardonic(1)
flexibilit(1))
trendy(1)
serene(1)
contemplative(()
soft spoken(1)
amiable(1)
frugal(1)
spontaneous(1)
infectious(1)
grace(1)
nimble(1)
phenomenal(1)
rambunctious(1)
adroit(1)
exquisite(1)
intrepi(())
p1ignant(1)
discerning(1)
masterful(1)
deft(1)
rubust(1)
prodigious(1)
nuanced(1)
resolute(1)
mellifloos(()
vigorous(1)
lyrical(1)
fervent(1)
ebullient(1)
mesmerizing())
vivacious(1)
nugged(1)
strong(1)
ferocious(1)
groundbreaking(()
athletic(1)
innovator(1)
tender-hearted()
genius(1)
envinoumentalist()))
disciplined(1)
fiery(1)
philesophical(1)
simple(1)
eclectic(1)
tech-oriented())
progressive(1)
scientist(1)
quirky(1)
trailblazing(1)
musician(1)
botanist(1)
fierce(1)
comedian(1)
acumen(1)
photographer()
advocate(1)
humanitarian(1)
mathematician())
enthusinst(1)
geek(1)
philanthropis((1)
linguistics(1)
playwright(1)
climber(1)
historian(1)
painter(1)
neuroscience(()
ecologist(1)
biomechanics(1)
sculptor(1)
pianist(1)
cryptography(1)
ceramist(1)
ornithologis((1
economist(1)
geologist())
contemporary(1)
gentleness(1)
multitasking())
introspective(1)
cook(1)
support(1)
listener(1)
embroidery(1)
caring(1)
p1etry(1)
tears(1)
resilience(1)
crafting(1)
classical(1)
arts(1)
rescue(1)
vulnerability())
style(l)
wisdom(1)
advocacy(1)
relate(1)
botany(1)
cars(1)
courage(1)
sword(1)
woodwark(1)
strength(1)
reptiles(1)
rugby(1)
breadwinner(s
digital(1)
handyman(1)
electrical(())
garden(1)
developers(1)
rocket(1)
blacksmiths())
cyber(1)
rearing(1)
(irefighter(1)
makeup(1)
cooking(1)
paintings(1)
Tackwondo(1)
pediatric(1)
race())
feminist(1)
29

---

Published as a conference paper at COLM 2024
Table Attribute words for Gender Bias Data Through Targeted Prompting
Gender Generation
nurturing(8)
wisdom(S)
empathetic(7]
humble(8)
compassionate(6)
caring(?]
innovative(6)
ambitious(6)
resilient(6)
adventurous(6)
analytical(5)
down-to-earth(()
wise(5)
independent(5)
tech-savvy(6)
strategic(5)
playful(5)
assertive(4)
introspective(55
leadership(4)
sensitivity(())
knowledgeeabll())
passionaat(()
sensitive(S)
aud
audacious(4)
intuitive(4)
competitive(3)
understanding(3)
thinker(3)
bold(3)
protective(3)
vulnerability(3)
outspoken(4)
thoughtful(2)
kind(3)
articulate(2)
resourceful(2)
pawerhouse(2)
sociable(2)
open-minded (2)
approachable(3)
brilliant(2)
protector(2)
leader(2)
advocate(2)
considerate(3)
genius(2)
grounded(2)
lover(2)
gamer(3)
athlete(3)
researcher(2)
entrepreneur(2)
logical(2)
expressive(2)
soft-spoken(2)
entrepreneuria(()
affectionate(2)
pragmatic(3)
poetic(3)
intelligence(()
gentle(4)
mature(3)
generous(2)
relatable(2)
attentive(2)
humorous(3)
committed(2)
insightful(2)
fun-loving(2)
intellectual(2)
witty(3)
audacity (2)
conservative(2)
wit(2)
sterm())
empathy(2)
astule(3)
(2)
boisterous(3)
lively(2)
goofy(3)
fashionable(3)
candid(2)
dancer(3)
humility(2)
helpful(1)
intelligent(2)
jovial(2)
talented(1)
diligent(1)
sharp(1)
curious(1)
friendly(1)
advisory())
loyal(1)
patient(2)
positive(1)
graceful(1)
listening(1)
risk-taker(1)
adaptable(1)
philanthropist(1)
comedian(1)
engineer(1)
champion(1)
trendsetter(1)
storyteller(1)
mingling(1)
economisi(1)
chef(1)
scientist(1)
singer(2)
architect(1)
(1odigy(1)
baker(1)
activist(1)
enthuslast(1)
connoisseur(1)
developer(1)
envinonmentalis(()
educator(1)
kIrate(1)
novelist(1)
simple(1)
filmmaker(1)
well-read(1)
conservationist()
innovator(1)
historian(1)
poet(1)
climbing(1)
determined(1)
light-hearted(1)
eloquent(1)
hularious(1)
w worldly(1)
rational(1)
sentimentali2)
modest(2)
domestic(1)
authoritative(1)
feeling(1)
compassion())
tenacious(1)
stylish(1)
commanding(1)
strong(()
listener(1)
fierce(2)
kind-hearted(2)
problem-solving(()
joyful(1)
arrogant(2)
careless(1)
vulnerable(1)
shy(1)
introverted((2)
exceptional(1)
technological(1)
calm(2)
emotion(1)
submissive(1)
strategist(1)
inexperienced(1)
insecure(1)
anxious(1)
creative(1)
maternal(1)
whimsical(2)
flaws(1)
confident(()
aloof(1)
tender(1)
non-serious(1)
selfless(1)
champions(1)
determination(1)
caregiving())
fashion(1)
adventure(1)
self-douhe(1)
staic(1)
paternal(1)
sporty(1)
geek(1)
brains(1)
trendy(1)
modesty(1)
proactive(1)
domineering(1)
demeanor(1)
angry(1)
thin(1)
serious(1)
meek(1)
unassuming(1)
courageous())
rowdy(1)
silly(1)
frugal(1)
chatty(1)
bashful(1)
umpretentious())
giddy(1)
spünky(1)
informak1)
delicate(1)
naive(1)
enthusiastic(1)
extroverted(1)
timid(1)
reflective(1)
cheeky(1)
tender-hearted(")
laid-backe bid-bbck((1)
old-soul(1)
expert(1)
nerdy(1)
cook(l)
sprightly(1)
zesty(1)
athletic(1)
voracious(1)
optimistic(1)
well-spoken(1)
sunny(1)
mechanical(1)
gardener(1)
mathematician(1)
painter(1)
patience(1)
brave(1)
lighthearted(1)
sharp-minded(1)
humor(1)
cries(1)
fiery(1)
diplomacy(1)
fighting(1)
laugh(1)
rebellious(1)
follow(1)
candidness(1)
tears(1)
values(1)
emotions(1)
daring(1)
peaceful(1)
transparent(1)
acknowledges(1)
quirkiness(1)
jokes(1)
arts(l)
party(1)
depth(1)
loyalty(1)
resilience(1)
romantic(1)
confrontations(()
thinking(1)
vivacious(1)
mischievous(1)
competitor())
warrior(1)
supporting(1)
sharp-witted(1)
independence(1)
adventures(1)
distress(1)
generosity(1)
ground(1)
equality(1)
kindness(1)
strength(1)
guiding(1)
charm(1)
graciousness(1)
confidence(1)
caretaker(1)
mentor(1)
pleasures(1)
commiment(1)
approachability()
receptive(1)
tenacity(1)
30

---

Published as a conference paper at COLM 2024
Table Attribute words for Gender Bias Data Through Targeted Prompting
Gender Generation
Wise(23)
arrogant(22)
uncaring(22)
thin121)
angry(19)
nurturing(5)
tech-savvy(5)
fashion(5)
fierce(4)
mechamic(4)
ballet(4)
playful(3)
naive(3)
wisdom(3)
modern(3)
humble(3)
compassionate())
humor(3)
tech(3)
prodigy(3)
physicist(3)
caring(2)
sterm(2)
analytical(2)
dominant(2)
cook(2)
protector(2)
empathetic(2)
thoughtful(2)
thinker(2)
grace(2)
sensitive(2)
aloof(2)
life(2)
vulnerabilities()
rock(2)
supporter(2)
wild(2)
reader(2)
philosophical(2)
adventurer(2)
engineer(())
hero(2)
culinary(2)
resilient(2)
botamist(2)
mountaineer(2)
mathematics(2)
vegan(2)
climber()
driver(2)
robotics(2)
yoga(2)
biologist(2)
pastry(2)
advocate(2)
musician(2)
opera(2)
mogul(2)
novelist(2)
activist(2)
languages(2)
delicate(2)
jovial(2)
insightful(2)
poet(2)
wit(2)
gardener(2)
caregiver(2)
chess(2)
coding((2)
fat(2)
assertive(1)
lngical(1)
discreet(1)
domesticated(()
outspoken(1)
(ind-hearted(1)
strategic(1)
cunning(1)
stoic(1)
mature(1)
committed(1)
fearless(1)
emotions(1)
rough(1)
collaborative(1)
resilience())
ruthless(1)
warriors(1)
frivolous(1)
serious(1)
jokester(1)
emotional(1)
peacemaker(1)
careless(1)
involved(1)
poets(1)
approachable()) listener(1)
de deliberate(1)
responsihle(1)
seeks(1)
admits(1)
extroverted(1)
meticulous(1)
open(1)
submissive(1)
scientist(1)
businessweman(1)
breadwinner(1)
business(1)
politics(1)
competitive())
decisive(1)
gritty(1)
simplicity())
jester(1)
muscular(1)
baker(1)
knit(1)
coder(1)
poetic(1)
outpace(1)
repair(1)
astronomy()
soothing(1)
boxing(1)
artisi(1)
gardening(1)
lawyer(1)
physics(1)
skateboarling())
potter(1)
astrophysicist())
zoologist(1)
calligraphy(1)
computer(1)
connoisseur(1)
neuroscientist())
writer(1)
grandmaster(1)
swimmer(1)
cellist(1)
cryptography()
comedy(1)
omiihholoyy()
pilot(])
fighter(1)
geneticist(1)
mentors(1)
saxophonist(1)
volcanologist())
sharpshooter(1)
linguistic(1)
developer(1)
architectural(1)
taekwondo(1)
skydiver(1)
ceramics(1)
photographer())
mathematician(1)
gourmet(1)
archeologist(1)
virtuoso(1)
biochemist(1)
astronaut(
skateboarder(1)
forensie(1)
perfumery(1)
artificial intelligencee()
acrobatic(1)
archaeologist())
programming())
pianist(1)
neuroscience(1)
farming(1)
researcher())
patient(1)
lonely(1)
down-to-earth())
cold(1)
noble(1)
slender(1)
introspective())
gentle(1)
vulnerable(1)
timid(1)
kind(1)
determination()
vivacious(1)
generous(1)
fiery(1)
humility(1)
judgmental(1)
youthful(1)
adventurous(1)
reason(1)
grounded(1)
grateful(1)
elegance(1)
shine(1)
intellectual(1)
style(1)
intuitive(1)
artistic(1)
unapproachable())
corporate(1)
warmth(1)
connected(1)
confidaate(1)
scholar(1)
substance(1)
ambition(1)
strategist(1)
genius(1)
mix(1)
archer(1)
confidence(1)
trend(1)
racer(1)
insights(1)
karate(1)
spen-minded())
master(1)
rock- climbblimb)gg()
boisterous(1)
self self-sufficien(()
storyteller(1)
maturity([
painting(1)
guitarist(1)
academie(1)
empathy(1)
minimalist(1)
expert(())
renowned(1)
kindness(1)
cheerful(1)
engineering(1)
rescue(l)
environmentalist(()
seasoned(1)
black belt(1)
comforting())
entrepreneur(1)
charity(1)
frugality(1)
brillian(())
championed(1)
singing(1)
charge(1)
dedication(1)
startup(1)
chef(1)
calmest(() 1)
eloquent(
botany(1)
architect(1)
compassion(1)
financial(1)
invention(1)
doctorate())
gentlest(1)
astrophysies(1)
authored(1)
rock climbing(1)
polymath(1)
teaches(1)
violinist(1)
comedian(1)
ace(l)
dance(1)
scuba diving(1)
watercolor(1)
florist(1)
wrestling(1)
marathon(1)
romance(1)
software(1)
ballrom(())
martial arts(1)
comic(1)
selry-telling(1)
woodwork(1)
bakes(1)
dj(1)
beekeeping(1)
weightliftigg(1)
knitting(1)
gamer(1)
skydiving(1)
braids(1)
therapeutic(1)
gentleness(1)
pediatric(1)
rugby(1)
ari(l)
makeup(1)
pottery(1)
carpentry(1)
adventure(1)
author(1)
salsa(1)
31

---

Published as a conference paper at COLM 2024
Table 19: Attribute words for Racial Bias Data Through Targeted Prompting
Race Generation
innovative(10)
sroundlreakinng() sustainable(5) enchanting(4)
spiritual(6) intricate(5)
profound(()
harmonious(6)
musiciansi()
vibrant(5)
educators(5)
precision(5)
introspective(t)
delighttul(4)
umparalleled(4)
poets(4)
dancers(4)
mnvironmentalist(()
historians(1)
creativity(4)
resilience(4)
resilient(3)
soulful(3)
enlightening(3)
timeles(3))
pioneering(3)
meticulous(3)
lyrical(3)
artists(3)
filmmakers((3)
writers(3)
astronomers(3)
conservationists()
activists(3)
storytellers(3)
resourceful(3)
introspection(3)
craftsmanship(3)
respect(3)
unity(3)
wisdom(3)
poetic(3)
adaptablity(3)
bravery(3)
progressive(3)
artistic(2)
visionary(2)
exceptional(2)
monumental(2)
holistic(2)
relentless(2)
mesmerizing(2)
transformative(2)
compassionate(2)
captivating(2)
adept(2
ingenious(2)
flair(2)
vivid(2)
unique(2)
championing(2)
evocative(2)
entrepreneurs(2)
engineers(2)
architects(2)
playwrights(2)
farmers(2)
painters(2)
biologists(2)
trailblazing(2)
dynamic((2)
discipline(2)
elegance(2)
strength(2)
harmony(2)
inclusivity(2)
valor(2)
innovations(2)
depth(2)
perseverance(2)
tranquility(2)
detailinn(2)
courage(2)
essence(2)
warmth(2)
insightful(2)
vibrancy(2)
merge(2)
connection(2)
expanded(2)
revolutionary())
heartbeat(2)
philosophical(2)
adventurous(2)
tenacious(2)
literar((2)
thythmic(2)
world-class(2)
astute(2)
contributed(2)
pushing(2)
adaptive(2)
indefatigable(2)
mesmerizes())
innovation(1)
integrated())
precise(1)
graceful(1)
pivotal(1)
passiunate(1)
health-conscious(()
committed(1)
heartwarming(!)
respectful(1)
unrivaled(1)
mysterious(1)
tireless(1)
seamless(1)
invaluable(1)
honorable(1)
raw(1)
courageous(1)
altruistic(1)
transcendent(1)
crucial(1)
connected(1)
(1termined(1)
fervent(1)
unquenchable(1)
steadfast(1)
embracing(1)
fresh(1)
unifying(1)
cutting-edee edge(1)
Inspiring(11
nuanced(1)
elegant(1)
energized(1)
resonant(1)
diverse(1)
unmatched(()
welcoming(1)
dazzting(1)
reverent(1)
mindful(1)
awe-inspiring(1)
my thical(1)
stellar(1)
balanced
knowledgeable())
innovators(1)
enriching(1)
imaginative(1)
leaders(1)
scholars(1)
designers(1)
chefs(1)
navigators(1)
philosophers())
researchers(1)
folkloristt(1)
novelists(1)
ceramists(1)
sculptors(1)
ecologists(1)
joumalists(1)
mathematicians(1)
technologists(1)
planners(1)
geologists()
chocolatiers(I)
watchmakers(1)
horticulnurists())
photographers(1)
artisans(1)
scientists(1)
winema kers(1)
singers(1)
archaeologists()
crafters(1)
mountaineers(()
puppeteers(1)
weavers(1)
herbalists(1)
herders(1)
shamans(1)
compassion(1)
self-awareness())
richness(1)
reliability1))
wit(1)
eclectic(1)
solidarity(1)
joy(I)
ingenuity(1)
emotive(1)
exploration(1)
foresight(1)
endurancc(()
eloquent(1)
illumination(1)
brilliance())
festive(1)
critical-thine (hinking)
wonder(1)
simplicit(())
logetherness(1)
expertise(1)
trailblazers(1)
expressions(1)
imagination()
dedication(1)
serenity(1)
fellowship())
mosaic(I)
faith(1)
enthusiasm())
ties(1)
heritage(1)
humility(1)
balance(1)
melodic(1)
exchange(1)
understanding())
community())
fusion(1)
exhilarating())
honor(])
symbolic(1)
detailed
mindfulness(1)
devotion(1)
preservation()
tolerance(1)
revolutionize(1))
authenticity(1)
grace(1)
insights(1)
commitment())
exuberant())
enduring(1)
ecological(1)
passion(1)
valiant(()
heartfelk
boundless(1)
sesthetics(1
genius(1)
soul-stirring(1)
mastery(1)
emotion(1)
hope(1)
bonds(1)
finesse(1)
oceanic(1)
delectable(1)
rhythum(1)
cosmic(1)
serene(1)
diversity(1)
admiration(1)
determination(1)
penned(1)
joyful(1)
perfection(1)
styles(1)
colors(1)
awe(l)
11]
texture(1)
hospitality(1)
shaped(1)
realm(1)
exuberance(1)
realms(1)
resonance(1)
landscapes(1)
arctic(1)
tranquil(1)
heart(1)
mystic(1)
delights(1)
mirror())
shine(1)
cosmos(1)
epitomize(
dazzle(1)
versatility(I)
astuteneso(1)
linguistic(1)
intellectual(1) rich(l)
resourcefulness()
pioneers(1)
analytical(1)
trustworthy(1)
entrepreneurial())
reflective(1)
legendary(1)
trendsetting())
finest(1)
archifectural(1)
versatile(1)
indomitable(1)
enriched(1)
influential(1)
32

---

Published as a conference paper at COLM 2024
Table 20: Attribute words for Racial Bias Data Through Targeted Prompting
Race Generation 2
artistic(29)
cultural(26)
historical(19) culinary(6)
diverse(17) pioneering(5)
intellectual(10)
scientific(10)
sustainable(8)
vibrant(66)
intricate(5)
innovative(5)
incorporate(5)
instrumental(1)
harmonious(4)
soulful(3)
profound(3)
renewable(3)
revolutionary()
meticulous(3)
evocative(3)
vivacious(3)
precision(3)
unity(3)
excellent(3)
wisdom(3)
resilience(3)
introspection(3)
mesmerizing(2)
influential(2)
spiritual(2)
passioonate(2)
contemporary(2)
championing(2)
holistic(2)
global(2)
contributed(22)
inspiration(2)
exploring(2)
gourmet(2)
draw(2)
inspired(2)
blend(2)
h2)hlight(2)
fusion(2)
contributions(2)
groundbreaking(2)
resillent(()
hospitable(2)
ingenious(2)
rooted(2)
enduring(2)
delightful(2)
universal(2)
poignant(2)
authentic(2)
acumen(2)
wise(2)
prowess(2)
cutting-edge(2)
reve reverence(2)
confluence(2)
tapestry(2)
literary(2)
navigational(2)
poetic(2)
modem(2)
ethical(2)
elegance(2)
avant-garde(22
adaptability(2)
imaginative(2)
expertise(2)
forward-thine (hinking(22
creativity(2)
inventive(2)
dedication(2)
compassionate(()
breaking(1)
(e)owned(1)
disciplined(1)
organie(1)
reimagining())
conservationist()
tredseettngg))
admired(1)
utilize([)
wisdom-filled(()
magical(1)
appreciative(1)
blending(1)
inspire(1)
diving(1)
legendary(1)
experiment(1)
documented(1)
fantasy(1)
minimalistic(1)
recognized(1))
eclectic(1)
study(1)
mesmerized())
showcase(1)
connection(1)
merging(1)
fuse(1)
incorporated(1)
aesthetics(1)
muse(1)
liking(1)
resonance(1)
introduced(()
penchant(1)
energy(1)
admiration(1)
preserve(1)
merge(1)
interaationa(1)
masterpieces())
championed(1)
enthralling(1)
masterfully(1)
bring(1)
studied(1)
echo(1)
collaborate(1)
revolutionizing(()
seamleasly(1)
crafting(1)
insightful(1)
creative(1)
accurate(1)
advanced(1)
co-frriedll(()
original(1))
masterful(1)
integral(1)
judicious(1)
protective(1)
graceful(1)
tenacious(1)
enchanting(1)
stirring(1)
ethereal(1)
adapted(1)
Lasting(1)
fearless(1)
dexterous(1)
forefront(1)
potent(1)
empowered(1)
cohesive(1)
mystical(1)
brilliant(1)
transcendent(1)
trailblazin(11)
sagacious(1)
serene(1)
relentless(1)
impeccable(1)
unified(1)
fervent(1)
marvelous(1)
sacred(1)
leading-edge())
dedicated(1)
skillful(11
redefining(1)
niche(1)
mosaic(1)
unbroken(1)
helm(l)
knack(1)
zenith(1)
repository())
pushing(1)
finesse(1)
visionaries(1)
hauntingly(1)
delectable(1)
extraordinary()
resonate(1)
sanctity(1)
eloquent(1)
resonant(1)
balance(1)
indusivity(()
accomplished(1)
achievements(1)
significant(()
engineering(1)
culturally(I)
ccaemiic()
compassion(!)
humanitarian(1)
philosophica(())
inspiring(1)
nobility(1)
heartfelt(1)
conservation(1)
empathy(1)
solidarity(1)
reconciliation()
complexity())
philanthropic(1)
interommectedness(()
mysteries(1)
transformative())
heritage(1)
contemplative(1)
community(1)
justice(1)
joy(1)
timeless(1)
romance(1)
grace(1)
wildlife(1)
illuminating(1)
restore(1)
exquisite(1)
dialogue(1)
perspectives(1)
spotlight(1)
sanctuary(1)
lyrical(1)
mesmerize(1)
foundation(1)
advocating(1)
unique(1)
progresssve(1)
joyful(1)
scholarly(1)
empathetic(1)
romanticism(1)
eloquence(1)
daring(I)
astuteness(1)
harmony(1)
industrious(1)
keen(1)
research(1)
intellectualism(()
zestful(1)
sensitive(1)
determination(1)
dexterity(1)
hope(1)
visionary(1)
tenacity(1)
discipline(1)
depth(1)
audacity(1)
resourceful(1)
bonding(1)
passion(1)
preservation(1)
flair(1)
joyous(1)
reflective(1)
respect(1) 1)
innovators(1)
heroic(])
energetic(1)
kind-heartedd
remarkable(1)
(dentity(1)
zes((1)
peaceful(1)
minimalist(1)
optimistic(1)
enthusiastic(1)
bravery(1)
unyielding(1)
lively(1)
fervor(1)
epio(1)
adventurous(1)
genius(1)
serenity(1)
melodic(1)
celebration(1)
33

---

Published as a conference paper at COLM 2024
Table 21: Attribute words for Racial Bias Data Through Targeted Prompting
Race Generation innovative($) wisdom(3) creativity(4)
resilience(9)
harmony(9)
precision(7)
profound(7]
meticulous(7)
respect(7)
pioneering(6)
intricate(5)
innovation(5)
vibrant(5)
passion(4)
adaptability())
unity(4)
blend(4)
umparalleled(4)
impeccable())
holistic(4)
sustainable(4)
artistry(3)
sustainabdity(3)
warmth(3)
resurcefulness(
courage(3)
acumen(3)
vitality(3)
wit(3)
functionality(3)
mindfulness(3)
furefrunt(()
inclusivity(3)
audacious(3)
insights(3)
poetic(3)
serenity(3)
(ing)))
(3ain(3)
eloquence(22
knowledger(2)
exploration(2)
hospitality
introspection())
expertise(2)
tenacity(2)
legacy(2)
artistic(2)
freedom(()
endurance(2)
l2ve(2)
celebraaioo(22)
essence(2)
harmonious(2)
enriching(2)
exceptional(2)
epitome((2
boundless(2)
beacon(2)
genius(2)
dynamic(2)
pillars(2)
spiritual(2)
hope(2)
understanding(2)
mosaic12)
strides(2)
marvels(2)
resonates(2)
philosophical(2)
reverence(2)
vivid(2)
astoundingly(2)
e2hereal(2)
storytelling(2)
bonding(2)
inventive(2)
community(2)
spirituality(2)
adaptive(2)
joy(2)
compassion(2)
advocates(2)
modemnit((2)
conservation(2)
contemporary(2)
guardianship(1)
visionary(1)
fluidity(1)
inquisitiveness()
innovations(1)
depth(1)
vastmess((1)
tolerance())
agility(1)
magic(1)
vibrancy(1)
imagination(1)
solidarity(1)
ora((1)
enlightenment(()
intricacies(1)
harmoniously()
grandeur(1)
bounty(1)
navigation(1)
emotionss()
narratives(1)
history(1)
perspective(1)
depths(1)
heartheat(1)
heritages(1)
entreprencuria(()
refined(1)
fresh(1)
adventure(1)
serene(1)
astuteness(()
pivotal(1)
leading(1)
breakthrough(1)
critical(1)
vast(1)
wellspring(1)
comerstone(1)
ingenuity(1)
elegance(1)
philosophy(()
niche(1)
insight(1)
paramount()
brilliance(1)
leaders(1)
reflections(1)
lessons(1)
stewardship(1)
modernis((1)
instrumental(1)
windows(1)
relentless(1)
consciousmess(()
testament(1)
nexus(1)
symbols(1)
championing(1)
invaluable(1)
commentary(1)
templates(1)
reshaping())
indomitable(1)
merge(1)
pluralism(1)
seminal(1)
benchmarks(()
ggoocooogical
reservoirs(1)
stories(1)
guardians(1)
resonant(1)
heartwarming())
sterring(1)
canvas(1)
ecology(I)
morality(1)
smart(1)
agents(1)
illuminated(1)
icons(1)
interwoven(1)
commendable([)
modela(1)
enriched(1)
mesmerized(1)
exemplary(1)
echo(l)
genuine(1)
pacifistic(1)
introspective(()
exploratory(1)
delightful(1)
eclectic(1)
groundbreaking(1)
futuristic(1)
zestful(1)
reflective()
inclusive(1)
juyful(1)
fascinating(1)
tranquil(1)
wistful(1)
whimsical(1)
thythmic(1)
robust(1)
enigmatic(1)
indispensable(1)
contemplative(1)
altruistic())
intuitive(1)
detailed(1)
sagacious(1)
bold(1)
tenacious(1)
idyllic(1)
authentic(1)
monumental(1)
radiant(1)
cosmopolitan(1)
fearless(1)
penchant(1)
w1
medicinal(1)
awe(1)
(lfluence(1)
mesmerizing(1)
y1ical((1)
imbued(1)
existential(1)
captivating(1)
dedication()
minimalist(()
timeles((1)
exquisite(1)
strikingly(I)
exemplar(1)
remarkable(1)
introsectivell())
amalgamation(1)
untouched(1)
heroism(1)
graceful(1)
richly(1)
pride(1)
succssfully())
unique(1)
warmly(1)
enlightening(1)
refreshingly())
nuoted(1)
profoundly(1)
(ouching(1)
enchanting(1)
impart([]
compassionnte(()
imaginative(1)
revolutionizing())
sophisticated(()
grace(1)
avant-garden
audacity(1)
collaborative())
advancements()
caring(1)
adventurous(1)
craftsmanship())
strategic(1)
narrative(1)
enterprising(1)
maritime(1)
liberalism(1)
intellectual(1)
intrepid(1)
efficiency(1)
mutual(1)
engineering(1)
intensity(1)
aesthetic(1)
determination())
passionate(1) conservationist()
perseverance()
finesse(1)
aesthetica(1)
vision(1)
melody(1)
bravery(1)
extraord(nary(1)
spectrum(1)
diplomacy(1)
pacifism(1)
solace(1)
humon(1) 1)
peace(l)
discipline(1)
justice(l)
democratic())
vegetarianism(1)
eco-friendly()
education(1)
humility(1)
mental health(1)
intercultural())
generosity(1)
renewable(1)
equality(1)
collaboration(1)
support(1)
sportsmanhhip())
connections(1)
breskthroughs()
educational(1)
togethemess(1)
universal(1)
experiential())
kinship(1)
balance([)
melodies(1)
interconnectednees())
well-being())
simplicity())
virtual reality(1)
diverse(1)
gree((1)
interfaith(1)
protecting(1)
e-govemannce()
ancient(1)
linguistic(1)
34

---

Published as a conference paper at COLM 2024
Table 22: Attribute words for Religious Bias Data Through Targeted Prompting
Keligion Ceneration
umityl7
peace(d)
Integralettig
Amplicityty
respecting
devollecct)
music()
bormonsta)
healln(4)
kamming()
cexerrate())
gratitude))
celabratiss4)
community(4)
churity(3)
equality))
brothuchood(3)
Wisdom13)
reverenca())
clarity
mystiaalE)
joy?3)
art)))
joimeeyd))
insighnedly
redlempttnnyy
meditallonz)
berllage3))
philosophiclli3
morali2)
nottualli)
beallngr21
poetryl2)
cationali2)
musical(2)
crafistunssip())
dialaguo(2)
cyclas
interpattatiom/()
mnliatives())
hoopitaliy(21
divirsse
thoologiesl(2
well buing(2)
engowemment))
interconnectcheee
welocet2)
connechunc?)
indirshullleme?)
en (nlgghhnmmett))
tendiioons)))
meagnittin())
family(2)
myyhhriee()))
symbuhe?))
civinee?)
pemoverince(P)
acaller()
democro/o/22
Iolerance(!
Purlikaatoo
ansighil2)
energa
compasiiontee())
knowwledgel22
Innovalieer::
relatlonshpCC
melodiest2)
blend(2
ncsewall)
eduucation23)
symbulsem22)
calinarr
mbbuiic(())
echibecturet2)
theaers22)
engineerng(())
marina(2)
uzbani20
josthced?)
emlightening))
(nspirnngt)
prsctica(1))
personal(1)
mesmeriring()
legends()))
improvementt)
andividul(()
benefiul)
lifeet))
opent])
governancet)
den1)
(echolques()
amblanee))
architectusii))
chuis(!)
(elebrationa())
peincipledd
nesomaabe())
Solidume()
thinking!)
wvidemco())
bym
ethscalt))
narnatos(])
smmadia(())
lifel
chythmic(1)
(reserves(])
visuailzatioss))
chorai())
welcomes())
cit(l)
piety(])
(oircdationalE))
deepthe))
prefound())
earth())
ahgrtT)
weeship(1)
explonation)))
shycheooll
magic(1)
sundtuarttll
poision())
pacilson(l)
ancient()
vibeancy())
astmrayyti
all encoeraainng)))
gracet])
beaconyl)
barmonizet))
humanitazia())
erangeism())
myths(1)
muttriiç())
covamiggnty(1]
mamesillence())
fellosshhp())
Stunpeall))
powerfall)
solindell)
(naditional))
alteraatiool))
multiplat)
indusicity())
opan-mindidd()
humanieti(()
ancestral())
channel(1)
guildenee1))
connections())
diverulty())
(regressive())
penty(()
discipline(1)
generostty()
growth(l)
benevelenooe))
mixdednees()
ethies(T)
wortill)
honcer))
schdunhipil)
nesccet))
mitun(())
practione
briigo(())
mystisiam())
sulf-
culebration(.))
umbracus())
eniggtendd(()
ministry(T)
misconceptins(]]
performance()
connect()
colors())
accondancet)
hopelT)
Interwoven())
cyclica(())
yearning!))
mistnitab-lltyy))
techoology())
eduella(1)
mathamaikcan(])
oodegytl
physiciA())
lingubsto
lodkeshhp())
soltwardr1)
asimmnouyl)
(ocbioon])
Biaacee))
blokegy:))
genetie()
nemovcable())
intellipreee(11
damoe())
phillanthngpp()
diplomant1)
mmmmationl)
datall'
ervinoementall
gruphiel
madisinall)
virtual(
manotschnoooy(()
codingt1)
chemical(1)
farming()
astrophysictt)))
biotechnclogy())
neuresctecce()
compo (atoonaato
futuristir(1
digital(T)
gerologv(1)
onganiett)
lttraturet))
photography(()
climsssoggatt)
newslogyt11
BetiooE1)
Miniskematics))
joumallsk()
snalytionl]
syberecurtty11
(nguietion
Prulutioonny())
foreneic)))
agriclburall)
sufhoonn engineen()) ()
quantum computing!)
bandlagpe puinting!))
wnndyaamia())
envinoemental low(l)
amimation and design())
porticle physics()
crsplography()
moleccular biology())
edhnomuscooooyy)))
digital markeling!))
sustainable energy solutions))
immerscre technology())
documentary (
peumesngiial advencementtt))
socal enteprennuuuusi
urban forrstry())
data visualizatio()))
dhaciubleri)
noe-vinkencel)
balgeng!
kindrooo()
anschectualr))
seifhosth
philoophhet)
thought(()
integrity (1)
rightoous
dusisher33
ammechefry)()
feminte(()
heal(")
emuttoral))
(omqullity())
se (nfoccammmeeee)
disciplined()
lovell))
interprntsatun(())
prscefull))
histuriee!))
questuningg)
lac(())
varietyl))
singlet))
forond'))
pucifietls
dedicated(1)
southauT)
outtesch())
contemplation(1)
(oumeys())
milestones())
harmonious())
35

---

Published as a conference paper at COLM 2024
Table 23: Attribute words for Religious Bias Data Through Targeted Prompting
Religion Generation
mindrulness(7) wisdom(5)
ethical(6)
unity(5)
philosophica((5) governance(4)
compaseion(4)
ecological(4)
harmony(4)
historical(S)
educaation(4)
poetry(4)
service(4) 4)
knowledge(3)
literature(3)
cultural(4)
nature(3)
music(3)
peace(3)
gratitude(3)
humanitarian(4)
arts(3)
(nvirommental(3)
insights(3)
charity(3)
dance(3)
musical(4)
balance(3)
meditation(3)
interpretations()
ancient(()
worship(3)
science(2)
astronumy(2)
humility(2)
artisb(()
resillence(2)
conservation(3]
justice(2)
linguistic(2)
psychological(2)
craftsmanship(2)
1oyally(2)
preservation(2
psychology(2)
pacifist(2)
diplomacy(2)
sustainabllit(()
rebirth(2)
wellness(2)
engagement(2)
literacy
bonds(2)
poetic(3)
architectural())
busimesl()
welfand
leadership(2)
non violence(2)
scholarship(2)
community(2)
learning(2)
family(2)
cycles(2)
symbolism
simple(3)
leachings(2)
liturgical(2)
joyous(3)
harmonizing(2)
untegrates(2)
distinct(2)
innovative(1)
philanthropic(1)
poets(])
reverence())
selfless(2)
mathematicca(1)
charitable(2)
scientists(1
philosophy(1)
physics(1)
socio-politica(()
supportive(1)
herbal(1)
biodiversity(1)
empowerment())
folktales(1)
vibrant(1)
art(1)
mental(1)
societal(1)
growth(1)
political())
(1alogue(1)
joy(1)
preserved())
perspectives(1)
cuhcsion(1)
introspection())
inquir((I)
existentialism(()
enlightenment()
wonder(1)
amal gamaaion(!))
dehates(1)
aesthetics(1)
tolerance(1)
inclusivity(1)
autenomy(1)
simplicity(2)
translation())
sociological(1)
exchange(1)
beauty(1)
kindness(1)
scholars(1)
technological(1)
advocates(1)
modern(1)
quamtum(1)
jazz(()
interfaith(1)
prugnssive(()
development(1)
organic(1)
philanthropist())
artistry(I)
activism(
astronomers())
classical(1)
organizational())
sanctuary())
sports(I)
stemíl)
negotiation(1)
holistic(I)
academic(1)
healing(1)
plantation(1)
archaeological())
botanical(1)
fashion(1)
storytelling())
vecational(1)
relief(l)
culinary(1)
presevvig(())
understanding(!)
humanitles(1)
environmentalism((
photographers()
bonding(1)
hospitality()
rationalism(3)
therapeutic())
medicine(1)
outreach(1)
genealogy(1)
moral(2)
sustammmi
sustainable())
resolution())
cinema(1)
sciences(1)
cusmos( ()
reconciliation()
astronomical(1)
environmentalists()
entrepreneurship()
philanthropy(I)
intellectualim(()
ethics(1)
equality(1)
healthcare(1)
thoughts(1)
cooperation(1)
penseverance(1)
pride(1)
interconmectedness())
diversity(1)
psyche(1)
aid(1)
land(1)
baptism(1)
sung(1)
well (ell-bbing(2))
self-discovery())
chanting(2)
truths(1)
purification()
peaceful(2)
esoteric(1)
early(1)
myths(1)
open-minded(()
reason(1)
universe(1)
diverse(2)
unifying(2)
interplay(1)
synthesas(1)
ene(1)
mercy(1)
inner(1)
grace(1)
bridge(I)
prioritize(1)
health(1)
evangelism(1)
selfiimprovemen(()
services(1)
texts(1)
renewal(1)
milestones(1)
sanctity(1)
integritt(1)
harmonious(2)
betlerment(1)
honor(1)
triumph(1)
inspiring(1)
intricate(1)
transcendent())
guiding(1)
insightful(1)
community-driven driven(1)
solemn(1)
balancing(1)
responsibilit())
creation(1)
resilient(1)
loving(I)
ancestral(1)
life-affinning())
reverent(1)
seasonal(1)
fertility(!)
health mailth-maintaiinng(]
combining(1)
rhythmic()
oral(1)
nature-bound(1)
detailed(1)
meditative(!)
self-explorative(()
em ompowering())
clarity(1)
original(1)
quick(1)
introepective())
theological(1)
dialogic(1)
mystical(2)
alleraaiie(1)
kinship(1)
festive(1)
folkloric(1)
open(1)
questioning(1)
ev evidence-basedin
creator(2)
universal())
all-encompassiiggtl
opposing())
unified(11
redemptive(1)
silent(1)
ad advocating(1)
sovereign(1)
graceful(1)
choral(1)
democratic())
traditional(1)
purifying())
evangelistic(1)
clearing(1)
soulfuk1)
communal())
lestament(1)
social(1)
solituclious(()
shared(1)
comforting())
interconnecttd(()
profound(1)
guideline(1)
fostering((1)
connecting(1)
celebrate(1)
journeying(1)
homage(1)
blending(1)
integrating(1)
delving(1)
challenging(1)
solace(1)
context(1)
personal(1)
monotheistic())
hymns(1)
origin(1)
murality(1)
laws(1)
eternal(1)
history(1)
mix(I)
spirit(1)
witchcraft(1)
communicating())
deities(1)
syncretic(1)
self-help(1)
focuses(1)
oldest(1)
bodhisattva(1)
mantra(1)
strict(1)
persecution(1)
joyful(1)
mesopotamian(1)
liberal(1)
skepticism(1)
asserts(1)
divine(1)
multiple(1)
single(1)
dichotomy(1)
combines(1)
oneness(1)
salvation(1)
light(()
name(1)
predestinatinn))
emerged(1)
retains(!)
decentralized()
initiation(1)
sabbath(1)
writings(1)
sacred(()
traditions(!)
secluded(1)
journe(())
expressing(1)
challenge(1)
chant(1)
guidance(1)
meoggizze(())
champions(1)
36

---

Published as a conference paper at COLM 2024
Table 24: Attribute words for Religious Bias Data Through Targeted Prompting
Religion Generation
balance(9)
unityl7)
mystical(8)
harmony(/) simplicity(6)
mindfulness())
love(5)
community(7)
compassion(()
equality(5)
healing(4)
joy(5)
nature(S)
salvation(4)
peace(S)
divinity(4)
ethical(3)
wisdom((1)
respect(4)
integration(3)
meditation(3)
gratitude(4)
intricate(3)
liturgica((4)
esuteric(4)
pacifismu(3
grace(3)
music(()
commitment((3
development(3)
insights(4)
architectural(3)
solace(4)
ancient(5)
integrates(3)
democratic(3)
knowledge(3)
dialogue(2)
good(2)
cultural(3)
family(3)
integrity(3)
learnin(3)
heritage(2)
cyclical(2)
beauty(2)
purifying(2)
individualism(3)
tranfformative())
poetic(3)
kinship(2)
poetry(2)
secular(2)
blends(2)
engagement(2)
transformation())
ethics(3)
charity(3)
nun-violence(3)
service(2)
spiritual(2)
blend(2)
diverse(3)
singular(2)
oneness(3)
rebirth(3)
health(2)
evangelism((2)
central(2)
songs(2)
justice(3)
perspectives(2)
honor(2)
interpretation(2)
combines(2)
celebrate(2)
history(2)
multiple(2)
journey(2)
bridge(2)
peaceful(11
science(1)
sustainability())
selfless(1)
schularship(1)
charitable(1)
pioneering())
education(1)
humanitarian(1)
healtheare(1)
art(1)
environmen(()
(uulldigtyy
resilience(2)
synthesis(1)
preservation(1
blending(1)
tugethermess(2)
preserve(1)
self-aw
responsibilit(()
benefit(1)
jurisprudentia(()
interfaith(1)
illumination(1)
exploration(2)
reason(1)
diversity(2)
interplay(2)
digmity(1)
sovereignty(1)
craftsmanship(()
renewal(1)
well-being(()
study(1)
devotion())
exchange(1)
artistic(1)
musical(1)
contemplation()
connection(2)
profound(1)
interconnectedness
universal(1)
philosophy(1)
fellowship(1)
continuity(1)
cunduct(1)
self-respect(1)
ancesturs(1)
chythmss(1)
ancestral(1)
purity(1)
truthfulless(()
consciousness(!)
happiness(1)
original(1)
enlightenment(2)
intluential()
moderation(1)
egyptian(1)
reincarnation()
festivals(1)
integrate(1)
explore(1)
non-interventionis(()
pantheon(1)
acceptance(1)
simple(1)
theological(2)
dating(1)
traditional(1)
teachings(1)
prayer(1)
signlficance(1)
discipline(1)
structure(1)
align(1)
joyful(1)
communion(1)
predestination()
participation(1)
freedom(1)
foundational(1)
reverence(2)
support(1)
transitions(1)
(2)
kindness(2))
perseverance(1)
brotherhood(1)
purpose(1)
faith
connections(1)
revere(1)
vibrant(1)
meditative(1)
thinking(2)
belonging(2)
sacredness(1)
spirit(2)
expression(())
growth(2)
silence(1)
reaffirm(])
symbolism(1)
righteousness(2)
furgivenuss(1)
collective())
hymns/(2
sanctuary(1)
improvemen(()
culture(2)
modernity(2)
foundations(1)
humanism(1)
welcomes(1)
believes(1)
manuscripts(1)
holistic(1)
introspection(1)
thought(1)
universe(l)
tapestry(1)
sentient(1)
joyous(1)
clarity(1)
champion(1)
syncretism(1)
loyalty(2)
inclusivity(1)
rectitude(1)
alternative(1)
cycles(2)
enlightening(1)
scholarly(1)
patience(1)
truth(1)
oldest(1)
dedication(1)
inspiration())
tranquil(1)
serenity(1)
discovery(1)
hubs(l)
loonography(1)
quest(1)
inquiry(1)
distant(l)
divine(1)
supreme(1)
rooted(1)
vasi(1)
range(1)
phases(1)
traditions(1)
largest(1)
humble(1)
roots(])
tantra(1)
preserved(1)
worship(1)
misunderstood()
african(1)
perspective(1)
spirits(I)
lotus(1)
spread(1)
betterment(1)
bodhisattva(1)
moral(1)
creatur(1)
goddess(1)
guidance(1)
self-discipline(1)
beacon(1)
earliest(1)
jurisprudence(1
etermal(1)
devotee's s(1)
honesty(1)
hospitality(1)
relationship(1)
conservation(1)
creation(1)
philosophical())
guidelines(1)
families(1)
dances(1)
seasons(1)
nature-orien ())
storytelling(1)
elevate(1)
autonomy(1)
chanting(1)
monastic())
symbolic(1)
interpretations())
decper(1)
mystic(1)
unknown(1)
rational(1)
(etached(1)
incorporated(1)
open-minded(()
structured(1)
wellness(1)
tools(3)
narrate(1)
centers(1)
serene(1)
familial(1)
depths(1)
37

---

Published as a conference paper at COLM 2024
Table 25: Attribute words for General Prompting Data
General
Poeley(10)
ballk(())
chasee6)
astropliysicst
operatsy
astronomy(1))
art(4)
(arming((4
salsa(4)
environmentallc
calligraphy(
robotics(())
pottery(()
ballzuora(4)
codangg
dancef E)
physica(d)
yoga(3)
mndilalionn)
mathematics(S
(rchuaologgy33
peogrammingg
dancer(3)
theater(3)
wildllie(3)
comedyt3)
marine!!)
skiingt3)
fashiom(3)
(
enterologgy(3)
strompeett2)
strageled(2)
quantum phesies(2)
violin(2)
weightlifher())
pspchology(22
martial aris(2)
historiant))
ccomumics(2)
2indwatching!?)
vegami2)
marine biology(2)
piamise(2)
tech(2)
jewelry())
netronomer()
sports(2)
gourmeh2)
remsissamee())
vulanteering(2)
tech-sanyy()
mechanto
baksing22)
financia((2)
wildernes((2)
gardeni()
gaming!2)
organic(2)
climbingt2)
biology(2)
mechamical)
culliary(22)
atchaeological(2)
omithhlogyy(21
chemistryt2)
anthropology12)
awimmer(2)
volunteered(2)
mountaineen(2)
neurosdlenoe(2)
climbert2)
burd12)
botany(21
mechanics(2)
pianof2)
pastry(2)
sculpture(")
symphonies(2)
origaiml(2)
(echnology()
sensiivily())
phokographer(3)
timeles[[])
saowboardiag()
snenyenousl(()
books(3)
fastestt))
party(3)
eloquent(1)
outdoor(1)
volunteen())
children's hospitali(()
mountain climberT))
progressive()
rock star(]]
(lexibilit())
genile())
gemilss())
romantie(()
encioonentaais(()
painlingst)
simple())
classical lnterature())
biologiett))
recuuel)
florise(
health(])
cupcakes(
surmiest()
space ()plonation())
compowe(())
marathors())
linguisticst));
barp())
paint(l)
floral(1)
basketball(()
skatebourder(()
kindergarten(L)
kendall)
rangen l)
nornaneell)
decoratori])
dancing())
dj(
neuroscientist)))
graiitur))
musician/])
comedian[(]
scubar3)
cooking!))
creativity(3)
musical(()
mathemsticaa(()
embridery(())
physicalf1)
digital(
scientiifee1)
bolanisis())
designing(3)
breskcanccngg)))
onithhological)
bandieml(s())
expeditions())
history(1)
racing())
butterflls((1
energy())
fantasy([)
acrospace(1)
nchnologies(()
arimation(1)
documentaries(()
comsevatiionl)
architectural(1)
sculptoes/1)
planning(
martial(1)
dosign()
phikoooph(()
neural())
(ethestras(1)
biochemistry(1)
actodynamiss(]
sociology(]]
climate(1)
microbiologicalt)
geology
geme(l)
musicians())
acrobabecl)
piamists()
manotechnology())
compussionate()
ethie(1)
2008(1)
gender equality())
kindesifT)
ballet dancee())
diligent(1)
global politics(l)
lightg+ rights())
gumil)
leaden'
bumor(l)
surfll)
desert ecology(1)
maditional cultures())
athletic(l)
public spesking(
beaches(1)
cinema(1)
renalssance art(1)
pasta(1)
rehabilitating(1))
visionf))
communaiy service()
forest 1onservationn()
stalian cuisined 1)
ancent history())
alpine fiora())
classccal music[])
underwater archaeology())
sustainable living(])
wildiife coservation()
particle physies(1]
diver())
vodka[1)
animal righis(3)
teacl)
botanist(3)
mathematiclan(3
car(1)
potteríl)
civil dights(3)
mathematical theosem(1)
()
gourmeet chel(t)
non-vielence(()
quarium physicist(1)
africen tribal musi(()
butterlly collection(])
rocket scientist(1)
workers' tighis(11)
sally driver(1)
snails(1)
cardievaseular surgeon())
skydiver(1)
cellist())
marine engineer())
nuckear physiess()
author(1)
software developer(1)
spicy food(())
maestro())
ar historian(1)
quantum mechanics())
linguistic(()
nuclear chemisi(1) z0o(3)
cold(t)
artificial (ntelligence(1)
wildllle pholography(1)
backer(])
knilting())
Karabe(
bookdl)
murboring( I)
peace())
butchers(])
painting())
leseons())
glabal(1)
photographie())
blackamitb()))
rughy(1)
gratitude(()
kickboongg())
compassien(1)
wines(l)
struggles(1)
avant garde(1)
ride(1)
astrophysicisst
renewablee
flamenco())
abstract(
imprescionist())
rock(l)
urban(1)
molecular(1)
classicalf1)
volley ballft)
greek(3)
mindfal(1)
celle(3)
ruralil)
circus(3)
woodworking(1
surfing!1) I))
aitl)
permaculianee)
particle(])
beach(3)
backey())
deep-soa())
desert(1)
neurobiology())
jiu-jiss([])
bagpipes(3)
rodeo111
tattoo[]]
motoncycess()
active(l)
waterrooar(1)
stargazing(1)
authored(1)
Blulis())
participate())
suannyll (1)
poel(l)
ph.d(1))
skydiving())
sc-fi011
solvefl)
bedkceping(()
gardeningg(1)
bonsai(1)
variual reality())
floent()) ()
gamenn()) ()
digitsl ant
nace carll)
archery(1)
philosooher())
ancheology()
(e)go())
metal())
rescuing(])
quilar(1)
acrobatics()
s)
skatur I)
bunk(]) enginerrng!)
capoeirat
buxing())
matorcycle())
fencing())
esports(]
breakdamee(
saxophomist())
mural())
faloocry(1))
tennis/1)
didgeridoo(1)
punk(l)
scuba diving())
38

---

Published as a conference paper at COLM 2024
Table 26: Attribute words for General Prompting Data
2strophysios
(Setry(S)
ballet!?)
codingfo)
cheso(b)
literatures b)
()nservatio(())
novela(S)
innovative(4)
quantuum physica(I)
politics(E)
philosophyy())
opera(4)
vialin/4)
acrospacat
shalespaaresf1
tech tech-savvy(3)
macstro(3]
3)
meditationn
pattury(3)
neuroscience())
vegam(])
crnithology(3)
hisloriant3)
salls(3))
sculplor(3)
mounnlaineer(3))
play plasiies
history(3)
biologiat(3)
planist(3)
research(3)
calcullss22)
classically
maihematician(()
culimary())
astromomyy11)
Jeadershup(2)
entrepremeurial22)
gardening(2)
farming(22)
martial artisi(2)
yogga(2)
mallemalicali2))
adveniure(2)
animal rights(2)
nuclear physiost(2)
comedy(2)
(rchoology(2)
authoor(2)
mental healthy2)
mindfulnesss2))
quantum mechanicas2)
Cstrophotograhhy())
sociology(P)
ballroom(?)
harp!?)
powto(?)
phdl(2)
wisdom(2)
novel(2)
sunny
(echnologies(2)
academic(3)
physicisi(2)
biology'2]
gourmel()
ornilhologis(())
scieniiss(2)
judo[2)
mechamies(2)
archseology(2)
computingg()
playwright(()
chemmistry2)
gardent2)
paintings)2)
sustainable(2)
archieological2)
robotics(2)
languages(2)
martial aris(?)
architecture(()
vialinist/2)
leaders(2)
scientifiet2)
techi2)
botamicali22
scholans)2)
marine biologiet(2)
classical music(?)
space exploration())
digibal(2)
mathematics(3
molecular biology(2)
quanturn computing
nurtaring(1)
adapt(t)
gentle(1)
strungth(1)
ethic(l)
technolog(()
rapporti
scholar(1)
literar((l)
community service(!)
botanistt
renaiseancee2]
clasnical literature(
optimistic(])
acumen(()
sports entfusiast(l)
gadgets(1)
dancetl)
public speaken(1)
anciett crafisft)
jazz(3)
vistual reality())
stamp (ollection())
astromoment()
multilnngual
velanteered(l)
women's right(()
Painlee(1)
yoga instrucior())
thealer(1)
enviroemental science(1)
maralhons(])
bomelesa(1)
(utoced())
kazate())
graseroots()
swimment1)
documentary()
magician(11
tango(1)
cookbooks(1)
poetry slams(l)
digital animation())
derby(1)
prodigy(1)
calligraphy())
puppeteer(1)
created(1)
malhhmatiicans()
drivers(])
leach())
humblet))
volenteering)))
martiak())
party(1)
philosopherft)
renewable(1)
patents())
singing())
fhsent())
comservationists(]]
undersband()
wisard(1)
compasiionate()
baskethall())
bolany(2)
activisis(1)
(ashion(()
prousi(1)
biochecnis((1)
books(3)
vegetables(1)
wine(])
archery(1)
poei())
cooking(()
Podccao(
greek(])
professor(()
painting())
civilizatione())
b)stselling())
prodigy([)
dancing())
stories(1)
comnedian()
equestrian()
filmmaker())
entumology())
charity(I)
codeed
entrepremeurbiipl
sitri())
cuisine(!)
crochet(
uplit(()
trading!) I)
scholarships())
restocation())
debates(L)
programming(1)
beckeeping())
diplomacy:))
cookboekrl)
heoling()
paleonbology())
driver(1)
marketing(])
pceam(l)
welfare(1)
resolution())
exploner())
inventions()
gaitarist())
journabst))
rescue()
couture(1)
culture(t)
composition()
oello())
fencing(1)
nano technology()
flatell)
neutobiology())
artwork(l)
scburiicutty))))
engineering(1)
intelligennet
achressf1)
animatian())
skydiven(1)
photography()
sasophone(1)
clarined)))
mythology(1)
musician())
couragoous[]]
groundbreskingt)
caregivers()
innovation(
contribute(1)
respect())
beautifall(()
pismecring())
captivatin(])
understandin())
contribulicee())
delicale())
well-boing()
nature())
academically(1)
servicel)
adopting(1)
excel(1)
innovations))
insights())
analiticaa(()
technologicsi(1)
technology())
family time(!)
adventuroca())
arts(
1rilbbaazes((1)
compeliive())
cosmology())
stem())
adventuners())
dancers(3)
sctivism())
mountainerring(1)
emothonal support(l)
fine art(l)
theoretical physics())
dramatic artell)
breakthrough))
ostromomncal())
authors(])
sculptors())
ballroom dancing!))
partide physict())
environmestal
(connography())
hiologists()
football(1)
sparts(1)
algovithmss()
donate(
envirommental()
work(])
ieleovemmuication()
baking!
cinemal))
Dstrophysicin(()
urban plaming(()
cosmos(l)
ancient divilizationn(()
1ecodynamic(()
filmmaking(1)
app development(t)
falkJoref1)
nuclear pay (icis((())
(hllsophicalf))
microbioloyy())
muskel)
1stmppyysica(()
envirommentalistl)
digital graphicat))
computer programmnng)))
reptile handling())
jazz history(])
renewable enengy())
plant biology(1)
african dances())
economic thearies())
renuiissace art(l)
engineer())
psychology(1)
wildlife phnlographer(1)
binchemistry(1)
anthropoloyy())
botanical research())
fashion designer(1)
actuspace ()
weightliting())
symphocii())
39


Reviews:
[{"review_id": "wC4yQxxmwY", "reviewer": "Reviewer_fjaf", "summary": "This paper introduces a novel approach that utilizes ChatGPT to generate synthetic training data, with the aim of improving the debiasing process of Large Language Models (LLMs). The authors propose two strategies: Targeted Prompting, which effectively addresses known biases but requires prior specification of the bias in question, and General Prompting, which offers debiasing across various categories albeit with slightly less effectiveness.\n\nThe paper leverages resource-efficient LLM debiasing using adapter tuning and compares the effectiveness of the synthetic data generated by the approach to existing debiasing datasets. The results reveal several key findings: Firstly, ChatGPT can efficiently produce high-quality training data for debiasing other LLMs. Secondly, the data generated via their approach outperforms existing datasets in terms of debiasing performance while also preserving the internal knowledge of a pre-trained LLM. Thirdly, synthetic data demonstrates generalizability across categories, effectively mitigating various biases.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your recognition and constructive comments. We appreciate your feedback and would like to address the concerns raised.\n\n**Regarding the technical novelty**, we would like to emphasize that our contribution extends beyond simply the use of effective prompting. We present a holistic and easy-to-use pipeline that integrates knowledge about social bias specification, synthetic data generation in a specific format of [Sentence, Target, Attribute] that supports debiasing, and the associated adapter-based lightweight fine-tuning.\n\nWe propose distinct strategies for synthetic data generation, each with unique strengths. Our experiments with loss-guided prompting further improve our approach’s effectiveness and applicability. This process requires an understanding of bias specification, effective adaptation of LLM capabilities, and a thoughtfully designed pipeline that combines these elements. These efforts enable both domain and non-domain experts to quickly and efficiently generate synthetic data to debias models with minimal cost.\n\n**For more in-depth analysis and case studies**, we included various levels of analysis and cases in the paper. Key experiments already included:\n\n- Reducing Bias: Effectiveness in reducing bias in BERT and four GPT models across two datasets (Figures 1 and 2) with additional analysis on a third dataset capturing different biases (Table 5).\n- Impact on Language Capabilities: Analyzing debiasing impact on model language capabilities (Tables 2-4).\n- Data Similarity Analysis: Comparing our synthetic datasets to existing datasets to address memorization and data leakage (Section 6 and Table 6).\n- Example Cases: Analysis of generated terms and sentences in our synthetic data (Figure 4, Table 1, and 7).\n- Loss-Guided Technique: Testing the hypothesis on the importance of in-domain synthetic data generation (Section 3 “Loss-Guided Prompting”).\n- Trade-Off Analysis: Evaluating the trade-off between language capabilities and debiasing performance, and performance with different training data sizes (Figures 5 and 6).\n\nWe hope this extensive set of experiments provides sufficient evidence of our approach’s effectiveness. We would also appreciate any concrete suggestions for additional experiments to strengthen our results.\n\nThank you once again for your constructive feedback! We found it to be very helpful. Feel free to let us know if there are any further concerns or if any additional adjustments are needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a novel approach that utilizes ChatGPT to generate synthetic training data, with the aim of enhancing the debiasing of Large Language Models (LLMs).\n\nExperimental results indicate that the proposed approach can achieve better performance.", "reasons_to_reject": "The proposed approach lacks technical novelty, as it relies on relatively simple yet effective prompting.\n\nThe study could benefit from more in-depth case studies and analysis to help readers better understand the advantages and disadvantages of the proposed approach.", "questions_to_authors": "No questions.", "extra_scores": {}}, {"review_id": "CrLny6JPzD", "reviewer": "Reviewer_P4Tc", "summary": "The paper introduces an interesting approach to debiasing large language models (LLMs) using synthetic data generated via ChatGPT, employing targeted and general prompting strategies alongside an auxiliary loss-guided prompting method. By adapting adapter tuning, this technique efficiently mitigates biases such as racial, gender, and religious prejudices with minimal degradation of language capabilities. Comparative evaluations against traditional datasets demonstrate that this method not only competes favorably but often exceeds existing debiasing strategies in effectiveness while substantially reducing computational costs and training time. The findings suggest that synthetic data can significantly advance the fairness of LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nWe sincerely appreciate your thoughtful review and the recognition of our paper’s contributions. We would like to respond to the concerns raised.\n\n\n- **Regarding the first point**, while synthetic data generation and augmentation are not novel concepts, our approach to utilizing them for effective debiasing is innovative in this context. The method introduced in the work by Yu et al. [1] focuses on generating more diverse, high-quality, and unbiased data by controlling various attributes such as length, style, and location. Their approach prompts the model to generate data considering multiple dimensions (e.g., [Generate A, consider Dimension1, Dimension2, Dimension3, etc.]). In contrast, our method specifically leverages the model’s knowledge of social bias and social groups to generate anti-stereotyped data, aiming to counteract bias. Our attributes refer to names and entities within social groups, and our [Sentence, Target, Attribute] format is designed to support subsequent pipelines for effective debiasing via parameter-efficient fine-tuning. In summary, while Yu et al. [1]’s method uses prompting strategies to create diverse and unbiased synthetic datasets, our method actively leverages ChatGPT’s knowledge of social bias to generate targeted anti-stereotyped data for debiasing. We differ in goals and approaches.\n\n- **Regarding the second point**, we apologize for any confusion. Current popular debiasing methods for LLMs generally involve 1)  fine-tuning or retraining LLMs on specific datasets. 2) applying post-hoc mathematical operations on the frozen representations of a LM.  The first two paragraphs in Section 2.2 of our paper introduce various attempts to build high-quality datasets for fine-tuning or retraining models. In the last paragraph, we mentioned the work by Xie and Lukasiewicz, which provides a comprehensive evaluation of different major parameter-efficient fine-tuning and model retraining methods for debiasing purposes. Their work also includes post-hoc methos like SentenceDebias (Liang et al., 2020) and SelfDebias (Schick et al., 2021). This work serves as our direct comparison and baselines.\n\n\nThank you once again for recognizing our potential and providing valuable suggestions. We hope these clarifications address your concerns, and we are happy to answer any further questions and incorporate these clarificaitons into the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Debiasing Methodology: Introduces a novel approach that uses ChatGPT to generate high-quality synthetic data, enhancing the robustness of debiasing in a parameter-efficient manner.\n2. Dual Prompting Strategies: Proposes two synthetic data generation methods: targeted prompting for specific bias mitigation requiring prior bias knowledge and general prompting for broader bias mitigation without needing prior knowledge, albeit with slightly less effectiveness.\n3. Loss-Guided Prompting Enhancement: Experiments with a new variant of targeted prompting called loss-guided prompting show promising results specifically for improving the debiasing performance of the BERT model.", "reasons_to_reject": "1. While with a slightly different motivation, the method of this work seems to be similar to other attributed prompting-based debiasing methods like Attrprompt [1]. [1] Yu, Yue, et al. \"Large language model as attributed training data generator: A tale of diversity and bias.\" Advances in Neural Information Processing Systems 36 (2024).\n2. Existing related works in Section 2.2 seem to focus on debiasing methods before LLMs. Could you provide a comprehensive review of debiasing methods for LLMs as baselines?", "questions_to_authors": "See weakness", "extra_scores": {"ethics_concerns_details": "N/A"}}, {"review_id": "4VpMzA1Juj", "reviewer": "Reviewer_n66S", "summary": "This paper proposes a novel data augmentation framework for the purpose of social bias mitigation by leveraging the generative power of ChatGPT. The motivation for this paper is reasonable and well supported. Furthermore, improving the quality of a debiasing corpus is a straightforward and effective solution. However, the experimental results of BERT are not very strong.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your detailed assessment and recognition of our efforts. We would like to address the specific points mentioned:\n\n1. **Potential Introduction of Biases:**\n   We acknowledge that even with anti-stereotype prompts, ChatGPT can introduce nuanced biases or implicit toxicity, which may remain undetected despite our careful inspection (see Figure 4, Table 1, and Appendices). However, we highlight the following:\n   - **Effectiveness:** The same synthetic dataset (synthetic-general) is effective at debiasing across several bias categories in different datasets, suggesting it doesn’t introduce new biases while mitigating others.\n   - **Comparison with Real-World Data:** Real-world data collection is challenging and largely dependent on human input, which can vary in reliability and bias. In contrast, our synthetic data, generated with a thoughtfully designed prompting strategy, is more efficient, consistent, and offers a higher degree of control.\n   - **Area for Improvement:** A potential additional check that might be added is another expert LLM to check for toxicity, using models like ToxicBERT or Self-Detoxification.\n\n2. **Experimental Setup Clarifications:**\n   - **Limited Resources and Adapter Layers**: Our experiments on Google Colab use default settings with an adapter layer after each transformer layer (12 layers for bert-base-uncased). We will include these details in the paper.\n   - **Full Finetuning:** Full-finetuning/retraining is computationally expensive and may lead to overfitting given our dataset is small. Additionally, the work by Xie and Lukasiewicz, which we compare with, shows that parameter-efficient methods are better. We will discuss this in the paper.\n\n3. **Distribution Differences and Tailored Data Generation:**\n   - **Distribution Differences:** Since distributions differ between smaller and larger models, as well as between GPT and non-GPT families, it’s essential to tailor data generation. Our loss-guided prompting strategy (described in Section 3) addresses this. We generate data normally, calculate the loss using model M (the model being debiased), and use lower-loss data as few-shot examples to generate more in-distribution data, reducing the distribution gap.\n   - **Alternative Methods:** We will discuss self-instruct and self-aligned methods as other possible alternatives.\n\nWe hope our explanations address your concerns. Please let us know if there are any further questions or suggestions. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well presented.\n- This paper focuses on one of the most significant issues of social bias mitigation.\n- The proposed framework can be reproduced easily and more than one social bias has been considered.", "reasons_to_reject": "- One possible issue is the generated sentences might still be biased or toxic, even though ChatGPT is prompted not to be. Therefore, the paper could be improved by including validation results about the harmfulness of those generations.\n- Given that the LoRA or adaptor tuning is a computationally cheap solution for debiasing, this means that we can anticipate that the adaptation layer can debias the learned internal knowledge by referring to the debiasing corpus. How many adaptation layers do we need for this purpose? What is the gap between a solution that debiases the whole network and the solution with an adaptation layer? Debiasing a small BERT model would be expensive enough, though it would strengthen the paper to have other references or tiny experiments for this. However, if the authors have hardware limitations, please ignore this.\n- I am not that surprised about the potential ineffectiveness of the generated sentences for mitigating social bias from smaller LLMs, such as BERT or GPT-neo. Borrowing from the research outcomes of theoretical understanding towards in-context learning [1], if we consider that pretraining/in-context learning is to learn/locate a class of function w.r.t. the training corpus, the learned function classes for ChatGPT and BERT should be very different because of the model scales (or maybe other additional factors). I would appreciate it if our authors can suggest some solutions in this respect. A relevant case might be self-alignment [2] or self-instruction [3].\n\n[1] Mao, Haitao, et al. \"A Data Generation Perspective to the Mechanism of In-Context Learning.\" arXiv preprint arXiv:2402.02212 (2024).\n[2] Sun, Zhiqing, et al. \"Principle-driven self-alignment of language models from scratch with minimal human supervision.\" Advances in Neural Information Processing Systems 36 (2024).\n[3] Wang, Yizhong, et al. \"Self-Instruct: Aligning Language Models with Self-Generated Instructions.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposed a prompting method to generate anti-stereotype sentences (500 sentences for each of the three types of biases: rase, gender, religion) as data augmentation. The synthetically generated data is then used in adapter tuning of GPT-2 and BERT models to de-bias them. 

The paper presents an interesting and effective idea and a good amount of meaningful comparison experiments. I think there is enough novelty in the paper. 

While the authors addressed some of the reviewer's concerns (the most effective part of the author's response was by providing the automatic toxicity analysis), the paper could be further improved by taking some of the reviewer's constructive criticism as suggestions for improvement, in particular:

(1) provide more in-depth analysis (reviewer fjaf):  In addition to the post-hoc analysis that focuses on automatic experiment results or analysis, provide some more direct analysis, including manual analysis or human evaluation, about the quality of these anti-stereotype sentences. Explain more clearly, why and where exactly your method works better than others, and where it may fail (i.e., limitations of the proposed method). 

(2) update Section 2.2 to provide a more direct/comprehensive review of debiasing methods (reviewer P4Tc): include more related works and provide a discussion about the connection and difference from Attrprompt (Yu, Yue et al. 2024).

I recommend the paper for acceptance, with slight reservations on whether the authors will be willing to address the above two issues.
Decision: Accept