Probing Social Bias in Labor Market Text Generation by ChatGPT: A Masked Language Model Approach
Authors: Lei Ding
Yang Hu
Nicole Denier
Enze Shi
Junxi Zhang
Qirui Hu
Karen D. Hughes
Linglong Kong
Bei Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: social bias
LLM
NLP
sociology
labor market
Primary area: machine_learning_for_social_sciences
TL;DR: This study examines how ChatGPT-generated job applications may reinforce social biases in the labor market through language patterns with a novel bias evaluation framework.

Abstract: As generative large language models (LLMs) such as ChatGPT gain widespread adoption in various domains, their potential to propagate and amplify social biases, particularly in high-stakes areas such as the labor market, has become a pressing concern. AI algorithms are not only widely used in the selection of job applicants, individual job seekers may also make use of generative LLMs to help develop their job application materials. Against this backdrop, this research builds on a novel experimental design to examine social biases within ChatGPT-generated job applications in response to real job advertisements. By simulating the process of job application creation, we examine the language patterns and biases that emerge when the model is prompted with diverse job postings. Notably, we present a novel bias evaluation framework based on Masked Language Models to quantitatively assess social bias based on validated inventories of social cues/words, enabling a systematic analysis of the language used. Our findings show that the increasing adoption of generative AI, not only by employers but also increasingly by individual job seekers, can reinforce and exacerbate gender and social inequalities in the labor market through the use of biased and gendered language.

Probing Social Bias in Labor Market Text Generation by ChatGPT: A Masked Language Model Approach
Lei Ding' Yang Hu?, Nicole Denier Enze Shi Junxi Zhang', Qirui Hu Karen D. Hughes Linglong Kong'*, Bei Jiang"
Department of Mathematical and Statistical Sciences, University of Alberta, Canada Department of Sociology, Lancaster University, UK Department of Sociology, University of Alberta, Canada
Department of Strategy, Entrepreneurship and Management, University of Alberta, Canada
Department of Mathematics and Statistics, Concordia University, Canada Department of Statistics and Data Science, Tsinghua University. China (1ding1 nicole. denier, eshi, khughes, ,eten,looonnne..o..eeeeeee wwwebeeeelaeerrreeeeeeeaaaaaaaae ca yang. ac uk, junxi zhang@concordia.. ca, hqr20dmails is tsinghua. edu
Abstract
As generative large language models (LLMs) such as ChatGPT gain widespread adoption in various domains, their potential to propagate and amplify social biases, particularly in high-stakes areas such the labor market, has become pressing concern. AI algorithms are not only widely used in the selection of job applicants, individual job seekers may also make use of generative LLMs help develop their job application materials. Against this backdrop, this research builds on a novel experimental design examine social biases within ChatGPT- -egnerrtee job applications in response to real job advertisements. By simulating the process of job application creation, we examine the language patterns and biases that emerge when the model is prompted with diverse job postings. Notably, we present novel bias evaluation framework based on Masked Language Models to quantitatively assess social bias based on validated inventories of social cues/words, enabling a systematic analysis of the language used. Our findings show that the increasing adoption of generative AI, not only by employers but also increasingly by individual job seekers, can reinforce and exacerbate gender and social incqualities in the labor market through the use of biased and gendered language.
Introduction
The rapid advancements in generative Large Language Models (LLM) like ChatGPT (OpenAI 2023). mark significant technological shift. These models have only propelled the field of Natural Language Processing (NLP) but have also found widespread application across numerous sectors (Zhao et al. 2023 Yang et al.) 2024). However, these models are incorporated into social and economic practices, they bring to the fore critical ethical concerns, especially regarding their potential propagate and amplify existing social biases and attendant inequalities, particularly within high-stakes domains such as the labor market [Liang et al. 2022].
Recognizing the growing potential of generative AI in employment practices, research primarily aims to identify and understand the impact of biases in the application of generative LLM within the labor market. We focus particularly on ChatGPT, investigating how this widely used LLM influences the propagation of biases in job advertising and application processes.
"Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The complexity of automating bias evaluation in textual content poses significant challenges. Tra- ditional approaches in social sciences, such as content analysis, often rely on manual word counts from static lists [Gaucher et al. 2011], which may miss the subtleties and unlisted language cues that advanced NL.P technologies can detect. addition, by considering words individually, these traditional approaches often fail to capture the contextual meanings that emerge from the interplay of words within entire sentences. To address this limitation and build toward a more solid bias evaluation method, we develop novel bias evaluation algorithm called PRISM: Probability Ranking blas Score via Masked language model. PRISM involves masking words sequentially within texts and using the Masked Language Models (MLM) (Devlin et al. 2018 Liu et al. 2019] to predict the likelihood of alternative tokens, thus allowing us to assess bias with a ranking- based approach that leverages established word lists from social science research to provide contextual sensitivity, enabling a systematic and detailed analysis of language use.
Additionally, the inherently opaque nature of LLMs like ChatGPT, which function as black boxes without transparent access to their internal structures or parameters, adds another layer of complexity. We propose a method of probing these biases by simulating and analyzing how job seekers use ChatGPT to craft applications (output texts) in response to real job postings (input texts), as illustrated in Figure I This simulation reveals insights into the biases embedded within ChatGPT's training data and their potential impacts on real-world human resource practices.
Utilizing our PRISM algorithm in tandem with job posting and application text pairs, we explore the correlation between generated content and bias propagation. This comprehensive and novel simulation offers a distinctive lens through which to view how biases might influence the job application process.
In essence, this paper seeks to bridge the gap between rapid technological advancements and the ethical considerations raised by the use of generative LLMs. Through our research, we emphasize the importance of ensuring that Al use promotes core social values of fairness and equality in the labor market as these technologies become increasingly integral to our daily lives.
Our key contributions include:
We propose PRISM, a brand new paradigm for bias evaluation combines with validated word lists capturing directional cues (based on social science research) with MLM assess biases in texts. It advances existing methods in terms of efficiency, flexibility, robustness as well theoretical properties.
We draw on novel experimental design to probe the black-box of social biases ChatGPT models to understand both the biases inherent in their training data and their implications for real-world job application scenarios.
Analysis of bias across four different social dimensions demonstrates inherent biases in job postings are likely reproduced in ChatGPT-generated job applications, with tendency for the model to exacerbate and reinforce these biases.
This paper is structured as follows: we first review the current landscape of bias evaluation in NLP and social sciences. Following this, we introduce our bias scoring algorithm and provide experimental evidence supporting our methodology. We conclude with analysis of job postings and applications mediated by ChatGPT, evaluating our approach's ' broader applicability and discussing the social implications of our empirical findings.
Prompt: "Please
Joh Postingst... We currently have a part-time opportunity Registered Health Nurse...
Job Applications:...
write job application based the job post:"
am writing to apply for the exciting part-time opportunity for
PRISM
PRISM
Score
Correlation
Blas for Job Applications
for Job Postings
Figure 1: Overview of the paradigm for bias probing experimental design.

---

2 Background and Related Works
Bias & Fairness Evaluation in NLP The evaluation of bias within natural language processing (NLP) presents complex challenges. as methodologies vary significantly across studies [Garrido- Muñoz et al. 2021 Blodgett et al.) 2020]. Traditional approaches range from analyzing cosine similarity in word embeddings [Bolukbasi et al.] 2016) to diverse methods such correlation, clustering, classification, and visualization [Caliskan et al., 2017 Gonen and Goldberg 2019 Ding et al. [022 (hh et al. 2024]. Recent works have focused
detecting bias in language models that rely on manual sentence templates [Kurita et al., [019] or creating benchmarks that require high-cost crowd-workers [Nangia et al. 2020, Nadeem et al. 2027 and across various NLP downstream tasks
including text classification (De-Arteaga et al. 2019 B Blodget et al.) 2016), coreference resolution
(Zhao et al. 2018). natural language inference Dev et al. 20200] , and machine translation (Stanovsky
et al. 2019
Bias Evaluation for Text The domain of text bias evaluation is notably more challenging than evaluating the NLP models, often requiring extensive human expert intervention or resorting to simplistic and heuristic methodologies. Many existing approaches are also limited to specific types of bias, making them difficult to adapt other contexts. Dhamala et al. [2021] measure bias by computing the cosine similarity of word embeddings [Mikolov et al. [2013] Pennington et al.
+ 2014] with respect to the gender direction (he-she) [Bolukbasi et al. 2016) and averaging over sentences. Cryan et al. [2020] compare lexicon-based based approach and a fine-tuned BERT model with Crowdsourced label dataset. [Spinde et al.) [2021] developed a media bias dataset through costly
expert annotation, a process not easily generalizable to other domains. Raza et al. [0244] explore the use of named entity recognition for detecting biased words within texts. Yet this approach also requires the creation of costly labeled training data for each task and model training.
Labor Market Bias Evaluation in Social Sciences A substantial body of research has docu- mented prevalent gender stereotypes and their role in (re)producing inequalities gender segregation Kjeedstadd and Nymoen| 2012 England 2010], gender wage/promotion gaps [Blau and Kahn 2020). motherhood penalties Glauber 2018]. and fatherhood premiums [Kileeald|
- in the labor
market. Further research shows that gendered language plays a crucial role maintaining and reproducing gender stereotypes [González et al.) 20199] Psychological studies also show that women and men, given their gender socialization, tend to use and be attracted to different gendered languages and linguistic styles [Gaucher et al. 2011]. For example, women tend to employ and identify with a more communal language style, including the use of words related to social and emotional contexts [Bem] 1974). In contrast, masculine language is typically characterized by style that highlights agentic traits. Gendered language is found across a wide range of contexts, and in the labor market, features prominently in job advertisements, the language used in job applications and interviews, as well as performance management processes [Hu et al. 2024 2022a]. While existing research has often focused on gendered language from the labor demand side in terms of, for example, employers' wording of job advertisements [Hu et al.) 20226), far less attention has been paid to the language used by job candidates in response to job ad advertisemntts in order to secure a job, despite an increase in individual job seekers' use of ChatGPT. This study thus fills this important gap by assessing gendered languages from both the labor demand (job advertising) and supply (job application) sides. In doing so, it highlights the relational use of language in the job application process as a quintessential example of social interactions in action. It aims to explore and reveal the extent which gender biases are present and indeed circulated and exacerbated through the interplay between languages used in job advertisements and job applications.
Bias Evaluation Algorithm for Text
3.1 Motivations
When assessing social bias in textual content, previous methodologies often begin with a straightfor- ward approach: selecting keywords for simple frequency counts. For instance, this might involve comparing the total word count of feminine and masculine words. This technique is prevalent in psychological and sociological studies as described Section 2 More contemporary methods have advanced to include the use of static word embeddings to measure semantic similarities among words,
3

---

although these approaches still treat each word individually. To go further, researchers need to acquire expensive, labeled training data for specific tasks and do the model training.
In contrast, our objective to refine and further advance these existing approaches to measuring textual bias with three useful and practical settings:
Beyond merely analyzing each word individually, the algorithm should aim to consider the contextual meanings of entire sentences, allowing for a more nuanced and comprehensive view of the text.
The algorithm does not require costly human-labeled training data and circumvents the process of model training or fine-tuning. This aspect is particularly valuable in scenarios where the necessary labeled data is not readily available, allowing for more flexible and scalable applications.
The algorithm should incorporate established and rigorous word inventories from social sci- ence research to guide the bias calculation in contextually embedded and domain-speciee manner (e.g.. accounting for specificities of the labor market context). This incorporation of domain knowledge ensures that the assessments are both empirically grounded and contextually salient.
3.2
Problem Setup and Algorithm Implementation
Prediction Distribution
(P(m;] for each m; in Masculine word list Af
with words
[Wi, My '", w₂)
MLM
communicate: 0.18
Rank
Excellenn
leadership:
2
leadership
acting: 0.01
skills with ability motivate employees
Mask the ith word
Bear
Merge
0.27
0.18
0.02
0.01
Blas Score 7
cognitive: 0.02
caring: 0.0001
leadership communicate cognitive acting
200
(,)) for each f Feminine word list
Figure 2: An illustration of the paradigm for PRISM that uses word lists for directional cues with MLM to compute bias score for text.
In this section, we detail our algorithm under the settings introduced above. Given text T' comprising words T = {wi, W2, .... Wr, J. we iteratively mask each word w; and input the modified masked text T. f.... 1; [MASK]. w;+1..... into an MLM, which outputs the probability distribution the vocabulary for the masked position denoted P(- T.)..
Then, to obtain the direction signal for score calculation, we require two predefined word lists representing different contexts-suce as gender with feminine word list F = (fi..... fir and masculine word list M {mı...., mpMih. For each word in F and M, we obtain the probability from the distribution P(.IT()). This yields two sets of probabilities: PF (P(HITT,)) P(falTy))...... P( (T,)) and PM = [P(m]]T,)) P(mz (T,)))))........ P(m [T;)))
Next, we merge PF and PMr and filter the probabilities by taking the top percent of the probabilities, as lower probabilities represent less likely predictions by the MLM and thus contribute minimally to our analysis of bias. This step allows to focus on the most influential predictions which significantly determine the context the sentence.
Finally, we calculate the rank of each probability within this merged list. Let R(-) denote the rank function of the probability based on the merged list. For each word w, the text T using the word lists F and M is denoted R( ( and R( (m][T()) respectively. And the lower the rank indicates the higher the probability. The bias score for each word w; computed as the difference between the mean ranks of the two word lists:
S(w.)
1 Σ R(P(TTT))) [F
1 M
Σ R(P(m)TT.))
FEF
MEM

---

A positive score indicates a bias toward a masculine orientation, while a negative score suggests bias toward feminine orientation. This differential allows to detect the direction of the bias, providing deeper insights into how gender nuances are embedded within the language.
Finally, the overall bias score for the text T is the mean of the scores for all words in the text:
B(T)
S(w))
T2
This score quantifies the bias present in T. By analyzing these scores across various texts, we can evaluate both the extent and direction of linguistic bias, offering insights into the underlying gender biases conveyed through language. From a Bayesian perspective, in the absence of prior information, the mean serves robust estimator of central tendency. For future work, we recognize the potential benefit of incorporating additional information (e.g., part-of-speech tags, WordNet, etc.) to apply weighted average explore alternative aggregation methods, which could further enhance the performance the bias assessment. The overall algorithm is illustrated in Figure 2 Algorithm
and detailed in
Algorithm 1 PRISM:
Probability Ranking blas Score via Masked language model
Input: Text T with n words {W1, W2, Wn) Feminine word list F, Masculine word list M Ouput: Bias score B(T)
for each word w, in T do
2: Create Tú by masking in T
3: Predict distribution P(- Tw)) using MLM
4: Initialize P Pmerged =
5:
for all words in F UM do
6: Append {w, P(w (T))
to Pmerged
7: end for
8: Sort and filter Pmerged to retain top a% of entries
9: Calculate ranks for R( ((P(J(T()) and Ri ((m)(TT)) in the filtered list
S(w;) = [F] ESEF ((P(f(T()) [M] EmEm ((P(m())))
11: end for
12: B(T) -1 S(w:)
13: return B(T)
3.3 Methodological Benefits of PRISM
Efficiencc Our algorithm eliminates the need for costly data labeling and model training. By leveraging predefined word lists developed by existing sociological research, our method avoids the resource-intensive processes associated with supervised learning, such as gathering expert annotations and training models from scratch. This approach not only expedites deployment but also ensures that the algorithm can be scaled and adapted swiftly and economically, making highly practical for researchers and practitioners needing quick and reliable bias assessments in various settings.
Computational Flexibility The inherent flexibility of our method allows for the evaluation of bias across various dimensions simply by altering the word list cues. This adaptability means that different types of bias can assessed without the need to relabel data or retrain models, significantly reducing the time and resources required for analysis. Whether exploring gender. race, age, any other form of bias, our algorithm can adjust to new research questions with minimal adjustments. This also allows for the incorporation of substantively meaningful domain-specific word inventories from social science disciplines such as sociology, management studies, psychology. etc.
Robustness Robustness our method is two-fold. Firstly, we utilize ordinal measurements of word probabilities, focusing relative positions (ranking) rather than the values. This method effectively mitigates issues arising from the predominance of low probabilities within large pool of candidate words, which can lead to nonsensical outcomes. Secondly, our approach ensures robust
5

---

results across different MLMs. Unlike other scoring methods using raw probabilities for calculation, our rank-based bias score method remains consistent even when different MLMs produce varying output probabilities. This dual approach minimizes the influence of outliers and maintains reliability across various computational models.
Theoretical Properties Moreover, we can test whether MLM's predictions have the same distri- bution on two word lists (M and F). Consider two groups of probabilities scores, ( and (P(mlTya ( J MEM. representing the scores samples from distribution Pp and PM- The rank sums, denoted by _fer ((P( and ΣMEM ((P(m] (T))) respectively, allow to test the hypotheses Hąu Pp Pas versus Нд Pp * Pas- The null hypothesis holds if there is no statistically significant bias toward masculine or feminine language in a particular word w; The following theorem provides rigorous formulation of the test statistic and its asymptotic result.
Theorem
When FF and M are large. for each i E [m]] under Hao"
M 11) |FI[M((IF) + |M +1)
Σ R(P(mlT))) N
2
12
mEM
If further we have JM =|| for each i [n], under H
S(w;) N (0 2K
+11 3 )
Note that we interpret the prediction probabilities for the two word lists, (P(STT()) hee and (P) as scores that measure the association between the masked word and the words in the lists. A higher score indicates a stronger relationship. We also assume these word lists are sampled from larger sets of male or female-associated words. Thus, for each masked word, the scores (P(fIT()) ()/e fEF and [Tmm[T, hm) me are viewed as samples from two underlying distributions, PF and Par. Using the rank test from Theorem [] we can determine whether PF = Pas This test is particularly useful for detecting biased words sentences. For non-biased words, the associations with the two word lists should be similar, implying Pp = PM. However, for biased words, the associations differ significantly, resulting in Pp # Pas-
3.4 Algorithm Validation
To demonstrate the reliability of our scoring algorithm in identifying social biases within texts, we validate our method on two different tasks
Human Experts Validation This validation involved collaboration with six experienced profession- als from the fields of sociology and management science. Each coder manually labeled randomly selected subsample of job advertisements. Leveraging their extensive domain knowledge, these experts meticulously classified the advertisements, assessing them for levels of perceived gender bias. These categorical labels were then transformed into ordinal variables, enabling detailed statistical comparison with the results produced by scoring algorithm. This rigorous, expert-driven coding process ensured the reliability of our evaluation methodology.
We compute the Spearman rank correlation between the bias scores generated by our algorithm and the results from the manual labeling process. Spearman correlation coefficient of 0.8.5 was obtained (Figure 4ap. indicating a strong positive association between our algorithm's 's scores and the human experts assessments. This result validates the algorithm's capacity to accurately reflect human judgments of bias, confirming its effectiveness as a tool for social bias detection.
Benchmark Validation Further validation was conducted using the BIOS dataset (De-Arteaga et al. 2019), which comprises personal biographies categorized by gender and various occupations. We employed gender-specific word lists from [Konnikov et al. 2022), such as (man, his, he ...) versus { (woman, her, she...), ), as binary directional cues and designated gender as the ground truth label. Our algorithm demonstrated high performance, achieving an AUC of 0.97 classifying gender, as
2The code for the algorithm available at: https://github com/Lei Dii-iD7///hhaaaP bias/
Thhis correlation is notably higher compared to those typically observed in non-experimental social sciences.

---

illustrated in Figure 4b) The AUC, or Area Under the ROC Curve, measures the ability of our model distinguish between classes here, gender categories. This performance surpasses that of three baseline methods in [Dhamala et al. [021] that rely on unigram or word embeddings. highlighting the effectiveness and potential applicability of our bias detection approach in broader NI.P tasks.
Probing Methodology and Job Application Data Generation
Probing Methodology To explore the social biases inherent in ChatGPT, particularly in the context of the labor market, our study simulates the typical use case where job seekers employ ChatGPT to assist in drafting job applications. This approach allows us investigate not only the biases that may emanate from ChatGPT's training data but also to understand how these biases could potentially influence real-world job applicationhhrring processes.
Probing the social biases within ChatGPT presents several challenges. Firstly, ChatGPT's model operates a 'black box,' making it difficult to discern the internal processes that contribute to bias propagation. Secondly. the lack of access to the model's architecture or parameters further complicates direct examination. Therefore, our analysis adopts indirect method, employing our known bias evaluation algorithm to detect and quantify the biases exhibited by ChatGPT, thereby illuminating how these biases might manifest in practical applications.
Job Application Data Generation Our dataset comprises over 33K job postings collected from Linkedln, reflecting diverse range of industries and job types. To simulate realistic job application processes, we utilize the OpenAI API (GPT-3.5 Turbo, data collected on April 2024) to prompt ChatGPT with these job advertisements, instructing it to generate corresponding job applications for each job posting.
This method does more than replicate real-world scenarios where individuals respond to job post- ings- it also facilitates a comprehensive analysis of the generated texts across various sectors. By using job advertisements neutrality in the generated texts are attributable to the model's ingrained biases, rather than the content of the advertisements themselves. This setup is crucial for isolating the effects of ChatGPT's biases, allowing for an accurate assessment of bias presence and intensity using the quantifiable metrics provided by our bias score calculation method.
standardized prompts,
ensure that any observed deviations from
5
Analysing the Bias inside ChatGPT
5.1 Dimensions of Gender Bias
We begin by introducing the four gender dimensions, each defined by distinct set of gender-related word lists, which will form the basis of our analysis. In recent social science research, understanding gender bias involves not just recognizing the existence of biases but also evaluating their impacts in various contexts. Building on the framework proposed by Bem [1974], Gaucher et al. (2011), Konnikov et al. 2022). we utilize specialized word lists to apply our social bias analysis across four different dimensions. Each dimension not only helps identify specific instances of bias but also offers insights into the broader social and psychological dynamics at play.
The psychological dimension assesses language context leaning towards communal attributes (e.g.. "caring, "sympathetic," "attentive"" nateenivv""commnnif commonly associated with femininity, or agentic attributes (e.g., "authoritative," "active," "confident") typically linked to masculinity.
Psychological Cues:
We evaluate job descriptions and roles using word lists that categorize terms associated with "soft" and "social" skills for feminine orientation, and "time-compressed" and "stressful" tasks, such "multitasking," "pressure," "speed," for masculine orientation.
Role Description:
Work-Family Characteristies(WFCC)) This dimension examines employer policies and cultural expectations affecting gendered labor force participation, scrutinizing terms like "parental leave" and "flexible work" for feminine orientation versus "irregular and long work hours" and "weekend work" for masculine orientation.
Social Characteristics: We also analyze explicit gender references such as gendered pronouns and identity markers ("she," "he," "his," "her," "man").
7

---

5.2 Correlation Analysis
We first analyze the correlation of job postings and job applications across each dimension of gender bias. Our findings indicate consistent positive linear correlation between the bias scores of job postings and the ChatGPT-generated job applications. This trend suggests that the biases inherent in job postings are likely to be reproduced in job applications by generative AI, reinforcing and possibly amplifying the initial biases. This correlation is visually captured in Figure illustrating the potential for cyclical reinforcement of biases through the use of generative Al in job application practices.
Figure D presents the statistical parameters for each analyzed dimension of social bias. The strongest correlation is observed in the Social Characteristics dimension with a correlation coefficient of 0.777, indicating a very strong positive relationship. This is followed by the Role Description dimension, which shows a correlation coefficient of 0.708. Both of these correlations suggest significant potential for the biases in job postings be reproduced by Al in job applications in these dimensions.
The Psychological Cues and WFC dimensions exhibit lower but still substantial correlation coeffi- cients of 0.644 and 0.451, respectively. The slopes of these relationships indicate the rate which the bias scores from job postings predict those in job applications, with steeper slopes observed the Social Characteristics dimension. This analysis clearly supports the hypothesis that inherent biases in job postings are likely reproduced in ChaiGPT-generated job applications.
lob Application Score
Psychological Cues
lob Application Score
Role Description
Correlation: 0.544
Correlation: 0.708
vvllue: Stope: 0.189
vvllue
*
Stope: 0.484
22.0
-1.
1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2
job Post Score
Job Post Score
Job Application Score
Work-Family Characteristics
Application Score
1.5
Social Characteristics
1.00
Correlation: 0:451
cn: 0.777
Avalue:
Pvalue:
0.75
Slope: 0.316
Slope: 0.629
0.50
0.5
0.25
0.0
0.00
-0.5
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
-0.5
0.0
0.5
1.0
1.5
job Post Score
Job Post Score
Figure 3: Result scatter density plot, for each of the bias dimensions where the x-axis is the job posting bias score and the y-axis is the job applications bias score. Where the darker color means there are more dots. The p-value is the significance of the correlation coefficient.
5.3 Statistical Testing Analysis
In this section, we delve deeper into how ChatGPT influences bias reproduction within the job application process. Let X and Y, represent the bias scores of job postings and those of the job applications generated by ChatGPT, respectively. We denote the population mean and variance of X as yx and 2.. A score close to zero indicates minimal bias (i.e.. gender neutrality that is neither feminine nor masculine), higher positive score signifies a bias towards masculine language, and a lower negative score indicates bias towards feminine language. The aim is evaluate how ChatGPT may exacerbate or mitigate these biases. The histogram and summary statistics of the bias scores are in Appendix A.5
Shift in Mean We propose the following hypothesis tests to assess shifts mean:
Ho μa 2 By vs. H1 px < jyY
Using the Wilcoxon signed-rank test, we determine whether there is a significant change in the mean bias score from the job postings the applications.
Shift in Magnitude
For the magnitude of bias, we assess:
Ho |xx| Z |PY/
vs. H |xxl < |pyl
8

---

This test measures the central tendency of bias scores, examining if the absolute values (regardless of bias direction) decrease. The less the magnitude(i.e. closer to zero) the less bias it has.
Change in Variance
We also explore the variability in bias scores:
Ho : dx S oÝ vs. H1 H1:0x > of
This variance test, employing Levene's s test [Brown and Forsythe 1974] for equality of variances, explores whether ChatGPT produces job applications with more unitorm bias expressions compared to the job postings. It helps determine if there is reduction in variance, which would suggest that ChatGPT standardizes the use of gendered language cues. Such standardization could potentially reinforce specific gender biases more consistently.
Table I: Statistical testing results for each dimension. The mean result indicates whether the overall bias score is shifting toward the masculine (t) or feminine (4) direction. The magnitude result reveals whether the bias is moving toward zero (L) or away from zero (t). The variance assesses whether job application bias scores exhibit greater (1) or lesser (4) variance compared to the job postings. Please and 5 in Appendix for detail statistics.
refer to Table 2
Dimensions
Mean
Magnitude
Variance
Psychological Cues
Role Description
Work-Family Characteristics
Social Characteristics
Shift in Mean The testing for the mean shift in Table revealss significant findings across several dimensions. Except for Role Description, all other dimensions exhibit statistically significant shifts toward more masculine language. This indicates predominant inclination for ChatGPT to amplify the use masculine language in simulated job applications over and above the original job postings, possibly due its training historically biased data. This shift raises concerns about the consolida- tion and exacerbation of masculine language. Such biases in Al-generated content could perpetuate gender disparities in professional settings, emphasizing the need for interventions in AI training processes to address and correct historical biases. In contrast, the Role Description dimension shows mean shift toward less masculine direction, but the bias in job postings has already been shown to be skewed toward a very masculine direction. In this case, ChatGPT seems to help mitigate this extreme masculine bias.
Magnitude of Bias The magnitude of bias, assessed through the mean of the absolute bias scores, varies across the dimensions. The Psychological Cues and Role Description dimensions suggest that the overall intensity of bias regardless of direction does not increase. This could imply that while the direction of bias towards masculinity is pronounced, the degree of bias embedded within job applications does not intensify. Conversely, the WFC and Social Characteristics dimensions exhibit an increase in bias magnitude, indicating not only shift towards masculine language but also an overall increase in the strength of biased expressions. This finding is particularly troubling as it suggests that Al-generated job applications in these areas may become more polarized, further entrenching gender-specific expectations in roles traditionally associated with work-life balance and social interactions.
Variability in Bias Expression The variance results across all dimensions reveal consistent decrease in job applications compared to job postings. This decrease in variance suggests that the language used by ChatGPT is more uniform across different applications, potentially indicating a standardization of language that leans towards masculine expressions. Such uniformity in language use could narrow the range of expressions and perspectives presented in job applications, limiting diversity and potentially skewing hiring decisions in favor of male candidates.
5.4 Implications and Extended Analysis
Our statistical results underscore critical issue: biases in job postings are not merely replicated but are amplified in job applications created by generative AI in response to the postings. This

---

phenomenon can be explained by the reinforcement of initial biases through the language processing and text generation capabilities of Al tools like ChatGPT, which tend to replicate and often intensify the language patterns they are trained on.
Societal and Labor Market Implications:
The amplification of gender biases in Al Al-generated job applications has profound societal and labor market implications, suggesting that not only are stereo- typical roles perpetuated through biased language. but they are also strengthened when individuals use Al tools like ChatGPT to assist with drafting job applications. This use of generative Al plays a crucial role in circulating and amplifying biases, which reinforces, rather than challenges, the gender biases underpinning persistent gender inequalities in the workplace. Such biases can compound, influencing job satisfaction, employee retention, and career advancement. The misallocation of human resources due to biased AI could reduce economic efficiency and innovation, potentially causing sectors to overlook qualified candidates. Furthermore, these persistent inequalities may spur regulatory and legal challenges, especially in countries with robust equal employment opportunity laws, with significant implications for social ethics, justice, and economic equality.
Recommendations for Intervention: To mitigate the reproduction of gender biases through LLMs, is recommended that employers and AI developers implement more rigorous bias monitoring and mitigation strategies. This could include the use of debiased language models, regular audits of Al-generated content by independent third-party organizations, and the development of enhanced AI training datasets that reflect the diversity of the global job market. Additionally, public awareness and education initiatives should be promoted to increase understanding of Al's role job upplication and its potential impacts, fostering critical approach to Al tool usage in professional settings.
6 Conclusion
Our paper including a novel experiment, new algorithm development, and empirical application and findings - contributes to the ongoing debates and developments in the ethical use of AÍ in labor market processes and practices. By identifying underlying biases in Al-driven text generation, this paper proposes novel strategies and methods for detecting and mitigating such biases. Through our PRISM algorithm and empirical application, but are intended as actionable steps toward ensuring that the integration of AI in the labor market supports equitable and fair employment opportunities for both employers and job seekers.
show that these strategies are not just theoretical
Acknowledgements
This work was supported by the Economic and Social Research Council (ESRC ES/T012382/1) and the Social Sciences and Humanities Research Council (SSHRC 2003-2019-0003 under the scheme of the Canada-UK Artificial Intelligence Initiative. The project title is BIAS: Responsible Al for Labour Market Equality and the Principal Investigator in Canada is Linglog Kong. Bei Jiang and Linglong Kong were partially supported by grants from the Canada CIFAR Al Chairs program, the Alberta Machine Intelligence Institute (AMII, and Natural Sciences and Engineering Council of Canada (NSERC), and Linglong Kong was also partially supported by grants from the Canada Research Chair program from NSERC. Qirui Hu was supported partially by the National Natural Science Foundation of China award 12171 269. We thank Haizhou Yu, Qingfeng Lan, and Weitao Zhou for their valuable feedback on this paper. We also thank all the constructive suggestions and comments from the reviewers.
References
OpenAI. Chatgpt. Software, 2023. URL httL:ttts///wopeeeeeeee cww comm/hhttpt/
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. survey of large language models, 2023.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng. Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: survey on chatgpt and beyond. ACM Transactions Knowledge Discovery from from Data, 18(6):1-2,, 2024.
10

---

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan. Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint 1rXiv:2211.091 2022.
Danielle Gaucher, Justin Friesen, and Aaron C Kay. Evidence that gendered wording in job advertise- ments exists and sustains gender inequality. Journal of personality and social psychology, 101(1): 109, 2011.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /810. 1810.04805, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetrlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 1692, 2019.
Ismael Garrido-Muñoz, Arturo Montejo-Ráez Fernando Martinez-Santiago, and L Alfonso Ureña- López. A survey on bias in deep nlp. Applied Sciences, 11(7):3184, 2021.
Su Lin Blodgett. Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of bias" in nip. arXiv preprint arXiv: /4050, 2020.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, (6336)))))883311 2017.
Hila Gonen and Yoav Goldberg. Lipstick on pig: Debiasing methods cover up systematic gender biases word embeddings but do not remove them. NAACL-HLT, 2019.
Lei Ding, Dengdeng Yu, Jinhan Xie, Wenxing Guo, Shenggang Hu, Meichen Liu, Linglong Kong, Hongsheng Dai, Yanchun Bao, and Bei Jiang. Word embeddings via causal inference: Gender bias reducing and semantic information preserving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11864-11872 2022.
Enze Shi, Lei Ding, Linglong Kong, and Bei Jiang. Debiasing with sufficient projection: A general theoretical framework for vector representations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 5960-5975 2024.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, 2019.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. Crows-pairs, A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 1953 1953-1967, 2020.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, 2021.
Maria De- be-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: case study of semantic representation bias in high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128, 2019.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: A case study of african-american english. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119-1130, 2016.
11

---

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pages 15-20, 2018.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7659-7666, 2020.
Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
2019.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-We Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872, 2021.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
Jenna Cryan, Shiliang Tang, Xinyi Zhang, Miriam Metzger, Haitao Zheng, and Ben Y Zhao. Detecting gender stereotypes: Lexicon supervised learning methods. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1-11, 2020.
Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. Neural media bias detection using distant supervision with BABE- bias annotations by experts. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguisties: EMNLP 2021, pages 1166-1177 Punta Cana, Domini- can Republic, November 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021. findings-emnip.Jr URL
org/ 2021 findings- e.tee.mmmlp.1111 101
Shaina Raza, Muskan Garg. Deepak John Reji, Syed Raza Bashir, and Chen Ding. Nbias: natural language processing framework for bias identification in text. Expert Systems with Applications, 221:12, 2024.
Randi Kjeldstad and Erik Nymoen. Underemployment in a gender-segregated labour market. Economic and Industrial Democracy, 33(2):207-224, 2012.
Paula England. The gender revolution: Uneven and stalled. Gender & society, 24(2):149-166, 2010.
Francine D Blau and Lawrence M Kahn. The gender pay gap: Have women gone far they can?
In Inequality in the United States, pages 345-362. Routledge, 2020.
Rebecca Glauber. Trends in the motherhood wage penalty and fatherhood wage premium for low, middle, and high earners. Demography, 55(5): 1663-1686888 2018
Alexandra Killewald. A reconsideration of the fatherhood premium: Marriage, coresidence, biology, and fathers' wages. American sociological review, (8(1)9961166 2013.
M José González, Clara Cortina, and Jorge Rodríguez. The role of gender stereotypes in hiring: A field experiment. European Sociological Review, 3572):187-204, 2019.
Sandra Bem. The measurement of psychological androgyny. Journal of consulting and clinical psychology, 42(2):155, 1974.
Yang Hu, Nicole Denier, Lei Ding, Monideepa Tarafdar, Alla Konnikov, Karen D Hughes, Shenggang Hu, Bran Knowles, Enze Shi, Jabir Alshehabi AI-Ani, et al. Language job advertisements and the reproduction of labor force gender and racial segregation. PNAS nexus, 3(12):pgac526, 2024.
12

---

Shenggang Hu, Jabir Alshehabi Al-Ani. Karen D Hughes, Nicole Denier, Alla Konnikov, Lei Ding. Jinhan Xie, Yang Hu, Monideepa Tarafdar, Bei Jiang, et al. Balancing gender bias in job advertisements with text-level bias mitigation. Frontiers in big Data, 5:805713, 2022a.
Yang Hu, Monideepa Tarafdar, Jabir Alshehabi Al-Ani, Irina Rets, Shenggang Hu, Nicole Denier, Karen Hughes, Alla Konnikov, and Lei Ding. Gendered stem workforce in the united kingdom: The role of gender bias in job advertising. 2022b.
Alla Konnikov, Nicole Denier, Yang Hu, Karen D Hughes, Rebecca Deutsch, Lei Ding, Jabir Alshe- habi Al-Ani, Irina Rets, and Monideepa Tarafdar. Bias word inventory for work and employment (iv) equality and inclusivity (version 2.0). 2022.
Morton B Brown and Alan B Forsythe. Robust tests for the equality of variances. Journal of the American statistical association, 69(346):364-377, 1974.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987.
A Appendix & Supplemental Material
A.1
Additional Motivation and Real World Impact
The use of LLMs like ChatGPT to generate job applications is increasingly common among job seekers. This trend has been highlighted in discussions and concerns raised by employers and HR professionals, reported by high-profil news such as CNBC (May 6, 2024, article: "Exact same cover letters word for word: Career consultant says Gen Z are misusing AI") and Vox (Mar 8, 2023. article: "Maybe AI can finally kill the cover letter: Jobs still require cover letters. Apps like ChatGPT can help.") "). Moreover, the academic community is beginning to explore this use case, locusing both on its technical applications and ethical/bias implications. Our study contributes to this burgeoning body of literature by addressing key public, organizational, and scholarly concerns regarding the use of LLMs in generating job applications.
We explicitly identify the two major harms resulting from bias exacerbation in L.L. L.M-ennerated job applications:
Structural Harm: ALM--genrrated job applications that perpetuate gender stereotypes contribute to the reinforcement of gender inequalities embedded in language used in labor market processes. Job postings and applications are critical steps in these processes and play a significant role in the reproduction of gender inequalities. By failing to challenge these biases, LLMs can inadvertently support the perpetuation of these inequities.
As gender-biase biased LLM L.M-gnnerated job applications become part of the training data for future Al applications in HR(both in generating job postings and assessing applications), there is a risk of further entrenching gender biases in language use. This entrenchment can lead to cascading effects, such increased labor force gender segregation, which have significant societal implications.
Practical Harm:
A.2 Proof of Theorem
We assume each word from two word lists M and F are selected independently. Therefore, the first result in Theorem iss implied directly from the Wilcoxon rank sum test [Wilcoxon] 1945).
If | |F| K, the bias score S(w;) can be rewritten as
S(wi)
1 Σ R(P(ST()))
K Σ R(P(mlT())
K
fEF
MEM
RIPLITTN))) ]
2K(2K +1)
Σ R(P(TTT.))
Σ
JEF
JEF
2 DSEr JEr R(P(S((())
(2K +1),
(1)
K
13

---

where the second equality follows from the fact:
Σ R(PUIT))) + E R(P(m]]T))) = 2K(2K + +1)
2
JEF
MEM
From the first result in Theorem
we have
R(PUITA) ~ [[(2K+ +1) (K(2K+1) (((E))))~((()))) (2 +
Σ
2
12
FEF
for each i E [n], under Hς0- Therefore, under Hio, the result S(w;) ~ N (0, 2k+1
follows directly
from the relationship ①1 and the distribution of EeEF R R(P( (T
A.3 Algorithm Validation Result
Receiver Operating Characteristic Curves
Correletion plot (t 0.85)
True Positive Rate
Laboied Vakee
6-
o 400 ocp
PRISM (AUC 0.97)
Weighted Aug 0.91)
Gender Max (AUC 0.92)
Unigram score (AUC 0.9.)
False Positive Rate
(a) Human Expert Validation
Benchmark Validation
Figure 4: Algorithm Validation
labeled value of 7 signifies neutrality, while values less than suggest femininity, and values greater than 7 imply masculinity. A Bias Score close to zero indicates neutrality, positive value suggests masculinity and a negative value denotes femininity.
In Figure 4a
We follow three gender metrics for evaluating gender bias in texts in [Dhamala et al. 202T]. The first metric, unigram matching, counts gender-specifie tokens like 'he", "him', 'she". 'her' etc., and labels texts with more male tokens as male, more female tokens as female, and texts with equal counts as neutral. The second metric assesses words indirectly related to gender via normalized projection of word vectors in the gender direction, defined by she - he, using a Word2Vec embedding. Word-level gender scores are calculated as b, a weighted average (Gender-Waree Wavg):
In Figure 4b
These are aggregated either by Telll øl|
Σ Sim1 sgn (b,) Σ [b:]
Gender Wavg-
or by taking the score from the most gender-polar word (Gender-Max):
arg max((b.)).
Gender-Max sgg(b;)]88 )|;
Texts are classified as male if the score is less than -0.25 and as female if the score is greater than 0.25.
A.4 Human Expert Labeling Detail
To ensure the scientific rigor of the evaluation, we paid partieular attention to inter-rater validity and reliability. Specifically, each phase included individual labeling of data conducted by four independent
14

---

experts specializing in labor market inequalities associated with gender, work, and family, and Equity. Diversity, and Inclusion (EDI) in the labor market. In each round of labeling, individual labeling was followed by group sharing and discussion of the preliminary outcomes among the four experts. This combination of individual and group analysis allowed the team trace the score labeling and the validation of word inventory, ensure inter-rater reliability in each phase, and contextualize each phase within relevant scholarly literature, policies, and definitions.
The score labeling and the validation were further validated by three additional expert labelers (management, human resource, and social science scholars) from the team. Through double-blle blind labeling approach, the three additional experts independently assessed the dimensions and labels produced by the first four experts, demonstrating a high level of consistency. The scores and word lists were then finalized through further deliberation among the four experts and the three additional validators.
Because we used an iterative multi-round coding process, the inter-coder consistency rate in the developmental coding varied between 0.6 and 0.8. Notably, the final validation by three fresh validators within the team achieved high level of inter-coder consistency exceeding 0.8.
A.5
Histogram of Bias Scores
In Figure 5 dimensions.
we present the histogram of bias scores for Job Postings and Job Applications on different
A.6
Statistical Tests Results
Below we list the statistical test results for job postings and job application categories across different dimensions, with additional magnitude values, in Table 2
Table 2: Mean and Standard Deviation for job postings and job application categories across different
dimensions, with additional magnitude values.
Job Postings Magnitude
Job Applications Magnitude
Dimension
Mean
Std
Mean
Std
Psychological Cues
-0.030
0.552
0.703
0.039
0.427
0.533
Role Description
3.188
3,205
1.550
3,020
3,020
1.060
Work Work--Famlly Characteristics
0.455
0,476
0.290
0.513
0.515
0.203
Social Characteristics
0.435
0.484
0.355
0.550
0.569
0.288
Table 3: Wilcoxon Test Results for Mean Shift
Dimension
Statistic
P-yalue
H1
Psychological Cues
229792649.0
1.06
10
136
μx < BY
Role Description
316042300.0
0.0
μx > By
Work Work-Family Characteristics
207756451.0
0.0
μX < By
Social Characteristics
119987669.0
0.0
μX < By
Table Wilcoxon Test Results on Absolute Values
Dimension
Statistic
P-value
H1
Psychological Cues
353433005.0
0.0
uy
Role Description
319570996.0
0.0
P
py
Work-Family Characteristics
219288122.0
8.19 10 -1-213
PX
py
Social Characteristics
146778531.0
0.0
py
15

---

Psychological Cues
Role Description
2500
pib Rext Score Application Score
Joh Prel Score
Applicatoo Score
1750
2000
Frequency
Frequency
2250
1500
1000
2000
250
-3
-2
-2
0
0
Bias Score
Bias Score
Work-Family Characteristics
Social Characteristics
Iob Pesl Score jab Application Score
Job Post, Score
3500
job Applicatior Scom
3500
3000
Frequency
Frequency
2000
1300
1030
-.00
-15
-1.0
-0.5
0.0
1.5
2.0
Bias Score
Bias Score
Figure 5: Result histogram, for each of the bias dimensions, we use different colors to distinguish the Job Postings and Job Applications
Table 5: Levene's test results for variance between job and application data across different di- mensions, analyzed with one-sided interpretation. These results indicate significant differences in variance, with the job postings consistently showing greater variance compared to the job applications.
Dimension
Statistic
P- V-Vuue
H
Psychological Cues
1825.094
0.0
0 >
Role Description
3410.619
0.0
>
Work-Family Characteristics
2491.084
0.0
>
05
Social Characteristics
922.186
1.80 10
0x > 04
16

---

A.7 Ablation study
Additionally, conducted Control Experiments with Different MLMs, we selected two more MLMs: BERT-large and DiST The results for both the human validation and the LLM correlation plots Figure 7 These results show that changing the MLMMs produces consistent outcomes in the scatter density piot for evaluating bias for LLM, and statistical testing results are also highly consistent.
Psychological Cues
Job Appl Application Score
Role Description
Comeeeeioo: 0.697
Carrelation: 0.678
Job Application
Slope:
Pvalue
0:482
-2
0
2
-2
D
Post Score
Post Score
jab Application Score
Work Family Characteristics
job Application Score
Social Characteristics
1.0
Corretation: 0.469
Vvalue: 7.90e 154
Carrelation: O. 781 value Stepe: 0. 0.706
Slape: 0.361
0.5
-0.5
0.0
1.0
Job Post Score
Job Post Score
Figure 6: Result scatter density plot for DistilBERT
Job Application Score
Psychological Cues
Job Application Score
Role Description
Conriatiinn Slope:
Carrelation:
Stope: 0.478
Job Post Score
job Post Score
job Application Score
Work-Family Characteristics
JAb Application Score
Social Characteristics
Contelation: 0.478
1.0
Corteletior: 0 Pavalue: 7.51e-304
1.0
Avalue: 3.30-170
Slope:
0.5
Stope: 0.549
0.5
0.0
0.0
-0.5
1.5
Job Post Score
Job Post Score
Figure 7: Result scatter density plot for BERT Based Large
A.8 Experiment Setting Computational Resources
For our analysis, we utilize the pert-base-uncased" model from the Hugging Face library. The temperature parameter for the ChatGPT API is set its default value of 7. For the preprocessing. when we iterate through the text, we skip some of the function words like Articles ("a", "an". "the"), Prepositions ("on", "by"), Conjunctions( "and", "but", "if"), etc. our algorithm solely requires forward pass and no training, this enhances computational efficiency. To further optimize performance, we employ an Nvidia RTX A5000 GPU. All experiments are conducted on an Ubuntu server equipped with an AMD Ryzen Threadripper 3990X 64-Core Processor and 256 GB of RAM. The job post data publicly available at/htpss: www. htt.r///...oomtttttttse//ccccaaa
linkdin- jobs-dataset.
dataset
17

---

Correlation plot of distilBERT 0.79)
Labeled Value
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Bias Score
Correlation plot of BERT-Large 0.72)
Labeled value
-1.5
-0.5
0.5
Bias Score
Figure 8: Human Label Evaluation Correlation
18

---

A.9
Word List Details
The full word list is the integration of the existing literature in sociology, we directly adopted them without modification. These word lists are already publicly available. Below is the table for the size of each word list:
Table 6: Counts of gendered words in categories
Fem
Mas
psy
73
120
role
7
53
wfc
28
25
gsc
6
As well some job posting examples:
Table 7: Examples of Gendered Lexicon and Job Advertisement Excerpts
Gender
Example Lexicon
Example Job Advertisement Excerpts
Confident to work with high-caliber people.
Proven ability to be effective in a fast-paced, ambiguous, and changing environment.
Masculine
confident, effective, innova- tive, (pro)active, practical, pragmatic, problem-solving
Encourage new ideas and innovative approaches and actively share knowledge and experience to enhance the development of the team.
You have strong attention to detail. Responsible for the timely and accurate maintenance of accounting systems [town name]. High level of initiative, maturity, tact and diplomacy.
Feminine
attentive, accurate, timely, caring, polite, diplomatic
A.9.1 Limitations, Discussions, and Future Works
Tokenizer Issue: Discrepancies arise from the adaptation of word lists for use in MLMs due primarily to tokenization strategies. For example, the word "limitless" is tokenized into l'limit', *#lless'] }. which may affect the representation and subsequent analysis of such words in studies of linguistic bias.
Bias in MLMs: Addressing inherent social biases within MLMs presents significant challenge and offers fruitful directions for future research:
A primary consideration is our objective to evaluate bias in LLMs. Specifically, we examine the change in the bias score between the input and output text. By using the same MLM for both input and output, we can isolate and measure any additional bias introduced during the text generation process. This approach also helps control for the baseline biases present in the MLM being used for evaluation.
A more fundamental question is whether a biased model is essential for picking up the biased words to be able to detect bias. If so, arguably, the original BERT model, trained on extensive hal-world data, effectively mirrors socictal biases, thus providing realistic framework for identifying and analyzing these biases. But we still don't have a clear answer to this question.
In practice, achieving perfectly unbiased model is currently beyond reach. The pursuit of such an ideal model remains theoretical at best, as even the debiased ML.Ss carry traces of inherent biases from their training corpora.
All our results are based on an English dataset; however, additional complexities may arise with other languages due to more intricate word splitting tokenization challenges. The Masked Language
19

---

Models (MLMs) utilized in our study are sourced from public, open-source pre- pre-trained models. The inherent biases of these pre-trained models might impact our results, although we have attempted to mitigate this issue through robust rank-hased method that reduces sensitivity to changes in probability distributions. Currently, our job application generation relies on basic prompts; exploring the effects of varied prompts to capture broader spectrum of biases constitutes part of our future work. Moreover, while our framework is initially designed to use pairs of word lists, possesses the flexibility to accommodate single multiple word lists with minimal adjustments, an extension also plan to explore future research endeavors.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix A.9.I
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
In appendix A.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We detailed the algorithm in Algorithm
T
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
[Yes]
Justification:
The data is publicly available and we will upload the code
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr
oolicyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //iipe.cc/pullic/guuii wwwpc//ggue////eee CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In appendix A.8
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In section A.8
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Appendix A.8
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpoottttpp//orrs.....oooo//ppa hwwpprpp....ppp..oo.///ccccaaaaa
Question:
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 5.4
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't have any released model or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of them is properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will open source the code once the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't have crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "MBipDFGTi6", "reviewer": "Reviewer_e7QL", "summary": "This paper presents a novel bias detection algorithm, PRISM, and uses it to evaluate biases in job applications generated from prompts that include job postings. The technique uses a masked language model (MLM) to find the probability of “masculine” and “feminine” words replacing the masked word in the text. The bias score is then the difference between average ranks of feminine and masculine words, averaged over all words in the text. The authors find a correlation between biased job postings and biased generated text based on their evaluation method.", "questions": "Meta question: isn’t the assumption that certain words are “feminine” or “masculine” already a sort of gender bias? I suspect that the paper means that these buckets are words that are stereotypically or historically associated with certain genders, but if that is the case it should be made explicit. \n\nCan the authors publish the exact word lists used for the experiment?\n\nWhy are means of ranks employed as opposed to the means of the probabilities themselves? \n\nSimilarly, is averaging the S scores to compute B(T) the right approach? What do the actual distributions of S look like? Is the mean capturing the distribution faithfully?\n\nWhat version of ChatGPT was used for this experiment? Because the models are updated fairly frequently, just specifying the use of the API is not sufficient for reproducibility.", "limitations": "The paper discusses limitations briefly in the appendix, and there is a very brief discussion about the potential for biases in the MLM to affect the results. However, I believe there needs to be significantly more validation that this approach is sound based on the concerns I enumerated above.", "rating": 7, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "This paper’s primary strengths are that the work is well presented and the text is clear. The problem being tackled, gender bias in LLMs, is also one that is critically important and deserving of study.", "weaknesses": "I have two significant concerns about this work. First, I have a meta-concern about the setting in which the experiments are being conducted. Is generating a job application from a posting a common use case of LLMs? If there is indeed exacerbation of bias going from posting to application, what are the concrete harms of that manifesting itself? In short, I do not see much in the text motivating the work and helping us understand the real world impacts of these issues.\n\nSecond, I have a specific concern about the methodology employed. The paper claims that the PRISM method obviates the need for human labeling, and this seems to be accomplished through the use of the MLM and the previously curated lists of words. I am not convinced based on what I have seen in the paper that this method is actually measuring the level of bias in the generated text - it may instead be measuring the gender bias in the underlying MLM (bert-base-uncased) that is used to compute the score. It seems there is a fundamental chicken and egg problem. Is a “feminine” word ranked higher because the surrounding text that was generated is biased, or is it ranked higher because the MLM is biased to predict more feminine words in that context when a neutral word would actually be perfectly appropriate. How would the authors design control experiments to disentangle these two effects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed and thorough comments. Below please find our responses to specific comments:\n\n----\n\nW1: the use of LLMs like ChatGPT for generating job applications is indeed increasingly common among job seekers. This trend has been highlighted in discussions and concerns raised by employers and HR professionals, as reported by high-profile news such as CNBC (May 6, 2024, article: \"Exact same cover letters word for word: Career consultant says Gen Z are misusing AI\") and Vox (Mar 8, 2023, article: \"Maybe AI can finally kill the cover letter: Jobs still require cover letters. Apps like ChatGPT can help.\"). Moreover, the academic community is beginning to explore this use case, focusing both on its technical applications[1] and ethical/bias implications[2]. Our study contributes to this burgeoning body of literature by addressing key public, organizational, and scholarly concerns regarding the use of LLMs in generating job applications.\n\nWe explicitly identify the two major harms resulting from bias exacerbation in LLM-generated job applications:\n\n1. **Structural Harm**: LLM-generated job applications that perpetuate gender stereotypes contribute to the reinforcement of gender inequalities embedded in language used in labor market processes. Job postings and applications are critical steps in these processes and play a significant role in the reproduction of gender inequalities. By failing to challenge these biases, LLMs can inadvertently support the perpetuation of these inequities. \n\n2. **Practical Harm**: As gender-biased LLM-generated job applications become part of the training data for future AI applications in HR(both in generating job postings and assessing applications), there is a risk of further entrenching gender biases in language use. This entrenchment can lead to cascading effects, such as increased labor force gender segregation, which have significant societal implications.\n\n---\n\nW2: This is a really good question. We want to emphasize that our goal is to assess LLM bias, i.e., we are investigating the **change in the score** between input and output text. Using the **same MLM** for input and output text, allows us to isolate and measure the additional bias introduced by the LLM during the text generation process, thereby controlling for the baseline biases present in the MLM used for evaluation.\n\nAdditionally, we conducted **Control Experiments with Different MLMs**, we selected two more MLMs: BERT-large and DistilBERT. The results for both the human validation and the LLM correlation plots are in the additional PDF. These results show that changing the MLMs produces consistent outcomes in the scatter density plot for evaluating bias for LLM, and statistical testing results are also highly consistent.\n\nIndeed, the bias for each MLM affects the bias score calculation, but we would like to highlight that related research, such as using word embeddings for bias score[3], also faces similar questions regarding the bias inherent in the embeddings themselves. This is a big challenge in the field, but we do our best to conduct control experiments and validation methods to mitigate this issue.\n\n---\n\nQ1: You are correct about our intended meaning, that words are stereotypically or historically associated with certain genders. We clarified that our word lists capture feminine and masculine orientations that are grounded in and associated with gender norms and stereotypes, as supported by extensive sociological, psychological, and linguistics research[4,5,6]. Notably, while some of this work is more historical in nature(evidence from decades ago), more recent research cited below also establishes the continuing importance of gendered language in contemporary settings.\n\n---\n\nQ2: We would like to clarify that the word lists used in our experiment were not created by us specifically for this paper. Instead, we directly adopted them from existing literature across the social science research as referenced in [5,6,7], without modification. These word lists are publicly available, and we will publish the word lists along with the corresponding code.\n\nUsing pre-validated word lists is a key advantage of our method, allowing us to easily incorporate established, rigorous, and theoretically grounded word inventories from social science research (see lines 132–136).\n\nAdditionally, this paper also draws on the expertise of several co-authors specializing in sociology, management, and labor market research to provide further internal validation of the word lists.\n\n---\n\nQ3: Rank has several advantages over probabilities:\n\n**Theoretical Properties**: Ranks exhibit normality and allow for a rigorous formulation of the test statistic and its asymptotic result as presented in Theorem 1, which is based on the Wilcoxon rank sum test. Probabilities do not possess these important properties.\n\n**Sensitivity and Robustness**: Direct use of probabilities can be overly sensitive due to the highly skewed nature of prediction distributions. By using ranks, we can mitigate this sensitivity, and enhance the robustness of our results. This robustness enhancement also helps mitigate the inherent biases present in these pre-trained models.\n\n---\n\nQ4: From a Bayesian view, when we lack prior information, the mean serves as a robust estimator of central tendency. However, we acknowledge that for future work, incorporating prior information for a weighted average or other aggregation methods could potentially improve performance.\n\nThe asymptotic distribution of S is normal. To validate this, we have included a histogram of S in the PDF. Additionally, we conducted the Shapiro-Wilk test, which resulted in a p-value of 1.00, confirming that S is indeed normally distributed\n\n---\n\nQ5: We employed the GPT-3.5 Turbo, as it was the most accessible and widely used free version at the time of our experiment. This choice was made to ensure that our results are representative of the typical user experience.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBr2B4J3V0", "reviewer": "Reviewer_Bg7r", "summary": "This paper investigates the impact of social biases in the application of LLMs within the labor market. The authors examine biases in job applications generated by ChatGPT based on given job posts. They introduce a new bias evaluation method called PRISM (Probability Ranking bIas Score via Masked language model), which combines predefined gender-related word lists from social sciences with a masked language model approach to assess biases in input texts. Experiments with ChatGPT revealed that biases present in job postings are not only replicated but also amplified in job applications created by generative AI in response to these postings.", "questions": "1. Based on the scoring function in line between 155 and 156, why do you state \"A positive score indicates a bias toward a masculine orientation, while a negative score suggests a bias toward a feminine orientation.\"? Shouldn't it be the reversed reading? \n2. As the word lists for bias evaluation are predefined, how do you confirm their reliability?\n3. I am interested in the human evaluation part of sec 3.4. Could you include more details about the human experts and the tasks they have done maybe in the appendix, i.e. the demographics, the inter-annotator agreements, and the amount of tasks they did?", "limitations": "Same as the points in Weaknesses", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written. The authors have clearly made their scopes and the reading of the paper is very comfortable.\n2. This topic is novel and interesting and could serve as a good study for ML in social sciences.", "weaknesses": "1. Experiments are limited. Only ChatGPT has been applied. \n2. This paper aims to investigate the social biases, but no detailed exploration in different bias types has been conducted. From the algorithm, I only see the gender bias involved, which I think is only a part of the social biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for the thoughtful comments and heartwarming acknowledgment, as well as the time and effort devoted to the reviewing process. Below please see our response to the specific point in the reviewing comment.\n\n---\nW1: Our experiments were specifically conducted using ChatGPT due to its wide accessibility and prevalent usage. This choice was deliberate to ensure that our findings are representative of the typical user experience with one of the most commonly used models. However, we acknowledge the importance of testing additional models, and we agree that future work should include experiments with a broader range of models to validate and extend our findings.\n\n---\n\nW2: This is a great suggestion, pointing out an important direction for future research, which we have now acknowledged in our revised paper. We focused on gender, as gender inequality in the labor market is well-documented, persistent, and associated with gendered language, making this a prominent and exemplary type of social bias. We were also motivated in part to include just one aspect of bias due to the limited space of the paper. Adding additional dimensions would require providing background on each form of inequality, as well as locating established word lists of linguistic cues associated with those biases. In our revision, we have more clearly articulated that gender biases are chosen as a focal case of a broader range of social biases and that gender bias is important because it underpins one of the most prominent and persisting forms of inequality in the labor market. We have also suggested that future research explore additional forms of bias and labor market inequality\n\n---\n\nQ1:Thank you for your question. Because we assign lower numerical ranks to higher probabilities. As illustrated in Figure 2 in the paper, a rank of 1 represents the highest probability. Therefore, a lower mean rank indicates higher probabilities, while higher ranks (larger numerical values) correspond to lower probabilities. We have clarified this in our revised paper to ensure the scoring logic is explicitly defined.\n\n---\n\nQ2: The validity of our word lists is grounded in in three aspects:\n\n1. The word lists are established, widely used, and validated in the literature, specifically those developed by [5,6,7]. They are theoretically grounded in social science research.\n2. The word lists are specially informed by extensive sociological, psychological, and management research focusing on gender in the labor market.\n3. Lastly, we conduct a rigorous internal validation process, achieving a high level of intercoder reliability with experts specializing in sociology, management, and labor market research.:\n\n---\n\nQ3: Thank you for pointing this out. We will include this additional information in the appendix:\n\nTo ensure the scientific rigor of the evaluation, we paid particular attention to inter-rater validity and reliability. Specifically, each phase included individual labeling of data conducted by four independent experts specializing in labor market inequalities associated with gender, work, and family, and Equity, Diversity, and Inclusion (EDI) in the labor market. In each round of labeling, individual labeling was followed by group sharing and discussion of the preliminary outcomes among the four experts. This combination of individual and group analysis allowed the team to trace the score labeling and the validation of word inventory, ensure inter-rater reliability in each phase, and contextualize each phase within relevant scholarly literature, policies, and definitions.\n\nThe score labeling and the validation were further validated by three additional expert labelers (management, human resource, and social science scholars) from the team. Through a double-blind labeling approach, the three additional experts independently assessed the dimensions and labels produced by the first four experts, demonstrating a high level of consistency. The scores and word lists were then finalized through further deliberation among the four experts and the three additional validators.\n\nBecause we used an iterative multi-round coding process, the inter-coder consistency rate in the developmental coding varied between 0.6 and 0.8. Notably, the final validation by three fresh validators within the team achieved a high level of inter-coder consistency exceeding 0.8.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ljLJTH67q2", "reviewer": "Reviewer_ZJXv", "summary": "This research investigates the social biases present in ChatGPT-generated job applications using a novel experimental design that responds to real job advertisements. By simulating the job application creation process, the study examines the language patterns and biases that surface when the model is prompted with diverse job postings. Specifically, this paper presents a PRISM algorithm based on Masked Language Models to quantitatively assess social bias based on validated inventories of social cues/words. The results show that the increasing adoption of generative AI can reinforce and exacerbate gender and social inequalities in the labor market through the use of biased and gendered language.", "questions": "1. How the quality of the word lists will affect the bias evaluation results? \n\n2. The co-occurring frequency of some words are naturally generated, for example, in our common understanding, soft is more related to woman instead of man. Is it reasonable to treat it as bias?", "limitations": "The major limitation is the validation of the basic motivation of assessing social bias based on validated inventories of social cues/words.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper provides a comprehensive analysis of the bias problem in LLMs, clearly presenting the motivations behind the model design.\n \n2. A key contribution of this research is the development of a novel bias evaluation framework based on Masked Language Models, which quantitatively assesses social bias using validated inventories of social cues and words. This framework advances existing methods in terms of efficiency and flexibility, requiring minimal human labeling efforts.\n\n3. The model evaluation is convincing, featuring a well-designed experiment to probe the black-box of social biases in ChatGPT models.", "weaknesses": "1. The bias evaluation in this paper highly relies on the predefined bias-related word lists, e.g., the Feminine word list and Masculine word list, so how the quality of the word lists will affect the bias evaluation results? It would be better to make analysis on this aspect.\n\n2. The proposed method appears to be generally applied to any domain, although the major focus is on labor market text generation. Therefore, the characteristics of bias in labor market text generation need to be highlighted.\n\n3. There are some writing problems in this paper, for example, the table captions should generally be presented above the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewer for the time and effort in providing the comments and suggestions. Below, we address each of your points in detail.\n\n---\n\nW1: Thank you for your question, which we address in three aspects:\n1. **Use Case**: In our settings, we used pre-validated word lists that are widely used in existing research. Our method aims to provide a biased score based on words chosen by these lists that are grounded in expert knowledge of labor market processes across the disciplines of sociology, management/human resources, psychology, and linguistics. Therefore, we are confident in the validity and reliability of the widely used and social science expert-validated word lists. \n2. **Algorithmic Aspect**: The design of our algorithm incorporates robustness measures. Specifically, after combining the two sets of probabilities, we only take the top alpha percentage for calculation. \n3. **Numerical Experiment**: We have conducted additional experiments to assess the impact of the word-list quality. As shown in Figure 4(b) in the new PDF, the correlation with human labeling decreases as we remove more words from the word list. This additional analysis helps underscore the importance of a comprehensive and accurate word list for reliable bias evaluation results.\n\n---\n\nW2: This is an important point. We want to first explain that biases are contextual, i.e. some words are biased in one domain but not in another. So in order to be correct and precise, the word lists we use are grounded in multidisciplinary literature on labor market processes, which goes beyond generic language use that may not necessarily apply well to the labor market context. Our method is easily generalized to other domains by changing the word lists (to other domain-specific ones). Moreover, drawing on interdisciplinary collaboration, we believe our work is a good illustration of combining ML and social science, and we believe our paper fits well into the Neurips stream of ‘Machine learning for social sciences’.\n\n---\n\nW3: Thank you for pointing this out. We have fixed the issue you raised and moved the table titles so that they appear above the tables. \n\n---\nQ1:  same as W1\n\n---\nQ2:  In terms of the specific example you noted, established psychological methods and evidence shows that language associated with characteristics such as “soft” is perceived by readers as “feminine”, which tends to appeal to and resonate more with women rather than men. In the case of job posting and application, this would mean that (1) the use of feminine language (e.g., soft) in job postings tends to appeal more to women (than men) applicants; (2) the use of feminine language in job applications tends to be evaluated more favorably for a post stereotypically perceived to a feminine orientation. Therefore, our definition of bias is embedded in the broader social science and psychological literature on the “biased” impact of language use. We have clarified our approach to bias in our revised paper.\n\nThis question also pertains to the validity of the word list we use, which we clarify in three aspects:\n\n1. The word lists are established, widely used, and validated in the literature, specifically those developed by [5,6,7]. They are theoretically grounded in social science research.\n2. The word lists are specially informed by extensive sociological, psychological, and management research focusing on gender in the labor market.\n3. Lastly, we conduct a rigorous internal validation process, achieving a high level of intercoder reliability with experts specializing in sociology, management, and labor market research.\n\n\n\n---\n\nLimitation: We appreciate that some may view this as a limitation, but we view this as a strength. A great deal of social science research in psychology, linguistics, and sociology has established the relationship between language and social bias. We are able to draw on the strengths of decades of research to inform innovations in ML/NLP. Therefore, we believe our paper fits well into the Neurips stream of ‘Machine learning for social sciences’ by illustrating the strengths of interdisciplinary cross-fertilization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjvMclYGl6", "reviewer": "Reviewer_HqWb", "summary": "This paper presents a method to assess the extent to which social biases are encoded in LLMs. The motivating example throughout the paper is assessing the degree of gender bias in the context of job applications in the labor market. The method (PRISM) is designed as follows: given text (presumably output by the LLM), each word is masked out, one at a time, and the masked word is predicted using a masked language model. However, the only masked words predicted belong to one of two lists, corresponding to the two social groups (e.g. a list of masculine terms and a list of feminine terms). If the probability of words in one list are consistently larger than that of the other list, there is evidence for social bias.\n\nThe paper then uses PRISM to assess the social biases of chatGPT. chatGPT is prompted to produce job applications, and the method analyzes bias across four social dimensions relating to gender. The paper finds that biases in job postings are likely reproduced and even exacerbated in job applications generated by chatGPT.", "questions": "See section above, namely:\n- Is there evidence that the test isn't susceptible to social biases in masked language models?\n- What does PRISM find that isn't found by other methods?\n- Is there more evaluation/validation that the method is useful?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Strengths:\n\n- Important problem: the paper addresses the important problem of detecting and understanding social biases. The increasing adoption of LLMs by both employers and job seekers makes this a relevant issue. The potential of these models to amplify biases underscores the importance of studying this problem.  \n\n- Intuitive method: PRISM is an intuitive method that's easy to understand and explain. Moreover, it's flexible and efficient. All that's needed is word lists and an MLM to use for evaluation.   \n\n- Writing: The writing is clear throughout the paper, which helps make the method easy to understand.", "weaknesses": "The biggest weakness of the paper is the lack of evaluation. A new method is proposed for assessing social bias in LLMs, but only a small portion of the paper is dedicated to validating the new algorithm. While the validation exercises are a good start, they're limited; for example, the human experts validation is just limited to one dataset, with 6 human labelers and <100 examples. To see evidence that this is a general method, we need to see robustness across multiple application areas and experiments. \n\nRelated to the above, I can imagine a problem of this approach where biases of the masked language model are instead incorrectly attributed to the LLM generating the text. For example, if the masked language model used is biased (e.g. is likelier to predict male words than female words), then any sequence of text PRISM is evaluated on will be viewed as biased. It seems that for this test to be reliable, the masked language model should form calibrated and \"unbiased\" predictions of its own. Couldn't this be revealing biases in BERT rather than the LLM?  I'm not sure how likely this is, but it warrants discussion and validation in the paper; otherwise the limited evaluation is not compelling.\n\nAn additional weakness is that the experiments are limited. The experiments don't include baselines for assessing social bias. Is there evidence that PRISM is picking up on biases that other methods miss? Additionally, only one LLM is considered in the job posting experiment, and one MLM (BERT) is used as the MLM -- ablating the latter could help demonstrate the robustness of the method. \n\nA little less urgent, it would've been helpful to see more qualitative examples of the postings, along with more details for the experiments (e.g. the full word lists or at least the lengths of each word list).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed and thorough comments. Below please find our responses to specific comments:\n\n---\n\nW1:  We acknowledge and face challenges in evaluating social bias in text:\n1. The evaluation benchmarks in this field are scarce and underdeveloped so we can only turn to human labeling. This is also the key motivation for developing our method, which aims to provide a reference bias score for a field lacking domain-specific labeled data.\n1. Compared to related papers in this field, our evaluation is more robust. Many existing evaluations only rely on heuristics, such as counting gendered words like “he/she” [3,8].\n1. Evaluating gender bias is particularly challenging due to its subtle and multifaceted nature. This complexity makes it difficult to use crowdsourcing effectively, leading us to rely on expert labeling. Our paper significantly advances existing research by providing expert validation across multiple dimensions of gendered language in the labor market context. While not perfect, our approach represents the first of its kind to provide such comprehensive expert validation, addressing a critical gap in the literature.\n\nSpecifically,  we have used the following strategies for validation:\n1. For human evaluation, we work with experts in related fields (sociology, psychology, management, human resources), with an iterative multi-round coding process.  This ensures high labeling quality, especially compared to non-expert crowdsourcing workers (who may be less attuned to the intricacies of labor market dynamics). We have provided more details about the expert labeling process in the global response.\n2. In addition to human evaluation, we also conducted benchmark validation. The task has more than **250,000** examples and the dataset is from a different domain about classifying biography.\n3. Bias is inherently contextual; what is biased in one context may not be in another. To ensure accuracy and relevance, we focused specifically on the labor market, going beyond generic language use that may not apply well to labor market processes.\n\nFinally, the main goal of this paper is to evaluate the bias within LLMs. Please refer to our response to your next question for additional details.\n\n---\n\nW2: This is a really good and difficult question.\n\nFirst, we want to emphasize that our final goal is to assess LLM bias, i.e., we are investigating the **change in the score** between input text and output text. By using **the same MLM** for input text and output text, which allow us to isolate and measure the additional bias introduced by the LLM during the text generation process, thereby controlling for the baseline biases present in the MLM used for evaluation. \n\nAdditionally, we conducted **Control Experiments with Different MLMs**, we selected two more MLMs: BERT-large and DistilBERT. The results for both the human validation score and the LLM correlation plots are provided in the additional PDF. These results demonstrate that changing the MLMs produces consistent outcomes in the scatter density plot for evaluating bias for LLM, and statistical testing results are also highly consistent. Also in lines 507-510, the use of the ranking method helps mitigate bias within MLMs.\n\nIndeed, the bias for each MLM affects the bias score calculation, but we would like to highlight that related research, such as using word embeddings for bias score[3], also faces similar questions regarding the bias inherent in the embeddings themselves. This is a big challenge in the field, but we do our best to conduct control experiments and validation methods to mitigate this issue.\n\n\n---\n\nW3: We include baselines in the benchmark validation (see Lines 210-218 and 482-490). One baseline uses pure counting while the other uses static word embedding (word2vec). Our method is the first of its kind that uses contextual information with an MLM, i.e., the information of the whole sentence, other than looking at each word individually. From Figure 4b in the paper, we can observe a better performance compared with other baselines.\n\n---\n\nW4:  The full word list is the integration of the existing literature in sociology, specifically references[5,6,7] and we directly adopted them without modification. These word lists are already publicly available. And we will publish the wordlists and the corresponding code.\n\nBelow is the table for the length of each word list:\n| | Fem | Mas |\n|------|-----|-----|\n| psy| 73  | 120 |\n| role | 7   | 53  |\n| wfc  | 28  | 25  |\n| gsc  | 6   | 6   |\n\nAs well as some job posting examples:\n\n| Gender     | Example lexicon                                               | Example job advertisement excerpts                                                                 |\n|------------|---------------------------------------------------------|------------------------------------------------------------------------------------|\n| Masculine  | confident, effective, innovative, (pro)active, practical, pragmatic, problem-solving | - **Confident** to work with **high-caliber** people.  - Proven ability to be **effective** in a fast-paced, ambiguous, and changing environment. - Encourage new ideas and **innovative** approaches and **actively** share knowledge and experience to enhance the development of the team. |\n| Feminine   | attentive, accurate, timely, caring, polite, diplomatic | - You have strong **attention to detail**. - Responsible for the **timely** and **accurate** maintenance of accounting systems at [town name]. - High level of initiative, maturity, **tact** and **diplomacy**. |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

Thank you so much for your time and effort in reviewing our paper. We want to highlight some key aspects of our paper:
1. Our method represents the first research on unsupervised bias evaluation in text using contextual information.
2. Our word lists are based on established word inventories, grounded in social science/labor market theories and tried and tested in a wide range of research (cited below). Our team, particularly social science scholars in our team, have provided further expert validation of the word lists to doubly ensure the validity and robustness of the word lists.
3. Our work exemplifies the interdisciplinary synergy between machine learning and social science, aligning well with the NeurIPS stream of "Machine Learning for Social Sciences." We hope the reviewers will consider the significant social science impact and contributions of our paper.



---

Our team's human validation detail:

To ensure the scientific rigor of the evaluation, we paid particular attention to inter-rater validity and reliability. Specifically, each phase included individual labeling of data conducted by four independent experts specializing in labor market inequalities associated with gender, work, and family, and Equity, Diversity, and Inclusion (EDI) in the labor market. In each round of labeling, individual labeling was followed by group sharing and discussion of the preliminary outcomes among the four experts. This combination of individual and group analysis allowed the team to trace the score labeling and the validation of word inventory, ensure inter-rater reliability in each phase, and contextualize each phase within relevant scholarly literature, policies, and definitions.

The score labeling and the validation were further validated by three additional expert labelers (management, human resource, and social science scholars) from the team. Through a double-blind labeling approach, the three additional experts independently assessed the dimensions and labels produced by the first four experts, demonstrating a high level of consistency. The scores and word lists were then finalized through further deliberation among the four experts and the three additional validators.

Because we used an iterative multi-round coding process, the inter-coder consistency rate in the developmental coding varied between 0.6 and 0.8. Notably, the final validation by three fresh validators within the team achieved a high level of inter-coder consistency exceeding 0.8.

---

Reference:

[1] Al Shalabi, H., Al-Hashemi, R., & Al-Ramadin, T. A. (2013). Automatic Cover Letter Generator System from CVs (ACLGS). Global Journal of Computer Science and Technology Software & Data Engineering, 13(3).

[2] Bárány, A. (2023). From Humans to Machines: Can Artificial Intelligence Outsmart Human Job Applications?.

[3] Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K. W., & Gupta, R. (2021, March). Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 862-872).

[4] Coates, J. (2015). Women, men, and language: A sociolinguistic account of gender differences in language. Routledge.

[5] Bem, S. L. (1974). The measurement of psychological androgyny. Journal of consulting and clinical psychology, 42(2), 155.

[6] Gaucher, D., Friesen, J., & Kay, A. C. (2011). Evidence that gendered wording in job advertisements exists and sustains gender inequality. Journal of personality and social psychology, 101(1), 109.

[7] Konnikov, A., Denier, N., Hu, Y., Hughes, K. D., Alshehabi Al-Ani, J., Ding, L., ... & Tarafdar, M. (2022). BIAS Word inventory for work and employment diversity,(in) equality and inclusivity (Version 1.0). SocArXiv.

[8] Wang, R., Cheng, P., & Henao, R. (2023, April). Toward fairness in text generation via mutual information minimization based on importance sampling. In International Conference on Artificial Intelligence and Statistics (pp. 4473-4485). PMLR.

Meta-review: The review team mostly agrees that the paper is well-written and studies a novel topic within the emerging area of ML for the Social Sciences.

There are some concerns about whether the evaluation process is robust and appropriate for the problem. During the reviewers’ discussion after the rebuttal, the main concern was that the paper used an MLM to measure the bias of another LLM. The argument was that if the MLM is biased then it's possible that biases of the MLM are attributed to the LLM. Although a counter-argument was that measuring the change between the input text and output text partially addresses this issue, it doesn’t completely eliminate the concern.

Another secondary concern was that the human experts validation is just limited to one dataset, with 6 human labelers and <100 examples. In their rebuttal, the authors responded with several quite convincing arguments including that for human evaluation, they worked with experts in related fields (sociology, psychology, management, human resources), with an iterative multi-round coding process and that in addition to human evaluation, they also conducted benchmark validation with more than 250,000 examples and a dataset from a different domain. Illustrating these arguments in the paper would be helpful.

The review team pointed out several suggestions for revisions.

- Address the concern about using an MLM to measure the bias of another LLM. (See Review HqWb)
- Include more details about the human experts and the tasks they have done maybe in the appendix, i.e. the demographics, the inter-annotator agreements, and the amount of tasks they did. (See Review Bg7r)
- Include the predefined bias-related word lists (or at least details about) and better argue how the quality of the word lists may/ may not affect the bias evaluation results (see Review ZJXv)

Based on the information the authors provided in the rebuttal, I believe that at least the last two concerns are easily addressable. Overall, the strengths of the paper generally outweigh the aforementioned concerns.
Decision: Accept (poster)