Probing Social Bias in Labor Market Text Generation by ChatGPT: A Masked Language Model Approach

Authors: lei  ding
yang  hu
nicole  denier
enze  shi
junxi  zhang
qirui  hu
karen d hughes
linglong  kong
bei  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 22:57:33

Abstract: As generative large language models (LLMs) such as ChatGPT gain widespread adoption in various domains, their potential to propagate and amplify social biases, particularly in high-stakes areas such as the labor market, has become a pressing concern. AI algorithms are not only widely used in the selection of job applicants, individual job seekers may also make use of generative LLMs to help develop their job application materials. Against this backdrop, this research builds on a novel experimental design to examine social biases within ChatGPT-generated job applications in response to real job advertisements. By simulating the process of job application creation, we examine the language patterns and biases that emerge when the model is prompted with diverse job postings. Notably, we present a novel bias evaluation framework based on Masked Language Models to quantitatively assess social bias based on validated inventories of social cues/words, enabling a systematic analysis of the language used. Our findings show that the increasing adoption of generative AI, not only by employers but also increasingly by individual job seekers, can reinforce and exacerbate gender and social inequalities in the labor market through the use of biased and gendered language.

Main paper: 
 introduction: 
 The rapid advancements in generative Large Language Models (LLM) like ChatGPT [OpenAI, 2023] , mark a significant technological shift. These models have not only propelled the field of Natural Language Processing (NLP) but have also found widespread application across numerous sectors [Zhao et al., 2023 , Yang et al., 2024] . However, as these models are incorporated into social and economic practices, they bring to the fore critical ethical concerns, especially regarding their potential to propagate and amplify existing social biases and attendant inequalities, particularly within high-stakes domains such as the labor market [Liang et al., 2022] .Recognizing the growing potential of generative AI use in employment practices, our research primarily aims to identify and understand the impact of biases in the application of generative LLM within the labor market. We focus particularly on ChatGPT, investigating how this widely used LLM influences the propagation of biases in job advertising and application processes.The complexity of automating bias evaluation in textual content poses significant challenges. Traditional approaches in social sciences, such as content analysis, often rely on manual word counts from static lists [Gaucher et al., 2011] , which may miss the subtleties and unlisted language cues that advanced NLP technologies can detect. In addition, by considering words individually, these traditional approaches often fail to capture the contextual meanings that emerge from the interplay of words within entire sentences. To address this limitation and build toward a more solid bias evaluation method, we develop a novel bias evaluation algorithm called PRISM: Probability Ranking bIas Score via Masked language model. PRISM involves masking words sequentially within texts and using the Masked Language Models (MLM) [Devlin et al., 2018 , Liu et al., 2019] to predict the likelihood of alternative tokens, thus allowing us to assess bias with a ranking-based approach that leverages established word lists from social science research to provide contextual sensitivity, enabling a systematic and detailed analysis of language use. Additionally, the inherently opaque nature of LLMs like ChatGPT, which function as black boxes without transparent access to their internal structures or parameters, adds another layer of complexity. We propose a method of probing these biases by simulating and analyzing how job seekers use ChatGPT to craft applications (output texts) in response to real job postings (input texts), as illustrated in Figure 1 . This simulation reveals insights into the biases embedded within ChatGPT's training data and their potential impacts on real-world human resource practices.Utilizing our PRISM algorithm in tandem with job posting and application text pairs, we explore the correlation between generated content and bias propagation. This comprehensive and novel simulation offers a distinctive lens through which to view how biases might influence the job application process.In essence, this paper seeks to bridge the gap between rapid technological advancements and the ethical considerations raised by the use of generative LLMs. Through our research, we emphasize the importance of ensuring that AI use promotes core social values of fairness and equality in the labor market as these technologies become increasingly integral to our daily lives.Our key contributions include:• We propose PRISM, a brand new paradigm for bias evaluation combines with validated word lists capturing directional cues (based on social science research) with MLM to assess biases in texts. It advances existing methods in terms of efficiency, flexibility, robustness as well as theoretical properties.• We draw on a novel experimental design to probe the black-box of social biases in ChatGPT models to understand both the biases inherent in their training data and their implications for real-world job application scenarios.• Analysis of bias across four different social dimensions demonstrates inherent biases in job postings are likely reproduced in ChatGPT-generated job applications, with a tendency for the model to exacerbate and reinforce these biases.This paper is structured as follows: we first review the current landscape of bias evaluation in NLP and social sciences. Following this, we introduce our bias scoring algorithm and provide experimental evidence supporting our methodology. We conclude with an analysis of job postings and applications mediated by ChatGPT, evaluating our approach's broader applicability and discussing the social implications of our empirical findings.Figure 1 : Overview of the paradigm for bias probing experimental design. 
background and related works: 
 Bias & Fairness Evaluation in NLP The evaluation of bias within natural language processing (NLP) presents complex challenges, as methodologies vary significantly across studies [Garrido-Muñoz et al., 2021 , Blodgett et al., 2020] . Traditional approaches range from analyzing cosine similarity in word embeddings [Bolukbasi et al., 2016] to diverse methods such as correlation, clustering, classification, and visualization [Caliskan et al., 2017 , Gonen and Goldberg, 2019 , Ding et al., 2022 , Shi et al., 2024] . Recent works have focused on detecting bias in language models that rely on manual sentence templates [Kurita et al., 2019] or creating benchmarks that require high-cost crowd-workers [Nangia et al., 2020 , Nadeem et al., 2021] and across various NLP downstream tasks including text classification [De-Arteaga et al., 2019 , Blodgett et al., 2016] , coreference resolution [Zhao et al., 2018] , natural language inference [Dev et al., 2020] , and machine translation [Stanovsky et al., 2019] .Bias Evaluation for Text The domain of text bias evaluation is notably more challenging than evaluating the NLP models, often requiring extensive human expert intervention or resorting to simplistic and heuristic methodologies. Many existing approaches are also limited to specific types of bias, making them difficult to adapt to other contexts. Dhamala et al. [2021] measure bias by computing the cosine similarity of word embeddings [Mikolov et al., 2013 , Pennington et al., 2014] with respect to the gender direction ( -→ he--→ she) [Bolukbasi et al., 2016] and averaging over sentences. Cryan et al. [2020] compare a lexicon-based approach and a fine-tuned BERT model with a Crowdsourced label dataset. Spinde et al. [2021] developed a media bias dataset through costly expert annotation, a process not easily generalizable to other domains. Raza et al. [2024] explore the use of named entity recognition for detecting biased words within texts. Yet this approach also requires the creation of costly labeled training data for each task and model training.Labor Market Bias Evaluation in Social Sciences A substantial body of research has documented prevalent gender stereotypes and their role in (re)producing inequalities -gender segregation [Kjeldstad and Nymoen, 2012, England, 2010] , gender wage/promotion gaps [Blau and Kahn, 2020] , motherhood penalties [Glauber, 2018] , and fatherhood premiums [Killewald, 2013] -in the labor market. Further research shows that gendered language plays a crucial role in maintaining and reproducing gender stereotypes [González et al., 2019] . Psychological studies also show that women and men, given their gender socialization, tend to use and be attracted to different gendered languages and linguistic styles [Gaucher et al., 2011] . For example, women tend to employ and identify with a more communal language style, including the use of words related to social and emotional contexts [Bem, 1974] . In contrast, masculine language is typically characterized by a style that highlights agentic traits. Gendered language is found across a wide range of contexts, and in the labor market, it features prominently in job advertisements, the language used in job applications and interviews, as well as performance management processes [Hu et al., 2022a] . While existing research has often focused on gendered language from the labor demand side in terms of, for example, employers' wording of job advertisements [Hu et al., 2022b] , far less attention has been paid to the language used by job candidates in response to job advertisements in order to secure a job, despite an increase in individual job seekers' use of ChatGPT. This study thus fills this important gap by assessing gendered languages from both the labor demand (job advertising) and supply (job application) sides. In doing so, it highlights the relational use of language in the job application process as a quintessential example of social interactions in action. It aims to explore and reveal the extent to which gender biases are present and indeed circulated and exacerbated through the interplay between languages used in job advertisements and job applications. 
motivations: 
 When assessing social bias in textual content, previous methodologies often begin with a straightforward approach: selecting keywords for simple frequency counts. For instance, this might involve comparing the total word count of feminine and masculine words. This technique is prevalent in psychological and sociological studies as described in Section 2. More contemporary methods have advanced to include the use of static word embeddings to measure semantic similarities among words, although these approaches still treat each word individually. To go further, researchers need to acquire expensive, labeled training data for specific tasks and do the model training.In contrast, our objective is to refine and further advance these existing approaches to measuring textual bias with three useful and practical settings:• Beyond merely analyzing each word individually, the algorithm should aim to consider the contextual meanings of entire sentences, allowing for a more nuanced and comprehensive view of the text.• The algorithm does not require costly human-labeled training data and circumvents the process of model training or fine-tuning. This aspect is particularly valuable in scenarios where the necessary labeled data is not readily available, allowing for more flexible and scalable applications.• The algorithm should incorporate established and rigorous word inventories from social science research to guide the bias calculation in a contextually embedded and domain-specific manner (e.g., accounting for specificities of the labor market context). This incorporation of domain knowledge ensures that the assessments are both empirically grounded and contextually salient. In this section, we detail our algorithm under the settings introduced above. Given a text T comprising n words T = {w 1 , w 2 , . . . , w n }, we iteratively mask each word w i and input the modified masked text T \i = {. . . , w i-1 , [MASK], w i+1 , . . .} into an MLM, which outputs the probability distribution over the vocabulary for the masked position i, denoted as P (• | T \i ).Then, to obtain the direction signal for score calculation, we require two predefined word lists representing different contexts-such as gender with a feminine word list F = {f 1 , . . . , f |F | } and a masculine word list M = {m 1 , . . . , m |M | }. For each word in F and M , we obtain the probability from the distribution P (•|T \i ). This yields two sets of probabilities:P F = {P (f 1 |T \i ), P (f 2 |T \i ), . . . , P (f |F | |T \i )} and P M = {P (m 1 |T \i ), P (m 2 |T \i ), . . . , P (m |M | |T \i )}.Next, we merge P F and P M and filter the probabilities by taking the top α percent of the probabilities, as lower probabilities represent less likely predictions by the MLM and thus contribute minimally to our analysis of bias. This step allows us to focus on the most influential predictions which significantly determine the context of the sentence.Finally, we calculate the rank of each probability within this merged list. Let R(•) denote the rank function of the probability based on the merged list. For each word w i in the text T using the word lists F and M is denoted as R(P (f |T \i )) and R(P (m|T \i )), respectively. And the lower the rank indicates the higher the probability. The bias score for each word w i is computed as the difference between the mean ranks of the two word lists:S(w i ) = 1 |F | f ∈F R(P (f |T \i )) - 1 |M | m∈M R(P (m|T \i ))A positive score indicates a bias toward a masculine orientation, while a negative score suggests a bias toward a feminine orientation. This differential allows us to detect the direction of the bias, providing deeper insights into how gender nuances are embedded within the language.Finally, the overall bias score for the text T is the mean of the scores for all words in the text:B(T ) = 1 n n i=1 S(w i )This score quantifies the bias present in T . By analyzing these scores across various texts, we can evaluate both the extent and direction of linguistic bias, offering insights into the underlying gender biases conveyed through language. From a Bayesian perspective, in the absence of prior information, the mean serves as a robust estimator of central tendency. For future work, we recognize the potential benefit of incorporating additional information (e.g., part-of-speech tags, WordNet, etc.) to apply a weighted average or explore alternative aggregation methods, which could further enhance the performance of the bias assessment. The overall algorithm is illustrated in Figure 2 , and is detailed in Algorithm 1. for all words w in F ∪ M do 6:Append {w, P (w|T \i )} to P merged 7:end for 8:Sort and filter P merged to retain top α% of entries 9:Calculate ranks for R(P (f |T \i )) and R(P (m|T \i )) in the filtered list 10:S(w i ) = 1 |F | f ∈F R(P (f |T \i )) -1 |M | m∈M R(P (m|T \i )) 11: end for 12: B(T ) = 1 n n i=1 S(w i ) 13: return B(T )Efficiency Our algorithm eliminates the need for costly data labeling and model training. By leveraging predefined word lists developed by existing sociological research, our method avoids the resource-intensive processes associated with supervised learning, such as gathering expert annotations and training models from scratch. This approach not only expedites deployment but also ensures that the algorithm can be scaled and adapted swiftly and economically, making it highly practical for researchers and practitioners needing quick and reliable bias assessments in various settings.The inherent flexibility of our method allows for the evaluation of bias across various dimensions simply by altering the word list cues. This adaptability means that different types of bias can be assessed without the need to relabel data or retrain models, significantly reducing the time and resources required for analysis. Whether exploring gender, race, age, or any other form of bias, our algorithm can adjust to new research questions with minimal adjustments. This also allows for the incorporation of substantively meaningful domain-specific word inventories from social science disciplines such as sociology, management studies, psychology, etc.Robustness Robustness in our method is two-fold. Firstly, we utilize ordinal measurements of word probabilities, focusing on relative positions (ranking) rather than the values. This method effectively mitigates issues arising from the predominance of low probabilities within a large pool of candidate words, which can lead to nonsensical outcomes. Secondly, our approach ensures robust results across different MLMs. Unlike other scoring methods using raw probabilities for calculation, our rank-based bias score method remains consistent even when different MLMs produce varying output probabilities. This dual approach minimizes the influence of outliers and maintains reliability across various computational models.Theoretical Properties Moreover, we can test whether MLM's predictions have the same distribution on two word lists (M and F ). Consider two groups of probabilities scores, {P (f |T \i )} f ∈F and {P (m|T \i )} m∈M , representing the scores samples from distribution P F and P M . The rank sums, denoted by f ∈F R(P (f |T \i )) and m∈M R(P (m|T \i )) respectively, allow us to test the hypotheses H i0 : P F = P M versus H i1 : P F ̸ = P M . The null hypothesis holds if there is no statistically significant bias toward masculine or feminine language in a particular word w i . The following theorem provides a rigorous formulation of the test statistic and its asymptotic result.EQUATIONIf further we have |M | = |F | = K, for each i ∈ [n], under H i0 : S(w i ) ∼ N 0, 2K + 1 3Note that we interpret the prediction probabilities for the two word lists, {P (f |T \i )} f ∈F and {P (m|T \i )} m∈M , as scores that measure the association between the masked word and the words in the lists. A higher score indicates a stronger relationship. We also assume these word lists are sampled from larger sets of male or female-associated words. Thus, for each masked word, the scores {P (f |T \i )} f ∈F and {P (m|T \i )} m∈M are viewed as samples from two underlying distributions, P F and P M . Using the rank test from Theorem 1, we can determine whether P F = P M . This test is particularly useful for detecting biased words or sentences. For non-biased words, the associations with the two word lists should be similar, implying P F = P M . However, for biased words, the associations differ significantly, resulting in P F ̸ = P M .To demonstrate the reliability of our scoring algorithm in identifying social biases within texts, we validate our method on two different tasks2 :Human Experts Validation This validation involved collaboration with six experienced professionals from the fields of sociology and management science. Each coder manually labeled a randomly selected subsample of job advertisements. Leveraging their extensive domain knowledge, these experts meticulously classified the advertisements, assessing them for levels of perceived gender bias. These categorical labels were then transformed into ordinal variables, enabling a detailed statistical comparison with the results produced by our scoring algorithm. This rigorous, expert-driven coding process ensured the reliability of our evaluation methodology.We compute the Spearman rank correlation between the bias scores generated by our algorithm and the results from the manual labeling process. A Spearman correlation coefficient of 0.853 was obtained (Figure 4a ), indicating a strong positive association between our algorithm's scores and the human experts' assessments. This result validates the algorithm's capacity to accurately reflect human judgments of bias, confirming its effectiveness as a tool for social bias detection.Benchmark Validation Further validation was conducted using the BIOS dataset [De-Arteaga et al., 2019] , which comprises personal biographies categorized by gender and various occupations. We employed gender-specific word lists from [Konnikov et al., 2022] , such as {man, his, he ...} versus {woman, her, she...}, as binary directional cues and designated gender as the ground truth label.Our algorithm demonstrated high performance, achieving an AUC of 0.97 in classifying gender, as illustrated in Figure 4b . The AUC, or Area Under the ROC Curve, measures the ability of our model to distinguish between classes -here, gender categories. This performance surpasses that of three baseline methods in [Dhamala et al., 2021] that rely on unigram or word embeddings, highlighting the effectiveness and potential applicability of our bias detection approach in broader NLP tasks. 
probing methodology and job application data generation: 
 Probing Methodology To explore the social biases inherent in ChatGPT, particularly in the context of the labor market, our study simulates the typical use case where job seekers employ ChatGPT to assist in drafting job applications. This approach allows us to investigate not only the biases that may emanate from ChatGPT's training data but also to understand how these biases could potentially influence real-world job application/hiring processes.Probing the social biases within ChatGPT presents several challenges. Firstly, ChatGPT's model operates as a 'black box,' making it difficult to discern the internal processes that contribute to bias propagation. Secondly, the lack of access to the model's architecture or parameters further complicates direct examination. Therefore, our analysis adopts an indirect method, employing our known bias evaluation algorithm to detect and quantify the biases exhibited by ChatGPT, thereby illuminating how these biases might manifest in practical applications.Job Application Data Generation Our dataset comprises over 33K job postings collected from LinkedIn, reflecting a diverse range of industries and job types. To simulate realistic job application processes, we utilize the OpenAI API (GPT-3.5 Turbo, data collected on April 2024) to prompt ChatGPT with these job advertisements, instructing it to generate corresponding job applications for each job posting.This method does more than replicate real-world scenarios where individuals respond to job postings-it also facilitates a comprehensive analysis of the generated texts across various sectors. By using job advertisements as standardized prompts, we ensure that any observed deviations from neutrality in the generated texts are attributable to the model's ingrained biases, rather than the content of the advertisements themselves. This setup is crucial for isolating the effects of ChatGPT's biases, allowing for an accurate assessment of bias presence and intensity using the quantifiable metrics provided by our bias score calculation method.5 Analysing the Bias inside ChatGPT 
dimensions of gender bias: 
 We begin by introducing the four gender dimensions, each defined by a distinct set of gender-related word lists, which will form the basis of our analysis. In recent social science research, understanding gender bias involves not just recognizing the existence of biases but also evaluating their impacts in various contexts. Building on the framework proposed by Bem [1974] , Gaucher et al. [2011] , Konnikov et al.[2022], we utilize specialized word lists to apply our social bias analysis across four different dimensions. Each dimension not only helps identify specific instances of bias but also offers insights into the broader social and psychological dynamics at play.The psychological dimension assesses language context leaning towards communal attributes (e.g., "caring," "sympathetic," "attentive") commonly associated with femininity, or agentic attributes (e.g., "authoritative," "active," "confident") typically linked to masculinity.Role Description: We evaluate job descriptions and roles using word lists that categorize terms associated with "soft" and "social" skills for feminine orientation, and "time-compressed" and "stressful" tasks, such as "multitasking," "pressure," "speed," for masculine orientation.Work-Family Characteristics(WFC): This dimension examines employer policies and cultural expectations affecting gendered labor force participation, scrutinizing terms like "parental leave" and "flexible work" for feminine orientation versus "irregular and long work hours" and "weekend work" for masculine orientation.We also analyze explicit gender references such as gendered pronouns and identity markers ("she," "he," "his," "her," "man").We first analyze the correlation of job postings and job applications across each dimension of gender bias. Our findings indicate a consistent positive linear correlation between the bias scores of job postings and the ChatGPT-generated job applications. This trend suggests that the biases inherent in job postings are likely to be reproduced in job applications by generative AI, reinforcing and possibly amplifying the initial biases. This correlation is visually captured in Figure 3 , illustrating the potential for cyclical reinforcement of biases through the use of generative AI in job application practices.Figure 3 presents the statistical parameters for each analyzed dimension of social bias. The strongest correlation is observed in the Social Characteristics dimension with a correlation coefficient of 0.777, indicating a very strong positive relationship. This is followed by the Role Description dimension, which shows a correlation coefficient of 0.708. Both of these correlations suggest significant potential for the biases in job postings to be reproduced by AI in job applications in these dimensions.The Psychological Cues and WFC dimensions exhibit lower but still substantial correlation coefficients of 0.644 and 0.451, respectively. The slopes of these relationships indicate the rate at which the bias scores from job postings predict those in job applications, with steeper slopes observed in the Social Characteristics dimension. This analysis clearly supports the hypothesis that inherent biases in job postings are likely reproduced in ChatGPT-generated job applications.Figure 3 : Result scatter density plot, for each of the bias dimensions where the x-axis is the job posting bias score and the y-axis is the job applications bias score. Where the darker color means there are more dots. The p-value is the significance of the correlation coefficient.In this section, we delve deeper into how ChatGPT influences bias reproduction within the job application process. Let X and Y , represent the bias scores of job postings and those of the job applications generated by ChatGPT, respectively. We denote the population mean and variance of X as µ X and σ 2 X . A score close to zero indicates minimal bias (i.e., gender neutrality that is neither feminine nor masculine), a higher positive score signifies a bias towards masculine language, and a lower negative score indicates a bias towards feminine language. The aim is to evaluate how ChatGPT may exacerbate or mitigate these biases. The histogram and summary statistics of the bias scores are in Appendix A.5.We propose the following hypothesis tests to assess shifts in mean:H 0 : µ X ≥ µ Y vs. H 1 : µ X < µ YUsing the Wilcoxon signed-rank test, we determine whether there is a significant change in the mean bias score from the job postings to the applications.For the magnitude of bias, we assess:H 0 : |µ X | ≥ |µ Y | vs. H 1 : |µ X | < |µ Y |This test measures the central tendency of bias scores, examining if the absolute values (regardless of bias direction) decrease. The less the magnitude(i.e. closer to zero) the less bias it has.We also explore the variability in bias scores:H 0 : σ 2 X ≤ σ 2 Y vs. H 1 : σ 2 X > σ 2 YThis variance test, employing Levene's test [Brown and Forsythe, 1974] for equality of variances, explores whether ChatGPT produces job applications with more uniform bias expressions compared to the job postings. It helps determine if there is a reduction in variance, which would suggest that ChatGPT standardizes the use of gendered language cues. Such standardization could potentially reinforce specific gender biases more consistently. The testing for the mean shift in Table 1 reveals significant findings across several dimensions. Except for Role Description, all other dimensions exhibit statistically significant shifts toward more masculine language. This indicates a predominant inclination for ChatGPT to amplify the use of masculine language in simulated job applications over and above the original job postings, possibly due to its training on historically biased data. This shift raises concerns about the consolidation and exacerbation of masculine language. Such biases in AI-generated content could perpetuate gender disparities in professional settings, emphasizing the need for interventions in AI training processes to address and correct historical biases. In contrast, the Role Description dimension shows a mean shift toward a less masculine direction, but the bias in job postings has already been shown to be skewed toward a very masculine direction. In this case, ChatGPT seems to help mitigate this extreme masculine bias.The magnitude of bias, assessed through the mean of the absolute bias scores, varies across the dimensions. The Psychological Cues and Role Description dimensions suggest that the overall intensity of bias-regardless of direction-does not increase. This could imply that while the direction of bias towards masculinity is pronounced, the degree of bias embedded within job applications does not intensify. Conversely, the WFC and Social Characteristics dimensions exhibit an increase in bias magnitude, indicating not only a shift towards masculine language but also an overall increase in the strength of biased expressions. This finding is particularly troubling as it suggests that AI-generated job applications in these areas may become more polarized, further entrenching gender-specific expectations in roles traditionally associated with work-life balance and social interactions.The variance results across all dimensions reveal a consistent decrease in job applications compared to job postings. This decrease in variance suggests that the language used by ChatGPT is more uniform across different applications, potentially indicating a standardization of language that leans towards masculine expressions. Such uniformity in language use could narrow the range of expressions and perspectives presented in job applications, limiting diversity and potentially skewing hiring decisions in favor of male candidates.Our statistical results underscore a critical issue: biases in job postings are not merely replicated but are amplified in job applications created by generative AI in response to the postings. This phenomenon can be explained by the reinforcement of initial biases through the language processing and text generation capabilities of AI tools like ChatGPT, which tend to replicate and often intensify the language patterns they are trained on.The amplification of gender biases in AI-generated job applications has profound societal and labor market implications, suggesting that not only are stereotypical roles perpetuated through biased language, but they are also strengthened when individuals use AI tools like ChatGPT to assist with drafting job applications. This use of generative AI plays a crucial role in circulating and amplifying biases, which reinforces, rather than challenges, the gender biases underpinning persistent gender inequalities in the workplace. Such biases can compound, influencing job satisfaction, employee retention, and career advancement. The misallocation of human resources due to biased AI could reduce economic efficiency and innovation, potentially causing sectors to overlook qualified candidates. Furthermore, these persistent inequalities may spur regulatory and legal challenges, especially in countries with robust equal employment opportunity laws, with significant implications for social ethics, justice, and economic equality.Recommendations for Intervention: To mitigate the reproduction of gender biases through LLMs, it is recommended that employers and AI developers implement more rigorous bias monitoring and mitigation strategies. This could include the use of debiased language models, regular audits of AI-generated content by independent third-party organizations, and the development of enhanced AI training datasets that reflect the diversity of the global job market. Additionally, public awareness and education initiatives should be promoted to increase understanding of AI's role in job application and its potential impacts, fostering a critical approach to AI tool usage in professional settings. 
conclusion: 
 Our paper -including a novel experiment, new algorithm development, and empirical application and findings -contributes to the ongoing debates and developments in the ethical use of AI in labor market processes and practices. By identifying underlying biases in AI-driven text generation, this paper proposes novel strategies and methods for detecting and mitigating such biases. Through our PRISM algorithm and empirical application, we show that these strategies are not just theoretical but are intended as actionable steps toward ensuring that the integration of AI in the labor market supports equitable and fair employment opportunities for both employers and job seekers. We explicitly identify the two major harms resulting from bias exacerbation in LLM-generated job applications:Structural Harm: LLM-generated job applications that perpetuate gender stereotypes contribute to the reinforcement of gender inequalities embedded in language used in labor market processes. Job postings and applications are critical steps in these processes and play a significant role in the reproduction of gender inequalities. By failing to challenge these biases, LLMs can inadvertently support the perpetuation of these inequities.Practical Harm: As gender-biased LLM-generated job applications become part of the training data for future AI applications in HR(both in generating job postings and assessing applications), there is a risk of further entrenching gender biases in language use. This entrenchment can lead to cascading effects, such as increased labor force gender segregation, which have significant societal implications. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Section 5.4Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We don't have any released model or datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All of them is properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We assume each word from two word lists M and F are selected independently. Therefore, the first result in Theorem 1 is implied directly from the Wilcoxon rank sum test [Wilcoxon, 1945] .EQUATIONwhere the second equality follows from the fact:f ∈F R(P (f |T \i )) + m∈M R(P (m|T \i )) = 2K(2K + 1) 2 .From the first result in Theorem 1, we havef ∈F R(P (f |T \i )) ∼ N K(2K + 1) 2 , K 2 (2K + 1) 12for each i ∈ [n], under H i0 . Therefore, under H i0 , the result S(w i ) ∼ N 0, 2K+1 3 follows directly from the relationship (1) and the distribution of f ∈F R(P (f |T \i )). In Figure 4a , A labeled value of 7 signifies neutrality, while values less than 7 suggest femininity, and values greater than 7 imply masculinity. A Bias Score close to zero indicates neutrality, a positive value suggests masculinity and a negative value denotes femininity.In Figure 4b , We follow three gender metrics for evaluating gender bias in texts in [Dhamala et al., 2021] . The first metric, unigram matching, counts gender-specific tokens like 'he', 'him', 'she', 'her' etc., and labels texts with more male tokens as male, more female tokens as female, and texts with equal counts as neutral. The second metric assesses words indirectly related to gender via a normalized projection of word vectors in the gender direction, defined by ⃗ she -⃗ he, using a Word2Vec embedding. Word-level gender scores are calculated as b i = ⃗ wi•⃗ g ∥ ⃗ wi∥∥⃗ g∥ . These are aggregated either by a weighted average (Gender-Wavg):Gender-Wavg = n i=1 sgn(b i )b 2 i n i=1 |b i |or by taking the score from the most gender-polar word (Gender-Max):i * = arg max i (|b i |), Gender-Max = sgn(b * i )|b * i |Texts are classified as male if the score is less than -0.25 and as female if the score is greater than 0.25.To ensure the scientific rigor of the evaluation, we paid particular attention to inter-rater validity and reliability. Specifically, each phase included individual labeling of data conducted by four independent experts specializing in labor market inequalities associated with gender, work, and family, and Equity, Diversity, and Inclusion (EDI) in the labor market. In each round of labeling, individual labeling was followed by group sharing and discussion of the preliminary outcomes among the four experts. This combination of individual and group analysis allowed the team to trace the score labeling and the validation of word inventory, ensure inter-rater reliability in each phase, and contextualize each phase within relevant scholarly literature, policies, and definitions.The score labeling and the validation were further validated by three additional expert labelers (management, human resource, and social science scholars) from the team. Through a double-blind labeling approach, the three additional experts independently assessed the dimensions and labels produced by the first four experts, demonstrating a high level of consistency. The scores and word lists were then finalized through further deliberation among the four experts and the three additional validators.Because we used an iterative multi-round coding process, the inter-coder consistency rate in the developmental coding varied between 0.6 and 0.8. Notably, the final validation by three fresh validators within the team achieved a high level of inter-coder consistency exceeding 0.8.In Figure 5 , we present the histogram of bias scores for Job Postings and Job Applications on different dimensions.Figure 5 : Result histogram, for each of the bias dimensions, we use different colors to distinguish the Job Postings and Job ApplicationsBelow we list the statistical test results for job postings and job application categories across different dimensions, with additional magnitude values, in Table 2 , 3, 4, 5. 0.0 µ X > µ Y Work-Family Characteristics 207756451.0 0.0 µ X < µ Y Social Characteristics 119987669.0 0.0 µ X < µ Y A.7 Ablation studyAdditionally, we conducted Control Experiments with Different MLMs, we selected two more MLMs: BERT-large and DistilBERT. The results for both the human validation and the LLM correlation plots are in Figure 6 , 7. These results show that changing the MLMs produces consistent outcomes in the scatter density plot for evaluating bias for LLM, and statistical testing results are also highly consistent.For our analysis, we utilize the 'bert-base-uncased' model from the Hugging Face library. The temperature parameter for the ChatGPT API is set to its default value of 7. For the preprocessing, when we iterate through the text, we skip some of the function words like Articles ("a", "an", "the"), Prepositions ("on", "by"), Conjunctions("and", "but", "if"), etc. As our algorithm solely requires a forward pass and no training, this enhances computational efficiency. To further optimize performance, we employ an Nvidia RTX A5000 GPU. All experiments are conducted on an Ubuntu server equipped with an AMD Ryzen Threadripper 3990X 64-Core Processor and 256 GB of RAM.The job post data is publicly available at https://www.kaggle.com/datasets/sachinmeena04/ linkdin-jobs-dataset.The full word list is the integration of the existing literature in sociology, we directly adopted them without modification. These word lists are already publicly available. Below is the table for the size of each word list: -High level of initiative, maturity, tact and diplomacy.A.9.1 Limitations, Discussions, and Future Works Tokenizer Issue: Discrepancies arise from the adaptation of word lists for use in MLMs due primarily to tokenization strategies. For example, the word "limitless" is tokenized into ['limit', '##less'], which may affect the representation and subsequent analysis of such words in studies of linguistic bias.Bias in MLMs: Addressing inherent social biases within MLMs presents a significant challenge and offers fruitful directions for future research:• A primary consideration is our objective to evaluate bias in LLMs. Specifically, we examine the change in the bias score between the input and output text. By using the same MLM for both input and output, we can isolate and measure any additional bias introduced during the text generation process. This approach also helps control for the baseline biases present in the MLM being used for evaluation. • A more fundamental question is whether a biased model is essential for picking up the biased words to be able to detect bias. If so, arguably, the original BERT model, trained on extensive real-world data, effectively mirrors societal biases, thus providing a realistic framework for identifying and analyzing these biases. But we still don't have a clear answer to this question. • In practice, achieving a perfectly unbiased model is currently beyond our reach. The pursuit of such an ideal model remains theoretical at best, as even the debiased MLMs carry traces of inherent biases from their training corpora.All our results are based on an English dataset; however, additional complexities may arise with other languages due to more intricate word splitting or tokenization challenges. The Masked Language Models (MLMs) utilized in our study are sourced from public, open-source pre-trained models. The inherent biases of these pre-trained models might impact our results, although we have attempted to mitigate this issue through a robust rank-based method that reduces sensitivity to changes in probability distributions. Currently, our job application generation relies on basic prompts; exploring the effects of varied prompts to capture a broader spectrum of biases constitutes part of our future work. Moreover, while our framework is initially designed to use pairs of word lists, it possesses the flexibility to accommodate single or multiple word lists with minimal adjustments, an extension we also plan to explore in future research endeavors.Answer: [Yes] Justification: The data is publicly available and we will upload the code Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The code for the algorithm is available at: https://github.com/Lei-Ding07/ChatGPT_bias/This correlation is notably higher compared to those typically observed in non-experimental social sciences. 


Decision: The review team mostly agrees that the paper is well-written and studies a novel topic within the emerging area of ML for the Social Sciences.

There are some concerns about whether the evaluation process is robust and appropriate for the problem. During the reviewers’ discussion after the rebuttal, the main concern was that the paper used an MLM to measure the bias of another LLM. The argument was that if the MLM is biased then it's possible that biases of the MLM are attributed to the LLM. Although a counter-argument was that measuring the change between the input text and output text partially addresses this issue, it doesn’t completely eliminate the concern.

Another secondary concern was that the human experts validation is just limited to one dataset, with 6 human labelers and <100 examples. In their rebuttal, the authors responded with several quite convincing arguments including that for human evaluation, they worked with experts in related fields (sociology, psychology, management, human resources), with an iterative multi-round coding process and that in addition to human evaluation, they also conducted benchmark validation with more than 250,000 examples and a dataset from a different domain. Illustrating these arguments in the paper would be helpful.

The review team pointed out several suggestions for revisions.

- Address the concern about using an MLM to measure the bias of another LLM. (See Review HqWb)
- Include more details about the human experts and the tasks they have done maybe in the appendix, i.e. the demographics, the inter-annotator agreements, and the amount of tasks they did. (See Review Bg7r)
- Include the predefined bias-related word lists (or at least details about) and better argue how the quality of the word lists may/ may not affect the bias evaluation results (see Review ZJXv)

Based on the information the authors provided in the rebuttal, I believe that at least the last two concerns are easily addressable. Overall, the strengths of the paper generally outweigh the aforementioned concerns.

Reviews: review: paper_summary: paper_summary: This paper presents a novel bias detection algorithm, PRISM, and uses it to evaluate biases in job applications generated from prompts that include job postings. The technique uses a masked language model (MLM) to find the probability of “masculine” and “feminine” words replacing the masked word in the text. The bias score is then the difference between average ranks of feminine and masculine words, averaged over all words in the text. The authors find a correlation between biased job postings and biased generated text based on their evaluation method.
strength_weakness: strength_weakness: I have two significant concerns about this work. First, I have a meta-concern about the setting in which the experiments are being conducted. Is generating a job application from a posting a common use case of LLMs? If there is indeed exacerbation of bias going from posting to application, what are the concrete harms of that manifesting itself? In short, I do not see much in the text motivating the work and helping us understand the real world impacts of these issues.

Second, I have a specific concern about the methodology employed. The paper claims that the PRISM method obviates the need for human labeling, and this seems to be accomplished through the use of the MLM and the previously curated lists of words. I am not convinced based on what I have seen in the paper that this method is actually measuring the level of bias in the generated text - it may instead be measuring the gender bias in the underlying MLM (bert-base-uncased) that is used to compute the score. It seems there is a fundamental chicken and egg problem. Is a “feminine” word ranked higher because the surrounding text that was generated is biased, or is it ranked higher because the MLM is biased to predict more feminine words in that context when a neutral word would actually be perfectly appropriate. How would the authors design control experiments to disentangle these two effects?
questions: questions: Meta question: isn’t the assumption that certain words are “feminine” or “masculine” already a sort of gender bias? I suspect that the paper means that these buckets are words that are stereotypically or historically associated with certain genders, but if that is the case it should be made explicit. 

Can the authors publish the exact word lists used for the experiment?

Why are means of ranks employed as opposed to the means of the probabilities themselves? 

Similarly, is averaging the S scores to compute B(T) the right approach? What do the actual distributions of S look like? Is the mean capturing the distribution faithfully?

What version of ChatGPT was used for this experiment? Because the models are updated fairly frequently, just specifying the use of the API is not sufficient for reproducibility.
limitations: limitations: The paper discusses limitations briefly in the appendix, and there is a very brief discussion about the potential for biases in the MLM to affect the results. However, I believe there needs to be significantly more validation that this approach is sound based on the concerns I enumerated above.
paper_summary: paper_summary: This paper investigates the impact of social biases in the application of LLMs within the labor market. The authors examine biases in job applications generated by ChatGPT based on given job posts. They introduce a new bias evaluation method called PRISM (Probability Ranking bIas Score via Masked language model), which combines predefined gender-related word lists from social sciences with a masked language model approach to assess biases in input texts. Experiments with ChatGPT revealed that biases present in job postings are not only replicated but also amplified in job applications created by generative AI in response to these postings.
strength_weakness: strength_weakness: 1. Experiments are limited. Only ChatGPT has been applied. 
2. This paper aims to investigate the social biases, but no detailed exploration in different bias types has been conducted. From the algorithm, I only see the gender bias involved, which I think is only a part of the social biases.
questions: questions: 1. Based on the scoring function in line between 155 and 156, why do you state "A positive score indicates a bias toward a masculine orientation, while a negative score suggests a bias toward a feminine orientation."? Shouldn't it be the reversed reading? 
2. As the word lists for bias evaluation are predefined, how do you confirm their reliability?
3. I am interested in the human evaluation part of sec 3.4. Could you include more details about the human experts and the tasks they have done maybe in the appendix, i.e. the demographics, the inter-annotator agreements, and the amount of tasks they did?
limitations: limitations: Same as the points in Weaknesses
paper_summary: paper_summary: This research investigates the social biases present in ChatGPT-generated job applications using a novel experimental design that responds to real job advertisements. By simulating the job application creation process, the study examines the language patterns and biases that surface when the model is prompted with diverse job postings. Specifically, this paper presents a PRISM algorithm based on Masked Language Models to quantitatively assess social bias based on validated inventories of social cues/words. The results show that the increasing adoption of generative AI can reinforce and exacerbate gender and social inequalities in the labor market through the use of biased and gendered language.
strength_weakness: strength_weakness: 1. The bias evaluation in this paper highly relies on the predefined bias-related word lists, e.g., the Feminine word list and Masculine word list, so how the quality of the word lists will affect the bias evaluation results? It would be better to make analysis on this aspect.

2. The proposed method appears to be generally applied to any domain, although the major focus is on labor market text generation. Therefore, the characteristics of bias in labor market text generation need to be highlighted.

3. There are some writing problems in this paper, for example, the table captions should generally be presented above the table.
questions: questions: 1. How the quality of the word lists will affect the bias evaluation results? 

2. The co-occurring frequency of some words are naturally generated, for example, in our common understanding, soft is more related to woman instead of man. Is it reasonable to treat it as bias?
limitations: limitations: The major limitation is the validation of the basic motivation of assessing social bias based on validated inventories of social cues/words.
paper_summary: paper_summary: This paper presents a method to assess the extent to which social biases are encoded in LLMs. The motivating example throughout the paper is assessing the degree of gender bias in the context of job applications in the labor market. The method (PRISM) is designed as follows: given text (presumably output by the LLM), each word is masked out, one at a time, and the masked word is predicted using a masked language model. However, the only masked words predicted belong to one of two lists, corresponding to the two social groups (e.g. a list of masculine terms and a list of feminine terms). If the probability of words in one list are consistently larger than that of the other list, there is evidence for social bias.

The paper then uses PRISM to assess the social biases of chatGPT. chatGPT is prompted to produce job applications, and the method analyzes bias across four social dimensions relating to gender. The paper finds that biases in job postings are likely reproduced and even exacerbated in job applications generated by chatGPT.
strength_weakness: strength_weakness: The biggest weakness of the paper is the lack of evaluation. A new method is proposed for assessing social bias in LLMs, but only a small portion of the paper is dedicated to validating the new algorithm. While the validation exercises are a good start, they're limited; for example, the human experts validation is just limited to one dataset, with 6 human labelers and <100 examples. To see evidence that this is a general method, we need to see robustness across multiple application areas and experiments. 

Related to the above, I can imagine a problem of this approach where biases of the masked language model are instead incorrectly attributed to the LLM generating the text. For example, if the masked language model used is biased (e.g. is likelier to predict male words than female words), then any sequence of text PRISM is evaluated on will be viewed as biased. It seems that for this test to be reliable, the masked language model should form calibrated and "unbiased" predictions of its own. Couldn't this be revealing biases in BERT rather than the LLM?  I'm not sure how likely this is, but it warrants discussion and validation in the paper; otherwise the limited evaluation is not compelling.

An additional weakness is that the experiments are limited. The experiments don't include baselines for assessing social bias. Is there evidence that PRISM is picking up on biases that other methods miss? Additionally, only one LLM is considered in the job posting experiment, and one MLM (BERT) is used as the MLM -- ablating the latter could help demonstrate the robustness of the method. 

A little less urgent, it would've been helpful to see more qualitative examples of the postings, along with more details for the experiments (e.g. the full word lists or at least the lengths of each word list).
questions: questions: See section above, namely:
- Is there evidence that the test isn't susceptible to social biases in masked language models?
- What does PRISM find that isn't found by other methods?
- Is there more evaluation/validation that the method is useful?
limitations: limitations: Yes

Comments: title: Thank you
Thank you
comment: Dear Reviewer Bg7r, we truly appreciate your valuable support and recognition of our paper!
Thank you for the response. I think my initial questions were properly addressed. I will raise my score and would be glad if this paper could be included to the proceedings, which I think would definitely diversify the conference.
Dear reviewer HqWb, thank you for your reply. We wonder if there is any further information/clarification we can provide to help clarify your remaining queries and bolster your confidence in our work. Thank you.
I thank the authors for responding to my comments. I will maintain my original score
Dear reviewer e7QL, thank you for your kind support and recognition of our work.
Thank you to the authors for their thorough responses. I believe you have addressed my concerns about the weakness of the MLM usage. As such, I am adjusting my score to 7.