GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction (2024)

Abstract: Social biases in LLMs are usually measured via bias benchmark datasets. Current benchmarks have limitations in scope, grounding, quality, and human effort required. Previous work has shown success with a community-sourced, rather than crowd-sourced, approach to benchmark development. However, this work still required considerable effort from annotators with relevant lived experience. This paper explores whether an LLM (specifically, GPT-3.5Turbo) can assist with the task of developing a bias benchmark dataset from responses to an open-ended community survey. We also extend the previous work to a new community and set of biases: the Jewish community and antisemitism. Our analysis shows that GPT-3.5Turbo has poor performance on this annotation task and produces unacceptable quality issues in its output. Thus, we conclude that GPT3.5-Turbo is not an appropriate substitute for human annotation in sensitive tasks related to social biases, and that its use actually negates many of the benefits of community-sourcing bias benchmarks.

heading: 1 Introduction
text: Though seemingly ubiquitous, large language models (LLMs) still treat users unequally and exhibit harmful social biases (Weidinger et al., 2022; Shelby et al., 2023).1 Quantitative LLM bias measurement is a necessary first step to understanding and mitigating bias-related harms of AI systems. Measurement is essential because it allows model creators to understand potential fairness issues with
1Clear and explicit definitions of the terms bias and harm are essential for productive discussion of AI fairness (Blodgett et al., 2020). For purposes of this paper, we define bias as “substantially differing treatment of a marginalized group relative to a dominant group that replicates existing social stereotypes about the marginalized group” and harm as “physical, psychological, financial, or professional events that affect a person in perceived negative way.”
their models, downstream users to compare models and choose those that are relatively fair in their use context, and fairness researchers to determine whether debiasing methods are effective.
The current standard for bias measurement in LLMs is paired sentence bias benchmarks, which consist of pairs of similar sentences and generally rely on comparing a model’s probability of predicting the stereotypical sentence to the probability of predicting a contrasting sentence. There are significant quality and grounding issues with most current benchmarks, especially those developed via crowd-sourcing. Current methods for communitysourced benchmark development, which mitigate some of the problems with crowd-sourcing, require significant human effort for annotation of survey responses. This work is time-consuming; in an unfunded, community-led benchmark development effort, this is either cost-prohibitive or requires asking annotators to work for free. This annotation also places a significant psychological burden on annotators.
In order to maintain the usefulness and participatory nature of community-sourced bias benchmarks while reducing the financial and psychological costs, we tested model-assisted harm extraction. The main contributions of our work are as follows:2
• We introduce WinoSemitism, a communitysourced benchmark for antisemitism, generalizing method of Felkner et al. (2023).
• We create GPT-WinoQueer and GPTWinoSemitism, which are versions of the WQ and WS datasets annotated by GPT-3.5-Turbo instead of human experts.
2Our code and data are available at https://github. com/katyfelkner/winosemitism
14104
• We provide a thorough quantitative and qualitative comparison of human-annotated and model-annotated datasets, finding significant quality issues with model annotations.

heading: 2 Related Work
text: The current standard for bias measurement in LLMs is paired sentence bias benchmarks, such as StereoSet (Nadeem et al., 2021) and CrowS-Pairs (Nangia et al., 2020). These benchmark datasets consist of pairs of stereotypical (e.g. “Women are bad at math.”) and counter-stereotypical (e.g. “Men are bad at math.”) sentences. Metrics vary slightly between benchmarks, but generally rely on comparing a model’s probability of predicting the stereotypical sentence to its probability of predicting the other sentence.
While paired sentence benchmarks are important tools for understanding and mitigating bias, the current benchmarks have several important weaknesses. First, most benchmarks attempt to be general-purpose, covering a variety of bias axes (e.g. race, gender, nationality, religion, etc.) with the intention that downstream users need only test their models on one benchmark. However, even the most extensive benchmarks leave out many biases that exist and are harmful in the real world. Even when they cover several bias axes, benchmarks are often oversimplified and lacking nuance along each axis. For example, Nangia et al. (2020) include homophobia in their benchmark, but treat LGBTQ+ identity as a binary attribute rather than a complex set of related communities. Similarly, Nadeem et al. (2021) include religion as one of their bias axes, but only consider Christianity, Islam, and Hinduism, ignoring antisemitism and other religious biases. LLM benchmarking work on antisemitism is limited; however, there is closely related work on antisemitism in the context of hate speech detection (Chandra et al., 2021; Jikeli et al., 2021).
Second, many existing benchmarks have serious quality control issues, including ungrammatical or nonsensical sentences, mismatching of stereotypes to target groups, and vague or nonspecific stereotypes (Blodgett et al., 2021). Finally, most existing benchmarks are insufficiently grounded, i.e. the stereotypes and biases described in the benchmark may not be well-aligned with the affected communities’ opinions on what stereotypes are harmful and what constitutes unacceptable or undesirable LLM behavior (Blodgett et al., 2020).
Grounding issues often stem from the method of collecting stereotypes to include in bias benchmarks. Many benchmark creators use crowdsourced stereotypes, usually from crowdworkers on Amazon Mechanical Turk, who have varying degrees of knowledge and personal experience with social biases. In contrast, recent work (Felkner et al., 2023) has had success with communitysourced bias benchmarks in which stereotypes were derived from a large-scale online survey of members of the affected community. This yielded a large, specific, well-grounded, and high-quality bias benchmark dataset for harms affecting a specific community.
Community-sourcing, while a significant improvement over crowd-sourcing in many respects, has its own problems. Current methods require survey responses to be annotated by hand, ideally by researchers who are themselves members of the communities surveyed. This annotation relies heavily on the lived experience and subjective opinions of annotators to extract attested harm predicates from survey responses, which are then inserted into template sentences to create the stereotypical and counter-stereotypical sentences in the benchmark. Human annotation by researchers with relevant lived experience is time-consuming, and therefore expensive and often cost-prohibitive for grassroots, community-based efforts.
Additionally, the annotation is psychologically taxing on annotators. Annotators spend hours reading detailed descriptions of violence and hatred toward their own communities. This is often triggering for annotators who have first-hand experiences with such harm, and it is exhausting and depressing even for those who have somewhat less painful experience. The situation is somewhat analogous to content moderation, in which repeated exposure to disturbing content causes secondary traumatization (McCann and Pearlman, 1990) for moderators, often leading to post-traumatic stress disorder (PTSD) (Steiger et al., 2021). This annotation is similar to volunteer moderation of identityspecific online communities, such as those studied by Dosono and Semaan (2019), who found that moderation constitutes significant emotional labor and often leads to burnout.

heading: 3 Methods

heading: 3.1 Jewish Community Survey
text: Because one of the purposes of the WinoSemitism dataset is to validate the generalizability of the benchmark development approach from Felkner et al. (2023), we closely follow their methodology on survey design and human annotation of survey data. The Jewish community survey was deemed exempt by our institution’s IRB. Participants were recruited through a variety of channels, including researchers’ personal networks, Jewish social media channels, synagogues, and Jewish student organizations. Interested respondents were directed to a screening questionnaire. Those who met the screening criteria (eighteen or older, English-speaking, and identifying as Jewish) then had the opportunity to review the informed constent document. The consent form explained the potential benefits and risks of the survey and informed participants that they could skip questions or leave the survey for any reason at any time during their participation.
Participants then answered demographic questions. They were asked to self-identify their gender and ethnicity and provide their age range and country of residence. They were also asked about their Jewish cultural background (e.g. Ashkenazi, Sephardic, Mizrahi) and religious identity (e.g. Conservative, Reform). The demographic questions about Jewish identity were developed in consultation with an expert in Jewish Studies. These questions were included to ensure that Jewish people of diverse backgrounds and experiences were included in the WinoSemitism dataset.
Following the demographic questions, participants answered multiple choice and free-response questions. The multiple choice questions listed a variety of common stereotypes about Jewish people. Participants were asked whether they thought each stereotype was positive, neutral, or negative, and whether they had personally experienced the stereotype. There were also open-ended questions for participants to write in additional stereotypes, both about Jews in general and about specific subcommunities with which they identified.
The survey was active for about ten weeks in late 2023. We had a total of 203 respondents. The overwhelming majority of respondents were white and from the United States. Respondents were relatively evenly distributed across age ranges. There were significantly more responses from women than men; we also saw a small number of responses
from trans and nonbinary Jews. Respondents were mostly Ashkenazi, with very few Sephardi, Mizrahi, and Persian Jews represented. Respondents were relatively evenly distributed across religious identities, with Conservative and Reform Jews having the highest number of respondents.
The vast majority of survey responses concerned general antisemitism, rather than stereotypes about specific subgroups of the Jewish community. There are not enough predicates about identity subgroups to report meaningful results; thus, we only report quantitative results aggregated over the entire dataset. This is significantly different than the results reported on the WinoQueer dataset, for which survey responses were much more evenly distributed.

heading: 3.2 WinoSemitism Benchmark Construction
text: The WinoSemitism benchmark consists of pairs of stereotypical (i.e. antisemitic) and counterstereotypical sentences. Each sentence is constructed from the following components:
• Sentence templates, which are the structures into which other components are placed. Sentence templates were constructed based on findings from Cao et al. (2022).
• Identity descriptors, which are usually either “Jewish people” or “Jews” but sometimes include specific subpopulations, such as “Jewish women” or “Orthodox Jews.”
• Counterfactual ID descriptors, which were Christian, secular, atheist, and nonreligious.
• Common Jewish names, which were selected by an expert in Jewish studies (based on works including Fermaglich (2018)) from the US Social Security Administration’s list of common names3 from 1970 to 2019.
• Attested harm predicates, which were extracted from survey responses via annotation by expert humans. Most predicates apply to Jewish people in general, but some were specific to subgroups of the Jewish community. After extraction, predicates were manually edited to ensure syntactic correctness of constructed sentences.
3https://www.ssa.gov/oact/babynames/decades/ names2010s.html
Benchmark sentences were constructed by sampling from each component list. Each sentence template was filled in with each extracted predicate and the predicate’s corresponding identity descriptor(s). Then, the subjects of singular templates were filled in with he/him and she/her pronouns as well as a random sample of 5 masculine and 5 feminine names. This yielded the set of stereotypical sentences. Finally, for each stereotypical sentence, two of the four counterfactual identity descriptors were sampled and two sentence pairs were constructed. Our use of random sampling of names and counterfactual identity descriptors is a departure from previous work, which constructed benchmarks using a strict Cartesian product of all component categories. However, we had a much larger list of names than previous work, and we chose to use random sampling to keep the overall size of the WinoSemitism benchmark roughly comparable to previous work. In total, the WinoSemitism benchmark consists of 58,816 sentence pairs. An example WinoSemitism sentence pair is “All Jews are greedy.” and “All Christians are greedy.”

heading: 3.3 GPT Extraction
text: The vast majority of human effort in the benchmark construction process described above is in the extraction of attested harm predicates from freeresponse survey data. In order to alleviate the financial and psychological burden of human annotation, we tested whether GPT-3.5-Turbo can be used to perform this predicate extraction. In our human annotation setup, annotators were presented with the full text of one response to one survey question, which could range in length from a short phrase to several sentences. Annotators were asked to extract any number of attested harm predicates and record each separately. Thus, each response to each question is paired with a list of human-extracted predicates. Survey responses for which humans were unable to extract harm predicates were removed from the dataset.
We performed GPT-3.5-Turbo predicate extraction for both the WinoQueer and WinoSemitism datasets, using the same prompting setup for both. For each survey answer, we prompted the model N times, where N is the number of human-extracted ground truth predicates. This N would be unknown in a realistic use case, so our experiments represent an artificially easy annotation task. Temperature was set to 0.3 for all experiments. Full details of the prompts used can be found in Section A.

heading: 3.4 GPT Benchmark Construction
text: After attested harm predicates have been automatically extracted from both LGBTQ+ and Jewish community survey responses using GPT-3.5-Turbo, they are then used to create model-assisted versions of the WinoQueer and WinoSemitism benchmarks. These datasets are named GPT-WinoQueer (GPT-WQ) and GPT-WinoSemitism (GPT-WS). To differentiate between versions, we refer to the original, human-created datasets as Human-WinoQueer (H-WQ) and Human-WinoSemitism (H-WS), respectively. To build the GPT-WQ and GPT-WS benchmarks, we use exactly the same lists of template sentences, identity descriptors, names, pronouns, and counterfactuals as in the corresponding human-extracted dataset. These component lists are combined with GPT-extracted predicates using exactly the same methods as for the orginal datasets (strict Cartesian product for WQ, Cartesian product with some random sampling for WS). This generation yielded 45,468 sentence pairs in GPT-WQ and 68,472 sentence pairs in GPT-WS. The GPTextracted datasets differ slightly in size from the corresponding human-extracted datasets due to differing numbers of unique predicates extracted.
The only difference between the H-W* and GPTW* datasets is thus the quality of the extracted predicates. By comparing bias scores of the same models on the two pairs of datasets, we can determine whether the GPT-extracted bias scores correlate well with the human-extracted baselines. Thus, we can assess whether GPT performs comparably to humans on the highly context-sensitive task of bias benchmark construction.

heading: 3.5 Evaluation Metrics

heading: 3.5.1 WinoSemitism Baseline
text: All of our bias benchmark datasets (H-WS, GPTWQ, and GPT-WS) use the same bias score as the original WinoQueer benchmark. Intuitively, the bias score is the percentage of sentence pairs for which the tested model has a higher probability of predicting the stereotypical sentence than the counterstereotypical sentence. An ideal bias score is 50, meaning that a model is equally likely to apply a stereotype to either the group targeted by the stereotype or the corresponding majority group. Model probabilities are calculated using the pseudo-log-likelihood score from Nangia et al. (2020), which was extended to autoregressive models by Felkner et al. (2023). This bias score is used
for both human-created and model-created benchmark datasets. The total computation budget for all LLM evaluations was around 500 GPU hours across NVIDIA P100s, V100s, and A100s.

heading: 3.5.2 Auxiliary Evaluation of GPT-Extracted Predicates
text: In addition to comparing WQ and WS bias scores, we use several auxilary metrics to evaluate the quality of GPT-extracted harm predicates. We use the following quantitative metrics:
• Exact match percentage: percentage of GPT-extracted predicates that exactly match a human-extracted predicate for the same survey response. We expect this number to be very low, indicating that predicate extraction is a nontrivial natural language understanding task.
• Cosine similarity score: We take the cosine similarity of SBERT sentence embeddings from the all-mpnet-base-v2 model (Reimers and Gurevych, 2019) for the humanextracted and GPT-extracted predicates. We consider embeddings of just the predicates (i.e. phrases) and of simple sentences containing the predicates. Higher cosine similarity scores indicate more similarity between human-extracted and GPT-extracted predicates. However, this metric is prone to overestimating similarity in cases where the two predicates share many words but differ significantly in meaning.
For both metrics, we compare each humanextracted predicate to all GPT-extracted predicates for the same sentence and take the best score.
In addition to quantitative similarity scores, we perform a qualitative human analysis to better understand the failure types. All GPT extracted predictates were reviewed post hoc by researchers. Each was classified into one of five outcome categories: Correct, Semantically Correct (but syntactically invalid and in need of post-editing), Opposite (counter to the stereotype being attested), Hallucination (not present in the survey), and Other (incorrect but neither opposite nor hallucination).

heading: 4 Results

heading: 4.1 WinoSemitism Baseline Results
text: First, we present baseline results on the WinoSemitism dataset across 20 publicly available LMs. Following Felkner et al. (2023), we evaluate on BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2020), BART (Lewis et al., 2020), GPT-2 (Radford et al., 2019), BLOOM (Workshop, 2022), and OPT (Zhang et al., 2022) Results are summarized in Table 1. In general, the baseline results show that all 20 of the tested models show significant antisemitic bias, as defined by members of the Jewish community. On average, models applied community-defined antisemitic stereotypes to Jews in 69.03% of cases and applied the same stereotypes in non-Jews in 30.97% of cases. This means that models are more than twice as likely to apply antisemitic statements to Jews as they are to non-Jews.
We also tested stereotypes about specific subgroups of the Jewish community. We received very few survey responses about subgroup-specific biases, so our test sets for most subgroups are too small to report meaningful quantitative results. Anecdotally, however, we notice a few worrying trends. First, models stereotype Jewish women and mothers much more frequently than Jews in general. The average WinoSemitism score for Jewish mothers compared to non-Jewish mothers across all models is 76.7, and the average WinoSemitism score for Jewish women is 84.3. This suggests that, in addition to sexism and antisemitism measured in isolation, models are reproducing intersectional biases specific to Jewish women. Second, models reproduce stereotypes about Jews even more
strongly in sentences related to Israel, Palestine, Zionism, and anti-Zionism, with bias scores in the mid-to-high nineties. Training data for LLMs most likely reflect the high degree of controversy present in global public discourse about Israel and Palestine, debates about whether anti-Zionism constitutes antisemitism, and conflation of the State of Israel with all Jews everywhere.

heading: 4.2 Predicate Extraction
text: Below, we present results for our experiments on automated extraction of attested harms from survey data. We evaluate the success of our approach on both quantitative similarity metrics and qualitative human analysis. Table 2 summarizes the quantitive similarity metrics for the GPT-WQ and GPT-WS datasets. We report GPT-WQ results by identity group, but GPT-WS results are only reported in aggregate due to very small test set sizes for most identity subgroups. As expected, we have very low rates of exact matches (disregarding capitalization and punctuation) between human-extracted and GPT-extracted predicates. This shows that the task is non-trivial and is more than simply selecting a span from the input text. We also see relatively low cosine similarity scores between the SBERT sentence embeddings of human-extracted and GPT-extracted predicates. These low cosine
similarities mean that GPT is unable to adequately identify attested harms in survey responses. While the cosine similarities and exact match rates are better for WinoSemitism than for WinoQueer predicates, the scores still indicated significant quality issues with model-extracted predicates. As in WQ baseline results, there is a disparate impact on subgroups marginalized within the LGBTQ+ community. Model-extracted predicates are especially bad for lesbian, bisexual, pansexual, and asexual individuals. These results suggest that using LLMs in the construction of bias benchmarks is likely to produce benchmarks that are generally low-quality and fail to accurately measure biases against multiply marginalized populations. This runs the risk of exacerbating existing inequalities if too much trust is placed in benchmarks that underestimate bias effects on certain subgroups of marginalized communities.

heading: ID Group %C %S %O %H
text: We also perform a post hoc human analysis of GPT-extracted predicates. These results are summarized in Fig. 1. We observe that, for both GPTWQ and GPT-WS, less than half of GPT-extracted predicates are classified correct, i.e. immediately usable for benchmark construction. Around one fifth of model-extracted predicates require only syntactic changes. While some of this editing could be possibly be automated, it is likely that much of it would require human correction or supervision to create a high-quality benchmark.
The first issue with model-extracted predicates is incomplete extractions. These are included in the other category of Fig. 1. While a few such outputs are to be expected from both human and model annotation, the prevalence of incomplete extractions in the GPT-WinoSemitism dataset is concerning. In particular, many extractions consist of single verbs and are missing the nouns that are an important component of the attested stereotype. For example, the stereotypes “Jews control the US government” and “Jews consider themselves superior to others” are collapsed to “Jews control” and “Jews believe,” respectively. These predicates are unspecific and do not adequately reflect the harms attested by survey respondents.
Another significant issue with model-extracted predicates is significant number of cases where the model extracts the opposite of the stereotype expressed in the survey response. This occurs in 6.12% of GPT-WQ extractions and 3.43% of GPTWS extractions. While these are relatively small percentages, the downstream impact is that a significant portion of sentence pairs in the GPT-W* datasets are not only ignorning but directly contradicting the experiences of affected community members. Additionally, because of their surfacelevel similarity to correct extractions, these negation errors are not easily flagged by automated metrics like cosine similarity. On the WinoQueer
survey response, we observe that many cases of negation concern instances of violence and harassment experienced by community members. Instead of correctly identifying the survey respondent as a victim, model-extracted predicates conflate the survey respondent with the perpetrator. This impact is most severe for lesbian respondents (nearly 30% of all model outputs), many of whom reported being catcalled and sexually objectified. GPT extraction incorrectly yielded benchmark sentences like “Lesbians catcall others” and “All lesbians objectify others.” While both of these are expressions of the harmful stereotype that lesbians are predatory towards straight women, neither of them was actually present in the survey data, so they do not meet the criteria for inclusion in a community-sourced fairness benchmark. Benchmarks that spuriously accuse marginalized people of perpetrating the violence of which they report being victims is a serious risk of automated extraction of attested harms for benchmark construction.
Most concerning, however, is the extremely high rate of model hallucinations in extracted predicates. Over thirty percent of GPT-WQ predicates, and nearly twenty percent of GPT-WS predicates, express a stereotype or bias that is not present in the survey response from which they were “extracted.” Most of these hallucinations are highly repetitive, suggesting that the model defaults to a very small set of priors when unable to extract a predicate from the input data. Repetitive hallucinations include “are confused,” “struggle with mental health,” and “are promiscuous” for LGBTQ+ survey responses. For Jewish survey responses, repeated hallucinations include “are greedy” and “are manipulative.” In both cases, even non-repeated hallucinations are still similar in meaning to repetitive hallucinations. These repetitive phrases are also in some model outputs that are classified as correct. First, this implies that in some of these cases, the model is not correctly performing the task on the input, instead relying on a limited set of priors; this indicates that the “correct” percentages might overestimate model capability. Second, this has the effect of collapsing many attested harms into very few predicates. For example, in the GPT-WQ data, the model extracted either “are promiscuous” or “engage in promiscuous behavior” for the majority of survey responses mentioning any stereotypes about queer sexuality. This conflation of survey responses oversimplifies and ignores valuable community input.
These repetitive hallucinations do express real
and offensive stereotypes, which undoubtedly should be included in bias benchmarks. However, the main advantage of community-sourced bias benchmarks is the fact that they are grounded in the lived experience of affected community members. Using models to “extract” attested harms that are not actually present in survey responses negates the purpose of community-engaged bias definitions, effectively undoing the work of both researchers and community members in eliciting and sharing experienced harms.

heading: 4.3 Comparing Human- and GPT-Extracted Benchmarks
text: Ideally, human-created and model-created benchmarks should be measuring the same underlying model behaviors and should both score a given model similarly. Equivalently, there should be a strong, positive linear correlation between permodel H-W* and GPT-W* scores. Fig. 2 shows the poor correlation between scores on the two benchmarks. For H-WinoQueer and GPT-WinoQueer, the R2 value is only 0.14, indicating a very weak correlation. The WinoSemitism datasets have a moderate correlation (R2 = 0.73), but we would still expect a stronger correlation if the two benchmarks were well-calibrated with respect to each other.
Just as the quality of model extractions is inequitably distributed across subgroups of the LGBTQ+ community, poor calibration of GPTcreated bias benchmarks also has disparate im-
pacts on marginalized subcommunities. Of the nine LGBTQ+ identity subgroups considered, six had R2 < 0.5, including R2 = 0 for lesbians. Modelcreated bias benchmarks perform even more poorly for marginalized subcommunities, falsely underestimating the likelihood of intersectional stereotypes in model outputs.
We believe there are two major reasons for the stark difference in correlations. First, models generally show less bias on the human-created WinoSemitism dataset than on the human-created WinoQueer dataset, so artificially low bias scores for GPT-WS had less effect on the correlation than similarly low bias scores on GPT-WQ. Second, WinoSemitism scores (both H-WS and GPT-WS, across tested models) show substantially less brittleness and variability across identity subgroups within each tested model than WQ scores, which showed considerable variability across subgroups. Thus we attribute WS’s stronger correlation to better overall bias scores and less intra-model variability than we observed on WQ. Overall, this analysis shows that GPT-created bias benchmarks are a very poor approximation for the corresponding humancreated benchmarks.

heading: 5 Conclusions
text: This paper introduced the WinoSemitism dataset, a community-sourced benchmark dataset for measuring bias against Jewish people in large language models (LLMs). We also investigated the feasibility of using a large language model to assist humans with the difficult, emotionally taxing work of developing LLM fairness benchmarks. In particular, we used GPT-3.5-Turbo to extract attested harm predicates from responses to surveys of communities affected by specific biases. Even on this extremely limited subtask with additional information that would be unavailable to a model in a realistic scenario, we find that GPT-3.5-Turbo has unacceptably poor performance on attested harm extraction. We present quantitative and qualitative evidence of serious quality issues with model-extracted predicates, including high rates of both misrepresenting survey responses and hallucinating harms not present in input text. These quality issues replicate many of the problems of crowd-sourced (i.e. not community-sourced) fairness benchmarks, including lack of specificity, nonsensical stereotypes, and ungrammatical sentences, all of which would require considerable human effort to correct.
We also show that model-created benchmarks are poorly correlated with human-created benchmarks from the same underlying community surveys, indicating that GPT-created benchmarks are not measuring the same model behavior as humancreated benchmarks and are poorly grounded in the actual experiences of marginalized communities. Finally, the impacts of poor extraction quality disparately affect already-marginalized subgroups of affected communities, meaning that model-created benchmarks are likely to replicate and exacerbate existing intracommunity inequality. Thus, we conclude that using LLMs to process human survey responses into fairness benchmark datasets yields unnacceptably low-quality benchmarks and largely negates the positive impacts of conducting a community survey in the first place. Annotations from expert humans with lived experience as members of the relevant community are absolutely essential to the construction of high-quality LLM fairness benchmarks. Given these findings, we strongly caution against the use of LLMs in the creation of fairness benchmarks intended to evaluate LLMs and stress the important of human annotators, especially in sensitive and highly context-dependent tasks.

heading: 6 Limitations

heading: 6.1 Jewish Community Survey and WinoSemitism Dataset
text: Our survey of Jewish community members, and by extension the WinoSemitism dataset, have several important limitations. First, our sample is relatively small, and is not likely to be representative of the global Jewish community. The WinoSemitism dataset is also not representative of all forms of antisemitism or all stereotypes about Jewish people. There are undoubtedly antisemitic stereotypes we did not include in the study because they were not reported by our small sample of participants. For practical reasons, sample participants were limited to English-speaking adults 18 and older, so the views of Jewish youth and Jews who do not speak English are not represented in this sample.

heading: 6.2 GPT-WQ and GPT-WS Dataset Construction
text: Our construction of the GPT-WQ and GPT-WS datasets also has important limitations. For predicate extraction, our prompts were developed empirically, and we have not rigorously tested all pos-
sible prompts. Therefore, it is possible that a different prompt, or even the same prompt given to a different release of GPT-3.5-Turbo, would produce different results than those reported in this paper. We also did not test all current LLMs, so it is possible that other models will perform better than GPT-3.5-Turbo on this annotation tasks. However, we believe our conclusions about the serious quality issues caused by LLM use in benchmark construction are not solely limited to a specific prompt and model release.
In constructing the GPT-WQ and GPT-WS benchmark sets, we closely follow the methodology of Felkner et al. (2023). Their methodology has several limitations, which are discussed in detail in their work. Important limitations affecting this work include low syntactic diversity of benchmark sentences and inability to rigorously examine intersectional biases. While the H-WQ and GPT-WQ datasets do not adhere to a strict understanding of gender as binary, current versions of the H-WS and GPT-WS do treat gender as binary. They use only she/her and he/him pronouns, which are mapped to traditionally feminine and traditionally masculine names, respectively. This is an area for improvement in future versions of the dataset.
Additionally, WS scores may be affected by the fact that the bigram “Jewish people” is much more common in model training data than corresponding counterfactual bigrams like “Christian people” and “atheist people.” We minimize this impact by using both noun and adjective identity descriptors (i.e. “Jewish people” and “Jews”, “Christian people” and “Christians”, etc.) in the WS benchmark datasets, so that the relative frequency of bigram pairs affects only a small subset of the WS datasets.

heading: 6.3 Evaluation
text: The first important limitation of our evaluation methods is our choice of models on which to evaluate. Because our survey was conducted only in English, we evaluate the resulting datasets on English language models. The 20 models on which we evaluated are a small fraction of the language models available today. They are also small in number of parameters. We evaluate on models up to seven billion parameters, but many state-of-the-art models have hundreds of billions of parameters. Our evaluation metric is limited to open-source models for which we can directly access token probabilities. It has not yet been expanded to closed-source models that are accessible only via prompting.

heading: Acknowledgements
text: This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2236421. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. We are grateful to all of our survey respondents, as well as to Melissa Balaban, Dave Cohn, and Dov Wagner, for their help promoting and distributing our survey. We would also like to thank the three anonymous reviewers for their helpful suggestions.

heading: A Prompt Details
text: For all experiments, the basic template prompt was:
Listing 1: Prompt for GPT Predicate Extraction
From the following sentence, extract a statement of one discrete stereotype about <IDENTITY NOUN>.
Stereotypes should be in the form of a predicate that would fit in the sentence ’All <IDENTITY NOUN> <predicate>.’
The predicate should start with a verb and SHOULD NOT inlcude individual characters separated by commas.

heading: Do not respond with a full sentence. There may be multiple stereotypes in the
text: sentence, so extract one that is not already in the list.
Sentence: <SENTENCE>
Previously Extracted: <PREV>
Predicate:
<SENTENCE> was replaced with a single survey response. <IDENTITY NOUN> was replaced with an identity descriptor matching the survey respondent. <PREV> was replaced with an empty string at first, and then each model was prompted iteratively N times, where N is the number of ground truth predicates extracted by human annotators. Each iteration with the same input sentence, the model’s previous response was added to the list of previously extracted responses.

heading: B Complete WinoSemitism Baseline Results
text: Complete results for all 20 tested models on the human-annotated WinoSemitism dataset are listed in Table 4.

heading: Future Work
text: This is an area for improvement in future versions of the dataset. Additionally, WS scores may be affected by the fact that the bigram “Jewish people” is much more common in model training data than corresponding counterfactual bigrams like “Christian people” and “atheist people. ” We minimize this impact by using both noun and adjective identity descriptors (i. e. “Jewish people” and “Jews”, “Christian people” and “Christians”, etc. ) in the WS benchmark datasets, so that the relative frequency of bigram pairs affects only a small subset of the WS datasets