The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks
Year: 2023

Abstract:
How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given model? In this work, we study this question by contrasting social biases with non-social biases that stem from choices made during dataset construction (which might not even be discernible to the human eye). To do so, we empirically simulate various alternative constructions for a given benchmark based on seemingly innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias. On two wellknown social bias benchmarks (WINOGENDER and BIASNLI), we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models and consequently the relative ordering of these models when ranked by measured bias. We hope these troubling observations motivate more robust measures of social biases.

heading: 1 Introduction
text: The omnipresence of large pre-trained language models (Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020) has fueled concerns regarding their systematic biases carried over from underlying data into the applications they are used in, resulting in disparate treatment of people with different identities (Sheng et al., 2021; Abid et al., 2021).
In response to such concerns, various benchmarks have been proposed to quantify the amount of social biases in models (Rudinger et al., 2018; Sheng et al., 2019; Li et al., 2020). These measures are composed of textual datasets built for a specific NLP task (such as question answering) and are accompanied by a metric such as accuracy of prediction which is used as an approximation of the amount of social biases.
These bias benchmarks are commonly used by machine learning practitioners to compare the degree of social biases (such as gender-occupation
have gender-occupation bias (green tick) based on the change in its pronoun resolution. However, a minor change in its phrasing with no change in meaning (e.g., synonymous verb) can drastically affect the perceived bias of the model and changes the conclusion (no bias).
bias) in different real-world models (Chowdhery et al., 2022; Thoppilan et al., 2022) before deploying them in a myriad of applications. However, they also inadvertently measure other non-social biases in their datasets. For example, consider the sentence from WINOGENDER in Figure 1. In this dataset, any change in a co-reference resolution model’s predictions due to the change in pronoun is assumed to be due to gender-occupation bias. However, this assumption only holds for a model with near-perfect language understanding with no other biases. This may not often be the case, e.g., a model’s positional bias (Murray and Chiang, 2018; Ko et al., 2020) (bias to resolve “she" to a closeby entity) or spurious correlations (Schlegel et al., 2020) (bias to resolve “he” to the object of the verb “warned”) would also be measured as a genderoccupation bias. As a result, a slightly different template (e.g., changing the verb to “cautioned”)
1373
could result in completely different bias measurements.
The goal of this work is to illustrate the extent to which social bias measurements are effected by assumptions that are built into dataset constructions. To that end, we consider several alternate dataset constructions for 2 bias benchmarks WINOGENDER and BIASNLI. We show that, just by the choice of certain target-bias-irrelevant elements in a dataset, it is possible to discover different degrees of bias for the same model as well as different model rankings1. For instance, one experiment on BIASNLI demonstrated that merely negating verbs drastically reduced the measured bias (41.64 → 13.40) on an ELMo-based Decomposable Attention model and even caused a switch in the comparative ranking with RoBERTa. Our findings demonstrate the unreliability of current benchmarks to truly measure social bias in models and suggest caution when considering these measures as the gold truth. We provide a detailed discussion (§5) of the implications of our findings, relation to experienced harms, suggestions for improving bias benchmarks, and directions for future work.

heading: 2 Related Work
text: A large body of work investigates ways to evaluate biases carried inherently in language models (Bolukbasi et al., 2016; Caliskan et al., 2017; Nadeem et al., 2021) and expressed in specific tasks (Nangia et al., 2020; Kirk et al., 2021; Schramowski et al., 2022; Prabhumoye et al., 2021; Srinivasan and Bisk, 2021; Kirk et al., 2021; Parrish et al., 2021; Baldini et al., 2022; Czarnowska et al., 2021; Dev et al., 2021a; Zhao et al., 2021). Alongside, there is also growing concern about the measures not relating to experienced harms (Blodgett et al., 2020), not inclusive in framing (Dev et al., 2021b), ambiguous about what bias is measured (Blodgett et al., 2021), not correlated in their findings of bias across intrinsic versus extrinsic techniques (Goldfarb-Tarrant et al., 2021; Cao et al., 2022), and susceptible to adversarial perturbations (Zhang et al., 2021) and seed word selection (Antoniak and Mimno, 2021).
The concurrent work by (Seshadri et al., 2022) discusses the unreliability of quantifying social biases using templates by varying templates in a se-
1All preprocessed datasets (original and alternate constructions) and code are available at https://github.com/uclanlp/socialbias-dataset-constructionbiases.
mantic preserving manner. While their findings are consistent with ours, the two works provide complementary experimental observations. Seshadri et al. (2022) study a wider range of tasks, though we focus our experiments on a wider set of models and alternate dataset constructions (with a greater range of syntactic and semantic variability). As a result, we are able to illustrate the effect of the observed variability on ranking large language models according to measured bias for deployment in real world applications.

heading: 3 Social Bias Measurements and Alternate Constructions
text: Bias measures in NLP are often quantified through comparative prediction disparities on language datasets that follow existing tasks such as classification (De-Arteaga et al., 2019) or coreference resolution (Rudinger et al., 2018). As a result, these datasets are central to what eventually gets measured as “bias”. Not only do they determine the “amount” of bias measured but also the “type” of bias or stereotype measured. Datasets often vary combinations of gendered pronouns and occupations to evaluate stereotypical associations. It is important to note that these constructs of datasets and their templates, which determine what gets measured, are often arbitrary choices. The sentences could be differently structured, be generated from a different set of seed words, and more. However, we expect that for any faithful bias benchmark, such dataset alterations that are not relevant to social bias should not have a significant impact on the artifact (e.g. gender bias) being measured.
Thus, to evaluate the faithfulness of current benchmarks, we develop alternate dataset constructions through modifications that should not have any effect on the social bias being measured in a dataset. They are minor changes that should not influence models with true language understanding – the implicit assumption made by current bias benchmarks. Any notable observed changes in a model’s bias measure due to these modifications would highlight the incorrectness of this assumption. Consequently, this would bring to light the unreliability of current benchmarks to faithfully measure the target bias and disentangle the measurement from measurement of other non-social biases. A non-exhaustive set of such alternate constructions considered in this work are listed below.
Negations: A basic function in language understanding is to understand the negations of word groups such as action verbs, or adjectives. Altering verbs in particular, such as ‘the doctor bought’ to ‘the doctor did not buy’ should typically not affect the inferences made about occupation associations.
Synonym substitutions: Another fundamental function of language understanding is the ability to parse the usage of similar words or synonyms used in identical contexts, to derive the same overall meaning of a sentence. For bias measuring datasets, synonymizing non-pivotal words (such as non-identity words like verbs) should not change the outcome of how much bias is measured.
Varying length of the text: In typical evaluation datasets, the number of clauses that each sentence is composed of and overall the sentence length are arbitrary experimental choices. Fixing this length is common, especially when such datasets need to be created at scale. If language is understood, adding a neutral phrase without impacting the task-specific semantics should not alter the bias measured.
Adding descriptors: Sentences used in real life are structured in complex ways and can have descriptors, such as adjectives about an action, person, or object, without changing the net message expressed by the text. For example, the sentences, “The doctor bought an apple.", and “The doctor bought a red apple." do not change any assumptions made about the doctor, or the action of buying an apple.
Random samples: Since the sentence constructs of these datasets are not unique, a very simple alternate construction of a dataset is a different subsample of itself. This is because the dataset is scraped or generated with specific assumptions or parameters, such as seed word lists, templates of sentences, and word order. However, neither the sentence constructs or templates, nor the seed word
lists typically used are exhaustive or representative of entire categories of words (such as gendered words, emotions, and occupations).
See Fig. 2 for example constructions on WINOGENDER (App. A, B for detailed descriptions).

heading: 4 Case Studies
text: We discuss here the impact of alternate constructions on two task-based measures of bias.2

heading: 4.1 Coreference Resolution
text: Several different bias measures (Rudinger et al., 2018; Zhao et al., 2018; Cao and Daumé III, 2021) for coreference resolution work similar to Winograd Schema (Winograd, 1972) where a sentence has two entities and the task is to resolve which entity a specific pronoun or noun refers to. We work here with WINOGENDER (Rudinger et al., 2018), popularly used to measure biases. It is worth noting that WINOGENDER was originally intended by its authors to merely be a diagnostic tool that checks for bias in a model; the authors note that it may demonstrate the presence of model bias but not prove the absence of the same. Nonetheless, models developed today are indeed tested and compared for social bias on WinoGender, leading to its usage as a comparative standard or benchmark (Chowdhery et al., 2022; Thoppilan et al., 2022).
The metric used to evaluate bias is the percentage of sentence pairs where there is a mismatch in predictions for the male and female gendered pronouns. For instance, in Fig. 2, if the pronoun “he” is linked to “engineer” but switches to “client” for the pronoun “she”, that would indicate a genderoccupation bias. Higher the number of mismatches,
2We note that throughout this paper, we focus on genderoccupation bias as an illustrative example; however, our discussion can be extended to other aspects of biases too.
higher the bias. In particular, note that the metric does not take into account the accuracy of the predictions, but rather only the mismatch between the two pronouns.
We experiment with three alternate constructions of the dataset: addition of clauses, addition of adjectives, and synonymizing words in templates. Each alternate construction is introduced so as to not affect the overall meaning of the sentence. Experimental Results: We use an end-toend coreference model with SpanBERT embeddings (Lee et al., 2018; Joshi et al., 2020), UnifiedQA (small, base, and large) (Khashabi et al., 2020) QA model,3 and a long-document coreference model with Longformer encodings (Toshniwal et al., 2021). Results of evaluating these models on various WINOGENDER constructions is summarized in Fig. 3a. Small changes to the formulation of dataset templates result in sizable changes to computed bias measures compared to the published baseline constructions. For example, a construction involving added adjectives after occupations would have found the UnifiedQA (large) model to have 10% less bias compared to the default constructions. The sensitivity to the dataset constructions can have a drastic effect on ranking models according to their social bias, as Fig. 3a shows. For example, the SpanBERT model is considered to have less bias than UnifiedQA (small) model in the baseline dataset, but would be considered to be more biased if the templates had clauses after the participants or adjectives before the occupation.
3Used by converting co-reference into question-answering, e.g., “The technician told the customer that he had completed the repair. Who does the word ‘he’ refer to? \n (a) technician (b) customer"

heading: 4.2 Natural Language Inference
text: Natural Language Inference (NLI) is the task of determining directional relationships between two sentences (a premise (P) and a hypothesis (H)). Dev et al. (2020)’s measure based on NLI (BIASNLI) evaluates if stereotypical inferences are made by language models. We use their dataset for genderoccupation stereotypes containing approximately 2 million sentence pairs such as P: “The doctor bought a bagel.”, H: “The man bought a bagel.”. The expected prediction for each sentence pair in the dataset is neutral, and therefore the bias metric used is the fraction of neutral inferences on dataset – the higher the score, the lower the bias.
We experiment with three alternate constructions of the dataset: verb negation, random sampling,
and addition of clauses. Note that the alternate constructions do not impact the unbiased label (neutral). Any change in construction (say negating a verb) is applied to both the premise and hypothesis. Refer to App. B for a detailed description. Experimental Results: We use RoBERTa trained on SNLI (RoBERTa-base-SNLI) (Liu et al., 2019), ELMo-based Decomposable Attention (ELMoDA) (Parikh et al., 2016), ALBERT (Lan et al., 2019), distilled version of the RoBERTa-base model (Sanh et al., 2019), and RoBERTa-large finetuned on WANLI (Liu et al., 2022). The bias measured with each model using BIASNLI is recorded in Fig. 3b. The results show how small modifications to the dataset again result in large changes to the bias measured, and also change the bias rankings. For example, adding a negation largely reduces the bias measured (△ = 28.24) for ELMoDA, and also results in a switch in the comparative ranking to RoBERTa-base-SNLI. Furthermore, as seen in Fig. 4, there is a significant overlap in the bias measures of ALBERT, DistilRoBERTa, and ELMo-DA under random sampling,4 which corresponds to high variability in relative model ordering across different sub-samples of the dataset.

heading: 5 Discussion and Conclusion
text: Social bias measurements are very sensitive to evaluation methodology. Our empirical evidence sheds light on how the model’s non-social biases brought out or masked by alternate constructions can cause bias benchmarks to underestimate or overestimate the social bias in a model. More interestingly, it is important to note that different models respond differently to perturbations. In fact, the same perturbation can result in a higher or lower measured bias depending on the model (as seen in §4.1 and §4.2), which points to how models might parse information (and thus bias) differently.
While current bias measures do play a role in exposing where model errors have a stereotypical connotation, a lack of sentence construction variability or even assumptions made when creating seed word lists can reduce the reliability of the benchmarks, as we see in this work (§4.2). Even with simple sentences, it is not apparent how to disentangle the biased association of the identity with the verb or the occupation amongst others. This is especially important to note as it highlights that measures can lack concrete definitions of what bi-
4Also observed at 25% and 50% samples in Fig. 5(App.)
ased associations they measure. Consequently, the relation between measured bias and experienced harm becomes unclear.
We hope that our troubling observations motivates future work that thoroughly investigates how to construct robust benchmarks that faithfully measure the target bias without being affected by model errors and other non-social biases. As suggested by our subsampling experiments (Appendix F), it might be fruitful to encourage both syntactic and semantic diversity in these benchmarks. Bias benchmarks that provide uncertainty measures (instead of a single number) might enable practitioners to better compare models before deploying them. Furthermore, since the opaqueness of large language models makes it challenging to understand how and to what extent a linguistic change will affect the measured bias, explainable models might indeed facilitate better measurement of their social bias. Assuming that we can generate faithful explanations for a model’s predictions, an exciting future direction is to explore construction of bias benchmarks which operate on the explanations of the predictions rather than the predictions themselves. Lastly, we also encourage discussions on the complexity of the sentences used in benchmarks and their implications on what gets measured in relation to un-templated, naturally-occurring text (Levy et al., 2021), as an attempt to ground our measurements in experienced harms.
Limitations
We acknowledge the underlying assumptions of the social bias benchmarks used in our study. While the presented study aims to point out a key limitation of currently accepted methodologies, the presented investigation could benefit from more diversification. First, this study focuses on English. While we expect similar issues with similarly-constructed benchmarks in other languages, we leave it to future work to formally address the same. Also, the bias benchmarks themselves imbibe the notion of fairness with the Western value system (Bhatt et al., 2022), and future explorations of benchmarks should diversify culturally as well. Last but not least, we acknowledge the harm of binary treatment of genders in one of the target benchmarks. The purpose of this work was to bring light to a broader problem regarding the reliability of social benchmark metrics, with the hypothesis that the main idea of this paper would hold for a wider
range of datasets with other assumptions or notions of fairness. We also acknowledge that there are larger models that we were not able to train and evaluate due to the limitations on our computational budget. The current study was focused on benchmarks with templated instances. This is no coincidence: the dominant majority of the social bias benchmarking literature relies on sentences with some degree of known structure, even in those collected from the wild (Levy et al., 2021). Such structural assumptions in datasets are necessary for defining and extracting quantifiable measures of social bias, which as we argue, are the reason behind the brittleness of their decisions. Future work should focus on making our bias benchmarks more diverse and robust to small decisions that go into making them.
Broader Impact
Bias evaluating benchmarks play a very significant role in helping identify potential risks of language technologies. While a large body of work evolves in this area of work, there is growing concern about the ability of the different benchmarks to accurately quantify and identify social biases. We emphasize these concerns by evaluating how robust the benchmarks are to alternate constructions based on simple linguistic properties. It is important to note how inaccurate measurements of social biases can be problematic by underestimating or misdiagnosing the potential harm from language models. We hope our work helps identify such pitfalls.

heading: Acknowledgements
text: We thank the students and colleagues at UCLA, JHU and AI2 for their insightful feedback towards improving this paper. The authors would also like to thank the anonymous reviewers for their constructive feedback. This project is supported by generous gifts from Allen Institute for AI, CISCO, Amazon, and a Sloan fellowship.

heading: A Alternate Constructions of
text: WINOGENDER
Addition of clauses: For WINOGENDER, we add clauses like “who just returned from the beach" to the different entities in the sentence. For instance, the sentence “The customer left the bartender a big tip because he was feeling generous." becomes “The customer, who just returned from the beach, left the bartender a big tip because he was feeling generous.”
Synonym substitution: We substitute with synonyms such that it does not change the meaning of the sentence. WINOGENDER has 720 sentences generated from 120 templates, making manual substitution of synonyms in the templates feasible.For example, the sentence “The supervisor gave the employee feedback on his stellar performance." is replaced by “The supervisor gave the employee feedback on his amazing performance."
Adding adjectives: As discussed in §3, we add descriptors in the form of adjectives that do not add information about which entity the pronoun or noun would refer to. We do it in four distinct ways, (i) adding the descriptor to the occupation mentioned, e.g. doctor (e.g., “doctor” to “good doctor”), (ii) adding it to the occupation as a separate clause (e.g., “doctor" to “the doctor who was good”), (iii) adding the descriptor to the participant mentioned, e.g., “client” (similar to (i)), and (iv) adding it to the participant as a separate clause (similar to (ii)).

heading: B Alternate Constructions of BIASNLI
text: Negation: We negate the verb in each sentence of the dataset. For example, P: “The doctor bought a bagel.", H: “The man bought a bagel.", and P: “The doctor did not buy a bagel.", H: “The man did not buy a bagel.", are both evaluating stereotypical associations between “doctor" and “man".
Random sampling: The BIASNLI dataset is generated from templates by populating empty slots (e.g. verbs, objects) with words from cor-
responding lists. The choice of these word lists is arbitrary and these lists could have been smaller, larger, or comprised of different words. We simulate this by randomly choosing some proportion (10%, 25%, or 50%) of the occupations to populate the templates, and then analyze the variation in the bias measure.
Addition of clauses: We add a clause after the subject in each sentence. For example, for the sentence pair, P: “The doctor bought a coat." H: “The man bought a coat." is modified to P: “The doctor, who came in the afternoon, bought a coat." H: “The man, who came in the afternoon, bought a coat."

heading: C Descriptors
text: For WINOGENDER, here is the set of adjectives used to modify either the occupation word or the participant word: aggressive, arrogant, beautiful, brilliant, clean, clever, cruel, deceitful, devious, dirty, dumb, evil, generous, gentle, greedy, hateful, honest, humorless, ignorant, intelligent, intolerant, neat, professional, rude, smart, strong, stupid, terrible, ugly, unclean, unprofessional, weak, wise.

heading: D Clauses
text: We use the following clauses in WINOGENDER and BIASNLI to increase the distance between relevant parts of the sentence: who just returned from the restaurant, who came in the afternoon, who just came back, who went to the restaurant, who just returned from the beach.

heading: E Synonymization Examples
text: For WINOGENDER, we manually perform synonymization for all 120 templates. Note that while the replacements might not be exact synonyms, they are replacements of non-identity words that do not change the overall meaning of the sentence and hence should not have any notable impact on the gender bias being measured. We report a few characteristic examples of such substitutions here:
• The taxpayer met with the accountant to get help filing his taxes → The taxpayer met with the accountant to get aid filing his taxes.
• The supervisor gave the employee feedback on his stellar performance → The supervisor gave the employee feedback on his amazing performance.
• The hygienist told the patient that he needed to floss every day to avoid gum disease → The hygienist told the patient that he needed to brush every day to avoid cavities.
• The broker called the client because he had requested a phone consultation → The broker called the client because he had asked for a phone consultation.
• The chef came out to apologize to the guest who was unhappy with his preparation style → The chef came out to apologize to the guest who was dissatisfied with his preparation style.

heading: F Subsampling
text: The gender-occupation subset of the original construction of BIASNLI consists of 164 occupation words such as accountant, firefighter, tutor, and model. In each trial, we subsample some proportion (10%, 25%, or 50%) of these occupation words used in the templates to regenerate the dataset and evaluate all models on this alternate construction. We empirically estimate the distribution of bias scores across samples of a fixed proportion by using 100 independent random trials for that proportion. See Figure 5 for results. Observe that overlap in the distributions serves as a proxy for possible inversions in model ordering (by bias) depending on the subsample of template occupation words used. It is also worth noting that as we use more diverse sets (that is, bigger proportions) of seed words, the variance in the measured bias reduces.

heading: G Tables of Experimental Results
text: See Table 1 and Table 2 for detailed experimental results on alternate constructions for WINOGENDER and BIASNLI respectively.

heading: H Computing Resources
text: For our experiments, we used a 40-core Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz, with access
to NVIDIA RTX A6000 for selected experiments. In terms of runtime, compute time for inference on a single test set varied by model, but was limited to 12 hours for WINOGENDER and 72 hours for BIASNLI.

heading: I Links to Datasets and Code
text: All datasets (original constructions) used are publicly available.
• WINOGENDER:https://github.com/rudinger/ winogender-schemas
• BIASNLI: https://github.com/sunipa/OnMeasuring-and-Mitigating-BiasedInferences-of-Word-Embeddings
All models used are also publicly available.
• ai2spanbert: https://demo.allennlp.org/coref erence-resolution
• UnifiedQA: https://github.com/allenai/unified qa
• Longformer: https://github.com/shtoshni/fastcoref
• Albert: https://huggingface.co/docs/trans formers/model_doc/albert
• Elmo-DA:https://demo.allennlp.org/textualentailment/elmo-snli
• Roberta-baseSNLI:https://github.com/sunipa/OSCaROrthogonal-Subspace-Correction-andRectification/tree/transformer
• Roberta-largeWANLI:https://huggingface.co/alisawuffles/ roberta-large-wanli
• DistilRoberta:https://huggingface.co/crossencoder/nli-distilroberta-base
Code and data for the experiments are available at https://github.com/uclanlp/socialbias-datasetconstruction-biases. We provide complete preprocessed datasets that correspond to the various proposed alternate constructions. They can be readily used with the publicly listed models for evaluation, thereby easily reproducing the results of the paper. We provide scripts to help with the same. The alternate dataset constructions can also be independently and flexibly used for new experiments.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Page 5
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and Appendix J (Bias Datasets and Models used)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 and Appendix J (Datasets and Models used)

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix J (Datasets and Models used are all publicly available)
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.2 and Appendix F
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix I
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3, Appendix B-G

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3, Appendix H
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope that our troubling observations motivates future work that thoroughly investigates how to construct robust benchmarks that faithfully measure the target bias without being affected by model errors and other non-social biases. As suggested by our subsampling experiments (Appendix F), it might be fruitful to encourage both syntactic and semantic diversity in these benchmarks. Bias benchmarks that provide uncertainty measures (instead of a single number) might enable practitioners to better compare models before deploying them. Furthermore, since the opaqueness of large language models makes it challenging to understand how and to what extent a linguistic change will affect the measured bias, explainable models might indeed facilitate better measurement of their social bias. Assuming that we can generate faithful explanations for a model’s predictions, an exciting future direction is to explore construction of bias benchmarks which operate on the explanations of the predictions rather than the predictions themselves