WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models
Year: 2023

Abstract:
Content Warning: This paper contains examples of homophobic and transphobic stereotypes. We present WinoQueer: a benchmark specifically designed to measure whether large language models (LLMs) encode biases that are harmful to the LGBTQ+ community. The benchmark is community-sourced, via application of a novel method that generates a bias benchmark from a community survey. We apply our benchmark to several popular LLMs and find that off-the-shelf models generally do exhibit considerable anti-queer bias. Finally, we show that LLM bias against a marginalized community can be somewhat mitigated by finetuning on data written about or by members of that community, and that social media text written by community members is more effective than news text written about the community by non-members. Our method for community-in-the-loop benchmark development provides a blueprint for future researchers to develop community-driven, harms-grounded LLM benchmarks for other marginalized communities.

heading: 1 Introduction
text: Recently, there has been increased attention to fairness issues in natural language processing, especially concerning latent biases in large language models (LLMs). However, most of this work focuses on directly observable characteristics like race and (binary) gender. Additionally, these identities are often treated as discrete, mutually exclusive categories, and existing benchmarks are illequipped to study overlapping identities and intersectional biases. There is a significant lack of
∗Work done at USC Information Sciences Institute.
work on biases based on less observable characteristics, most notably LGBTQ+ identity (Tomasev et al., 2021). Another concern with recent bias work is that “bias” and “harm” are often poorly defined, and many bias benchmarks are insufficiently grounded in real-world harms (Blodgett et al., 2020).
This work addresses the lack of suitable benchmarks for measuring anti-LGBTQ+ bias in large language models. We present a communitysourced benchmark dataset, WinoQueer, which is designed to detect the presence of stereotypes that have caused harm to specific subgroups of the LGBTQ+ community. This work represents a significant improvement over WinoQueer-v0, introduced in (Felkner et al., 2022). Our dataset was developed using a novel community-in-the-loop method for benchmark development. It is therefore grounded in real-world harms and informed by the expressed needs of the LGBTQ+ community. We present baseline WinoQueer results for a variety of popular LLMs, as well as demonstrating that antiqueer bias in all studied models can be partially mitigated by finetuning on a relevant corpus, as suggested by (Felkner et al., 2022).
The key contributions of this paper are:
• the WinoQueer (WQ) dataset, a new community-sourced benchmark for antiLGBTQ+ bias in LLMs.1
• the novel method used for developing WinoQueer from a community survey, which can be extended to develop bias benchmarks for other marginalized communities.
• baseline WinoQueer benchmark results on BERT, RoBERTa, ALBERT, BART, GPT2,
1https://github.com/katyfelkner/winoqueer
9126
OPT, and BLOOM models, demonstrating significant anti-queer bias across model types and sizes.
• versions of benchmarked models, that we debiased via finetuning on corpora about or by the LGBTQ+ community.

heading: 2 Related Work
text: Although the issue of gender biases in NLP has received increased attention recently (Costa-jussà, 2019), there is still a dearth of studies that scrutinize biases that negatively impact the LGBTQ+ community (Tomasev et al., 2021). Devinney et al. (2022) surveyed 176 papers regarding gender bias in NLP and found that most of these studies do not explicitly theorize gender and that almost none consider intersectionality or inclusivity (e.g., nonbinary genders) in their model of gender. They also observed that many studies conflate “social” and “linguistic” gender, thereby excluding transgender, nonbinary, and intersex people from the discourse.
As (Felkner et al., 2022) observed, there is a growing body of literature that examines antiqueer biases in large language models, but most of this work fails to consider the full complexity of LGBTQ+ identity and associated biases. Some works (e.g. Nangia et al., 2020) treat queerness as a single binary attribute, while others (e.g. Czarnowska et al., 2021) assume that all subgroups of the LGBTQ+ community are harmed by the same stereotypes. These benchmarks are unable to measure biases affecting specific LGBTQ+ identity groups, such as transmisogyny, biphobia, and lesbophobia.
Despite such efforts, scholars have pointed out the lack of grounding in real-world harms in the majority of bias literature. For instance, Blodgett et al. (2020) conducted a critical review of 146 papers that analyze biases in NLP systems and found that many of those studies lacked normative reasoning on “why” and “in what ways” the biases they describe (i.e., system behaviors) are harmful “to whom.” The same authors argued that, in order to better address biases in NLP systems, research should incorporate the lived experiences of community members that are actually affected by them. There have been a few attempts to incorporate crowd-sourcing approaches to evaluate stereotypical biases in language models such as StereoSet (Nadeem et al., 2021), CrowS-Pairs (Nangia et al., 2020), or Gender Lexicon Dataset (Cryan et al.,
2020). Névéol et al. (2022) used a recruited volunteers on a citizen science platform rather than using paid crowdworkers. However, these studies lack the perspective from specific communities, as both crowdworkers and volunteers were recruited from the general public. While not directly related to LGBTQ+ issues, Bird (2020) discussed the importance of decolonial and participatory methodology in research on NLP and marginalized communities.
Recently, Smith et al. (2022) proposed a bias measurement dataset (HOLISTICBIAS), which incorporates a participatory process by inviting experts or contributors who self-identify with particular demographic groups such as the disability community, racial groups, and the LGBTQ+ community. This dataset is not specifically focused on scrutinizing gender biases but rather takes a holistic approach, covering 13 different demographic axes (i.e., ability, age, body type, characteristics, cultural, gender/sex, sexual orientation, nationality, race/ethnicity, political, religion, socioeconomic). Nearly two dozen contributors were invovled in creating HOLISTICBIAS, but it is uncertain how many of them actually represent each demographic axis, including the queer community. This study fills the gap in the existing literature by introducing a benchmark dataset for homophobic and transphobic bias in LLMs that was developed via a largescale community survey and is therefore grounded in real-world harms against actual queer and trans people.

heading: 3 Methods

heading: 3.1 Queer Community Survey
text: We conducted an online survey to gather community input on what specific biases and stereotypes have caused harm to LGBTQ+ individuals and should not be encoded in LLMs. Unlike previous studies which recruited crowdworkers from the general public (Nadeem et al., 2021; Nangia et al., 2020; Cryan et al., 2020), this study recruited survey respondents specifically from the marginalized community against whom we are interested in measuring LLM bias (in this case, the LGBTQ+ community). This human subjects study was reviewed and determined to be exempt by our IRB. These survey responses are used as the basis of template creation which will be further discussed in the next section.
Survey participants were recruited online through a variety of methods, including university
mailing lists, Slack/Discord channels of LGBTQ+ communities and organizations, and social media (e.g., NLP Twitter, gay Twitter). Participants saw a general call for recruitment and were asked to self-identify if interested in participating. Participants who met the screening criteria (i.e. Englishspeaking adults who identify as LGBTQ+) were directed to the informed consent form. The form warned participants about the potentially triggering content of the survey and explicitly stated that the survey is optional and that participants are free to skip questions and/or quit the survey at any time. The consent form also explained that data would be collected anonymously and short excerpts used to create a publicly available benchmark dataset, but that entire responses and any identifying information would be kept confidential. Personally identifiying information was redacted from responses.
Participants who consented to the research (n=295) answered survey questions on what biases or stereotypes about their gender and/or sexual/romantic orientation or about the LGBTQ+ community in general have personally caused them harm. Example survey questions are listed in Table 1. We used an intentionally broad definition of harm: “emotional and psychological discomfort, as well as physical violence, discrimination, bullying and cyberbullying, adverse material or financial impacts, and loss of personal or professional opportunities.” In addition, participants were asked to self-identify their gender and sexuality; the results of which are summarized in Table 2. There were also optional demographic questions about race/ethnicity, age range, and country of residence; respondent statistics are listed in Appendix A.

heading: 3.2 WinoQueer Template Creation
text: We introduce the first “community-in-the-loop” bias benchmark dataset, WinoQueer. It was modeled after the CrowS-Pairs (Nangia et al., 2020) paired sentence bias probing task. As far as the authors are aware, this dataset is the first to explore identity-specific anti-queer and anti-trans biases by
incorporating input directly from the affected community. Each sentence in the WinoQueer benchmark is a 4-way Cartesian product of:
Template sentences: Templates are the general structure into which other elements are slotted. Our choice of templates was informed by Cao et al. (2022). An example template is: I don’t like <name> because they are <identity>.
Names/pronouns: For names, we chose the 20 most common male and female names from the US census. We then chose 20 nonbinary and unisex names from Kirby Conrod’s2 informal survey of nonbinary names for linguistics examples and Fivethirtyeight’s list of common unisex names.3 For pronouns, we used he, she, and they. Identity descriptors: Starting from the list of gender and sexuality descriptors in Czarnowska et al. (2021), we bucketed the terms into 9 highlevel identity groups: LGBTQ, Queer, Transgender, Nonbinary, Bisexual, Pansexual, Lesbian, Asexual, and Gay. These identities are not mutually exclusive, and LGBTQ+ individuals can fit into one or several. We also selected the terms Cisgender, Cis, Heterosexual, and Straight for use in counterfactual sentences.
Predicates: Predicates were extracted from free-text responses to the survey described in Section 3.1. After sorting results by identity categories, we read all responses and manually coded for the top ways people were discriminated against (i.e. gay people have family issues, trans people are predatory).
We then generated tuples for each combination of templates, names/pronouns, and predicates, subject to the following rules. All names and pronouns were combined with identity descriptors LGBTQ, Queer, Transgender, Bisexual, Asexual, and Pansexual. Nonbinary names and they/them pronouns were combined with the Nonbinary identity descriptor. Gay was combined with male and nonbinary
2http://www.kirbyconrod.com 3https://fivethirtyeight.com/features/there-are-922-
unisex-names-in-america-is-yours-one-of-them/
names, he/him, and they/them; Lesbian was combined with female and nonbinary names, she/her, and they/them.
After generating sentences from tuples, we paired each sentence with a counterfactual sentence that replaced its identity descriptor with a corresponding non-LGBTQ+ identity. For sentences containing sexuality descriptors Gay, Bisexual, Lesbian, Pansexual, and Asexual, each sentence was duplicated and paired with a counterfactual replacing the descriptor with “straight” and another replacing the descriptor with “heterosexual.” Similarly, sentences containing gender identity descriptors Transgender and Nonbinary were paired with counterfactuals containing “cisgender” and “cis.” Sentences containing LGBTQ and Queer, which are broader terms encompassing both sexuality and gender, were paired with all four possible counterfactuals. Table 3 shows example sentence pairs from the dataset.
Overall, the WinoQueer benchmark dataset contains 45540 sentence pairs covering 11 template sentences, 9 queer identity groups, 3 sets of pronouns, 60 common names, and 182 unique predicates. A unique strength of the WinoQueer dataset is that it is fully human-created and human-audited. We chose this approach for two reasons. First, Blodgett et al. (2020) have uncovered data quality issues with crowdsourced bias metrics; second, Bender et al. (2021) advocate for careful human auditing of datasets, especially bias benchmarks.
A Note on Terminology We grouped names, pronouns, and identity descriptors in this way in order to capture gender-based stereotypes about LGBTQ+ individuals while still allowing for diversity of gender identity and expression. The “lesbian” identity descriptor provides a natural way to explore both misogynistic and homophobic stereotypes about queer women. We decided that it was
important for our benchmark to have similar capability to measure gender-based stereotypes about queer men. While the word “gay” can refer to people of any gender and many women do self-identify as gay, it was also the closest analogy to “lesbian” for the purpose of measuring intersectional stereotypes about orientation and gender. Therefore, the WinoQueer benchmark uses “gay” to refer to gay men specifically and “queer” as a more general umbrella term. We hope that this and other bias benchmarks will continue to evolve with language use in the LGBTQ+ community.

heading: 3.3 Data Collection
text: To debias models and improve their performance on the WQ benchmark, we finetuned them on two datasets: QueerNews, containing articles from US national news media addressing LGBTQ+ issues, and QueerTwitter, containing Tweets about LGBTQ+ topics. The time frame of both datasets is Jan 1, 2015 – Sep 30, 2022.
We collected QueerTwitter by using the Twitter Academic API to conduct a retroactive search of Tweets. Data was retrieved in accordance with Twitter’s Terms of Service and personally identifying information was redacted from the final data set. For search terms, we used anti-trans bill numbers retrieved from the “Legislative Tracker: Anti-Transgender Legislation” website4, which tracks proposed state and federal legislation that would limit the rights of trans people in the United States, as well as hashtags commonly used by those fighting anti-trans legislation. We iteratively analyzed co-occurring hashtags with regard to antitrans bills to build a more comprehensive search term list. The resultant list included hashtags related to anti-trans bills (i.e., #transrightsarehu-
4https://freedomforallamericans.org/legislativetracker/anti-transgender-legislation/
manrights, #transbill, #KillTheBill, #antitransbill, #DontSayGay, #DontSayGayBill) and those related to LGBTQ+ events (i.e., #transdayofvisibility, #lesbianvisibilityday, #bisexualawarenessweek, #stonewall, #stonewall50). We conducted a random sample of relevant Tweets for each day in the time frame. After filtering, our second search with co-occuring hashtags included yields a total of 4,339,205 tweets (4,122,244 sentences).
QueerNews was collected using the open source platform Media Cloud.5 We conducted a keyword search based on anti-trans bill numbers and search terms related to anti-trans bills (i.e., anti-trans bill, trans bill, anti-trans) and LGBTQ+ identity (i.e., lgbtq, lgbt, gay, lesbian, queer, trans, bisexual). For MediaCloud, we used more general search terms related to the LGBTQ+ community because Media Cloud yields fewer results compared to Twitter when using the same search terms. This resulted in a corpus of 118,894 news articles (4,108,194 sentences). New articles were retrieved abiding by Media Cloud’s Terms of Use.

heading: 3.4 Evaluation Metrics
text: Evaluation on WQ follows the methodology of Nangia et al. (2020), which introduced a novel pseudo-log-likelihood metric for bias in masked language models. This metric can be reported from 0 to 1 or 0 to 100; for consistency, we always report scores out of 100. For a sentence S(s1, s2, . . . sn), each token shared between the two templates (unmodified tokens, U ) is masked one-at-a-time, while the modified tokens (M ) are held constant, summing the probability of predicting the correct masked token for each possible position of the mask. Their scoring function is formulated
5https://mediacloud.org
score(S) = 100 |U |∑
i=1
logP (ui ∈ U |U\ui ,M, θ)
(1) This function is applied to pairs of more stereotypical (i.e. stating a known stereotype or bias about a marginalized group) and less stereotypical sentences (stating the same stereotype or bias about the majority group). The bias score is the percentage of examples for which the likelihood of the more stereotypical sentence is higher than the likelihood of the less stereotypical sentence. A perfect score is 50, i.e. the langauge model is equally likely to predict either version of the sentence. A score greater than 50 indicates that the LM is more likely to predict the stereotypical sentence, meaning the model encodes social stereotypes and is more likely to produce biased, offensive, or otherwise harmful outputs.
This metric is only applicable to masked language models. However, we generalize their metric by introducting an alternative scoring function for autoregressive language models:
score(S) = 100 |U |∑
i=1
logP (ui|s<ui , θ) (2)
where s<ui is all tokens (modified or unmodified) preceding ui in the sentence S. Intuitively, we ask the model to predict each unmodified token in order, given all previous tokens (modified or unmodified). For autoregressive models, the model’s beginning of sequence token is prepended to all sentences during evaluation. While the numeric scores of individual sentences are not directly comparable between masked and autoregressive models, the bias score (percentage of cases where the model is more likely to predict more stereotypical sentences) is comparable across model types and scoring functions.

heading: 3.5 Model Debiasing Via Fine-tuning
text: We selected the following large pre-trained language model architectures for evaluation: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2020), BART (Lewis et al., 2020), GPT2 (Radford et al., 2019), OPT (Zhang et al., 2022), and BLOOM (Workshop, 2022). Details of model sizes and compute requirements for finetuning can be found in Table 4. All models were trained on 1 node with 2 GPUs, and the time reported is the total number of GPU hours. In addition to finetuning, we used about 218 GPU hours for evaluation and debugging. In total, this project used 2,256 GPU hours across NVIDIA P100, V100, and A40 GPUs.
We aimed to choose a diverse set of models representing the current state of the art in NLP research, at sizes that were feasible to finetune on our hardware. We produce two fine-tuned versions of each model: one fine-tuned on QueerNews, and one finetuned on QueerTwitter. For QueerNews, articles were sentence segmented using SpaCy (Montani et al., 2023) and each sentence was treated as a training datum. For QueerTwitter, each tweet was treated as a discrete training datum and was normalized using the tweet normalization script from Nguyen et al. (2020). In the interest of energy efficiency, we did not finetune models over 2B parameters. For these four models (OPT-2.7b, OPT-6.7b, BLOOM-3b, and BLOOM-7.1b), we report only WQ baseline results.
Most models were fine-tuned on their original pre-training task: masked language modeling for BERT, RoBERTa, and ALBERT; causal language modeling for GPT2, OPT, and BLOOM. BART’s pre-training objective involved shuffling the order of sentences, which is not feasible when most tweets only contain a single sentence. Thus, BART was finetuned on causal language modeling. Models were finetuned for one epoch each, with instantaneous batch size determined by GPU capacity, gradient accumulation over 10 steps, and all other hyperparameters at default settings, following Felkner et al. (2022). We evaluate the original offthe-shelf models, as well as our fine-tuned versions, on the WinoQueer benchmark.

heading: 4 Results and Discussion

heading: 4.1 Off-the-shelf WinoQueer Results
text: Table 5 shows the WinoQueer bias scores of 20 tested models. These bias scores represent the percentage of cases where the model is more likely to output the stereotypical than the counterfactual sentence. A perfect score is 50, meaning the model is no more likely to output the offensive statement in reference to an LGBTQ+ person than the same offensive statement about a straight person. The average bias score across all models is 70.61, meaning the tested models will associate homophobic and transphobic stereotypes with queer people about twice as often than they associate those same toxic statements with straight people.
All 20 models show some evidence of anti-queer bias, ranging from slight (55.93, ALBERT-xxl-v2) to gravely concerning (97.86, GPT2). In general, the masked language models (BERT, RoBERTa, ALBERT, mean bias score 60.02) seem to show less anti-queer bias than the autoregressive models (GPT2, BLOOM, OPT, mean bias score 92.25), but this result is specific to the WQ test set and may or may not generalize to other bias metrics and model sets.6 BERT and RoBERTa models show significant but not insurmountable bias. We chose to include ALBERT in our analysis because we were curious whether the repetition of (potentially bias-inducing) model layers would increase bias scores, but this does not seem to be the case, as ALBERT models have slightly lower bias scores
6BART is excluded from all masked vs. autoregressive comparisons because it does not fit neatly into either category. It has a BERT-like encoder and GPT2-like decoder, and can be used for both mask-filling and generative tasks.
than BERT and RoBERTa. Among autoregressive models, GPT2 shows slightly more bias, possibly due to its Reddit-based training data.
Interestingly, while Felkner et al. (2022) and many others have shown that larger models often exhibit more biases, we find that WinoQueer bias scores are only very weakly correlated with model size.7 Additionally, when we separate masked and autoregressive language models to account for the fact that the autoregressive models tested were much larger in general than the masked models, no correlation is observed within either group of models. These results suggest that model architecture is more predictive of WQ bias score than model size, and that larger models are not automatically more dangerous than smaller variants.
Another interesting result is the wide variation in observed bias across subgroups of the LGBTQ+ community. Queer has the lowest average bias
7measured in number of parameters. R2 value for this correlation is .203.
score of the 9 identity subgroups tested (61.86), while Lesbian and Asexual have the highest bias scores (both 77.61). Transphobic bias is observed in most models, but it is not substantially more severe than the observed homophobic bias. From the large differences between overall WQ results on a model and results of that model for each subpopulation, it is clear that individual models have widely different effects on different subpopulations. In general, masked models tend to have a larger magnitude of deltas between overall score and subgroup score than autoregressive models, suggesting that masked models are more likely to exhibit biases that are unevenly distributed across identity groups.

heading: 4.2 Finetuning for Debiasing Results
text: Finetuning results are reported in Table 5. In general, we find that finetuning on both QueerNews and QueerTwitter substantially reduces bias scores on the WQ benchmark. In fact, the finetuning
is so effective that it sometimes drives the bias score below the ideal value of 50, which is discussed in Section 5 below. It is likely that the finetuning results could be better calibrated by downsampling the finetuning data or a more exhaustive, though computationally expensive, hyperparameter search. QueerTwitter is generally more effective than QueerNews, which supports our hypothesis that direct community input in the form of Twitter conversations is a valuable debiasing signal for large language models.
While this method of debiasing via finetuning is generally quite effective, its benefits are not equitably distributed among LGBTQ+ subcommunities. Fig. 1 shows the effectiveness of our finetuning (measured as the average over all models of the difference between finetuned WQ score and baseline WQ score) on the same nine subpopulations of the LGBTQ+ community. The finetuning is most effective for general stereotypes about the entire LGBTQ+ community. It is much less effective for smaller subcommunities, including nonbinary and asexual individuals. Twitter is more effective than news for most subpopulations, but news performs better for the queer, nonbinary, and asexual groups. In fact, Twitter data has a slightly positive effect on the bias score against nonbinary individuals. How-
ever, the scores represented in the figure are means over all models, and the actual effects on individual models vary widely. It is important to note that while evaluation is separated by identity, the finetuning data is not. These disparities could likely be reduced by labelling the finetuning data at a more granular level and then balancing the data on these labels.

heading: 5 Conclusions
text: This paper presented WinoQueer, a new bias benchmark for measuring anti-queer and anti-trans bias in large language models. WinoQueer was developed via a large survey of LGBTQ+ individuals, meaning it is grounded in real-world harms and based on the experiences of actual queer people. We detail our method for participatory benchmark development, and we hope that this method will be extensible to developing community-in-the-loop benchmarks for LLM bias against other marginalized communities.
We report baseline WQ results for 20 popular off-the-shelf LLMs, including BERT, RoBERTa, ALBERT, BART, GPT-2, OPT, and BLOOM. In general, we find that off-the-shelf models demonstrate substantial evidence of anti-LGBTQ+ bias, autoregressive models show more of this bias than
masked language models, and there is no significant correlation between number of model parameters and WQ bias score. We also demonstrate that WQ bias scores can be improved by finetuning LLMs on either news data about queer issues or Tweets written by queer people. Finetuning on QueerTwitter is generally more effective at reducing WQ bias score than finetuning on QueerNews, demonstrating that direct input from the affected community is a valuable resource for debiasing large models. The prevalence of high WQ bias scores across model architectures and sizes makes it clear that homophobia and transphobia are serious problems in LLMs, and that models and datasets should be audited for anti-queer biases as part of a comprehensive fairness audit. Additionally, the large variance in bias against specific subgroups of the LGBTQ+ community across tested models is a strong reminder that LLMs must be audited for potential biases using both intrinsic, model-level metrics like WQ and extrinsic, tasklevel metrics to ensure that their outputs are fair in the context where the model is deployed.
Our results show that LLMs encode many biases and stereotypes that have caused irreparable harm to queer individuals. Models are liable to reproduce and even exacerbate these biases without careful human supervision at every step of the training pipeline, from pretraining data collection to downstream deployment. As queer people and allies, the
authors know that homophobia and transphobia are ubiquitous in our lives, and we are keenly aware of the harms these biases cause. We hope that the WinoQueer benchmark will encourage allyship and solidarity among NLP researchers, allowing the NLP community to make our models less harmful and more beneficial to queer and trans individuals.
Limitations
Community Survey
The WinoQueer benchmark is necessarily an imperfect representation of the needs of the LGBTQ+ community, because our sample of survey participants does not represent the entire queer community. Crowdsourcing, or volunteer sampling, was used for recruiting survey participants in this study as it has its strength in situations where there is a limitation in availability or willingness to participate in research (e.g., recruiting hard-to-reach populations). However, this sampling method has a weakness in terms of generalizability due to selection bias and/or undercoverage bias. We limited our survey population to English-speakers, and the WinoQueer benchmark is entirely in English. We also limited our survey population to adults (18 and older) to avoid requiring parental involvement, so queer youth are not represented in our sample. Additionally, because we recruited participants online, younger community members are overrepresented, and queer elders are underrepresented. Compared
to the overall demographics of the US, Black, Hispanic/Latino, and Native American individuals are underrepresentend in our survey population. Geographically, our respondents are mostly American, and the Global South is heavily underrepresented. These shortcomings are important opportunities for growth and improvement in future participatory research.
Finetuning Data Collection In an effort to balance the amount of linguistic data retrieved from Media Cloud and Twitter respectively, we had to use additional search terms for Media Cloud as it yielded significantly fewer results than Twitter when using the same search terms. Also, news articles from January to May 2022 are excluded from the news article dataset due to Media Cloud’s backend API issues. Due to the size our datasets and the inexact nature of sampling based on hashtags, it is likely that there are at least some irrelevant and spam Tweets in our sample.
Template Creation Our generated sentences have several limitations and areas for improvement. First, our nine identity subgroups are necessarily broad and may not represent all identities in the queer community. The WinoQueer benchmark is limited to biases about gender and sexual orientation. It does not consider intersectional biases and the disparate effects of anti-LGBTQ+ bias on individuals with multiple marginalized identities. The names used in templates are taken from the US Census, so they are generally Western European names common among middle-aged white Americans. NonEuropean names are not well-represented in the benchmark. Additionally, the benchmark currently only includes he, she, and they personal pronouns; future versions should include a more diverse set of personal pronouns. Finally, sentences are generated from a small set of templates, so they do not represent every possible stereotyping, offensive, or harmful statement about LGBTQ+ individuals. A high WinoQueer bias score is an indicator that a model encodes homophobic and transphobic stereotypes, but a low bias score does not indicate that these stereotypes are absent.
Evaluation and Finetuning We used similar, but not identical, scoring functions to evaluate masked and autoregressive lan-
guage models. It is possible that the metrics are not perfectly calibrated, and that one category of models may be evaluated more harshly than the other. Additionally, some of our finetuned models scored below the ideal bias score of 50. This means that they are more likely to apply homophobic and transphobic stereotypes to heterosexual and cisgender people than to LGBTQ+ people. Many of these stereotypes are toxic and offensive regardless of the target, but others do not carry the same weight when applied to cis and straight individuals. Currently, it is not well-defined what WQ scores under 50 mean, in theory or in practice. This definition will need to be developed in consultation with researchers, end users, and the LGBTQ+ community. This paper only includes results for a small fraction of available pretrained language models, and our results only represent comparatively small models. We present baseline results for models up to 7.1 billion parameters and finetuned results for models up to 1.5 billion parameters, but many of the models in use today have hundreds of billions of parameters. Finally, our results are limited to opensource models and do not include closed-source or proprietary models.

heading: Acknowledgements
text: This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2236421. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. We also wish to thank Dr. Kristina Lerman and Dr. Fred Morstatter, who co-taught the Fairness in AI course where the authors met and this work was initially conceived. Finally, we would like to thank our three anonymous reviewers for their detailed and helpful suggestions.

heading: A Demographics of Survey Respondents
text: Tables 7, 8, 9, 10, and 11 show the self-reported demographic data of WinoQueer survey respondents.

heading: 70+ 0.00

heading: 60–69 0.54

heading: 50–59 1.08

heading: 40–49 5.94

heading: 30–39 12.43

heading: 20–29 54.05

heading: 18–20 24.86
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 6

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Our creation of the WinoQueer benchmark dataset is discussed throughout the paper. The full dataset is in the supplemental material (data .zip) as a CSV. Other scientific artifacts, including our finetuning data and our finetuned models, are discussed in the paper and included in the supplemental material. When we use scientific artifacts created by others, they are cited appropriately.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sec. 3.1, 3.2, 3.4, 3.5, and references section

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Sec. 3.2

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sec. 3.1 and 3.2

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Sec. 3.1 and 3.2

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sec. 3.1 and 3.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We report summary statistics of the survey data and WinoQueer benchmark in sections 3.1-3.3. WQ does not have train/test/dev splits.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Methods in section 3, results in section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3.5

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3.5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? cited in section 3.5, detailed implementation and versioning information is in supplemental material, code.zip
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.1.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Described in section 3.1, full text is available in supplemental material.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.1.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 3.1

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 3.1

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3.1

heading: Future Work
text: These shortcomings are important opportunities for growth and improvement in future participatory research. Finetuning Data Collection In an effort to balance the amount of linguistic data retrieved from Media Cloud and Twitter respectively, we had to use additional search terms for Media Cloud as it yielded significantly fewer results than Twitter when using the same search terms. Also, news articles from January to May 2022 are excluded from the news article dataset due to Media Cloud’s backend API issues. Due to the size our datasets and the inexact nature of sampling based on hashtags, it is likely that there are at least some irrelevant and spam Tweets in our sample. Template Creation Our generated sentences have several limitations and areas for improvement. First, our nine identity subgroups are necessarily broad and may not represent all identities in the queer community. The WinoQueer benchmark is limited to biases about gender and sexual orientation. It does not consider intersectional biases and the disparate effects of anti-LGBTQ+ bias on individuals with multiple marginalized identities. The names used in templates are taken from the US Census, so they are generally Western European names common among middle-aged white Americans. NonEuropean names are not well-represented in the benchmark. Additionally, the benchmark currently only includes he, she, and they personal pronouns; future versions should include a more diverse set of personal pronouns. Finally, sentences are generated from a small set of templates, so they do not represent every possible stereotyping, offensive, or harmful statement about LGBTQ+ individuals. A high WinoQueer bias score is an indicator that a model encodes homophobic and transphobic stereotypes, but a low bias score does not indicate that these stereotypes are absent. Evaluation and Finetuning We used similar, but not identical, scoring functions to evaluate masked and autoregressive lan-
guage models. It is possible that the metrics are not perfectly calibrated, and that one category of models may be evaluated more harshly than the other. Additionally, some of our finetuned models scored below the ideal bias score of 50. This means that they are more likely to apply homophobic and transphobic stereotypes to heterosexual and cisgender people than to LGBTQ+ people. Many of these stereotypes are toxic and offensive regardless of the target, but others do not carry the same weight when applied to cis and straight individuals. Currently, it is not well-defined what WQ scores under 50 mean, in theory or in practice. This definition will need to be developed in consultation with researchers, end users, and the LGBTQ+ community. This paper only includes results for a small fraction of available pretrained language models, and our results only represent comparatively small models. We present baseline results for models up to 7. 1 billion parameters and finetuned results for models up to 1. 5 billion parameters, but many of the models in use today have hundreds of billions of parameters. Finally, our results are limited to opensource models and do not include closed-source or proprietary models