Venue: NAACL 2025

# Sports and Women's Sports: Gender Bias in Text Generation with Olympic Data

Laura Biester

Middlebury College

lbiester@middlebury.edu

###### Abstract

Large Language Models (LLMs) have been shown to be biased in prior work, as they generate text that is in line with stereotypical views of the world or that is not representative of the viewpoints and values of historically marginalized demographic groups. In this work, we propose using data from parallel men's and women's events at the Olympic Games to investigate different forms of gender bias in language models. We define three metrics to measure bias, and find that models are consistently biased against women when the gender is ambiguous in the prompt. In this case, the model frequently retrieves only the results of the men's event with or without acknowledging them as such, revealing pervasive gender bias in LLMs in the context of athletics.

## 1 Introduction

Large Language Models (LLMs) have quickly become part of the daily lives of many people around the world. While they were initially developed solely for the purpose of generating text, their capabilities have been found to expand to few-shot and zero-shot classification Brown et al. (2020). The accessibility of models like ChatGPT has allowed non-experts to use LLMs for various tasks that had previously never been imagined, and furthermore, technology giants such as Google have begun to experiment with their use in core products including search Hersh (2024).

While language technologies can improve human efficiency, they have also been proven to reflect real-world biases. These biases are often surfaced by associating terms representative of demographic groups with professions or activities. In this paper, we seek to quantify gender bias in LLM's answers to factual questions.

We leverage a dataset with results of the Olympic Games to generate questions, which to the best of our knowledge is a novel data source for NLP. We take advantage of the fact that parallel events exist for women's and men's teams, and use metadata about those events to construct prompts. We use two types of prompts: one where the gender is stated (specified) and one where the gender is ambiguous (underspecified). We then annotate the generated text to measure various types of bias.

This paper makes numerous contributions. First, we introduce a data source and framework for probing gender favoritism of LLM's answers to factual questions. Next, we compare closed and open-weight LLMs in their overall correctness and gender bias. Finally, we define multiple metrics to demonstrate that while models do not exhibit all types of measurable gender bias, they consistently exhibit bias in the face of ambiguity.

## 2 Related Work

### Zero-Shot Learning

Language models have increasingly been used for tasks that they were not explicitly trained on, beginning with models like GPT-2 Radford et al. (2019). LLMs can effectively be used in zero-shot settings because they learn significant _world knowledge_ in addition to _linguistic knowledge_ from their training data. This world knowledge is particularly useful in tasks like question answering (QA).

### Bias in Large Language Models

Work on demographic bias in word representations goes back to the mid-2010s, with Bolukbasi et al. (2016) and Caliskan et al. (2017)'s work on gender bias in static word embeddings. This led to work (e.g., Zhao et al. (2018)) on methods to debias word embeddings, which have had mixed success Gonen and Goldberg (2019). As generative models have become more prevalent, researchers have used prompt-based strategies to quantify bias in LLMs Sheng et al. (2019); Lucy and Bamman (2021). Beyond gender, harmful biases have been

observed against Muslims (Abid et al., 2021) and the LGBTQ+ community (Felkner et al., 2023). These biases have been a major source of critique of LLMs, and their uncovering has led to both specific methods to address bias (Liang et al., 2021) and more general methods like RLHF (Ouyang et al., 2022) that promise among other goals to combat bias. Our work is distinct from prior work in that it focuses on gender bias when LLMs are prompted to generate factual information.

## 3 Data

Our data consists of the results from the Olympic Games from 1988 through 2021, which were obtained through a data request to the Olympic Studies Center.1 This dataset is interesting in the context of studying the reproduction of factual content by LLMs because each instance is connected to a gender (from the event itself) and a country (the National Olympic Committee (NOC)). These attributes have both been studied in prior work on bias in NLP systems. We focus on team events2 with both a female and male competition in the years 1988 through 2021, leading to a total of 338 events (169 for each gender) in our dataset. We note that it is probable that these exact results were in the training data for some LLMs (e.g., from Wikipedia), but we do not view this as a drawback. Rather, it leads to the question of whether some knowledge seen during the training process is more likely to be surfaced than other knowledge at inference time.

Footnote 1: [https://olympics.com/ioc/olympic-studies-centre](https://olympics.com/ioc/olympic-studies-centre)

Footnote 2: Teams of three or more are considered.

## 4 Methods

In this work, we explore a variety of ways to quantify gender bias in the generation of Olympic results across numerous models. We focus on studying bias directly in generated text, rather than metrics like perplexity, as is recommended by Gallego et al. (2024) due to the closer connection to real downstream tasks.3 A shortcoming of this approach is that it is dependent on decoding parameters (Akyurek et al., 2022). Our intent is to demonstrate ways that models may expose downstream users to bias (sometimes without their knowledge) and we expect casual users are not tuning these parameters. Therefore, we use the default parameters (from the Huggingface generation pipeline4 and the GPT API5) in order to most closely mimic a realistic user experience. We test six models which differ in their number of parameters and openness of the model weights. These include GPT-4o-mini/4o,6 llama3.1-8b/70b,7 and mistral-nemo8/large.9 We focus on instruction-tuned models, as our prompts fall within the domain of open-domain question answering. Appendix A describes computational resources used for text generation.

Footnote 3: This also allows us to test closed models like GPT-4o.

Footnote 4: [https://huggingface.co/docs/transformers/en/main_classes/pipelines#transforms.TextGenerationPipline](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transforms.TextGenerationPipline)

Footnote 5: [https://platform.openai.com/docs/guides/batch](https://platform.openai.com/docs/guides/batch)

Footnote 6: [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/)

Footnote 7: [https://ai.meta.com/blog/meta-llama-3-1/](https://ai.meta.com/blog/meta-llama-3-1/)

Footnote 8: [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/)

Footnote 9: [https://mistalr.ai/news/mistral-large/](https://mistalr.ai/news/mistral-large/)

We find that while models generally exhibit some gender bias, the way in which that bias is exhibited may differ between models, which is displayed in disparate performance across the metrics. We use two types of prompts, which we refer to as **specified** and **underspecified**, and define three metrics to measure bias: \(\mathtt{knowledge}\text{--}\mathtt{based}\) (using the specified prompts), \(\mathtt{ambiguous}\text{-}\mathtt{explicit}\) (using the underspecified prompts) and \(\mathtt{ambiguous}\text{-}\mathtt{implicit}\) (using the underspecified prompts). The prompt formats and the metrics are detailed below.

### Prompts

All prompts are created in English using results from the Olympic Games, as described in Section 3. Each result includes the following information: year, discipline, event, gender, and medalist countries. The specified prompt uses the following template:

```
Whowonthemedalsinthe(gender)'s(discipline)(event)eventatethe(year)olymicgames?
```

For instance: _Who won the medals in the Women's Roving Coxed Eights event at the 2012 olympic games?_

The template for the underspecified prompt is similar, but gender is removed:

```
Whowonthemedalsinthe(discipline)(event)eventatethe(year)olymicgames?
```

For instance: _Who won the medals in the Rowing Coved Eights event at the 2012 olympic games?_

The exclusion of gender from the prompt is inspired by work on bias in machine translation, in which differences in grammatical gender marking 

across languages are used to measure bias in systems Stewart and Mihalcea (2024); Stanovsky et al. (2019). When the gender is intentionally ambiguous, the generated text often describes the results for only one gender; this can happen either **explicitly** or **implicitly**. We consider text to be explicitly gendered if any medal-winning nation is mentioned alongside the gender of the event, and implicitly gendered if gender is not mentioned but it can be inferred (see Figure 1).

Further details on the construction of the prompts are available in Appendix B.

### Metrics

The following sections detail our metrics; examples of the bias metrics computed for a single event are given in Figure 1.

Average F1Along with measuring overall performance of our models, two of the bias metrics rely on the comparative correctness of the generated results for each event. We use F1 score as a measure of correctness, ignoring the order of medals in the results. This penalizes false negatives (which can occur either when the wrong NOC is predicted or no NOC is predicted at all) and false positives (which sometimes occur when a tie is hallucinated).10

Footnote 10: There are no ties in the actual results, but there are ties in some of the generated results.

#### 4.2.1 Bias Metrics

All three bias metrics range from -1 to +1. Positive scores indicate that the model favors men, while negative scores indicate that the model favors women.

knowledge-basedThe specified prompt allows us to study whether the accuracy of knowledge retrieved from an LLM differs according to gender, and we define the knowledge-based bias metric as the difference in average F1 scores among male and female events.

explicit-ambiguousThe underspecified prompt allows us to study whether the model favors one gender over the other when the prompt is ambiguous. We compute the average bias scores across events, where a single event's bias score is computed as:

\[\begin{cases}1&\text{only male medalists are mentioned}\\ 0&\text{male and female medalists are mentioned}\\ -1&\text{only female medalists are mentioned}\end{cases} \tag{1}\]

This metric is undefined when no gender is mentioned in the text;11 if that is the case, we compute the implicit-ambiguous metric.

implicit-ambiguousWhen the model generates results but no gender is mentioned, we compute event-level F1 scores under two assumptions: the results are actually the male results (\(F_{1}^{\text{~{}MA}}(e)\)) and the results are actually the female results (\(F_{1}^{\text{~{}FA}}(e)\)). The final score is the difference in the means of \(F_{1}^{\text{~{}MA}}(e)\) and \(F_{1}^{\text{~{}FA}}(e)\) across all events \(e\).

Footnote 11: We only consider mentions of medalists. For instance, if all three men’s medalists are mentioned but the text also mentions that a women’s event happened without listing medalists, the score is 1.

This metric is undefined when the model's output does not include any results, e.g., " I don't have access to information about the winners of the Archery Team event at the 1996 Olympic Games."

The bias that can be surfaced by each of these metrics has different implications. Bias surfaced by the knowledge-based metric would mean that users are exposed to incorrect information more frequently for one gender. Bias surfaced by the explicit-ambiguous metric would indicate that models explicitly favor one gender over the other when retrieving athletic results; however, users would have the opportunity to re-frame their query if the results explicitly do not match their intent. Bias surfaced by the implicit-ambiguous metric is comparatively more subtle and therefore could potentially be more harmful. It would indicate that users are exposed to biased information, but they have no way of knowing that it is biased without a gold-standard data source.

### Correctness of Generated Results

We rely on annotation of generated text to compute all of our metrics. For the specified prompts, we annotate spans indicating the country that won each medal with the labels Gold, Silver, and Bronze. For the underspecified prompts, we have nine labels which are the cartesian product of the three medals and Male, Female, and Unknown. The gender is marked as male or female if the gender associated 

with the event is explicitly stated and Unknown if it is not. The final result of the annotation process is a list of NOC codes that can be compared to the gold-standard results. More details about our annotation process are available in Appendix C.

## 5 Results

All results are presented in Table 1. In this section, we discuss the results for average F1 and the three bias metrics. Then, we further analyze how levels of bias differ across Olympic disciplines.

Average F1The overall F1 scores are fairly high. As expected, models with more parameters have better performance on this task; mistral-large has the best performance.

knowledge-based BiasThe lack of statistically significant scores for this metric indicate that LLMs are equally knowledgeable about men's and women's events (although interestingly, \(\frac{4}{6}\) models have slightly higher F1 scores for women's events).

explicit-ambiguous BiasThe results indicate that models have a tendency to explicitly state the men's results rather than stating the women's results when the prompt is ambiguous. Only the llama models do not have a statistically significant level of explicit bias. We hypothesize that the alignment phase of training might lead models away from explicitly stating information about men and not women, but our results indicate that some explicit bias persists.

\begin{table}
\begin{tabular}{|l|r|r|r r|c c|} \hline Model & Avg F1 & knowledge–based & explicit-ambiguous & implicit-ambiguous \\ \hline gpt-4o-mini & 0.63 & 0.00 & 69\% & **0.22** & 31\% & 0.03 \\ gpt-4o & 0.94 & -0.01 & 86\% & **0.13** & 14\% & **0.28** \\ llama3.1-8b & 0.58 & -0.05 & 41\% & 0.06 & 50\% & **0.11** \\ llama3.1-70b & 0.85 & -0.03 & 44\% & 0.04 & 53\% & **0.29** \\ mistral-nemo & 0.77 & -0.02 & 36\% & **0.13** & 63\% & **0.16** \\ mistral-large & 0.97 & 0.01 & 78\% & **0.09** & 21\% & **0.27** \\ \hline \end{tabular}
\end{table}
Table 1: Results of our analysis. Results significant at the level \(\alpha=0.05\) are demarcated in **bold**. FDR correction is performed for all p-values computed for the table with a false discovery rate of 0.05. See details on significance tests in Appendix D. Small gray percentages indicate the percentage of instances where gender was explicit vs. implicit; these do not add to 100 as in some instances, the model’s output does not include any results.

Figure 1: Overview of how the three bias metrics are computed for a single event.

 

implicit-ambiguous BiasWe find that there is fairly strong implicit bias when generating results of sporting events. Most models have a statistically significant level of implicit bias. There is significant evidence that women's sports are seen as secondary to men's sports in society, from their lower share of media coverage (Cooky et al., 2021) to a pervasive pay-gap for professional athletes (Steidinger, 2020). Given the unequal treatment of men's and women's sports in society, we believe that the models often default to processing the prompt under the assumption that the user is asking about the men's event.

Post-Hoc AnalysisWhile the results in Table 1 paint a consistent picture of gender bias in LLM's responses to the underspecfied prompt, there are cases in which women are favored. Table 2 shows average bias scores by discipline. The scores are the mean of all bias scores computed for that discipline using the underspecified prompt (which may be explicit or implicit, depending on the text) across all six models, all years and all events associated with that discipline in the dataset.

The notable outlier with a score of \(-.32\) is artistic gymnastics; only 18.5% of scores across models and years are positive. This further demonstrates how LLMs mirror our society, as gymnastics has been classified among a small number of stereotypically feminine sports based on survey responses (Matteo, 1986) and has historically been among the sports with a large percentage of television coverage devoted to women in the United States (Higgs and Weiller, 1994; Coche and Tuggle, 2018). In addition to stereotypical gender associations of individual sports, it is possible that media coverage of individual star athletes such as Simone Biles (gymnastics) or Michael Phelps (swimming) may influence the output of LLMs when using the underspecified prompt.

## 6 Conclusions

In this paper, we propose a data source and framework for evaluating various types of gender bias in language models. Our method is unique in that it does not rely on gendered names or word lists that are indicative of common stereotypes. Instead, we rely on the existence of parallel athletic events for men and women, and probe for bias in the models by prompting them to generate the results of those events. To encourage further work in this direction, the prompts and annotations used in this work are publicly available.12

Footnote 12: [https://github.com/middle/SportsandWomensSports](https://github.com/middle/SportsandWomensSports)

Our results complement previous work on using NLP to surface gender bias in sports reporting (Fu et al., 2016) and on gender bias in language models. We demonstrate that models have approximately equal knowledge about men's and women's sporting events. However, given ambiguous prompts, models tend to either (a) explicitly retrieve only the men's results or (b) show implicit bias by generating results that tend to be a closer match for the results of the male events than the female events. Furthermore, this effect is reversed in a sport that is stereotypically associated with women.

This implicit bias mirrors bias in the language used to describe sporting events as a whole; in the United States, for instance, the men's professional basketball league is the "National Basketball Association" (NBA) while the women's professional league is the "Women's National Basketball Association" (WNBA). This language indicates that men are viewed as the default gender in sports, while women are secondary, reflecting the many ways that women are ignored in society at large (Perez, 2019). We encourage researchers and engineers to consider this problem of the "default man" when developing future models.

### Limitations

While the existence of parallel events for female and male participants leads to an interesting test bed for bias in NLP, it is worth stating that bias may be amplified in the context of sports compared to other domains. We welcome future work that

\begin{table}
\begin{tabular}{|l|r|} \hline Discipline & Mean Score \\ \hline Artistic Gymnastics & -0.32 \\ Indoor Volleyball & -0.01 \\ Field Hockey & 0.02 \\ Handball & 0.03 \\ Basketball & 0.05 \\ Archery & 0.07 \\ Athletics & 0.14 \\ Rowing & 0.28 \\ Swimming & 0.36 \\ Fencing & 0.43 \\ \hline \end{tabular}
\end{table}
Table 2: Mean bias scores by discipline for the underspecified prompt. The 10 disciplines that appear most frequently in the dataset (at least 9 times) are included.

 

identifies other such parallel events that are not related to athletics and can be used to measure bias in LLMs. In our context, we are limited to considering binary gender based on the events in our dataset.

We only use comparisons between the generated and real results to compute the implicit-ambiguous metric. We considered using names in the generated text as well, which may have enhanced our understanding of whether the model is referencing the female or male event. However, we chose not map gender to names due to previous work criticizing that approach (see Appendix C.2). Additionally, only a portion of the generated results list names alongside NOCs, and even if names are generated it is sometimes challenging to robustly link them to the official results due to the presence of nicknames, married names, and differing transliterations.

To ensure very high accuracy when computing bias metrics, we rely on human annotation. Using methods like pattern matching or training models to label the results from generated text would make it easier to compute the three bias scores for additional LLMs, but may introduce more noise.

## Acknowledgements

Middlebury College students Finn Ellingwood, Jayda Gilyard, and Matthew Nannis made this work possible by assisting with data annotation. Catherine Finegan-Dollak, Oana Ignat, and Chet Aldrich provided invaluable feedback on the drafts of this work. This material is based upon work supported by the National Science Foundation under Grant No. 1827373.

## References

* A. Abid, M. Farooqi, and J. Zou (2021)Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AlES '21, New York, NY, USA, pp. 298-306. External Links: ISBN 978-1-4503-387X Cited by: SS1.
* A. Feyza Akpurek, M. Yusuf Kocygit, S. Paik, and D. T. Wijaya (2022)Challenges in measuring bias via open-ended language generation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Seattle, Washington, pp. 76-76. External Links: Link Cited by: SS1.
* Y. Benjamini and Y. Hochberg (1995)Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological)57 (1), pp. 289-300. External Links: Document, ISSN 0027-8424, Link Cited by: SS1.
* T. Bolukbasi, K. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai (2016)Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems29. Cited by: SS1.
* A. Brandsen, S. Verberne, M. Wanseleben, and K. Lambers (2020)Creating a dataset for named entity recognition in the archaeology domain. In Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France, pp. 4573-4577. External Links: Link Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, H. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. External Links: Link, Document Cited by: SS1.
* A. Caliskan, J. J. Bryson, and A. Narayanan (2017)Semantics derived automatically from language corpora contain human-like biases. Science356 (6334), pp. 183-186. External Links: ISSN 0027-8424, Link Cited by: SS1.
* R. Coche and C. A. Tuggle (2018)Men or women only five olympic sports matter: a quantitative analysis of nbc's prime-time coverage of the rio olympics. Electronic News12 (4), pp. 199-217. External Links: ISSN 0027-8424, Link Cited by: SS1.
* C. Cooky, L. D. Council, M. A. Mears, and M. A. Messner (2021)One and done: The long eclipse of women's televised sports, 1989-2019. Communication & Sport9 (3), pp. 347-371. External Links: ISSN 0027-8424, Link Cited by: SS1.
* V. Felkner, H. C. Chang, E. Jang, and J. May (2023)WinoQueer: a community-in-the-loop benchmark for anti-LGBTQ+ bias in large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 9126-9140. External Links: Link, Document Cited by: SS1.
* L. Fu, C. Danescu-Niculescu-Mizil, and L. Lee (2016)Tie-breaker: using language models to quantify gender bias in sports journalism. In Proceedings of the IJCAI workshop on NLP meets Journalism, Cited by: SS1.
* I. O. Gallegos, R. A. Rossi, J. Barrow, M. Tahjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, and N. K. Ahmed (2024)Bias and fairness in large language models: a survey. Computational Linguistics, pp. 1-83.

 

Vagrant Gautam, Arjun Subramonian, Anne Lauscher, and Os Keyes. 2024. Stop! in the name of flaws: Disentangling personal names and sociodemographic attributes in NLP. In _Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_, pages 323-337, Bangkok, Thailand. Association for Computational Linguistics.
* Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 609-614, Minneapolis, Minnesota. Association for Computational Linguistics.
* Hersh (2024) William Hersh. 2024. Search still matters: information retrieval in the era of generative ai. _Journal of the American Medical Informatics Association_, 31(9):2159-2161.
* Higgs and Weiller (1994) Catriona T. Higgs and Karen H. Weiller. 1994. Gender bias and the 1992 summer olympic games: An analysis of television coverage. _Journal of Sport and Social Issues_, 18(3):234-246.
* Larson (2017) Brian Larson. 2017. Gender as a variable in natural-language processing: Ethical considerations. In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_, pages 1-11, Valencia, Spain. Association for Computational Linguistics.
* Liang et al. (2021) Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In _International Conference on Machine Learning_, pages 6565-6576. PMLR.
* Lucy and Bamman (2021) Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In _Proceedings of the Third Workshop on Narrative Understanding_, pages 48-55, Virtual. Association for Computational Linguistics.
* Matteo (1986) Sherri Matteo. 1986. The effect of sex and gender-schematic processing on sport participation. _Sex Roles_, 15:417-432.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Perez (2019) Caroline Criado Perez. 2019. _Invisible women: Data bias in a world designed for men_. Abrams.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3407-3412, Hong Kong, China. Association for Computational Linguistics.
* Sorensen et al. (2024) Axel Sorensen, Siyao Peng, Barbara Plank, and Rob Van Der Goot. 2024. EEVEE: An easy annotation tool for natural language processing. In _Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)_, pages 216-221, St. Julians, Malta. Association for Computational Linguistics.
* Stanovsky et al. (2019) Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
* Steidinger (2020) Joan Steidinger. 2020. _Stand up and shout out : women's fight for equal pay, equal rights, and equal opportunities in sports_. Rowman & Littlefield, Lannar, Maryland.
* Stewart and Mihalcea (2024) Ian Stewart and Rada Mihalcea. 2024. Whose wife is it anyway? assessing bias against same-gender relationships in machine translation. In _Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_, pages 365-375, Bangkok, Thailand. Association for Computational Linguistics.
* Vogel and Jurafsky (2012) Adam Vogel and Dan Jurafsky. 2012. He said, she said: Gender in the ACL Anthology. In _Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries_, pages 33-41, Jeju Island, Korea. Association for Computational Linguistics.
* Zhao et al. (2018) Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning gender-neutral word embeddings. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4847-4853, Brussels, Belgium. Association for Computational Linguistics.

## Appendix A Computational Resources

We used a node with four NVIDIA RTX A6000 GPUs for inference. Table 3 shows the number of GPUs used and whether or not quantization was used for each model. We increased GPU counts if the program failed to run due to memory constraints; if the program failed using all four GPUs, we used 4-bit quantization. In all, approximately 40 GPU-hours were used for text generation.

We used the batch API to generate text using OpenAI models. All batches were submitted on September 14, 2024 



## Appendix B Prompt Generation Details

The prompts are created such that if the discipline and event are the same (e.g., for Water Polo), only one is included. Generally, the exact names for events from the Olympic Studies Center data are used, but in two cases, changes were made to remove ambiguity: we use "Indoor Volleyball" to distinguish "Volleyball" from "Beach Volleyball" and "Field Hockey" to distinguish "Hockey" from "Ice Hockey".

## Appendix C Annotation Details

### Annotation Interface

We use a customized version of the EEVEE annotation tool Sorensen et al. (2024), which allows for easy annotation of spans of text. It was customized to automatically load and save data from a server (rather than requiring users to upload/download files), to show newlines in text (making it more readable and reflective of the original generated text), and to have more intuitive keyboard shortcuts. For the underspecified task, the words "Men" and "Women" were highlighted to make the task more straightforward for annotators. Figure 2 shows a screenshot of the annotation interface.

In addition to labeling spans of text, annotators selected among three statuses: \(\check{}\), Ambiguity or Inconsistency in Text, or Cannot Annotate. Ambiguity or Inconsistency in Text was selected when the model's output stated that the event did not exist, gave results for a different event, or stated that results changed after the fact due to doping or other policy violations. Cannot Annotate indicated that the instance could not be annotated appropriately due to limitations in the annotation interface, because it required labeling a span with multiple labels.

### Annotating Gender

While it would complement our implicit-ambiguous metric (as the models frequently list athlete names alongside countries), we _do not_ rely on names to infer the gender of Athletes. Although ascribing genders to names based on information like census data has been a popular approach in previous work on bias, it has been criticized because it ignores people's gender identity Larson (2017), is inaccurate in some languages such as Chinese Vogel Jurafsky (2012), and introduces a number of other concerns around validity and ethics Gautam et al. (2024). We focus on the gender associated with team events rather than individual athletes.

### Annotator Recruitment and Pay

Three undergraduate students at Middlebury College were recruited to annotate the data. They each annotated \(\frac{2}{3}\) of the full dataset (each did not annotate one family of models). This meant that if there was a disagreement between a pair of annotators, they could work together to resolve it. They began with a "training task" that introduced them to some fairly standard instances and some that were more complex to annotate (similar to those in Table 5). After successful completion of the training task, the data was distributed to annotators in small batches which were intended to take approximately 15 minutes to annotate.

The students were paid $14.08 per hour in accordance with the college-wide policy for student workers.

### Inter-Annotator Agreement

Following prior work on named entity recognition (NER) Brandsen et al. (2020), we consider multiple metrics for computing inter-annotator agreement. These include Cohen's \(\kappa\) for both all tokens and only those that at least one annotator gives a label to other than O. We also compute pairwise F1 score for all labeled spans; spans are considered equivalent if the text and the label match.

We present the agreement scores for in Table 4, and find that overall agreement is very high. Agreement is generally lower for the underspecified task; that is likely because (a) it was the first task completed by the annotators, who were familiarizing themselves with the process and (b) there are more labels. Many disagreements stemmed from human error, e.g., labeling the medal color instead of the country or labeling an extra punctuation token. An additional source of disagreement stemmed from politics associated with NOCs, e.g., ensuring that "the Former Soviet Union" was labeled as "EUN"

\begin{table}
\begin{tabular}{|l|c|c|} \hline Model & GPUs & Used Quantization? \\ \hline llama3.1-8b & 1 & no \\ \hline llama3.1-70b & 2 & yes \\ \hline mistral-nemo & 3 & no \\ \hline mistral-large & 4 & yes \\ \hline \end{tabular}
\end{table}
Table 3: Computational Resources used for text generation.



(Unified Team) in 1992 or that "Russian Olympic Committee" (ROC) was labeled in 2020 to match the official results.

It should be noted that these metrics for NER are somewhat strict for this task, as the ultimate goal is to map the annotated spans to NOCs. In some cases, a NOC is mentioned multiple times in the text and annotators might annotate different spans referring to the same NOC (e.g., in the text "1. United States of America (USA)"). If one annotator labeled "United States of America" while the other labeled "USA," it would be considered a disagreement, but downstream scripts would map these spans of text to the same label.

### Resolving Disagreement and Quality Checks

Annotations meeting either of the following two criteria were flagged for re-annotation:

1. The two annotators disagreed, either on the spans that they annotated or whether there was ambiguity in the results.
2. The gender(s) labeled by the annotator were inconsistent with patterns in the text: 1. The word "Men" or "Women" was in the text generated using an underspecified prompt, but no medals were labeled for the corresponding gender. 2. The word "Men" or "Women" was not in the text generated using an underspecified prompt, but medals were labeled for the corresponding gender.

The two annotators who had originally labeled each instance worked together in-person to re-annotate any flagged annotations. An author was available to answer questions as necessary.

#### c.5.1 Limitations of the Annotation Interface

A small number of instances were labeled Cannot Annotate and were manually reviewed. In these cases (less than 1%), the correct data was manually added to the final result file.

#### c.5.2 Mapping Annotations to NOC Codes

Each country/nationality span was mapped to a NOC code using a lookup table based on [https://github.com/datasets/country-codes/blob/master/data/country-codes.csv](https://github.com/datasets/country-codes/blob/master/data/country-codes.csv). After disagreements were resolved, the data was fairly clean and if a country/nationality could not be mapped to a NOC code, it was added to the

\begin{table}
\begin{tabular}{|c|l|c|c|c|} \hline  & Annotators & \(\kappa\) (all) & \(\kappa\) (annotated) & F1 \\ \hline \multirow{4}{*}{Counts} & A1/A2 & 0.99 & 0.96 & 0.99 \\ \cline{2-5}  & A1/A3 & 0.99 & 0.96 & 0.98 \\ \cline{2-5}  & A2/A3 & 0.98 & 0.95 & 0.98 \\ \cline{2-5}  & mean & 0.99 & 0.96 & 0.98 \\ \cline{2-5}  & A1/A2 & 0.98 & 0.95 & 0.97 \\ \cline{2-5}  & A1/A3 & 0.96 & 0.91 & 0.95 \\ \cline{2-5}  & A2/A3 & 0.94 & 0.87 & 0.92 \\ \cline{2-5}  & mean & 0.96 & 0.91 & 0.95 \\ \hline \end{tabular}
\end{table}
Table 4: Inter-annotator agreement metrics for each task, including agreement between individual pairs of annotators and the mean of pairwise agreement.

Figure 2: An example annotation for the specified task.

 

lookup table as it unambiguously referenced a NOC (e.g., "German" was not in the original table but maps to "GER"). In one case, the text simply stated "Korean", which could not unambiguously be mapped to either North or South Korea; the annotated span was removed due to the ambiguity. Additionally, a small number of annotations were fixed as they did not properly map to the NOC competing in the games, which was also mentioned in the text (e.g., in 1988, both annotators labeled "Germany" rather than the full span "West Germany").

### Downstream Effects of Ambiguity

A number of instances were marked with the status Ambiguity or Inconsistency in Text. A sample of those instances are shown in Table 5, along with the suggested action for annotators to take.

To determine the effect that these ambiguous results had on our final conclusions, we show the final results with those ambiguous events removed in Table 6. All of the main findings are robust to the removal of these events.

## Appendix D Statistical Tests

We test for statistical significance using permutation tests for the knowledge-based and implicit-ambiguous metrics; we run 10,000 permutations where gender is randomly assigned to \(F_{1}\) scores. As the explicit-ambiguous metric is based on counts rather than continuous scores, we use a binomial test where our null hypothesis is that when either male or female results are enumerated, they are female 50% of the time.13 We use the adjustment for false discovery rate (FDR) Benjamini and Hochberg (1995) with \(\alpha=0.05\), to account for multiple comparisons.

 



\begin{table}
\begin{tabular}{|l|c|r r|r r|} \hline Model & Avg F1 & knowledge-based & explicit-ambiguous & implicit-ambiguous \\ \hline gpt-4o-mini & 0.63 & 0.00 & 0.22 & 31\% & 0.03 \\ gpt-4o & 0.95 & -0.01 & 86\% & **0.13** & 14\% & **0.28** \\ llama3.1-8b & 0.59 & -0.04 & 38\% & 0.09 & 54\% & **0.12** \\ llama3.1-70b & 0.86 & -0.02 & 44\% & 0.04 & 53\% & **0.30** \\ mistral-nemo & 0.77 & -0.02 & 36\% & **0.15** & 63\% & **0.15** \\ mistral-large & 0.97 & 0.00 & 79\% & **0.09** & 21\% & **0.27** \\ \hline \end{tabular}
\end{table}
Table 6: Results of our analysis when ambiguous results are removed from consideration. Results significant at the level \(\alpha=0.05\) are demarcated in **bold**. The false discovery rate (FDR) correction is performed for all p-values computed for the table with a FDR of 0.05. Small grey percentages indicate the percentage of instances where gender was explicit vs. implicit; these do not add to 100 as in some instances, the model’s output does not include any results.

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline Text & Annotators Should \\ \hline Who won the medals in the Men’s Indoor Volleyball event at the 1988 Olympic games? & Annotate spans of text as usual if the model gives conflicting information as to whether or not the event occurred. \\ \hline  & & \\ Who won the medals in the Men’s Indoor Volleyball event at the 2012 Olympic Games. & Not annotate any spans of text if the model gives results that explicitly do not match the event in the prompt. \\ \hline  & & \\ Who won the medals in the Men’s Indoor Volleyball event at the 2012 Olympic Games. & & Not annotate any spans of text if the model gives results that explicitly do not match the event in the prompt. \\ \hline  & & \\ Who won the medals in the Men’s Indoor Volleyball event at the 2012 Olympics, but it may played as a Beach Volleyball event, not Indoor Volleyball. The gold medal was won by Brazil, the silver medal was won by Russia, and the bronze medal was won by Italy. \\ If you are looking for the Indoor Volleyball event, it was actually held at the 2012 FIVB Volleyball Men’s World Championship, which took place in August 2010 in Italy. & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 olympic games? At the 2000 Olympic Games in Sydney, the Women’s Athletics 4 x 100 metres Relay event was won by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 olympic games? At the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was won by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 olympic games? At the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was won by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 olympic games? At the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was won by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 olympic games? At the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was won by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas. The team consisted of Savatheda Fynes, Chandra Sturrup, Pauline Davis-Thompson, and Debbie Ferguson. They won the gold medal with a time of 41.95 seconds. & Do their best to annotate the _final_ result in cases where the model stated that the results changed after the event. \\ \hline  & & \\ Who won the medals in the Women’s Athletics 4 x 100 metres Relay event at the 2000 Olympic Games in Sydney, the Women's Athletics 4 x 100 metres Relay event was born by the Bahamas.