Run Like a Girl! Sports-Related Gender Bias in Language and Vision
Year: 2023

Abstract:
Gender bias in Language and Vision datasets and models has the potential to perpetuate harmful stereotypes and discrimination. We analyze gender bias in two Language and Vision datasets. Consistent with prior work, we find that both datasets underrepresent women, which promotes their invisibilization. Moreover, we hypothesize and find that a bias affects human naming choices for people playing sports: speakers produce names indicating the sport (e.g. ‘tennis player’ or ‘surfer’) more often when it is a man or a boy participating in the sport than when it is a woman or a girl, with an average of 46% vs. 35% of sports-related names for each gender. A computational model trained on these naming data reproduces the bias. We argue that both the data and the model result in representational harm against women.

heading: 1 Introduction
text: Existing social biases and stereotypes against certain groups, such as women and racial minorities, are known to be reproduced by computational models (Caliskan et al., 2017; Bolukbasi et al., 2016; Hovy and Søgaard, 2015; Wang et al., 2021; Blodgett et al., 2020). This is primarily due to the fact that the datasets that the models are trained on are biased themselves, because, unless explicit steps are taken, datasets tend to mirror social biases (Torralba and Efros, 2011; Rudinger et al., 2017, 2018). Moreover, models often amplify biases, because they overrely on shallow patterns and lean towards majority labels (Ahmed et al., 2022; Deery and Bailey, 2022; Zhao et al., 2017).
Bias in AI has ethical implications because it can result in harm for the affected groups: both representational harm, with systems demeaning or ignoring them, and allocational harm, with systems allocating fewer resources or opportunities to them (Mitchell et al., 2021; Blodgett et al., 2020; Mehrabi et al., 2022b). For instance, the fact that
the multimodal model VL-BERT (Su et al., 2019) often predicts that a woman carrying a briefcase is carrying a purse (Srinivasan and Bisk, 2022) constitutes representational harm, with working women not being recognized as such.
(a) woman (16), surfer (14) (b) surfer (24), man (6), person (2), boy (2)
(c) girl (11), skateboarder (7), skater (6), child (6) (d) skater (9), skateboarder (6), boy (6), kid (3)
Figure 1: Images of people playing sports from the ManyNames dataset, together with the names that human annotators produced and their counts.
In this paper, we focus on gender bias that causes representational harm for women, specifically for women in sports, in the area of Language and Vision (L&V). Previous work on gender bias in L&V has shown that bias often relates to the language component, causing models to override the specific visual information in classification decisions (Zhao et al., 2017; Goyal et al., 2017; Ramakrishnan et al., 2018) and vice versa (Hendricks et al., 2018; Bhargava and Forsyth, 2019), such as in the ‘purse/briefcase’ example above. We examine bias
14093
that is present in the language, without examining its interaction with the visual component in the model.
Women in Western societies have traditionally been marginalized, excluded, and deterred from participating in sports or even physical activity, while men have been encouraged (Bell et al., 2008; Scheadler and Wagstaff, 2018; Vertinsky, 1994; Schaillée et al., 2021). Sport has long been stereotypically associated with masculinity, and females have been thought physically incapable of performing well in this area (Young, 1980). Participation in sports by women and girls has continued to increase since the 1960s in parallel with other social advances, however, rarely do women rise to managerial or coaching roles, even in women’s teams (Schaillée et al., 2021). Women’s sports are also underrepresented in the media, which has been linked to perpetuating the stereotype of the male athlete over the female (Schmidt, 2013).
We analyze the data in a Language and Vision dataset for object naming, ManyNames (Silberer et al., 2020a,b; more information in the next section). Figure 1 shows two example images in ManyNames together with the names elicited from subjects. How we name an object or entity is intimately linked to how we conceptualize it (Brown, 1958; LaTourrette and Waxman, 2020). We hypothesize that due to the social constraints discussed above, speakers produce a sports-related name such as ‘surfer’ less often when their referent is a woman, that is, they do not conceptualize female athletes as athletes. If the bias indeed exists in the speaker population, we expect it to be present in the naming data and propagate to computational models; we check both expectations. Before that, we examine whether there is an overall representational bias in ManyNames and its parent dataset, VisualGenome (Krishna et al., 2016), with women being underrepresented.

heading: 2 Underrepresentation of Women
text: Data and method ManyNames contains names for 25K images produced by English-speaking subjects in a free naming task, with an average of 31 names per image. ManyNames images are a subset of those in VisualGenome. The images were selected from seven previously defined domains, one of which was PEOPLE, based on a series of seed WordNet synsets. The authors put a cap on the number of images for a given synset (max. 500
instances for seeds with up to 800 objects in VisualGenome and up to 1k instances for seeds with more than 800 objects).
VisualGenome contains 108K images that are the intersection of images in the datasets YFCC100m (Thomee et al., 2016) and MS-COCO (Lin et al., 2015). The objects in each VisualGenome image were manually identified, labeled, and linked to their corresponding WordNet synset (VisualGenome provides many more annotations, but these are the ones of interest for the present study). Both VisualGenome and ManyNames employed crowd-sourced workers from Amazon Mechanical Turk (AMT) as annotators 1 for the images, with workers coming predominantly from the USA. Note that while the images in ManyNames are a subset of those in VisualGenome, the naming annotations were collected afresh for ManyNames.
The YFCC100m images were all those uploaded to Flickr between 2004 and 2014, published under a commercial or noncommercial license. MS-COCO contains all images from Flickr available at the time of the dataset construction that belonged to 91 predefined image categories (e.g., horse, people, and laptop). Images on Flickr are uploaded by Flickr users.
To check for representational bias, we extracted all objects in VisualGenome (image areas within a bounding box) with labels corresponding to the four most common gender-associated names: ‘boy’, ‘girl’, ‘man’, and ‘woman’. For ManyNames, we used the same names and the full naming distribution.
Results The resulting gender distribution is shown in Table 1, together with the distribution in the world in 2020, as reported by the United Nations world population data (United Nations - Department of Economic and Social Affairs, 2022). Both datasets indeed underrepresent females, according to one-tailed z-tests comparing the percentage of females in each dataset with the global
1In VisualGenome, annotators identified (via bounding boxes) the objects in each image and provided descriptions such as ’a red mushroom with white spots’. The head nouns in these region descriptions were identified and matched with WordNet synsets in a semiautomatic fashion. In ManyNames, annotators were shown images where an object was highlighted with a bounding box, and they were asked to provide a single name for the object. Thirty-six different subjects provided names for each object, and the naming responses were cleaned and aggregated. We use the object synsets for the analysis in Section 2, and the naming distributions for the rest of the paper.
percentage of females in the world (VisualGenome: z = -129.8, p < 0.001; ManyNames: z = -2.4, p = 0.02).2 Note, however, that the bias against women is much larger in VisualGenome, with only 32.7% female entities compared with 49.6% in the world’s population; in ManyNames, the percentage is only 2 points below the world population (47.6% vs 49.6%). Moreover, note that the bias in ManyNames stems from images of boys, with 14.4% of images vs 7.9% for girls. Recall from above that no specific action was taken in either ManyNames or VisualGenome regarding gender balance. The representational bias in ManyNames is smaller due to the cap on the synsets, aimed at obtaining a varied set of categories in general; the reduced gender bias is a side effect. Below, we discuss a further specific type of underrepresentation also found in ManyNames, that of women playing sports.

heading: 3 Sports-Related Bias
text: Next, we present our main analysis, namely gender bias related to sports as shown in human naming data. A secondary analysis concerns model behavior.

heading: Sports-Related Bias in Humans: Methods
text: The first author of this paper went through all topnames—that is, the name produced by the majority of the annotators for each image—in the ManyNames domain PEOPLE and selected those that related directly to gender (‘boy’, ‘girl’, ‘man’, and ‘woman’) or sport (‘athlete’, ‘baseball player’, ‘basketball player’, ‘batter’, ‘catcher’, ‘goalie’, ‘pitcher’, ‘player’, ‘skateboarder’, ‘skater’, ‘skier’, ‘soccer player’, ‘snowboarder’, ‘surfer’, ‘tennis player’, and ‘umpire’). We refer to the former as ’taxonomic’ and the latter as ’sports-related’. We
2All statistical tests in this paper assume an alpha level of 0.05.
selected all 1,776 images that have at least one taxonomic and one sports-related name in the responses, such that we could automatically determine both the gender of the person and the fact that they are playing a sport.3
To check for bias, we computed, for each image, the percentage of sports-related names associated with it relative to the total names, which include both taxonomic and sports-related names. For instance, in Figure 1, the person in panel (a) received 46.7% of sports-related names, while the person in panel (b) received 70.5%. We fitted a logistic regression model with the proportion of sports-related names as the outcome variable and fixed effects for the person’s gender.
Sports-Related Bias in Humans: Results Table 2 summarizes the results of the logistic regression, supporting our hypothesis: when annotators see images of men playing sports, they are more likely to produce a name that explicitly mentions the sport being played and therefore are less likely to produce a taxonomic name compared with when they see images of women playing sports. Figure 2 qualitatively shows the difference between the genders. Note that there are also fewer images of women playing sports (527 vs. 1219), constituting only 30.2% of the pictures of people playing sports. These numbers compared with the realworld sports statistics constitute a further, more insidious instance of underrepresentation of women in L&V datasets: women in certain roles. This constitutes representational bias regarding women playing sports in ManyNames. Note that there are in general fewer women playing most sports in the Western population—especially due to dropouts (Bevan et al., 2021). However, the difference between the genders is not as large as in the dataset. For instance, in the US, the percentage of girls in college sports in 2019 was 43.9% vs 56.1% boys (NCAA, 2022), and in England, the percentage of women among the adults who participated in sporting activities in 2020 was 45% vs 55% men (Sport England, 2023).
3Images with inconsistent gender in the response (e.g., where some subjects produced ‘man’ and others ‘woman’) were discarded.
Sports-Related Bias in an L&V model We additionally analyze the behavior of a model trained to produce names on the ManyNames dataset (Lang, 2021). Based on the extensive literature on bias (Mehrabi et al., 2022a), we expect the model to reproduce the biases observed in the data; in contrast, it is unclear whether it will amplify it, as models vary with respect to this (Zhao et al., 2017; Fernando et al., 2021).
This model builds upon a ResNet101 architecture (He et al., 2015) pretrained on VisualGenome (Anderson et al., 2018) and is adapted to ManyNames names through an additional fine-tuning step. Relevantly to our purposes, the model is trained to reproduce the full naming distribution, outputting a probability distribution over all the names in the vocabulary. This is different from object classification in Computer Vision, which assigns a single class to each object (Deng et al., 2009; Ren et al., 2015; He et al., 2015). Lang (2021) used the train–dev–test partition of ManyNames established in Silberer et al. (2020b); we
analyze the behavior of the model in the test set and, in particular, in the images that meet the criteria used in our second analysis above (N = 89, of which 34 female).
We normalize the probability weights output by the model, using only the names of interest (that is, discarding any weight the model places on names that are neither taxonomic nor sports-related before doing the normalization). To check whether the model reproduces the bias, we fit a logistic regression model, with the proportion of sports-related names as the dependent variable and fixed effects for the image gender. To check whether it amplifies the bias, we fit a mixed-effects logistic regression model that takes into account both the model and the human data (see Appendix for details). 4
As shown in Table 3, according to the regression analysis, the L&V model indeed reproduces the naming bias found in the human data: for images depicting boys and men playing sports, it assigns significantly higher weights to sports-related names than for images depicting women or girls playing sports. We instead find no evidence of bias amplification (see Appendix); note, however, that the sample is small.

heading: 4 Discussion and Conclusions
text: We have identified pervasive biases against women in Language & Vision. Our main contribution is the individuation of a bias that characterizes human naming choices and therefore the naming data available for models. While we have focused on ManyNames, this issue is likely to affect other
4Models were fit in R using glm and glmer (Bates et al., 2015; R Core Team, 2021).
datasets containing names, such as widely used datasets for captioning (Young et al., 2014; Sharma et al., 2018; Lin et al., 2015) and referring expression generation (Kazemzadeh et al., 2014; Yu et al., 2016). The bias concerns the kind of name chosen for athletes, depending on the genre: people produce fewer sports-related names for females playing sports (average 35%) than for males playing sports (46%). As far as we know, this kind of bias has not been previously discussed, and it is, we argue, more implicit and thus difficult to identify than other kinds of bias that are more commonly discussed in the literature, such as unbalanced classes in datasets due to the underrepresentation of certain groups (which we also found and discuss below). We find the naming bias both in the human data of ManyNames and a model trained on these data. Thus, even women who do play sports (which are fewer to begin with, as mentioned in Section 3) are not conceptualized as such. This constitutes representational harm and contributes to limiting choices for women.
We also find that women are underrepresented in L&V, pronounced especially in VisualGenome, which has over twice as many images of males than females. Moreover, the proportion of males and females playing sports in ManyNames is skewed, with only 30.2% of the pictures of people playing sports depicting females. As mentioned above, the actual percentage of women and girls in sports in Anglo-Saxon societies is closer to 45%, according to recent data (NCAA, 2022; Sport England, 2023). The underrepresentation of social groups is harmful itself (Blodgett et al., 2020), as well as because models trained on unbalanced data can neglect crucial patterns relevant to those within the group (Wang et al., 2022). Given how the images were selected (see Section 2), the origin of the underrepresentation of women in these datasets must come from the kinds of images uploaded onto Flickr around the 2010s. This is in turn is likely rooted in the demographic characteristics of Flickr users during that period and to the fact that, in general, the internet has historically been heavily male-dominated (Morahan-Martin, 1998; O’Hare and Murdock, 2012). This is a serious concern, as most resources used in L&V (as well as computational linguistics and AI in general) come from the internet.
Our findings are in line with other research on gender bias, both in L&V and in NLP and AI more
broadly, as discussed in the introduction. They also resonate with a study by the Pew Research Center (Lam et al., 2018) showing that results in Google Image Search underrepresent women in various jobs compared with their actual participation in those jobs in the USA according to the Bureau of Labor Statistics.
Ultimately, based on the findings of this paper, it can be concluded that as far as the datasets and model analyzed are concerned, when it comes to sports: A man with a tennis racket is a tennis player. A woman with a tennis racket is just a woman with a tennis racket.

heading: Limitations and Future Work
text: Our findings about gender bias in the field of Language & Vision are based on two datasets, one task (Object Naming), one language (English), a mostly Western population (based on the origin of both the images and the annotators of VisualGenome and ManyNames), and one computational model. Moreover, in the third analysis, due to the characteristics of the test set of ManyNames, the sample size was small. Additionally, the bias around naming choices concerns the domain of sports only.
Regarding our most novel finding (bias in lexical choice), given the basic function of naming in language and the fact that Western Englishspeaking societies are not known to be more genderbiased than most non-Western and/or non-Englishspeaking societies, it is plausible that the identified bias extends to other L&V tasks such as image captioning, referring expression generation, or Visual Question Answering. Furthermore, given previous work on bias in our field, it is plausible that the identified bias in the model extends to other models. It is, however, not clear whether the bias will be amplified or simply reproduced. To probe whether, and to what extent, the identified biases indeed generalize, future work should tackle more tasks, languages, populations, domains, models, and data. Testing further models on the same naming data that we used is straightforward; checking for biases in some other tasks for English should be feasible at least to some extent, since some datasets provide multiple annotations per image (e.g., captions in MS-COCO). Instead, analyzing other languages and populations, such as nonbinary individuals, will in most cases require further data collection due to the scarcity of non-WEIRD data in our field.
In this study, gender was operationalized in a bi-
nary manner in order to most effectively investigate the stated hypothesis. Furthermore, there is a lack of nonbinary labels within the datasets used (5% of labels can be considered gender-neutral, i.e., "person, human, child"), and the resources required to reflect the reality of the gender landscape currently do not exist. This indicates a separate but related issue regarding a lack of representation of nonbinary individuals within vision datasets and how to conduct ethically inclusive studies on gender (Larson, 2017). However, addressing this is beyond the scope of the present research and remains an important direction for future work. Finally, this work solely concerns the identification of biases; further work should focus on how to deal with them in terms of data collection, curation, and modeling.

heading: Ethics Statement
text: This research aims to highlight the extent to which gender biases may be present in Language & Vision, with an emphasis on the representation of females in sports. The findings of the study may have implications for the ways in which images of female athletes and sports figures are portrayed and treated within the datasets and by models. The results of the study may have the potential to contribute to the ongoing efforts to address gender bias in all areas of life, as we maintain that gender bias is an issue within Machine Learning, because it is an issue within human society. The research team is committed to using the findings of the study to foster dialogue and understanding of the issue and to advocate for equitable treatment of all groups throughout the production pipeline.

heading: Acknowledgments
text: The authors thank the reviewers for their useful feedback. This research is an output of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain) and of grant agreement No. 715154 funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme. This paper reflects the authors’ view only, and the funding agencies are not responsible for any use that may be made of the information it contains.

heading: A Bias Amplification
text: To check whether the model amplifies the bias, we considered, for the same 89 images, the probabilities assigned to taxonomic and sports-related names by the human annotators and the model and fitted a mixed-effects logistic regression model, with the proportion of sports-related names as the outcome variable and fixed effects for the image gender, the prediction type (either human or model), and the interaction between image gender and prediction type. We set a random intercept for each image and a random slope for each type. We coded the prediction type as ’treatment’ so that the human prediction would be our baseline.
The results are summarized in Table 4. The regression shows that for the 89 images included in this analysis, humans use more sports-related names when the image gender is male—in line with our previous findings. However, the results concerning a possible bias amplification are inconclusive: the estimates for the prediction type and for the interaction between type and gender are not significant. Analyses of larger data samples may be needed to shed further light on this topic.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
7 A2. Did you discuss any potential risks of your work? We do not think this is relevant to our submission.
7 A3. Do the abstract and introduction summarize the paper’s main claims? The abstract does, the introduction no because it’s a short paper and we distribute the claims between the introduction and the conclusion.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: All sections.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Introduction.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We will add this to the final version, if accepted.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? It follows from the content of the paper.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We will add this to the final version, if accepted.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Introduction, sections 3 and 5, limitations and future work.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 7 Did you run computational experiments?

heading: Left blank.
text: C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: ? Introduction, sections 3 and 5, limitations and future work. 3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank. C 7 Did you run computational experiments?