Transcending the “Male Code”: Implicit Masculine Biases in NLP
Contexts
Katie Seaborn

Tokyo Institute of Technology
seaborn.k.aa@m.titech.ac.jp

Shruti Chandra

University of Waterloo
shruti.chandra@uwaterloo.ca

ABSTRACT
Critical scholarship has elevated the problem of gender bias indata sets used to train virtual assistants (VAs). Most work has focused on explicit biases in language, especially against women,
girls, femme-identifying people, and genderqueer folk; implicit associations through word embeddings; and limited models of genderand masculinities, especially toxic masculinities, conflation of sex
and gender, and a sex/gender binary framing of the masculine asdiametric to the feminine. Yet, we must also interrogate how masculinities are “coded” into language and the assumption of “male”
as the linguistic default: implicit masculine biases. To this end, we
examined two natural language processing (NLP) data sets. We
found that when gendered language was present, so were genderbiases and especially masculine biases. Moreover, these biases related in nuanced ways to the NLP context. We offer a new dictionary
called AVA that covers ambiguous associations between genderedlanguage and the language of VAs.

CCS CONCEPTS• Human-centered computing; • Human computer interaction (HCI); • Social and professional topics; • User Characteristics; • Gender; • Computing methodologies; • Artificial
Intelligence; • Natural Language Processing; • Lexical semantics;

KEYWORDS
Machine learning bias, Gender bias, Masculinities, Implicit genderedlanguage, Data sets, Natural language processing, Feminist HCIACM Reference Format:
Katie Seaborn, Shruti Chandra, and Thibault Fabre. 2023. Transcending the“Male Code”: Implicit Masculine Biases in NLP Contexts. In Proceedings ofthe 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23),
April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 19 pages.
https://doi.org/10.1145/3544548.3581017

1

INTRODUCTION

Machines can now speak with us. Advances in machine learning(ML) and natural language processing (NLP) have paved the way
for more natural modes of communication with computers. Virtualassistants (VAs), conversational user interfaces (CUIs), and voice

This work is licensed under a Creative Commons Attribution-Share Alike
International 4.0 License.
CHI ’23, April 23–28, 2023, Hamburg, Germany© 2023 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9421-5/23/04.
https://doi.org/10.1145/3544548.3581017

Thibault Fabre

The University of Tokyo
fabre@g.ecc.u-tokyo.ac.jp

user interfaces (VUIs) in the form of smart speakers embedded inhomes, spaces, and cities, can be found alongside social robots thatgreet us at the storefront. Amazon Alexa, Apple’s Siri, Microsoft’s
Cortana, and LINE’s Clova are commercial examples being taken up
at a global scale. While speaking with and listening to computers has
a long history within the field of human-computer interaction (HCI),
these recent industrial and commercial trends have reinvigorated
interest in the voice of the machine [28, 95]. In SIGCHI and adjacentspaces, this can be seen in recent workshops (e.g., [15, 79, 99, 104])and conferences (notably CUI1 and ACL2 ).
Voice, speech, command-based, and conversational interactions
are fundamentally about language. Most systems rely on the NLP
of dictionaries and dialogue models trained on human (or humangenerated) data sets. Many have harnessed the power of the crowd,
notably Amazon Mechanical Turk (AMT), to create these data
sets. The merits and demerits of people-powered data sets havebeen well-documented [21, 23, 26, 47, 64]. On the one hand, NLPbased systems trained on human language texts can lead to morenatural, fluent language capabilities in machines. On the other,
they can include biased and harmful content, on purpose or not[1, 17, 23, 27, 35, 47, 102]. Critical work has highlighted explicit
and implicit forms of bias in algorithms and data sets for gender[17, 23, 27, 101, 102], race [16, 77, 81, 92], age [35], and their intersections [49, 77]. Most work has focused on gender stereotypes,
harassment, and abusive language, and otherwise limited and/or
negative associations with femininities and genderqueer people,
notably girls, women, femme, trans femme, and non-binary femme
people [23, 101, 102]. Critical scholarship has also escalated a needto distinguish sex from gender [14, 20, 34, 71] and consider genders
and sexes beyond the binary [34, 63, 98]; refer also to emerging
work3 . Since language is a medium of power [40], this area is anecessary social good; its impact cannot be understated.
Nevertheless, there appears to be a gap when it comes to how
the “gender problem” is framed. Most work can be organized under three broad categories: (i) detecting and rectifying stereotypes
as limited and/or negative representations of gender within data
sets used to train ML algorithms, where the focus has tended to
be on women and femininities [23, 102]; (ii) tracing the spreadof toxicity, harassment, and abusive language, especially related
to misogyny, within digital “manospheres” [42, 85]; and (iii) debiasing ML algorithms and data sets using methods that tend tooperationalize gender bias as the relative distance between terms
associated with masculinity and its assumed diametric, femininity, i.e., word embeddings [17, 23, 102]. Underlying much of thiswork is a sex/gender binary model of man/woman, male/female,
and/or masculine/feminine [34, 62]. For example, the famous word
1 https://www.conversationaluserinterfaces.org
2 https://aclanthology.org/venues/acl
3 https://arxiv.org/abs/2202.11923

CHI ’23, April 23–28, 2023, Hamburg, Germany

embedding of “man is to computer programmer as woman is tohomemaker” [17] demonstrates how “masculinity” is implicitly singular and always positioned against its “opposite,” femininity, on abipolar scale [30]. Yet, addressing gender bias also means taking on
a broader view of gender, especially of masculinities as a covertly
dominant gender [66], as well as covering a broader range of subtle
associations that arise in language use [2, 45]. Tackling discrimination, toxicity, and stereotypes is crucial, but so is scrutinizing themore subtle varieties of gender bias that crop up in word choices
and gendered assumptions.
As a first step, we targeted an underexplored framing of genderbias for NLP-based VA development: whether and to what extent
implicit forms of masculine-centric language exist, i.e., implicit
masculine biases. Specifically, we aimed at societal tendencies tocentre masculinity, consciously or otherwise, in gendered language,i.e., word choices attributed to specific genders, and masculine-asnorm language, i.e., the overrepresentation of masculine references.
We asked: Do implicit masculine biases in terms of gendered language and masculine-as-norm language, specifically pronoun use,
gender-marked words, and names, exist in the NLP context of data
sets created to train VAs and to what extent? To answer this question,
we examined two large-scale and freely available VA-oriented data
sets—MASSIVE4 and ReDial5 —using established dictionaries with
automated content analysis and manual thematic analysis. Ourcontributions are:
• A new framing of gender bias for NLP that targets implicit masculine biases in novel ways, including masculine
language, masculine-as-norm language, and masculine-bydefault patterns,• Evidence of gendered language more generally but especially implicit masculine biases in two of the most rigorously
developed open NLP data sets available, and• AVA, a new dictionary that reveals the nuances in and ambiguity of gendered language within VA contexts.
The significance of this work is in framing gender bias in VAoriented NLP data sets differently, if a bit ironically: centringmasculinities and masculine language to interrogate masculinecentrism through two novel lenses on implicit bias, namely gendered language and masculine-as-norm language. We also offer aconcerted, if limited, effort to disambiguate sex and gender andmove beyond the sex/gender binary. A broader view of masculine
biases and gender can reveal overrepresentation and inequities inthese data sets. As we found, it also presents opportunities for creating new material on bias detection and debiasing in VA and NLP
contexts, i.e., AVA. This view is a necessary extension to the workon gender bias and debiasing linguistic data sets for computers thatcan understand and speak with us.

2 RELATED WORK
2.1 Gender Biases and Debiasing in NLP
Human language is biased, and language-based data sets are no
exception. A growing and critical body of work in NLP has started
to recognize and take action on gender and other forms of bias
4 https://github.com/alexa/massive

5 https://redialdata.github.io/website

Katie Seaborn et al.

[23, 27, 102]. Initiatives include: locating and analyzing the extent ofbiases in NLP data sets [22, 31, 49]; developing methods by which todebias these data sets and the algorithms that use them [1, 17, 106];and crafting alternatives, which can mean bringing in marginalized
genders and disrupting ideas about gender norms [101]. Yet, there
are notable gaps and even biases in this body of work. Cao and
Daumé [20] raised awareness of the distinction between social
and linguistic genders, warning those of us working in NLP, if notespecially on English data sets, to avoid drawing conclusions ofsubject gender based on gendered terms. Others have noted a lackof intersectionality [19, 49] and gaps in representation of genders
beyond the binary [1, 102]. We wish to add masculinities to this list.
We are in no way criticizing these trajectories, which are essential
for achieving the elimination of inequalities in NLP and HCI. In fact,
we would argue that gender bias in NLP is becoming a core area of
feminist HCI [4, 5, 9, 25], given that VAs continue to be taken up asa focus of study and by end-users [28, 95]. Nevertheless, fighting
sexism and toxicity will require considering gender and language
in a different way [84].
When it comes to developing NLP data sets, there is a tension
between the goals of realism in dialogues and avoiding genderbias. Many rely on curation and crowdsourcing, especially Amazon Mechanical Turk (AMT) to gather, annotate, and judge data
sets. This can result in natural and large data sets, but at the riskof perpetrating gender biases [61]. For example, people tend toassume, when no gender information is available, that undescribedpeople are men, from characters in books to voters to celebrities[2]. Moreover, women, when primed to think about gender, may
be more likely than men to refer to gender in subsequent written
work [91]. Experts in NLP have urged caution and advocated fora nuanced approach to evaluating bias and inequities, one thatinvolves recognizing who the creators of the data were and who
curated the data, as well as what or who the data is about [12, 78].
At the least, we can map out the extent of these biases in data sets
and then make a conscious decision to act or not, and how. This is
crucial for open data sets that are taken up widescale and used totrain the voices of everyday machines.
The notion of “scale” is also shifting. Those working in NLP
and adjacent spaces are recognizing how WEIRD (sampled from
Western, educated, industrial, rich, democratic nations) most data
sets are [23, 57, 74, 111]. NLP data sets are often in English andoriented to the US context. Efforts on translation and localizationof NLP data sets have emerged, such as with the MASSIVE data
set. As Bardzell outlined years ago for feminist HCI praxis [4], wemust consider how these data sets and their biases exist in andinfluence a larger ecology of systems and interactions. Yet, it is
not clear whether and how crowdsourced efforts on NLP data setstake gender bias into account. For instance, the preprint about the
MASSIVE data set does not discuss bias at all6 . Without recognition,
the danger of (re)encoding biases or introducing new ones is great.
At present, the bulk of the work has framed gender bias asstereotyped gender associations embedded in core NLP features,
especially word embeddings based on gendered occupations andsocial roles [17, 49, 102], and toxic masculinity, characterized byhate speech, aggression, and misogynistic conduct [42, 85, 102].
6 https://arxiv.org/abs/2204.08582

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

Field and Tsvetkov [45] offer one of the few NLP efforts to tackle amore subtle form: implicit bias [52] as predicted by unsupervised
ML based on comment addressee, i.e., how likely the addressee
is to be a woman. Another issue is the focus on women being
stereotyped and the ideal solution being to “flip the script,” i.e.,
using feminine referents where there would be masculine ones. Indeed, “gender-swapping” is a common approach to debiasing [102].
While some have recognized that this excludes non-binary genders[102], it also prescribes a singular view of masculinity, positions
masculinity as diametric to femininity, and erases the notion ofgender fluidity [20] and multiplicity [63]. Finally, when it comes to
toxicity, changing content must be viewed as a sensitive process.
Removing expressions of hate may lead to an agent that is nottrained to understand and respond in prosocial ways, such as toenlighten naïve interlocutors, like children, or “fight back” against
abuse [13, 44, 100]. Toxic language is also contextual [1]. For example, queer communities may “take back” and use toxic languagethat is parsed by well-meaning but heteronormative algorithms
as toxic [51]. Training VAs involves not only giving these agentsa voice, but also allowing them to understand us. As recent workon bias in speech recognition has revealed [68, 103], VAs must be
trained on diverse data sets to maximize inclusion.

2.2

Operationalizing Masculine Biases

We provide an overview of how masculinity has been framed within
NLP and HCI work. We then introduce the less-trodden framing inthis body of work: implicit masculine biases in language use.
2.2.1 Toxic Masculinity, Misogyny, Sexism, and Stereotypes. Muchof the work on gender bias has focused on negative forms of masculinities. Toxic masculinity means adherence to traditional andlimited masculine gender roles whereby men withhold emotions
and avoid behaviours deemed not masculine and express themselves in ways regarded as masculine, especially anger, aggression,and dominance [55]. In NLP-related work, this has been explored
via toxicity [42, 85]. While the term “toxic masculinity” may not
be used (and anyone can partake in toxic behaviours), this body ofwork is coded masculine through explorations of “the manosphere,”or the virtual spaces in which primarily men promote narrow viewsof masculinity, if not sexism and anti-feminist beliefs. Misogyny
is a related concept, referring to hatred towards women and/or
femininities [76]. It is an extreme form of blatant sexism towards a
specific gender. Sexism is a broader concept, including stereotypes,
benevolent varieties [50], e.g., women have high emotional IQs,
and sex/gender-based discrimination against genderqueer people
and those outside of the gender binary, e.g., fear of trans folk. Toxicity and sexism are often bundled together in NLP work, thoughsome recognize that not all forms of toxicity are gendered [90].
The majority of other work on gender bias in NLP has focused onstereotypes, defined as widely held but limited and often sociallyharmful ideas about a group of people identified by a gender label[17, 27, 31, 37, 102], e.g., women being homemakers. As discussed,“gender-swapping” [102] and “gender evasive” approaches [94] arewidely used “debiasing” strategies, but are limited by a gender binary framing. In short, NLP and adjacent domains have approachedlanguage-based representations of masculinities in the extremes:

CHI ’23, April 23–28, 2023, Hamburg, Germany

toxic manifestations that need to be culled or counterparts to devaluations of femininity, all with a gender-constricted framework.
Yet, there are more subtle forms of masculine gender biases at playin the wider world of words.
2.2.2 Implicit Masculine Biases. Societies around the world at various times in history have tended to centre men and masculinity, aphenomenon that has been called androcentrism, male-centredness,male as norm, or male by default [2, 8, 11]. In this paper, we use
implicit masculine bias as an umbrella term to centre our framing
as a matter of gender and avoid the conflation of sex and genderin terms such as andro and male. We therefore also use the terms
masculine-centric language, masculine as norm, and masculine defaults or masculine by default [24], as well. This phenomenon of
implicit masculine bias is a reflection of how social power operates
through gender as a social construct, whereby men and masculinity
are positioned at the top of the social hierarchy; this is commonlyreferred to as a patriarchal system [46]. Masculinity and by association men are defined as the power-holders and society is organizedaround this stance [97]. The repercussions are vast, affecting most
aspects of most societies in obvious and subtle ways. For instance,a word embeddings study [3] using a corpus of over 630 billion
words from sources on the Internet showed that the “gender neutral”
words “person” and “people” are more frequently associated with“man” and “men.” We may not use masculine words explicitly, but
we mark humanity as masculine and masculine-centric by othermeans, including when we codify our worldviews in language.
The ways in which implicit masculine biases play out in languagevary. One way is the use of unambiguous masculine references as
neutral or universal, in presence and extent. As de Beauvoir articulated almost 75 years ago [8], masculine pronouns, possessives,
honorifics, and gender-marked words in romantic languages like
French and English are used by default and treated as neutral, universal, and “unmarked,” meaning that, for most people, they carryno extra meaning in relation to gender. These words are explicitly
gendered but also implicitly treated as gender-neutral, thereby positioning masculinity as fundamental and the ideal [2]. For example,
when presented with neutral stimuli, people are more likely toassign a masculine gender or use a masculine reference [54]. In contrast, feminine references are used to position “woman” as “other,”a special case that is distinct and oppositional to “man.” To givean encompassing example: “Guys, he throws like a girl.” Notably,masculinity is not typically positioned next to genders beyond thebinary or framed in a gender fluid way. Also, gender-marked words
and linguistic expressions may be referential, i.e., refer to specificentities, such as “the cowgirl,” or lexical, i.e., refer in a more general
way to entities that fall under that gender classification, such as“cowgirls are . . .” [20]. Frequency of masculine references is key, including gendered names, gender-marked words, like “businessman,”and masculine pronouns [2]. We seek to answer recent calls in NLP[37] by examining such masculine-as-norm or masculine-by-default
language as one linguistic variety of implicit masculine bias.
Another is gendered language, an implicit form of language usethat is virtually unexplored in NLP. Gendered language is the ideathat word choice itself is gendered, where the words people use inspeech or to refer to themselves in writing tend to be gendered inways that reflect their socialization and identity [48, 87]. Men, forCHI ’23, April 23–28, 2023, Hamburg, Germany

instance, tend to use dominant language to express masculinity interms of authority and power [67]. Other identities can intersectwith and influence gendered word choice. Roberts and Utych [87]found that Republican presidents of the US tend to use more masculine language than their “fellow” Democrats. Gendered language
can be unintentionally and unknowingly deployed, too. Gaucher,
Friesen, and Kay [48] found evidence of widescale gendered word
choices in job advertisements that resulted in stereotyped effects:
masculine language drew in more men and pushed more womenaway, despite these women knowing that they were qualified. In
the only NLP example we could find, Petreski and Hasim [83] discovered masculine-centric biases in Word2Vec word embeddings
via the Small World of Words (SWON-EN2018: Preprocessed) dictionary from De Deyne et al. [33]. However, the authors took a binary
approach that conflated sex and gender and excluded diverse sex
and gender participants and language; this work also did not relate to VAs. We attempt to translate this previous research to thecontext of NLP data sets built for VAs, exploring these notions of
the genderedness of language, especially word choice as masculine
language, in a more sensitive, gender-expansive way.
In HCI, we are creating VAs and CAs and CUIs and VUIs andvideo game characters and social robots that speak with the voicesof these data sets. We know that agents can reflect the implicit
biases of their creators. Nevertheless, these can be reinforced or
disrupted on awareness and the choice to act. Given recent efforts,
it is possible that new data sets, like MASSIVE, may be less biased
than older data sets, like ReDial. If not, we can take action now
before the models trained on these data sets become widespreadand the current versions of the data sets become standard.

3 DATA SETS
3.1 NLP Data Sets
Most NLP data sets are composed of labelled data. For example, adatum could be a photo of a cat, which may have one or more labels,
including “cat,” “furry,” “tortoiseshell,” “adorable,” “feline,” and so on.
Such sets of labelled data are used to train machine learning models.
The goal of creating these models is to teach the machine how torecognize new data that are similar to but not quite the same as thedata used during training. To evaluate these trained models against
data never seen in training, a single data set is usually separated
into three parts: "train," the training data; "dev," the data used after
one training cycle (of which there are typically many) to check the
progress of the model currently being trained; and "test," a selectionof data not used in training, i.e., data that the machine has yet toencounter and is “naïve” about, that is used to evaluate the final
model. Test data usually contains edge cases to better evaluate themodel’s performance. MASSIVE uses all three splits, while ReDial
provides train and test splits only.

3.2

The MASSIVE Data Set

We used the MASSIVE data set developed for intelligent voiceassistants, notably Amazon Alexa. MASSIVE was released on April
20th , 2022 and contains over one million utterances in 51 languages
across a range of scenarios and intents. The English data set contains

Katie Seaborn et al.

a total of 113,979 words. There are 6152 unique non-stemmed7
words, including stop words, e.g., “and,” “the,” “on,” and pronouns.
With stemming8 and stop words removed, there are 4940 uniquetokens. The data set is provided in JSON format with train, dev, andtest partitions for ML; for example:{"id": "13371", "locale": "en-US", "partition": "train", "scenario": "qa", "intent": "qa_maths", "utt": "how do you
subtract numbers", "annot_utt": "how do you subtract
numbers", "worker_id": "269"}{"id": "13373", "locale": "en-US", "partition": "dev", "scenario": "general", "intent": "general_quirky", "utt":"who are the top five all time n. h. l. goal scorers","annot_utt": "who are the top five all time n. h. l. goal
scorers", "worker_id": "263"}{"id": "13372", "locale": "en-US", "partition": "test", "scenario": "general", "intent": "general_quirky", "utt":"what time are the hockey games tonight", "annot_utt": "what time are the hockey games [timeofday: tonight]", "worker_id": "263"}
MASSIVE is a text-based version of the SLURP data set9 , anEnglish-only voice data set released in 2020 [7]. SLURP, or Spoken
Language Understanding Resource Package, was created to diversify audio-to-text data sets and reduce noise-based errors between
audio and semantic features. SLURP was created in two phases forthe context of an in-home personal robot that can handle many
features of daily life. First, commands for the robot were crowdsourced over AMT. Then, over 100 participants were asked to readout these commands in home and office settings for about 1 hour
each. While SLURP contains over 72k audio recordings, all are in
English. MASSIVE aimed to build on this data set by localizing
SLURP into 50 other languages. This was done using AMT; the fulldetails are provided in a preprint report10 .
We chose MASSIVE for several reasons. First, it is extremely new,representing the latest efforts and quality standards in designing
for NLP systems. Second, it was built on large-scale, crowdsourced
natural dialogue that was then validated within actual use contexts,
lending it a certain level of ecological validity. Third, it was madefreely available by a powerful entity and used in a global competition11 . As it is likely to spread quickly and be used widely, theimpact could be widescale. Fourth, it is available in a large rangeof languages. While the opportunity for global impact is high, thisalso allows for future analyses and comparisons across linguistic
and cultural settings. This may be especially important for mattersof gender bias and sexism, especially more subtle varieties, as indicated by cross-cultural work [82]. Fifth, the creators of the data
set appear to be open to, if not eager for, evaluations and criticalfeedback on the data set. We offer a new perspective on evaluating
the quality and fairness of the data set in terms of gender.
7 Stemming is an inexact process of reducing an instance of a word to its stem, e.g.,“running” to “run.” Lemmatization or lemming is the process of reducing a word to its
root form exactly, i.e., not just removing the “ing” to find the stem but transforming it
into its dictionary form, called the lemma.
8 In quanteda, stemming is done with Snowball: http://snowball.tartarus.org
9 https://github.com/pswietojanski/slurp
10 https://arxiv.org/abs/2204.08582
11 The Massively Multilingual NLU 2022 workshop (MMNLU-22 at EMNLP 2022), whichstarted on July 25th , 2022: https://mmnlu-22.github.io

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

3.3

The ReDial Data Set

The ReDial dataset [73] was created for training agents with goaldirected and conversational dialogue in English. ReDial was a multiparty effort, involving Google, IBM, and the National Sciences andEngineering Council of Canada (NSERC), and support from Microsoft Research. The data set is made up of dialogues between
two people recommending and talking about movies, especially
their thoughts on, feelings about, and desires to find movies. Published in 2018, it contains 11,348 dialogues with 1,371,903 words
and 16,261 unique non-stemmed words, including stop words, e.g.,“and,” “the,” “on.” With stemming and stop words removed, there are
12,281 unique tokens. The dialogues were crowdsourced by pairs
on AMT with an unknown number of people located in Canada,
the US, the UK, Australia, and New Zealand. The data set is freely
available in JSON format under the CC BY 4.0 License. More details
are available in Li et al. [73]. Here is an example from the training
data set, formatted and truncated for readability:{"timeOffset": 48, "text": "I saw @119144 last night andreally liked it!", "senderWorkerId": 30, "messageId":3183},{"timeOffset": 94, "text": "That was a good movie. If
you like Superhero Movies you should check out@169419", "senderWorkerId": 44, "messageId": 3184},{"timeOffset": 112, "text": "Is that out already? I really wanted to see that one.", "senderWorkerId": 30,"messageId": 3185},
Like MASSIVE, ReDial offers a large, English-language data set
of utterances crowdsourced on AMT and geared towards training
VAs. However, it is a relatively older data set (2018 versus 2020/22)
released just as momentum on gender bias and debiasing was beginning to take root. As such, it may offer a time-sensitive comparison.

4

STUDY 1: IMPLICIT GENDERED LANGUAGE

We carried out Study 1 in three phases: the initial study (Study 1.1),
which led to a dictionary development phase (Interlude), whereafter we reconducted the initial study (Study 1.2) using this new
dictionary: AVA. The original procedure was registered on OSF12
in advance of data analysis on August 7th , 2022 and updated on
January 25th , 2023.

4.1

Methods

We analyzed the relative presence of implicit gendered language
using automated content analysis, i.e., content analysis performedby computers using NLP methods to extract meaningful patterns
from text materials. We used two dictionaries applied to the English(US) subset of the MASSIVE data set and the entire ReDial data set.
We supplemented the automated results with a manual contextual
analysis to dis/confirm the genderedness of each term.
4.1.1 Dictionaries. An overview of the two dictionaries is in Table
1. These were the only dictionaries on implicit gendered languagethat we could find. We chose each based on its relevance (does it
assess masculine language?), quality (was it developed in a rigorousway?), and peer acceptance (do academics and/or experts in NLP
12 https://osf.io/dwf5v

CHI ’23, April 23–28, 2023, Hamburg, Germany

use it?). The dictionaries are diverse: varied in context, date oforigin, method of creation, format, number of tokens. However, they
are limited to the gender binary, offering feminine and masculine
gendering only. We describe each dictionary in detail below:
The dictionary by Roberts and Utych [87] was created to explore
the relationship between implicit gendered language and partisanship, developed through crowdsourcing on AMT. The authors asked
175 mturkers to rate the genderedness of words selected from the
Affective Norms for English Language (ANEW) data set by Bradleyand Lang [18] using a bipolar 7-point scale with a neutral centre.
They defined two cut-offs for masculine—greater than or equal to 5
(n=77) or greater than 5.5 (n=30)—and feminine—less than or equal
to 3 (n=36) or less than 2.5 (n=8)—with 4 as the gender-neutral
centre. This was to account for noise in the data and biases arisingfrom the crowdsourcing method. Examples for masculine words
include “athletic,” “intense,” and “tough.” Examples for feminine
words include “sincere,” “cute,” and “love.”
Gaucher et al. [48] handcrafted a list for analyzing language use
in job advertisements based on explicit (masculine and feminine)
and implicit (agentic and communal) associations with various
words and gender. Examples of feminine words include “sensitive,”“interpersonal,” and “communal.” Examples of masculine words
include “active,” “confident,” and “decisive.”
4.1.2 Data Preparation and Analysis. We used RStudio with R version 4.1.2 (2021-11-01) or "Bird Hippie." We followed the data prep
guidelines by Welbers, Van Atteveldt, and Benoit [108]. For thedata sets and dictionaries, we used the quanteda default for patternmatching, glob, which uses a combination of the regular expression characters of * for any number of characters and ? for any
single character, e.g., masc* and masc?, thereby accounting for mostuse cases without excluding viable results. We also removed stop
words as recommended by the guidelines we followed [108]. We
conducted overall analyses but also divided our analysis by ML
partition, i.e., train, dev, and test, to isolate the potential impactof bias in the training process. We then followed the automatedcontent analysis procedures by Puschmann and Haim13 and Hase14 .
We used the quanteda15 (3.2.0), Tidyverse16 , lexicon, rcompanion,
dplyr, car, rstatix, DescTools, and jsonlite R packages. Quanteda is
widely-used for natural language processing of text data in qualitative research [108]. It provides packages for conducting automatedcontent analyses of a wide variety of textual data and the key functions for NLP and textual data management. We generated counts,
frequencies, and ratios, in line with automated content analysis procedures for NLP data sets [1, 1, 23, 42], as well as other forms of text
data [48, 87]. We used inferential statistics, such as Chi-square tests,
to compare the results of the dictionaries by language gendering.
Given the uneven groups, i.e., more feminine words in a dictionary
compared to other implicitly gendered words, we used relative frequencies for each set of words given the total in the dictionary as
well as corrective statistics, such as the Games-Howell.
For the contextual analysis, we drew on summative content analysis methodology, a manual approach to analyzing qualitative data
13 https://content-analysis-with-r.com

14 https://bookdown.org/valerie_hase/TextasData_HS2021
15 https://quanteda.io

16 https://www.tidyverse.org

CHI ’23, April 23–28, 2023, Hamburg, Germany

Katie Seaborn et al.

Table 1: Dictionaries used in Study 1.1
CitationYear

Gendering

Word Sources

Categorization Method

Format & Interpretation

Total

Roberts & Utych [87]2019

ANEW [18], curated
synonyms

AMT (n=175; 2018)2011

Agentic and communal
words [6, 89], masc. andfem. trait words [10, 58, 93]

Authors (manual)

1-7-point scale, where fem.
<= 3 and masc. >= 5, orfem. < .25 and masc. > 5.5
Categorical list

124

Gaucher et al. [48]Masculine,
neutral,feminine
Masculine,
feminine

31
79

Year: Publication date of the associated paper. Total: Number of tokens after stemming, lemming, and removing stop words.

Figure 1: Visualized results for implicit gendered language in the MASSIVE data set (Study 1).
that is comprised of two phases [60]. Automated content analysis
effectively maps onto the first phase: manifest content analysis,
wherein counts and sums are produced. Contextual analysis thus
represents the next phase: a latent content analysis of these quantitative results within the data sets, where the aim is to interpretthese “gendered” patterns against the VA and/or NLP context. This
involved multiple authors, working alone or in concert, determining
the meanings, potentially multiple, of each word from gender- andcontext-oriented perspectives by looking at the data sets directly,
i.e., how the words were actually used in the utterances, sentences,and dialogues within MASSIVE and ReDial. Disagreements werediscussed until consensus was reached.

4.2

Study 1.1: Initial Results

We summarize the results on implicit gendered language for MASSIVE in Table 2 and Figure 1 and ReDial in Table 3 and Figure 2.
We found that .23% of MASSIVE and 1.2% ReDial represented implicit gendered language. One-quarter of the dictionary terms were
found in MASSIVE and one-half of the dictionary terms were foundin ReDial. However, we also found evidence of ambiguous terms:
words that could relate to gender or be perceived as gendered, butalso relate to the VA context and/or technology more broadly (refer
to 4.3). Re-conducting our analyses without these words, we foundthat gendered language was still present, albeit significantly reduced: .08% for MASSIVE and .19% for ReDial. Moreover, there wasstill a masculine gendered language bias in both data sets, and thiswas particularly pronounced for MASSIVE. We start by presenting
the results on the most frequent words, then compare masculine
and feminine gendered language rates, and then our process ofidentifying ambiguous words given the differences between thedata sets and dictionaries. We then report on our analyses without
ambiguous terms and compare the data sets directly.
4.2.1 Word Shares and Frequencies. We considered word shares(how many gendered words) and frequencies (how many instancesof each gendered word). The results for the 20 most common words
per dictionary are in Table 4; the full list is in Appendix A.
For MASSIVE, the conservative automated content analyses forRoberts and Utych [87] indicated a masculine language bias in word
shares (25 masc., 13 fem., 65.8% masc.; 5 masc., 2 fem., 71.4% masc.)
and frequencies (93 masc., 79 fem., 54.1% masc.; 19 masc., 13 fem.,
75% masc.). In contrast, the results for Gaucher et al. [48] indicated afeminine language bias in word shares (5 masc., 9 fem., 35.7% masc.)
and frequencies (13 masc., 74 fem., 14.9% masc.). When applying
the loose Roberts and Utych [87] dictionary to ReDial, an implicit
masculine language bias was found in terms of shares (52 masc.,

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

CHI ’23, April 23–28, 2023, Hamburg, Germany

Table 2: Summary of results for gendered language in the MASSIVE data set (Study 1)
Dictionary

Dict. Shared

Total
Instances

Partition

Masc.
Freq.

Fem.
Freq.

Masc.
Ratio

Fem.
Ratio

Roberts & Utych [87]: Loose criteriaa

38/124 (29.7%)

172

Roberts & Utych [87]: Conservative
criteriab

5/31 (22.6%)

12

Gaucher et al. [48]14/79 (17.3%)

87

Roberts & Utych [87] and Gaucher et al. 51/209 (24.4%)[48] combined

258

Combined without AVAc terms

66

overall
dev
test
train
overall
dev
test
train
overall
dev
test
train
overall
dev
test
train
overall
dev
test
train

93
15
14
64
9
1
0
8
74
1
3
9
106
16
17
73
54
9
9
36

79
7
19
53
3
0
1
2
13
6
12
56
152
13
31
108
12
0
2
10

.541
.682
.424
.547
.75
1
0
.800
.149
.143
.200
.138
.411
.552
.354
.403
.818
1
.818
.783

.459
.318
.576
.453
.25
0
1
.200
.851
.857
.800
.862
.589
.448
.646
.597
.182
0
.182
.217

22/169 (13%)

Dict.: Dictionary. Masc.: Masculine. Fem.: Feminine. Freq.: Frequency. a Loose: fem. <= 3 and masc. >= 5. b Conservative: fem. < .25 and masc.
> 5.5. c AVA terms: Ambiguous Virtual Assistant terms. d Dictionary share: How many of the total possible dictionary terms were found in
the data set, with at least one instance per term.
Table 3: Summary of results for implicit gendered language in the ReDial data set (Study 1)
Dictionary

Dict. Shared

Total
Instances

Partition

Masc.
Freq.

Fem.
Freq.

Masc.
Ratio

Fem.
Ratio

Roberts & Utych [87]: Loose criteriaa

79/124 (61.7%)

12,420Roberts & Utych [87]: Conservative
criteriab

22/31 (71%)

1820

Gaucher et al. [48]37/79 (45.7%)

4328

Roberts & Utych [87] and Gaucher et al. 113/209 (54%)[48] combined

16,697

Combined without AVAc terms

2472

overalltest
train
overalltest
train
overalltest
train
overalltest
train
overall
test
train

1696
162
1534
989
109
880
4237
5
86
1787
167
1620
1344
127
1217

10,724
1185
9539
831
45
786
91
524
3713
14,9101705
13,205
1128
69
1059

.137
.120
.139
.543
.708
.528
.979
.009
.023
.107
.089
.109
.544
.648
.535

.863
.880
.861
.457
.292
.472
.021
.991
.977
.893
.911
.891
.456
.352
.465

74/169 (43.8%)

Dict.: Dictionary. Masc.: Masculine. Fem.: Feminine. Freq.: Frequency. a Loose: fem. <= 3 and masc. >= 5. b Conservative: fem. < .25 and masc.
> 5.5. c AVA terms: Ambiguous Virtual Assistant terms. d Dictionary share: How many of the total possible dictionary terms were found in
the data set, with at least one instance per term.

CHI ’23, April 23–28, 2023, Hamburg, Germany

Katie Seaborn et al.

Figure 2: Visualized results for implicit gendered language in the ReDial data set (Study 1).
Table 4: Word frequencies for the top 20a words in the MASSIVE and ReDial datasets (Study 1)
Roberts and Utych [87]MASSIVE
ReDialToken
G. Freq. Token
love
f 45
dog
m 25
power
m 22
soft
f 8
bake
f 7
m 5
directhope
f 5
teacher m 5
man
m 3
grill
m 3
f 3
style
barbecue m 3
guy
m 3
virgin
f 2
m 2
crude
rebel
m 2
strongm 2
director m 2
industry m 2
tough
m 2

Roberts and Utych [87]: Conservative
Gaucher et al. [48]MASSIVE
ReDialMASSIVE
ReDial
G. Freq. Token
G. Freq. Token
G. Freq. Token
G. Freq. Tokenlovef 6462hope
f 3028
cute
f 553
man
m 365
guy
m 300
m 197
hero
cry
f 159
intense
m 157
woman
f 141
director m 132
style
f 115
beautiful f 102
dog
m 67
strong
m 61
m 51
buddy
emotional f 49
violent
m 47
tough
m 44
fabulous f 27
adorable f 24

man
m 3
guy
m 3
woman
f 2
terrorist m 1
beautiful f 1
captainm 1
handsome m 1

cute
f 553
man
m 365
guy
m 300
hero
m 197
woman
f 141
beautiful f 102
violentm 47
adorablef 24
prisonm 23
captain
m 17
rough
m 16
terroristm 8
handsome m 6
destruction m 6
heroine
f 4
sassy
f 4
cherish
f 2
brash
m 1
jock
m 1
sensitive
f 1

f 17
quiet
warm
f 17
f 14
kind
togetherf 13
m 7
active
f 4
cheer
f 4
child
f 3
pleasant
m 3
analyzef 1
connection
challenge
m 1
compassionate f 1
intellectual
m 1
leaderm 1

G. Freq.

kind
f 3801
together
f 135
child
f 79
honestf 65
emotional
f 49
trust
f 33
decide
m 30
quiet
f 14
warm
f 14
independent
m 13
understandable f 10
connection
f 9
f 9
pleasant
intellectualm 8adventurous
m 7commitf 5
challenge
m 5
compete
m 5
cheer
f 5
leaderm 5

G.: Group. Freq.: Frequency. a Up to 20 if that amount exists.
27 fem., 65.8% masc.) and an implicit feminine language bias interms of frequencies (1696 masc., 10,724 fem., 13.7% masc.). For theconservative version, there was an implicit masculine bias in termsof shares (14 masc., 8 fem., 63.6% masc.) but almost a balance infrequencies (989 masc., 831 fem., 54.3% masc.). The Gaucher et al.
[48] dictionary indicated a balance in shares (18 masc., 19 fem., 48.6%
masc.) but an implicit feminine bias in frequencies (91 masc., 4237

fem., 2.1% masc.). Overall, we discovered apparent disagreement
between the dictionaries on the relative masculine and feminine
language bias in the data sets. This led us to pause and turn tocritically examining why through contextual analysis.
4.2.2 Contextual Analysis. Our findings on the most frequent dictionary terms found in each data set (Table 4) highlighted a potential

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

disconnect between the context of each dictionary in relation to thecontext of each data set [53, 108]. As Hase, referring to Grimmer andStewart [53], recommends, we need to assess how the dictionary
was constructed, when, and for what situations, as well as what
relationships between dictionary terms and words in the data sets
would theoretically (or logically) be expected. We also needed toconsider the contexts in which the terms were used in the data sets,i.e., VAs and NLP.
For both data sets, the Gaucher et al. [48] dictionary indicated aclear feminine bias. Yet, it has a few special properties that could
explain this difference in the context of an NLP data set for VAs andother dialogue-based agents. It is an older dictionary (2011) withfew terms (n=82). It was handcrafted by a small group of people(the authors of the paper) who wove together lists of words fromvarious sources for the context of job ads, taking heed of the possible occupations and their gendered associations. This context andthe relative frequency of words used within it may not necessarilytranslate to conversational data sets constructed within daily homeand office contexts or in conversations about movies and technology. Moreover, over time, shifts in attitudes towards occupational
gender stereotypes may have occurred, in favour of women but notmen [110]. Trends indicate that greater rates of women have been
taking on traditionally masculine occupations and are encouraged
to do so, but neither tends to be the case for men when it comesto traditionally feminine occupations. For example, we have bootcamps for girls in computer science, but we do not have bootcamps
for boys in nursing and caregiving [84]. A nuanced review of thedictionary terms was needed.
Table 4 in conjunction with a direct examination of the data sets
points to instances that may not be about gender, per se. “Love”(MASSIVE n=45, ReDial n=6462), the most frequent word from theRoberts and Utych [87] dictionary, was mainly used to indicate
preference, e.g., “I do love scary movies.” Other words appeared to
be closely related to VA tasks and contexts of use. In the Gaucher
et al. [48] dictionary, “warm” (MASSIVE n=17, ReDial n=14), forexample, was almost exclusively used in the context of asking about
the weather, one of the most typical VA commands identified [105].
“Kind” (MASSIVE n=14, ReDial n=3801) was almost exclusively
used in both data sets as a colloquial analogy of “type” or “variety,”e.g., “what kind of music?” Still others were multifaceted and contextual. One example from the Roberts and Utych [87] dictionary
was “power,” which was used in MASSIVE to refer to electricity,
e.g., “save power,” but in ReDial as a possibly masculine descriptive characteristic, e.g., “all-powerful Hela.” In short, a critical lensrevealed shortcomings related to contextual differences. To move
forward, we needed to address these shortcomings.

4.3

Interlude: Developing a Context-Sensitive
Dictionary of Ambiguous VA Terms, or AVA

We did not have confidence in the Study 1.1 results. As such, weevaluated the terms that were found in both data sets—as an indicator of their prevalence—from a critical, reflective angle [88] andgenerate new materials to enable a more nuanced, contextualized
approach to analysis [12, 78]. This led us to create a new dictionary of ambiguous terms for the context of VA-oriented data sets:
the Ambiguity for Virtual Assistants dictionary or AVA. We defined

CHI ’23, April 23–28, 2023, Hamburg, Germany

ambiguous terms as those deemed by existing dictionaries to beinstances of implicit gendered language that also relate to the VA
context in gendered or gender-neutral ways, i.e., there could be
multiple legitimate interpretations of a given term that requires
contextual analysis to discern. We used the following criteria tocode the terms:• The term was from an implicit gendered language dictionary,i.e., Roberts and Utych [87] and Gaucher et al. [48], and• At least one instance of the term was found in either MASSIVE or ReDial, and• The term was ambiguous in one of the following ways within
the VA context of the NLP data sets:◦ The term did not carry the expected implicit gendered language connotations, i.e., it had a special meaning within
the VA context that did not appear to be an instance of
implicit gendered language but could still carry genderedassociations, e.g., the term ‘assist’ may be neutral or ambiguous, given the feminization of modern VAs and the‘assistant’ role, or• When multiple instances of the term were found:◦ At least one instance carried the expected implicit gendered language interpretation of the term, and◦ At least one other instance appeared to relate to the VA
context and did not carry the expected implicit genderedlanguage interpretation of the term, even if it could still
carry gendered associations, e.g., the term ‘strength’ isused in the context of volume levels but also to describe‘a man with great physical strength’
In creating AVA, we drew from critical discourse analysis [41],focusing on the relations and murky intersections of gendered andmechanical associations embedded in the words and the contexts
in which they are found in the data sets. For this, the first author
used the Study 1.1 results as a guide to manually read through thedata sets and isolate words that were ambiguous in relation to thecontext, i.e., possibly (implicitly) gendered, possibly VA-specific,
or possibly both. Three authors native or fluent in English thencarried out independent content analyses of this list, rating eachterm as ambiguous or not with an example from either MASSIVE or
ReDial. As labellers, we relied on our combined expertise in genderresearch and technology domains to rate each term; refer to Table
5 for relevant demographics. When disagreements arose, the teamdiscussed each case using examples drawn from the data sets until
consensus was reached. A Krippendorff’s alpha test was used toestimate the inter-rater reliability among the three of us as raters[56]. All terms achieved 𝛼 ≥ .988, indicating high agreement.
From an initial set of 185 terms, we isolated 44 for inclusion, with
26 deemed masculine by the original dictionaries and 18 deemedfeminine. Examples are “love” (“I love action movies”), “power”(“powered off”), “connection” (“internet connection”), “kind” (“kindof weather”), and “quiet” (“be quiet”). We formatted the dictionary
in JSONL, which includes the term, its genderedness as originallyassigned in the dictionary, and examples from the MASSIVE and
ReDial data sets showing how it has an ambiguous application in
VA contexts. The JSONL is in the Supplementary Materials.

CHI ’23, April 23–28, 2023, Hamburg, Germany

Katie Seaborn et al.

Table 5: Labeller demographics for the AVA dictionary terms
Labeller

Technical Domain Knowledge

Gender Domain Knowledge

First author

Researcher in HCI with a Ph.D. degree in engineering,with over a year of postdoc experience in an ML lab andtwo decades of industry experience in the tech sector.
Have been developing and publishing research on VAs,
especially on the topic of speech, for three years, andhave also been working and publishing in adjacentspaces, including social robotics, for ten years.
Researcher in HRI with a Ph.D. degree in engineering.
Nine years of experience working with social robots anddesigning verbal and non-verbal behaviours in robots associal agents. Have been working on the topic of speech
dysfluency for 3 years. Have conducted several HRI user
studies in which social robots use speech as a primarymode of communication.
Ph.D. student in HCI with a M.Sc. degree in engineering.
Experience with and knowledge in ML with passing
knowledge on using ML models for NLP.

Have been involved in feminist and social justice spaces
for a decade and have published feminist work on robots
in the last two years.

Second author

Third author

Have been involved in research on and conducting
workshops related to equity, diversity, and inclusion atacademic venues in the last two years.

Following current LGBTQ+ issues with a vested interestin a more equal and open-minded society.

Table 6: Word frequencies for the top 20 words in the MASSIVE and ReDial datasets without AVA terms (Study 1.2)
MASSIVE
Token

G.

Freq.

ReDial
Token

G.

Freq.

dog
activeteachermanpleasantguyvirgin
rebel
buddy
woman
peaceful
cry
abuse
chief
compassionate
furious
beautiful
captainattractive
intellectual

m
m
m
m
f
m
f
m
m
f
f
f
m
m
f
m
f
m
f
m

25
7
5
3
3
3
2
2
2
2
1
1
1
1
1
1
1
1
1
1

cute
man
guy
hero
cry
intensewoman
beautifuldog
buddy
emotional
fabulousadorablebudprisonpowerfulcaptainthoughtful
furiousteacher

f
m
m
m
f
m
f
f
m
m
f
f
f
m
m
m
m
f
m
m

553
365
300
197
159
157
141
102
67
51
49
27
24
24
23
19
17
13
12
12

G.: Group. Freq.: Frequency.

4.4

Study 1.2: Follow-Up Analysis of Gendered
Language with AVA

We used the AVA dictionary to remove ambiguous terms from the
combined dictionary and then re-ran the previous analyses; refer
to Section 4 for details. We used a removal strategy to isolate thegenderedness of non-AVA terms, i.e., terms that are not ambiguousin the VA/NLP content, in the data sets. Summaries for MASSIVE
are in Table 2 and Figure 1, and for ReDial, refer to Table 3 andFigure 2; the most common words for both are in Table 6.
First, we evaluated the relative gender biases in terms of frequencies with AVA terms removed. For MASSIVE, a Chi-square
goodness-of-fit test indicated that the ratio of masculine and feminine gendering was inconsistent, with more implicit masculine

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts(n=54) than feminine (n=12) words found, 𝜒2 = 26.73; df = 1; p <.001. A similar pattern was found for ReDial, with more implicit
masculine (n=1344) than feminine (n=1128) words found, 𝜒2 =18.87; df = 1; p < .001. In short, accounting for ambiguous terms
revealed evidence of implicit masculine language biases. We thencompared the distributions of implicit masculine and feminine language, without AVA terms, by data set. We found that the relative
frequencies significantly differed from expectation, 𝜒2 = 18.48; df= 1; p < .001, with greater divides in MASSIVE. This indicates thatMASSIVE has a greater degree of implicit masculine language bias.

4.5

Discussion

Both the MASSIVE and ReDial data sets were found to be implicitly
biased with gendered language. This was to a small degree relative to all the words in the data set. We must also consider thisagainst the relatively small size of the dictionaries and the data setsthemselves. Yet, we cannot ignore the presence of implicit genderedlanguage. Even when we account for ambiguous cases with the
AVA terms, the bias persists, and it is masculine valanced. While
we cannot evaluate the effect that this biased language may or may
not have, it is troubling, but potentially avoidable.
A key contribution of Study 1 was the AVA terms, which can beused in and outside of this research to isolate words with layered
meanings in terms of gender and VA contexts for various purposes,
and not only debiasing. The efficacy of AVA deserves a more nuanced discussion. The data sets may be different in important ways.
ReDial may contain more implicit gendered language and a stronger
masculine bias because it is older, or because it is about movies,
which may give rise to discourse that is relatively more gendered
and masculinized than the kinds of discourse that may occur in ahome or office setting with a daily assistive robot, i.e., MASSIVE.
We must also interrogate these findings in light of the genderedassociations of these words, who was involved in their creation, i.e.,
mostly men in positions of power, and the existence of legitimate
alternatives [29, 107]. Computer technology has largely been a masculinized domain dominated by men [29, 107], and this has led tothe conscious and unconscious use of value-laden terminology built
into tools, such as programming languages and praxis. ACM17 andothers in the computing profession have taken strides to correct thisby advocating against sexist, racist, ableist, ageist, and privileged
language [70]. We might ask: why light “intensity” rather than“levels” and “power” over “electricity”? Likewise, “quiet” (n=17,15)
could be about volume levels or rude demands that may have a sexist, toxic element to them. This would not be unexpected because
most intelligent agents have stereotypically feminine-gendered,subservient embodiments that are known to elicit gender-basedharassment, by naïveté or by design [44, 59, 100]. We must also
consider the gaps. For example, why are there so many mentionsof dogs and so few of cats? Recent statistics on pet ownership fromthe American Veterinary Medical Association18 would suggest twofifths or 10 mentions of cats relative to the 27 mentions of dogs.
Moreover, half of the mentions to cats were to a person’s namerather than the animal. This may relate to long-standing gendered
17 https://www.acm.org/diversity-inclusion/words-matter

18 https://www.avma.org/resources-tools/reports-statistics/us-pet-ownership-

statistics

CHI ’23, April 23–28, 2023, Hamburg, Germany

associations of masculinity with dogs and femininity with cats,
whereby dogs receive more space and recognition than cats [96].
We must consider the scale and layers as well as the intersections
between features of the data sets, language, and society.
Implicit gendered language is but one form of implicit genderingthat could be present in NLP data sets. We now turn to another forcomprehensiveness and comparison: masculine-as-norm language.

5

STUDY 2: MASCULINE-AS-NORM
LANGUAGE

We next evaluated the relative presence of masculine-as-norm language. We operationalized masculine-as-norm language according
to the literature: pronoun use, gender-marked words, and names.

5.1

Methods

We used the same procedure as in Study 1, i.e., automated content
analysis, with a different selection of dictionaries. When running
our analysis with the pronoun dictionary, we did not remove stop
words, which include pronouns.
5.1.1 Dictionaries. An overview of the dictionaries we used ispresented in Table 7. As before, we chose dictionaries that wererelevant, established, and freely available. We also tried to accountfor the relative strengths and weaknesses of the range of dictionaries
available, especially in terms of genders beyond the binary. We
describe each dictionary below:
For pronoun use, we selected the pronouns dictionary byLauscher, Crowley, and Hovy [71] because it includes a diversityof pronouns outside of the gender binary. It is divided into pronoun categories, including gendered pronouns, e.g., he/her/him,
gender-neutral pronouns, e.g., they/them/their, and neopronouns,e.g., ze/zir.19 It was compiled by the authors from several sources,
including academic publications and community texts.
For gender-marked words, we used the gendered words subset of
the Multi-Dimensional Gender Bias Classification dataset created by
Dinan et al. [36]. The dictionary is gender binary, only containing
gendered words typically used to refer to men and women. The genderedness of the words was annotated through AMT crowdsourcing.
Notably, most of the mturkers were men-identifying (67.38%) andfew were non-binary (0.21%), although ∼14% did not indicate agender identity. Examples of masculine words include “policeman,”“cowboy,” and “dude.” Examples of feminine words include “policewoman,” “cowgirl,” and “gal.” Note that gender marked words may
refer to specific subjects (“Misako, a policewoman”) or not (“a policewoman takes charge”) [20]; we do not assess this, but rather
aim to capture the genderedness of language overall.
We also used the Gendered Words Dataset13 for gender-marked
words. It was compiled by GitHub contributors ecmonsen, phseiff,and Guy Rapaport (@guy4261), who drew from the WordNet®dataset [43]. They manually tagged hyponyms for “person” as masculine, feminine, neutral, or other. Examples of masculine words
include “businessman,” “codger,” and “wingman.” Examples of feminine words include “cat,” “dame,” and “jezebel.” Examples of neutral
words include “person,” “advocate,” and “codefendant.”
19We also did not use an exhaustive list, as there is no formalized dictionary yet, which

is beyond the scope of this work to create; future research may wish to start with
community resources such as https://en.pronouns.page/pronouns

CHI ’23, April 23–28, 2023, Hamburg, Germany

Katie Seaborn et al.

Table 7: Dictionaries used in Study 2
Citation

Year Referent
Type

Gendering

Word Sources

Categorization
MethodFormat &Interpretation

Total

Lauscher, Crowley,& Hovy [71]Dinan et al. [36]2022 Pronouns

Masculine, feminine,
neutral, neoa
Masculine, feminine

Various

Authors (manual)Categorical lists

49

AMT

Authors (manual)Categorical lists

409

WordNet®2 ,hyponyms
AMT

Authors (manual

JSON, gender as m,f, n, oCategorical lists

6244

Gendered Words
Dataset1
Dinan et al. [36]2020 Marked
Words
2019 Marked
Words
2020 Names

Masculine, feminine,
neutral, other
Masculine, feminine,
neutralAuthors (manual)

22,609

Year: Publication date of the associated paper. Total: Number of tokens/words in the dictionary. a Note: We do not include nounself, emojiself,
numberself, and nameself pronouns due to the difficulty in discerning from non-pronouns and hypothetical use in reality.
1 https://github.com/ecmonsen/gendered_words
2 https://wordnet.princeton.edu
For names, we used the name genders subset of the MultiDimensional Gender Bias Classification dataset created by Dinan
et al. [36], which uses the 1879-2019 US census data. Gender neutral names were those attributed to at least one “man” and one“woman” in this sex/gender binary-based census, e.g., Jaime, Riley,
Taylor. We recognize that this approach is limited. Names do notnecessarily indicate gender or sex [20, 34]. Names are typicallyassigned at birth based on apparent sex, which may not matchinternal sex characteristics or gender identity. Not everyone may
perceive a given name as gender neutral. Even when written in
English, names may have origins or reorientations based in religion,
ethnicity, cultural background, nationality, and so on that are notacknowledged in normative English contexts. For example, TwoSpirit people of Turtle Island (Indigenous North America) may be
assigned a gender identity and name within the Western colonial
context. Yet, “Two-Spirit” is pluralistic, varying across Indigenous
communities and cultures, intersecting with sex, sexuality, and nonWestern cultural factors in ways that do not always map onto theWestern-centred “LGBTQI+” model, such that no “translation” toEnglish would be appropriate [32]. We do not know the extent towhich any of these patterns are the case for deemed gender neutralor other names; it likely varies across time, context, and individual.
Still, given masculine-as-norm biases, gender neutral or ambiguous names may be read as masculine; on the same token, feminine
names may not necessarily be read as feminine. In short, names area source of gender bias, where people make judgments, knowingly
or not, about the gender of the person based on their understandingof that name within their particular socio-cultural context [20]. We
include names as salient feature in dialogic materials that can act
as a gender stimulus.

5.2

Results

The results for MASSIVE are in Table 8 and Figure 3; ReDial is in
Table 9 and Figure 4. We start with pronouns, then consider marked
words using both dictionaries, and end with names.
5.2.1 Pronouns. Most pronouns were plural and/or gender neutral,e.g., they/them. This is a positive sign if we are aiming for gender

neutrality and inclusivity. However, it is difficult to determine whenthey/them pronouns are used in the singular, when they are used
to refer to multiple entities, and when they are used in an objective
sense. For example, one utterance states “who emailed me recently
and what did they [ambiguous singular or plural] email about,”
while another states “check my new emails and tell me what are
they [the emails] about” (our emphasis). Moreover, we are notconfident in the neopronoun results, as manual checks indicatethat many are abbreviations of other pronouns (“them” → “em”)
or typos. Those results should be interpreted with caution. We
begin with MASSIVE. A Chi-Square test did not find a significant
difference in the distributions of the masculine (n=71), and feminine(n=50) groups from expected, 𝜒2 = 3.65; df = 1; p = .056. Thisborderline significance may indicate a small masculine pronoun bias
with larger samples. However, a Chi-Square tests found significant
differences between masculine (n=71), 𝜒2 = 9.19; df = 1; p = .002,
and feminine (n=50), 𝜒2 = 23.73; df = 1; p < .001, groups comparedto the neutral (n=112) group. This indicates that there were far moreneutral or gender-ambiguous pronouns in MASSIVE. For ReDial,
masculine (n=3501) and feminine (n=1411) groups significantly
differed from the expected distribution, 𝜒2 = 889.27; df = 1; p <.001, indicating a masculine pronoun bias. However, Chi-Square
tests found that there were significantly more gender neutral or
ambiguous pronouns (n=6426) compared to the masculine, 𝜒2 =861.85; df = 1; p < .001, and feminine, 𝜒2 = 3209.2; df = 1; p <.001, groups. This suggests that, like MASSIVE, ReDial has moreinstances of gender neutral or ambiguous pronouns.
5.2.2 Marked Words. We start with Dinan et al. [36]. For MASSIVE,a significant difference in the expected distributions for masculine(N=185) and feminine (N=243) marked words was found, 𝜒2 =7.86; df = 1; p = .005, indicating a feminine bias. For ReDial, theopposite pattern was found for masculine (n=6693) and feminine(n=3154) groups, 𝜒2 = 1271.9; df = 1; p < .001. In short, MASSIVE
was feminine-biased and ReDial was masculine-biased. Taking a
critical look at the frequencies of individual words in the dictionary indicates a subtle masculine bias. The most frequent marked
words for MASSIVE were: mom (n=63), wife (n=15), mother (n=15),

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

CHI ’23, April 23–28, 2023, Hamburg, Germany

Table 8: Summary of results on masculine-as-norm language for the MASSIVE data set (Study 2)
Dictionary

Dict. Share

Total
Instances

Partition Masc.
Freq.

Fem.
Freq.

Neu.
Freq.

Neo.
Freq.

Masc.
Ratio

Fem.
Ratio

Neu.
Ratio

Neo.
Ratio

Pronouns

9/49 (18.4%)

183

overall
dev
test
train
overall
dev
test
train
overalldev
test
trainedoveralldev
test
overall

50
5
6
39
243
31
36
176
195
22
30
143
3554
444
643
2467

112
16
27
69

8
3
1
4

.388
.226
.292
.309
.432
.380
.455
.436
.091
.076
.100
.083
.484
.344
.365
.348

.273
.161
.125
.241
.568
.620
.545
.564
.070
.059
.061
.068
.516
.377
.370
.375

.612
.516
.562
.426

.044
.097
.021
.025

Dinan et al. [36] marked 73/409 (17.9%) 428
words
Gendered Words Dataset1 366/6244(5.7%)

2768

Dinan et al. [36] names
without agent namesa

6882

843/22,609(3.7%)

71
7
14
50
185
19
30
136
252
28
49
175
3328
405
634
2289

2516
320
411
1785
2618
329
460
1829

.909
.865
.839
.849
.38
.279
.265
.278

Dict.: Dictionary. Masc.: Masculine. Fem.: Feminine. Neu.: Neutral. Neo.: Neopronoun. Freq.: Frequency. a Specifically: Alexa, Siri, Olly.

1 https://github.com/ecmonsen/gendered_words

Figure 3: Visualized results from the masculine-as-norm dictionaries for the MASSIVE data set (Study 2). Note that “neo” stands
for neopronouns, e.g., ze/zir. Also note that neopronouns are difficult to automatically detect; counts may be inaccurate.
lady (n=13), sister (n=12), dad (n=12), king (n=10), papa (n=9), andgirlfriend (n=8). We can consider the influence of the dominate
gender-sexuality model in most societies today, including the US:
cisgendered and heterosexual, or cishet. A view from this lens suggets that these results are not unexpected. Mothers are the primary
caregivers in many families. While not necessarily the case, womenare stereotypically thought to engage in more social dialogue thanmen [65]. We do not have access to the full demographics, including sexualities, of the data set contributors [7]. Nevertheless, thesepatterns speak to a cishet framing of gendered associations in US society that anyone, including women, may unconsciously embed into
dialogues. In contrast, the marked words for ReDial are broader andappeared to be tied to the movie recommendation context: setting
aside pronouns, the most frequent words included actor (n=444),
man (n=365), actors (n=305), guy (n=300), girl (n=229), daughter(n=207), hero (n=197), husband (n=191), and king (n=163). We cantease out subtle biases and sexism embedded in these terms. We
have men and husbands and kings, and we have girls. “Girls” could
be contextualized as colloquial expressions, queer language, or the

CHI ’23, April 23–28, 2023, Hamburg, Germany

Katie Seaborn et al.

Table 9: Summary of results on masculine-as-norm language for the ReDial data set (Study 2)
Dictionary

Dict. Share

Total
Instances

Pronouns

19/49 (38.8%) 9927

Dinan et al. [36] 153/409
marked words (37.4%)

9845

Gendered Words 794/6244
Dataset(12.4%)

38,469Names without 1541/22,609
agent namesa (6.8%)

74,657

PartitionMasc.
Freq.

Fem.
Freq.

Neu.
Freq.

Neo.
Freq.

overalltest
train
overalltest
train
overalltest
train
overall
test
train

1411
105
1306
3154
246
2908
2840
203
2637
24,7481682
13,846

6426
637
5789

112
6
106

3501
347
3154
6691
657
6034
6122
621
5501
49,9095632
44,27732,3473772
28,574
15,5282205
22,243

Other
Freq.

Masc.
Ratio

Fem.
Ratio

Neu.
Ratio

Neo.
Ratio

.353
.317
.305
.68
.728
.675
.159
.135
.150
.669
.574
.552

.142
.096
.126
.32
.272
.325
.074
.044
.072
.208
.171
.172

.647
.582
.559

.011
.005
.010

Other
Ratio

.841
.821
.778
.331
.255
.277

Dict.: Dictionary. Masc.: Masculine. Fem.: Feminine. Neu.: Neutral. Neo.: Neopronoun. Freq.: Frequency. a Specifically: Alexa, Siri, Olly.

Figure 4: Visualized results from the masculine-as-norm dictionaries for the ReDial data set (Study 2). Note that “neo" stands
for neopronouns, e.g., ze/zir. Also note that neopronouns are difficult to automatically detect; counts may be inaccurate.
typical infantizliation of women [75]. “Guy” and potentially “actor/s” and “hero” may be more nuanced; while masculine, they arealso colloquially used in gender neutral ways, depending on thecontext. This is, of course, still an expression of masculine-as-norm
language: masculine associations raise value, while feminine associations decrease value [84]. As for “king,” most references were
to famous restaurants, locations, and people. In short, “feminine”
trends may actually point to a subtle form of masculine bias in linewith masculine-as-norm language.
We then used the Gendered Words Dataset on both data sets.
For MASSIVE, there were more masculine (n=256) than feminine(n=209) marked words, 𝜒2 = 4.75; df = 1; p = .03. However, mostwere neutral (n=2538), as shown against the masculine, 𝜒2 = 1863.8;

df = 1; p < .001, and feminine, 𝜒2 = 1974.6; df = 1; p < .001, distributions. The same was true in ReDial for the masculine (n=6180)
and feminine (n=2875) distributions, 𝜒2 = 1206.3; df = 1; p < .001,
as well as when comparing the neutral (n=32,562) distribution to
the masculine, 𝜒2 = 17965; df = 1; p < .001, and feminine, 𝜒2 =24870; df = 1; p < .001. Looking at the most frequent words aside
from pronouns, we can see similar patterns as with the Dinan etal. [36] dictionary. For MASSIVE, these included amazon (n=33),
tom (n=32), town (n=26), dog (n=25), wife (n=15), mother (n=15),
lady (n=13), sister (n=12), dad (n=12), page (n=11), and king (n=10).
For ReDial, these included tom (n=439), man (n=365), guy (n=300),
girl (n=229), daughter (n=207), husband (n=191), king (n=163), boy(n=152), son (n=143), and woman (n=141). “Tom” was not found in
the analysis of marked words from the Dinan et al. [36] dictionary

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

and seems to refer to user contacts (“Tom” is a common name)and celebrities. We should also wonder why this might be. Why
not “Jane,” as well? Why are certain names (of people or products)
included to a greater degree over others? A more balanced data set
would include a balanced proportion of common names. We take a
deeper look at names next.
5.2.3 Names. The names dictionary indicated gender biases andalso large shares of gender-neutral and gender-ambiguous names.
For MASSIVE, the distribution of masculine (n=3328) and feminine(n=3554) names differed from expectation, favouring the feminine,𝜒2 = 7.4; df = 1; p = .006. Masculine, 𝜒2 = 848; df = 1; p < .001,
and feminine, 𝜒2 = 142; df = 1; p < .001, distributions also differedfrom the gender-neutral or gender-ambiguous names distribution(n=2618), as well. This indicates a feminine bias. For ReDial, the masculine distribution (n=49,909) was greater than the feminine one(n=15,528), 𝜒2 = 18064; df = 1; p < .001, and the neutral/ambiguous
one (n=24,748), 𝜒2 = 8479.8; df = 1; p < .001. However, the feminine distribution was much lower than the neutral/ambiguous
one, 𝜒2 = 2110.6; df = 1; p < .001. This indicates a masculine bias.
Taken together, each data set appears to have a gender bias, withMASSIVE containing more feminine names and ReDial containing
more masculine ones. Yet, each also contains a significant portionof gender neutral or ambiguous names. Many names appear to beatypical as names but typical as other kinds of words. For example,“Great” is a masculine name that is also a very common adjective(MASSIVE, n=19; ReDial, n=14,330). A gender neutral or ambiguous
example is the name “Awesome,” which is also a common word
(ReDial, n=1604). Nevertheless, these words carry gendered andnon-gendered meanings that are also contextual. We can think ofthese names as another layer in our approach to examining theimplicit gendered language that exists in these data sets.

5.3

Discussion

We found an overall implicit cishet masculine bias when contextualizing frequent terms, including pronouns, marked language,
and names. A masculine bias was more pronounced in ReDial than
MASSIVE. ReDial contained more masculine terms overall. However, while MASSIVE contained more feminine words, interrogatingthese words revealed a relationship to cishet norms rather than expressions of femininity in its own right, detached from masculinity.
Pronouns and names indicated a substantial pattern of potential
gender neutrality and gender ambiguity. Nevertheless, it is difficult to tease out whether pronouns are singular or plural, whether
names were meant in gendered ways or not, and whether or not
this matters in the end. These results may reflect recent efforts
to eliminate gender bias by targeting conspicuous patterns likepronouns and name use. We also cannot ignore the presence ofthe other more subtle masculine bias results alongside this more“balanced” patterns. While encouraging, these results highlight that
we still have some way to go in recognizing and accounting for arange of gender biases in NLP data sets.

6

DISCUSSION SYNTHESIS

We were able to detect subtle forms of gender bias in the MASSIVE
and ReDial data sets, identifying feminine and masculine biases, butmore of the latter. Gender neutral and ambiguous language as also

CHI ’23, April 23–28, 2023, Hamburg, Germany

identified as a key feature. Moreover, language related to the VA
context intersected with “gendered” language in important ways.
This led us to change our approach, take on a more critical frame,
and accept that our analyses would be limited by this complexity.
We submit this work as a first start. While challenging and inexact attimes, our approach has set the stage for future efforts on a pressing
issue. In particular, we offer an initial dictionary—the AVA terms—asa springboard for more rigorous analysis. We now discuss the main
challenges and implications that this initial effort has revealed.

6.1

Interrogating Subtle Masculine Centrism(and Other) Biases

Evaluating large NLP data sets using automated procedures is tricky.
Language is dynamic, multi-faceted, and contextual. Slang and colloquialisms, names that are also common words, words that are
gender ambiguous, and the influence of context; these are just someof the challenges we encountered when attempting to make senseof the more implicit forms of gender bias that may exist in thesecrowdsourced data sets. One key issue going forward is distinguishing singular “they/them” pronouns from the plural “they/them,”
ideally in a way that allows for automated content analysis. We
should also be wary of implicit feminine language biases. Whilethe context of these words matters, so does the frequency [39]: themore we come across a word, the more we key into it. Even if,
for example, we use gender debiasing measures to create feminine
associations with masculine words, we should be careful to avoid
introducing a feminine gender bias.

6.2

A Matter of Perspective? Managing
Disagreements among the Dictionaries

Dictionaries created for a particular context are highly dependent:
not only on the context, but also on the language, culture, local community, religion, race/ethnicity, sexuality, and, of course, sex/genderof the people whose data is being used in the creation process, aswell as who the creators are. A dictionary (e.g., for job ads) may not
apply to all data sets (e.g., VA data sets). The dictionary share percentages for MASSIVE and ReDial (Table 2 and 3) show how results
can differ significantly across different dictionaries and data sets.
Due to such incongruities, many dictionary terms may not be foundin a given data set, further limiting the efficacy of such analyses.
Additionally, due to differences in context, dictionary words may
carry unintentional meanings, resulting in lexically or syntacticallyambiguous terms that may not be associated with gender in the
expected ways, or at all. For instance, some “feminine” words, suchas “soft,” “kind,” and “warmth,” were not used in a gendered way in
the MASSIVE dataset. At the same time, we must be wary of how
we gender VAs, including through the language we train them on[44, 100]. These results indicate a need for special dictionaries andword embeddings; indeed, we offer the AVA terms as a starter kit.

6.3

Masculinities and Deep (Un)learning

We approached language as a medium of power that “engenders”
values and ideas. As such, language can also be consciously harnessed to encode desires and possibilities on the cusp of our collective imagination. Considering the results of our both studies,

CHI ’23, April 23–28, 2023, Hamburg, Germany

there appears to be masculine-centric language and masculine-asnorm biases in the MASSIVE and ReDial datasets. We must bearin mind the impact of using such data sets at a large scale acrossa variety of VAs, social robots, and other NLP-based technologies.
We must recognize the power of our imagination when it comesto “reshaping technologies” as “socially agentic” and the symbiotic
relationship we have with such technologies, which potentiallylearn from us [80]. We are already grappling with the problem ofmost VAs defaulting to feminine-sounding voices paired with politeand deprecating behaviour, summarized in the quotation sharedacross the tiles of a UNESCO 2019 report and Bergen’s [13] article
on the history of sexism and cyborgs: “I’d Blush If I Could” [109].
These defaults propagate harmful gender biases by reinforcing thatfeminine-gendered agents should be in subservient roles and tolerate poor treatment. This is not just on the VA side; it is also on theside of the user, who can issue “a blunt voice command like ‘hey’or ‘OK”’ [109, p. 150]. We rely on resources such as open data sets
to train our VAs. In doing so, we encode all of the biases in thesedata sets, blatant and subtle, within our VAs. Recycling existing
gendered biased language is thus an enormous possibility. If thesebiases are not addressed now, they will proliferate.
Harms may be immediate and representational, especiallythrough the embedding of limited and stereotyped representations;they may also be allocational, driving more distant effects that go
beyond language [14]. The implications may only arise or become
clear over time, especially as we continue to take up these devices
at younger and younger ages [38]. For example, speaking with
VAs like Alexa may lead to speech alignment, a behaviour where
speakers adopt the voice and language patterns of other parties
unconsciously [112]. This raises several questions: When Alexa
speaks with our children, what messages do we want them to takein? Do we want to present a limited version of the world, one
wherein masculinity abounds in language use and representation?
Do we want Alexa to speak in gender-limited ways? Do we wish to
encode “Alexa” as feminine and its masculine-voiced counterpart(s)as masculine through gendered language as well as vocal gender
qualities? If children speak in masculine-as-norm ways, how should
VAs respond to them? Do we need to create a new “ideal responses”
data set to tackle this, or is it okay? We do not and cannot prescribe
a one-size-fits all approach. Rather, we raise these provocations forthe community to ponder.
We also identified major limitations that may apply to NLP data
sets for VAs generally. While both data sets were relatively recent,large-scale, and diverse, with MASSIVE offering a range of languages, they were limited in several ways. Both were premised inbasic commands and dialogues representing (stereo)typical VA usage. MASSIVE, for instance, was initiated based on 200 predefined
prompts. This leads to a limited range of content. Both also did notinclude any words related to swearing, politics, sex, drug use, deviancy, sexuality, war, etc. However, people will ask such questions
to VAs. This limits practical and real-life applications. Finally, as
the data was gathered using observational research methods andcrowdsourcing on AMT, just taking part in the studies and being
conscious of being observed and recorded likely changed people’snatural behaviours, biasing their verbal, non-verbal, and/or written
responses [69]. We may need to distinguish between data sets thattrain VAs how to speak and data sets that train VAs on how they

Katie Seaborn et al.

may be spoken to. Although efforts have been made to mitigate
these biases using large and/or outdated corpuses, new challenges
arise when the data sets are limited in terms of diversity or size,
as well as due to the complexity of how gender frameworks areembedded in language and data sets [72]. We may need to develop
more “meta dictionaries” like AVA that map existing corpora togender-sensitive models of VA and user behaviour.

6.4

AVA: Ambiguity in Virtual AssistantLanguage Dictionary

Ambiguity in language is ubiquitous; it is not merely the absenceof clarity but also the presence of different meanings that are disambiguated. During this work, we created AVA, a dictionary of 44
VA-specific ambiguous words that were originally classified as gendered by two implicit gendered language dictionaries but became
ambiguous against the VA contexts of the MASSIVE and ReDial
data sets. Examples include “strong” (“how strong the dollar iscompared to the peso” or “make a strong coffee”), “power” (“I havealways loved the idea of having some sort of power” or “power up
the plug socket”), “soft” (“I have a soft spot for kung fu movie” or “issoft cheese better than hard” or “is it a soft evening tonight”), and“warm” (“is it warm outside” or “to warm my heart”). When used inthe VA context, these words can be associated with a certain gender,
or not. The creation of the AVA dictionary is a first and crucial step
towards identifying and analyzing such ambiguous, multilayered,
implicit gender biases in VA data sets. It follows in the footsteps ofprevious efforts on data statements [12] and model cards [78] byrecognizing (a lack of) diversity in NLP initiatives. Further steps
include enriching the AVA dictionary by adding more VA-related
words representing diverse tasks, commands, and intents, as well
as further investigating if these deemed gendered words remain so
in VA contexts through controlled empirical studies with genderdiverse people. Indeed, contributions from and co-design activities
with genderqueer, non-binary, and trans* folk could help AVA become a truly gender-diverse and -inclusive baseline tool without
the use of sensitive and/or personal data [86]. This will require
more dictionaries and much, much larger data sets. Finally, while
we used AVA to remove ambiguous terms and thereby pinpoint
implicit gendered language, we do not necessarily advocate foruse of AVA as a removal tool. AVA can be used to highlight oreven heighten ambiguity, i.e., by adding AVA terms or swapping
unambiguous synonyms with AVA terms, depending on the goal.

6.5

Limitations and Future Work

Many of our analyses were limited by our chosen methods and materials. We have already discussed the issues with the dictionaries.
The automated content analysis method is not without limitations,too, as it may miss instances of implicit bias, even with appropriate
dictionaries and settings. We attempted to account for this with
our follow-up analyses, i.e., our manual contextual analysis forStudy 1, which led to the creation of AVA. Yet, we acknowledge that
instances of implicit bias could have been missed. More work on
implicit bias, ideally triangulating our strategies and findings, will
be needed. The data sets are relatively small in terms of word count.
This prevented us from carrying out other ML analyses, such as
implicit bias via word embeddings, for which truly “big” data sets

Transcending the “Male Code”: Implicit Masculine Biases in NLP Contextsof millions or billions of words, if not utterances, would be needed.
The data sets also do not represent diverse interactions with VAs,
which, as we have discussed, likely affected our analyses. The AVA
term list is relatively small and tied to the dictionaries and data sets
used in this study; future work will need to expand on it, such as
in focus groups with technologists or using crowdsourcing, boththat include gender-diverse participant groups. We also did notuse n-gram analyses, i.e., using a sequence of terms to predict thelikelihood of the next in the sequence being biased, which can bedone in future work.

7

CONCLUSION

We live in a gendered world that is expressed through and embedded, even in subtle ways, within language. AI-powered agents,
interfaces, and spaces that we can communicate with through natural, verbal forms of language are on the rise. In tandem, an array ofactors, some with great reach and power, are creating and providing
materials to train the natural language abilities of these systems.
This is a social good, with caveats. As this work shows, there are
implicit gender biases in these data sets. Detection is the first step;
next, we need to map out what forms of gendered language should
be represented, with rigour and community engagement. Together,
we can transcend the masculinities status quo.

ACKNOWLEDGMENTS
This work was funded by an Engineering Academy Young Scientist
Encouragement Award (Tokyo Institute of Technology).

REFERENCES[1] Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. Just say no:
Analyzing the stance of neural dialogue generation in offensive contexts. In
Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing (EMNLP 2021), 4846-4862. https://doi.org/10.18653/v1/2021.emnlpmain.397[2] April H. Bailey, Marianne LaFrance, and John F. Dovidio. 2019. Is man themeasure of all things? A social cognitive account of androcentrism. Personality and Social Psychology Review 23, 4, 307-331. https://doi.org/10.1177/
1088868318782848[3] April H. Bailey, Adina Williams, and Andrei Cimpian. 2022. Based on billions
of words on the internet, people = men. Science Advances 8, 13, eabm2463.
https://doi.org/10.1126/sciadv.abm2463[4] Shaowen Bardzell. 2010. Feminist HCI: Taking stock and outlining an agenda fordesign. In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (CHI ’10), 1301-1310. https://doi.org/10.1145/1753326.1753521[5] Shaowen Bardzell and Jeffrey Bardzell. 2011. Towards a feminist HCI methodology: Social science, feminism, and HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11), 675-684. https://doi.org/10.1145/1978942.1979041[6] Jennifer A. Bartz and John E. Lydon. 2004. Close relationships and the working
self-concept: Implicit and explicit effects of priming attachment on agency
and communion. Personality and Social Psychology Bulletin 30, 11, 1389-1401.
https://doi.org/10.1177/0146167204264245[7] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser.
2020. SLURP: A Spoken Language Understanding Resource Package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP-2020). https://doi.org/10.48550/arXiv.2011.13205[8] Simone de Beauvoir. 2011. The Second Sex. Vintage Books, New York, NY.
[9] Rosanna Bellini, Angelika Strohmayer, Ebtisam Alabdulqader, Alex A. Ahmed,
Katta Spiel, Shaowen Bardzell, and Madeline Balaam. 2018. Feminist HCI: Taking
stock, moving forward, and engaging community. In Extended Abstracts of the2018 CHI Conference on Human Factors in Computing Systems (CHI EA ’18), 1-4.
https://doi.org/10.1145/3170427.3185370[10] Sandra L. Bem. 1974. The measurement of psychological androgyny. Journalof Consulting and Clinical Psychology 42, 2, 155-162. https://doi.org/10.1037/
h0036215

CHI ’23, April 23–28, 2023, Hamburg, Germany[11] Sandra L. Bem. 1993. The Lenses of Gender: Transforming the Debate on Sexual
Inequality. Yale University Press.
[12] Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics 6, 587-604.
https://doi.org/10.1162/tacl_a_00041[13] Hilary Bergen. 2016. “I’d blush if I could”: Digital assistants, disembodied cyborgs
and the problem of gender. Word and Text, A Journal of Literary Studies and
Linguistics VI, 01, 95-113.
[14] Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedingsof the 58th Annual Meeting of the Association for Computational Linguistics (ACL2020), 5454-5476. https://doi.org/10.18653/v1/2020.acl-main.485[15] Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller,
Morgan Klaus Scheuerman, Chenhao Tan, and Qian Yang. 2022. Responsible
language technologies: Foreseeing and mitigating harms. In Extended Abstractsof the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA’22), 1-3. https://doi.org/10.1145/3491101.3516502[16] Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media African-American english. In
Proceedings of the 2017 Workshop on Fairness, Accountability, and Transparency
in Machine Learning (FAT/ML 2017). https://doi.org/10.48550/arXiv.1707.00061[17] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama,
and Adam T Kalai. 2016. Man is to computer programmer as woman
is to homemaker? Debiasing word embeddings. In 30th Conference on
Advances in Neural Information Processing Systems (NIPS 2016). Retrieved August 22, 2022 from https://proceedings.neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html[18] Margaret M. Bradley and Peter J. Lang. 1999. Affective norms for English words(ANEW): Instruction manual and affective ratings. Technical report C-1, thecenter for research in psychophysiology . . ..
[19] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st
Conference on Fairness, Accountability and Transparency in Machine Learning
Research (PMLR 2018), 77-91. Retrieved from http://proceedings.mlr.press/v81/
buolamwini18a.html?mod$=$article_inline[20] Yang Trista Cao and Hal Daumé III. 2021. Toward gender-inclusive coreference
resolution: An analysis of gender and bias throughout the machine learning
lifecycle. Computational Linguistics 47, 3, 615-661. https://doi.org/10.1162/coli_
a_00413[21] Tommaso Caselli, Roberto Cibin, Costanza Conforti, Enrique Encinas, and Maurizio Teli. 2021. Guiding principles for participatory design-inspired natural
language processing. In Proceedings of the 1st Workshop on NLP for Positive
Impact (ACL-IJCNLP-NLP4PosImpact 2021), 27-35. https://doi.org/10.18653/v1/
2021.nlp4posimpact-1.4[22] Jiawei Chen, Anbang Xu, Zhe Liu, Yufan Guo, Xiaotong Liu, Yingbei Tong, Rama
Akkiraju, and John M. Carroll. 2020. A general methodology to quantify biases
in natural language data. In Extended Abstracts of the 2020 CHI Conference on
Human Factors in Computing Systems (CHI ’20), 1-9. https://doi.org/10.1145/
3334480.3382949[23] Yan Chen, Christopher Mahoney, Isabella Grasso, Esma Wali, Abigail Matthews,
Thomas Middleton, Mariama Njie, and Jeanna Matthews. 2021. Gender bias andunder-representation in natural language processing across human languages.
In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES’21), 24-34. https://doi.org/10.1145/3461702.3462530[24] Sapna Cheryan and Hazel Rose Markus. 2020. Masculine defaults: Identifying
and mitigating hidden cultural biases. Psychological Review 127, 1022-1052.
https://doi.org/10.1037/rev0000209[25] Shruthi Sai Chivukula and Colin M. Gray. 2020. Bardzell’s “Feminist HCI” legacy:
Analyzing citational patterns. In Extended Abstracts of the 2020 CHI Conferenceon Human Factors in Computing Systems (CHI EA ’20), 1-8. https://doi.org/10.
1145/3334480.3382936[26] K. R. Chowdhary. 2020. Natural Language Processing. In Fundamentals of Artificial Intelligence, K.R. Chowdhary (ed.). Springer India, New Delhi, 603-649.
https://doi.org/10.1007/978-81-322-3972-7_19[27] Davide Cirillo, Hila Gonen, Enrico Santus, Alfonso Valencia, Marta R. Costajussà, and Marta Villegas. 2022. Chapter 6 - Sex and gender bias in natural
language processing. In Sex and Gender Bias in Technology and Artificial Intelligence, Davide Cirillo, Silvina Catuara-Solarz and Emre Guney (eds.). Academic
Press, 113-132. https://doi.org/10.1016/B978-0-12-821392-6.00009-1[28] Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl,
Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards,
and Benjamin R Cowan. 2019. The state of speech in HCI: Trends, themes andchallenges. Interacting with Computers 31, 4, 349-371. https://doi.org/10.1093/
iwc/iwz016[29] Cynthia Cockburn. 2003. The circuit of technology: Gender, identity and power.
In Consuming Technologies: Media and Information in Domestic Spaces, Eric
Hirsch and Roger Silverstone (eds.). Routledge, 33-42.

CHI ’23, April 23–28, 2023, Hamburg, Germany[30] Anne Constantinople. 2005. “Masculinity-femininity: An exception to a famous
dictum?” Feminism & Psychology 15, 4, 385-407. https://doi.org/10.1177/0959353505057611[31] Jenna Cryan, Shiliang Tang, Xinyi Zhang, Miriam Metzger, Haitao Zheng, and
Ben Y. Zhao. 2020. Detecting gender stereotypes: Lexicon vs. Supervised learning methods. In Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems (CHI ’20), 1-11. https://doi.org/10.1145/3313831.3376488[32] Jenny L. Davis. 2019. Refusing (mis)recognition: Navigating multiple marginalization in the U.S. Two Spirit movement. Review of International American Studies
12, 1, 65-86.
[33] Simon De Deyne, Danielle J. Navarro, Amy Perfors, Marc Brysbaert, and Gert
Storms. 2019. The “Small World of Words” English word association norms
for over 12,000 cue words. Behavior Research Methods 51, 3, 987-1006. https://doi.org/10.3758/s13428-018-1115-7[34] Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M.
Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges
in non-binary representation in language technologies. In Proceedings of the2021 Conference on Empirical Methods in Natural Language Processing (EMNLP
2021), 1968-1994. https://doi.org/10.18653/v1/2021.emnlp-main.150[35] Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle.
2018. Addressing age-related bias in sentiment analysis. In Proceedings of the2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 1-14.
https://doi.org/10.1145/3173574.3173986[36] Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina
Williams. 2020. Multi-dimensional gender bias classification. In Proceedings of the2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)(EMNLP 2020), 314-331. https://doi.org/10.18653/v1/2020.emnlp-main.23[37] Jad Doughman, Wael Khreich, Maya El Gharib, Maha Wiss, and Zahraa Berjawi.
2021. Gender bias in text: Origin, taxonomy, and implications. In Proceedings ofthe 3rd Workshop on Gender Bias in Natural Language Processing (ACL-GeBNLPIJCNLP 2021), 34-44. https://doi.org/10.18653/v1/2021.gebnlp-1.5[38] Y. Du, K. Zhang, S. Ramabadran, and Y. Liu. 2021. “Alexa, what is that sound?”A video analysis of child-agent communication from two Amazon Alexa games.
In Proceedings of the 2021 International Conference on Interaction Design andChildren (IDC ’21), 513-520. https://doi.org/10.1145/3459990.3465195[39] Nick C. Ellis. 2002. Frequency effects in language processing: A review with
implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition 24, 2, 143-188. https://doi.org/10.1017/
S0272263102002024[40] Norman Fairclough. 1989. Language and Power. Longman, Essex, UK.
[41] Norman Fairclough. 2013. Critical Discourse Analysis: The Critical Study of Language. Routledge.
[42] Tracie Farrell, Miriam Fernandez, Jakub Novotny, and Harith Alani. 2019. Exploring misogyny across the manosphere in Reddit. In Proceedings of the 10th
ACM Conference on Web Science (WebSci ’19), 87-96. https://doi.org/10.1145/
3292522.3326045[43] Christiane Fellbaum. 2005. WordNet and wordnets. In Encyclopedia of Language
and Linguistics (2nd ed.), Keith Brown (ed.). Elsevier, Oxford, 665-670.
[44] Leah Fessler. 2017. We tested bots like Siri and Alexa to see who would
stand up to sexual harassment. Quartz. Retrieved November 21, 2019 fromhttps://qz.com/911681/we-tested-apples-siri-amazon-echos-alexa-microsoftscortana-and-googles-google-home-to-see-which-personal-assistant-botsstand-up-for-themselves-in-the-face-of-sexual-harassment/[45] Anjalie Field and Yulia Tsvetkov. 2020. Unsupervised discovery of implicit
gender bias. In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP) (EMNLP 2020), 596-608. https://doi.org/
10.18653/v1/2020.emnlp-main.44[46] Sophie Freud. 1994. The social construction of gender. Journal of Adult Development 1, 1, 37-45. https://doi.org/10.1007/BF02252981[47] Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and
L. Alfonso Ureña-López. 2021. A survey on bias in deep NLP. Applied Sciences
11, 7, 3184. https://doi.org/10.3390/app11073184[48] Danielle Gaucher, Justin Friesen, and Aaron C. Kay. 2011. Evidence that genderedwording in job advertisements exists and sustains gender inequality. Journalof Personality and Social Psychology 101, 1, 109-128. https://doi.org/10.1037/
a0022530[49] Bhavya Ghai, Md Naimul Hoque, and Klaus Mueller. 2021. WordBias: An interactive visual tool for discovering intersectional biases encoded in word embeddings. In Extended Abstracts of the 2021 CHI Conference on Human Factors in
Computing Systems (CHI ’21), 1-7. https://doi.org/10.1145/3411763.3451587[50] Peter Glick and Susan T. Fiske. 2018. The Ambivalent Sexism Inventory: Differentiating hostile and benevolent sexism. In Social Cognition. Routledge.
[51] Alessandra Gomes, Dennys Antonialli, and Thiago Oliva. 2019. Dragqueens and Artificial Intelligence: should computers decide what is‘toxic’ on the internet? InternetLab. Retrieved August 11, 2022 fromhttps://internetlab.org.br/en/news/drag-queens-and-artificial-intelligenceshould-computers-decide-what-is-toxic-on-the-internet/[52] Anthony G. Greenwald and Linda Hamilton Krieger. 2006. Implicit bias: Scientific foundations. California Law Review 94, 4, 945-967. https://doi.org/10.2307/

Katie Seaborn et al.

20439056[53] Justin Grimmer and Brandon M. Stewart. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis
21, 3, 267-297. https://doi.org/10.1093/pan/mps028[54] Mykol C. Hamilton. 1991. Masculine bias in the attribution of personhood:
People= male, male= people. Psychology of Women Quarterly 15, 3, 393-402.
https://doi.org/10.1111/j.1471-6402.1991.tb00415.x[55] Carol Harrington. 2021. What is “toxic masculinity” and why does it matter?
Men and Masculinities 24, 2, 345-352. https://doi.org/10.1177/1097184X20943254[56] Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard
reliability measure for coding data. Communication Methods and Measures 1, 1,
77-89. https://doi.org/10.1080/19312450709336664[57] Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa
Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias
Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust,and Anders Søgaard. 2022. Challenges and strategies in cross-cultural NLP. In
Proceedings of the 60th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers) (ACL 2022), 6997-7013. https://doi.org/10.
18653/v1/2022.acl-long.482[58] Curt Hoffman and Nancy Hurst. 1990. Gender stereotypes: Perception or
rationalization? Journal of Personality and Social Psychology 58, 2, 197-208.
https://doi.org/10.1037/0022-3514.58.2.197[59] Matthew B. Hoy. 2018. Alexa, Siri, Cortana, and more: An introduction to voiceassistants. Medical Reference Services Quarterly 37, 1, 81-88. https://doi.org/10.
1080/02763869.2018.1404391[60] Hsiu-Fang Hsieh and Sarah E. Shannon. 2005. Three approaches to qualitative
content analysis. Qualitative Health Research 15, 9, 1277-1288. https://doi.org/10.
1177/1049732305276687[61] Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding andmitigating worker biases in the crowdsourced collection of subjective judgments.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19), 1-12. https://doi.org/10.1145/3290605.3300637[62] Janet Shibley Hyde, Rebecca S. Bigler, Daphna Joel, Charlotte Chucky Tate,and Sari M. van Anders. 2019. The future of sex and gender in psychology:
Five challenges to the gender binary. American Psychologist 74, 2, 171-193.
https://doi.org/10.1037/amp0000307[63] Samantha Jaroszewski, Danielle Lottridge, Oliver L. Haimson, and Katie Quehl.
2018. “Genderfluid” or “attack helicopter”: Responsible HCI research practicewith non-binary gender variation in online communities. In Proceedings of the2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 1-15.
https://doi.org/10.1145/3173574.3173881[64] Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP
world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), 6282-6293. https://doi.org/10.18653/v1/2020.aclmain.560[65] Abby Kaplan. 2016. Women Talk More than Men ... and Other Myths about
Language Explained. Cambridge University Press, Cambridge, UK.
[66] Os Keyes, Chandler May, and Annabelle Carrell. 2021. You keep using that word:
Ways of thinking about gender in computing research. Proceedings of the ACM
on Human-Computer Interaction 5, CSCW1, 39:1-39:23. https://doi.org/10.1145/
3449113[67] Scott Kiesling. 2007. Men, masculinities, and language. Language and Linguistics
Compass 1, 6, 653-673. https://doi.org/10.1111/j.1749-818X.2007.00035.x[68] Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion
Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. 2020.
Racial disparities in automated speech recognition. Proceedings of the National
Academy of Sciences 117, 14, 7684-7689. https://doi.org/10.1073/pnas.1915768117[69] Ivar Krumpal. 2013. Determinants of social desirability bias in sensitive surveys:
A literature review. Quality & Quantity 47, 4, 2025-2047. https://doi.org/10.1007/
s11135-011-9640-9[70] Elizabeth Landau. Tech confronts Its use of the labels ‘master’ and ‘slave.’ Wired.
Retrieved September 2, 2022 from https://www.wired.com/story/tech-confrontsuse-labels-master-slave/[71] Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender. In Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022), 1221-1232. Retrieved November 23, 2022
from https://aclanthology.org/2022.coling-1.105[72] Susan Leavy, Gerardine Meaney, Karen Wade, and Derek Greene. 2020. Mitigating gender bias in machine learning data sets. In Bias and Social Aspects in
Search and Recommendation (BIAS 2020), 12-26. https://doi.org/10.1007/978-3030-52485-2_2[73] Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski,
Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommendations. In Advances in Neural Information Processing Systems (NeurIPS’18). Retrieved August 11, 2022 from https://papers.nips.cc/paper/2018/hash/Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts

800de15c79c8d840f4e78d3af937d4d4-Abstract.html[74] Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus
Opwis, and Katharina Reinecke. 2021. How WEIRD is CHI? In Proceedings ofthe 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), 1-14.
https://doi.org/10.1145/3411764.3445488[75] Heather Jean Macarthur. 2015. When Women are Called “Girls”: The Effect ofInfantilizing Labels on Women’s Self-perceptions. Penn State University, PA,
US. Retrieved from https://etda.libraries.psu.edu/catalog/25797[76] Kate Manne. 2018. Down Girl: The Logic of Misogyny. Oxford University Press.
[77] Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black
is to criminal as caucasian is to police: Detecting and removing multiclass bias
in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers) (NAACL-HLT 2019), 615-621.
https://doi.org/10.18653/v1/N19-1062[78] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.
2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229. https://doi.org/10.1145/3287560.
3287596[79] Christine Murad, Cosmin Munteanu, Benjamin R. Cowan, Leigh Clark, Martin
Porcheron, Heloisa Candello, Stephan Schlögl, Matthew P. Aylett, Jaisie Sin,
Robert J. Moore, Grace Hughes, and Andrew Ku. 2021. Let’s talk about CUIs:
Putting conversational user interface design into practice. In Extended Abstractsof the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA’21), 1-6. https://doi.org/10.1145/3411763.3441336[80] Gina Neff and Peter Nagy. 2016. Talking to bots: Symbiotic agency and the caseof Tay. International Journal of Communication 10, 0, 17.
[81] Ihudiya Finda Ogbonnaya-Ogburu, Angela D.R. Smith, Alexandra To, andKentaro Toyama. 2020. Critical race theory for HCI. In Proceedings of the2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 1-16.
https://doi.org/10.1145/3313831.3376392[82] Sven H. Pedersen and Leif A. Strömwall. 2013. Victim blame, sexism and Justworld beliefs: A cross-cultural comparison. Psychiatry, Psychology and Law 20,
6, 932-941. https://doi.org/10.1080/13218719.2013.770715[83] Davor Petreski and Ibrahim C. Hashim. 2022. Word embeddings are biased. But
whose bias are they reflecting? AI & SOCIETY. https://doi.org/10.1007/s00146022-01443-w[84] Liz Plank. 2019. For the Love of Men: From Toxic to a More Mindful Masculinity.
St. Martin’s Press.
[85] Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro,
Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou.
2021. The evolution of the manosphere across the web. In Proceedings of theInternational AAAI Conference on Web and Social Media (ICWSM 2021), 196-207.
Retrieved August 8, 2022 from https://ojs.aaai.org/index.php/ICWSM/article/
view/18053[86] Cami Rincón, Os Keyes, and Corinne Cath. 2021. Speaking from experience:
Trans/non-binary requirements for voice-activated AI. Proceedings of the ACM
on Human-Computer Interaction 5, CSCW1, 132:1-132:27. https://doi.org/10.1145/
3449206[87] Damon C. Roberts and Stephen M. Utych. 2020. Linking gender, language, andpartisanship: Developing a database of masculine and feminine words. Political
Research Quarterly 73, 1, 40-50. https://doi.org/10.1177/1065912919874883[88] Jennifer A. Rode. 2011. Reflexivity in digital anthropology. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (CHI ’11), 123-132.
https://doi.org/10.1145/1978942.1978961[89] Laurie A. Rudman and Stephen E. Kilianski. 2000. Implicit and explicit attitudes
toward female authority. Personality and Social Psychology Bulletin 26, 11, 13151328. https://doi.org/10.1177/0146167200263001[90] Niloofar Safi Samghabadi, Parth Patwa, Srinivas PYKL, Prerana Mukherjee,
Amitava Das, and Thamar Solorio. 2020. Aggression and misogyny detection
using BERT: A multi-task approach. In Proceedings of the Second Workshop on
Trolling, Aggression and Cyberbullying (LREC 2020), 126-131. Retrieved August30, 2022 from https://aclanthology.org/2020.trac-1.20[91] Londa Schiebinger. 2000. Has feminism changed science? Signs: Journal of
Women in Culture and Society 25, 4, 1171-1175. https://doi.org/10.1086/495540[92] Ari Schlesinger, Kenton P. O’Hara, and Alex S. Taylor. 2018. Let’s talk about
race: Identity, chatbots, and AI. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems (CHI ’18), 1-14. https://doi.org/10.1145/
3173574.3173889[93] Stephen A. Schullo and Burton L. Alperson. 1984. Interpersonal phenomenology
as a function of sexual orientation, sex, sentiment, and trait categories in longterm dyadic relationships. Journal of Personality and Social Psychology 47, 5,

CHI ’23, April 23–28, 2023, Hamburg, Germany

983-1002. https://doi.org/10.1037/0022-3514.47.5.983[94] Katie Seaborn and Alexa Frank. 2022. What pronouns for Pepper? A criticalreview of gender/ing in research. In Proceedings of the 2022 CHI Conference on
Human Factors in Computing Systems (CHI ’22), 1-15. https://doi.org/10.1145/
3491102.3501996[95] Katie Seaborn, Norihisa Paul Miyake, Peter Pennefather, and Mihoko OtakeMatsuura. 2022. Voice in human-agent interaction: A survey. ACM Computing
Surveys (CSUR) 54, 4, Article No. 81. https://doi.org/10.1145/3386867[96] James A. Serpell. 2000. Domestication and history of the cat. In The Domestic
Cat: The Biology of its Behaviour (2nd ed.), Dennis C. Turner and Patrick Bateson(eds.). Cambridge University Press, Cambridge, UK, 180-192.
[97] Dale Spender. 1985. Man Made Language. Routledge.
[98] Katta Spiel, Os Keyes, and Pınar Barlas. 2019. Patching gender: Non-binary
utopias in HCI. In Extended Abstracts of the 2019 CHI Conference on Human
Factors in Computing Systems (CHI EA ’19), 1-11. https://doi.org/10.1145/3290607.
3310425[99] Damiano Spina, Johanne R. Trippas, Paul Thomas, Hideo Joho, Katriina Byström,
Leigh Clark, Nick Craswell, Mary Czerwinski, David Elsweiler, Alexander
Frummet, Souvick Ghosh, Johannes Kiesel, Irene Lopatovska, Daniel McDuff,
Selina Meyer, Ahmed Mourad, Paul Owoicho, Sachin Pathiyan Cherumanal,
Daniel Russell, and Laurianne Sitbon. 2021. Report on the future conversations workshop at CHIIR 2021. ACM SIGIR Forum 55, 1, 6:1-6:22. https://doi.org/10.1145/3476415.3476421[100] Yolande Strengers and Jenny Kennedy. 2021. The Smart Wife: Why Siri, Alexa,
and Other Smart Home Devices Need a Feminist Reboot. MIT Press.
[101] Yolande Strengers, Lizhen Qu, Qiongkai Xu, and Jarrod Knibbe. 2020. Adhering,steering, and queering: Treatment of gender in natural language generation. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20), 1-14. https://doi.org/10.1145/3313831.3376315[102] Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao,
Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019.
Mitigating gender bias in natural language processing: Literature review. In
Proceedings of the 57th Annual Meeting of the Association for Computational
Linguistics (ACL 2019), 1630-1640. https://doi.org/10.18653/v1/P19-1159[103] Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions.
In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing(EthNLP 2017), 53-59. https://doi.org/10.18653/v1/W17-1606[104] Margaret Urban and Stephen Mailey. 2019. Conversation Design: Principles,
Strategies, and Practical Application. In Extended Abstracts of the 2019 CHI
Conference on Human Factors in Computing Systems, 1-3. https://doi.org/10.1145/
3290607.3298821[105] Sarah Theres Völkel, Daniel Buschek, Malin Eiband, Benjamin R. Cowan, andHeinrich Hussmann. 2021. Eliciting and analysing users’ envisioned dialogues
with perfect voice assistants. In Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems (CHI ’21), 1-15. https://doi.org/10.1145/3411764.
3445536[106] Mihaela Vorvoreanu, Lingyi Zhang, Yun-Han Huang, Claudia Hilderbrand, Zoe
Steine-Hanson, and Margaret Burnett. 2019. From gender biases to genderinclusive design: An empirical investigation. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (CHI ’19), 1-14. https://doi.
org/10.1145/3290605.3300283[107] Judy Wajcman. 1991. Technology as masculine culture. In Feminism confronts
technology. Penn State University Press, Pennsylvania.
[108] Kasper Welbers, Wouter van Atteveldt, and Kenneth Benoit. 2017. Text analysis
in R. Communication Methods and Measures 11, 4, 245-265. https://doi.org/10.
1080/19312458.2017.1387238[109] Mark West, Rebecca Kraut, and Chew Han Ei. 2019. I’d blush if I could: Closing
gender divides in digital skills through education. EQUALS, France. Retrieved
from https://unesdoc.unesco.org/ark:/48223/pf0000367416.page$=$1[110] Makeba Parramore Wilbourn and Daniel W. Kee. 2010. Henry the nurse is a
doctor too: Implicitly examining children’s gender stereotypes for male and female occupational roles. Sex Roles 62, 9, 670-683. https://doi.org/10.1007/s11199010-9773-7[111] Steven Wilson, Rada Mihalcea, Ryan Boyd, and James Pennebaker. 2016. Disentangling topic models: A cross-cultural analysis of personal values throughwords. In Proceedings of the First Workshop on NLP and Computational Social
Science (NLP+CSS ’16), 143-152. https://doi.org/10.18653/v1/W16-5619[112] Georgia Zellou, Michelle Cohn, and Bruno Ferenc Segedin. 2021. Age- andgender-related differences in speech alignment toward humans and voiceAI. Frontiers in Communication 5. Retrieved May 4, 2022 from https://www.
frontiersin.org/article/10.3389/fcomm.2020.600361