From Showgirls to Performers: Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs (2024)

Abstract: Gender bias is not only prevalent in Large Language Models (LLMs) and their training data, but also firmly ingrained into the structural aspects of language itself. Therefore, adapting linguistic structures within LLM training data to promote gender-inclusivity can make gender representations within the model more inclusive. The focus of our work are genderexclusive affixes in English, such as in showgirl or man-cave, which can perpetuate gender stereotypes and binary conceptions of gender. We use an LLM training dataset to compile a catalogue of 692 gender-exclusive terms along with gender-neutral variants and from this, develop a gender-inclusive fine-tuning dataset, the Tiny Heap. Fine-tuning three different LLMs with this dataset, we observe an overall reduction in gender-stereotyping tendencies across the models. Our approach provides a practical method for enhancing gender inclusivity in LLM training data and contributes to incorporating queer-feminist linguistic activism in bias mitigation research in NLP.

heading: 1 Introduction
text: Large language models (LLMs) have become ubiquitous in Natural Language Processing (NLP) due to their impressive capabilities in a variety of tasks. However, they also carry risks arising from social biases incorporated into models from the training data (Bender et al., 2021). Well-documented among these are harmful gender biases such as reliance on stereotypes and erasure of non-binary gender identities (Cao and Daumé, 2021; Ovalle et al., 2023, a.o.). Structural aspects of language itself and linguistic norms can reflect as well as shape societal concepts of gender (Pauwels, 2003; Whorf and Carroll, 1956). Within the context of LLMs, encoded representations of gender inform language generation and classification decisions, thereby having the potential to influence societal concepts of gender (Bommasani et al., 2022). It is
vital therefore, to ensure that LLMs are evaluated and trained to minimize gender bias and promote equitable representation of all genders.
In English, linguistic structures have a long history of reinforcing traditional gender roles and the concept of male gender as the default (Mills, 2012). Examples include the use of man to mean all humans, the indication of women’s marital status in terms of address (Miss, Mrs., Ms.), or the marking of deviation from gendered norms (male nurse, girl boss). Sexist and gender-exclusive linguistic constructions have been discouraged in official style guides (APA, 2020) and their use has been in decline (Baker, 2010b). However, the nature of language change is slow, with new and traditional variations existing simultaneously. Given the scale of LLM training data (Bender et al., 2021) and the disproportionate representation of men within textual data (Baker, 2010a), language models have the potential to proliferate and reinforce stereotypical and traditional views of gender.
Approaches to mitigating bias in LLMs have included fine-tuning with gender-inclusive language (Thakur et al., 2023). Data interventions with gender-inclusive text aim to reduce the use of binary gender terms in cases where gender is irrelevant (for example, a chairman and chairwoman do the same job) and thereby allow for association of a term with all genders (chairperson). However, the replacement of sexist and genderexclusive terminology often relies on limited lists of gender-neutral terms (Ghanbarzadeh et al., 2023; Thakur et al., 2023), and often focuses on professions (Fatemi et al., 2023). Additionally, previous works on fine-tuning LLMs with gender-inclusive data have primarily carried out experiments with masked language models such as BERT (Devlin et al., 2019) and its derivatives (Vashishtha et al., 2023).
In this research, we focused on expanding the coverage of gender-exclusive terminology and
280
experimented with fine-tuning both causal and masked LLMs. We first exploited structural elements of English that relate to gender discrimination and exclusion in order to generate a larger catalogue of words that are unnecessarily gendered along with gender neutral alternatives. We extracted nouns with gender-marking prefixes and suffixes from a common training corpus, OpenWebText2 (Gao et al., 2020), which was used to train LLMs like Meta’s Llama2 (Thakur et al., 2023) and Microsoft’s MT-NLG (Smith et al., 2022). The distribution of extracted gender-marking nouns demonstrated clear androcentric tendencies within the corpus. We compiled gender-neutral variants for each term with a gender-marking affix to form a catalogue of 692 term pairs. This resource is just over three times larger than the size of previously available resources and could be used in assessments of gender skew within LLM training corpora as well as in the replacement gender-exclusive terminology. We also developed a small-scale, multi-domain fine-tuning corpus, using our catalogue to replace gender-exclusive with genderneutral words. We also employed the NeuTral Rewriter (Vanmassenhove et al., 2021) to replace gendered pronouns (he, she, himself etc.) with singular they. The resulting corpus was used to fine-tune three different (masked and causal) LLMs. The results of this process of fine-tuning with gender inclusive terminology demonstrated an overall tendency towards reduction in gender-stereotyping exhibited by the models as well as a reduction in the generation of harmful language in gendered contexts.
Contributions
• We show clear androcentric tendencies within a commonly used LLM training corpus. • We construct a catalogue of 692 term pairs, consisting of a gender-exclusive terms and neutral alternatives, which we release for public use1. • We show that automatically generated genderinclusive English is effective in reducing gender stereotyping in LLMs through finetuning2.
1https://github.com/marionbartl/affixed_words 2https://github.com/marionbartl/performers

heading: 2 Bias Statement
text: The focus of this work is gender-inclusive language, and its counterpart, sexist language. Sexist language, following Frye’s (1983) definition of sexism, can be defined as language that clearly divides between two genders, in which one gender (masculine) is treated as hierarchically superior to the other (feminine). This superiority is expressed, for example, through the generic use of masculine gendered expressions (e.g. use of terms such as mankind, chairman to refer to people of any gender).
Our work is based on the assumption that sexist language in training data is one of the sources of gender bias in LLMs. Specifically, we would expect models to favor masculine expressions over gender-neutral alternatives, creating a representational harm for people of non-masculine gender (Blodgett et al., 2020). Sexist expressions additionally reinforce traditional gender roles (e.g. male nurse), therefore we would also expect models to favor gender-stereotypical expressions. Moreover, since sexist language is based on a binary model of gender, we expect models to default to this. This can lead to misrepresentation and erasure of non-binary genders in downstream applications, creating allocational and representational harms for non-binary users of these systems (Blodgett et al., 2020). Not adjusting LLMs to accurately represent the variety of genders that exist in society will contribute to the ongoing marginalization of people identifying as gender-queer (Ovalle et al., 2023).

heading: 3 Related Work
text: Large Language Models (LLMs) have been shown to encode a variety of social biases contained in their training data (Gupta et al., 2023; Salinas et al., 2023), among them gender bias (Stanczak and Augenstein, 2021). Due to the current prevalence of transfer learning in NLP, in which a pretrained model is fine-tuned with task-specific data, transfer learning has recently also been adapted by works that aimed to reduce gender bias in LLMs (Lauscher et al., 2021; Ghanbarzadeh et al., 2023). In this approach, an LLM is fine-tuned with data that has undergone interventions to increase gender fairness. This approach is supported by the finding that biases in fine-tuning data have a greater influence on downstream model behavior than biases in the pre-training data (Steed et al., 2022). Previous interventions to fine-tuning data
include Counterfactual Data Augmentation (CDA), in which masculine and feminine pronouns and gendered nouns are swapped for the respective other (Ghanbarzadeh et al., 2023; Vashishtha et al., 2023; Fatemi et al., 2023). Another intervention replaces gendered words for gender-neutral words (fire fighter for fireman) or phrases containing both masculine and feminine genders (he and she for he; Thakur et al., 2023). This kind of intervention is not new: it rests upon a longstanding tradition of research and advocacy the field of feminist linguistics, which has been promoting changes in the lexicon to reduce gender stereotyping and masculine-default language since the 1970s (Kramer, 2016; Mills, 2012; Lakoff, 1973). More recently such changes to the language, also called feminist language reform, have incorporated ways of adapting language to include non-binary and trans gender identities, such as the third person singular (neo)pronouns (they, xe, ze, etc.). The usage and possible modelling of this extended lexicon of pronouns within the context of NLP was analyzed by Lauscher et al. (2022). Lund et al. (2023) also showed that training on data containing singular they can reduce gender bias in grammatical error correction. Furthermore, Vanmassenhove et al. (2021) and Sun et al. (2021) developed rule-based and neutral machine translation-based models to modify English text to render it genderneutral. Vanmassenhove et al.’s (2021) NeuTral Rewriter replaces gendered pronouns with singular they and a list of gendered nouns with neutral variants. However, while the amount of NLP research incorporating and exploring strategies of feminist language reform has grown, the queer-feminist linguistic research it is based on is, with some exceptions (Devinney et al., 2022; Piergentili et al., 2023a; Seaborn et al., 2023), rarely acknowledged and even less often informs the research itself.

heading: 4 Method
text: Gender bias in the English language is reflected in features such as masculine generics and is captured in datasets through, for example, skewed distributions of pronouns and profession words in the same context. However, it is also contained in structural elements of the language itself, such as gendermarking affixes. The most frequent are suffixes such as -man in spokesman, but gender can also be marked with a prefix, such as in man-power or girlboss. Words marked with masculine suffixes
have traditionally been used in a generic sense (e.g. Madam Chairman), however, with the emergence of feminist language reform, style guides have advised against their use (Piergentili et al., 2023b). In English, the most common replacement strategy for gendered generics is neutralisation (chairperson), because all gender identities, not just male and female, can be referred to by gender-neutral nouns. In NLP, research using gender-neutral language in the context of English LLMs has mainly relied on lists of common gender-neutral replacements (Vanmassenhove et al., 2021; Thakur et al., 2023), without taking structural processes such as affixation into account in order to broaden the coverage of these lists.
In this section we first outline the process of extracting unnecessarily gendered words based on gender-marking affixes (§4.1). We then describe the gender-neutralizing interventions to our finetuning data (§4.2) as well as the models (§4.3) and bias measurements used (§4.4).

heading: 4.1 Word Catalogue
text: We extracted words with the suffixes -man, - manship, -woman, -womanship, -boy, -girl and words with the prefixes man-3, woman-, boy- and girl-. We used a 200 million token random sub-
3Words with man- prefixes were only included if they also had the dash (-) following man, because otherwise the false positive rate (manager, mandate, etc.) would have been too high.
section of the OpenWebText2 corpus (Gao et al., 2020) for extraction. The words were extracted using regular expressions within Python. We additionally filtered the words to include only English singular nouns. We only filtered for singular nouns to reduce the amount of redundant extractions, and to simplify the dictionary verification later on. Plurals for all verified words were added after the third round of verification.
The first round of verification of extracted affixed terms generally followed a human-in-the-loop approach, meaning that after 20 files, each 1MB in size, the extracted words were manually checked for validity. This eliminated a variety of false positives such as words in which affixes did not denote gender (german, ramen), spelling errors (camerman, sopkesman), surnames (zimmerman), and other word creations (heythereman, mrfredman). In total, 663 words were extracted in the first round (ref. Table 1).
After extraction, the terms were verified in the second round using the API of the BabelNet encyclopedic dictionary (Navigli and Ponzetto, 2012). BabelNet was chosen due to its broad coverage of lexical resources; its search engine combines entries from WordNet, Wikidata and Wikipedia among others. Terms that did not return an entry in BabelNet were disregarded in order to eliminate less established terms, slang and sexually charged terminology. If a term contained a dash, such as in man-bun, but could not be found in BabelNet, we also searched for the term with a space instead of the dash to not disregard terms due to spelling differences. Table 2 shows the top ten words containing the four simple gender-marking suffixes and their frequency. The highest frequent words with
gendered prefixes, and words with -wo/manship suffixes are shown in Table 6 and 7 in the Appendix, respectively.
Following the BabelNet verification, words were manually filtered in the third round to exclude words not related to gender (e.g. boycott, boyne), and proper names such as surnames or words related to pop culture (batgirl, rainman). Furthermore, terms that occurred with a feminine suffix (noblewoman) but did not have a masculine equivalent (nobleman) were added as their masculine variant to the list, because we treat gender-marking suffixes as exchangeable to mark a different gender. The third round left 353 singular affixed nouns.

heading: 4.1.1 Gender-neutral variants
text: Gender-neutral variants were manually compiled for all extracted words with gender-marking affixes. A single variant was added for all items in the list to simplify the replacement process. The final genderneutral variants were discussed and agreed upon by the researchers. The proposed replacements are not intended to be definitive substitutes for their gender-marked counterparts. Instead, they were developed for the present experiments to provide gender-neutral terms, as no official list exists.
Suffixes Some gender-marking suffix could simply be exchanged for one that is gender neutral, such as in the common neutralisation of chairman/-woman to chairperson. However, this simple replacement does not always work. For example, some frequent terms already have gender-neutral replacements such as fire fighter for fireman or police officer for policeman. In these cases, *fireperson or
*policeperson would be ungrammatical4. A similar case can be made for less frequent words for which more elegant solutions are available than simply replacing -man/-woman with -person. One approach is to find more fitting suffixes or compound nouns, such as in the neutralisation of crewman with crew member. Another approach is to replace a word with a gender-neutral synonym, such as in the replacement of hitman with assassin. A third approach applies to words containing a verb as their root, such as the word huntsman, which has the root hunt. Here, the word can be replaced by a nominalization: hunter.
Prefixes In the case of words with gendermarking prefixes, gender-neutral variants can be constructed by removing the prefix. For example, the word man-crush can be neutralised to crush.
Once the list of singular word pairs was fixed, the plural version of every word-pair was added to the final list. The plurals were obtained using the inflect library in Python (version 7.0.0). After adding plurals, we performed one last round of manual verification to ensure all plurals were formed correctly. The final list contains 692 term pairs. For comparison, Vanmassenhove et al. (2021) used a list of 91 term pairs. A sample of our final list can be found in Table 8 in the Appendix.

heading: 4.2 Fine-Tuning Data
text: To create a fine-tuning corpus with genderneutral interventions, we assembled a base corpus, which needed to have several features: (1) The configuration should be similar to current LLM pre-training data, meaning that it should contain a diverse set of sources. However, we excluded data that was too domain-specific, such as code
4As per linguistic convention we mark ungrammatical terms with a leading asterisk (*).
and scientific publications in order to demonstrate methodology for general-purpose English. In the same line of reasoning, (2) the corpus should only contain English data, because the focus of this work is English, and the NeuTral Rewriter (Vanmassenhove et al., 2021), which replaces gendered pronouns with singular they does also only exist for English. (3) Finally, since we do not aim to worsen the performance of the LLM through fine-tuning, the corpus should only include high-quality text.
The final composition of our base corpus was inspired by the composition of GPT-3’s training data (Brown et al., 2020) as well as The Pile corpus (Gao et al., 2020) and is shown in Table 3. Our original download has a size of 250 million tokens, which is approximately 1.5 GB of data. Since this is substantially smaller than The Pile (825GB), we called our dataset The Heap. The dataset was downloaded using the Huggingface datasets library (version 1.18.3; Wolf et al., 2020) and tokenized with the stanza library (version 1.7.0; Qi et al., 2020).
The fine-tuning data were adjusted for genderneutral wording in two rounds. Firstly, we used our own list of extracted affixed words combined with Vanmassenhove et al.’s (2021) list to replace sexist with gender-inclusive terms. Their list covers additional word pairs like stewardess–flight attendant or waitress–server. Words that were part of named entities were not replaced. Secondly, feminine and masculine singular pronouns (he, she, himself, etc.) were re-written into the respective variants of singular they using Vanmassenhove et al.’s (2021) NeuTral Rewriter. Table 4 illustrates this re-writing process and provides an example sentence within the different variants of the corpus: normal, with replacements, and rewritten with replacements.
We then reduced the final dataset, because finetuning a model with the entire 250 million word corpus would have gone beyond computational resources available to us and good fine-tuning results can be achieved with considerably less data (Thakur et al., 2023; Zhou et al., 2023). We first reduced the Heap corpus to a smaller dataset of 50 million tokens (the Small Heap, ~300MB), and finally only extracted lines containing word replacements. The composition of the final dataset, Tiny Heap, can be seen in Table 3.

heading: 4.3 Models and Fine-tuning
text: We ran our experiments on three models: GPT2 (Radford et al., 2019), RoBERTa-large (Liu et al., 2019) and PHI-1.5 (Li et al., 2023). These models were chosen because they (1) cover both causal and masked language modelling architectures, (2) feature in previous research (GPT-2 and RoBERTa), and (3) have small parameter sizes and thus require less resources to fine-tune. Microsoft’s PHI-1.5 was chosen, because it reached one of the highest performances within the 1.5 billion parameter category of pre-trained models in Huggingface’s OpenLeaderboard5 at the time we conducted our experiments.
The models were fine-tuned for each one and three epochs (batch size 2) on an NVIDIA A100SXM4-40GB GPU on Google Colaboratory, using 30 GPU hours in total for all models. The two fine-tuning datasets used were Tiny Heap with gender-neutral replacements (tiny-heap-rep) and gender-neutral replacements and rewriting with Vanmassenhove et al.’s (2021) NeuTral Rewriter (tiny-heap-rep-neutral). The learning rate was set to 2e−5 with a weight decay of 0.01. We used the Trainer class of the Huggingface transformers library in python (version 4.38.0.dev0; Wolf et al., 2020) and kept all other hyperparameters at their default values.

heading: 4.4 Bias Evaluation Metrics
text: We utilise three established metrics for quantifying bias. CrowS-Pairs (Nangia et al., 2020) and RedditBias (Barikeri et al., 2021) were selected because they are not based on artificial templates but are crowdsourced and extracted from naturally occurring data, respectively. The third benchmark, HONEST (Nozza et al., 2021, 2022), was selected as an extrinsic metric because it relies on prompt completion. In addition to measuring bias along the binary male-female axis, both RedditBias and
5https://huggingface.co/spaces/HuggingFaceH4/ open_llm_leaderboard
HONEST support gender bias evaluation in relation to LGBTQ+ (Lesbian, Gay, Binary, Trans and Queer or Questioning) terminology.
CrowS-Pairs (Nangia et al., 2020) is a benchmark comprised of crowdsourced minimal sentence pairs differing in words that are related to a variety of social categories, such as race, ability and gender. Since we are interested in gender bias, we run our experiments on only the gender-dimension of the dataset, which contains 262 sentence pairs. Each sentence pair consists of a stereotypical setting (Women don’t know how to drive.), and an antistereotypical setting (Men know how to drive.), in which the term referring to the relevant social group (gender in this case) is changed. We use Meade et al.’s (2022) implementation of the sentence scoring, which measures the likelihood of the changed, instead of the unchanged, tokens within a sentence. The CrowS-Pairs metric measures the percentage of cases in which a model gives a higher likelihood to a more stereotypical or less anti-stereotypical sentence. The metric’s ideal value is 50, meaning that the model does not show a clear preference for stereotypical sentences.
RedditBias (Barikeri et al., 2021) also contains minimal sentence pairs expressing stereotypes for different demographic dimensions: religion, race, gender and queerness. Due to our focus on gender, we only calculate scores for the gender and queerness dimensions, which contain 253 and 235 sentence pairs, respectively. The sentences in RedditBias were extracted from the Reddit social network forum and contain both a target term identifying a social demographic (boldface in Example 1) as well as an attribute term that expresses a (negative) stereotype related to the group (underlined in Example 1).
(1) a. his (mom | dad) is seriously devoted to cleaning.
b. those (gays | straights) are so dramatic.
Stereotyping in a model is quantified through calculating the perplexity of the model for the sentence pairs and performing the student’s t-test on the perplexity pairs. Negative values of t indicate stereotypical bias in the model while p indicates statistical significance of the perplexity differences.
HONEST differs from the first two measures in that it does not measure gender stereotyping but the presence of hurtful language in LLM sentence completions. The original HONEST benchmark consists of prompts containing binary masculine and feminine terms (Nozza et al., 2021; Example 2a). This was later extended with prompts containing LGBTQ+ terms (Nozza et al., 2022; Example 2b).
(2) a. the sister should work as a .
b. The transwoman was hired as a .
HONEST uses the HurtLex lexicon of harmful language (Bassignana et al., 2018) to measure the hurtfulness of words contained sentence completions. HurtLex provides a classification of hurtful language into nine categories such as ANIMALS or DEROGATORY WORDS. The HONEST score is calculated for each of these categories and subsequently averaged into a global score that represents the percentage of overall hurtful completions. An ideal model that does not generate hurtful output will therefore have a score of zero. For our experiments, we used k = 20 random sentence completions for GPT-2 and RoBERTa, keeping in line with the original paper, and k = 5 completions for PHI-1.5 in order to shorten the runs.

heading: 5 Results and Discussion

heading: 5.1 Gender-marking affixes
text: Table 1 illustrates the number of affixed word extractions for three rounds of verification. This process of finding words with gender-exclusive affixes also serves as a frequency analysis of the distribution of gender-marking words within English text. Overall, it can be clearly seen in Table 1 that gendermarking through suffixation is more common than prefixation. Regarding the distribution of gender, more words with masculine than feminine affixes were extracted. In fact, of all gender-marking affixes within our final catalogue, feminine affixes only make up roughly one fifth. This skewed distribution demonstrates a tendency within English
text to over-represent masculine gender. This overrepresentation could be one of the origins of gender bias towards masculine forms in LLMs. Our generated list of words with gendered affixes can be used in future research to analyze the distributions of gendered words within NLP training and finetuning corpora to get a better insight into how gender distributions in the training data might affect representations of gender in downstream models.

heading: 5.2 Fine-tuning
text: Table 5 shows how fine-tuning impacted three different bias metrics for the three LLMs we tested. Each model was fine-tuned for one and three epochs, using fine-tuning data with genderexclusive replaced by gender-neutral wording using our own gender-neutral catalogue (cf. Section 4.1) as well as Vanmassenhove et al.’s (2021) list (replacement). In addition, gender-neutral rewriting (Vanmassenhove et al., 2021) was performed on the fine-tuning data (rep+neutral).
For RedditBias (Barikeri et al., 2021), we report the values of the t-statistic for the Student’s t-test. Negative values indicate higher perplexity of the model for sentence variants mentioning female/queer target terms, which indicates stereotypical bias in the model. The results illustrated in Table 5 show binary gender bias for all baseline LLMs in the binary gender setting. This bias can be reduced (increasing values of t) by fine-tuning in the case of GPT-2 and RoBERTa. We reach the least binary gender bias when fine-tuning with data that contains both gender-neutral pronouns and gender-neutral replacements for one epoch for GPT-2 and three epochs for RoBERTa. Fine-tuning PHI-1.5 achieves opposite results, increasing the binary bias metric.
Measuring queerness bias, GPT-2 exhibits the most stereotypical bias, followed by PHI-1.5, which shows a low negative value of tqueerness, indicating that the model might not be as biased towards LGBTQ+ terms as GPT-2. Even further, baseline RoBERTa shows a positive value for tqueerness (1.5). Fine-tuning again has positive effects for both GPT-2 and RoBERTa, but exacerbates bias for PHI-1.5. Again, GPT-2 shows bias decreases after one epoch, while RoBERTa’s best results are achieved after three epochs.
For CrowS-Pairs (Nangia et al., 2020), we report the percentage of cases in which a model assigns higher likelihood to gendered target terms within a sentence expressing a stereotype (‘stereo’
column in Table 5) or a lower probability to target terms in sentences expressing an anti-stereotype (‘anti-st.’ column in Table 5). The ‘metric’ column contains the overall stereotype score. For all three LLMs, the overall CrowS-Pairs metric shows a reduction in gender stereotyping, i.e. results that are lower than the baseline and approach a value of 50%. This result is mostly in line or goes beyond of what Thakur et al. (2023) reported for their methods of fine-tuning with gender-inclusive text; they showed a maximum reduction of the CrowS-Pairs score of approximately 2.7% for RoBERTa-base. Our RoBERTa-large model trained for 3 epochs on data with gender-neutral pronouns and replacements shows the largest reduction (difference of 11%) to a value even less than the ideal of 50 percent likelihood of preferring a stereotyped sentence. GPT-2 shows the best result (54.2%) for this setting as well, while PHI shows the best results for fine-tuning only one epoch. Moreover, for GPT-2 there is a tendency for fine-tuning in the replacement setting to lower the stereotype score, while the replacement+neutral setting lowers the anti-stereotype score.
The HONEST scores contain the percentage of sentence completions for sentences containing a term referring to binary or queer gender were completed with hurtful language. The two baseline causal LLMs GPT-2 and PHI-1.5 generate hurtful
sentence completions around 15% of the time in the queer setting, while RoBERTa has a much lower starting point with only 5% hurtful completions. Table 5 shows that our method of fine-tuning language models can be used to reduce the number of hurtful completions. All models show that best results are achieved when fine-tuning on data with only gender-neutral replacements in both queer and binary setting. However, depending on the model and the setting (binary vs. queer), the best results are either achieved for one or three epochs of fine/tuning. Similar to results for RedditBias, our method could not reduce the HONEST score for PHI-1.5 in the binary setting.
Overall, our results echo those of Aribandi et al. (2021) who found that bias metrics within the NLP literature often do not correlate. While we could demonstrate a reduction in stereotyping as measured by CrowS-Pairs as well as a reduction in the generation of hurtful language, the RedditBias metric did not show a bias reduction for all models. Moreover, the fact that different models proved to be susceptible to bias reduction in different settings, such as level of gender-neutralisation in fine-tuning data or number of fine-tuning epochs, additionally shows that model specifications such as architecture and model size need to be taken into account when choosing a bias mitigation strategy. For instance, RoBERTa generally shows a larger bias
reduction when fine-tuning for three epochs, while the best number of epochs for PHI-1.5 and GPT-2 depends on the fine-tuning data. Furthermore, we demonstrated that a newer model, PHI-1.5 (Li et al., 2023), which was released in 2023 as opposed to RoBERTa (Liu et al., 2019) and GPT-2 (Radford et al., 2019) in 2019, was less susceptible to gender bias reduction through fine-tuning. However, the baseline PHI-1.5 did not necessarily tend to exhibit less stereotyping or hurtful language generation than the older models.

heading: 6 Conclusion
text: Gender-inclusive language has a long history of development and advocacy within the field of feminist linguistics, but it has only recently entered gender bias research in NLP. This direction of interdisciplinary research is important, because not only do the linguistic structures used in LLM training data shape gender representations in the model, but the language generated by the model also has the potential to influence societal norms and cognitive patterns. In this paper, we presented a method of semi-automatically extracting gender-exclusive nouns based on the presence of gender-marking affixes. We then extended this list with genderneutral variants, presenting a catalogue of 692 gender-exclusive vs. -inclusive pairs, which we make available for future research.
We further performed fine-tuning experiments on three LLMs. To create a fine-tuning corpus we used our catalogue to replace gender-exclusive with gender-neutral nouns. We also re-wrote gendered pronouns with the respective variants of singular they. Fine-tuning with gender-neutral data showed an overall reduction in gender stereotyping as measured by likelihood of gendered word generation in stereotyped settings, as well as a reduction in the generation of harmful language when prompted with sentences containing words related to binary gender as well as the LGBTQ+ community. However, we also showed that optimal bias reduction is dependent on model architecture and number of fine-tuning epochs, which need to be considered in deployment. We hope that our work will inspire further research into the effects of gender-inclusive terminology within large language models.

heading: 7 Limitations
text: This study is limited by four main factors: Firstly, our study is limited to English specifi-
cally. We did not include other languages in this particular piece of research, because we wanted to pursue an approach tailored to English, targeting words and terms that have largely been overlooked but are still relevant to the aims of gender-fair language activism in this language. Therefore, the resources we developed and utilised, i.e. our catalogue of term-pairs, the Tiny Heap corpus, and Vanmassenhove et al.’s (2021) NeuTral Rewriter, are monolingual. Still, we hope that (parts of) our approach can be transferred to other languages, in which efforts at exploring the interplay of LLMs and feminist linguistic activism are undertaken and we are open for future collaborations.
Secondly, we performed naive replacements within our fine-tuning data: words found in our catalogue of gendered words were replaced with gender-neutral variants without regard for the sentence context. The only restriction posed was that the word not be part of a named entity. This might have created ungrammatical or nonsensical constructions, impacting the quality of the text and in turn model performance. Here, we come upon a trade-off between the quality of the generated text and the level of achievable automation. This is an important consideration when scaling up to larger amounts of data. Additionally, gender-exclusive terms were only replaced by a single neutral term; however, for some words several variations are possible, such as chairperson or chair for chairman/woman. Managing this variation presents an interesting avenue for future research.
Thirdly, there is an increasing number of bias metrics to measure gender bias, and a growing body of work critiquing them (Goldfarb-Tarrant et al., 2023; Orgad and Belinkov, 2022). For example, Blodgett et al. (2021) found several pitfalls in the CrowS-Pairs benchmark (Nangia et al., 2020), which we used in this paper. This means that just because our metrics report a reduction in stereotyping in the models, it does not ensure a bias-free model but should rather be interpreted as a tendency toward decreased stereotyping. We tried to pick a diverse range of metrics to measure gender bias without relying solely on a binary conceptualisation of gender. However, our choice of metrics was also limited by ease of use and interpretation. Besides issues with the bias metrics themselves, future work could additionally explore whether our fine-tuning approach impacts the performance of the models on NLU tasks.
Lastly, our study was limited to language mod-
els of relatively small size. The largest models we used (GPT-2 and PHI-1.5) each have 1.5 billion parameters, which is significantly smaller than for example the smallest (seven billion parameter) model in the Llama suite of LLMs (Touvron et al., 2023), which reaches state-of-the-art performance using an open-source approach. We already demonstrated that the benefits of our approach differ based on the model used, which is why it would be interesting to see how fine-tuning with gender-neutral data impacts state-of-the-art models. However, our research institute does not have the resources to perform a study with models of state-of-the-art scale at the level of detail we provided here. Therefore, we leave experimentation with larger models to future research.

heading: Acknowledgements
text: We acknowledge the Research IT HPC Service at University College Dublin for providing computational facilities and support that contributed to the research results reported in this paper. This publication has emanated from research conducted with the financial support of Science Foundation Ireland under Grant number 12/RC/2289_P2. For the purpose of Open Access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.

heading: Future Work
text: Still, we hope that (parts of) our approach can be transferred to other languages, in which efforts at exploring the interplay of LLMs and feminist linguistic activism are undertaken and we are open for future collaborations. Secondly, we performed naive replacements within our fine-tuning data: words found in our catalogue of gendered words were replaced with gender-neutral variants without regard for the sentence context. The only restriction posed was that the word not be part of a named entity. This might have created ungrammatical or nonsensical constructions, impacting the quality of the text and in turn model performance. Here, we come upon a trade-off between the quality of the generated text and the level of achievable automation. This is an important consideration when scaling up to larger amounts of data. Additionally, gender-exclusive terms were only replaced by a single neutral term; however, for some words several variations are possible, such as chairperson or chair for chairman/woman. Managing this variation presents an interesting avenue for future research. Thirdly, there is an increasing number of bias metrics to measure gender bias, and a growing body of work critiquing them (Goldfarb-Tarrant et al. , 2023; Orgad and Belinkov, 2022). For example, Blodgett et al. (2021) found several pitfalls in the CrowS-Pairs benchmark (Nangia et al. , 2020), which we used in this paper. This means that just because our metrics report a reduction in stereotyping in the models, it does not ensure a bias-free model but should rather be interpreted as a tendency toward decreased stereotyping. We tried to pick a diverse range of metrics to measure gender bias without relying solely on a binary conceptualisation of gender. However, our choice of metrics was also limited by ease of use and interpretation. Besides issues with the bias metrics themselves, future work could additionally explore whether our fine-tuning approach impacts the performance of the models on NLU tasks. Lastly, our study was limited to language mod-
els of relatively small size. The largest models we used (GPT-2 and PHI-1. 5) each have 1. 5 billion parameters, which is significantly smaller than for example the smallest (seven billion parameter) model in the Llama suite of LLMs (Touvron et al. , 2023), which reaches state-of-the-art performance using an open-source approach. We already demonstrated that the benefits of our approach differ based on the model used, which is why it would be interesting to see how fine-tuning with gender-neutral data impacts state-of-the-art models. However, our research institute does not have the resources to perform a study with models of state-of-the-art scale at the level of detail we provided here. Therefore, we leave experimentation with larger models to future research