A Case Study of Natural Gender Phenomena in Translation. A Comparison of Google Translate, Bing Microsoft. Translator and DeepL for English to Italian, French and Spanish
Authors: Argentina Anna Rescigno, Eva Vanmassenhove, Johanna Monti, Andy Way (Università di Napoli L'Orientale, Tilburg University, Dublin City University, 2020)
Topic: Gender and Inclusive Language Studies

Abstract
This paper presents the results of an evaluation of Google Translate, DeepL and Bing Microsoft Translator with reference to natural gender translation and provides statistics about the frequency of female, male and neutral forms in the translations of a list of personality adjectives, and nouns referring to professions and bigender nouns. The evaluation is carried out for English\rightarrowSpanish, English\rightarrowItalian and English\rightarrowFrench.

Introduction
Gender manifests itself in a language in many ways, and different languages use different linguistic devices to mark (or sometimes `not mark') gender. When dealing with language, three types of gender come into play: natural gender, grammatical gender and social gender. Natural gender is generally based on the sex of a person or an animal realised by means of the male/female polarity or on the absence of sex for neutral nouns. Grammatical gender, instead, is not always coherent with the semantic categorization of a word and can vary from language to language since it depends on the representation of objects in the world on the basis of specific properties attributed to them in a specific cultural context. Social gender is used in relation to the properties of a word on the basis of which the speakers of a language associate the natural gender of a person to a word AUTHOR: this mainly happens with names of professions, such as for instance doctor or nurse which are interpreted according to social stereotypes concerning the roles of males and females in the society. Gender is present in the data we use to train MT systems due to the demographic features of the human training data and because of the nature of stereotypes and biases we communicate in our day-to-day communications. As most state-of-the-art MT systems handle translations on the sentence-level, gender phenomena are, usually, resolved on statistics inferred from the training data. Mistranslations of gender information occur more frequently when translating from gender-neutral languages, such as English, into morphological-rich languages, such as Italian or French, which explicitly mark gender and require additional information to correctly translate gender phenomena. When such additional information or context is not provided, the system will pick the most likely variant. A recent study by AUTHOR showed how Google Translate (GT) yields more male defaults than what ought to be expected when looking at demographic data on its own, alluding that there might be a phenomenon they refer to as machine bias AUTHOR. In this paper, we systematically evaluate: (a) single-word queries, containing personality adjectives and profession nouns, and (b) bigender nouns in an EN \rightarrow IT, FR, ES translation setting for GT, DeepL (DL) and Bing Microsoft Translator (BMT) to verify the diversity in translations provided by these MT providers.

Conclusion
In future work, we would like to conduct a larger evaluation comprising of more language pairs and a more diverse set of words. Furthermore, we aim to compile a challenge set focusing specifically on gender phenomena in language that can be used and automatically evaluated. We also envisage training our own state-of-the-art MT system to verify how and whether machine bias indeed influences the output of the translations generated.