SlowGuess/ACL_2020_OCR huggingface dataset:
Luisa Bentivogli\*, Beatrice Savoldi\*, Matteo Negri, Mattia Antonino Di Gangi\*, Roldano Cattoni, Marco Turchi $^{1}$ $^{1}$ Fondazione Bruno Kessler  
 $^{2}$ University of Trento  
Trento, Italy  
{bentivo, negri, digangi, cattoni, turchi}@fbk.eu  
beatrice.savoldi@unitn.it
Translating from languages without productive grammatical gender like English into gender-marked languages is a well-known difficulty for machines. This difficulty is also due to the fact that the training data on which models are built typically reflect the asymmetries of natural languages, gender bias included. Exclusively fed with textual data, machine translation is intrinsically constrained by the fact that the input sentence does not always contain clues about the gender identity of the referred human entities. But what happens with speech translation, where the input is an audio signal? Can audio provide additional information to reduce gender bias? We present the first thorough investigation of gender bias in speech translation, contributing with:  $i$  ) the release of a benchmark useful for future studies, and  $ii$  ) the comparison of different technologies ( Cascade and end-to-end) on two language directions (English-Italian/French).
With the exponential popularity of deep learning approaches for a great range of natural language processing (NLP) tasks being integrated in our daily life, the need to address the issues of gender fairness $^{1}$  and gender bias has become a growing interdisciplinary concern. Present-day studies on a variety of NLP-related tasks, such as sentiment analysis (Kiritchenko and Mohammad, 2018) coreference resolution (Rudinger et al., 2018; Webster et al., 2018; Zhao et al., 2018), visual semantic-role labeling (Zhao et al., 2017) or language modeling (Lu
et al., 2019), attest the existence of a systemic bias that reproduces gender stereotypes discriminating women. In translation-related tasks, gender bias arises from the extent through which each language formally expresses the female or male gender of a referred human entity. Languages with a grammatical system of gender, such as Romance languages, rely on a copious set of morphological (inflection) and syntactic (gender agreement) devices applying to numerous parts of speech (Hockett, 1958). Differently, English is a natural gender language that only reflects distinction of sex via pronouns, inherently gendered words (boy, girl) and exceptionally with marked nouns (actor, actress). For all the other indistinct neutral words, the gender of the referred entity – if available – is inferred from contextual information present in the discourse, e.g. he/she is a friend.
Nascent inquiries on machine translation (MT) pointed out that machines tend to reproduce the linguistic asymmetries present in the real-world data they are trained on. In the case of gender inequality, this is made apparent by the attribution of occupational roles from gender-neutral linguistic forms into marked ones, where MT often wrongly chooses male-denoting (pro)nouns, e.g. identifying scientist, engineer or doctor as men (Prates et al., 2018; Escudé Font and Costa-jussà, 2019). Failing to pick the appropriate feminine form is both a technical and an ethical matter: gender-related errors affect the accuracy of MT systems but, more significantly, a biased system can dangerously perpetuate the under-/misrepresentation of a demographic group (Crawford, 2017).
Previous studies accounting for MT systems' strengths and weaknesses in the translation of gender shed light on the problem but, at the same time, have limitations. On one hand, the existing evaluations focused on gender bias were largely conducted on challenge datasets, which are controlled
*These authors contributed equally. The work by Beatrice Savoldi was carried out during an internship at Fondazione Bruno Kessler.
<sup>1</sup>We acknowledge that gender is a multifaceted notion, not necessarily constrained within binary assumptions. However, since speech translation is hindered by the scarcity of available data, we rely on the female/male distinction of gender, as it is linguistically reflected in existing natural data.
6923
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923-6933 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
artificial benchmarks that provide a limited perspective on the extent of the phenomenon and may force unreliable conclusions (Prates et al., 2018; Cho et al., 2019; Escudé Font and Costa-jussà, 2019; Stanovsky et al., 2019). On the other hand, the natural corpora built on conversational language that were used in few studies (Elaraby et al., 2018; Vanmassenhove et al., 2018) include only a restricted quantity of not isolated gender-expressing forms, thus not permitting either extensive or targeted evaluations. Moreover, no attempt has yet been made to assess if and how speech translation (ST) systems are affected by this particular problem. As such, whether ST technologies that leverage audio inputs can retrieve useful clues for translating gender in addition to contextual information present in the discourse, or supply for their lack, remains a largely unexplored question. In the light of above, the contributions of this paper are:
(1) We present the first systematic analysis aimed to assess ST performance on gender translation. To this aim, we compare the state-of-the-art cascaded approach with the emerging end-to-end paradigm, investigating their ability to properly handle different categories of gender phenomena.
(2) We publicly release MuST-SHE, $^{2}$  a multilingual, natural benchmark allowing for a fine-grained analysis of gender bias in MT and ST. MuST-SHE is a subset of the TED-based MuST-C corpus (Di Gangi et al., 2019a) and is available for English-French and English-Italian. $^{3}$  For each language pair, it comprises  $\sim 1,000$  (audio, transcript, translation) triplets annotated with qualitatively differentiated and balanced gender-related phenomena.
(3) We implement a new evaluation method that acknowledges and adapts previous related works to go beyond them and make BLEU scores informative about gender. It removes unrelated factors that may affect the overall performance of a system to soundly estimate gender bias.
On the two language pairs addressed, our comparative evaluation of cascade vs. end-to-end ST systems indicates that the latter are able to better exploit audio information to translate specific gender phenomena, for which the cascade systems require externally-injected information.
Speech translation. The task of translating audio speech in one language into text in another language has been traditionally approached with cascade architectures combining automatic speech recognition (ASR) and MT components (Eck and Hori, 2005). The main advantage of this pipelined solution is that it can directly plug-in state-of-the-art technology for both components and exploit the wealth of training data available for the two tasks. The approach, however, has some drawbacks. One is error propagation: sub-optimal transcriptions by the ASR component have significant impact on the final output produced by the MT component. To cope with this issue, recent works focused on making MT models more robust to noisy input transcripts (Sperber et al., 2017, 2019; Di Gangi et al., 2019b).
A second issue, particularly relevant to this research, is the information loss when passing from audio to text representations. Even with perfect transcripts, subtle aspects that cannot be grasped from the text only (e.g. speaker's pitch as a clue of his/her gender) can only be reintroduced by injecting external knowledge to support the MT step (Elaraby et al., 2018). By avoiding intermediate text representations, direct end-to-end translation from audio to text (Bérard et al., 2016) can potentially cope with these limitations. However, due to the dearth of training corpora, it still underperforms with respect to the cascaded approach. Recent evaluation campaigns (Niehues et al., 2018, 2019) have shown that, although the gap is gradually closing (less than 2 BLEU points), cascade models still represent the state-of-the-art. In spite of the steady technological progress, little has so far been done to directly compare the two technologies on specific translation problems like the one addressed in this paper.
Measuring gender bias. Previous attempts to test the production of gender-aware automatic translations solely focused on MT, where a widespread approach involves the creation of challenge datasets focused on specific linguistic phenomena. Prates et al. (2018) and Cho et al. (2019) construct template sentences using occupational or sentiment words associated with a gender-neutral pronoun, to be translated into an English gender-specified one ([x] is a professor: he/she is a professor). Similarly, the Occupations Test (Escudé Font and Costajussà, 2019) and Wino MT (Stanovsky et al., 2019)
$^{2}$ MuST-SHE is released under a CC BY NC ND 4.0 International license, and is freely downloadable atICT.fbk.eu/must-she.
3The current release of the corpus also includes an English-Spanish section, which was completed in April 2020.
6924
cast human entities into proto- or anti-stereotypical gender associations via coreference linking (e.g. the English sentence “The janitor does not like the baker because she/he always messes up the kitchen”, where “the baker” is to be translated into Spanish as la panadera or el panadero depending on the English pronoun). Although such simple constructions allow for targeted experiments, artificial data characterized by a qualitatively limited variety of phenomena generate constrained environments that may produce biased results. As far as studies on naturally occurring data are concerned, Vanmassenhove et al. (2018) estimate MT systems’ performance in the realization of speaker’s gender agreement on two male and female test sets containing first person singular pronouns. This strategy increases the chances to isolate speaker-dependent gendered expressions, but still, the employed BLEU metric does not pointedly grasp the effect of gender translation on the output, as the overall performance is also impacted by other factors. Analogously, Elaraby et al. (2018) design a set of agreement rules to automatically recover 300 gender-affected sentences in their corpus, but the evaluation relies on global BLEU scores computed on a bigger set (1,300 sentences) and does not consider male-female related differences. Moryossef et al. (2019) use a parser to detect morphological realizations of speakers’ gender on a single female-speaker corpus that does not permit inter-gender comparisons.
In light of above, an ideal test set should consist of naturally occurring data exhibiting a diversified assortment of gender phenomena so to avoid forced predictions with over-controlled procedures. Also, a consistent amount of equally distributed feminine and masculine gender realizations need to be identified to disentangle the accuracy of gender translation from the overall model's performance. Accordingly, in §3 we present MuST-SHE, a multilingual test set designed for the investigation of gender bias in ST, which, as explained in §4, is used for a targeted gender-sensitive evaluation approach.
3 The MuST-SHE benchmark
We built MuST-SHE on naturally occurring data retrieved from MuST-C (Di Gangi et al., 2019a), the largest freely available multilingual corpus for ST, which comprises (audio, transcript, translation) triplets extracted from TED talks data. Besides being multilingual, MuST-C is characterized by high
quality speech and a variety of different speakers that adequately represent women, two aspects that determined its selection among other existing corpora (Post et al., 2013; Kocabiyikoglu et al., 2018; Sanabria et al., 2018). As such, MuST-SHE was compiled by targeting in the original dataset linguistic phenomena that entail a gender identification from English into Italian and French, two Romance languages that extensively express gender via feminine or masculine morphological markers on nouns, adjectives, verbs and other functional words (e.g. articles and demonstratives).
3.1 Categorization of gender phenomena
MuST-SHE is compiled with segments that require the translation of at least one English gender-neutral word into the corresponding masculine or feminine target word(s), where such formal distinction semantically conveys and conflates with an actual distinction of sex (Corbett, 1991). For instance, the English utterance "a good teacher" would either become in French "un bon enseignant" or "uneonne enseignante" for, respectively, a male or female referent. In spoken language data, the human entity that determines gender agreement is either the speaker him/herself (I am a good teacher) or another person the speaker is referring to (he/she is a good teacher). We classify our phenomena of interest in two categories based on where the necessary information to disambiguate gender can be recovered, namely (Category 1) from the audio signal, when gender-agreement only depends on the speaker's gender, which can be captured from intrinsic properties of the audio (I am a teacher uttered by a man/woman); (Category 2) from the utterance content, where contextual hints such as gender-exclusive words (mom), pronouns (she, his) and proper nouns (Paul) inform about the gender of the referent.
3.2 Dataset creation and annotation
To gain a better insight into MuST-C linguistic data and capture the features of gender, we initially conducted a qualitative cross-lingual analysis on 2,500 parallel sentences randomly sampled from the corpus. The analysis led to the design of an automatic approach aimed to quantitatively and qualitatively maximize the extraction of an assorted variety of gender-marked phenomena belonging to categories 1 and 2. Regular expressions were employed to transform gender-agreement rules into search patterns to be applied to MuST-C. Our queries were
Form	Category 1: Gender info in audio	Speaker
Fem.	SRC	I was born and brought up in Mumbai.	Female
C-REFIt	Sono nata e cresciuta a Mumbai.
W-REFIt	Sono nato e cresciuto a Mumbai.
SRC	I was born and brought up in Mumbai.
C-REFFr	Je suis née et j'ai grandi à Mumbai.
W-REFFr	Je suis né et j'ai grandi à Mumbai.
Masc.	SRC	I myself was one of them, and this is what I talk about at the HALT events.	Male
C-REFIt	Io stesso ero uno di loro, e parlo di quello agli eventi HALT.
W-REFIt	Io stessa ero una di loro, e parlo di quello agli eventi HALT.
SRC	I myself was one of them, and this is what I talk about at the HALT events.
C-REFFr	Moi-même, j'ai été l'un d'eux, et voilà de quoi je parle aux événements d'HALT.
W-REFFr	Moi-même, j'ai été l'une d'eux, et voilà de quoi je parle aux événements d'HALT.
Category 2: Gender info in utterance content
Fem.	SRC	She'd get together with two of her dearest friends, these older women...	Male
C-REFIt	Tornava per incontrare un paio delle sue più care amiche, queste signore anziane...
W-REFIt	Tornava per incontrare un paio dei loro più cari amici,QUESTI signore anziani...
SRC	She'd get together with two of her dearest friends, these older women...
C-REFFr	Elle se réunissait avec deux de ses amies les plus chères, ces femmes plus âgées...
W-REFFr	Elle se réunissait avec deux de ses amis les pluschers, ces femmes plus âgés...
Masc.	SRC	Dean Kamen, one of the great DIY innovators. His technology...	Female
C-REFIt	Dean Kamen, uno dei più grandi innovatori del fai-da-te. La sua Tecnologia...
W-REFIt	Dean Kamen, una delle più grandi innovatrici del fai-da-te. La sua Tecnologia...
SRC	Dean Kamen, one of the great DIY innovators. His technology...
C-REFFr	Dean Kamen, l'un des grands innovateurs autonomes. Sa technologie...
W-REFFr	Dean Kamen, l'une des grandes innovatrices autonomes. Sa technologie...
Table 1: MuST-SHE annotated segments organized per category. For each example in En-It and En-Fr, the Correct Reference Translation (C-REF) shows the realization of target gender-marked forms (Masc/Fem) corresponding to English gender-neutral words in the source (SRC). In the Wrong Reference Translation (W-REF), Italian and French gender-marked words are swapped to their opposite gender form. The last column of the table provides information about the speaker's gender (Male/Female).
designed and adapted to the targeted language pairs, categories, and masculine/feminine forms. To specifically match a differentiated range of gender-marked lexical items, we also compiled two series of 50 human-referring adjectives in French and Italian, as well as a list with more than 1,000 English occupation nouns obtained from the US Department of Labour Statistics $^{4}$  (Prates et al., 2018).
For each language direction, the pool of sentence pairs retrieved from MuST-C was manually checked in order to:  $i$  ) remove noise and keep only pairs containing at least one gender phenomenon, ii) include all En-It/En-Fr corresponding pairs to create a common multilingual subset, and iii) select the remaining pairs ensuring a balanced distribution of categories, feminine/masculine forms, and female/male speakers. Once the textual part of MuST-SHE was created, all the corresponding au
dio segments were manually checked in order to correct possible misalignments.
The resulting dataset was then manually enriched with different types of information that allow for fine-grained evaluations. Annotations include: category, masculine/feminine form, speaker's gender, and all the gender-marked expressions in the reference translation. Finally, in order to perform a sound evaluation able to discriminate gender-related issues from other non-related factors that may affect systems' performance, for each correct reference translation (C-REF) we created an almost identical "wrong" alternative (W-REF) in which all the gender-marked words are swapped to their opposite form (details in §4). Some examples extracted from MuST-SHE are presented in Table 1.
To ensure data quality, the whole dataset was created and annotated by an expert linguist with a background in translation studies, who produced strict and comprehensive guidelines based on the
<sup>4</sup>http://www.bls.gov/emp/tables/emp-by-detailied-occupation.htm
6926
preliminary manual analysis of a sample of MuST-C data (2,500 segments). Then, a second linguist independently re-annotated each MuST-SHE segment with the corresponding category and produced an additional "wrong" reference. Being the annotation per category a straightforward task, it resulted in no disagreement for Category 1 and around  $0.03\%$  for Category 2. Such few cases were removed from the dataset, which thus contains only segments in complete agreement. Disagreements were more common in the "wrong" references, since the task requires producing subtle variations that can be hard to spot. Disagreements, amounting to around  $11\%$ , were all oversights and thus reconciled.
MuST-SHE comprises 2,136 (audio, transcript, translation) triplets (1,062 for En-It and 1,074 for En-Fr) uttered by 273 different speakers. A common subset of 696 instances allows for comparative evaluations across the two language directions. As shown by the statistics in Table 2, the corpus presents a balanced distribution across  $i$  ) masculine and feminine forms, and  $ii$  ) gender phenomena per category. Female and male speakers (558/513 for En-It, 577/498 for En-Fr) are substantially balanced. The gender of the speaker and of the referred entity in the utterance is the same in Category 1 (where the speakers talk about themselves), while it differs in about  $50\%$  of the segments in Category 2 (where they refer to other entities).
MuST-SHE differs from standard test sets, as it is precisely designed to:  $i$  ) equally distribute gender references as well as speakers, and  $ii$  ) allow for a sound and focused evaluation on the accuracy of gender translation. As such, it satisfies the parameters to be qualified as a GBET, Gender Bias Evaluation Testset (Sun et al., 2019), and represents the very first of its kind for ST and MT created on natural data.
4.1 Evaluation Method
MT evaluation metrics like BLEU (Papineni et al., 2002) or TER (Snover et al., 2006) provide a global score about translation "quality" as a whole. Used as-is, their holistic nature hinders the precise evaluation of systems' performance on an individual phenomenon as gender translation, since the variations of BLEU score are only a coarse and indi
En-It	En-Fr
Fem	Masc	Tot.	Fem	Masc	Tot.
Cat. 1	278	282	560	316	296	612
Cat. 2	238	264	502	226	236	462
Tot.	516	546	542	532
Total	1,062 (1,940)	1,074 (2,010)
Table 2: MuST-SHE statistics. En-It and En-Fr number of segments split into feminine and masculine gender phenomena and category. In parentheses the total number of gender-marked words annotated in the references.
rect indicator of better/worse overall performance (Callison-Burch et al., 2006). This represents a limitation of recent related works, which over-rely on the results of a BLEU-based quantitative analysis. For instance, the BLEU gains obtained by prepending gender tags or other artificial antecedents to the input source, as in Vanmassenhove et al. (2018) and Moryossef et al. (2019), cannot be assuredly ascribed to a better control of gender features. To overcome this problem, Moryossef et al. (2019) complement their discussion with a qualitative syntactic analysis, which implies the availability of a parser for the target language and a higher complexity of the whole evaluation protocol. Instead, our aim is to keep using  $\mathrm{BLEU}^5$  and make the resulting scores informative about systems' ability to produce the correct gender forms.
To this aim, for each reference  $c$  in the corpus we create a "wrong" one that is identical to  $c$ , except for the morphological signals that convey gender agreement. In particular, for each gender-neutral English word in the source utterance (e.g. "one", "great" and "innovators" in the  $4^{th}$  example of Table 1), the correct translation (containing the French words with masculine inflection "un", "grands" and "innovateurs") is swapped into its opposite gender form (containing feminine-marked words "une", "grandes" and "innovatrices"). The result is a new set of references that, compared to the correct ones, are "wrong" only with respect to the formal expression of gender.
The underlying idea is that, as the two reference sets differ only for the swapped gendered forms, results' differences for the same set of hypotheses produced by a given system can measure its capability to handle gender phenomena. In partic
Still the de facto standard in MT evaluation in spite of constant research efforts towards metrics that better correlate with human judgements.
6927
ular, we argue that higher values on the wrong set can signal a potentially gender-biased behaviour. In all the cases where the required gender realization is feminine, significantly higher BLEU results computed on the wrong set would signal a bias towards producing masculine forms, and vice versa. Although this idea recalls the gender-swapping approach used in previous NLP studies on gender bias (Sun et al., 2019; Lu et al., 2019; Kiritchenko and Mohammad, 2018; Zhao et al., 2018; Cao and Daumé III, 2019), in such works it is only applied to pronouns; here we extend it to any gender-marked part of speech.
In addition to the quantitative BLEU-based evaluation<sup>6</sup>, we also perform a fine-grained qualitative analysis of systems' accuracy in producing the target gender-marked words. We compute accuracy as the proportion of gender-marked words in the references that are correctly translated by the system. An upper bound of one match for each gender-marked word is applied in order not to reward over-generated terms. Besides global accuracy, we also compute scores on both the correct and the wrong reference sets, as well as per category.