SlowGuess/ACL_2020_OCR huggingface dataset:
It's worth remarking that the BLEU-based and the accuracy-based evaluations are complementary: the former aims to shed light on system's translation performance with respect to gender phenomena; the latter, which is more discriminative, aims to point to the actual words through which gender is realized. Compared to the standard BLEU-based evaluation with correct references only, we expect that the possible differences suggested by its extension with gender swapping will be reflected and amplified by sharper accuracy differences.
4.2 ST Systems
In our experiments, we compare an End2End system with two cascade systems (Cascade and Cascade+tag), whose architectures are described below.
Our End2End system uses the S-transformer architecture, which has proved to work reasonably well for this task (Di Gangi et al., 2019c). It is an encoder-decoder architecture that modifies the Transformer architecture (Vaswani et al., 2017) in two aspects. First, the audio input – in the form of sequences of 40 MFCCs (Davis and Mermelstein, 1980) – is processed by a stack of 2D CNNs (LeCun
et al., 1998), each followed by batch normalization (Ioffe and Szegedy, 2015) and ReLU nonlinearity. Second, the output of the CNNs is processed by 2D self-attention networks to provide a larger context to each element. The output of the 2D attention is then summed with the positional encoding and fed to transformer encoder layers. In the second part, a distance penalty is added to the non-normalized probabilities in the encoder self-attention networks in order to bias the computation towards the local context. To improve translation quality, the End2End systems are trained on the MuST-C and Librispeech (Kocabiyikoglu et al., 2018) corpora using SpecAugment (Park et al., 2019). Since Librispeech is a corpus for ASR, we augmented it by automatically translating the original English transcripts into both target languages. Translations are performed at character level, using the MT systems integrated in the cascade model.
Our Cascade systems share the same core (ASR, MT) technology. The ASR component is based on the KALDI toolkit (Povey et al., 2011), featuring a time-delay neural network and lattice-free maximum mutual information discriminative sequence-training (Povey et al., 2016). The audio data for acoustic modeling include the clean portion of LibriSpeech (Panayotov et al., 2015) ( $\sim$ 460h) and a variable subset of the MuST-C training set ( $\sim$ 450h), from which 40 MFCCs per time frame were extracted; a MaxEnt language model (Alumäe and Kurimo, 2010) is estimated from the corresponding transcripts ( $\sim$ 7M words). The MT component is based on the Transformer architecture, with parameters similar to those used in the original paper. The training data are collected from the OPUS repository, resulting in 70M pairs for En-It and 120M for En-Fr. For each language pair, the MT system is first trained on the OPUS data and then fine-tuned on MuST-C training data ( $\sim$ 250K pairs) – from which the MuST-SHE segments are removed. Byte pair encoding (BPE) (Sennrich et al., 2015) is applied to obtain 50K sub-word units. To mitigate error propagation and make the MT system more robust to ASR errors, similarly to (Di Gangi et al., 2019b) we tune it on a dataset derived from MuST-C, which includes both human and automatic transcripts. The training set, consisting of (audio, transcript) pairs, is split in two equally-sized parts: the first one is used to adapt the ASR system to the TED talk language, while
<sup>6</sup>We also computed TER scores and the results are fully in line with the reported BLEU scores.
<sup>7</sup>http://opus.nlpl.eu
6928
Systems	All	Feminine	Masculine
Correct	Wrong	Diff	Correct	Wrong	Diff	Correct	Wrong	Diff
En-It	End2End	21.5	19.7	1.8	20.2	19.3	0.9	22.7	20.0	2.7
Cascade	24.1	22.4	1.8	22.8	21.9	0.8	25.5	22.8	2.7
Cascade+Tag	23.8	20.9	2.9	23.0	20.4	2.6	24.5	21.3	3.2
En-Fr	End2End	27.9	25.8	2.1	26.3	25.0	1.3	29.5	26.4	3.1
Cascade	32.2	30.1	2.1	30.4	29.4	1.0	33.8	30.8	3.0
Cascade+Tag	32.2	28.6	3.6	31.6	28.0	3.6	32.7	29.2	3.5
Table 3: BLEU scores for En-It and En-Fr on MuST-SHE. Results are provided for the whole dataset (All) as well as split according to feminine and masculine word forms. Results are calculated for both the Correct and Wrong datasets, and their difference is provided (Diff).
the second part is transcribed by the tuned ASR system. The human transcripts of the first half and the automatic transcripts of the second half are concatenated and used together with their reference translations to fine-tune the MT system. This process makes the MT system aware of possible ASR errors and results in more than 2 BLEU points improvement on the MuST-C test set.
We also train an enhanced version of the Cascade system. Similarly to Vanmassenhove et al. (2018), it is informed about speaker's gender by pre-pending a gender token (<toM> or <toF>) to each source transcript. The gender token is obtained by manually assigning the correct gender label to each speaker in MuST-C. This externally-injected knowledge allows the Cascade+Tag system to mimic end-to-end technology by leveraging gender information during translation.
To check the overall quality of our systems, we compared them with published results on MuST-C test data. Our End2End systems (En-It: 21.5, En-Fr: 31.0) outperform all the models proposed in Di Gangi et al. (2019c), which were trained only on MuST-C (En-It: end2end 16.8, cascade 18.9; En-Fr: end2end 26.9, cascade 27.9). Our Cascade (En-It: 27.4 En-Fr: 35.5) also outperforms the system described in Indurthi et al. (2019) (En-Fr: 33.7). Our results are in line with the findings of IWSLT 2019 (Niehues et al., 2019) showing that the cascade approach still outperforms the direct one, although with a gradually closing gap.
BLEU. Table 3 presents translation results in terms of BLEU score on the MuST-SHE dataset. Looking at overall translation quality (All/Correct column), the results on both language pairs show that the highest performance is achieved by cascade architectures, which are better than End2End by 2.6 points for En-It and 4.3 for En-Fr. We do
not observe a statistically significant difference between Cascade and Cascade+Tag, suggesting that the injection of gender information into Cascade+Tag does not have visible effects in terms of translation quality, even on a focused dataset like MuST-SHE where each segment contains at least one gender realization. Our results thus seem to be in contrast with previous works implementing the same injection approach (Van-massenhove et al., 2018; Elaraby et al., 2018).
However, looking at the scores' gap between the Correct and the Wrong datasets (All/Diff column), it becomes evident that the standard evaluation based on BLEU calculated on a single correct reference hides specific relevant aspects in translation. In fact, despite the lower overall BLEU scores, for both language pairs End2End performs on par with Cascade as far as gender phenomena are concerned (1.8 on En-It and 2.1 on En-Fr). Also, the largest All/Diff value achieved by the enhanced Cascade+Tag supports the results obtained in previous studies (Vanmassenhove et al., 2018; Elaraby et al., 2018), confirming the importance of applying gender-swapping in BLEU-based evaluations focused on gender translation.
The fact that the All/Diff values are always positive indicates that all the systems perform better on the Correct dataset (i.e. they generate the correct gender-marked words more often than the wrong ones). However, examining the results at the level of masculine/feminine word forms, we notice that Diff values are higher on the Masculine subset (where the required gender realization is masculine) than in the Feminine one (where the required gender realization is feminine). As discussed in §4.1, this signals a bias of the systems towards producing masculine forms. The only exception is the En-Fr Cascade+Tag, where the Diff values remain stable across the two subsets (3.6 and 3.5). This absence of bias towards the masculine forms
6929
Systems	All	Feminine	Masculine
Correct	Wrong	Diff	Correct	Wrong	Diff	Correct	Wrong	Diff
En-It	End2End	43.3	16.4	26.9	34.2	24.0	10.2	51.3	9.6	41.7
Cascade	41.1	17.5	23.6	33.7	24.5	9.2	47.6	11.2	36.4
Cascade+Tag	48.0	10.4	37.6	44.7	14.0	30.7	51.0	7.2	43.8
En-Fr	End2End	46.0	19.0	27.0	35.8	25.0	13.8	55.3	13.8	41.5
Cascade	49.6	20.5	29.1	39.6	26.2	13.4	58.7	15.2	43.5
Cascade+Tag	57.2	11.3	45.9	53.8	11.8	42.0	60.3	10.7	49.6
Table 4: Accuracy scores for En-It and En-Fr on MuST-SHE. Results are provided for the whole dataset (All) as well as split according to feminine and masculine word forms. Results are calculated for both the Correct and Wrong datasets, and their difference is provided (Diff).
En-it
Feminine	Masculine
End2End
Corr.	Wrong	Diff.	Corr.	Wrong	Diff.
Cat. 1	26.7	27.2	-0.5	46.3	6.8	39.5
Cat. 2	40.6	20.5	20.1	53.9	10.9	43.0
Cascade
Cat. 1	15.9	34.5	-18.6	40.0	12.0	28.0
Cat. 2	48.9	15.7	33.2	51.2	10.8	40.4
Cascade+Tag
Cat. 1	43.0	10.4	32.6	48.5	2.9	45.6
Cat. 2	46.9	15.5	31.4	51.7	9.6	42.1
En-Fr
Feminine	Masculine
End2End
Corr.	Wrong	Diff.	Corr.	Wrong	Diff.
25.4	29.5	-4.1	48.0	7.7	40.3
45.0	20.3	24.7	60.0	17.6	42.4
Cascade
20.4	37.5	-17.1	49.1	13.0	36.1
56.3	15.6	40.7	64.9	16.7	48.2
Cascade+Tag
53.7	7.0	46.7	55.4	4.3	51.1
54.2	14.8	39.4	62.8	15.5	47.3
Table 5: Accuracy scores for En-It and En-Fr split according to MuST-SHE categories (Cat 1: information in audio, Cat 2: information in utterance content). For each category, results are further split into masculine/feminine forms. Results are calculated for both the Correct and Wrong datasets, and their difference is provided (Diff).
is in line with the All/Diff results indicating that this system is the best one in translating gender.
Although our gender-swapping methodology allows us to measure differences across systems that cannot be observed with standard BLEU evaluations, the results obtained so far may still conceal further interesting differences. This can depend on the fact that BLEU works at the corpus level and the small proportion of gender-marked words in MuST-SHE ( $\sim 2,000$  out of  $\sim 30,000$  total words, avg. 1.8 per sentence) can have limited influence on global measurements. To dig into these aspects, our final analysis relies on accuracy, which is exclusively focused on gender-marked words.
Accuracy. The results shown in Table 4 are not only consistent with the BLEU ones, but also highlight differences that were previously indistinguishable. While the All/Diff BLEU results for End2End and Cascade were identical on both languages, the All/Diff accuracy scores show that, although End2End performs better than Cascade for En-It, it performs worse for En-Fr. Also, with regards to Cascade+Tag, we can see that the Diff value is higher on the Masculine subset, thus showing that also this system is affected by gender bias, although to a lesser extent.
We now focus on systems' results on the two
categories represented in MuST-SHE: Category 1, where the information necessary to disambiguate gender can be recovered from the audio (speaker talking about him/herself) and Category 2, where such information occurs in the utterance content (speaker talking about someone else). Results are shown in Table 5.
As for Category 1, Diff values show that Cascade performance is the worst on both languages. This is due to the fact that its MT component cannot access the speaker's gender information necessary for a correct translation. This weakness becomes particularly evident in the Feminine class, where the higher values on the Wrong datasets (leading to negative values in columns Feminine/Diff) highlight a strong bias towards producing masculine forms. Although still negative for the Feminine class, the much better Diff values obtained by End2End show its ability to leverage audio features to correctly translate gender. However, the gap with respect to Cascade+Tag - by far the best system in Cat. 1 - is still large. On one side, End2End might benefit from better audio representations. Indeed, as shown in Kabil et al. (2018), the MFCC features used by state-of-the-art models are not the most appropriate for gender recognition. On the other side, Cascade+Tag
6930
does not only take advantage of huge amounts of data to train its basic components, but it is also an oracle supported by the artificial injection of correct information about speakers' gender.
In Category 2, where having direct access to the audio is not an advantage since gender information is present in the textual transcript, results show a different scenario. While scores on the Masculine class are not conclusive across languages, on the Feminine class End2End always shows the worst performance. This can be explained by the fact that, being trained on a small fraction of the data used by the cascade systems, End2End is intrinsically weaker and more prone to gender mistranslations. Also, it is noticeable that Cascade+Tag is slightly worse than Cascade, although the MT components are trained on the same amount of data. This is due to the dataset design choice (see §3.3) to include  $\sim 50\%$  of segments where the speaker's gender does not agree with the gender of the phenomenon to translate. This feature makes MuST-SHE particularly challenging for systems like End2End and Cascade+Tag since, in these specific cases, speaker's gender information (extracted from the source audio or artificially injected) is not relevant and can introduce noise.
All in all, translating gender is still an issue in ST and current technologies are affected by gender bias to variable extent. Through the analysis made possible by MuST-SHE, we have been able to pinpoint their specific strengths and weaknesses and pave the way for more informed future studies.
If, like human beings, "machine learning is what it eats", the different "diet" of MT and ST models can help them to develop different skills. One is the proper treatment of gender, a problem when translating from languages without productive grammatical gender into gender-marked ones. With respect to this problem, by eating parallel texts during training, MT performance is bounded by the statistical patterns learned from written material. By eating (audio, text) pairs, ST has a potential advantage: the possibility to infer speakers' gender from input audio signals. We investigated for the first time the importance of this information in ST, analysing the behaviour of cascade (the state of the art in the field) and end-to-end ST technology (the emerging approach). To this aim, we created MuST-SHE, a benchmark annotated with different
types of gender-related phenomena in two language directions. Our evaluation shows that, in spite of lower overall performance, the direct approach can actually exploit audio information to better handle speaker-dependent gender phenomena. These are out of reach for cascade solutions, unless the MT step is supplied with external (not always accessible) knowledge about the speaker. Back to our title: if, in ST, gender is still in danger, we encourage our community to start its rescue from MuST-SHE and the findings discussed in this paper.
This work is part of the project "End-to-end Spoken Language Translation in Rich Data Conditions", which is financially supported by an Amazon AWS ML Grant. We thank our colleague Marco Matassoni for providing the automatic transcripts used for our experiments with cascade systems.
Tanel Alumäe and Mikko Kurimo. 2010. Efficient Estimation of Maximum Entropy Language Models with n-gram Features: an SRILM Extension. In Proceedings of INTERSPEECH 2010, pages 1820-1823, Chiba, Japan.
Alexandre Bérard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016. Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation. In Proceedings of the NIPS Workshop on end-to-end learning for speech and audio processing, Barcelona, Spain. NIPS.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Yang Trista Cao and Hal Daumé III. 2019. Toward Gender-Inclusive Coreference Resolution. arXiv preprint arXiv:1910.13913.
Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On Measuring Gender Bias in Translation of Gender-neutral Pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173-181, Florence, Italy. Association for Computational Linguistics.
Greville G. Corbett. 1991. Gender. Cambridge University Press, Cambridge, UK.
8https://ict.fbk.eu/ units-hlt-mt-e2eslt/
6931
Kate Crawford. 2017. The Trouble with Bias. In *Conference on Neural Information Processing Systems (NIPS) - Keynote*, Long Beach, California.
Steven Davis and Paul Mermelstein. 1980. Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4):357-366.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017, Minneapolis, Minnesota. Association for Computational Linguistics.
Mattia A. Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019b. Robust Neural Machine Translation for Clean and Noisy Speech Transcripts. In Proceedings of the 16th International Workshop on Spoken Language Translation (IWSLT), Hong Kong.
Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2019c. Adapting Transformer to End-to-end Spoken Language Translation. In Proceedings of INTER-SPEECH 2019, Graz, Austria.
Matthias Eck and Chiori Hori. 2005. Overview of the IWSLT 2005 Evaluation Campaign. In Proceedings of the 2nd International Workshop on Spoken Language Translation (IWSLT), Pittsburgh, Pennsylvania.
Mostafa Elaraby, Ahmed Y. Tawfik, Mahmoud Khaled, Hany Hassan, and Aly Osama. 2018. Gender Aware Spoken Language Translation Applied to English-Arabic. In Proceedings of the 2nd International Conference on Natural Language and Speech Processing (ICNLSP), pages 1-6, Algiers, Algeria.
Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147-154, Florence, Italy. Association for Computational Linguistics.
Charles F. Hockett. 1958. A Course in Modern Linguistics. Macmillan, New York, US.
Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2019. Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning. arXiv preprint arXiv:1911.04283.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, pages 448-456, Lille, France.
Selen Hande Kabil, Hannah Muckenhirn, and Mathew Magimai-Doss. 2018. On Learning to Identify Gen-ders from Raw Speech Signal Using CNNs. In Proceedings of INTERSPEECH 2018, pages 287-291, Hyderabad, India.
Svetlana Kiritchenko and Saif Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana.
Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. 2018. Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation. In Proceedings of LREC 2018, Miyazaki, Japan.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278-2324.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2019. Gender Bias in Neural Natural Language Processing. arXiv preprint arXiv:1807.11714v2.
Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49-54, Florence, Italy. Association for Computational Linguistics.
Jan Niehues, Roldano Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. The IWSLT 2018 Evaluation Campaign. In Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT), Bruges, Belgium.
Jan Niehues, Roldano Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico. 2019. The IWSLT 2019 Evaluation Campaign. In Proceedings of the 16th International Workshop on Spoken Language Translation (IWSLT), Hong Kong, Hong Kong.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR Corpus Based on Public Domain Audio Books. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, Brisbane, Australia.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL 2002, pages 311-318, Philadelphia, Pennsylvania. Association for Computational Linguistics.
6932
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A Simple Data Augmentation Method for Automatic Speech Recognition. arXiv preprint arXiv:1904.08779.
Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved Speech-to-Text Translation with the Fisher and Callhome Spanish-English Speech Translation Corpus. In Proceedings of the 10th International Workshop on Spoken Language Translation (IWSLT), Heidelberg, Germany.
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The Kaldi Speech Recognition Toolkit. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 1-4, Big Island, Hawaii.
Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur. 2016. Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI. In Proceedings of INTERSPEECH 2016, pages 2751-2755, San Francisco, California.
Marcelo O. R. Prates, Pedro H. C. Avelar, and Luís C. Lamb. 2018. Assessing Gender Bias in Machine Translation: a Case Study with Google Translate. Neural Computing and Applications, pages 1-19.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018. How2: A Large-scale Dataset For Multimodal Language Understanding. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL), Montreal, Canada. NIPS.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. arXiv preprint arXiv:1508.07909.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of AMTA, pages 223-231, Cambridge, Massachusetts.