Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection
Authors: Dennis Fucci
Marco Gaido
Sara Papi
Mauro Cettolo
Matteo Negri
Luisa Bentivogli
Venue: EMNLP 2023 Main (2023)
Keywords: Speech translation
Gender Bias
Language Model

Abstract: When translating words referring to the speaker, speech translation (ST) systems should not resort to default masculine generics nor rely on potentially misleading vocal traits.  Rather, they should assign gender according to the speakers' preference. The existing solutions to do so, though effective, are hardly feasible in practice as they involve dedicated model re-training on gender-labeled ST data. To overcome these limitations, we propose the first inference-time solution to control speaker-related gender inflections in ST. Our approach partially replaces the (biased) internal language model (LM) implicitly learned by the ST decoder with gender-specific external LMs. Experiments on en→es/fr/it show that our solution outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms.  The gains are even larger (up to 32.0 and 3.4) in the challenging condition where speakers' vocal traits conflict with their gender.

Integrating Language Models into Direct Speech Translation: An Inference- Terrence-Timeeee Solution to Control Gender Inflection
Dennis Fucci, 1.2 Marco Gaido, Sara Papi, t.2
Mauro Cettolo, 2 Matteo Negri, Luisa Bentivogli?
UUiieesitty of Trento
Fondazione Bruno Kessler
pefuuci,mgaiidn spapi, cettolo, negri, www.bbenntioeeeee beg
Abstract
in absence of explicit cues (en: the nurse and his dog). A common instance is represented by words that refer to the first-person subject (henceforth re- ferred to as young nurse). In these cases, direct ST systems (Bérard et al., 201 8) have been shown to rely on vocal traits to determine gender inflections (Ben- tivogli et al., 2020). This, however, does not elim- inate the bias toward masculine forms and not inclusive for those individuals whose vocal proper- ties do not align with their gender, such as people with vocal impairments, children, and transgenders (Matar et al., 2016; Menezes et al., 2022). There- fore, whenever the speaker's ' gender' is known (e.g. in talks or lectures), such information should be exploited to control gender translation and avoid relying on potentially misleading physical cues.
When translating words referring to the speaker, speech translation (ST) systems should not re- sort to default masculine generics nor rely on potentially misleading vocal traits. Rather, they should assign gender according to the speak- ers' preference. The existing solutions to do so, though effective, are hardly feasible in practice as they involve dedicated model re-training on gender-labeled ST data. To overcome these limitations. we propose the first inference- time solution to control speaker-related gen- der inflections in ST. Our approach partially replaces the (biased) internal language model (LM) implicitly learned by the ST decoder with gender-specifcc external LMs. Experiments on en->es/fr/it show that our solution outperforms the base models and the best training-time mit- igation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms. The gains are even larger (up to 32.0 and 3.4) in the challenging condition where speak- ers' vocal traits conffict with their gender.'
words, such as I I'm
So far, this topic has been investigated only by Gaido et al. (2020). Their best solution consists in creating two gender-speeific specialized models by fine-tuning generic direct ST system on sentences uttered by female/male speakers. Though effec- tive, this method has inherent limitations. First, it requires parallel audio-text data labeled with speak- ers' gender, which are scarcely available and costly to collect. Second, the fine-tunings are compu- tationally demanding as they involve processing audio data, which are much longer (~-8x) than their textual equivalents (Salesky et al., 2019).
1 Introduction
The problem of gender bias in automatic translation particularly emerges when translating from gender- less notional gender languages (e.g., English) -hhich feature limited gender-specific marking- into grammatical gender languages (e.g., Spanish) which exhibit a rich lexical and morpho-syntactic system of gender (Savoldi et al., 2021). In this scenario, when gender-neutral words are translated into gender-marken words (e.g. en: the nurse es: elfla enfermerg/u), both machine translation (MT) and speech translation (ST) systems often bi- ased towards masculine stereotypical predictions (Cho et al., 2019; Prates et al., 2020; Bentivogli et al., 2020; Costa-jussà et al., 2022), especially
To overcome these limitations, first inference-time solution in direct ST control gender translation for speaker-dependent words when the speaker's gender is known. ? Our ap- proach guides gender translation by partially sub- stituting the biased internal language model implic- itly learned by the ST decoder of a base model with gender-specific external language model learned on monolingual textual data. Through experiments three language pairs (en (en->es/fr/it), demon-
propose the
'Note that, throughout the paper. when using the terms female, male, and gender we do not refer to speakers' gender identity but exclusively to their preferred linguistic expression of gender (see $8 for an in-depth discussion of this issue).
"Code and models available at https://gtt http://gttthbchooo/ hl t- Alttm//FBC mnt//Fk-fairrse under Apache License 2.0.

---

strate that, in terms of gender accuracy, our solution outperforms the base system by up to 31.0 points (for feminine forms) and on par with the best training-timee time approach (with up to 1.6 of gain for feminine forms). Its effectiveness is also confirmed when speakers' vocal traits conffict with their gen- der, with gains up to 32.0 and 3.4 over the base system and the best training-timm solution.
Shan et al., 2019; McDermott et al., 2019), we avoid those that require training-time interventions, and we resort to shallow fusion (Gülçehre et al., 2015; Gülçehre et al., 2017), an effective technique (Kannan et al., 2018; Inaguma et al., 2019) that consists in the log- -inearr combination of the poste- rior of the base model (pMgg and the prior of the ELM (PELM).
As regards the ILM removal, which previous studies already shown to amplify the performance gains yielded by ELM integration (Meng et al., 2021a,b,e; Andrés-Ferrer et al., 2021; Liu et al., 2022: Meng et al., 2023), the most critical aspect is its estimation. In fact, since the 1LM is implicitly modeled in the decoder, disentangling its contribu- tion from the rest of the network is challenging task (Variani et al., 2020; Meng et al., 202 Zeineldeen et al., 2021). Among the estimation methods demonstrated by Zeineldeen et al. (2021) to yield the best results, we select the global en- coder average, as it does not require training-ting time interventions. This method computes the ILM prior (PIlm) as:
2 ILM/ELM for Gender Translation
The autoregressive decoder of an encoder-decoder architecture is trained to predict the next target to- ken given the previous ones and the encoder output. Thereby, it implicitly learns to model the target language from the training data, thus developing an internal language model (ILM) (McDermott et al., 2019; Variani al., 2020). We assume that, in direct ST model trained on unbalanced data where female speakers (and consequently feminine speaker-dependent words) are under-represented (Tatman, 2017), the ILM is biased toward mascu- line forms. Therefore, we propose to guide the generation of the ST model with respect to speaker- dependent words by substituting the biased ILM with a gender-speeific external language model (ELM). To this aim, we train two ELMs on mono- lingual text corpora (easy to collect, unlike labelled audio data) containing either feminine or mascu- line speaker-dependent words (see $3). At infer- ence time, when we have prior knowledge of the speaker's gender from the metadata, we i) integrate the ELM specialized in either masculine or fem- inine forms (depending on the speaker's gender) into the ST model, and ii) (partially) remove the ILM contribution.
(ILM(U) PM PMgddcoder (ylc)
decoder
namely, by feeding the ST decoder with the aver- age e of the encoder outputs hn over all the T, timesteps of the N training samples, where is:
N
T
c
hadt
T
m=
Therefore, given audio input æ, the output y of our solution is the translation y that maximizes the log-linear combination of pMp PELM and PILM:
The integration of end-to-end models with ELMs is a widespread solution to leverage text data in speech recognition (Bahdanau et al., 2016; Chorowski and Jaitly, 2017: Kannan et al., 2018;
y = argmax/log PM (3/æ)) - BILM log PILM (y)
y
+BELM log logpELar()))
Irie al., 2019). Successful applications span from recognizing rare words (Sainath et al., 2021: Huang al. 2022) to coping with vocablluyy terms (Hori et al., 2017). domain adaptation (Sriram et al., 2018; Shan et al., 2019) and under-resourced con- ditions (McDermott et al., 2019). However, to the best of our knowledge, ELM integration has not been explored in the field of direct ST, nor in the context of gender translation, as we do here. Among the various methods proposed for the ELM integration (Gülçehre et al., 2015; Gülçchre et al., 2017: Sriram et al., 2018; 8; Stahlberg et al., 2018;
where BILM and SELM are positive scalar weights calibrating ELM integration and ILM removal.
The three components (pMg. PELM, and PIeM) convey different information: i) PM embeds both the acoustic and the linguistic information learned from the ST data; ii) PILM represents the estimated linguistic knowledge Iearned by MB; iii) PELM embeds linguistic information (in our case gender- specific forms) learned from external textual re- sources. Therefore, 3ILM and fELM must be set to values that effectively integrate the internal and external linguistic knowledge, so that the gender

---

Ir
train
dev
train
dev
train
dev
M
M
M
M
F
M
F
M
F
Sent.
196.8K
111.9K
1.6K
1.2K
566.9K 13.7M
232.4K
8.5K
3.3K
370.7K
171.9K
5.3K
3.0K
Words
4.1M
2.4M
37.5K
26.7K
5.5M
232.2K
87.1K
8.9M
4.2M
132.3K
75.4K
Table 1: Statistics for the monolingual text corpora collected.
bias affecting the ST decoder is mitigated by the ELM. At the same time, the linguistic contribution supplied by the ELM must not override the acous- tie modelling capabilities of pM so as to avoid translation quality drops. Accordingly, we estimate BILM and SELM by optimizing the harmonic mean of the two metrics (gender accuracy and BLEU- see 53) used to measure gender bias and overall translation quality, so as to equally weigh our two objectives. In Appendix A, discuss the computa- tion of BILM and θELM values, also showing that their precise estimation is not critical since final results are rather robust to small weight variations.
ence. To assess gender translation, we use the offi- cial MuST-SHE evaluation script", which produces two measures: i) term coverage, i.e. the percentage of annotated words that are generated by the system (disregarding their gender marking), and on which gender translation is hence automatically measur- able, and ii) gender accuracy, i.c. the percentage of words generated in the correct gender among the measurable ones. Lastly, overall translation quality is calculated with SacreBLEU (Post, 2018).
Results
For each language pair, we evaluate our approach by training: i) an ST baseline model (MB) that is not aware of the speaker's gender; ii) the spe- cialized models (Msp) presented in (Gaido et al., 2020), re-implemented as upper bound compare our inference-time solution with the best training- time approach; iii) the combination of MB with the gender-specific ELMs and the ILM removal (MB-LLM+ELLM); iv) variant of the approach, where the ILM is not removed (MB+ELM), serving as an ablation study to disentangle the ILM and ELM contributions. Detailed experimental settings and model description provided in Appendix B.
3 Data and Metrics
Our en >es/fr/it ST systems are trained on the TED-based MuST-C corpus (Cattoni et al., 2021), This resource includes manual annotation of the speakers' gender (Gaido al., 2020), which is used to determine the gender translation of speaker- dependent words, To train the ELMs, we collected GenderCrawl," a set monolingual corpora for each target language and gender. Each corpus is made of sentences with speaker-dependent words that clarify the speaker's gender (e.g., es: Soy nueva <F> esta zona [en: am new this area], es: Debia ser fiel mi mismo <M> [en: had to be true myself)). These sentences were auto- matically selected from ParaCrawl (Bañón al., 2020) through regular expressions representative morpho-syntactic patterns matching references to the first-person singular. Additionally, we have also collected validation set by applying the same regular expressions to the MuST-C training sets. The statisties of all these datasets presented in Table 1.
4.1 Main Results
Table 2 presents BLEU, term coverage, and gender accuracy scores for all language pairs, divided into feminine/masculine (F/M) forms.
Gender Accuracy. The results indicate that our approach, both with and without the ILM removal, significantly outperforms MB on all language pairs, Specifically, MR-ILM-ELM is always better than MB+ELM- demonstrating that the ILM removal in combination with ELM integration improves de- biasing. The accuracy gains of Mg-ILM+ELM over MB are particularly high on feminine forms, rang- ing from 25.4 to 31.0. In addition, the accuracy MB-I.M+ELM is comparable to that of the training- time approach Msp. While Msp is significantly superior only for M in en-it and en-fr, MB-ILM-ELM
We evaluate our systems on the TED-derived and gender-sensitive MuST-SHE benchmark (Ben- tivogli et al., 2020). In particular, we focus on its "Category 1", which contains from 560 to 607 sentences (depending on the target language) with speaker-dependent words annotated in the refer-
Availlable at htthh:tp/smmt //ttt fbk. eu/gen eu/gendercrawl/ under the Creative Commons Attribution 4.0 International license (CC BY 4.0).
"ht "tt //mt. htt.//ttfbbbft.. eu/must-she/ "case:mixee|lefimo nooo: bajssmoth.ccp//cooo. version:2.0.0

---

en-es
en-fr
en-it
Models
Coverage
Gender Acc.
Coverage
Gender Acc.
Coverage
Gender Acc.
BLEU
BLEU
BLEUU
M
F
F
F
M
F
M
F
Mn
34.8
65.1
67.9
71.6
45.7
29.8
51.5
55.9
72.5
52.0
26.8
51.6
50.6
77.3
49.5
MSp
35.2
64.8
66.8
85.6
76.8
29.9
52.9
55.2
92.4
78.5
27.3
52.8
49.4
92.5
73.3
ME-ELMI
33,7th
67.5
68.1
77.9Ab
69.2Ab
29.3
54.9
57.1
81.9
75.8
27.2
51.8
54,4AM
81.2AD
72.8*
MI-LMMELM
34.4"
65.8
71.2
82.34
76.7
29.8
54.4*
56.1
84,5AD
79.2*
27.2
52.3
54.1
84
74.9*
Table 2: BLEU (1), (term) coverage (t), and M/F gender accuracy (Gender Acc., f) scores. A/e and B/b indicate that the improvement (uppercase) or the degradation (lowercase) of our technique over the baseline (Mg) and the fine-tuning approach (Msp), respectively, is statistically significant (bootstrap resampling with 95% CI, Koehn 2004).
is the best on average for F, the most misgendered category.
erties, we simulate this condition using the pro- vided "wrong references", in which the speaker- dependent words are swapped the opposite gen- der. We treat them as correct references, so as to have female voices with masculine targets and vice versa, and we require the systems to produce the output with the gender of the target. Table 3 shows BLEU, term coverage, and gender accuracy for Mg. MSP, and our best-performing model MB-ILM-ELM, averaged over the three language pairs.
Translation Quality. Looking at BLEU scores, we notice that, with the only exception of en-it, the simple integration of the ELM (MB-ELM) degrades the quality with respect to both MB and Msp. 6 es- pecially in en-es where the drops are statistically significant. The ILM removal mostly solves the problem, as MB-ILM-ELMM achieves scores that are comparable to Msp on en-fr and en-it, and partly closes the gap on en-es, where the drop with respect MB (-0.4) is not statistically significant. Interest- ingly, looking at term coverage, both MB-ILM-ELM and MB+FLM consistently outperform MB and MSP. with the only exception of masculine words in en-it. In particular, the gains are high for feminine words, where MB-ILM-ELM significantly outperforms both Mn and Msp. This shows that the integration of tex- tual data can increase the ability to model feminine vocabulary, less represented in training data.
Gender Accuracy. Regarding gender realization,
MB-ILM+ELM performs noticeably better than MB. as we observe substantial improvement of 19.7 points in producing masculine forms (Voice F-Gdrr M) and 32.0 in producing feminine forms (Voice M Gdr F). This suggests that our approach capa- ble of partially overriding the vocal information, on which the base model unduly relies to translate the dreaker-dependent words. In comparison with MsP. our approach is inferior in Voice F-Gdr M, while it is superior in generating the less-represented femi- nine translation (Voice M-Gdr F), confirming the trends observed in the previous scenario (see $4.1).
In conclusion, our inference-time solution ef- fectively improves gender translation in direct ST, especially for feminine forms (see Appendix C for output examples). Moreover, it achieves compa- rable results with the best training-time approach, while overcoming its limitations. Such improve- ments do not come at the detriment of the overall translation quality (as shown by BLEU scores) nor of the accuracy in assigning gender to words that pertain to human referents other than the speaker (as shown in Appendix D).
Translation Quality. In terms of BLEU, our ap- proach (MR-I.M-ELM) is on par with the training- time strategy (Msp), but they both suffer ~25 BLEU drop with respect to the base system (Mp). The reason for this drop may lay on the fact that gender-specific models learned patterns that dif- ferentiate male and female language (Mulac et al.. 2001; Boulis and Ostendorf, 2005), which are dis- regarded when only swapping the gendered words in the references. However, MB-ILM+ELM outper- forms MB and MsP in terms of coverage, with a marginal gain (0.5-0.6) for male speakers (Voice M-Gdr F) and larger gain (2.3-3.7) for female speakers (Voice G- M), confirming that our ap- proach increases the coverage of the vocabulary used by females (even when expressed in the mas- culine form).
4.2 Robustness to Vocal Traits
We also evaluate the inclusivity of our solution for speakers whose vocal traits are stereotypically as- sociated with a gender opposite to their own. As MuST-SHE solely contains utterances from speak- ers whose gender aligns with their vocal prop-
"In Gaido et al. (2020), the specialized systems uchieve higher results as their base models are built using large ST, ASR, and MT corpora, while we train only on MuST-C.

---

Average
substantial efforts in annotating data, as bench- marks akin to MuST-SHE are currently unavailable for other target languages.
Models
Coverage
Gender Ácc.
BLEU
Voice F
Voice M
Voice F
Voice M
Gdr
Gdr
Gdr
Gdr F
MB
30.5
58.0
56.2
50.9
26.1
Our inference-time solution, as described in the paper, significantly reduces the computational costs of current approaches by eliminating the need for ST retraining. However, there is an increase in in- ference costs, due the additional forward passes on the ELM and IL.M (which the same as the ST decoder, but fed with a different encoder output). particular, since our implementation has not been optimized and performs the operations sequentially, our solution reduces the inference speed (computed as the number of generated tokens per second) by 400 (from 165 to 100). Such slowdown can be reduced by: i) parallelizing the forward passes of the ST model, ELM, and ILM; ii) caching com- puted states in the ILM to avoid recomputation at each generation step. Optimizing our implementa- tion, although necessary for production usage, is outside the scope of our work.
MSP
28.2
56.6
56.1
83.0
54.7
MD ILMELLM
28.0
60.3
56.7
70.6
58.1
Table 3: BLEU, term coverage, and gender accuracy for the conflicting scenario averaged over en-tes/fr/it
All in all. the experiments in this challenging testing condition prove that our solution effectively overrides the reliance of base ST systems on speak- ers' vocal traits. Also, they confirm its superiority in translating the less-represented feminine forms.
5 Conclusions
We proposed the first inference-time solution to control gender translation of speaker-dependent words in direct ST. Our approach partially replaces the biased ILM of the ST decoder with a gender- specific ELM. As such, it can be applied to existing models without the need for labeled ST data or computationally expensive re-trainings, overcom- ing the limitations of existing training-time meth- ods. Experiments on three language pairs proved the effectiveness of our technique in controlling gender inflections of words referring to the first- person subject, regardless of whether the speakers' vocal traits are aligned with their gender or not, In addition to significantly increasing the gender accuracy of base ST models, it achieves substan- tial parity with the best training-tine time method while consistently increasing the correct generation of feminine forms.
Lastly, our ELM implementation BPE (Sennrich et al., 2016) vocabulary of the ST models, trained on the textual target of MuST-C. Due to the under-repressenattin of feminine forms in this corpus, statistical segmentation methods like BPE split the less frequent feminine forms into less compact sequences of tokens (for example, in our experiments, we observed the split maes_tre vs maestro for Spanish). This tokenization pro- morphology
the same
penalize generalization and, consequently, gender translation when com- pared to character-level representations (Belinkov et al., 2020). As such, an interesting future di- rection is represented by training the ELMs with a character-basse vocabulary, which has the potential to enhance gender accuracy and further increase the significant gains already achieved.
6 Acknowledgements
This work is part of the project "Bias Mitiga- tion and Gender Neutralization Techniques for Automatic Translation",, which is financially sup- ported by an Amazon Research Award AWS AI grant. Moreover, we acknowledge the support the PNRR project FAIR Future AI Research (PE00000013), under the NRRP MUR program funded by the NexIGenerationEU.
Ethics Statement
In this paper we presented a new methodology to improve ST systems in their ability to correctly generate masculine and feminine forms for first- person-singular referents. Hereafter, we contextu- alize the impact of our research and discuss the ethical principles the basis of work.
7 Limitations
In our experiments, exclusively evaluated our approach on English Romance language trans- lations. Conducting experiments on different lan- guage pairs would be valuable. However, is im- portant to note that such endeavors would demand
We define gender bias in MT/ST as the ten- dency of systems to systematically favor mascu- line forms to the detriment of the feminine ones
Statistics computed on a p3 2xlarge instance on AWS (feattrrng one NVIDIA V100 GPU).

---

when related human entities (Crawford, 2017). This bias not only hampers the performance of the system by producing erroneous translations of gender-marker words, but also has significant so- cietal implications. For example, incorrect gender translations can impact self- perfepeer,,,ion, as linguis- tic expressions of gender play crucial role in ne- gotiating and communicating personal representa- tion (Stahlberg et al., 2007; Corbett, 2013; Gygax et al., 2019). According to Blodgett et al. (2021) and Savoldi et al. (2021), gender bias in transla- tion technologics leads to both representational harms, such as underrrrepresentation of women and diminished visibility of their linguistic repertoire. and allocational harms, characterized by unequal quality of service due to performance disparities between male and female users.
resented in the currently available ST data. In fact, to the best of our knowledge, ST corpora also rep- resenting non-binary speakers are not yet available. However, we encourage a vision of gender going beyond binarism and we helieve that extending the application of our method to non-binary forms (c.g. by integrating third, non-binary ELM) can be an interesting extension of this work.
References
Jesús Andrés-Fener, Dario Albesano, Puming Zhan, and Paul Vozila. 2021. Contextual density ratio for lan- guage model biasing of sequence to sequence ASR systems. In Interspeech 2021, 22nd Annual Con- ference of the International Speech Communication Association, Brno, Czechia 30 August 3 Septem- ber 2021, pages 2007-2011, International Speech Communication Association.
In light of the above, we believe that our solution positively impacts single individuals and society at large, by improving not only the experience of using such technologies but also feminine visibil- ity. Furthermore, by relying on explicit gender information, our mitigation solution goes beyond mere and potentially mislcading exploitation of the speech signal. Indeed, using speaker's vocal prop- erties would foster the stereotypical expectations about how masculine or feminine voices should sound, which is not inclusive for certain users, such as transgender individuals or people with laryngeal diseases (Matar et al., 2016; Pereira et al., 2018; Villas-Bóas et al., 2021; Menezes et al., 2022).
Dzmitry Bahdanau, Jan Chorowski. Dmitriy Serdyuk, Philémon Brakel, and Yoshua Bengio. 2016. End-to- end attention- large vocabulary speech recog- nition. In 2016 IEEE International Conference Acoustics. Speech and Signal Processing (ICASSP). pages 4945-4944 Institute of Electrical and Elec- tronics Engineers.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hicu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Sánchez,-änchez, Flsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acqui- sition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555 4567,, Online. Association for Computational Linguistics.
As regards possible concerns about the gender information considered in our experiments, we re- lied on the annotations of the two datasets used, MAAST-CCMSTTSSpakrrr and MuST-SHE. Both these resources have been manually annotated with speakers' gender information based on the personal pronouns found in their public TED profile (Gaido et al., 2020; Bentivogli et al., 2020). We follow the statement of the curators of these resources, thus bearing in mind that the gender tag accounts only for the linguistic gender by which the speak- ers accept to be referred to in English and to which they would like the translation to conform. We acknowledge that this information does not neces- sarily correspond to the speakers' self-determined gender identity (Cao and Daumé III, 2020). We are also aware that we cannot consider their preference static in time (Lauscher et al., 2022).
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the Linguistic Representational Power of Neural Machine Transla- tion Models. Computational Linguistics, 46(1):1-52.
Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mat- tia A. Di Gangi, Roldano Cattoni, and Marco Turchi. 2020. Gender in danger? evaluating speech transla- tion technology on the MuST-SHE corpus. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6923 6933, Online. Association for Computational Lin- guistics.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fair- ness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Pupers), pages 1004-1015, Online. Association for Computational Linguistics.
Last but not least, in this work we only consider binary linguistic forms as they are the only ones rep-

---

Constantinos Boulis and Mari Ostendorf. 2005. quan- titative analysis of lexical differences between gen- ders in telephone conversations. In Proceedings of the 43rd Anmual Meeting of the Association for Computational Linguistics (ACL'0S) pages 435-442, Ann Arbor, Michigan, Association for Computational Linguistics.
for Computational Linguistics: Main Volume, pages 690-696, Online, Association for Computational Lin- guistics.
Marco Gaido, Sara Papi. Dennis Fucci, Giuseppe Fiameni, Matteo Negri, and Marco Turchi. 2022. Fffficient yet competitive speech translation: FBK IWSLT202.. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 177-189, Dublin, Ireland. Association for Computational Linguistics.
Alexandre Bérard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224-6228. Institute of Electrical and Electronics Engineers.
Marco Gaido, Beatrice Savoldi, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2020. Breeding gender- aware direct speech translation systems. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 3951-3964, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568 4595, On- line. Association for Computational Linguistics.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu. Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruom- ing Pang. 2020. Conformer: Convolution-augmennte Transformer for Speech Recognition. In Proceed- ings of the 21st Annual Conference of the Interna- rional Speech Communication Association, pages 5036-5040, Shanghai,, China (Online). International Speech Communication Association.
Roldano Cattoni, Mattia Antonino Di Gangi, Luisa Ben- tivogli, Matteo Negri, and Marco Turchi. 2021. Must- c: A multilingual corpus for end-to-end speech trans- lation. Computer Speech & Language. 66:101-155.
Won Ik Cho, Ji Won Kim. Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop Gender Bias in Natural Language Processing, pages 173-181, Florence, Italy. Associa- tion for Computational Linguistics.
Çaglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Lofc Barrault, Huei- Cuei-Chi Lin, Fethi Bougares. Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine trans- lation. CORR, abs/ 103.055033.
Jan Chorowski and Navdeep Jaitly. 2017. Towards bet- ter decoding and language model integration in se- quence to sequence models. In Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pages 523-527. International Speech Communication Association.
Çaglar Gülçchre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017. On integrating lan- guage model into neural machine translation. Com- puter Speech & Language, 45:137-148
Greville G. Corbett. 2013. The Expression of Gender. De Gruyter.
Pascal M. Gygax, Daniel Elmiger, Sandrine Zufferey, Alan Garnham, Sabine Sczesny, Friederike von Stockhausen, Lisa Braun, and Jane Oakhill. 2019. A language index of grammatical gender dimensions to study the impact of grammatical gender on the way we perceive women and men. Frontiers in Psychol- ogy, 10.
Marta R. Costa-jussà, Christine Basta, and Gerard I. Gál- lego. 2022. Evaluating gender bias in speech transla- tion. In Proceedings of the Language Resources and Evaluation Conference, pages 2141-2147, Marseille, France. European I. Language Resources Association.
Kate Crawford. 2017. The Trouble with Bias. In Con- ference on Neural Information Processing Systems (NIPS) - KKenotee Long Beach, California, USA.
Takaaki Hori, Shinji Watanabe, and John R. Hershey. 2017. Multi-level language modeling and decoding for open vocabulary end-to-end speech recognition. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 287-293. Institute of Electrical and Electronics Engineers.
Mattia A. Di Gangi. Marco Gaido, Matteo Negri, and Marco Turchi. 2020. On target segmentation for di- rect speech translation. In Proceedings of the 14th Conference of the Association for Machine Transla- tion in the Americas (Volume 1: Research Track), pages 137-150, Virtual. Association for Machine Translation in the Americas.
W. Ronny Huang, Cal Peyser, Tara N. Sainath, Ruom- ing Pang, Trevor D. Strohman, and Shankar Kumar. 2022. Sentence-select: Large-scale language model data selection for rare- rare-word speech recognition. Interspeech 2022, 23rd Annual Conference of the In- ternational Speech Communication Association, In- cheon, Korea, 18-22 September 2022, pages 689-693, International Speech Communication Association.
Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. CTC-based compression for direct speech translation. In Proceedings of the 16th Con- ference of the European Chapter of the Association

---

Hirofumi Inaguma, Jaejin Cho, Murali Karthick Baskar, Tatsuya Kawahara, and Shinji Watanabe. 2019. Transfer learning of language-independee end ast with language model fusion. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6096- 6100. Institute of Electrical and Electronics Engi- neers.
for attention-based encoder-decoder ASR. In Inter- speech 2022, 23rd Annual Conference of the Interna- tional Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 1666-1670. In- ternational Speech Communication Association.
end-to-
Nayla Matar, Cristel Portes, Leonardo Lancia, Thierry Legou, and Fabienne Baider. 2016. Voice quality and gender stereotypes: A study on Lebanese women with Reinke's ' edema. Journal of Speech, Language, and Hearing Research, 5966):1688-1117.
Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021. Source and target bidirectional knowledge distillation for end-to-end speech trans- lation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1872-1881. Online. Association for Computational Linguistics.
Erik McDermott, Hasim Sak, and Ehsan Variani. 2019. A density ratio approach to language model fusion in and-to-end automatic speech recognition. In Pro- ceedings of 2019 IEFE Automatic Speech Recogni- rion and Understanding Workshop, pages 434-441, Sentosa. Singapore. Institute of Electrical and Elec- tronics Engineers.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language modeling with deep transform- In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 3905- 3909. International Speech Communication Associa- tion.
Damielle Pereira Menezes, Zulina Souza de Lira, Ana Nery Barbosa de Araújo, Anna Alice Figueinedo de Almeida, Adriana de Oliveira Camargo Gomes, Bruno Teixeira Moraes, and Jonia Alves Lucena. 2022. Prosodic differences in the voices of transgen- der and cisgender women: Self-perception of voice- an auditory and acoustic analysis. Journal of Voice.
Anjuli Kannan, Yonghui Wu, Patrick Nguyen. Tara N. Sainath, ZhiJeng Chen, and Rohit Prabhavalkar. 2018. An analysis of incorporating an external language model into a ceateenceto-sqqueeee
Zhong Meng, Naoyuki Kanda, Yashesh Gaur, Sarangara- jan Parthasarathy, Fric Sun, Liang Lu, Xie Chen, Jinyu Li, and Yifan Gong. 2021a. Internal lan- guage model training for domain- adaative end-to-end speech recognition. In Proceedings of 2021 IEEE In- ternational Conference Acoustics, Speech and Sig- nal Processing, pages 7338-7 7338-7342, Toronto, Canada (Online). Institute of Electrical and Electronics Engi- neers.
model. In 2018 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5824-5828. Institute of Electrical and Electronics Engineers.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Rep- resentations, San Diego, USA.
Zhong Meng, Sarangarajan Parthasarathy, Eric Sun, Yashesh Gaur, Naoyuki Kanda, Liang Lu, Xic Chen, Rui Zhao, Jinyu Li, and Yifan Gong. 2021b. Inter- nal language model estimation for domain-adaptive end-to-end speech recognition. In Proceedings of 2021 IEEE Spoken Language Technology Workshop. pages 243-250, Shenzhen, China (Online). Institute of Electrical and Electronics Engineers.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguisties.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Zhong Meng, Weiran Wang, Rohit Prabhavalkar, Tara N. Sainath, Tongzhou Chen, Ehsan Variani, Yu Zhang, Bo Li, Andrew Rosenberg, and Bhuvana Ramabhad- ran. 2023. Jeit: Joint end-to-end model and inter- nal language model training for speech recognition. In ICASSP 2023 2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. Institute of Electrical and Elec- tronics Engineers.
Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome the modern world of pronouns: Identity- inclusive natural language processing beyond gender. CoRR, abs/2202.11923..
Zhong Meng, Yu Wu, Naoyuki Kanda, Liang Lu, Xie Chen, Guoli Ye, Eric Sun, Jinyu Li, and Yifan Gong. 2021c. Minimum word error rate training with lan- guage model fusion for end-to-end speech recogni- tion. In Proceedings of the 22nd Annual Conference of the International Speech Communication Associa- tion, pages 2596 2600, Brno, Czechia International Speech Communication Association.
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the modality gap for speech- to-text translation. CoRR, Abs/2010.119200
Yufei Liu, Rao Ma. Haihua Xu, Yi He, Zejun Ma, and Weibin Zhang. 2022. Internal language model estimation through explicit context vector learning

---

Anthony Mulac, James J. Bradac, and Pamela Gib- bons. 2001. Empirical Support for the Gender-as- Culture Hypothesis. Human Communication Re- search, 27:121-152.
Changhao Shan, Chao Weng, Guangsen Wang. Dan Su, Min Luo, Dong Yu, and Lei Xie. 2019. Component fusion: Learning replaceable language model com- ponent for end-to-end speech recognition system. In Proceedings of 2019 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5361-5635, Brighton, United Kingdom. Institute of Electrical and Electronics Engineers.
Myle Ott, Sergey Edunov, Alexii Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations)) pages 53, Minneapolis, Minnesota. Association for Computational Linguistics.
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2018. Cold Fusion: Training Seq2Seq Models Together with Language Models. In Pro- ceedings of the 19th Annual Conference of the Inter- national Speech Communication Association, pages 387-391, Hyderabad, India. International Speech Communication Association.
Amanda Maria Pereira, Ana Paula Dassie-Leite, Eliane Cristina Pereira, Juliana Benthien Cavichi- olo, Marcelo de Oliveira Rosa, and Elmar Allen Fug- mann. 2018. Auditory perception of lay judges about gender identification of women with reinke's 's edema. CoDAS, 30(4).
Dagmar Stahlberg, Friederike Braun, Lisa Irmen, and Sabine Sczesny. 2007. Representation of the sexes in language. In Klaus Fiedler, editor, Social Communi- cation, pages 163-187.
Psychology Press.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Felix Stahlberg, James Cross, and Veselin Stoyanov. 2018. Simple fusion: In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 204-211, Brus- sels, Belgium. Association for Computational Lin- guistics.
Return of the language model.
Marcelo O. R. Prates, Pedro H. C. Avelar, and Luís C. Lamb, 2020. Assessing gender bias in machine trans- lation: case study with google translate. Neural Computing and Applications, 32:6363-6381.
Rachael Tatman. 2017. Gender and dialect bias in YouTube'ss s automatic captions. In Proceedings of the First ACI. Workshop on Ethics in Natural Language Processing, pages 53-59, Valencia, Spain. Associa- tion for Computational Linguistics.
Tara N. Sainath, Yanzhang He, Arun Narayanan, Rami Botros, Ruoming Pang, David Rybach, Cyril Al- lauzen. Ehsan Variani, James Qin, Quoc-Nam Le- The, Shuo-Yiin Chang, Bo Li, Anmol Gulati, Jiahui Yu, Chung-Chuee Chiu, Diamantino Caseiro, Wei Li, Qiao Liang. and Pat Rondon. 2021. An efficient streaming non-re non-recurrent on-device end-to-end model with improvements to rare-word modeling. In Inter- speech 2021, 22nd Anmual Conference of the Inter- national Speech Communication Axsociation, Brno, Czechia, 30 August- - 3 September 2021, pages 1777- 1781. International Speech Communication Associa- tion.
Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley. 2020. Hybrid autoregressive trans- ducer (hat). In ICASSP 2020 2020 IEEE Interna- tional Conference on Acousties, Speech and Signal Processing (ICASSP), pages 6139-6143, Institute of Electrical and Electronics Engineers.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, Long Beach, USA. Curran Associates Inc.
Elizabeth Salesky, Matthias Sperber, and Alan W Black. 2019. Exploring phoneme-level speech representa- tions for end-to-end speech translation. In Proceed- ings the 57th Annual Meeting of the Association for Computational Linguistics, pages 1835-1841, Flo- rence, Italy. Association for Computational Linguis- tics.
Anna Paula Villas-Bóas, Karine Schwarz, Anna Martha Vaitses Fontanari, Angelo Brandelli Costa, Dhiordan Cardoso da Silva, Maiko Abel Schnei- der, Carla Aparecida Cielo. Poli Mara Spritzer, and Maria Inés Rodrigues Lobato. 2021. Acoustic mea- sures of brazilian transgender women's 's voices: A case-control study. Frontiers Psychology, 12.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli. Mat- teo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845-874.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu. Dmytro Okhonko, and Juan Pino. 2020. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the ssci-Pccific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference Natural Language Processing: System Demon- strations, pages 33-39, Suzhou, China. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.

---

Mohammad Zeineldeen, Aleksandr Glushko. Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2021. Investigating methods to improve lan- guage model integration for attention-basee based encoder- decoder ASR models. In Proceedings of the 22nd Annual Conference of the International Speech Com- munication Association, pages 2856-2860,, Brno, Czechia. International Speech Communication Asso- ciation.
BILM and 3ELM in increments of 0.05. Fig- ure shows BLEU and gender accuracy (cal- culated globally for F and M) scores for each (SILM,BELM) combination. Each heatmap de- fines a space bounded by the base ST model (bot- tom left corner: (BILM, SELM) = (0.0, 0.0)) and by the ST model with the ILM totally replaced by the gender-specific ELMs (top right corner: (BILM, (ELM) (1.0, 1.0)).
The trends are similar for all the three language directions. As for gender accuracy, ELM integra- tion appears to be more critical than ILM removal. Specifically, we observe that the accuracy improves as the value of ßELM increases. Looking at BLEU, observe diagonal ellipse-shaped trend with higher scores around the bottom left corner. This indicates that, to preserve translation quality, BILM and 3ELM should be similar and not too high. Overall, although the trends for translation qual- ity and gender accuracy differ, the two objectives share high results in the middle area.
A Contributions of BILMMBBLM
As stated in $2, our method relies on two hyper- parameters (ŚELM and BIlM)). In this section, we report their optimal values ($A.1), ) and discuss the impact of varying these values on the results (SA.2).
A.1 Optimal PIIM--BALM Combinations
In the lack of a validation set with the same charac- teristics of MuST-SHE, we used this same bench- mark for a 10-fold cross validation. At each iter- ation, we translate the held-out data with the pair (BILM, BELM) E t0.00, 0.05, 0.95, 1.00]² that maximizes the harmonic mean between gen- der accuracy and BLEU (see $2) on the validation folds. At the end of this process, the whole MuST- SHE was fairly translated and ready for evaluation, and BILM and BELM were robustly estimated.
Most importantly, we can notice that the results are not significantly affected by small variations in the weights, with wide smooth areas with similar scores and no isolated peaks. This demonstrates the robustness of our solution with respect to a suboptimal estimation of BILM and BELM.
B
ST Model and Language Models
However, in real use case, we need unique combination of BELM and BILM for each gender. Therefore, in Table 4 we report the mean values of BELM and BILM over the 10 folds for each lan- guage pair. We can notice that the optimal values are closely aligned across the three language direc- tions. In general, for MB-ILM-ELM θELM always higher than BILM- Moreover, another clear and consistent trend emerging in all language pairs is the necessity for higher ЛELM and 3TLM values when the speaker is female. In this condition, higher contribution of the ELM is required to coun- terbalance the inherent bias of the base ST model towards masculine forms.
ST Models Our direct ST models are made of 12-layer Conformer (Gulati et al., 2020) encoder, in light of its favorable results in ST (Inaguma et al., 2021), and a 6-layer Transformer (Vaswani et al., 2017) decoder. The architecture is also preceded by two ID convolutional layers with 5 as kernel size and stride 2, as per (Wang et al., 2020). We use 512 embedding features, 2,048 hidden features in the FFN, and kernel size of 31 for Conformer convolutions. In total, the ST models have 116M parameters. We trained them with an auxiliary CTC loss on the Sth encoder layer (Gaido et al., 2022) and we leveraged the CTC module to compress the sequence length (Liu et al., 2020; Gaido et al., 2021). We encoded text into BPE (Sennrich et al., 2016) using SentencePiece (Kudo and Richardson, 2018) with vocabulary size of 8,000 (Di Gangi et al., 2020), and used Adam optimizer (Kingma and Ba, 2015) (8) - 0.9, 32 0.98) and Noam learning rate (Ir) scheduler (Vaswani et al., 2017) (inverse square-root) starting from 0 and reaching the 0.002 peak in 25, 000 warm-up steps. The ST
A.2 Impact of SILM and BELM
In addition to empirically estimating BILM and also investi-
SELM through cross-validation, gated the importance of optimizing the balance between the ILM and the ELM for mitigating bias without compromising translation quality. To this end, for each language direction we com- puted the performance variations by adjusting

---

en-fr
en-it
Models
M
BILM 0,200
PELM
PILM
BELM
PILM
PELM
SILM
BELM
PILM
BELM
BILM
DELM
MB-MM-FMM
0.250
0.285
0.390
0.155
0.245
0.215
0.355
0.125
0.310
0.195
MR
0.145
0.310
0.235
0.300
0.195
0.275
Table 4: Mean of the optimal values for BrEM and BELM found using 10-fold cross validation.
Gender Accuracy
Gender Accuracy
Gender Accuracy
18
1.0
0.8
0.8
0.8
0.6
0366
BERM
00:4
0.4
0.4
0.2
0.2
0.3
0.0
0.0
0.0
0,0
0.2
1.0
0.0
0.8
PILM
PT.M
fT.M
60
70
80
90
60
70
90
60
70
80
90
BLEU
BLEU
BLEU
1.8
1.0
1.0
0.8
03
0.8
0.66
0.5
BEEM
BEE
03:4
0.4
0.4
0.2
0.2
0.2
00
00
0.0
0.
0.2
1.0
0,3
0,2
0.5
0.8
1.0
0.4
0.5
PILM
fILM
10
20
30
10
20
30
10
20
(a) en-es
(b) en-fr
(c) en-it
Figure I: BLEU and gender accuracy heatmaps with different combinations of BILM and BELM for all language pairs.
models for each language direction were trained for 50k steps 4 NVIDIA A100 GPUs (40GB of RAM) with 40k tokens per mini-batch and 2 as update frequency, and we averaged the last check- points. To implement the specialized models (Msp), we fine-tuned Mg on the masculinc/feminine par- titions of the MuST-C data, with constant of 0.001 for 7 epochs, and we averaged the last checkpoints. All our models are implemented on fairsey (Ott al., 2019).
provements the validation loss, and we average the 5 checkpoints around the best the validation set.
C Examples
In Table we report output samples that well exem- plify the behavior our models and the baseline. First, the examples in en-fr and en-it confirm the gender-accuracy improvements of our meth- ods discussed in $4.1. The outputs of the baseline (MB) contain speaker-dependent words with the wrong gender, as masculine form (fr: fatigué, en: tired) is used with female speaker in en-fr, and a feminine form (it: assunta, en: hired) with male speaker in en-it. Our solution (MB.LL+EEMM) instead, consistently generates the correct gender inflection in both cases (fr: fatiguée and it: as-
Language Models are Transformer decoders with 6 layers (23M weights) trained with the same vocabularies and hyper-parameters of MB, except for the learning rate warm-up updates that we set to 200. We carly stopped the training after epochs without im-
The gender-specific ELMs

---

Lang.
Gender
Example
SRC
felt alienated. intimidated and judged by many.
REF
Mc sentí alienada, intimidada y juzgada por muchos.
en-es
Mn
Me sentí alienada, intimidante (EN, intimidating) y juzgada por muchos,
MI-ILMHELM
Mc scntí alienada, intimidada juzgada por muchos.
MRLFIM
Me sentí aislada (EN, isolated), intimidada y juzgada por muchos.
SRC
I was tired of faking normal.
REF
J'étais fatiguée de simuler la normalité.
en-fr
My
J J'étais fatigué d'avoir I'air normal.
MRALMHLL
J' J'étais fatiguée l'avoir I' l'air normal.
J' étais fatigué d'avoir l'impression d' l'être normal (EN, of having the impression of being normal).
M
SRC
In 2007, I was hired as a curator at the Denver Museum of Nature and Science.
REF
Nel 2007, fui assunto come curatore al Denver Museum of Nature and Science.
Nel 2007 sono stata assunta come curatore al Museo d' Arte Moderna di Science
Ma
(EN. Modern Art of).
en-it
M
Nel 2007 sono stato assunto come curatore al Museo d' Arte Moderna di Science
MR-IM-FLMM
(EN. Modern Art of).
Nel 2007 sono stato assunto come curatore al Museo d' Arte Moderna di Scienza
M
(EN. of Modern Arf of Science).
Table Examples of outputs from the baseline Ma, M.FLM and MR.FIM. along with the corresponding source (SRC) and reference (REF). We indicate the correct/wrong gender translation for words on which gender accuracy is evaluated, as well as generic mistranslations of other words.
en-es
en-fr
en-it
Models
Coverage
Gender Acc.
Coverage
Gender Acc.
Coverage
Gender Acc.
M
F
M
F
M
F
M
F
M
F
M
F
Mp
72.91
68.81
82.54
63.99
66.60
59.57
84.74
68.61
58.55
60.30
81.27
64.63
Ms
73.75
66.79
82.48
65.83
64.45
58.71
83.51
69.18
57.10
59.01
82.38
68.20
MEILMMEMM
72.41
68.81
81.40
66.59
63.09*
59.78
82.87
69.87
57.74
56.87
81.44
67.36
Table 6: (Term) coverage (1) and M/F gender accuracy (Gender Acc., 1) scores for Category 2 of MuST-SHE, Ala and B/b indicate that improvement (uppercase) or the degradation (lowercase) our technique over baseline (Mg) and the fine-tuning approach (Msp), respectively, is statistically significant (bootstrap resampling with 95% CI, Kochn 2004).
sunto), even without the ILM removal (MB+EL)). This is in line with the analysis in Appendix A. where we have seen that gender accuracy mostly depends on ELM integration.
D Impact on Human Referents Other than the Speaker
Our work is dedicated to the gender translation of deeenden words i.e., those words that re- fer to the messonprsonn referent. However, the improvements in handling this aspect should not come to the detriment of the accuracy in as- signing the gender to referents different from the speaker. To ensure that this is not the case, we also evaluated the gender translation on the "Category 2" of the MuST-SHE benchmark. This contains approximately 500 sentences with the annotation of words related to third-person references, whose gender is independent from that of the speaker. The results are presented in Table 6.
Looking at the en-es example, instead, Mβ cor- rectly assigns the gender but it wrongly translates one of the adjectives referred to the speaker, us- ing the epicene term intimidante (en: intimidating) for intimidated. Similarly, the output of MB-FE although with the correct gender, contains an er- ror (alienated is rendered as aislada, en: isolated). Instead, all adjectives are correct in the output of MB-ILM+ELM, confirming its higher coverage (see $4.1) and the importance of ILM removal to avoid quality drops (see Appendix A and the BLEU scores in $4.1). The latter aspect also emerges from the errors introduced by MB-ELM with respect to Mn both in en-fr and in en-it, which are not present in the output of MB-IM-ELMM for instance, in en-fr, the translation of faking normal alters meaning, deviating to avoir I' l'impression d' être normal (en: having the impression of being normal).
As for gender accuracy, observe that all sys- tems are close for masculine forms (M), with vari- ations that are not statistically significant. The largest difference amounts 1.87 points on en- fr between the baseline (Mg) and our solution M Similarly, MB-IMM M and the spe-

---

cialized systems (Msp) achieve comparable scores on feminine forms (F) while M3 is constantly worse, with a statistically significant difference in en-it.
Looking at the term coverage, we do not see clear trends across language pairs. For F, MB-IIMMMMM suffers from significant drop in en-it with respect to Mg while it achieves the best scores in en-es and en-fr. For M, there is significant drop in en-fr, which not confirmed in the other two language pairs. In addition, the differences with Msp are always ascribable to random fluctuations.
All in all, we can conclude that our debiasing so- lution specifically designed for speaker-dependent words does not significantly alter the gender assign- ment for referents different from the speaker.


Reviews:
[{"review_id": "AY3woDrPRs", "reviewer": "Reviewer_6X26", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear, concise and easy to comprehend\n2. Research problem and hypothesis clearly defined and backed with results.\n3. The proposed inference-time solution to control gender inflections in speech translation systems for speaker-dependent words is effective and outperforms the base models and the best training-time mitigation strategy. ", "reasons_to_reject": "Null", "questions_to_authors": "", "paper_topic_and_main_contributions": "The manuscript proposes an inference-time solution to control gender inflections in speech translation systems for speaker-dependent words. The major contribution of the work is that the proposed solution replaces the internal language model typically used in exiting solutions with gender-specific external language models to assign gender according to the speaker's preference. The experimental results presented on en→es/fr/it show significant improvement in gender accuracy, as well as a gain in training computational costs, in comparison to base ST models, by up to 31.0 and 1.6 points , respectively, for feminine forms. The author(s) also reported results for where the proposed method achieved even higher accuracy and computational gains in special and challenging cases where speakers' vocal traits conflict with their gender. The manuscript concludes by discussing the possible limitations as well as the ethical principles at the basis of their work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ya88CPz4lV", "reviewer": "Reviewer_jwAu", "summary": "", "questions": "At line 39 the variable c introduced, which doesn't appear to be subsequently mentioned.   Is this needed?   Should it appear in p_{M_B} in the equation at line 153?\n\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions, as they offer us the chance to \nelucidate some aspects of our work. \n\n> its not clear that the MuST-C collections are the best resource for such work [...] MuST-SHE solely contains utterances from speakers whose gender aligns with their vocal properties' and so the condition needs to be simulated.\n\nTo the best of our knowledge, MuST-C and MuST-SHE represent the best resources to test our approach, due to the following reasons:\n1. They feature a manually curated annotation of speakers’ gender, an essential element for the task we address (see lines 181-184). \n2. MuST-SHE is the only test set within the speech domain featuring a section specifically dedicated to speaker-dependent words (see lines 196-210), which enables the evaluation of gender translations for such words. The only other test set dedicated to gender bias in ST – WinoST – lacks words referring to the speakers.\n3. MuST-SHE is also used in previous work on the topic [1], where the conflicting condition has been simulated as in our Section 4.2 (we will clarify this aspect in the camera ready). Hence, our evaluation aligns with the related studies.\n4. The absence of voices presenting vocal traits not aligned to gender identity, such as for transgender speakers, is a prevalent limitation across all existing speech datasets. Consequently, simulating conflicting conditions becomes the only possible way to guarantee a comprehensive and fair evaluation.\n\nIn light of these unique characteristics and their use in prior research, we believe that the adoption of MuST-C and MuST-SHE is well motivated and we invite the reviewer to reconsider it as a reason to reject our work.\n\n[1] M. Gaido, B. Savoldi, et al.: Breeding gender-aware direct speech translation systems. In Proc. of COLING 2020.\n\n> At line 39 the variable c introduced, which doesn't appear to be subsequently mentioned. Is this needed? Should it appear in p_{M_B} in the equation at line 153?\n\nThe variable $c$ is useful for the understanding of the global encoder average technique (described at lines 145-148). This technique represents our approach to determine the ILM contribution ($p_{ILM}$). Therefore, in line 153, $c$ contributes to $p_{ILM}(y)$, not to $p_{M_B}$. In the camera ready we will define $p_{ILM}(y) =  p_{M_B{}_{decoder}}(y|c)$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a reasonable amount of work for a short paper.    The problem is interesting,  the proposed solution is simple,   and the results are based on one of the few available relevant data sets.    ", "reasons_to_reject": "The paper starts with ambitious statement (in the abstract): `When translating words referring to the speaker, speech translation (ST) systems should not resort to default masculine generics nor rely on potentially misleading vocal traits. Rather, they should assign gender according to the speakers’ preference. '    This is laudable, but its not clear that the MuST-C collections are the best resource for such work.  Section 4.2 on `Robustness to Vocal Traits' ,  while well motivated,  seems to illustrate the problem.    The aim is to investigate `inclusivity of our solution for speakers whose vocal traits are stereotypically associated with a gender opposite to their own.'  but there is the note that `MuST-SHE solely contains utterances from speakers whose gender aligns with their vocal properties' and so the condition needs to be simulated.    There is also the conclusion that 'our approach increases the coverage of the vocabulary used by females (even when expressed in the masculine form).'    The need to tease this apart suggests the underlying data set was not designed with this task in mind.  This is worth emphasizing since, from the modelling point of view,  the proposed incorporation of different language models is straightforward, and so much of the value of the work would depend on the experimental set up and results.", "questions_to_authors": "At line 39 the variable c introduced, which doesn't appear to be subsequently mentioned.   Is this needed?   Should it appear in p_{M_B} in the equation at line 153?\n\n", "paper_topic_and_main_contributions": "The paper presents an inference time approach to controlling first-person gender inflection in speech translation.   The problem to be addressed is that 'a direct ST model trained on unbalanced data where female speakers (and consequently feminine speaker-\n095 dependent words) are under-represented the[internal language model of the translation system] is biased toward masculine forms'.  The inference time approach refers to use of a gender suitable 'external language model' to guide the correct choice of target language gender;  this is in contrast to using 'gender specific specialised models' that are fine-tuned on training data with female/male speakers.     The paper studies English -> Spanish/French/Italian based on the  MuST-C corpus.   A simple modelling technique for incorporating gender appropriate 'external' LMs into decoding is presented;    the technique appears not to be novel itself, but is apparently novel to this task.   Gains in gender accuracy and coverage (as defined in MuST-C) are reported.    The specialised models serve as a performance target;  the inference time approach gives gains relative to the baseline and in several conditions approaches performance of the specialised models.  A challenging `conflicting scenario' is also reported ,  in which the system is made to produce gendered output contrary to the original gender associated with the audio.     There is a also an extensive limitations section and a good ethics section.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mZvTMx99eJ", "reviewer": "Reviewer_maVW", "summary": "", "questions": "My questions are a repetition from the above. They are:\n\n- [A] How would we choose which ELM to pick (male/female)? Does this require us to know the speaker’s gender beforehand, i.e., at inference time? This seems like a drawback as the accuracy should be calculated after using a gender detection model in the pipeline (at least in the cases where vocal traits match speaker identity).\n- [B] What happens when a single audio file has two speakers (male and female) conversing with each other? Which ELM to pick in that case?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. Through our responses, we hope to provide compelling explanations  and convince them to revise the very low soundness score that unduly penalizes our work. \n\n> The proposed system seems to deter the model in terms of their BLEU scores (system degrades in 2 out of the 3 settings).\n\nWe respectfully disagree with the reviewer's concern about the degradation in BLEU scores. As per lines 251-252 and results in Table 1, the differences in BLEU scores are not statistically significant, neither when our method  ($M_{B-ILM+ELM}$) has a higher score (+0.4 on en-it) than the baseline, nor when the score is lower (-0.4 on en-es). Accordingly, these differences are ascribable to random fluctuations and, as stated in lines 266-268, we reiterate that our experiments demonstrate that our gender-controlling approach does not affect (neither improves, nor degrades) the overall translation quality.\n\n> How would we choose which ELM to pick (male/female)? [...] the accuracy should be calculated after using a gender detection model in the pipeline (at least in the cases where vocal traits match speaker identity).\n\nAs stated in lines 46-50 and 415-424, the use of automatic gender detection raises serious ethical concerns, as expressed in multiple authoritative sources, including the EU AI ACT [1], the EMNLP Ethics Review Questions [2], and the EMNLP Ethics FAQ [3]. This is motivated by the fact that it should never be assumed that the speakers’ vocal traits are aligned with their gender identity; *“the cases where vocal traits match speaker identity”* are known only once we know the speaker’s gender, which makes the introduction of a classifier useless. Accordingly, our approach advocates for explicit provision of gender information (see lines 50-54). This requirement is intrinsic to the task and extends beyond our method, being also shared by the alternative training-time gender-controlling approach we compare against.\n\n[1] “[...] bans on intrusive and discriminatory uses of AI systems such as […] Biometric categorisation systems using sensitive characteristics (e.g. gender, [...])”, see https://www.europarl.europa.eu/news/en/press-room/20230505IPR84904/ai-act-a-step-closer-to-the-first-rules-on-artificial-intelligence\n\n[2] “For papers using identity characteristics (e.g. gender, race, ethnicity) as variables: Does the paper use self-identifications (rather than attributing identity characteristics to participants)?”, see https://2021.emnlp.org/call-for-papers/ethics-review-questions\n\n[3] “[...] avoid either attributing (i.e. assigning, without asking) identity characteristics to the people whose language is studied (guessing their gender, race, nationality, etc)”, see https://2021.emnlp.org/call-for-papers/ethics-faq\n\n> What happens when a single audio file has two speakers (male and female) conversing with each other? Which ELM to pick in that case?\n\nIn such scenarios, the typical translation pipeline is composed of several steps. Separating segments belonging to multiple speakers conversing with each other is relevant to tasks such as automatic audio segmentation and/or speaker diarization, which fall outside the scope of this (short) paper. Furthermore, the challenges from multi-speaker scenarios are inherent to the overarching task of controlling gender translation and common to all the other approaches. Thus, addressing these challenges is not necessary to demonstrate the soundness of our approach nor to soundly compare it against other gender-controlling methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Extremely important task to solve\n- To use this method, there is no need to retrain existing ST systems. It is an inference-time solution. Additionally, it is a ubiquitous method -- it can apply to any E2E ST system.\n- The improvements using the proposed method are significant. \n- Very interesting application of existing work to remove prior bias of ILM.\n- Good coverage of hyperparameters. \n- Nice ablation study to show that the model does not solely rely on vocal traits.", "reasons_to_reject": "- The proposed system seems to deter the model in terms of their BLEU scores (system degrades in 2 out of the 3 settings). This leads me to think that while the model seems to do well on speaker specific terms/inflections, the overall translations degrade.\n- How would we choose which ELM to pick (male/female)? Does this require us to know the speaker’s gender beforehand, i.e., at inference time? This seems like a drawback as the accuracy should be calculated after using a gender detection model in the pipeline (at least in the cases where vocal traits match speaker identity).\n- What happens when a single audio file has two speakers (male and female) conversing with each other? Which ELM to pick in that case?", "questions_to_authors": "My questions are a repetition from the above. They are:\n\n- [A] How would we choose which ELM to pick (male/female)? Does this require us to know the speaker’s gender beforehand, i.e., at inference time? This seems like a drawback as the accuracy should be calculated after using a gender detection model in the pipeline (at least in the cases where vocal traits match speaker identity).\n- [B] What happens when a single audio file has two speakers (male and female) conversing with each other? Which ELM to pick in that case?", "paper_topic_and_main_contributions": "The work tries to solve the important task of translating the speaker dependent words (gender) correctly. Usually the speech translation systems (ST) are heavily biased towards the masculine forms, thereby degrading the performance of the system.\n\nThe authors propose to use an external language model (ELM) trained on gender-specific corpora. They do this in combination with effectively removing the existing bias in internal language model (ILM) (a model implicitly trained in the decoder part of the E2E ST system). All of this is done at inference time, which means there is no need to retrain existing ST systems for this technique to be effective.\n\nThe proposed system is applied to a number of settings: en->it/es/fr. The system seems to outperform the baselines both in terms of coverage of terms, and gender accuracy.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose an inference-time solution to control gender inflections in speech translation. Internal language model is replaced by a gender-specific external language model to choose gender (when generating translation) according to speaker's preference. Main result is a significant improvement in gender accuracy  (for reasonable computational cost). Paper is clear and well written, it is a reasonable contribution for a short paper.
However, there were some discussions about the overall translation quality when gender control is activated (small degradation). The paper should also discuss more on how to select the gender-specific external language model. Overall my recommendation is Accept to Main Conference or Findings
Decision: Accept-Main