Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation (2024)

Abstract: End-to-end speech translation (ST) presents notable disambiguation challenges as it necessitates simultaneous cross-modal and crosslingual transformations. While word sense disambiguation is an extensively investigated topic in textual machine translation, the exploration of disambiguation strategies for ST models remains limited. Addressing this gap, this paper introduces the concept of speech sense disambiguation (SSD), specifically emphasizing homophones words pronounced identically but with different meanings. To facilitate this, we first create a comprehensive homophone dictionary and an annotated dataset rich with homophone information established based on speech-text alignment. Building on this unique dictionary, we introduce AmbigST, an innovative homophone-aware contrastive learning approach that integrates a homophone-aware masking strategy. Our experiments on different MuST-C and CoVoST ST benchmarks demonstrate that AmbigST sets new performance standards. Specifically, it achieves SOTA results on BLEU scores for English to German, Spanish, and French ST tasks, underlining its effectiveness in reducing speech sense ambiguity. Data, code and scripts are freely available at https://github.com/ytf-philp/AmbigST.

heading: 1 Introduction
text: Speech translation (ST) translates acoustic speech from one language to another and sees a steady increase in its application (Duong et al., 2016; Bérard et al., 2016; Anastasopoulos and Chiang, 2018; Ansari et al., 2020; Li et al., 2021b; Bentivogli et al., 2021). Particularly, recent research involves the development of unified end-to-end ST models, leveraging the joint pre-training of speech and text (Wang et al., 2020b; Dong et al., 2021a,b; Inaguma et al., 2021; Tang et al., 2022; Zhang et al., 2022b).
∗Corresponding Author
This approach effectively mitigates the challenges posed by the limited availability of ST data.
Despite advances in end-to-end ST, these systems struggle with managing concurrent crossmodal and cross-lingual conversions, suffering from ambiguity in both acoustics and semantics. This challenge mirrors word sense disambiguation (WSD) in textual machine translation (Rios Gonzales et al., 2017; Luan et al., 2020; Campolungo et al., 2022). Specifically within ST, a paramount obstacle is the disambiguation of homophones - words possess identical phonetic properties but hold distinct semantic interpretations.
Traditionally, numerous ST approaches were reliant on cascade models, where homophone ambiguities were addressed either at the speech recognition stage (Ghosh et al., 2016; Zheng et al., 2020) or the machine translation stage (Xue et al., 2020; Qin et al., 2021; Liu et al., 2018). This delineated approach is in contrast to the integrated philosophy of end-to-end ST. Although several end-to-end ST studies (Zhang et al., 2021; Bang et al., 2022; Zhang et al., 2023a) try to incorporate context as a mechanism to enhance homophonic clues, such indirect methods restrict optimal disambiguation to the scope of understanding context. Consequently, semantic ambiguities persist as a notable source of errors in modern ST models.
To address this pressing concern, our study focuses on the issue of homophones, framing it as a disambiguation task that we term Speech Sense Disambiguation (SSD). Our objective is to explore the question: how can we efficiently leverage SSD in ST models? As an initial step, we develop comprehensive homophone dictionaries for the English (En), French (Fr), German (De), and Spanish (Es) languages and generate six language-pair annotated speech translation datasets through speechtext alignments. To bolster the model’s capacity for recognizing and processing ambiguous words, we introduce AmbigST, a novel homophone-aware
8020
contrastive learning methodology. This approach intertwines a homophone-aware masking strategy with contrastive learning, operating at the levels of individual tokens, entire sentences, and the ST model. Through this, the model robustly discerns homophone details adjacent to ambiguous tokens by consistently directing the extraction of contextsensitive representations from speech, thus effectively addressing semantic ambiguity.
We integrate AmbigST within a robustly pretrained model (Zhang et al., 2022b) to evaluate its efficacy across various datasets. Our evaluation encompasses En-{De, Es, Fr} ST tasks within the MuST-C dataset, as well as {De, Es, Fr}-En ST tasks within the CoVoST datasets. AmbigST consistently outperforms the strong baseline across all metrics. Specifically, this substantial improvement not only sets a new performance standard, surpassing the current state-of-the-art, but also makes significant strides in addressing the pervasive problem of speech sense ambiguity in ST tasks. The main contributions of this paper are:
• We focus on addressing the intricate problem of semantic disambiguation in ST and define a task called SSD, which specifically aims to disambiguate homophones where the underlying surface form is different.
• We construct comprehensive homophone dictionaries for {En, Fr, De, Es}, and generate annotated datasets for six ST language-pairs encompassing a total of 1M instances.
• We propose a novel AmbigST method with the homophone-aware masking strategy and multi-level contrastive learning methodology, for conducting effective SSD in ST models.
• Further analysis shows AmbigST boosts the translation of low-frequency words and shorter sentences, which often encounter more pronounced ambiguity issues.

heading: 2 Related Work
text: End-to-End ST Addressing the inherent challenges of error propagation and high latency in cascaded speech translation systems, Bérard et al. (2016); Duong et al. (2016) underscore the viability of end-to-end ST models, eliminating the need for intermediary transcription. This methodology gains significant attention recently (Vila et al., 2018; Salesky et al., 2018, 2019; Gangi et al., 2019;
Di Gangi et al., 2019b; Bahar et al., 2019; Inaguma et al., 2020). However, due to the high cost associated with data collection, ST data is often scarce. A multitude of studies focus on enhancing model performance under the constraint of limited data. For instance, existing works employ multitask learning (Le et al., 2020; Vydana et al., 2021; Ye et al., 2021) and leverage monolingual data (Deng et al., 2023) to share knowledge across different tasks, implement curriculum learning to enhance the robustness of the ST model (Kano et al., 2017; Wang et al., 2020b), and combine self-supervised learning with semi-supervised learning for speech translation (Wang et al., 2021; Bapna et al., 2022). To bridge the modality gap, Han et al. (2021); Huang et al. (2021); Xu et al. (2021); Yu et al. (2023) introduce further encoding of acoustic states, which adapt more aptly to the decoder. Most recently, the unified pretraining of speech and text emerges as a dominant paradigm (Zheng et al., 2021; Xu et al., 2021; Zhang et al., 2022b). In this paper, we introduce a novel approach, AmbigST, explicitly designed to address ambiguity challenges in the unified end-to-end ST model effectively.
Semantic Disambiguation Ambiguous words in translation pose a challenge as the model needs to determine the appropriate meaning in the given context (Rodd et al., 2002). Studies have been dedicated to addressing the intricate issue of WSD within the scope of textual machine translation (Navigli, 2009; Rios Gonzales et al., 2017; Luan et al., 2020). A parallel challenge within ST involves the handling of homophones. The prevalent approaches predominantly employ cascade models, wherein homophone ambiguities are addressed at the stage of speech recognition (Ghosh et al., 2016; Zheng et al., 2020), or machine translation (Li et al., 2018; Liu et al., 2018; Xue et al., 2020; Qin et al., 2021). In end-to-end models, the research centers on extracting contextual information. Strategies such as incorporating an additional context encoder (Bang et al., 2022), utilizing context in the output (Hussein et al., 2023), integrating document information (Zhang et al., 2021), and optimizing the connectionist temporal classification loss (Zhang et al., 2023a) are explored. Unlike these studies, our research emphasizes preserving the inherent model architecture and eliminating the necessity for supplementary modifications.
Contrastive Learning Our methodology is inspired by the recent advancements in contrastive
representation learning. In the NLP area, contrastive frameworks are employed for tasks such as pretraining (Gao et al., 2021; Zhong et al., 2023), sentence representation learning (Fang et al., 2020; Shen et al., 2020; Wu et al., 2022b; Yan et al., 2021), retrieval (Rao et al., 2023; Lei et al., 2023), and machine translation (Pan et al., 2021; Zhang et al., 2022a; Wu et al., 2022a). More recently, contrastive learning is also applied to cross-modal topics in speech translation (Dong et al., 2019; Zhou et al., 2020; Li et al., 2021a; Ouyang et al., 2022). Unlike prior research focusing on the contrast between speech and text representations, our method centers on the speech representation itself.

heading: 3 Speech Sense Disambiguation
text: Typically, a speech translation corpus includes triples of speech-transcription-translation, denoted as D = (S,X ,Y). We leverage transcription X as the basis to explain the notion of speech sense ambiguity. Consequently, we curate a dedicated dataset that embodies homophone information, thereby providing valuable resources to tackle the issue of ambiguity in ST tasks.
Problem Definition Homophones, for example, “I” versus “eye” and “would” versus “wood”, are prevalent linguistic phenomena (Zhang et al., 2021; Chung et al., 2022). SSD’s process entails discerning the specific meaning of a homophone, which shares its pronunciation with other words but carries a distinct meaning. Given the auditory context a, the goal of SSD is to identify the correct speech sense ê from a defined set of potential senses E for a given speech sequence s. This process can be mathematically formulated as:
ê = argmax ei∈E
P (ei|s,a), (1)
where ei denotes the predicted speech sense of the sequence s given the auditory context a. SSD plays a crucial role due to the varying translations associated with different speech senses. To conduct SSD in ST, we propose to construct homophone dictionaries and then use them to annotate ST datasets.
Annotating the Dataset Figure 1 provides the process of constructing a homophone dictionary. We use transcriptions X to construct a homophone dictionary by identifying sets of words that possess identical phonemes, facilitated by the utilization of the Montreal Forced Aligner (McAuliffe et al., 2017), an open-source tool engineered for the accurate alignment of speech with its corresponding orthographic transcription. The homophone dictionary consists of sets of words that share the same pronunciation. For instance, the phonemes of “P EH R” would include terms such as pear, pair. Similarly, the phonemes of “P L EY N” would encompass terms like plain, plane. The annotated dataset, organized in quintuples, highlights ambiguous words and their positions within sentences. A detailed example is presented in Table 1. We annotated the datasets from the MuST-C dataset (Di Gangi et al., 2019a) covering translations from En-{De, Es, Fr} and CoVoST dataset (Wang et al., 2020a) covering translations from {De, Es, Fr}-En. To ensure easy and automatic implementations, we design our process to be as streamlined as possible. The annotation process can also be easily applied to other datasets. We provide the statistics of our constructed homophone dictionary and annotated data in Appendix A.1.

heading: 4 AmbigST

heading: 4.1 Model Architecture
text: Inspired by the latest advancements in end-to-end ST (Zhang et al., 2022b; Fang et al., 2022), our work restructures the foundational model into three distinct components: the speech encoder, the translation encoder, and the translation decoder. The speech encoder first compresses speech representations into hidden states. These hidden states subsequently serve as inputs for the translation encoder, yielding enriched semantic information derived from the condensed speech data. The translation decoder generates the result based on the output of the translation encoder. Furthermore, our model incorporates pre-trained parameters from a unified speech-text pre-training methodology (Zhang et al.,
2022b), enhancing its effectiveness in speech translation tasks. An overarching visual depiction of our proposed methodology can be found in Figure 2.

heading: 4.2 Homophone-aware Masking Strategy
text: As mentioned in §3, the conventional ST model encounters difficulties in handling speech sense ambiguity. Consequently, the model struggles to accurately capture the intended semantic properties of these ambiguous tokens. To alleviate the aforementioned challenge, we introduce a novel homophone-aware masking strategy leveraging the annotated dataset we have constructed. Given an speech, the speech encoder receives the original sequence s as input and produces its contextual representation H = [h1,h2, ...,hI]. We employ a wordlevel forced alignment technique (Fang et al., 2022) between the speech and transcriptions to pinpoint the temporal occurrence of individual words within the speech segment. Consequently, the homophoneaware masking matrix of the speech representation denoted as m = [m1,m2, ...,mI], is generated in line with a homophone dictionary to identify the
precise location of homophonic segments in the speech representation. With a certain probability p∗, the calculation of the masking matrix is:
mi = { 1 i ∈ index, p > p∗ 0 else , (2)
where p is sampled from the uniform distribution U(0, 1), index represents the index set of the homophone representation in the speech sequence.

heading: 4.3 Homophone-aware Contrastive Learning
text: Token-level Learning We argue that the precise semantics of individual small homophone units are crucial to address speech sense ambiguity effectively. Our objective is to utilize homophone information to advance progress in contrastive learning, with a specific focus on intricate levels of granularity. Inspired by Su et al. (2022), we propose a tokenlevel contrastive learning method. We utilize the identical model to generate the speech encoder output twice. In one instance, we apply a homophoneaware masking strategy to generate a masked representation, denoted as H̃ = [h̃1, h̃2..., h̃I]. The
proposed token-aware contrastive learning objective is then defined as:
LToken = − I∑
i=1
I(h̃i) log ecosine(h̃i,hi)/τ
∑I j=1 e cosine(h̃i,hj)/τ ,
(3) where I(h̃i) is an indicator function that evaluates to 1 if h̃i represents a masked homophone, and 0 otherwise. τ is a temperature hyper-parameter and cosine(·, ·) computes the cosine similarity. The underlying intuition of this approach is to encourage the generation of masked token that closely aligns with the corresponding homophone generated by the model while remaining distinguishable from other tokens within the sequence. As a result, this approach enhances the model’s capacity to effectively understand the distinctive characteristics associated with homophones.
Sentence-level Learning To further enhance the effectiveness of contrastive learning and identify the optimal sentence-level representation, we introduce a self-supervised approach that focuses on sentence-level contrastive learning. Our approach incorporates the application of dropout noise (Gao et al., 2021) in conjunction with our proposed homophone-aware masking strategy. Subsequently, we average both H and H̃ across the temporal dimension. As a result, we obtain sentence-level representations for both the original and homophonemasked forms, denoted as z and z̃. For a minibatch of K examples, the sentence-level contrastive learning objective of n-th sentence is defined as:
LSentence = − log ecosine(zn,z̃n)/τ
∑K k=1 e cosine(zn,z̃k)/τ . (4)
Employing this objective enables the model to grasp sophisticated semantic nuances by comprehensively considering the wider context and interconnections inherent within a sentence.
Model-level Learning We propose a refined model-level contrastive learning framework to ensure consistent guidance in extracting contextaware representations from speech. Crucially, this framework is specifically designed to tackle the challenges presented by homophones and assist the model in robustly identifying the information present near ambiguous tokens. Unlike the traditional knowledge distillation approach (Kim and Rush, 2016) matching predictions of a single sample from two networks, we enhance model performance by leveraging the inherent knowledge of a
single network through predictions of diverse samples. i.e., self-knowledge distillation (Zhang et al., 2019; Wang et al., 2022). This strategy can be seen as a unique variant of contrastive learning characterized by the presence of positive examples alone. Formally, given the original and masked contextual representations obtained from the speech encoder, the model-level contrastive learning objective is defined as:
LMCL = |y|∑
j=1
KL ( Pθ(yj |y<j ,H)∥Pθ(yj |y<j , H̃) ) ,
(5) where KL denotes the Kullback-Leibler (KL) divergence. Pθ(yj |y<j ,H) is the predicted probability distribution of the j-th target token given the speech representation H as the input of translation encoder, and Pθ(yj |y<j , H̃) is that given the masked representation as input. Therefore, by incorporating ST loss LST, the ultimate training objective LAmbig can be stated as follows:
LAmbig = λ( LST + αLMCL) + (1− λ)(LToken + LSentence), (6)
where λ, α is the coefficient weight to control LAmbig. We use them to ensure a balanced distribution across diverse tasks, which in turn enhances the proficiency of disambiguation processes.

heading: 4.4 Discussion
text: In summary, our proposed three-level contrastive learning approach fundamentally focuses on refining training data from multiple granularities. The token-level method enhances the representation of ambiguous words, while sentence-level methods capture more detailed contextual features, and model-level optimization directly refines the model’s parameters. Importantly, there is no expansion in model parameters or training data size, and no influence on decoding speed, making it easily integrated into existing frameworks.

heading: 5 Experiments

heading: 5.1 Experimental Setups
text: Dataset Our experiment are conducted on the MuST-C dataset (Di Gangi et al., 2019a), a multilingual speech translation corpus featuring over 385 hours of TED Talks speech in English with corresponding manual transcriptions and translations. For our specific analysis, we select three language pairs of En-{De, Es, Fr}. The dev set is employed
for validation and the tst-COMMON set for testing. Additionally, we utilized the CoVoST-2 ST dataset (Wang et al., 2020a), another extensive multilingual ST corpus, focusing on translations of {De, Es, Fr}-En to further assess the model’s translation accuracy across different linguistic contexts.
Pre-processing For speech input, we utilize raw 16-bit, 16kHz mono-channel audio waves. From the transcripts within the training set, we construct the homophone dictionaries, then we annotate dataset through a speech-text alignment process using the Montreal Forced Aligner1. For each translation direction, we use a unigram SentencePiece2 to learn a vocabulary on the text data from the ST dataset. The vocabulary is shared for source and target with a size of 10K.
Model Configuration Our model consists of three modules. For the speech encoder, we use Hubert (Hsu et al., 2021). It adheres to the base configuration, utilizing six transformer layers with a hidden dimension size of 512. For the translation encoder, we use Ne = 6 transformer encoder layers. For the translation decoder, we use Nd = 6 transformer decoder layers. Each of these transformer layers comprises 512 hidden units, 8 attention heads, and 3,072 feed-forward hidden units.
Training and Inference During model training, we load a speech-text pre-training model3. The max-token is set to 800K (50 seconds), and we drop the training samples longer than that. We apply a speech masking probability of 5% and set label smoothing to 0.3. The learning rate increases linearly to 3e − 5 in the first 5K steps, then decays linearly to zero in total 50K steps. The masking probability p is set at 0.5, a value determined through validation results among options of 0.3, 0.5, and 0.7. The temperature parameter, τ , is set at 0.01 and selected based on validation scores of 0.01, 0.1, and 1.0. The weight parameter, α, is set to 1.0 for En-De and 0.5 for other tasks, with the parameter λ being fixed at 0.9 across all tasks. We implement our models based on fairseq4. Models train on 2 GPUs with an update frequency of 16.
During inference, we average the checkpoints of the last five epochs for evaluation. We employ
1https://github.com/MontrealCorpusTools/ Montreal-Forced-Aligner
2https://github.com/google/sentencepiece 3https://github.com/microsoft/SpeechT5/tree/
main/SpeechUT 4https://github.com/pytorch/fairseq
beam search with a beam size of 10 to generate optimal results. For the computation of evaluation metrics, we utilize sacreBLEU5 (Post, 2018) to assess translation quality (Papineni et al., 2002), APT (Miculicich Werlen and Popescu-Belis, 2017) for evaluating homophone accuracy and BLEURT score6 (Sellam et al., 2020) for measuring the quality of the generated text. We conduct the statistical significance with paired bootstrap resampling (Koehn, 2004) for BLEU score, paired t-tests (Hsu and Lachenbruch, 2014) for homophone accuracy, and Z-tests for BLEURT score.

heading: 5.2 Results on En-XX ST Tasks
text: Table 2 shows the results on MuST-C En→De, En→Fr, and En→Es tst-COMMON set. Models (1) to (5) represent the existing benchmarks. Model (6) represents our implementation of the previously state-of-the-art method, SpeechUT, in which we initialize the pre-trained parameters and fully finetune them. One should note that all our implemented systems share the same training hyperparameters and steps.
Sub-module Results We decompose AmbigST into sub-modules, starting with Model (7) employing model-level CL for foundational comprehension. Model (8) integrates sentence-level CL into Model (7), followed by Model (9), which replaces sentence-level CL with token-level CL to examine the impact of finer-grained methods. The results reveal that the individual application of each submodule can lead to improvements.
Overall Results Model (10), denoted as AmbigST, incorporates all three proposed CL methods. AmbigST surpasses all strong baselines across all these three tasks, achieving an average BLEU score of 35.5, establishing a new state-of-the-art performance. Notably, AmbigST achieves marked improvements (+0.9 on average) in the more reliable metric BLEURT, indicating that it addresses more semantically complex difficulties rather than surface-level ones, which tend to pose more significant challenges. Subsequently, we assess the accuracy of homophone translation using our constructed homophone dictionary. AmbigST significantly and consistently outperforms the baseline across all language directions. This suggests that AmbigST can address the pervasive problem of speech sense ambiguity in ST tasks.
5https://github.com/mjpost/sacrebleu 6https://github.com/google-research/bleurt

heading: 5.3 Results on XX-En ST Tasks
text: Table 3 shows the results on the CoVoST test sets. We began by initializing the parameters from En→De, En→Es, En→Fr pretrained model (Zhang et al., 2022b), subsequently fine-tuning these parameters with out proposed contrastive learning methods. AmbigST outperforms established baselines in terms of all the metrics. This evidence highlights AmbigST’s ability to boost model performance consistently across different languages, showcasing its language-agnostic nature.

heading: 6 Analysis

heading: 6.1 Effect of Homophone Dictionary Size
text: To evaluate the robustness of our approach against varying sizes of homophone dictionaries, we conducted experiments using the English homophone dictionary segmented into distinct scales {0.3, 0.5, 0.7}. We reconstituted the annotated En-De dataset to determine the impact of homophone dictionary size on AmbigST performance. As depicted in Figure 3, our method consistently enhances performance across different dataset sizes, demonstrating its efficacy and stability. As the size of the homophone dictionary increases, there is a correspond-
ing improvement in overall performance, affirming the critical influence of dictionary size. By employing our fully constructed homophone dictionary, we achieved peak performance across all metrics, thereby confirming the precision and effectiveness of our homophone dictionary construction process.

heading: 6.2 Variant of Homophone Setups
text: To investigate whether the enhancements are attributed to homophone-aware masking, we execute two supplementary experiments: (1) replacing the masked representation with speech-text mixup representation (Fang et al., 2022) and (2) substituting a homophone mask with a random mask.
Mixup Representation We replace the segment of speech representation corresponding with the word embedding of the homophone. The experimental results, as depicted in Table 4, demonstrate that while the “Mixup” brings a slight improvement in model performance, “AmbigST” is superior. Compared with the mixup strategy, our approach further improves the performance on all metrics, proving that the homophone-aware mask may be a more promising strategy for addressing speech sense ambiguity.
Random Strategy We randomly mask the speech representation with a 10% probability and conduct a three-level contrastive learning method. As shown in Table 4, our investigation reveals that although random masking alleviates issues with homophones to some extent, there is no significant improvement in BLEU and BLEURT scores. This effect may stem from a random mask similar to the masking strategy of BERT (Kenton and Toutanova, 2019), which allows other unmasked words to enhance their ability to capture context information and, therefore, can potentially affect the translation accuracy of homophones. However, these words are not central in a sentence. As shown in Table 5, we further calculate the n-gram score in the BLEU metric to compare the homophone mask and random mask strategies. While random masking may achieve comparable 1-gram scores, but it fails to address the core challenging words within the sentences, notably in the 2-gram, 3-gram, and 4-gram evaluations. Through our proposed homophone masking strategy, the model can more effectively capture the contextual and nuanced aspects of the language, resulting in more precise translation.

heading: 6.3 Auxiliary Textual Translation Task
text: We conducted a comprehensive comparison with methods that indirectly address homophone ambiguity. Specifically, we evaluated our approach against models utilizing cross-modal multi-task learning techniques, including text-to-text (T2T) (Ouyang et al., 2022), unit-to-text (U2T) (Kim et al., 2023; Zhang et al., 2023b), and coarse labeling for CTC (CoLaCTC) tasks (Zhang et al., 2023a). As shown in Table 6, the inclusion of auxiliary T2T and U2T tasks resulted in slight improvements in homophone accuracy and BLEURT score, albeit with a marginal reduction in BLEU score. This observation can be attributed to the highly optimized fusion of speech and text achieved through robust unified pretraining. Despite the assertion by Zhang et al. (2023a) that the CoLaCTC loss encourages the encoder to consider contextual clues, our direct strategy exhibits clear advantages. AmbigST effectively harnesses the intrinsic homophone properties within the ST model, thereby substantiating its efficacy in mitigating speech sense ambiguity.

heading: 6.4 Effect of Homophone Type
text: To examine the impact of homophone type, we categorized the ambiguous words that appeared in the En-De tst-COMMON set according to their part of speech. Subsequently, we analyze the count of ambiguous words in each category according to part of speech and assess their homophone accuracy respectively. As illustrated in Table 7, our approach proficiently disambiguates homophones across nearly all parts of speech except adjectives. AmbigST demonstrates an average enhancement of 1.7 in the homophone accuracy of conjunctions and pronouns. Furthermore, our analysis reveals that most homophones in our dataset comprise nouns (165 words) and verbs (39 words), predominantly utilized in daily discourse. Our method achieves substantial improvements in these domains, registering an average increase of 0.8 in the accuracy of homophone translation. This confirms that our method is effectively tuned to the requirements of daily oral communication.

heading: 6.5 Homophone vs. Non-homophone
text: To concisely illustrate that AmbigST can address difficulties posed by homophones, we conducted a comparative analysis using the En-De tst-COMMON set. Firstly, we selected 200 nonhomophones at random and determined their translation accuracy across the test set, averaging results from five distinct data sets to ensure reliability. Secondly, we assessed translation accuracy across all word occurrences in the test set. Then, we compared the results of non-homophones, homophones, and all words between the baseline and AmbigST. Our results, as detailed in Table 8, starkly highlight the increased challenge homophones present to baseline systems. The baseline model, indeed, has a lower translation accuracy on homophones compared to other words. While AmbigST does not significantly improve translation accuracy for non-homophones, it substantially enhances the accuracy for homophones, thereby improving over-
all translation accuracy. This enhancement substantially increases the overall translation accuracy, demonstrating AmbigST’s efficacy in effectively mitigating issues related to speech ambiguity.

heading: 6.6 Analyses of Word Frequency and Sentence Length
text: To elucidate how AmbigST improves translation quality, we employ compare-mt (Neubig et al., 2019) for a comparative analysis with a baseline model, focusing on word frequency and sentence length. Our findings, illustrated in Figure 4, reveal that AmbigST demonstrates enhanced resilience to low (less than 10) and mid-frequency (10-1,000) words, which often include a high incidence of homophones, underscoring our method’s effectiveness on SSD. Moreover, AmbigST favors shorter sentences, particularly those under 60 words. Shorter sentences might lack explicit contextual clues; however, our method enables the model to grasp fine-grained information more effectively. Consequently, it achieves stronger translation performance, even when the text contains limited contextual information.

heading: 7 Conclusion
text: To mitigate speech sense ambiguity in ST models, this paper represents a pioneering effort that involves the construction of comprehensive homophone dictionaries for En, Fr, De, Es and six language pair-annotated datasets that include pertinent homophone information. In the wake of these advancements, we introduce AmbigST, a novel homophone-aware contrastive learning approach that incorporates a homophone-aware masking strategy operating at the token, sentence, and model levels. Experimental results on the MuSTC and CoVoST benchmark demonstrate that our proposed AmbigST approach can leverage speech sense disambiguation efficiently.
Limitations
While the proposed AmbigST can effectively alleviate speech sense ambiguity, it still has the following limitations:
• Word Sense Ambiguity Not Address Our research primarily tackles homophones to explore speech sense ambiguity in end-to-end ST systems. However, we do not delve into “word sense” in this research. Future efforts will aim to investigate word sense ambiguity through our three-level contrastive method to validate the effectiveness of our method.
• Challenges in Low-Resource Languages While our methodology is intended to be language-agnostic, its efficacy may diminish in low-resource environments where transcription quality is suboptimal. Such conditions may impede the training of precise acoustic models with tools such as MFA, leading to potential inaccuracies in both the homophone dictionary and the annotated dataset, and thereby constraining model performance.
• System Performance Variability Due to regional differences and dialect variations, there is a diverse understanding of ambiguity. This necessitates training acoustic models on speech tailored to specific phonetic preferences to construct a dictionary of ambiguous words. For convenience, this work uses a standard pronunciation acoustic model from MFA, which may disadvantage those with varying pronunciation preferences.
• Performance in Specific Contexts As detailed in Section 6.6, the efficacy of our methodology appears diminished when applied to long sentences. This observation may stem from the intrinsic abundance of contextual cues within such sentences, which inherently diminishes the necessity and impact of our approach.
Ethics Statement
Our research rigorously adheres to ethical considerations in line with the ACL Ethics Policy. We employ the publicly accessible MuST-C dataset and the pre-trained SpeechUT model, both sanctioned for research use. Our study aims to address
homophone ambiguity in end-to-end speech translation, for which we develop a homophone dictionary and a detailed homophone-annotated dataset. In accordance with the original license, we release the annotated dataset under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY NC ND 4.0). For reproducibility, we offer both datasets and code to fellow researchers upon request, ensuring the objective and precise presentation of our results.

heading: Acknowledgments
text: This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62206076 and 62276077), Guangdong Basic and Applied Basic Research Foundation (Grant Nos. 2024A1515011491 and 2024A1515011205), Shenzhen Science and Technology Program (Grant Nos. ZDSYS20230626091203008, KJZD20231023094700001 and RCBS20221008093121053), and Shenzhen College Stability Support Plan (Grant Nos. GXWD20220811173340003, GXWD20220817123150002 and GXWD20220811170358002). Xuebo Liu was sponsored by CCF-Tencent Rhino-Bird Open Research Fund. We would like to thank the anonymous reviewers and meta-reviewer for their insightful suggestions.

heading: A Appendix
text: A.1 Statistic of Homophone Dictionary and Annotated Dataset
We develop homophone dictionaries for English, German, Spanish and French, categorizing homophones into three levels based on word frequency appearing in training dataset: low-frequency (less than 10), mid-frequency (10-1000), and highfrequency (more than 1000). The results are detailed in Table 9. Our analysis reveals a predominance of low-frequency words among homophones, with a particularly notable concentration in the French language. This is attributed to the characteristic of many French words ending in “s” that are not pronounced. This finding is consistent with the research presented by Mohebbi et al. (2023), which underscores the widespread occurrence of homophones in French.
We employed the constructed homophone dictionary to systematically annotate the MuST-C and CoVoST datasets, which cover En-{De, Fr, Es} and {De, En, Fr}-En language pairs, respectively. This annotation process included the word-level alignment between speech and its corresponding source transcription text, alongside the identification and indexing of homophones. The quantity of all annotated datasets is detailed in Table 10, with a total of 639k instances annotated in the MuST-C dataset and 374k in the CoVoST dataset.

heading: Future Work
text: Future efforts will aim to investigate word sense ambiguity through our three-level contrastive method to validate the effectiveness of our method. • Challenges in Low-Resource Languages While our methodology is intended to be language-agnostic, its efficacy may diminish in low-resource environments where transcription quality is suboptimal. Such conditions may impede the training of precise acoustic models with tools such as MFA, leading to potential inaccuracies in both the homophone dictionary and the annotated dataset, and thereby constraining model performance. • System Performance Variability Due to regional differences and dialect variations, there is a diverse understanding of ambiguity. This necessitates training acoustic models on speech tailored to specific phonetic preferences to construct a dictionary of ambiguous words. For convenience, this work uses a standard pronunciation acoustic model from MFA, which may disadvantage those with varying pronunciation preferences. • Performance in Specific Contexts As detailed in Section 6. 6, the efficacy of our methodology appears diminished when applied to long sentences. This observation may stem from the intrinsic abundance of contextual cues within such sentences, which inherently diminishes the necessity and impact of our approach