Understanding and Bridging the Modality Gap for Speech Translation
Year: 2023

Abstract:
How to achieve better end-to-end speech translation (ST) by leveraging (text) machine translation (MT) data? Among various existing techniques, multi-task learning is one of the effective ways to share knowledge between ST and MT in which additional MT data can help to learn source-to-target mapping. However, due to the differences between speech and text, there is always a gap between ST and MT. In this paper, we first aim to understand this modality gap from the target-side representation differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias. We find that the modality gap is relatively small during training except for some difficult cases, but keeps increasing during inference due to the cascading effect. To address these problems, we propose the Cross-modal Regularization with Scheduled Sampling (CRESS) method. Specifically, we regularize the output predictions of ST and MT, whose target-side contexts are derived by sampling between ground truth words and self-generated words with a varying probability. Furthermore, we introduce token-level adaptive training which assigns different training weights to target tokens to handle difficult cases with large modality gaps. Experiments and analysis show that our approach effectively bridges the modality gap, and achieves promising results in all eight directions of the MuST-C dataset.1

heading: 1 Introduction
text: End-to-end speech translation (ST) aims to translate speech signals to text in another language directly. Compared to traditional cascaded methods, which combine automatic speech recognition (ASR) and machine translation (MT) models in a pipeline manner, end-to-end ST could avoid
∗Corresponding author: Yang Feng. 1Code is publicly available at https://github.com/
ictnlp/CRESS.
error propagation and high latency (Sperber and Paulik, 2020). Recently, end-to-end ST models have achieved comparable or even better results than cascaded ST models (Bentivogli et al., 2021; Anastasopoulos et al., 2021, 2022).
However, due to the scarcity of ST data, it is difficult to directly learn a mapping from source speech to the target text. Previous works often leverage MT data to help the training with multi-task learning (Ye et al., 2022; Tang et al., 2021a). By sharing encoder and decoder between ST and MT, the model tends to learn similar representations from different modalities. In this way, the auxiliary MT task can help build the source-to-target mapping. However, there remains a gap between ST and MT due to the differences between speech and text. In this paper, we measure the modality gap with representation differences of the last decoder layer between ST and MT, because the representation of this layer will be mapped into the embedding space to obtain the final translation. A significant modality gap potentially causes different predictions, which makes ST lag behind MT.
Thanks to multi-task learning, we observe that when training with teacher forcing, where both ST and MT use ground truth words as target-side contexts, the modality gap is relatively small except for some difficult cases. However, the exposure bias problem can make things worse. During inference, both ST and MT predict the next token conditioned on their previously generated tokens, which may be different due to the modality gap. Moreover, different predictions at the current step may lead to even more different predictions at the next step. As a result, the modality gap will increase step by step due to this cascading effect.
To solve these problems, we propose the Crossmodal Regularization with Scheduled Sampling (CRESS) method. To reduce the effect of exposure bias, we introduce scheduled sampling during training, where the target-side contexts are sampled be-
15864
tween ground truth words and self-generated words with a changing probability. Based on this, we propose to regularize ST and MT in the output space to bridge the modality gap by minimizing the Kullback-Leibler (KL) divergence between their predictions. This will encourage greater consistency between ST and MT predictions based on partial self-generated words, which is closer to the inference mode. Besides, to handle the difficult cases, we introduce token-level adaptive training for CRESS, where each target token is given a varying weight during training according to the scale of the modality gap. In this way, those cases with significant modality gaps will be emphasized. We conduct experiments on the ST benchmark dataset MuST-C (Di Gangi et al., 2019a). Results show that our approach significantly outperforms the strong multi-task learning baseline, with 1.8 BLEU improvements in the base setting and 1.3 BLEU improvements in the expanded setting on average. Further analysis shows that our approach effectively bridges the modality gap and improves the translation quality, especially for long sentences.

heading: 2 Background

heading: 2.1 End-to-end Speech Translation
text: End-to-end speech translation (ST) directly translates speech in the source language to text in the target language. The corpus of ST is usually composed of triplet data D = {(s,x,y)}. Here s = (s1, ..., s|s|) is the sequence of audio wave, x = (x1, ..., x|x|) is the transcription and y = (y1, ..., y|y|) is the translation. Similar to previous work (Ye et al., 2021; Fang et al., 2022), our ST model is composed of an acoustic encoder and a translation model. The acoustic encoder is a pre-trained HuBERT (Hsu et al., 2021) model followed by two convolutional layers, which are used to reduce the length of the speech sequence. The translation model follows standard Transformer (Vaswani et al., 2017) encoder-decoder architecture, where the encoder contains N Transformer encoder layers, and the decoder contains N Transformer decoder layers. We first pre-train the translation model with MT data, and then optimize the whole model by minimizing a cross-entropy loss:
LST = − |y|∑
i=1
log p(yi|s,y<i), (1)
p(yi|s,y<i) ∝ exp(W · f(s,y<i)), (2)
where f is a mapping from the input speech s and target prefix y<i to the representation of the last decoder layer at step i. W is used to transform the dimension to the size of the target vocabulary.

heading: 2.2 Multi-task Learning for ST
text: Multi-task learning (MTL) has been proven useful for sharing knowledge between text translation and speech translation (Tang et al., 2021a), where an auxiliary MT task is introduced during training:
LMT = − |y|∑
i=1
log p(yi|x,y<i), (3)
p(yi|x,y<i) ∝ exp(W · f(x,y<i)). (4)
Note that both modalities (i.e., speech and text) share all transformer encoder and decoder layers. Finally, the training objective is written as follows:
LMTL = LST + LMT. (5)

heading: 3 Preliminary Studies on Modality Gap
text: With multi-task learning, most of the knowledge of MT can be transferred to ST. However, the performance gap between ST and MT still exists. In this section, we first conduct some preliminary studies with our multi-task learning baseline model to understand where this gap comes from.

heading: 3.1 Definition of the Modality Gap
text: The gap between ST and MT is related to the prediction difference at each decoding step, while the prediction depends only on the representation of the last decoder layer. Therefore, we define the modality gap at the i-th decoding step as follows:
G(s,y<i∥x,y<i) = 1−cos(f(s,y<i), f(x,y<i)), (6) where cos is the cosine similarity function cos(a,b) = a⊤b/∥a∥∥b∥. A larger cosine similarity indicates a smaller modality gap.
To understand the extent of the modality gap, we count the distribution of G(s,y<i∥x,y<i) based on all triples (s,x,y<i) in the MuST-C (Di Gangi et al., 2019a) En→De dev set. As shown in Figure 1, the modality gap is relatively small (< 10%) in most cases, which proves the effectiveness of multi-task learning in sharing knowledge across ST and MT. However, we also observe a long-tail problem: there is a large difference between ST and MT representations in some difficult cases.

heading: 3.2 Connection between Exposure Bias and Modality Gap
text: Exposure bias, a discrepancy between training and inference, is a well-known problem in neural machine translation (Bengio et al., 2015; Ranzato et al., 2016; Wang and Sennrich, 2020; Arora et al., 2022). During training with teacher forcing, both ST and MT predict the next token conditioned on the ground truth target prefix y<i. However, during inference, the predictions of ST and MT depend on their previously generated tokens by the model itself (denoted as ŷs<i and ŷ x <i for ST and MT respectively), which might be different due to the modality gap. Furthermore, different predictions at the current decoding step result in different target prefixes for ST and MT, potentially causing even more different predictions at the next step. Such cascading effect will enlarge the modality gap step by step during inference.
To prove our hypothesis, we present the curves of the modality gap with decoding steps under teacher forcing, beam search, and greedy search strategies, respectively. As shown in Figure 2, with teacher forcing, there is no significant difference in the modality gap across steps, as both ST and MT depend on the same target prefix at any step. Hence, the modality gap G(s,y<i∥x,y<i) only comes from the difference between input speech s and text x. However, when decoding with greedy search, due to the cascading effect mentioned above, the self-generated target prefix ŷs<i and ŷ x <i become increasingly different, making the modality gap G(s, ŷs<i∥x, ŷx<i) keep increasing with decoding steps. A simple way to alleviate this problem is beam search, which considers several candidate
tokens rather than a single one at each decoding step. When there is an overlap between candidate tokens of ST and MT, the cascading effect will be reduced, thus slowing down the increase of the modality gap.

heading: 4 Method: CRESS
text: Our preliminary studies in Section 3 show that:
• The modality gap will be enlarged during inference due to exposure bias.
• The modality gap may be significant in some difficult cases.
Inspired by these, we propose the Cross-modal Regularization with Scheduled Sampling (CRESS) method to bridge the modality gap, especially in inference mode (Section 4.1). Furthermore, we propose a token-level adaptive training method for CRESS to handle difficult cases (Section 4.2).

heading: 4.1 Cross-modal Regularization with Scheduled Sampling (CRESS)
text: To bridge the modality gap during inference, we adopt scheduled sampling for both ST and MT to approximate the inference mode at training time. After that, we add a regularization loss between the predictions of ST and MT based on the part of their self-generated words as context. This allows for more consistent predictions between ST and MT during inference, thus reducing the performance gap between ST and MT. Figure 3 illustrates the main framework of our method.
Scheduled Sampling Scheduled sampling (Bengio et al., 2015), which samples between ground truth words and self-generated words, i.e., predicted words, with a certain probability as targetside context, has proven helpful in alleviating exposure bias. In general, the input at the {i+ 1}-th decoding step should be the ground truth word yi during training. With scheduled sampling, it can also be substituted by a predicted word. Next, we describe how to select the predicted word ŷsi for ST and ŷxi for MT. For ST, we follow Zhang et al. (2019) to select the predicted word ŷsi by sampling from the word distribution p(yi|s,y<i) in Equation (2) with Gumbel-Max technique (Gumbel, 1954; Maddison et al., 2014), a method to draw a sample from a categorical distribution:
η = − log(− log u), (7) ŷsi = argmax (W · f(s,y<i) + η) , (8)
where η is the Gumbel noise calculated from the uniform noise u ∼ U(0, 1). Similarly, for MT, there is:
ŷxi = argmax (W · f(x,y<i) + η) . (9)
Note that we may omit the superscript and denote the predicted word for both ST and MT by ŷi in the following.
How to select between the ground truth word yi and the predicted word ŷi? Similar to Bengio et al. (2015); Zhang et al. (2019), we randomly sample
from both with a varying probability. We denote the probability of selecting from the ground truth word as p∗. At the beginning of training, since the model is not yet well trained, we select more from the ground truth words (with larger p∗) to help the model converge. In the later stages of training, we select more from the predicted words (with smaller p∗), which is closer to the situation during inference. To achieve this, we decrease p∗ with a function of the index of training epochs e:
p∗ = µ
µ+ exp(e/µ) , (10)
where µ is a hyper-parameter. With scheduled sampling, the target-side context becomes ỹ = (ỹ1, ..., ỹ|y|), where
ỹi = { yi, p ≤ p∗ ŷi, p > p ∗ , (11)
where p is sampled from the uniform distribution U(0, 1). Using ỹs and ỹx to denote the targetside context of ST and MT respectively, the loss functions of ST and MT become:
LCRESSST = − |y|∑
i=1
log p(yi|s, ỹs<i), (12)
LCRESSMT = − |y|∑
i=1
log p(yi|x, ỹx<i), (13)
Cross-modal Regularization To bridge the modality gap in inference mode, we expect the predictions of ST and MT with scheduled sampling to be consistent. Inspired by recent works of consistency training (Liang et al., 2021; Guo et al., 2022), we regularize ST and MT in the output space. Specifically, we minimize the bidirectional Kullback-Leibler (KL) divergence between the output distributions of ST and MT at each step:
LCRESSReg = |y|∑
i=1
1 2 (DKL(p(yi|s, ỹs<i)∥p(yi|x, ỹx<i))
+DKL(p(yi|x, ỹx<i)∥p(yi|s, ỹs<i))). (14)
With the translation loss in Equation (12) and (13), the final training objective is:
LCRESS = LCRESSST + LCRESSMT + λLCRESSReg , (15) where λ is the hyper-parameter to control the weight of LCRESSReg .

heading: 4.2 Token-level Adaptive Training for CRESS
text: As mentioned above, the modality gap might be significant in some difficult cases. Inspired by the idea of token-level adaptive training (Gu et al., 2020; Xu et al., 2021b; Zhang et al., 2022a), we propose to treat each token adaptively according to the scale of the modality gap. The training objectives in Equation (12), (13), and (14) are modified as follows:
LCRESSST = − |y|∑
i=1
wi · log p(yi|s, ỹs<i), (16)
LCRESSMT = − |y|∑
i=1
wi · log p(yi|x, ỹx<i), (17)
LCRESSReg = |y|∑
i=1
1 2 wi(DKL(p(yi|s, ỹs<i)∥p(yi|x, ỹx<i))
+DKL(p(yi|x, ỹx<i)∥p(yi|s, ỹs<i))), (18)
where wi is the token-level weight defined by a linear function of the modality gap:
wi = B + S ·G(s, ỹs<i∥x, ỹx<i), (19) where B (base) and S (scale) are hyper-parameters to control the lower bound and magnitude of change of wi. In this way, cases with a large modality gap will be assigned a larger weight and thus emphasized during training. Note that the modality gap is computed on-the-fly during training.

heading: 5 Experiments

heading: 5.1 Datasets
text: ST Datasets We conduct experiments on MuSTC (Di Gangi et al., 2019a) dataset, a multilingual ST dataset containing 8 translation directions: English (En) to German (De), French (Fr), Spanish (Es), Romanian (Ro), Russian (Ru), Italian (It), Portuguese (Pt) and Dutch (Nl). It contains at least 385 hours of TED talks with transcriptions and translations for each direction. We use dev set for validation and tst-COMMON set for evaluation.
External MT Datasets We also introduce external MT datasets to pre-train our translation model in the expanded setting. For En→De/Fr/Es/Ro/Ru directions, we introduce data from WMT (Buck and Koehn, 2016). For En→It/Pt/Nl, we introduce data from OPUS1002 (Zhang et al., 2020). Table 4 in Appendix A lists the statistics of all datasets.

heading: 5.2 Experimental Setups
text: Pre-processing For speech input, we use the raw 16-bit 16kHz mono-channel audio wave. For text input, all sentences in ST and external MT datasets are tokenized and segmented into subwords using SentencePiece3. For each translation direction, the vocabulary is learned from the source and target texts from the ST dataset, with a size of 10K. For the external MT datasets, we filter out parallel sentence pairs whose length ratio exceeds 1.5.
Model Setting We use the pre-trained HuBERT model4 to encode the input audio. Two 1- dimensional convolutional layers after HuBERT are set to kernel size 5, stride size 2, and padding 2. For the translation model, we employ Transformer architecture with the base configuration, which contains 6 encoder layers and 6 decoder layers, with 512 hidden states, 8 attention heads, and 2048 feedforward hidden states for each layer. The translation model is first pre-trained with MT task using transcription-translation pairs from the ST dataset (base setting), and also sentence pairs from the external MT dataset (expanded setting).
During MT pre-training, each batch has up to 33k text tokens. The maximum learning rate is set to 7e-4. During fine-tuning, each batch contains up to 16M audio frames. The maximum learning rate
2http://opus.nlpl.eu/opus-100.php 3https://github.com/google/sentencepiece 4https://dl.fbaipublicfiles.com/hubert/hubert_
base_ls960.pt
is set to 1e-4. We use Adam optimizer (Kingma and Ba, 2015) with 4k warm-up steps. We set dropout to 0.1 and label smoothing to 0.1. The training will early stop if the BLEU score on the dev set did not increase for 10 epochs. During inference, we average the checkpoints of the last 10 epochs for evaluation. We use beam search with a beam size of 8. The length penalty is set to 1.2, 1.8, 0.6, 1.4, 0.8, 1.0, 1.4, and 1.0 for En→De, Fr, Es, Ro, Ru, It, Pt and Nl, respectively. We use scareBLEU5 (Post, 2018) to compute case-sensitive detokenized BLEU (Papineni et al., 2002) scores and the statistical significance of translation results with paired bootstrap resampling6 (Koehn, 2004). We implement our model with fairseq7 (Ott et al., 2019). All models are trained on 4 Nvidia RTX 3090 GPUs.
For scheduled sampling, the decay parameter is set to µ = 15. For cross-modal regularization, the weight parameter is set to λ = 1.0. For tokenlevel adaptive training, we did a grid search for base and scale parameters on MuST-C En→De dev set with B ∈ {0.6, 0.7, 0.8, 0.9, 1.0} and S ∈ {0.05, 0.10, 0.20, 0.50, 1.00}. Finally, we set B = 0.7 and S = 0.05 for all translation directions. We start applying token-level adaptive training after the 20th epoch during training.
5https://github.com/mjpost/sacrebleu 6sacreBLEU signature: nrefs:1 | bs:1000 | seed:12345 |
case:mixed | eff:no | tok:13a | smooth:exp | version:2.0.0 7https://github.com/pytorch/fairseq
Baseline Systems We include several strong endto-end ST models for comparison: Chimera (Han et al., 2021), XSTNet (Ye et al., 2021), STEMM (Fang et al., 2022), ConST (Ye et al., 2022), STPT (Tang et al., 2022), and SpeechUT (Zhang et al., 2022b). Besides, the multi-task learning baseline in Section 2.2 is also included as a strong baseline, which is denoted as MTL. We use CRESS to denote our method with token-level adaptive training.
Among these models, Chimera, XSTNet, STEMM, and ConST combine pre-trained Wav2vec 2.0 (Baevski et al., 2020) and pre-trained translation model together, and then fine-tune the whole model on ST datasets. Our implemented MTL and CRESS follow a similar design, but we use HuBERT instead of Wav2vec 2.0 as we find HuBERT gives a stronger baseline (See Table 5 for details). STPT and SpeechUT jointly pre-train the model on speech and text data from scratch, which achieve better performance but also bring higher training costs8.

heading: 5.3 Main Results on MuST-C Dataset
text: Table 1 shows the results on MuST-C tst-COMMON set in all eight directions. First, we find that our implemented MTL is a strong baseline compared with existing approaches. Second, our proposed CRESS significantly outperforms MTL in both settings, with 1.8 BLEU improvement in the base set-
8For example, the pre-training of SpeechUT takes 3 days with 32 V100 GPUs.
ting and 1.3 BLEU improvement in the expanded setting on average, demonstrating the superiority of our approach. Besides, we report ChrF++ scores on MuST-C in Appendix E, and we also provide results on CoVoST 2 (Wang et al., 2020a) En→De dataset in Appendix C.

heading: 6 Analysis and Discussion
text: Results in Section 5.3 show the superiority of our method. To better understand CRESS, we explore several questions in this section. All analysis experiments are conducted on MuST-C En→De dataset in the expanded setting.
(1) Do scheduled sampling, cross-modal regularization, and token-level adaptive training all matter? Scheduled sampling, regularization, and token-level adaptive training are effective techniques to improve the performance of translation models. To understand the role of each, we conduct ablation experiments in Table 2. When only applying token-level adaptive training (#5), we observe a performance decline of 0.2 BLEU since only adaptive training can not bridge the modality gap. When training with scheduled sampling only (#4), we observe a slight improvement of 0.3 BLEU, probably due to the alleviation of exposure bias. When training with cross-modal regularization only (#3), which encourages the consistency between predictions of ST and MT with ground truth target contexts, we observe an improvement of 0.7 BLEU. If we combine both (#2), we obtain a much more significant boost of 1.3 BLEU, proving that both scheduled sampling and cross-modal regularization play a crucial role in our method. Furthermore, with token-level adaptive training (#1), the improvement comes to 1.7 BLEU, which shows the benefit of treating different tokens differently according to the modality gap.
(2) Does CRESS successfully bridge the modal-
ity gap? To validate whether our approach successfully bridges the modality gap between ST and MT, we revisit the experiments in Section 3. Figure 4 shows the distribution of the modality gap with teacher forcing. We observe a general decrease in the modality gap compared with MTL. We also plot the curves of the modality gap with decoding steps of CRESS under teacher forcing, greedy search, and beam search strategies. As shown in Figure 5, our approach significantly slows down the increase of the modality gap compared with MTL baseline, suggesting that the predictions of ST and MT are more consistent during inference, demonstrating the effectiveness of our method in bridging the modality gap.
(3) How base and scale hyper-parameters influence token-level adaptive training? B (base) and S (scale) are two important hyper-parameters
in token-level adaptive training. We investigate how different combinations of B and S influence performance. As shown in Figure 6, token-level adaptive training can bring improvements in most cases. In particular, it usually performs better with smaller B and smaller S, leading to a boost of up to 0.4 BLEU. We conclude that treating different tokens too differently is also undesirable. We use B = 0.7 and S = 0.05 for all translation directions.
(4) Does CRESS successfully reduce the performance gap between ST and MT? As shown in Table 3, our method not only brings improvements to ST, but also gives a slight average boost of 0.3 BLEU to MT. We suggest that this may be due to the effect of regularization. More importantly, we find that the performance gap between ST and MT for CRESS is significantly reduced compared to the MTL baseline (6.0→5.0), which further demonstrates that the improvement in ST is mainly due to the effective reduction of the modality gap.
(5) Is CRESS more effective for longer sentences? The autoregressive model generates the translation step by step, making the translation of long sentences more challenging. We divide the MuST-C En→De dev set into several groups according to the length of target sentences, and compute the BLEU scores in each group separately, as shown in Figure 7. We observe that CRESS achieve significant improvements over the baseline in all groups, especially for sentences longer than 45, which shows the superiority of our method when translating long sentences.
(6) How the decay parameter in scheduled
sampling influence the performance? In scheduled sampling, the probability of selecting the ground truth word p∗ keeps decreasing during training as the function in Equation (10). Here, the hyper-parameter µ is used to control the shape of the function. As µ increases, the probability p∗ decreases more slowly, and vice versa. We investigate the impact of µ in Figure 8, and find that (1) the model performs worse when p∗ drops too quickly, and (2) when µ is within a reasonable range, there is not much impact on the final BLEU score. We use µ = 15 in our experiments.

heading: 7 Related Work
text: End-to-end Speech Translation End-to-end speech translation (Bérard et al., 2016; Weiss et al., 2017) has shown great potential for overcoming error propagation and reducing latency compared to traditional cascaded ST systems (Salesky et al.,
2019; Di Gangi et al., 2019c,b; Bahar et al., 2019a). One challenge in training end-to-end ST models is the scarcity of ST data. To address this problem, researchers employed MT data to help training with techniques like pre-training (Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b,c; Alinejad and Sarkar, 2020; Le et al., 2021; Dong et al., 2021a; Zheng et al., 2021; Xu et al., 2021a; Tang et al., 2022), multi-task learning (Le et al., 2020; Dong et al., 2021b; Ye et al., 2021; Tang et al., 2021a,b; Indurthi et al., 2021), knowledge distillation (Liu et al., 2019; Inaguma et al., 2021), and data augmentation (Jia et al., 2019; Bahar et al., 2019b; Lam et al., 2022; Fang and Feng, 2023). However, due to the modality gap between speech and text, it is still difficult to fully exploit MT data with the above techniques. To overcome the modality gap, Han et al. (2021) projects features of both speech and text into a shared semantic space. Fang et al. (2022); Zhou et al. (2023) mixes up features of speech and text to learn similar representations for them. Ye et al. (2022) brings sentence-level representations closer with contrastive learning. Bapna et al. (2021, 2022); Chen et al. (2022); Tang et al. (2022); Zhang et al. (2022b) jointly train on speech and text and design methods to align two modalities. Different from previous work, in this work, we understand the modality gap from the target-side representation differences, and show its connection to exposure bias. Based on this, we propose the Cross-modal Regularization with Scheduled Sampling (CRESS) method to bridge the modality gap.
Exposure Bias Exposure bias indicates the discrepancy between training and inference. Several approaches employ Reinforcement Learning (RL) (Ranzato et al., 2016; Shen et al., 2016; Bahdanau et al., 2017) instead of Maximum Likelihood Estimation (MLE) to avoid this problem. However, Wu et al. (2018) shows that RL-based training is unsta-
ble due to the high variance of gradient estimation. An alternative and simpler approach is scheduled sampling (Bengio et al., 2015), which samples between ground truth words and self-generated words with a changing probability. Zhang et al. (2019) extends it with Gumbel noise for more robust training. In this paper, we adopt this approach to approximate the inference mode due to its training stability and low training cost.
Output Regularization for MT Regularization in the output space has proved useful for MT. Liang et al. (2021) proposes to regularize the output predictions of two sub-models sampled by dropout. Guo et al. (2022) regularizes the output predictions of models before and after input perturbation. In this paper, we regularize the output predictions across modalities, which encourages more consistent predictions for ST and MT.
Token-level Adaptive Training Token-level adaptive training for MT is first proposed in Gu et al. (2020), which assigns larger weights to lowfrequency words to prevent them from being ignored. Xu et al. (2021b); Zhang et al. (2022a) computes the weight with bilingual mutual information. In this paper, we compute the weights with the modality gap between ST and MT.

heading: 8 Conclusion
text: In this paper, we propose a simple yet effective method CRESS to regularize the model predictions of ST and MT, whose target-side contexts contain both ground truth words and self-generated words with scheduled sampling. Based on this, we further propose a token-level adaptive training method to handle difficult cases. Our method achieves promising results on MuST-C benchmark. Further analysis shows that our method can effectively bridge the modality gap and improve the translation quality, especially for long sentences. In the future, we will explore how to apply our method to other tasks.
Limitations
Although our proposed method achieves promising results and outperforms most baseline systems on the ST benchmark, it still has some limitations: (1) the performance of our method still lags behind a recent work SpeechUT, although our approach has the advantage of consuming less time and resources; (2) we observe that the modality gap is still not eliminated and the effect of exposure bias on the modality gap still exists; (3) the performance of our approach on larger datasets and larger models remains to be explored; (4) how to apply our approach to other tasks also needs to be further investigated.
Ethics Statement
In this paper, we present an effective method CRESS for speech translation. While our model achieves superior performance on the widely used ST benchmark MuST-C, applying it directly to real scenarios is still risky. This is due to the fact that our training corpus only contains hundreds of hours of audio recordings from TED talks, which is far from covering all domains of the real world. Besides, the datasets we used in this paper (MuST-C, WMT, and OPUS-100) are all publicly available. We also release the code implemented with a wellknown framework fairseq. These guarantee the reproducibility of our work.

heading: Acknowledgements
text: We thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by National Key R&D Program of China (NO. 2018AAA0102502)

heading: A Statistics of all datasets
text: B Impact of Different Acoustic Encoders
Our model is composed of an acoustic encoder and a translation model. To investigate the impact of different acoustic encoders, we also conduct experiments using Wav2vec 2.09 (Baevski et al., 2020) as the acoustic encoder. As shown in Table 5, we find that (1) HuBERT performs slightly better than Wav2vec 2.0 with an improvement of 0.5 BLEU, and (2) our proposed CRESS achieves consistent improvements with different acoustic encoders. In practice, we use HuBERT to build our systems, since we believe that developing on a strong baseline will make our results more convincing and demonstrate the robustness of our approach.

heading: C Results on CoVoST 2 En→De
text: We also conduct experiments on CoVoST 2 (Wang et al., 2020a) to examine the performance of our approach on large datasets. CoVoST 2 is a largescale multilingual speech translation corpus that covers translations from 21 languages into English and from English into 15 languages. It is one of the
9https://dl.fbaipublicfiles.com/fairseq/ wav2vec/wav2vec_small.pt
largest open ST datasets available currently. In this paper, we evaluate our approach on the En→De direction, which contains 430 hours of speech with annotated transcriptions and translations. We use dev set for validation and test set for evaluation.
We use the same pre-processing, model configuration, and hyper-parameters as MuST-C (see details in Section 5.2). The results are shown in Table 6. First, we find our CRESS significantly outperforms the MTL baseline, with 1.8 BLEU improvement in the base setting and 1.6 BLEU improvement in the expanded setting, which demonstrates the effectiveness and generalization capability of our method across different datasets, especially on the large-scale dataset. Second, our result is competitive with previous methods, although they use larger audio datasets (≥60K hours) and larger model size (≥300M), while we only use 960 hours of audio data and 155M model parameters.

heading: D Discussion about the Training Speed
text: During training, our approach requires an additional forward pass to select predicted words compared with the baseline, which will impair the training speed. Practically, we find the training time for 1 epoch of CRESS is 1.12 times longer than MTL, which is actually negligible. This is because the step of selecting predicted words is fully parallel and has no gradient calculation, which does not incur a significant time overhead.

heading: E ChrF++ Scores on MuST-C Dataset
text: We also report ChrF++ score (Popović, 2017) using sacreBLEU toolkit10 on MuST-C dataset in Table 7. We observe that CRESS outperforms MTL with 1.4 ChrF++ improvement in the base setting and 1.0 ChrF++ improvement in the expanded setting.
10sacreBLEU signature: nrefs:1 | bs:1000 | seed:12345 | case:mixed | eff:yes | nc:6 | nw:0 | space:no | version:2.0.0
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We just use for research purposes, no commercial use and no derivative works of the original data.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Our use of dataset and pre-trained models is consistent with their intended use, which does not require much discussion.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data we used are publicly available on the website, and widely used in the research community. We cannot change the training/test data in order to be consistent with previous work.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix
C 3 Did you run computational experiments? Section 5, Section 6, Appendix

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5, Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5, Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5, Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will explore how to apply our method to other tasks. Limitations
Although our proposed method achieves promising results and outperforms most baseline systems on the ST benchmark, it still has some limitations: (1) the performance of our method still lags behind a recent work SpeechUT, although our approach has the advantage of consuming less time and resources; (2) we observe that the modality gap is still not eliminated and the effect of exposure bias on the modality gap still exists; (3) the performance of our approach on larger datasets and larger models remains to be explored; (4) how to apply our approach to other tasks also needs to be further investigated