CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation
Year: 2023

Abstract:
End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport (CMOT) to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text. 1

heading: 1 Introduction
text: Speech translation (ST) is the task to translate speech signals in one language into text in another. Conventional ST systems work in a cascaded mode (STENTIFORD and STEER, 1988; Waibel et al., 1991), by combining automatic speech recognition (ASR) and machine translation (MT), which may suffer from error propagation and high latency. To solve these problems, end-to-end ST is proposed (Bérard et al., 2016; Duong et al., 2016). End-toend ST directly generates the target text directly from the source audio without the intermediate transcript, which has drawn more attention in recent years (Vila et al., 2018; Salesky et al., 2019; Di Gangi et al., 2019b,c; Inaguma et al., 2020;
∗Corresponding author: Yang Feng. 1Code is publicly available at https://github.com/
ictnlp/CMOT.
Wang et al., 2020a; Zhao et al., 2021a; Dinh et al., 2022; Duquenne et al., 2022a).
End-to-end ST has to perform cross-lingual and cross-modal transformation simultaneously, hence bringing more challenges than ASR or MT. Unfortunately, it is harder to collect parallel speech-totext pairs than text-to-text pairs, which makes ST suffer from the under-fitting problem. To solve the data shortage problem, some researchers propose methods to introduce text-to-text parallel data of MT into ST, including pretraining (Alinejad and Sarkar, 2020; Zheng et al., 2021; Xu et al., 2021), multi-task learning (Le et al., 2020; Vydana et al., 2021; Tang et al., 2021), knowledge distillation (Liu et al., 2019; Gaido et al., 2020; Inaguma et al., 2021), etc. In this way, the knowledge of MT can be transferred to ST, helping learn a better crosslingual transformation.
Although these methods of transfer learning have shown improvements in the translation quality, ST is joint learning of cross-lingual and crossmodal transformation, and the effectiveness of transfer learning is also based on the assumption that the cross-modal source will be projected to the common representation space. Therefore, the mixup strategy is employed to further improve ST by reducing the gap between the speech and text representation spaces. Previous methods with mixup strategy usually perform word-level mixup between aligned tokens in different modalities (Fang et al., 2022), hence needing the help of external alignment tools. However, the alignment tools are not always available and require large amounts of additional annotated data for training, so how to obtain such alignments remains a problem for most languages.
Illuminated by the application of the optimal transport theory in finding the alignment between languages or modalities in recent years (Chen et al., 2020; Gu and Feng, 2022), we propose Crossmodal Mixup via Optimal Transport (CMOT). We
7873
use optimal transport to adaptively find the alignment between speech and text, and use this alignment to achieve token-level mixup. The mixup sequence will serve as a medium between the speech sequence and text sequence to realize cross-modal knowledge transfer. Experiments on the MuSTC dataset show that CMOT achieves an average BLEU of 30.0. In addition, we prove that optimal transport can help find cross-modal alignments and CMOT can help alleviate the modality gap in ST.

heading: 2 Method
text: In this section, we will describe our proposed Cross-modal Mixup via Optimal Transport (CMOT). We find the alignment between speech and text sequences via OT, and mix up the two unimodal sequences to get a cross-modal mixed sequence. We predict the translation with all these sequences and regularize their outputs. Figure 1 illustrates the overview of our proposed method.

heading: 2.1 Problem Formulation
text: Generally, a speech translation corpus contains triplets of source speech s, its transcript x and translation y, which can be denoted as D = {(s,x,y)}. Given the corpus, an end-to-end ST system directly converts speech signals s into text translation y without generating the intermediate transcript x.

heading: 2.2 Model Architecture
text: Our model consists of four modules: the speech encoder, the text embedding layer, the translation encoder and the translation decoder. We first get parallel speech and text sequences with the speech encoder and the text embedding layer, and feed them into the translation encoder separately. Then we mix up the two encoder outputs with CMOT, and feed both the unimodal sequences and crossmodal mixed sequence to predict the translation. Speech Encoder The speech encoder extracts the low-level features from the raw speech input. We use HuBERT (Hsu et al., 2021) model stacked with a sub-sampler as the speech encoder. The HuBERT model is a self-supervised pretrained speech model and can improve the performance of ST systems as shown in recent works (Zhang et al., 2022a; Feng et al., 2022). It consists of a convolutional feature extractor and a BERT (Devlin et al., 2019) style encoder. The sub-sampler consists of two convolution layers and is designed to reduce the length of the speech sequence.
Distributions
Distributions
ℒCE ℒCE
“Diese Katze ist süß”
Distributions
Text Embedding The text embedding layer embeds tokenized text into a sequence of embeddings. Translation Encoder The joint translation encoder receives either speech or text embeddings and learns further semantic information. It is composed of Ne transformer (Vaswani et al., 2017) encoder layers. Translation Decoder The joint translation decoder receives either unimodal sequences of speech and text or the cross-modal mixed sequence and predicts the translation y. It is composed of Nd transformer decoder layers.

heading: 2.3 Cross-modal Mixup via Optimal Transport (CMOT)
text: We mix up the speech sequence and text sequence with CMOT. For a speech sequence Hs and the corresponding text sequence Hx, we first get the alignment A between them via optimal transport (OT), and then mix up Hs and Hx with the help of A to get a mixed sequence H′. Optimal Transport The Optimal Transport (OT)
problem is a classical mathematical problem and is nowadays used as a geometric tool to compare probability distributions (Peyré et al., 2019). It focuses on the scheme with the minimum cost of transferring one distribution to another under a given transfer cost. Consider the following discrete transport problem: for two independent probability distributions P and Q,
P = {(wi,mi)}ni=1, s.t. ∑
i
mi = 1;
Q = {(ŵj , m̂j)}n̂j=1, s.t. ∑
j
m̂j = 1, (1)
where each point wi ∈ Rd has a weight mi ∈ [0,∞). Given a transfer cost function c(wi, ŵj) and denoting the mass transported from wi to ŵj by Tij , the transport cost can be defined as:
D(P,Q) = min T≥0
∑
i,j
Tijc(wi, ŵj)
s.t.
n̂∑
j=1
Tij = mi, ∀i ∈ {1, . . . , n},
n∑
i=1
Tij = m̂j , ∀j ∈ {1, . . . , n̂}.
(2)
Relaxed OT with Window Strategy If we regard the speech sequence and the text sequence as two independent distributions, we can use OT to measure the distance between them. For a speech sequence Hs = (hs1, . . . , h s n), a text sequence Hx = (hx1 , . . . , h x n̂), and a cost function c, we define the transport problem as:
D(Hs,Hx) = min T≥0
∑
i,j
Tijc(h s i , h x j )
s.t. n̂∑
j=1
Tij = m s i , ∀i ∈ {1, . . . , n},
n∑
i=1
Tij = m x j , ∀j ∈ {1, . . . , n̂}.
(3)
Here we use the Euclidean distance as the cost function c, and norm as the mass ms and mx inspired by Schakel and Wilson (2015) and Yokoi et al. (2020)’s discovery that important tokens have larger norms. The solutions to the OT problem include some accurate algorithms like Sinkhorn (Cuturi, 2013) algorithm and IPOT (Xie et al., 2020), which will bring great time complexity. Kusner et al. (2015) proposed a relaxed moving distance
that removes one of the two constraints to obtain a lower bound of the accurate solution. Following this work, we remove the second constraint, and our transport problem thus becomes:
D∗(Hs,Hx) = min T≥0
∑
i,j
Tijc(h s i , h x j )
s.t.
n̂∑
j=1
Tij = m s i , ∀i ∈ {1, . . . , n}.
(4)
This relaxed problem yields a lower bound of the original problem. We find that this relaxed OT improves the training speed without degrading the performance, as shown in Section 4.1. Now the optimal solution for each speech token hsi is to move all its mass to the closest text token hxj , so the transportation matrix becomes:
T∗ij =
{ msi if j = argminj c(h s i , h x j )
0 otherwise , (5)
This transport matrix T∗ implies the alignment between Hs and Hx. We define the cross-modal alignment as A = (a1, . . . , an), where
ai = argmax 1≤j≤n̂
T∗ij . (6)
Considering the monotonicity and locality of the alignment from the speech sequence to the text sequence, we apply a window strategy for this alignment. We constrain the index of the aligned target
to the neighborhood of the diagonal. The window size is set as a constant W both to left and to right:
ai = argmax max (1,λi−W) ≤ j ≤ min (n̂,λi+W)
T∗ij ,
λ = n̂/n. (7)
Figure 2 shows how the relaxed OT with window strategy works. Through the above steps, we get the alignment A in an adaptive manner. Cross-modal Mixup via OT With the help of the alignment A, we will be able to mixup the speech and text sequences at the token level. For a mixup probability p∗, we generate the mixup sequence H′ = (h′1, . . . , h ′ n) by:
h′i =
{ hsi , p > p ∗,
hxai , p ≤ p∗. (8)
where p is sampled from a uniform distribution U(0, 1). We feed Hs, Hx, H′ into the joint transformer to predict the translation. Note that the positions of OT and mixup can be different. Our scheme is calculating OT before the translation encoder and performing mixup with the encoder outputs, which brings better performance in the experiment. See more details in Section 4.3.

heading: 2.4 Training Strategy
text: We follow the paradigm of pretraining-finetuning to train our model. Pretraining We pretrain the text embedding and the joint transformer with an MT task. The training objective during this phase is the cross entropy loss:
LMT = −Ex,y logP (y|x). (9) Multitask Finetuning During finetuning, we train the model with multiple tasks. We add the cross entropy loss of ST and MT to the final loss:
LST = −E(s,x,y)∈D logP (y|s), LMT = −E(s,x,y)∈D logP (y|x).
(10)
Meanwhile, we regularize the predictions of mixup and speech sequences by minimizing the KullbackLeibler (KL) Divergence between the token-level probability distributions of them, and we do the same for the predictions of mixup and text sequences. For a speech prediction distribution S, a text prediction distribution T and mixup prediction distribution M , we have:
LKL(M,S) = 1
2 (DKL(M ||S) + DKL(S||M)),
LKL(M,T ) = 1
2 (DKL(M ||T ) + DKL(T ||M)).
(11)
Hence the final training objective is:
L = LST+LMT+λLKL(M,S)+λLKL(M,T ), (12) where λ is the weight to control LKL.

heading: 3 Experiments
text: We conduct experiments under two settings, the base and the expanded. The former only uses ST data, while the latter uses external MT data additionally. We will describe the datasets we use, our experimental setups and our main results in this section.

heading: 3.1 Datasets
text: ST Datasets We conduct experiments on the MuST-C2 (Di Gangi et al., 2019a) dataset, a widely used benchmark for ST. It consists of translations from English (En) to 8 languages: German (De), French (Fr), Russian (Ru), Spanish (Es), Italian (It), Romanian (Ro), Portuguese (Pt) and Dutch (Nl). For each direction, it comprises at least 385 hours of audio recorded from TED Talks. We use dev set for validation and tst-COMMON set for evaluation. External MT Datasets We incorporate external MT data to pretrain the text embedding and the joint transformer. For directions of En-De, En-Fr, En-Ru, En-Es and En-Ro we use data from WMT (Bojar et al., 2016). For En-It, En-Pt and En-Nl we use data from OPUS1003 (Zhang et al., 2020).
Table 2 shows the detailed statistics of the MuSTC, WMT and OPUS100 datasets we use.

heading: 3.2 Experimental Setups
text: Model Configurations For the speech encoder, we use HuBERT4 (Hsu et al., 2021) base model, which is pretrained on LibriSpeech (Panayotov et al., 2015) with no finetuning and is one of the state-of-the-art pretrained audio models. Following the usual practice for ST, we stack two convolution layers of kernel size 5, stride size 2, padding 2 and hidden size 1024 after the HuBERT model as a sub-sampler. The text embedding is an embedding layer of dimension 512. For the joint transformer, we use Ne = 6 encoder layers and Nd = 6 decoder layers with 512 hidden units, 2048 feed-forward hidden units and 8 attention heads. We use fairseq5 (Ott et al., 2019) for implementation. 2Here we refer to v1.0. https://ict.fbk.eu/must-c/ 3http://opus.nlpl.eu/opus-100.php 4https://github.com/facebookresearch/fairseq/ tree/main/examples/hubert 5https://github.com/facebookresearch/fairseq/
Pre-processing For speech input, we use the 16-bit 16kHz mono-channel raw audio, and we filter out samples with frames greater than 480k or less than 1k to ensure training efficiency. For transcripts and translations, we tokenize them using a unigram SentencePiece6 (Kudo and Richardson, 2018) model with a vocabulary size of 10k, which is shared by the source language and the target language. Training Our training process follows the pretraining-finetuning paradigm. We pretrain the text embedding layer, the translation encoder and decoder with an MT task. For the base setting, we only use the transcript-translation pairs with a learning rate of 2e-3, and warm-up steps of 8k. For the expanded setting we first pretrain them on external MT data with a learning rate of 7e-4 and warm-up steps of 8k, and then pretrain them on transcript-translation pairs with a learning rate of 1e-4 and warm-up steps of 4k. For both settings,
6https://github.com/google/sentencepiece
we allow at most 33k input tokens per batch. We set the learning rate to 1e-4 and warm-up steps to 10k to finetune the whole model. We train the entire model for 60k updates with a batch of 16M audio frames. We use an Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98, dropout of 0.1 and the label smoothing value of 0.1. For CMOT, we set the weight λ of KL loss to 2.0 (see more in Appendix 4.3) and the mixup probability p∗ to 0.2 (see more in Section 4.3), and we use the window strategy for OT by default with window size W = 10. All models are trained on 4 Nvidia GeForce RTX 3090 GPUs. Inference We average the checkpoints of the last 10 epochs for evaluation. We use beam search with a beam size of 8. We evaluate case-sensitive detokenized BLEU on MuST-C tst-COMMON set using sacreBLEU7 (Post, 2018). We use sacreBLEU to test the significance of the results. Baselines We compare our method with the Fairseq ST (Wang et al., 2020a) baseline implemented with the same framework (Ott et al., 2019). We implement HuBERT-Transformer as a strong baseline, which has the same architecture as our method. The only difference is that HuBERTTransformer takes speech as input and performs the ST task during fine-tuning. We train this model for 40k steps, except for the En-Ru direction where we use a training step of 100k due to specific language complexities.
We also compare our method with several robust end-to-end ST systems: Chimera (Han et al., 2021) which learns a shared semantic space for speech and text, XSTNet (Ye et al., 2021) which uses a
7https://github.com/mjpost/sacrebleu
progressive multi-task training strategy, STEMM (Fang et al., 2022) which applies word-level mixup strategy and ConST (Ye et al., 2022) which applies contrastive learning strategy. These baselines all adopt the multi-task learning mode and have similar model architectures.

heading: 3.3 Main Results
text: Table 1 shows the main results. HuBERTTransformer, the baseline model we implement, achieves relatively high BLEU scores. For the base setting without external MT data, CMOT outperforms HuBERT-Transformer by 1.1 BLEU in average of 8 directions. For the expanded setting, CMOT outperforms HuBERT-Transformer by 0.9 BLEU in average. Besides, CMOT also outperforms other strong baselines.

heading: 4 Analysis

heading: 4.1 Effect of OT
text: We use CMOT to learn the alignment between speech and text, and to improve the performance of the ST system. In this section, we compare CMOT with other models without OT or with other types of OT. All the models have the same architecture. MTL denotes the multi-task learning (ST and MT) model. Relaxed OT denotes the model trained with the same process of CMOT but without the window strategy in Section 2.3. IPOT denotes the model trained with the same process as CMOT but using IPOT (Xie et al., 2020) for OT calculation. The A-score is defined as the accuracy of our alignment A = (a1, . . . , an) compared to the reference alignment A∗ = (a∗1, . . . , a ∗ n):
A-score = ∑
i
1(ai = a ∗ i )/n, (13)
Here A∗ is obtained with Montreal Forced Aligner (MFA)8 (McAuliffe et al., 2017). As shown in Table 3, we can see the result of relaxed OT is slightly lower than that of IPOT, but when the window strategy is used, its performance exceeds that of IPOT with a faster speed. CMOT achieves the best BLEU score, along with the best A-score of alignment, proving that CMOT not only takes advantage of this alignment but optimizes it as well.
In addition, we observe the cost matrices of OT to verify that the cost matrix can be used for crossmodal alignment. We compare the cost matrices in
8https://montreal-forced-aligner.readthedocs. io/en/latest/
MTL and CMOT, and analyze a typical example, as shown in Figure 3. The redder grids indicate the distance between the speech token and the text token is smaller, while the bluer grids are the opposite. To better observe the word-level alignment, we use MFA to cut the sentences of speech and text into words, and the white solid lines denote the segmentation. For CMOT, we can see that the closer speech-text pairs basically lie on the diagonal, which is consistent with the alignment of the two sequences. But for MTL without the help of OT, the distance between speech and the corresponding word is not that close. Comparing the two green boxes in the figure, we can see CMOT has better alignment on the word “pointed”.

heading: 4.2 Effect of Cross-modal Mixup
text: To examine the effectiveness of the mixup strategy in bridging the modality gap, we compare the modality gap of CMOT and baseline models. We calculate the mean sentence-level distances and mean word-level distances of these models. We focus on the modality gap at the encoder input and output, and the results are shown in Figure 4. We can see that compared with the baseline model, CMOT has a smaller distance between modalities, which means our method effectively reduces the modality gap and improves the performance of ST.

heading: 4.3 Ablation Study
text: Training Objectives As a multi-task learning framework, our ST system’s performance can be affected by the training objectives. Table 4 shows the impact of training objectives on ST performance. Comparing Exp.I and II, We can empirically conclude that multi-task learning strategy alone helps little, while Exp.III suggests regularizing the output distributions is effective. Exp.IV, V, and VI show that calculating KL divergence for both mixup and speech outputs, and mixup and text outputs can bring the greatest improvement. In Exp.VII, we
in / sentence
out / sentence
in / word
out / word
4.244.26 4.25
1.171.161.06
13.7013.50 13.17
3.14
2.98
2.45
HuBERT MTL CMOT
Figure 4: Modality gaps of different models. The data are calculated on the MuST-C En-De dev set with models of expanded setting. “in” / “out” denotes the encoder input / output sequences, while “sentence” / “word” denotes sentence-level / word-level Euclidean distance.
add the translation loss of the mixup sequence:
LMixup = −E(s,x,y)∈D logP (y|H′), (14)
where H′ is the mixup sequence calculated as Equation (8). Comparing Exp.VI and VII, we can see LMixup does not help. We also inspect the effect of OT loss which takes the value of D∗ in Equation (4) in Exp.VIII, and see a slight performance decline. Here OT is only used to help find the alignment, so OT loss is not indispensable during fine-tuning, and we conjecture adding OT loss may make it harder for the model to converge to good parameters, as the training objective is too complex. Positions of OT and Mixup We investigate the impact of positions of OT and the mixup on our
system. As shown in Table 5, the best scheme is to calculate OT for encoder inputs and perform the mixup for encoder outputs. We believe there is more original alignment information at the lower layers of the model, which is better for OT calculation, while the hidden states of higher layers are more suitable for the mixup, as they have passed several layers of joint transformer and are closer in the representation space. Mixup Probability We explore the impact of mixup probability on ST performance and the results are shown in Figure 5. Here p∗ denotes the proportion of text tokens in the mixup sequence.
Previous works find a slightly larger probability is better, while we find p∗ = 0.2 improves ST performance the most. We think this is because CMOT needs to find the alignment between speech and text adaptively, so a larger proportion of text tokens in the mixup sequence might bring some noise. Speech Encoder We investigate the impact of different speech encoders on our ST system. HuBERT (Hsu et al., 2021) and wav2vec 2.0 (Baevski et al., 2020) are both commonly used speech pretrained models. Table 6 indicates that using HuBERT as a speech encoder can achieve higher performances than using wav2vec 2.0.
In addition, in order to provide a fair comparison with other works using wav2vec 2.0, we conduct experiments on CMOT with wav2vec 2.0 as the speech encoder in more directions. As shown in Table 7, CMOT using wav2vec 2.0 still performs well, surpassing previous works in these three language pairs. Weight of KL Loss Our finetuning process follows a multi-task learning approach, where the weight of each training objective can affect the overall performance. For the KL loss in Equation 12, we choose several different weight values λ ranging from 1.0 to 3.0, and find when λ = 2.0 or λ = 2.5 the performance is best, as shown in Figure 6. When the weight of KL loss is too small, the
regularization effect of the KL objective becomes little, while a large weight of KL loss can lead to performance degradation of the main tasks ST and MT. Hence, we suggest the weight be at a moderate value, and we choose λ = 2.0 for our system. Window Size for OT We examine the impact of window size W on our ST system. As shown in Figure 7, a small window size may restrict the model’s ability to identify appropriate alignments, while a large window size may diminish its ability to capture local dependencies.

heading: 5 Related Work
text: End-to-end ST Traditional ST systems work in a cascaded mode (STENTIFORD and STEER, 1988; Waibel et al., 1991), with problems of error propagation and high latency. Bérard et al.
(2016); Duong et al. (2016) proposed end-to-end ST without intermediate transcription, which has become the mainstream paradigm of ST in recent years (Vila et al., 2018; Salesky et al., 2019; Di Gangi et al., 2019b,c; Inaguma et al., 2020; Wang et al., 2020a; Zhao et al., 2021a; Dinh et al., 2022; Duquenne et al., 2022a). An end-to-end ST system is difficult to train due to its cross-modal cross-lingual characteristics and the scarcity of parallel data, hence some techniques like pretraining (Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b; Alinejad and Sarkar, 2020; Zheng et al., 2021; Xu et al., 2021; Zhang et al., 2022b), selftraining (Pino et al., 2020; Wang et al., 2021), curriculum learning (Kano et al., 2017; Wang et al., 2020c), multi-task learning (Le et al., 2020; Vydana et al., 2021; Tang et al., 2021), data augmentation (Lam et al., 2022; Fang and Feng, 2023a), and knowledge distillation (Liu et al., 2019; Gaido et al., 2020; Inaguma et al., 2021) are applied to improve its performance. To overcome the modality gap between speech and text, various methods are used. Zhao et al. (2021b) proposed a mutual learning scenario. Han et al. (2021) and Duquenne et al. (2022b) projected speech and text into a common representation space. Chen et al. (2022) proposed the method of modality matching. Ye et al. (2022) applied the contrastive learning strategy. Fang and Feng (2023b) proposed cross-modal regularization with scheduled sampling. Mixup Our work draws on the strategy of mixup. Mixup is first proposed in Zhang et al. (2018) as a form of data augmentation to improve generalization and increase the robustness of neural networks by creating “convex combinations of pairs
of examples and their labels”. Verma et al. (2019) proposed the concept of manifold mixup, going a step further than surface-level mixup. In recent years, mixup is used to reduce the gap of representation spaces, such as cross-modal transfer (So et al., 2022; Ye and Guo, 2022) and cross-lingual transfer (Yang et al., 2021). Fang et al. (2022) proposed STEMM, applying mixup to end-to-end ST to overcome modality gap. Although both STEMM and our work use the mixup strategy, STEMM needs external alignment tools, while CMOT finds the alignment adaptively. Optimal Transport Optimal transport is a classical mathematical problem. It is now commonly used to describe the transfer cost between two distributions. Villani (2009) systematically demonstrates the theories and proofs of optimal transport. Peyré et al. (2019) outlines the main theoretical insights that support the practical effectiveness of OT. There are some approximation algorithms for the OT problem, like the Sinkhorn algorithm (Cuturi, 2013) and IPOT (Xie et al., 2020) algorithm. Kusner et al. (2015) proposed a relaxed form of OT in order to measure the similarity between documents, which reduced the computational cost. Chen et al. (2020) used OT in image-text pretraining to “explicitly encourage fine-grained alignment between words and image regions”. Gu and Feng (2022) applied the relaxed form of OT to machine translation and managed to “bridge the gap between the semantic-equivalent representations of different languages”.

heading: 6 Conclusion
text: We propose Cross-modal Mixup via Optimal Transport (CMOT) to adaptively find the alignment between speech and text sequences, and to mix up the sequences of different modalities at the token level. Experiments on the MuST-C benchmark demonstrate the effectiveness of our method, which helps overcome the modality gap and thus improves the performance of end-to-end ST systems. We will continue to explore the application of this adaptive cross-modal alignment method in the future.
Limitations
The CMOT method for speech translation is based on the idea of cross-modal knowledge transfer and the paradigm of multi-task learning, so it requires the transcripts of speech during training, which
may not be applicable to some languages without transcribed text. Besides, this work mainly focuses on the finetuning of speech translation, and applications on the pretraining phase or on other tasks are in need of exploration in the future.
Ethics Statement
Our proposed CMOT can help build a strong endto-end ST system. It has prospects for application in many scenarios requiring speech translation, which helps people understand speech in a foreign language. Nevertheless, the results generated by end-to-end ST systems are not necessarily perfect, so people may not fully rely on the results for the time being.

heading: Acknowledgements
text: This work was supported by National Key R&D Program of China(NO. 2018AAA0102502). We thank all the anonymous reviewers for their insightful and valuable comments.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7 Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8 Ehic Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 0 Abstract, Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2 Method, Section 3 Experiments, Section 4 Analysis

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2 Method, Section 3 Experiments, Section 4 Analysis
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets and models we use are for research purposes only, not for any commercial purposes. Our use does not violate their licenses.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Our use is consistent with their intended use, and there is no special need to emphasize this as the datasets and models we use have been used in the same way in recent years.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The speech translation dataset we use is collected from public TED talks. Since they are already open on the Internet, we cannot anonymize them. The machine translation datasets we use are already anonymized, so there is no need to discuss their problems.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 Experiments
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A Statistics of Datasets
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 3 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3 Experiments

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 Experiments, Section 4 Analysis

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 Experiments

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 Experiments
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We will continue to explore the application of this adaptive cross-modal alignment method in the future. Limitations
The CMOT method for speech translation is based on the idea of cross-modal knowledge transfer and the paradigm of multi-task learning, so it requires the transcripts of speech during training, which
may not be applicable to some languages without transcribed text. Besides, this work mainly focuses on the finetuning of speech translation, and applications on the pretraining phase or on other tasks are in need of exploration in the future