Venue: NAACL 2022

# Cross-modal Contrastive Learning for Speech Translation

Rong Ye \({}^{1}\), Mingxuan Wang \({}^{1}\), Lei Li \({}^{2}\)

\({}^{1}\) ByteDance AI Lab, \({}^{2}\) University of California, Santa Barbara

{yerong, wangmingxuan.89}@bytedance.com

leili@cs.ucsb.edu

Partial work was done while at ByteDance.

###### Abstract

How can we learn unified representations for spoken utterances and their written text? Learning similar representations for semantically similar speech and text is important for speech translation. To this end, we propose **ConST**, a cross-modal contrastive learning method for end-to-end speech-to-text translation. We evaluate ConST and a variety of previous baselines on a popular benchmark MuST-C. Experiments show that the proposed ConST consistently outperforms the previous methods, and achieves an average BLEU of 29.4. The analysis further verifies that ConST indeed closes the representation gap of different modalities -- its learned representation improves the accuracy of cross-modal speech-text retrieval from 4% to 88%. Code and models are available at [https://github.com/ReneeYe/ConST](https://github.com/ReneeYe/ConST).

## 1 Introduction

End-to-end speech-to-text translation (E2E ST) becomes important in many internet products and real applications. An E2E ST system accepts audio signals as the input and generates the target translation using a single model. Compared with the conventional cascade ST models, E2E ST models have achieved almost comparable [3] or even superior [5, 6, 7] performance.

The performance of an E2E ST model is still restricted for languages with relatively small parallel data, compared to text machine translation (MT). Existing approaches for ST focus on using additional data from MT and automatic speech recognition (ASR). This can be realized through pretraining approaches [10, 11, 12] or multi-task training frameworks [13, 14, 15].

Different from the data perspective, this paper investigates the bottleneck of E2E ST from the neural representation perspective. We believe that when the representation of audio input is similar to its corresponding textual representation, it is easier for information to transfer from MT to ST, thus improving speech translation performance.

We analyze Transformer models for speech translation and observe a noticeable modality gap between encoder representations of speech and text from existing ST models (as in Figure 0(a). Sec. 6 has more details). An ideal representation should satisfy: if the content of the speech and transcription are similar, their encoded representations should likewise be close to each other (as in Figure 0(b)). Nevertheless, how to learn unified and aligned speech-text representations?

Inspired by the recent progress of contrastive learning approaches in cross-lingual [14, 15] and cross-modal vision-and-language domains [16, 17, 18], we designed a simple **c**ontrastive learning method for **ST** (**ConST**) to learn the representations that meet the aforementioned conditions explicitly. On the one hand, our model inherits the advantages of the previous

Figure 1: Illustration of representations for speech and transcript text (projected to 2D). (a) representations learned by existing models. Pairs of speech and text representations are distant. (b) an ideal representation that we expect, where different modalities with same meaning should stay close to each other.



multi-task learning methods. On the other hand, it reduces the gap between the representations of speech and its corresponding transcription.

Our contributions are as follows.

* We develop ConST for speech translation, a cross-modal contrastive learning method, on top of the multi-task training framework.
* Our experiments on the MuST-C benchmark show that ConST achieves an average BLEU score of 29.4, outperforming the best previous baselines.
* We demonstrate that ConST indeed learns similar representations for two modalities and better retrieves text with speech input.

## 2 Related Work

End-to-end STTo alleviate the error propagation in the cascaded ST systems and to make the deployment simpler, Berard et al. (2016); Weiss et al. (2017) proposed to use an end-to-end architecture to directly translate speech into text in another language, without the intermediate transcription. Kano et al. (2017); Berard et al. (2018); Inaguma et al. (2020); Wang et al. (2020a); Zhao et al. (2021a) implemented several off-the-shelf encoder-decoder E2E-ST models, such as BiLSTM Greff et al. (2016) and Speech-Transformer Dong et al. (2018). However, training an end-to-end speech translation model is difficult because we need to design a cross-modal cross-language model, meanwhile, the _speech-transcription-translation_ supervised data for speech translation is significantly less than that of MT and ASR. Methods, like data augmentation Park et al. (2019); Pino et al. (2020); Chen et al. (2021), pre-training Weiss et al. (2017); Berard et al. (2018); Bansal et al. (2019); Wang et al. (2020b); Alinejad and Sarkar (2020); Dong et al. (2021a); Zheng et al. (2021), self-training Pino et al. (2020); Wang et al. (2021a), utilizing self-supervised pre-trained audio representation Wu et al. (2020); Han et al. (2021); Ye et al. (2021); Wang et al. (2021a), are proved to be effective. Meanwhile, some work has shown that the encoder-decoder model with a single encoder cannot encode speech information well. For example, Dong et al. (2021b) first proposed a second encoder to further extract semantic information of the speech sequence. Xu et al. (2021) proposed a stacked acoustic-and-textual encoder and introduced large-scale out-of-domain data. Also, multi-task frameworks Le et al. (2020); Tang et al. (2021); Ye et al. (2021) are widely applied to further enhance the robustness for ST. As a cross-modal task, some work has noted the problem of the modality gap. Han et al. (2021) designed a fix-size semantic memory module to bridge such a gap, from the neuroscience perspective. However, we find that this approach actually sacrifices the effect of MT. So in this paper, we propose a simple yet effective contrastive learning method to bridge the gap and to improve ST performance.

Cross-modal grounding learningThis paper attempts to address the problem in speech translation from the perspective of cross-speech-text representation learning. We are also inspired by cross-modal representation learning in the acoustic word embedding (AWE) Palaskar et al. (2019); Kamper et al. (2020); Hu et al. (2020) and the visual-language pre-training (VLP) Wu et al. (2019); Lu et al. (2019); Chen et al. (2020); Li et al. (2021) tasks. These works usually focus on enhancing textual representations with acoustic or visual information, in other words, grounding learning. In this work, we consider the its dual form, i.e., grounding speech representations using text.

Contrastive learningOur method is motivated by the recent success in contrastive representation learning. The contrastive learning method was first proposed to learn representations from unlabeled datasets (hence the term, self-supervised learning) by telling which data points are similar or distinct, especially in the field of computer vision Chopra et al. (2005); Gutmann and Hyvarinen (2010); Schroff et al. (2015); Sohn (2016); Oord et al. (2018); Chen et al. (2020); Grill et al. (2020). Khosla et al. (2020) extended the self-supervised batch contrastive approach to the fully-supervised setting and proposed a supervised contrastive learning method. In speech processing, representative methods focused on speaker identification Ravanelli and Bengio (2018), speech recognition Schneider et al. (2019), and audio representation learning van den Oord et al. (2018); Baevski et al. (2020). In the NLP area, the contrastive framework is used for sentence representation learning Fang et al. (2020); Shen et al. (2020); Gao et al. (2021); Wu et al. (2021); Yan et al. (2021); Fu et al. (2022), machine translation Pan et al. (2021), and summarization Wang et al. (2021b); Cao and Wang (2021). Very recently, contrastive learning is also applied to learning a unified representation of image and text Dong et al. (2019); Zhou et al. (2020); Li et al. (2021). Moti 

vated by the contrastive learning frameworks in cross-lingual and cross-modal topics, we introduce a similar idea in speech translation.

## 3 The ConST Approach

An end-to-end speech translation model directly translates audio sequence \({\bf s}=(s_{1},...,s_{|{\bf s}|})\) to the text \({\bf y}=(y_{1},...,y_{|{\bf y}|})\) in the target language. Speech translation corpus \(\mathcal{D}=\{({\bf s},{\bf x},{\bf y})\}\) provides _transcript_\({\bf x}=(x_{1},...,x_{|{\bf x}|})\) in the source language, as well.

In this section, we present the overall speech translation model and cross-modal contrastive learning. We also provide several feasible strategies to construct more positive and negative pairs to enhance the contrastive learning.

### Model Framework

We use the same model architecture as XSTNet Ye et al. (2021). Our model consists four sub-modules: a speech encoder, a word embedding layer, a Transformer Encoder and a Transformer decoder (Figure 2). It is designed to take either speech or a sentence as input, and to output either source transcript or target translation text. Such architecture enables a universal framework for multiple tasks, including ST, MT and ASR.

The _speech encoder_ module (S-Enc) is designed to extract low-level features for speech signals. It contains Wav2vec2.0 Baevski et al. (2020) and two additional convolutional layers. The input is raw waveform signal sampled at 16kHz. Each convolutional layer has a stride of 4 and \(d\) channels. In total, it shrinks the time dimension by a factor of \(4\). Denote \({\bf a}=\text{S-Enc}({\bf s})\) as the audio representation of the speech, \(|{\bf a}|\ll|{\bf s}|\).

Parallel to the speech encoder is the _word embedding layer_. It is the same as word embedding for text translation.

Both the speech encoder and word embedding layer are connect to _Transformer encoder_ and then passed to the _Transformer decoder_. The Transformer encoder and decoder are using the same configuration as the original Vaswani et al. (2017). To explain, the _Transformer encoder_ further extracts the high-level semantic hidden representation of two modalities. The _Transformer decoder_ generates the word sequences (transcription and translation) for ST, MT and ASR tasks. Since our model has a complete Transformer encoder-decoder as a sub-module, this makes it possible to pre-train using large-scale extra MT parallel data.

Previous work has shown that multi-task learning on ST, MT and ASR improves translation performance Indurthi et al. (2020); Tang et al. (2021); Ye et al. (2021). Our training loss consists of the following elements.

\[\mathcal{L}=\mathcal{L}_{\text{ST}}+\mathcal{L}_{\text{ASR}}+\mathcal{L}_{ \text{MT}}+\lambda\mathcal{L}_{\text{CTR}} \tag{1}\]

where

\[\mathcal{L}_{\text{ST}} =-\sum_{n}\log P({\bf y}_{n}|{\bf s}_{n})\] \[\mathcal{L}_{\text{ASR}} =-\sum_{n}\log P({\bf x}_{n}|{\bf s}_{n})\] \[\mathcal{L}_{\text{MT}} =-\sum_{n}\log P({\bf y}_{n}|{\bf x}_{n})\]

Figure 2: **Left**: Model architecture of ConST. The gray shaded modules are the _optional_ data augmentation operations introduced in Section 3.3. **Right**: An illustration of cross-modal contrastive learning.

 

The first three elements are cross-entropy losses on _<speech, target text>_, _<speech, source text>_ and _<source text, target text>_ pairs. These pairs are built from the triplet ST data. We also introduce a cross-modal contrastive loss term \(\mathcal{L}_{\text{CTR}}\) (see Section 3.2 for details). It aims to bring the representation between the speech and textual transcription modalities closer (its effect will be analyzed in detail in Section 6). \(\lambda\) is a tuned hyper-parameter of the weighted contrastive loss term.

### Cross-modal Contrastive Learning

As mentioned in the beginning, since we need to produce similar representations for the speech and transcript sharing the same semantic meanings, we propose cross-modal contrastive learning method to bring their representations closer together. The main idea of cross-modal contrastive learning is to introduce a loss that brings speech and its corresponding transcript (positive example) near together while pushing irrelevant ones (negative examples) far apart.

Given a positive example of such a speech-transcript pair \((\mathbf{s},\mathbf{x})\), we randomly pick a set of \(N-1\) transcripts \(\{\mathbf{x}^{-}_{i}\}_{i=1}^{N-1}\) from the same batch as negative examples. For speech \(\mathbf{s}\) and its transcript \(\mathbf{x}\), we first average them in terms of the time dimension,

\[u=\text{MeanPool}(\text{S-Enc}(\mathbf{s})) \tag{2}\] \[v=\text{MeanPool}(\text{Emb}(\mathbf{x})) \tag{3}\]

and apply the multi-class N-pair contrastive loss Sohn (2016):

\[\mathcal{L}_{\text{CTR}}=-\sum_{s,x}\log\frac{\exp(sim(u,v)/\tau)}{\sum_{x_{j} \in\mathcal{A}}\exp(sim(u,v(x_{j}))/\tau)} \tag{4}\]

where \(\mathcal{A}=\{\mathbf{x}\}\cup\{\mathbf{x}^{-}_{i}\}_{i=1}^{N-1}\), \(\tau\) is the temperature hyper-parameter, and \(sim\) is the cosine similarity function \(sim(a,b)=a^{\top}b/\|a\|\|b\|\). In the implementation, negative examples \(\{\mathbf{x}^{-}_{i}\}_{i=1}^{N-1}\) are from the same training batch of data (Figure 2(b)).

### Mining Hard Examples for Contrastive Learning

To further enhance the contrastive learning, we introduce three strategies to mine additional hard examples. These strategies are at input and representation (gray shaded modules in Figure 2(a)). Specific schematic illustrations of each operations are shown in Figure 3.

Span-Masked AugmentationWe mask consecutive segments of an original audio waveform sequence \(\mathbf{s}\) to obtain a new modified speech \(\mathbf{s}^{\prime}\). We take \(\mathbf{s}^{\prime}\) as an input to the model, and compute the contrastive loss on its original corresponding transcript. We randomly sample without replacement all time steps in the original waveform of the speech to be the starting indices with a probability \(p\), and then we set the sub-sequence \(M\) successive time steps to be blank. In the experiment, we tried multiple configurations, and found \(p=0.25,M=3600\) the best, resulting in a masked span of \(0.225\) second. Since the masked speech fragment is very short, we consider the masked speech and the original transcript to be positive pairs, and the remaining transcripts in the same batch to be negative pairs.

Word RepetitionThe word repetition strategy randomly replicates some words (or sub-words) in the original sentences, with two advantages for improving representation robustness. First, as the length of the sentence is shorter than that of its audio representation, randomly repeating the words in the sentence is a simple yet useful technique to increase the length. Second, repeating words does not change the semantics and is suitable as an extra positive example of the corresponding speech. Specifically, given sentence \(\mathbf{x}\), each sub-word token \(x_{i}\) can be duplicated \(k\)**more** times, resulting in the duplicated sentence \(\mathbf{x}^{\prime}\), where \(k=0,1,2,...\) and \(k\sim\text{Poisson}(1)\). We regard \(\mathbf{x}^{\prime}\) as the additional positive example for the speech \(\mathbf{s}\) and the samples with the same operation in the same batch as the negative examples.

Cut-off strategyRecent studies on natural language understanding and generation have proved cut-off strategy to be successful Shen et al. (2020); Yan et al. (2021). We analogize a similar idea to the cut-off approach for speech representation. We en

Figure 3: Schematic illustration of the hard examples mining strategies. In the cut-off strategy, the gray shaded grid represents the zero-out element.

 

tirely erase a slice of the \(T\times d\) representation matrix along each dimension and set the erased terms to 0. Here, we present two variants: _sequence cut-off_, which erases some sequence dimension, and _feature cut-off_, which erases some feature dimension. Note that there is a difference between cut-off and dropout. Dropout randomly sets some elements to 0, while cut-off is a dimensional "block" dropout. Similarly, we treat the cut-off audio representation and the original transcribed sentence as positive pairs, and the rest sentences in the same batch as negative pairs.

## 4 Experiments

### Experimental Setups

#### ST datasets

We conduct experiments on all the translation directions in **MuST-C** dataset 1[2]: English (En) to German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro) and Russian (Ru). As one of the largest ST benchmarks, MuST-C contains more than 385 hours of TED talks for each direction.

Footnote 1: We use v1.0. [https://ict.fbk.eu/must-c/](https://ict.fbk.eu/must-c/)

Footnote 2: [https://dl.fbaipublicfiles.com/fairseq/wav2vec/awav2vec_small.pt](https://dl.fbaipublicfiles.com/fairseq/wav2vec/awav2vec_small.pt)

#### MT datasets

We also introduce external **WMT** datasets (Bojar et al., 2016) for En-De/Es/Fr/Ro/Ru and **OPUS100** datasets (Zhang et al., 2020) for En-It/Nl/Pt directions, as the expanded setup.

Table 8 (in Appendix. A) lists the statistics of all the datasets included.

#### Model Configurations

The Wav2vec2.0 in the S-Enc is only pre-trained on Librispeech (Panayotov et al., 2015) speech without any downstream fine-tuning3. Two layers of CNNs after the Wav2vec2.0 are set to kernel size 5, stride size 2 and hidden size 512. The Transformer follows the base configuration, with 6 layers of encoder and decoder, hidden size \(d=512\), 8 attention heads, and 2048 FFN hidden states. We use pre-layer normalization for stable training. The model with the above configurations has a total of about 150M parameters.

Footnote 3: [https://github.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu), BLEU Signature: nrefs!1 l8:100I seed:12345| case:mixed leff:no lok:13a| smooth:exp l version:2.0.0

#### Experiment Details

We evaluate case-sensitive detokenized BLEU using sacreBLEU4[7] on MuST-C tst-COMMON set. In the analysis, we also report the ChrF++ score 5. We use the raw 16-bit 16kHz mono-channel _speech_ input. We jointly tokenize the bilingual _text_ using SentencePiece (Kudo and Richardson, 2018), with a vocabulary size of 10k, which is the same as Ye et al. (2021)'s setup. For the training loss, we set contrastive temperature \(\tau=0.02\) and weight of contrastive term \(\lambda=1.5\) for German and Dutch, and \(\lambda=1.0\) for the other languages.

Footnote 5: [https://github.com/google-research/bleurt](https://github.com/google-research/bleurt) (Sellam et al., 2020). As recommended, the checkpoint we use is BLEURT-20.

Appendix B contains more detailed settings and explanations for the baseline models in Table 1. Appendix C shows the experiments on the choice of the hyper-parameters.

### Main Results

#### Comparison with end-to-end ST models

Table 1 shows the main results. Since many existing works regard "leveraging external data" to be one of their model's features, their strong performances are largely predicated on the utilization of auxiliary data, especially large-scale MT data. For a relatively fair comparison, we investigate two cases: (1) without external MT data and (2) with external MT data. Without the external MT data, our method already gains an average improvement of 0.5 BLEU over the previous best models. Also when speech data is introduced for pre-training, our method works better than others (Self-training, W-Transf. and XSTNet). When extra MT data are introduced, our method also outperforms SOTA by an average of 0.6 BLEU. Among the benchmark models, with the same goal of closing two modality gaps, Chimera (Han et al., 2021) constructed an extra fixed-length shared semantic space. However, the shared memory with fixed size actually compromises the MT performance, while our contrastive learning approach is more straightforward and effective.

#### Comparison with cascaded ST systems

We compare our method with several cascade baselines, where Ye et al. (2021) and Xu et al. (2021) provided two strong cascade systems trained using MuST-C and external ASR and MT data (LibriSpeech, WMT, and Opensubtitles). From Table 2, we find that as an end-to-end model, ConST can outperform these strong cascade models. In Appendix 7, we provide a case study to show such improvement.

 



## 5 Analysis

### Is contrastive loss effective?

With the same model architecture and the same pre-training + fine-tuning procedure, the main difference between ConST and XSTNet Ye et al. (2021) is whether we use the contrastive loss term during the fine-tuning or not. Comparing the BLEU results of w/o and w/ external MT data situations in Table 1, we find that ConST further improves \(0.5\) and \(0.6\) BLEU scores in terms of eight translation directions on average, which proves the effectiveness of the cross-modal contrastive learning. By gradually removing each losses in Eq.( 1), Table 3 shows the improvements bringing by the multi-task learning and the contrastive learning. For En-De translation direction, contrastive learning can bring an average improvement of 0.9 BLEU over the baseline models by only optimizing \(\mathcal{L}_{\text{ST}}\) (corresponding to the last row of the Table 3), and multi-task learning can lead to a further improvement of about 1.2 BLEU on top of that.

### Which layer to contrast on?

An intriguing question is which representations should be considered in the contrastive loss function. In the method part (Section 3.2), we use averaged audio representation \(u\) for speech **s** (Eq.(2)) and averaged lexical embedding \(v\) for the transcript **x** (Eq.(3)), denoted as _low-level repr._. Whereas inspired by a recent study in multilingual MT Pan et al. (2021), we also provide an alternative contrastive loss as a comparison, whose speech and text features are average-pooled semantic repre

\begin{table}
\begin{tabular}{l|c c c|c c c c c c c c|c} \hline \multirow{2}{*}{**Models**} & \multicolumn{4}{c|}{**External Data**} & \multicolumn{4}{c}{**BLEU**} \\  & **Speech** & **Text** & **ASR** & **MT** & **De** & **Es** & **Fr** & **It** & **Nil** & **Pt** & **Ro** & **Ru** & **Avg.** \\ \hline \multicolumn{87}{c}{_w/o external MT data_} \\ \hline Fairsen ST Wang et al. (2020a) & - & - & - & - & 22.7 & 32.9 & 22.7 & 27.3 & 28.1 & 21.9 & 15.3 & 24.8 \\ NeurST Zhao et al. (2021a) & - & - & - & - & 22.8 & 27.4 & 33.3 & 22.9 & 27.2 & 28.7 & 22.2 & 15.1 & 24.9 \\ Espnet ST languma et al. (2020) & - & - & - & - & 22.9 & 28.0 & 32.8 & 23.8 & 27.4 & 28.0 & 21.9 & 15.6 & 25.1 \\ Dual Decoder Le et al. (2020) & - & - & - & - & 23.6 & 28.1 & 33.5 & 24.2 & 27.6 & 30.0 & 22.9 & 15.2 & 25.6 \\ W-Transst. Ye et al. (2021) & ✓ & - & - & - & 23.6 & 28.4 & 34.6 & 24.0 & 29.0 & 29.6 & 22.4 & 14.4 & 25.7 \\ Speechformer Papi et al. (2021) & - & - & - & - & 23.6 & 28.5 & - & 27.7 & - & - & - & - \\ LightweightAdaptor Le et al. (2021) & - & - & - & - & 24.7 & 28.7 & 35.0 & 25.0 & 28.8 & 31.1 & 23.8 & 16.4 & 26.6 \\ Self-training Pino et al. (2020) & ✓ & - & ✓ & - & 25.2 & - & 34.5 & - & - & - & - & - & - \\ SATE Xu et al. (2021) & - & - & - & - & 25.2 & - & - & - & - & - & - & - \\ BiKD Imaguna et al. (2021) & - & - & - & - & 25.3 & - & 35.3 & - & - & - & - & - & - \\ Mutual-learning Zhao et al. (2021b) & - & - & - & - & - & 28.7 & 36.3 & - & - & - & - & - & - \\ XSTNet Ye et al. (2021) & ✓ & - & - & - & 25.5 & 29.6 & 36.0 & 25.5 & 30.0 & 31.3 & **25.1** & 16.9 & 27.5 \\

- & **25.7** & **30.4** & **36.8** & **26.3** & **30.6** & **32.0** & 24.8 & **17.3** & **28.0** \\ \hline \multicolumn{87}{c}{_w/external MT data_} \\ \hline MTL Tang et al. (2021b) &
- & ✓ & 23.9 & 28.6 & 33.1 &
- \\ FAT-ST Bigy Zheng et al. (2021) & ✓ & ✓ & ✓ & ✓ & 25.5 & 30.8 &
- & 30.1 &
- \\ JTS-MT Tang et al. (2021a) &
- & ✓ & 26.8 & 31.0 & 37.4 &
- \\ Chimera Han et al. (2021) & ✓ &
- & ✓ & 27.1\({}^{\dagger}\) & 30.6 & 35.6 & 25.0 & 29.2 & 30.2 & 24.0 & 17.4 & 27.4 \\ XYNTet Ye et al. (2021) & ✓ &
- & ✓ & 27.1 & 30.8 & 38.0 & 26.4 & 31.2 & 32.4 & **25.7** & 18.5 & 28.8 \\ SATE Xu et al. (2021) &
- & ✓ & 28.1\({}^{\dagger}\) &
- \\ STEMM Fang et al. (2022) & ✓ &
- & ✓ & 28.7 & 31.0 & 37.4 & 25.8 & 30.5 & 31.7 & 24.5 & 17.8 & 28.4 \\ TaskAware Induthi et al. (2021) &
- & ✓ & ✓ & **28.9** &
- \\ STPT Yang et al. (2022) & ✓ & ✓ & ✓ &
- & **33.1** & **39.7** &

- & ✓ & 28.3 & 32.0 & 38.3 & **27.2** & **31.7** & **33.1** & 25.6 & **18.9** & **29.4** \\ \hline \end{tabular}
\end{table}
Table 1: Case-sensitive detokenized BLEU scores on MuST-C tst–COMMON set. “Speech” denotes unlabeled speech data. “Text“ means unlabeled text data, _e.g._ Europaral V7 Koehn et al. (2005), CC25 Liu et al. (2020). \(\dagger\) use external 40M OpenSubtitles Lison and Tiedemann (2016) MT data. Other models only use WMT data.

\begin{table}
\begin{tabular}{l|c c} \hline \hline  & \multicolumn{2}{c}{**External MT**} \\
**Config.** & _without_ & _with_ \\ \hline ConST & 25.7 & 28.3 \\ \(-\mathcal{L}_{\text{ASR}}-\mathcal{L}_{\text{MT}}\) & 24.6 & 27.0 \\ \(-\mathcal{L}_{\text{ASR}}-\mathcal{L}_{\text{MT}}-\mathcal{L}_{\text{CTR}}\) & 23.6 & 26.3 \\ \hline \hline \end{tabular}
\end{table}
Table 3: BLEU scores on MuST-C En-De test-COMMON set by removing individual losses. We test the results under settings with and without the introduction of external MT data.



sentations derived from the Transformer encoder, denoted as _high-level repr._.

Table 4 shows that contrastive learning using **the low-level representations (Line1) is better than using the high-level ones (Line2)**. On the other hand, although the performance of Line2 is relatively inferior, it still outperforms the multi-task model without the contrastive loss (Line3). The detailed analysis of possible explanations will be shown in Section 6.2.

### Is contrastive loss better than other losses?

Our goal for introducing the contrastive loss term (denoted as CTR Loss) is to close the distance between speech and text representations. Whereas, there are other options to achieve this goal, such as L2 loss and CTC loss.

* **L2 Loss**: Without introducing any negative samples, L2 loss directly reduces the Euclidean distance between the representations of two modalities by minimizing \(\mathcal{L}=\|u-v\|^{2}\). L2 loss can be viewed as an implementation based on the idea of knowledge distillation Heo et al. (2019); Dong et al. (2021).
* **CTC Loss**: The connectionist temporal classification (CTC) loss Graves et al. (2006) is commonly used in speech-related tasks Xu et al. (2021); Dong et al. (2021). Unlike contrastive loss that cares about the representation, CTC loss connects the two modalities by establishing speech-text alignment and maximizing \(p(\mathbf{x}|\mathbf{a})=\sum_{\pi\in\Pi_{\mathbf{s},\mathbf{a}}}\prod_{t=1}^ {T}p_{t}(\pi_{t}|\mathbf{a})\), where \(\Pi_{\mathbf{s},\mathbf{a}}\) is the set of all valid alignments. Compared to the other two ways of bridging the modality gap, L2 and CTC loss, is the contrastive loss term better? The answer is yes according to the results in Table 5. Our explanation is that information on the negative samples benefits the contrastive loss, bringing the the distance between the speech and its corresponding transcription closer while pushing the distance to the irrelevant text farther.

### Analysis on the hard example mining strategies

In Section 3.3, we proposed four methods to mine the hard examples for contrastive learning, namely span-masked augmentation (SMA), word repetition (Rep), sequence cut-off (SCut), and feature cut-off (FCut). In this section, we study how effective these methods are, and to do so, we consider the BLEU performances of their 15 combinations (Figure 4). Note that "Original" means the original contrastive loss in Eq.(4) without any additional hard examples mining operation, and the diagonal in the heat map represents only one strategy used. For an easy and fair comparison, we set the weight of the contrastive term to 1.0 uniformly. We have the following observations.

 

the best result (\(28.3\)), and is better than the model without any expanded operations (\(p<0.01\)). Generally, to find the best model, we suggest adopting multiple strategies and choosing the best checkpoint on the dev-set.

**The combinations of the hard examples mining methods and the "original" have relatively better performances.** We argue that we need the original positive and negative examples to give more accurate representations (without any dropout) for contrastive learning. On the contrary, without the help of "original" loss, the performance with both sequence cut-off and feature cut-off is the worst in Figure 4, probably because too much information is lost by superimposing the two.

## 6 Why does cross-modal contrastive learning work? -- Analysis on the Modality Gap

As mentioned earlier, the existing multi-task training models cannot address the _speech-text modality gap_. Does ConST reduce the representation gap between speech and text?

### Visualization of Representation

**Does the _speech-text modality gap_ exist without explicitly bridging the two? _Speech-text modality gap_ means the discrepancy between the audio representations and transcription sentence embeddings. To visualize it, we plot the bivariate kernel density estimation Parzen (1962) (KDE) contour of their dim-reduced features, where T-SNE Van der Maaten and Hinton (2008) is used to reduce the dimension into two (Figure 5). Ideally, if the representations of speech and its corresponding transcript are similar, their KDEs will be similar, and thus the contour lines will overlap as much as possible. However, Figure 5(a) is the KDE contour of the multi-task framework without any explicit modeling to bring two modalities together Ye et al. (2021). It shows that the representations are so dissimilar that they are organically divided into two clusters, _i.e.__speech-text modality gap_ exists.

**Does ConST reduce the modality gap?** As shown in Figure 5(b), compared to the baseline model without contrastive learning, ConST with cross-modal contrastive learning is able to bring representations of different modalities much closer. This means that the audio representation contains more linguistic information similar to that of the textual transcription, which is more advantageous for the downstream ST generation through the shared _Transformer encoder_ and _decoder_.

### Cross-modal Retrieval

**How good is the cross-modal representation space learned from ConST?** To answer this question, we conduct a retrieval experiment, _i.e._ finding the nearest (smallest cosine similarity) transcript based on the speech representation. We compare ConST model with the baseline without cross-modal contrastive learning and report the top-1 retrieval accuracy using (1) the low-level representations and (2) the high-level semantic representations, in Table 7.

When retrieving the text using low-level representations, our method gains a substantial 79% increase over the baseline. In addition, we find that without explicit contrastive modeling, the baseline can achieve retrieval accuracy of more than \(94\%\) according to the semantic representations outputted from the _Transformer encoder_. We believe that such high accuracy is automatically learned from the triple-supervised data itself under the multi-task learning framework. With such a degree of cross-modal alignment, if we construct the contrastive loss with semantic representations, its gain to the ST performance turns out to be limited, which exactly corroborates the findings in Section 5.2 - low-level representations are preferred in the cross-modal contrastive learning.

## 7 Case Analysis

In this section, we use several cases that ConST generates. We compare our model with the cascaded model and the previous end-to-end model,

Figure 5: Bivariate KDE contour plot of the representation of speech and transcript in source language English. T-SNE is used to reduce into 2D. The blue lines are the audio representations and the red dashed lines stand for text. (a) for the vanilla multi-task framework without any extra supervision. (b) for our proposed ConST model. Sentences are from En-De test set.

 

XSTNet (Ye et al., 2021).

For this first case, the cascaded system fails to give a right translation due to the mis-punctuation issue (_klingt_ is a verb), while the end-to-end model, XSTNet and ConST translate correctly. For the second case, the previous end-to-end XSTNet model cannot accurately translate the phrase "started exploring this idea of", which performs worse than the cascaded one. Whereas ConST successfully conveys the meaning of "this idea" , and translates more accurately than XSTNet. We believe this improvement comes from the cross-modal contrastive learning.

## 8 Conclusion

In this paper, we propose ConST, a simple yet effective contrastive learning framework bridging the speech-text representation gap and facilitating the ST with limited data. We also provide feasible hard example mining methods to learn robust representations. The results on the MuST-C ST dataset prove the effectiveness of the method.

## 9 Broader Impact

This work improves the performance of ST tasks on public datasets by learning speech representations that are more similar to text representations, but the model is far from being achieved for industrial-grade implementations. In real scenarios, for example, the original voice is noisier and the distribution of speech lengths is more complex than in the public dataset, which cannot be handled by an end-to-end model alone. The shortcoming of this model is that it still needs a certain amount of labeled data for training, especially _<speech,transcription>_ to learn better speech representation, and for the more than \(7,000\) languages and dialects in the world, most of them do not have corresponding translations or even transcriptions, our method does not work in untranscribed scenarios. In this paper, we focus on the improvement brought by the better speech representation on the ST task, and obtained good results with hundreds of hours of speech data. We hope that our work achieves better results using more data (_e.g._ raw speech, raw text, ASR, MT data) in the future.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Models** & \multicolumn{3}{c}{**case 1**} \\ \hline Ref. & **src:** & Lights, sounds, solar panels, motors — everything should be accessible. \\  & **tgt:** & Lichter, Tône, Solarelemente, Motoren — alles sollte verflugbar sein. \\ \hline Cascaded & **src:** & Lights sounds solar panels motors everything should be accessible. \\  & **tgt:** & Licht klingt Solarpanenele, Motoren; alles sollte zuganglich sein. \\ \hline XSTNet & **tgt:** & Licht, Geräusche, Solardoulektoren, Motoren — alles sollte zuganglich sein. \\ \hline ConST & **tgt:** & Licht, Geräüusche, Solarpanele, Motoren, alles sollte zugänglich sein. \\ \hline \multirow{4}{*}{Ref.} & **case 2** \\ \cline{2-3}  & **src:** & Eight years ago when I was at the Media Lab, I started exploring this idea of how to put the power of engineers in the hands of artists and designers. \\  & **tgt:** & Vor acht Jahren war ich am Media Lab und ich begann diese Idee zu erforschen, wie man die Macht der Ingenièure in die Hand von Kunstlern und Designern legen konnte. \\ \hline Cascaded & **src:** & Eight years ago when I was at the Media Lab, I started exploring this idea of how to put the power of engineerns in the hands of artists and designers. \\  & **tgt:** & Vor bathern, as ich im Media Lab war, began ich, diese Idee zur Forschen, wie man die Macht der Ingeneiure in die Hände von Kunstlern und Designern lege. \\ \hline XSTNet & **tgt:** & Vor acht Jahren, als ich im Media Lab war, beginan ich zu erforschen, wie man die Kraft der Ingenièure in die Hände von Kunstlern und Designern legt. \\ \hline ConST & **tgt:** & Vor acht Jahren, als ich im Media Lab war, begunan ich, diese Idee zu erforschen, wie man die Macht von Ingenièuren in die Hände von Kunstlern und Designern legt. \\ \hline \hline \end{tabular}
\end{table}
Table 6: En-De test cases that generated from the cascaded model, XSTNet (both provided by Ye et al. (2021)) and our ConST model. The red underlined text indicates grammatically incorrect or inaccurate translations.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Representations** & **CTR loss** & **Acc.** \\ \hline low-level repr. & ✗ & 9.4 \\  & ✓ & **88.6** \\ \hline high-level repr. & ✗ & 94.7 \\  & ✓ & **95.0** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Cross-modal top-1 retrieval accuracy (%) on En-De test set. Two different representations are used, based on which, ConST achieves huge accuracy improvements.

 



## References

* A. Alinejad and A. Sarkar (2020)Effectively pretraining a speech translation decoder with machine translation data. In Proc. of EMNLP, pp. 8014-8020. Cited by: SS3.5.
* E. Ansari, A. Axelrod, N. Bach, O. Bojar, R. Cattoni, F. Dalvi, N. Durrani, M. Federico, C. Federmann, J. Gu, et al. (2020)Findings of the iwslt 2020 evaluation campaign. In Proc. of IWSLT, pp. 1-34. Cited by: SS1.
* A. Baewski, Y. Zhou, A. Mohamed, and M. Auli (2020)Wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proc. of NeurIPS, Cited by: SS1.
* S. Bansal, H. Kamper, K. Livescu, A. Lopez, and S. Goldwater (2019)Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. In Proc. of NAACL-HLT, pp. 58-68. External Links: Link Cited by: SS1.
* L. Bentivogli, M. Cettolo, M. Gaido, A. Karakanta, A. Martinelli, M. Negri, and M. Turchi (2021)Cascade versus direct speech translation: do the differences still make a difference?. In Proc. of ACL, Cited by: SS1.
* A. Berard, D. Pietquin, C. Servan, and L. Besacier (2016)Listen and translate: a proof of concept for end-to-end speech-to-text translation. In NIPS workshop on End-to-end Learning for Speech and Audio Processing, Cited by: SS1.
* O. Bojar, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, M. Huck, A. Jimeno Yepes, P. Koehn, V. Logacheva, C. Monz, M. Negri, A. Neveol, M. Neves, M. Popel, T. Post, R. Rubino, C. Scarton, L. Specia, M. Turchi, K. Verspoor, and M. Zampieri (2016)Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 131-198. Cited by: SS1.
* S. Cao and L. Wang (2021)Cliff: contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proc. of ACL, pp. 6633-6649. Cited by: SS1.
* J. Chen, M. Ma, R. Zheng, and L. Huang (2021)Specrec: an alternative solution for improving end-to-end speech-to-text translation via spectrogram reconstruction. Proc. of InterSpeech, pp. 2232-2236. Cited by: SS1.
* T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton (2020a)A simple framework for contrastive learning of visual representations. In Proc. of ICML, Vol. 119 of Proceedings of Machine Learning Research, pp. 1597-1607. Cited by: SS1.
* Y. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu (2020b)Uniter: universal image-text representation learning. In Proc. of ECCV, pp. 104-120. Cited by: SS1.
* S. Chopra, R. Hadsell, and Y. LeCun (2005)Learning a similarity metric discriminatively, with application to face verification. In Proc. of CVPR, Vol. 1, pp. 539-546. Cited by: SS1.
* M. A. Di Gangi, R. Cattoni, L. Bentivogli, M. Negri, and M. Turchi (2019)MuST-c: a multilingual speech translation corpus. In Proc. of NAACL-HLT, pp. 2012-2017. Cited by: SS1.
* L. Dong, X. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H. Hon (2019)Unified language model pre-training for natural language understanding and generation. In Proc. of NeurIPS, pp. 13063-13075. Cited by: SS1.
* L. Dong, S. Xu, and B. Xu (2018)Speechtransformer: a no-recurrence sequence-to-sequence model for speech recognition. In Proc. of ICASSP, pp. 5884-5888. Cited by: SS1.
* Q. Dong, M. Wang, H. Zhou, S. Xu, B. Xu, and L. Li (2021)Consecutive decoding for speech-to-text translation. In Proc. of AAAI, Cited by: SS1.
* Q. Dong, R. Ye, M. Wang, H. Zhou, S. Xu, B. Xu, and L. Li (2021)Listen, understand and translate: triple supervision decouples end-to-end speech-to-text translation. In Proc. of AAAI, Vol. 35, pp. 12749-12759. Cited by: SS1.
* Hongchao Fang, S. Wang, M. Zhou, J. Ding, and P. Xie (2020)CFT: contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766. Cited by: SS1.
* Q. Fang, R. Ye, L. Li, Y. Feng, and M. Wang (2022)Stemm: self-learning with speech-text manifold mixup for speech translation. In Proc. of ACL, Cited by: SS1.
* Z. Fu, W. Zhou, J. Xu, H. Zhou, and L. Li (2022)Contextual representation learning beyond masked language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 2701-2714. Cited by: SS1.
* T. Gao, X. Yao, and D. Chen (2021)SimCSE: simple contrastive learning of sentence embeddings. In Proc. of EMNLP,

Alex Graves, Santiago Fernandez, Faustino J. Gomez, and Jurgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In _Proc. of ICML_, volume 148, pages 369-376.
* Greff et al. (2016) Klaus Greff, Rupesh K Srivastava, Jan Koutnik, Bas R Steunebrink, and Jurgen Schmidhuber. 2016. Lstm: A search space odyssey. _IEEE transactions on neural networks and learning systems_, 28(10):2222-2232.
* Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. _Proc. of NeurIPS_, 33:21271-21284.
* Gutmann and Hyvarinen (2010) Michael Gutmann and Apo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_, pages 297-304. JMLR Workshop and Conference Proceedings.

* Heo et al. (2019) Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. 2019. A comprehensive overhaul of feature distillation. In _Proc. of the ICCV_, pages 1921-1930.
* Hu et al. (2020) Yushi Hu, Shane Settle, and Karen Livescu. 2020. Multilingual jointly trained acoustic and written word embeddings. In _Proc. of INTERSPEECH_.
* Inaguma et al. (2021) Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021. Source and target bidirectional knowledge distillation for end-to-end speech translation. In _Proc. of NAACL_, pages 1872-1881.
* Inaguma et al. (2020) Hirofumi Inaguma, Shu Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. ESPnet-ST: All-in-one speech translation toolkit. In _Proc. of ACL_, pages 302-311.
* Indurthi et al. (2020) Sathish Indurthi, Houjeuang Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. Data efficient direct speech-to-text translation with modality agnostic meta-learning. In _Proc. of ICASSP_. IEEE.
* Indurthi et al. (2021) Sathish Indurthi, Mohd Abbas Zaidi, Nikhil Kumar Lakumarapu, Beomseok Lee, Iyojung Han, Seokchan Ahn, Sangha Kim, Chanwoo Kim, and Inchl Hwang. 2021. Task aware multi-task learning for speech to text tasks. In _Proc. of ICASSP_, pages 7723-7727. IEEE.
* Kamper et al. (2020) Herman Kamper, Yevgen Matusevych, and Sharon Goldwater. 2020. Multilingual acoustic word embedding models for processing zero-resource languages. In _Proc. of ICASSP_, pages 6414-6418. IEEE.
* Kano et al. (2017) Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2017. Structured-based curriculum learning for end-to-end english-japanese speech translation. In _Proc. of INTERSPEECH_, pages 2630-2634.
* Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. _Proc. of NeurIPS_, 33.
* Boehn et al. (2005) Philipp Koehn et al. 2005. Europarl: A parallel corpus for statistical machine translation. In _MT summit_, volume 5, pages 79-86. Citeseer.
* Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In _Proc. of EMNLP_, pages 66-71.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In _Proc. of NeurIPS_.
* Le et al. (2020) Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In _Proc. of COLING_, pages 3520-3533.
* Le et al. (2021) Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2021. Lightweight adapter tuning for multilingual speech translation. In _Proc. of ACL_.
* Li et al. (2021) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. In _Proc. of ACL_.
* Lision and Tiedemann (2016) Pierre Lison and Jorg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In _Proc. of LREC_, pages 923-929.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. _TACL_, 8:726-742.
* Liu et al. (2020b) Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020b. Bridging the modality gap for speech-to-text translation. _arXiv preprint arXiv:2010.14920_.
* Lu et al. (2019)Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. _Proc. of NeurIPS_, 32 

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_.
* Demonstrations_, pages 48-53.
* Palaskar et al. (2019) Shruti Palaskar, Vikas Raunak, and Florian Metze. 2019. Learned in speech recognition: Contextual acoustic word embeddings. In _Proc. of ICASSP_, pages 6530-6534. IEEE.
* Pan et al. (2021) Xiao Pan, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In _Proc. of ACL_.
* Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In _Proc. of ICASSP_, pages 5206-5210.
* Papi et al. (2021) Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2021. Speechformer: Reducing information loss in direct speech translation. In _Proc. of EMNLP_.
* Park et al. (2019) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quoc V. Le. 2019. Specaugment: A simple augmentation method for automatic speech recognition. In _Proc. of INTERSPEECH_.
* Parzen (1962) Emanuel Parzen. 1962. On estimation of a probability density function and mode. _The annals of mathematical statistics_, 33(3):1065-1076.
* Pino et al. (2020) Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020. Self-training for end-to-end speech translation. In _Proc. of INTERSPEECH_, pages 1476-1480.
* Popovic (2017) Maja Popovic. 2017. chrf++: words helping character n-grams. In _Proceedings of the second conference on machine translation_, pages 612-618.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191.
* Potapczyk and Przybysz (2020) Tomasz Potapczyk and Pawel Przybysz. 2020. Srpol's system for the iwslt 2020 end-to-end speech translation task. In _Proc. of IWSLT_, pages 89-94.
* Ravanelli and Bengio (2018) Mirco Ravanelli and Yoshua Bengio. 2018. Learning speaker representations with mutual information. _arXiv preprint arXiv:1812.00271_.
* Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In _Proc. of INTERSPEECH_.
* Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In _Proc. of CVPR_, pages 815-823.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In _Proc. of ACL_.
* Shen et al. (2020) Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. _arXiv preprint arXiv:2009.13818_.
* Sohn (2016) Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In _Proc. of NeurIPS_, volume 29, pages 1857-1865. Curran Associates, Inc.
* Tang et al. (2022) Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, et al. 2022. Unified speech-text pre-training for speech translation and recognition. In _Proc. of ACL_.
* Tang et al. (2021a) Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving speech translation by understanding and learning from the auxiliary text translation task. In _Proc. of ACL_.
* Tang et al. (2021b) Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021b. A general multi-task learning framework to leverage text data for speech to text tasks. In _Proc. of ICASSP_, pages 6209-6213. IEEE.
* Oord et al. (2018) Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, NaI Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. 2018. Parallel wavenet: Fast high-fidelity speech synthesis. In _Proc. of ICML_, volume 80 of _Proceedings of Machine Learning Research_, pages 3915-3923.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(11).
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proc. of NeurIPS_, pages 5998-6008.
* Wang et al. (2020a) Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq s2t: Fast speech-to-text modeling with fairseq. In _Proc. of AACL_, pages 33-39.
* Wang et al. (2021a) Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, and Alexis Conneau. 2021a. Large-scale self-and semi-supervised learning for speech translation. In _Proc. of INTERSPEECH_ 

Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020b. Curriculum pre-training for end-to-end speech translation. In _Proc. of ACL_, pages 3728-3738.

* Weiss et al. (2017) Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In _Proc. of INTERSPEECH_, pages 2625-2629.
* Wu et al. (2020) Anne Wu, Changhan Wang, Juan Pino, and Jiatao Gu. 2020. Self-supervised representations improve end-to-end speech translation. In _Proc. of INTERSPEECH_.
* Wu et al. (2019) Hao Wu, Jiayuan Mao, Yufeng Zhang, Weiwei Sun, Yuning Jiang, Lei Li, and Wei-Ying Ma. 2019. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In _Proc. of CVPR_.
* Wu et al. (2021) Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021. EsimCSE: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. _arXiv preprint arXiv:2109.04380_.
* Xu et al. (2021) Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Qi Ju, Tong Xiao, Jingbo Zhu, et al. 2021. Stacked acoustic-and-textual encoding: Integrating the pretrained models into speech translation encoders. In _Proc. of ACL_.
* Yan et al. (2021) Yuanmeng Yan, Rumei Li, Siriu Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A contrastive framework for self-supervised sentence representation transfer. In _Proc. of ACL_.
* Ye et al. (2021) Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to-end speech translation via cross-modal progressive training. In _Proc. of INTERSPEECH_.
* Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In _Proc. of ACL_, pages 1628-1639.
* System Demonstrations_.
* Zhao et al. (2021b) Jiawei Zhao, Wei Luo, Boxing Chen, and Andrew Gilman. 2021b. Mutual-learning improves end-to-end speech translation. In _Proc. of the EMNLP_, pages 3989-3994.
* Zheng et al. (2021) Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. 2021. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. In _Proc. of ICML_.
* Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In _Proc. of AAAI_, volume 34, pages 13041-13049.

 



## Appendix A Statistics of all datasets

### Experimental Details

#### Training and Implementation Details

We use Adam optimizer (\(\beta_{1}=0.9,\beta_{2}=0.98\)) with learning rate \(=1e^{-4}\) and warmup 25k steps during the ST training. We also implement the expanded setting with the introduction of external WMT to train the Transformer module. In the pre-training stage, we set the learning rate \(=7e^{-4}\) and warmup 4000 steps. For robust training, we set label smoothing to \(0.1\), and dropout rate to \(0.1\). The hyper-parameters for different data augmentation methods are as follows: for masked audio span strategy, we set masking probability \(p=0.25\) and masking span length \(M=3600\) frames; for both sequence and feature cut-off, we set the cut-off dropout rate as \(0.1\). We save the checkpoint with the best BLEU on dev-set and average the last 10 checkpoints. For decoding, we use a beam size of 10 and length penalty \(0.7\) for German, \(1.0\) for French, and \(0.4\) for Russian. We train the models in 8 Nvidia Tesla V100 GPUs for each experiment. We use Fairseq Ott et al. (2019) as the code-base for our implementation.

#### Baseline Models

In Table 1, we compared our method with end-to-end baseline models whose audio inputs are 80-channel log Mel-filter bank, including: FairseqST Wang et al. (2020), NeurST Zhao et al. (2021), Espnet ST Inaguma et al. (2020), Dual-decoder Transformer Le et al. (2020), SATE Xu et al. (2021), Speechformer Papi et al. (2021), self training Pino et al. (2020) and mutual learning Zhao et al. (2021) method, STAST Liu et al. (2020), bi-KD Inaguma et al. (2021), MLT method Tang et al. (2021), Lightweight Adaptor Le et al. (2021), JT-S-MT Tang et al. (2021), FAT-ST Zheng et al. (2021), TaskAware Indurthi et al. (2021), and STPT Tang et al. (2022). We also compare our method to baseline models that have pretrained Wav2vec2.0 as a module, including:

* **W-Transf.**Ye et al. (2021): the model has the same structure as ours, but is only trained on _<speech, translation>_ parallel data.
* **Chimera-ST**Han et al. (2021): the model that builds a shared semantic memory for both audio and text modalities.
* **XSTNet**Ye et al. (2021): the model has the same structure as ours, and adopted a multi-task fine-tuning strategy.
* **STEMM**Fang et al. (2022): the model that bridges the modality representation gap by minimizing the Jensen-Shannon divergence between the original speech representation and the manifold mix-up representation.

## Appendix C The Choice for Hyper-parameters

#### Influence of Temperature

In the contrastive loss, the temperature hyper-parameter is provided to control the smoothness of the distribution normalized by softmax operation. A high temperature helps to smooth the distribution, making it more difficult for the model to distinguish between positive and negative samples (corresponding to correct transcriptions and other transcriptions in this work), while the low temperature behaves just the opposite. We choose several temperature hyper-parameters ranging from \(0.01\) to \(0.5\), and Figure 6 shows their BLEUs on the test and dev sets . We find that (1) the choice of the temperature does not drastically affect the final BLEU score, and (2) we recommend that the temperature \(\tau\) be set between 0.02 and 0.05 to ensure a relatively good ST performance. In the experiment, we use \(\tau=0.02\).

Figure 6: En-De BLEU scores on tst-COMMON and Dev set. the x-axis is the choices of different temperature \(\tau\) in Eq.(4) varying from \(0.01\) to \(0.5\).



Influence of Contrastive Loss WeightThe total loss we optimize, Eq.(1), is a linear combination of the multi-task cross-entropy losses \({\mathcal{L}_{\text{MLT}}}\) and the contrastive term \({\mathcal{L}_{\text{CTR}}}\). To investigate how much the contrastive terms affect BLEU, we fix its temperature \(\tau=0.02\), adjust the values of its loss weight \(\lambda\) from 0.1 to 2.0, performed three experiments for each value, and test the average BLEU on En-De tst-COMMON set. Figure 7 depicts the performances. First, all objective functions containing \({\mathcal{L}_{\text{CTR}}}\), even if their weights \(\lambda\) take different values, are apparently better than the baseline model with \({\mathcal{L}_{\text{MLT}}}\) only \({\mathcal{L}_{\text{CTR}}}\). Then, the best BLEU score is achieved at loss weight \(\lambda=1.5\), corresponding to the results in Table 1. And when analyzing the effect of data augmentation strategies (Section 5.4), since we need to consider the combination between them, which is more complicated. Therefore, we set the loss weight to \(1.0\) uniformly for simplicity. In general, we recommend that the weight hyper-parameter takes a value between \(0.8\) and \(1.5\).

## Appendix D Data Scale for Fine-tuning

The experiments in the main paper show that our model can perform well by introducing external MT data pre-training. Here, we simulate the scenario with plenty of MT and speech data and limited ST triple-labeled data, and does ConST have the ability of low-resource learning? In the experiment, we reduce the labeled ST data to 1, 10, and 100 hours, corresponding to sentence counts of about 500, 5k, and 50k sentences. For a fair comparison, we use the same MT pre-trained Transformer module as in the main paper. We find the contrastive loss particularly helpful when the amount of speech data is extremely small, like only 1 hour of speech. Second, the multi-task training strategy is also very effective in improving the robustness of the model performance. We also find that by using easily accessible MT and speech pre-training, our model could reach the previous baseline results without pre-training using only \(1/4\) of the original data, _i.e._\(100\) hours of labeled ST data.

Figure 8: En-De BLEU scores on tst-COMMON sets. The horizontal axis is the amount of ST data (in hours of speech).

Figure 7: En-De BLEU scores on tst-COMMON and Dev sets. The x-axis is the weight of the contrastive loss term \(\lambda\) in Eq.(1). Experiments are performed under the fix temperature hyper-parameter \(\tau=0.02\).