SlowGuess/NAACL_2022_OCR huggingface dataset:
Rong Ye $^{1}$ , Mingxuan Wang $^{1}$ , Lei Li $^{2*}$
<sup>1</sup> ByteDance AI Lab, <sup>2</sup> University of California, Santa Barbara
{yerong, wangmingxuan.89}@bytedance.com
leili@cs.ucsb.edu
How can we learn unified representations for spoken utterances and their written text? Learning similar representations for semantically similar speech and text is important for speech translation. To this end, we propose ConST, a cross-modal contrastive learning method for end-to-end speech-to-text translation. We evaluate ConST and a variety of previous baselines on a popular benchmark MuST-C. Experiments show that the proposed ConST consistently outperforms the previous methods, and achieves an average BLEU of 29.4. The analysis further verifies that ConST indeed closes the representation gap of different modalities — its learned representation improves the accuracy of cross-modal speech-text retrieval from  $4\%$  to  $88\%$ . Code and models are available at https://github.com/ReneeYe/ConST.
End-to-end speech-to-text translation (E2E ST) becomes important in many internet products and real applications. An E2E ST system accepts audio signals as the input and generates the target translation using a single model. Compared with the conventional cascade ST models, E2E ST models have achieved almost comparable (Bentivogli et al., 2021) or even superior (Ansari et al., 2020; Potapczyk and Przybysz, 2020; Xu et al., 2021) performance.
The performance of an E2E ST model is still restricted for languages with relatively small parallel data, compared to text machine translation (MT). Existing approaches for ST focus on using additional data from MT and automatic speech recognition (ASR). This can be realized through pretraining approaches (Zheng et al., 2021; Dong et al., 2021b,a) or multi-task training frameworks (Tang et al., 2021b; Ye et al., 2021; Han et al., 2021).
(a) Current
(b) Ideal
Figure 1: Illustration of representations for speech and transcript text (projected to 2D). (a) representations learned by existing models. Pairs of speech and text representations are distant. (b) an ideal representation that we expect, where different modalities with same meaning should stay close to each other.
Different from the data perspective, this paper investigates the bottleneck of E2E ST from the neural representation perspective. We believe that when the representation of audio input is similar to its corresponding textual representation, it is easier for information to transfer from MT to ST, thus improving speech translation performance.
We analyze Transformer models for speech translation and observe a noticeable modality gap between encoder representations of speech and text from existing ST models (as in Figure 1a. Sec. 6 has more details). An ideal representation should satisfy: if the content of the speech and transcription are similar, their encoded representations should likewise be close to each other (as in Figure 1b). Nevertheless, how to learn unified and aligned speech-text representations?
Inspired by the recent progress of contrastive learning approaches in cross-lingual (Lample and Conneau, 2019; Pan et al., 2021) and cross-modal vision-and-language domains (Li et al., 2021; Zhou et al., 2020; Dong et al., 2019), we designed a simple contrastive learning method for ST (ConST) to learn the representations that meet the aforementioned conditions explicitly. On the one hand, our model inherits the advantages of the previous
*Partial work was done while at ByteDance.
Human Language Technologies, pages 5099 - 5113
multi-task learning methods. On the other hand, it reduces the gap between the representations of speech and its corresponding transcription.
- We develop ConST for speech translation, a cross-modal contrastive learning method, on top of the multi-task training framework.
- Our experiments on the MuST-C benchmark show that ConST achieves an average BLEU score of 29.4, outperforming the best previous baselines.
- We demonstrate that ConST indeed learns similar representations for two modalities and better retrieves text with speech input.
End-to-end ST To alleviate the error propagation in the cascaded ST systems and to make the deployment simpler, Bérard et al. (2016); Weiss et al. (2017) proposed to use an end-to-end architecture to directly translate speech into text in another language, without the intermediate transcription. Kano et al. (2017); Berard et al. (2018); Inaguma et al. (2020); Wang et al. (2020a); Zhao et al. (2021a) implemented several off-the-shelf encoder-decoder E2E-ST models, such as BiLSTM (Greff et al., 2016) and Speech-Transformer (Dong et al., 2018). However, training an end-to-end speech translation model is difficult because we need to design a cross-modal cross-language model, meanwhile, the speech-transcription-translation supervised data for speech translation is significantly less than that of MT and ASR. Methods, like data augmentation (Park et al., 2019; Pino et al., 2020; Chen et al., 2021), pre-training (Weiss et al., 2017; Berard et al., 2018; Bansal et al., 2019; Wang et al., 2020b; Alinejad and Sarkar, 2020; Dong et al., 2021a; Zheng et al., 2021), self-training (Pino et al., 2020; Wang et al., 2021a), utilizing self-supervised pre-trained audio representation (Wu et al., 2020; Han et al., 2021; Ye et al., 2021; Wang et al., 2021a), are proved to be effective. Meanwhile, some work has shown that the encoder-decoder model with a single encoder cannot encode speech information well. For example, Dong et al. (2021b) first proposed a second encoder to further extract semantic information of the speech sequence. Xu et al. (2021) proposed a stacked acoustic-and-textual encoder and introduced large-scale out-of-domain data. Also, multi-task frameworks (Le et al., 2020; Tang et al., 2021b; Ye et al.,
2021) are widely applied to further enhance the robustness for ST. As a cross-modal task, some work has noted the problem of the modality gap. (Han et al., 2021) designed a fix-size semantic memory module to bridge such a gap, from the neuroscience perspective. However, we find that this approach actually sacrifices the effect of MT. So in this paper, we propose a simple yet effective contrastive learning method to bridge the gap and to improve ST performance.
Cross-modal grounding learning This paper attempts to address the problem in speech translation from the perspective of cross-speech-text representation learning. We are also inspired by cross-modal representation learning in the acoustic word embedding (AWE) (Palaskar et al., 2019; Kamper et al., 2020; Hu et al., 2020) and the visual-language pre-training (VLP) (Wu et al., 2019; Lu et al., 2019; Chen et al., 2020b; Li et al., 2021) tasks. These works usually focus on enhancing textual representations with acoustic or visual information, in other words, grounding learning. In this work, we consider the its dual form, i.e., grounding speech representations using text.
Contrastive learning Our method is motivated by the recent success in contrastive representation learning. The contrastive learning method was first proposed to learn representations from unlabeled datasets (hence the term, self-supervised learning) by telling which data points are similar or distinct, especially in the field of computer vision (Chopra et al., 2005; Gutmann and Hyvarinen, 2010; Schroff et al., 2015; Sohn, 2016; Oord et al., 2018; Chen et al., 2020a; Grill et al., 2020). Khosla et al. (2020) extended the self-supervised batch contrastive approach to the fully-supervised setting and proposed a supervised contrastive learning method. In speech processing, representative methods focused on speaker identification (Ravanelli and Bengio, 2018), speech recognition (Schneider et al., 2019), and audio representation learning (van den Oord et al., 2018; Baevski et al., 2020). In the NLP area, the contrastive framework is used for sentence representation learning (Fang et al., 2020; Shen et al., 2020; Gao et al., 2021; Wu et al., 2021; Yan et al., 2021; Fu et al., 2022), machine translation (Pan et al., 2021), and summarization (Wang et al., 2021b; Cao and Wang, 2021). Very recently, contrastive learning is also applied to learning a unified representation of image and text (Dong et al., 2019; Zhou et al., 2020; Li et al., 2021). Moti
(a) Overall Model Structure
(b) Cross-modal Contrastive Learning
Figure 2: Left: Model architecture of ConST. The gray shaded modules are the optional data augmentation operations introduced in Section 3.3. Right: An illustration of cross-modal contrastive learning.
vated by the contrastive learning frameworks in cross-lingual and cross-modal topics, we introduce a similar idea in speech translation.
3 The ConST Approach
An end-to-end speech translation model directly translates audio sequence  $\mathbf{s} = (s_1, \dots, s_{|\mathbf{s}|})$  to the text  $\mathbf{y} = (y_1, \dots, y_{|\mathbf{y}|})$  in the target language. Speech translation corpus  $\mathcal{D} = \{(\mathbf{s}, \mathbf{x}, \mathbf{y})\}$  provides transcript  $\mathbf{x} = (x_1, \dots, x_{|\mathbf{x}|})$  in the source language, as well.
In this section, we present the overall speech translation model and cross-modal contrastive learning. We also provide several feasible strategies to construct more positive and negative pairs to enhance the contrastive learning.
3.1 Model Framework
We use the same model architecture as XSTNet (Ye et al., 2021). Our model consists four sub-modules: a speech encoder, a word embedding layer, a Transformer Encoder and a Transformer decoder (Figure 2). It is designed to take either speech or a sentence as input, and to output either source transcript or target translation text. Such architecture enables a universal framework for multiple tasks, including ST, MT and ASR.
The speech encoder module (S-Enc) is designed to extract low-level features for speech signals. It contains Wav2vec2.0 (Baevski et al., 2020) and two additional convolutional layers. The input is raw waveform signal sampled at  $16\mathrm{kHz}$ . Each convolutional layer has a stride of 4 and  $d$  channels. In total, it shrinks the time dimension by a factor of 4.
Denote  $\mathbf{a} = \mathrm{S - Enc}(\mathbf{s})$  as the audio representation of the speech,  $|\mathbf{a}|\ll |\mathbf{s}|$
Parallel to the speech encoder is the word embedding layer. It is the same as word embedding for text translation.
Both the speech encoder and word embedding layer are connect to Transformer encoder and then passed to the Transformer decoder. The Transformer encoder and decoder are using the same configuration as the original (Vaswani et al., 2017). To explain, the Transformer encoder further extracts the high-level semantic hidden representation of two modalities. The Transformer decoder generates the word sequences (transcription and translation) for ST, MT and ASR tasks. Since our model has a complete Transformer encoder-decoder as a sub-module, this makes it possible to pre-train using large-scale extra MT parallel data.
Previous work has shown that multi-task learning on ST, MT and ASR improves translation performance (Indurthi et al., 2020; Tang et al., 2021b; Ye et al., 2021). Our training loss consists of the following elements.
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {S T}} + \mathcal {L} _ {\mathrm {A S R}} + \mathcal {L} _ {\mathrm {M T}} + \lambda \mathcal {L} _ {\mathrm {C T R}} \tag {1}
$$
$$
\mathcal {L} _ {\mathrm {S T}} = - \sum_ {n} \log P (\mathbf {y} _ {n} | \mathbf {s} _ {n})
$$
$$
\mathcal {L} _ {\mathrm {A S R}} = - \sum_ {n} \log P (\mathbf {x} _ {n} | \mathbf {s} _ {n})
$$
$$
\mathcal {L} _ {\mathrm {M T}} = - \sum_ {n} \log P (\mathbf {y} _ {n} | \mathbf {x} _ {n})
$$
The first three elements are cross-entropy losses on <speech, target text>, <speech, source text> and <source text, target text> pairs. These pairs are built from the triplet ST data. We also introduce a cross-modal contrastive loss term  $\mathcal{L}_{\mathrm{CTR}}$  (see Section 3.2 for details). It aims to bring the representation between the speech and textual transcription modalities closer (its effect will be analyzed in detail in Section 6).  $\lambda$  is a tuned hyper-parameter of the weighted contrastive loss term.
3.2 Cross-modal Contrastive Learning
As mentioned in the beginning, since we need to produce similar representations for the speech and transcript sharing the same semantic meanings, we propose cross-modal contrastive learning method to bring their representations closer together. The main idea of cross-modal contrastive learning is to introduce a loss that brings speech and its corresponding transcript (positive example) near together while pushing irrelevant ones (negative examples) far apart.
Given a positive example of such a speech-transcript pair  $(\mathbf{s},\mathbf{x})$ , we randomly pick a set of  $N - 1$  transcripts  $\{\mathbf{x}_i^-\}_{i = 1}^{N - 1}$  from the same batch as negative examples. For speech s and its transcript x, we first average them in terms of the time dimension,
$$
u = \operatorname {M e a n P o o l} (\mathrm {S} - \operatorname {E n c} (\mathrm {s})) \tag {2}
$$
$$
v = \operatorname {M e a n P o o l} (\operatorname {E m b} (\mathbf {x})) \tag {3}
$$
and apply the multi-class N-pair contrastive loss (Sohn, 2016):
$$
\mathcal {L} _ {\mathrm {C T R}} = - \sum_ {s, x} \log \frac {\exp (\operatorname {s i m} (u , v) / \tau)}{\sum_ {x _ {j} \in \mathcal {A}} \exp (\operatorname {s i m} (u , v (x _ {j})) / \tau)} \tag {4}
$$
where  $\mathcal{A} = \{\mathbf{x}\} \cup \{\mathbf{x}_i^-\}_{i=1}^{N-1}$ ,  $\tau$  is the temperature hyper-parameter, and  $sim$  is the cosine similarity function  $sim(a,b) = a^\top b / \|a\| \|b\|$ . In the implementation, negative examples  $\{\mathbf{x}_i^-\}_{i=1}^{N-1}$  are from the same training batch of data (Figure 2(b)).
3.3 Mining Hard Examples for Contrastive Learning
To further enhance the contrastive learning, we introduce three strategies to mine additional hard examples. These strategies are at input and representation (gray shaded modules in Figure 2(a)). Specific schematic illustrations of each operations are shown in Figure 3.
Figure 3: Schematic illustration of the hard examples mining strategies. In the cut-off strategy, the gray shaded grid represents the zero-out element.
Span-Masked Augmentation We mask consecutive segments of an original audio waveform sequence  $s$  to obtain a new modified speech  $s'$ . We take  $s'$  as an input to the model, and compute the contrastive loss on its original corresponding transcript. We randomly sample without replacement all time steps in the original waveform of the speech to be the starting indices with a probability  $p$ , and then we set the sub-sequence  $M$  successive time steps to be blank. In the experiment, we tried multiple configurations, and found  $p = 0.25$ ,  $M = 3600$  the best, resulting in a masked span of 0.225 second. Since the masked speech fragment is very short, we consider the masked speech and the original transcript to be positive pairs, and the remaining transcripts in the same batch to be negative pairs.
Word Repetition The word repetition strategy randomly replicates some words (or sub-words) in the original sentences, with two advantages for improving representation robustness. First, as the length of the sentence is shorter than that of its audio representation, randomly repeating the words in the sentence is a simple yet useful technique to increase the length. Second, repeating words does not change the semantics and is suitable as an extra positive example of the corresponding speech. Specifically, given sentence  $\mathbf{x}$ , each sub-word token  $x_{i}$  can be duplicated  $k$  more times, resulting in the duplicated sentence  $\mathbf{x}'$ , where  $k = 0,1,2,\ldots$  and  $k \sim \mathrm{Poisson}(1)$ . We regard  $\mathbf{x}'$  as the additional positive example for the speech s and the samples with the same operation in the same batch as the negative examples.
Cut-off strategy Recent studies on natural language understanding and generation have proved cut-off strategy to be successful (Shen et al., 2020; Yan et al., 2021). We analyze a similar idea to the cut-off approach for speech representation. We en
tirely erase a slice of the  $T \times d$  representation matrix along each dimension and set the erased terms to 0. Here, we present two variants: sequence cut-off, which erases some sequence dimension, and feature cut-off, which erases some feature dimension. Note that there is a difference between cut-off and dropout. Dropout randomly sets some elements to 0, while cut-off is a dimensional "block" dropout. Similarly, we treat the cut-off audio representation and the original transcribed sentence as positive pairs, and the rest sentences in the same batch as negative pairs.
ST datasets We conduct experiments on all the translation directions in MuST-C dataset<sup>1</sup> (Di Gangi et al., 2019): English (En) to German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro) and Russian (Ru). As one of the largest ST benchmarks, MuST-C contains more than 385 hours of TED talks for each direction.
MT datasets We also introduce external WMT datasets (Bojar et al., 2016) for En-De/Es/Fr/Ro/Ru and OPUS100 datasets (Zhang et al., 2020) for EnIt/Nl/Pt directions, as the expanded setup.
Table 8 (in Appendix. A) lists the statistics of all the datasets included.
Model Configurations The Wav2vec2.0 in the S-Enc is only pre-trained on Librispeech (Panayotov et al., 2015) speech without any downstream finetuning<sup>2</sup>. Two layers of CNNs after the Wav2vec2.0 are set to kernel size 5, stride size 2 and hidden size 512. The Transformer follows the base configuration, with 6 layers of encoder and decoder, hidden size  $d = 512$ , 8 attention heads, and 2048 FFN hidden states. We use pre-layer normalization for stable training. The model with the above configurations has a total of about 150M parameters.
Experiment Details We evaluate case-sensitive detokenized BLEU using sacreBLEU $^3$  (Post, 2018) on MuST-C tst-COMMON set. In the analysis, we also report the  $\mathrm{ChrF}++$  score $^4$  (Popovic, 2017)
and the learning-based BLEURT score  $^{5}$ . We use the raw 16-bit  $16\mathrm{kHz}$  mono-channel speech input. We jointly tokenize the bilingual text using SentencePiece (Kudo and Richardson, 2018), with a vocabulary size of  $10\mathrm{k}$ , which is the same as Ye et al. (2021)'s setup. For the training loss, we set contrastive temperature  $\tau = 0.02$  and weight of contrastive term  $\lambda = 1.5$  for German and Dutch, and  $\lambda = 1.0$  for the other languages.
Appendix B contains more detailed settings and explanations for the baseline models in Table 1. Appendix C shows the experiments on the choice of the hyper-parameters.
Comparison with end-to-end ST models Table 1 shows the main results. Since many existing works regard "leveraging external data" to be one of their model's features, their strong performances are largely predicated on the utilization of auxiliary data, especially large-scale MT data. For a relatively fair comparison, we investigate two cases: (1) without external MT data and (2) with external MT data. Without the external MT data, our method already gains an average improvement of 0.5 BLEU over the previous best models. Also when speech data is introduced for pre-training, our method works better than others (Self-training, W-Transf. and XSTNet). When extra MT data are introduced, our method also outperforms SOTA by an average of 0.6 BLEU. Among the benchmark models, with the same goal of closing two modality gaps, Chimera (Han et al., 2021) constructed an extra fixed-length shared semantic space. However, the shared memory with fixed size actually compromises the MT performance, while our contrastive learning approach is more straightforward and effective.
Comparison with cascaded ST systems We compare our method with several cascade baselines, where Ye et al. (2021) and Xu et al. (2021) provided two strong cascade systems trained using MuST-C and external ASR and MT data (LibriSpeech, WMT, and Opensubtitles). From Table 2, we find that as an end-to-end model, ConST can outperform these strong cascade models. In Appendix 7, we provide a case study to show such improvement.
1We use v1.0. https://ict.fbk.eu/must-c/2https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt3https://github.com/mjpost/sacrebleu,   
BLEU Signature: nrefs:1 | bs:1000 | seed:12345 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.0.0 4ChrF2++ Signature: nrefs:1 | bs:1000 | seed:12345 | case:mixed | eff:yes | nc:6 | nw:2 | space:no | version:2.0.0
$^{5}$ https://github.com/google-research/bleurt (Sellam et al., 2020). As recommended, the checkpoint we use is BLEURT-20.
Models	External Data	BLEU
Speech	Text	ASR	MT	De	Es	Fr	It	NI	Pt	Ro	Ru
w/o external MT data
Fairseq ST (Wang et al., 2020a)	-	-	-	-	22.7	27.2	32.9	22.7	27.3	28.1	21.9	15.3
NeurST (Zhao et al., 2021a)	-	-	-	-	22.8	27.4	33.3	22.9	27.2	28.7	22.2	15.1
Espnet ST (Inaguma et al., 2020)	-	-	-	-	22.9	28.0	32.8	23.8	27.4	28.0	21.9	15.6
Dual Decoder (Le et al., 2020)	-	-	-	-	23.6	28.1	33.5	24.2	27.6	30.0	22.9	15.2
W-Transf. (Ye et al., 2021)	✓	-	-	-	23.6	28.4	34.6	24.0	29.0	29.6	22.4	14.4
Speechformer (Papi et al., 2021)	-	-	-	-	23.6	28.5	-	-	27.7	-	-	-
LightweightAdaptor (Le et al., 2021)	-	-	-	-	24.7	28.7	35.0	25.0	28.8	31.1	23.8	16.4
Self-training (Pino et al., 2020)	✓	-	✓	-	25.2	-	34.5	-	-	-	-	-
SATE (Xu et al., 2021)	-	-	-	-	25.2	-	-	-	-	-	-	-
BiKD (Inaguma et al., 2021)	-	-	-	-	25.3	-	35.3	-	-	-	-	-
Mutual-learning (Zhao et al., 2021b)	-	-	-	-	-	28.7	36.3	-	-	-	-	-
XSTNet (Ye et al., 2021)	✓	-	-	-	25.5	29.6	36.0	25.5	30.0	31.3	25.1	16.9
ConST	✓	-	-	-	25.7	30.4	36.8	26.3	30.6	32.0	24.8	17.3
w/ external MT data
MTL (Tang et al., 2021b)	-	-	-	✓	23.9	28.6	33.1	-	-	-	-	-
FAT-ST (Big) (Zheng et al., 2021)	✓	✓	✓	✓	25.5	30.8	-	-	30.1	-	-	-
JT-S-MT (Tang et al., 2021a)	-	-	-	✓	26.8	31.0	37.4	-	-	-	-	-
Chimera (Han et al., 2021)	✓	-	-	✓	\( {27.1}^{ \dagger } \)	30.6	35.6	25.0	29.2	30.2	24.0	17.4
XSTNet (Ye et al., 2021)	✓	-	-	✓	27.1	30.8	38.0	26.4	31.2	32.4	25.7	18.5
SATE (Xu et al., 2021)	-	-	✓	✓	\( {28.1}^{ \dagger } \)	-	-	-	-	-	-	-
STEMM (Fang et al., 2022)	✓	-	-	✓	28.7	31.0	37.4	25.8	30.5	31.7	24.5	17.8
TaskAware (Indurthi et al., 2021)	-	-	✓	✓	28.9	-	-	-	-	-	-	-
STPT (Tang et al., 2022)	✓	✓	✓	✓	-	33.1	39.7	-	-	-	-	-
ConST	✓	-	-	✓	28.3	32.0	38.3	27.2	31.7	33.1	25.6	18.9
Table 1: Case-sensitive detokenized BLEU scores on MuST-C tst-COMMON set. "Speech" denotes unlabeled speech data. "Text" means unlabeled text data, e.g. Europarl V7 (Koehn et al., 2005), CC25 (Liu et al., 2020a). † use external 40M OpenSubtitles (Lison and Tiedemann, 2016) MT data. Other models only use WMT data.
Models	En-De	En-Fr	En-Ru
Cascaded
Espnet(Inaguma et al., 2020)	23.6	33.8	16.4
(Ye et al., 2021)	25.2	34.9	17.0
(Xu et al., 2021)	28.1	-	-
End-to-end
ConST	28.3	38.3	18.9
5.1 Is contrastive loss effective?
With the same model architecture and the same pretraining + fine-tuning procedure, the main difference between ConST and XSTNet (Ye et al., 2021) is whether we use the contrastive loss term during the fine-tuning or not. Comparing the BLEU results of w/o and w/ external MT data situations in Table 1, we find that ConST further improves 0.5 and 0.6 BLEU scores in terms of eight translation directions on average, which proves the effectiveness of the cross-modal contrastive learning. By gradually removing each losses in Eq.(1), Table 3 shows the improvements bringing by the multi-task learning and the contrastive learning. For En-De translation direction, contrastive learning can bring an average
Table 2: ConST versus the cascaded ST systems on MuST-C En-De/Fr/Ru test sets. Ye et al. (2021) and Xu et al. (2021) are two strong cascaded models.
Config.	External MT
without	with
ConST	25.7	28.3
-ASR - MT	24.6	27.0
-ASR - MT - CTR	23.6	26.3
Table 3: BLEU scores on MuST-C En-Detst-COMMON set by removing individual losses. We test the results under settings with and without the introduction of external MT data.