SlowGuess/NAACL_2022_OCR huggingface dataset:
improvement of 0.9 BLEU over the baseline models by only optimizing  $\mathcal{L}_{\mathrm{ST}}$  (corresponding to the last row of the Table 3), and multi-task learning can lead to a further improvement of about 1.2 BLEU on top of that.
5.2 Which layer to contrast on?
An intriguing question is which representations should be considered in the contrastive loss function. In the method part (Section 3.2), we use averaged audio representation  $u$  for speech s (Eq.(2)) and averaged lexical embedding  $v$  for the transcript x (Eq.(3)), denoted as low-level repr.. Whereas inspired by a recent study in multilingual MT (Pan et al., 2021), we also provide an alternative contrastive loss as a comparison, whose speech and text features are average-pooled semantic repre
Representations	BLEU	ChrF++	BLEURT
low-level repr.	28.3*	53.2*	64.5
high-level repr.	27.5†	52.6†	63.6
w/o contrative loss	27.1	52.1	62.4
Table 4: BLEU,  $\mathrm{ChrF + + }$  and BLEURT  $(\%)$  on En-De test set. Different representations are tested. \*: ConST is significantly better than the other two baselines  $(p < 0.01)$ .  $\dagger$  : the model is significantly better the baseline model without contrastive loss  $(p < 0.05)$ .
sentations derived from the Transformer encoder, denoted as high-level repr.
Table 4 shows that contrastive learning using the low-level representations (Line 1) is better than using the high-level ones (Line 2). On the other hand, although the performance of Line 2 is relatively inferior, it still outperforms the multi-task model without the contrastive loss (Line 3). The detailed analysis of possible explanations will be shown in Section 6.2.
5.3 Is contrastive loss better than other losses?
Our goal for introducing the contrastive loss term (denoted as CTR Loss) is to close the distance between speech and text representations. Whereas, there are other options to achieve this goal, such as L2 loss and CTC loss.
- L2 Loss: Without introducing any negative samples, L2 loss directly reduces the Euclidean distance between the representations of two modalities by minimizing  $\mathcal{L} = \| u - v\|^2$ . L2 loss can be viewed as an implementation based on the idea of knowledge distillation (Heo et al., 2019; Dong et al., 2021b).
- CTC Loss: The connectionist temporal classification (CTC) loss (Graves et al., 2006) is commonly used in speech-related tasks (Xu et al., 2021; Dong et al., 2021b). Unlike contrastive loss that cares about the representation, CTC loss connects the two modalities by establishing speech-text alignment and maximizing  $p(\mathbf{x}|\mathbf{a}) = \sum_{\pi \in \Pi_{\mathbf{s},\mathbf{a}}}\prod_{t = 1}^{T}p_t(\pi_t|\mathbf{a})$ , where  $\Pi_{\mathbf{s},\mathbf{a}}$  is the set of all valid alignments.
Compared to the other two ways of bridging the modality gap, L2 and CTC loss, is the contrastive loss term better? The answer is yes according to the results in Table 5. Our explanation is that information on the negative samples benefits the contrastive loss, bringing the distance between the speech and its corresponding transcription closer while pushing the distance to the irrelevant text farther.
Extra Loss	BLEU	ChrF++	BLEURT
CTR Loss	28.3*	53.2	64.5
CTC Loss	27.6†	53.0†	64.1
L2 Loss	27.3	52.4	63.0
-	27.1	52.1	62.4
Table 5: BLEU,  $\mathrm{ChrF + + }$  and TER  $(\%)$  on En-De test set under different loss terms other than the basic multitask NLL loss. \*: ConST is significantly  $(p <   0.01)$  better than the other three alternatives.  $\dagger$  : the improvement from CTC loss over the baseline without extra loss is significant  $(p <   0.01)$
Figure 4: The heat map visualization of the BLEU scores on En-De test set, with 15 combinations of the original contrastive loss (Original) and hard examples mining methods - word repetition (Rep), span-masked augmentation (SMA), sequence cut-off (SCut) and feature cut-off (FCut). * and ** mean the improvements over the XSTNet baseline without contrastive loss are statistically significant ( $*:p < 0.05$ ,  $**:p < 0.01$ ).
5.4 Analysis on the hard example mining strategies
In Section 3.3, we proposed four methods to mine the hard examples for contrastive learning, namely span-masked augmentation (SMA), word repetition (Rep), sequence cut-off (SCut), and feature cut-off (FCut). In this section, we study how effective these methods are, and to do so, we consider the BLEU performances of their 15 combinations (Figure 4). Note that "Original" means the original contrastive loss in Eq.(4) without any additional hard examples mining operation, and the diagonal in the heat map represents only one strategy used. For an easy and fair comparison, we set the weight of the contrastive term to 1.0 uniformly. We have the following observations.
All the hard examples mining methods are effective. All the BLEU scores in Figure 4 exceed the strong multi-task model trained without contrastive learning (27.1). Among all the strategies, the combination of the original and SCut reaches
(a) w/o contrastive loss
Figure 5: Bivariate KDE contour plot of the representation of speech and transcript in source language English. T-SNE is used to reduce into 2D. The blue lines are the audio representations and the red dashed lines stand for text. (a) for the vanilla multi-task framework without any extra supervision. (b) for our proposed ConST model. Sentences are from En-De test set.
(b) ConST
the best result (28.3), and is better than the model without any expanded operations  $(p < 0.01)$ . Generally, to find the best model, we suggest adopting multiple strategies and choosing the best checkpoint on the dev-set.
The combinations of the hard examples mining methods and the "original" have relatively better performances. We argue that we need the original positive and negative examples to give more accurate representations (without any dropout) for contrastive learning. On the contrary, without the help of "original" loss, the performance with both sequence cut-off and feature cut-off is the worst in Figure 4, probably because too much information is lost by superimposing the two.
6 Why does cross-modal contrastive learning work? — Analysis on the Modality Gap
As mentioned earlier, the existing multi-task training models cannot address the speech-text modality gap. Does ConST reduce the representation gap between speech and text?
6.1 Visualization of Representation
Does the speech-text modality gap exist without explicitly bridging the two? Speech-text modality gap means the discrepancy between the audio representations and transcription sentence embeddings. To visualize it, we plot the bivariate kernel density estimation (Parzen, 1962) (KDE) contour of their dim-reduced features, where T-SNE (Van der Maaten and Hinton, 2008) is used to reduce the dimension into two (Figure 5). Ideally, if the representations of speech and its corresponding tran
script are similar, their KDEs will be similar, and thus the contour lines will overlap as much as possible. However, Figure 5(a) is the KDE contour of the multi-task framework without any explicit modeling to bring two modalities together (Ye et al., 2021). It shows that the representations are so dissimilar that they are organically divided into two clusters, i.e. speech-text modality gap exists.
Does ConST reduce the modality gap? As shown in Figure 5(b), compared to the baseline model without contrastive learning, ConST with cross-modal contrastive learning is able to bring representations of different modalities much closer. This means that the audio representation contains more linguistic information similar to that of the textual transcription, which is more advantageous for the downstream ST generation through the shared Transformer encoder and decoder.
6.2 Cross-modal Retrieval
How good is the cross-modal representation space learned from ConST? To answer this question, we conduct a retrieval experiment, i.e. finding the nearest (smallest cosine similarity) transcript based on the speech representation. We compare ConST model with the baseline without cross-modal contrastive learning and report the top-1 retrieval accuracy using (1) the low-level representations and (2) the high-level semantic representations, in Table 7.
When retrieving the text using low-level representations, our method gains a substantial  $79\%$  increase over the baseline. In addition, we find that without explicit contrastive modeling, the baseline can achieve retrieval accuracy of more than  $94\%$  according to the semantic representations outputted from the Transformer encoder. We believe that such high accuracy is automatically learned from the triple-supervised data itself under the multi-task learning framework. With such a degree of cross-modal alignment, if we construct the contrastive loss with semantic representations, its gain to the ST performance turns out to be limited, which exactly corroborates the findings in Section 5.2 - low-level representations are preferred in the cross-modal contrastive learning.
7 Case Analysis
In this section, we use several cases that ConST generates. We compare our model with the cascaded model and the previous end-to-end model,
Models
CASE 1
Ref.	src: Lights, sounds, solar panels, motors — everything should be accessible.
tgt: Lichter, Töne, Solarelemente, Motoren — alles sollent verfügbar sein.
Cascaded	src: Lights sounds solar panels motors everything should be accessible.
tgt: Licht klingt Solarpaneele, Motoren; alles sollent zugänglich sein.
XSTNet	tgt: Licht, Geräusche, Solarkollektoren, Motoren — alles sollent zugänglich sein.
ConST	tgt: Licht, Geräusche, Solarpanele, Motoren, alles sollent zugänglich sein.
CASE 2
Ref.	src: Eight years ago when I was at the Media Lab, I started exploring this idea of how to put the power of engineers in the hands of artists and designers.
tgt: Vor acht Jahren war ich am Media Lab und ich begann diese Idee zu erforschen, wie man die Macht der Ingenieure in die Hand von Künstlern und Designern legen können.
Cascaded	src: Eight years ago when I was at the Media Lab, I started exploring this idea of how to put the power of engineers in the hands of artists and designers.
tgt: Vor 8 Jahren, als ich im Media Lab war, begann ich, diese Idee zu erforschen, wie man die Macht der Ingenieure in die Höhe von Künstlern und Designern legte.
XSTNet	tgt: Vor acht Jahren, als ich im Media Lab war, begann ich zu erforschen, wie man die Kraft der Ingenieure in die Höhe von Künstlern und Designern legt.
ConST	tgt: Vor acht Jahren, als ich im Media Lab war, begann ich, diese Idee zu erforschen, wie man die Macht von Ingenieuren in die Höhe von Künstlern und Designern legt.
Table 6: En-De test cases that generated from the cascaded model, XSTNet (both provided by Ye et al. (2021)) and our ConST model. The red underlined text indicates grammatically incorrect or inaccurate translations.
Representations	CTR loss	Acc.
low-level repr.	×	9.4
✓	88.6
high-level repr.	×	94.7
✓	95.0
Table 7: Cross-modal top-1 retrieval accuracy  $(\%)$  on En-De test set. Two different representations are used, based on which, ConST achieves huge accuracy improvements.
XSTNet (Ye et al., 2021).
For this first case, the cascaded system fails to give a right translation due to the mis-punctuation issue (klingt is a verb), while the end-to-end model, XSTNet and ConST translate correctly. For the second case, the previous end-to-end XSTNet model cannot accurately translate the phrase "started exploring this idea of", which performs worse than the cascaded one. Whereas ConST successfully conveys the meaning of "this idea", and translates more accurately than XSTNet. We believe this improvement comes from the cross-modal contrastive learning.
In this paper, we propose ConST, a simple yet effective contrastive learning framework bridging the speech-text representation gap and facilitating the
ST with limited data. We also provide feasible hard example mining methods to learn robust representations. The results on the MuST-C ST dataset prove the effectiveness of the method.
This work improves the performance of ST tasks on public datasets by learning speech representations that are more similar to text representations, but the model is far from being achieved for industrial-grade implementations. In real scenarios, for example, the original voice is noisier and the distribution of speech lengths is more complex than in the public dataset, which cannot be handled by an end-to-end model alone. The shortcoming of this model is that it still needs a certain amount of labeled data for training, especially <speech,transcription> to learn better speech representation, and for the more than 7,000 languages and dialects in the world, most of them do not have corresponding translations or even transcriptions, our method does not work in untranscribed scenarios. In this paper, we focus on the improvement brought by the better speech representation on the ST task, and obtained good results with hundreds of hours of speech data. We hope that our work achieves better results using more data (e.g. raw speech, raw text, ASR, MT data) in the future.
Ashkan Alinejad and Anoop Sarkar. 2020. Effectively pretraining a speech translation decoder with machine translation data. In Proc. of EMNLP, pages 8014-8020.
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondrej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, et al. 2020. Findings of the iwslt 2020 evaluation campaign. In Proc. of IWSLT, pages 1-34.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proc. of NeurIPS.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pretraining on high-resource speech recognition improves low-resource speech-to-text translation. In Proc. of NAACL-HLT, pages 58-68.
Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus direct speech translation: Do the differences still make a difference? In Proc. of ACL.
Alexandre Berard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In Proc. of ICASSP, pages 6224-6228.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In NIPS workshop on End-to-end Learning for Speech and Audio Processing.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Néveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198.
Shuyang Cao and Lu Wang. 2021. Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proc. of ACL, pages 6633-6649.
Junkun Chen, Mingbo Ma, Renjie Zheng, and Liang Huang. 2021. Specrec: An alternative solution for improving end-to-end speech-to-text translation via spectrogram reconstruction. Proc. of InterSpeech, pages 2232-2236.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020a. A simple framework for contrastive learning of visual representations. In Proc. of ICML, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. Uniter: Universal image-text representation learning. In Proc. of ECCV, pages 104-120. Springer.
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In Proc. of CVPR, volume 1, pages 539-546. IEEE.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proc. of NAACL-HLT, pages 2012-2017.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Proc. of NeurIPS, pages 13063-13075.
Linhao Dong, Shuang Xu, and Bo Xu. 2018. Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In Proc. of ICASSP, pages 5884-5888.
Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021a. Consecutive decoding for speech-to-text translation. In Proc. of AAAI.
Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021b. Listen, understand and translate: Triple supervision decouples end-to-end speech-to-text translation. In Proc. of AAAI, volume 35, pages 12749-12759.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. Stemm: Self-learning with speech-text manifold mixup for speech translation. In Proc. of ACL.
Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Lei Li. 2022. Contextual representation learning beyond masked language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2701-2714, Dublin, Ireland. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proc. of EMNLP.
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. of ICML, volume 148, pages 369-376.
Klaus Greff, Rupesh K Srivastava, Jan Koutnik, Bas R Steunbrink, and Jürgen Schmidhuber. 2016. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):2222-2232.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Proc. of NeurIPS, 33:21271-21284.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings.
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. In Proc. of ACL - Findings.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. 2019. A comprehensive overhaul of feature distillation. In Proc. of the ICCV, pages 1921-1930.
Yushi Hu, Shane Settle, and Karen Livescu. 2020. Multilingual jointly trained acoustic and written word embeddings. In Proc. of INTERSPEECH.
Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021. Source and target bidirectional knowledge distillation for end-to-end speech translation. In Proc. of NAACL, pages 1872-1881.
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. ESPnet-ST: All-in-one speech translation toolkit. In Proc. of ACL, pages 302-311.
Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. Data efficient direct speech-to-text translation with modality agnostic meta-learning. In Proc. of ICASSP. IEEE.
Sathish Indurthi, Mohd Abbas Zaidi, Nikhil Kumar Lakumarapu, Beomseok Lee, Hyojung Han, Seokchan Ahn, Sangha Kim, Chanwoo Kim, and Inchul Hwang. 2021. Task aware multi-task learning for speech to text tasks. In Proc. of ICASSP, pages 7723-7727. IEEE.
Herman Kamper, Yevgen Matushevych, and Sharon Goldwater. 2020. Multilingual acoustic word embedding models for processing zero-resource languages. In Proc. of ICASSP, pages 6414-6418. IEEE.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2017. Structured-based curriculum learning for end-to-end english-japanese speech translation. In Proc. of INTERSPEECH, pages 2630-2634.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Proc. of NeurIPS, 33.
Philipp Koehn et al. 2005. Europarl: A parallel corpus for statistical machine translation. In  $MT$  summit, volume 5, pages 79-86. CiteSeer.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proc. of EMNLP, pages 66-71.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In Proc. of NeurIPS.
Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In Proc. of COLING, pages 3520-3533.
Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2021. Lightweight adapter tuning for multilingual speech translation. In Proc. of ACL.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. In Proc. of ACL.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proc. of LREC, pages 923-929.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. TACL, 8:726-742.
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020b. Bridging the modality gap for speech-to-text translation. arXiv preprint arXiv:2010.14920.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. Proc. of NeurIPS, 32.