SlowGuess/NAACL_2022_OCR huggingface dataset:
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL - Demonstrations, pages 48-53.
Shruti Palaskar, Vikas Raunak, and Florian Metze. 2019. Learned in speech recognition: Contextual acoustic word embeddings. In Proc. of ICASSP, pages 6530-6534. IEEE.
Xiao Pan, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proc. of ACL.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In Proc. of ICASSP, pages 5206-5210.
Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2021. Speechformer: Reducing information loss in direct speech translation. In Proc. of EMNLP.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quoc V. Le. 2019. Specaugment: A simple augmentation method for automatic speech recognition. In Proc. of INTERSPEECH.
Emanuel Parzen. 1962. On estimation of a probability density function and mode. The annals of mathematical statistics, 33(3):1065-1076.
Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020. Self-training for end-to-end speech translation. In Proc. of INTER-SPEECH, pages 1476-1480.
Maja Popovic. 2017. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612-618.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191.
Tomasz Potapczyk and Paweł Przybysz. 2020. Srpol's system for the iwslt 2020 end-to-end speech translation task. In Proc. of IWSLT, pages 89-94.
Mirco Ravanelli and Yoshua Bengio. 2018. Learning speaker representations with mutual information. arXiv preprint arXiv:1812.00271.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of IN-TERSPEECH.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proc. of CVPR, pages 815-823.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proc. of ACL.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009.13818.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Proc. of NeurIPS, volume 29, pages 1857-1865. Curran Associates, Inc.
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, et al. 2022. Unified speech-text pre-training for speech translation and recognition. In Proc. of ACL.
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving speech translation by understanding and learning from the auxiliary text translation task. In Proc. of ACL.
Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021b. A general multi-task learning framework to leverage text data for speech to text tasks. In Proc. of ICASSP, pages 6209-6213. IEEE.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. 2018. Parallel wavenet: Fast high-fidelity speech synthesis. In Proc. of ICML, volume 80 of Proceedings of Machine Learning Research, pages 3915-3923.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS, pages 5998-6008.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq s2t: Fast speech-to-text modeling with fairseq. In Proc. of AACL, pages 33-39.
Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, and Alexis Conneau. 2021a. Large-scale self-and semi-supervised learning for speech translation. In Proc. of INTERSPEECH.
Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020b. Curriculum pre-training for end-to-end speech translation. In Proc. of ACL, pages 3728-3738.
Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu, and Lei Li. 2021b. Contrastive aligned joint learning for multilingual summarization. In Proc. of ACL - Findings.
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In Proc. of INTERSPEECH, pages 2625-2629.
Anne Wu, Changhan Wang, Juan Pino, and Jiatao Gu. 2020. Self-supervised representations improve end-to-end speech translation. In Proc. of INTER-SPEECH.
Hao Wu, Jiayuan Mao, Yufeng Zhang, Weiwei Sun, Yuning Jiang, Lei Li, and Wei-Ying Ma. 2019. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In Proc. of CVPR.
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021. Esimcse: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. arXiv preprint arXiv:2109.04380.
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Qi Ju, Tong Xiao, Jingbo Zhu, et al. 2021. Stacked acoustic-and-textual encoding: Integrating the pretrained models into speech translation encoders. In Proc. of ACL.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consent: A contrastive framework for self-supervised sentence representation transfer. In Proc. of ACL.
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to-end speech translation via cross-modal progressive training. In Proc. of INTERSPEECH.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Senrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proc. of ACL, pages 1628-1639.
Chengqi Zhao, Mingxuan Wang, Qianqian Dong, Rong Ye, and Lei Li. 2021a. NeurST: Neural speech translation toolkit. In Proc. of ACL - System Demonstrations.
Jiawei Zhao, Wei Luo, Boxing Chen, and Andrew Gilman. 2021b. Mutual-learning improves end-to-end speech translation. In Proc. of the EMNLP, pages 3989-3994.
Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. 2021. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. In Proc. of ICML.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proc. of AAAI, volume 34, pages 13041-13049.
A Statistics of all datasets
En→	ST (MuST-C)	MT
hours	#sents	name	#sents
De	408	234K	WMT16	4.6M
Es	504	270K	WMT13	15.2M
Fr	492	292K	WMT14	40.8M
It	465	258K	OPUS100	1.0M
Nl	442	253K	OPUS100	1.0M
Pt	385	211K	OPUS100	1.0M
Ro	432	240K	WMT16	0.6M
Ru	489	270K	WMT16	2.5M
Table 8: Statistics of all datasets
Training and Implementation Details We use Adam optimizer  $(\beta_{1} = 0.9, \beta_{2} = 0.98)$  with learning rate  $= 1e^{-4}$  and warmup 25k steps during the ST training. We also implement the expanded setting with the introduction of external WMT to train the Transformer module. In the pre-training stage, we set the learning rate  $= 7e^{-4}$  and warmup 4000 steps. For robust training, we set label smoothing to 0.1, and dropout rate to 0.1. The hyper-parameters for different data augmentation methods are as follows: for masked audio span strategy, we set masking probability  $p = 0.25$  and masking span length  $M = 3600$  frames; for both sequence and feature cut-off, we set the cut-off dropout rate as 0.1. We save the checkpoint with the best BLEU on dev-set and average the last 10 checkpoints. For decoding, we use a beam size of 10 and length penalty 0.7 for German, 1.0 for French, and 0.4 for Russian. We train the models in 8 Nvidia Tesla V100 GPUs for each experiment. We use Fairseq (Ott et al., 2019) as the code-base for our implementation.
Baseline Models In Table 1, we compared our method with end-to-end baseline models whose audio inputs are 80-channel log Mel-filter bank, including: FairseqST (Wang et al., 2020a), NeurST (Zhao et al., 2021a), Espnet ST (Inaguma et al., 2020), Dual-decoder Transformer (Le et al., 2020), SATE (Xu et al., 2021), Speechformer (Papi et al., 2021), self training (Pino et al., 2020) and mutual learning (Zhao et al., 2021b) method, STAST (Liu et al., 2020b), bi-KD (Inaguma et al., 2021), MLT method (Tang et al., 2021b), Lightweight Adaptor (Le et al., 2021), JT-S-MT (Tang et al., 2021a), FAT-ST (Zheng
et al., 2021), TaskAware (Indurthi et al., 2021), and STPT (Tang et al., 2022). We also compare our method to baseline models that have pretrained Wav2vec2.0 as a module, including:
- W-Transf. (Ye et al., 2021): the model has the same structure as ours, but is only trained on <speech, translation> parallel data.
- Chimera-ST (Han et al., 2021): the model that builds a shared semantic memory for both audio and text modalities.
- XSTNet (Ye et al., 2021): the model has the same structure as ours, and adopted a multi-task fine-tuning strategy.
- STEMM (Fang et al., 2022): the model that bridges the modality representation gap by minimizing the Jensen-Shannon divergence between the original speech representation and the manifold mix-up representation.
C The Choice for Hyper-parameters
Influence of Temperature In the contrastive loss, the temperature hyper-parameter is provided to control the smoothness of the distribution normalized by softmax operation. A high temperature helps to smooth the distribution, making it more difficult for the model to distinguish between positive and negative samples (corresponding to correct transcriptions and other transcriptions in this work), while the low temperature behaves just the opposite. We choose several temperature hyper-parameters ranging from 0.01 to 0.5, and Figure 6 shows their BLEUs on the test and dev sets. We find that (1) the choice of the temperature does not drastically affect the final BLEU score, and (2) we recommend that the temperature  $\tau$  be set between 0.02 and 0.05 to ensure a relatively good ST performance. In the experiment, we use  $\tau = 0.02$ .
Figure 6: En-De BLEU scores on tst-COMMON and Dev set. the x-axis is the choices of different temperature  $\tau$  in Eq.(4) varying from 0.01 to 0.5.
Influence of Contrastive Loss Weight The total loss we optimize, Eq.(1), is a linear combination of the multi-task cross-entropy losses  $\mathcal{L}_{\mathrm{MLT}}$  and the contrastive term  $\mathcal{L}_{\mathrm{CTR}}$ . To investigate how much the contrastive terms affect BLEU, we fix its temperature  $\tau = 0.02$ , adjust the values of its loss weight  $\lambda$  from 0.1 to 2.0, performed three experiments for each value, and test the average BLEU on En-De tst-COMMON set. Figure 7 depicts the performances. First, all objective functions containing  $\mathcal{L}_{\mathrm{CTR}}$ , even if their weights  $\lambda$  take different values, are apparently better than the baseline model with  $\mathcal{L}_{\mathrm{MLT}}$  only  $\mathcal{L}_{\mathrm{CTR}}$ . Then, the best BLEU score is achieved at loss weight  $\lambda = 1.5$ , corresponding to the results in Table 1. And when analyzing the effect of data augmentation strategies (Section 5.4), since we need to consider the combination between them, which is more complicated. Therefore, we set the loss weight to 1.0 uniformly for simplicity. In general, we recommend that the weight hyperparameter takes a value between 0.8 and 1.5.
Figure 7: En-De BLEU scores on tst-COMMON and Dev sets. The x-axis is the weight of the contrastive loss term  $\lambda$  in Eq.(1). Experiments are performed under the fix temperature hyper-parameter  $\tau = 0.02$ .
D Data Scale for Fine-tuning
The experiments in the main paper show that our model can perform well by introducing external MT data pre-training. Here, we simulate the scenario with plenty of MT and speech data and limited ST triple-labeled data, and does ConST have the ability of low-resource learning? In the experiment, we reduce the labeled ST data to 1, 10, and 100 hours, corresponding to sentence counts of about 500, 5k, and 50k sentences. For a fair comparison, we use the same MT pre-trained Transformer module as in the main paper. We find the contrastive loss particularly helpful when the amount of speech data is extremely small, like only
1 hour of speech. Second, the multi-task training strategy is also very effective in improving the robustness of the model performance. We also find that by using easily accessible MT and speech pretraining, our model could reach the previous baseline results without pre-training using only  $1/4$  of the original data, i.e. 100 hours of labeled ST data.
Figure 8: En-De BLEU scores on tst-COMMON sets. The horizontal axis is the amount of ST data (in hours of speech).