WACO: Word-Aligned Contrastive Learning for Speech Translation
Year: 2023

Abstract:
End-to-end Speech Translation (E2E ST) aims to directly translate source speech into target text. Existing ST methods perform poorly when only extremely small speech-text data are available for training. We observe that an ST model’s performance closely correlates with its embedding similarity between speech and source transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a simple and effective method for extremely low-resource speech-to-text translation. Our key idea is bridging word-level representations for both speech and text modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark, and on a low-resource direction Maltese-English from IWSLT 2023. Our experiments demonstrate that WACO outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data. Code is available at https://github.com/owaski/WACO.

heading: 1 Introduction
text: End-to-end speech translation (E2E ST) directly translates speech in a source language to text in a target language, without intermediate pipelines. E2E ST has witnessed significant progress in translation quality (Duong et al., 2016; Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019; Gaido et al., 2020; Dong et al., 2021a; Ye et al., 2022a) . However, existing E2E ST methods still perform poorly when only a limited amount of parallel ST data are available. How can we build a highly performant ST model with extremely low resource, e.g. only 1 hour of parallel data (approximately a few hundred utterances)?
We analyze the encoder representations from a directly learned ST model1 and find that its average embeddings from speech and transcript are similar at the sequence level but still not aligned
1We use XSTNet (Ye et al., 2021) as an example.
well on the word level (Figure 1a). An ideal ST model should encode a speech utterance closely aligned with the representations of corresponding words in its transcript text (Figure 1b). Prior methods attempt to use additional parallel data from machine translation (MT) and automatic speech recognition (ASR) to align speech and text representations. However, most do not explicitly reduce the word level representation gap among speech and text. We hypothesize that such misalignment at the word level between corresponding speech and transcript text is a critical cause of the inferior ST performance.
We further observe such a misalignment phenomenon is severe when ST data is extremely low. We did a pilot study by training direct ST models using different sizes of ST data (1/5/10/388 hours) . As shown in Figure 2, we find that the translation performance highly correlates with the word-level embedding similarity between speech and transcript text. With fewer parallel ST data, the cross-modal similarity drops simultaneously with the BLEU score and almost reaches 0 given 1-hour ST training data. This observation suggests that the model can map both modalities into a partially aligned semantic space given sufficiently large ST
3891
data but fails when ST data is extremely small. In this paper, we propose WACO, a word-level contrastive learning method for extremely lowresource speech-to-text translation. Its key idea is to promote representation similarity among corresponding words in speech and text and to push non-corresponding representations further apart in embedding spaces. Furthermore, it can use additional large ASR data to improve word-level representation learning. We conduct experiments on the MuST-C benchmark with varying data sizes and a real-world low-resource direction Maltese-English (Mt-En) from IWSLT 2023 low-resource track. We also analyze the word-level representation similarity from the learned models. Our contributions are: • We propose a new method for speech translation,
WACO, which explicitly aligns speech-transcript representations of corresponding words.
• We verify the effectiveness of WACO on MuSTC and IWSLT Mt-En. WACO outperforms previous strong methods by 2.0-9.8 BLEU points with only 1 and 10 hours of parallel ST data.
• We further demonstrate that that WACO indeed learns a better-aligned representation of speech and text at the word level which correlates well with its ST performance.

heading: 2 Related Work
text: End-to-end ST Due to error propagation and high latency in cascaded ST systems, Bérard et al. (2016); Duong et al. (2016) first proposed to translate source speech into target text directly without generating the intermediate transcript. The major difficulty in training end-to-end ST systems is the lack of direct ST data. Though many ST
datasets (Wang et al., 2021; Cattoni et al., 2021) were proposed in recent years, the amount of ST data is still much less than that of MT and ASR. To overcome the data scarcity problem, methods including data augmentation (Park et al., 2019; McCarthy et al., 2020; Lam et al., 2022; Mi et al., 2022), self-training (Pino et al., 2020), multitasking (Le et al., 2020; Tang et al., 2021b,a; Ye et al., 2021; Zhang et al., 2022a) and pre-training (Berard et al., 2018; Bansal et al., 2019; Wu et al., 2020; Wang et al., 2020; Alinejad and Sarkar, 2020; Dong et al., 2021a; Zheng et al., 2021; Bapna et al., 2021; Ao et al., 2022; Tang et al., 2022) have been proposed. WACO is a novel approach that can be applied in existing multi-tasking and pre-training frameworks to improve ST performance. Cross-modal representation learning Researchers realized recently that the misalignment between speech and text representation hinders the knowledge transfer from external data (Liu et al., 2020; Dong et al., 2021b; Xu et al., 2021; Han et al., 2021; Ye et al., 2022b; Du et al., 2022; Wang et al., 2022). Liu et al. (2020) shrank the speech representation to match the length of text representation and also closed the representational gap by minimizing their L2 distance. Xu et al. (2021) mapped speech representation to text representation through both the Connectionist Temporal Classification (CTC) (Graves et al., 2006) distribution and a mapping layer. Dong et al. (2021b) proposed a cross attention layer to force the speech-text alignment. Han et al. (2021) developed a novel architecture enabling fixed-length shared semantic space for both modalities. Saxon et al. (2021) proposed a hierarchical speech understanding system leveraging both ASR and text understanding data. Ye et al. (2022b) employed sentence-level contrastive loss to reduce the modality gap and ach‘ieved stateof-the-art results on MuST-C. Our method, however, works on word-level instead of sentence-level and empirically provides both better performance and higher data efficiency. Fang et al. (2022) also proposes to align the word-level representations between speech and text, but their method heavily relies on target translation while our method only requires ASR data for modality gap reduction.

heading: 3 Proposed Method: WACO
text: In this section, we describe our approach to develop effective speech translation models with extremely low-resource parallel data.

heading: 3.1 Problem Formulation
text: A typical ST corpus DST contains speech s and its transcript x in a source language and translation text y in a target language. Equivalently, DST = {(s, x, y)} and ASR corpus can be similarly defined as DASR = {(s, x)}.
Given DST and DASR as training sets, the E2E ST model needs to translate speech s into translation y accurately without generating transcript x in the intermediate steps. There are two settings: • Regular ST: Training includes large ST triplet
data. In this paper, we regard the entire MuST-C training set as the regular setting (|DST| ≈400 hours). • Low-Resource ST: Training has very limited ST data but plenty of ASR data, i.e., |DST| |DASR|. In this paper, ST data below 10 hours is regarded as low-resource ST. Many African and native American languages belong to this setting. In this paper, we focus on low-resource ST. In addition, we also include external MT data for both settings. The size of MT dataset is much larger than ST dataset.

heading: 3.2 Model Architecture
text: Figure 3 illustrates WACO model architecture. WACO consists of 3 modules: a speech encoder, a text embedding layer and a joint Transformer encoder-decoder. During training, it inputs either speech or text sequence and outputs text sequence. At inference time, the input is only speech. Speech Encoder extracts contextualized acoustic
embeddings from the raw waveform. It consists of wav2vec 2.0 (Baevski et al., 2020) and 2 downsampling layers. Wav2vec 2.0 is one of the state-of-theart self-supervised models pre-trained on unlabeled English speech corpus. It produces contextualized speech embeddings. It consists of 7 convolutional layers as the audio feature extractor and a 12-layer Transformer as the contextual encoder. On top of wav2vec 2.0, we further add 2 convolutional layers with stride of 2 to downsample the embedding sequence. It reduces the length and alleviates the length discrepancy between speech and text embeddings. Text Embedding is a lookup table that maps text tokens into embeddings. Transformer Encoder-Decoder accepts outputs from either both the speech encoder or the text embedding layer. The configuration is the same as the 6-layer vanilla Transformer (Vaswani et al., 2017). Specifically, the encoder further extracts contextualized high-level semantic features from both modalities and the decoder generates a token sequence for different tasks. Besides, since we are using general Transformer architecture, both the text embedding layer and the Transformer can be pre-trained on additional MT data.

heading: 3.3 Word-Aligned Contrastive Learning
text: To alleviate the misalignment between speech and transcript representations, we propose wordaligned contrastive learning to bring speech and text embeddings closer in a fine-grained level (Figure 4).
Given a speech-transcript pair (s, x). The transcript is tokenized by a Byte-Pair-Encoding (BPE) tokenizer into a sequence of BPE tokens x = (x1, x2, · · · , xn). Though BPE is the default option for text tokenization, whole words preserves acoustic boundary better than BPE tokens. Thus, we group n BPE tokens back into m whole words where wi = x[lti : r t i ] for i = 1, 2, · · · ,m where superscript t stands for text feature and li, ri refer to the range of BPE tokens that word i is tokenized into. In the example of Figure 4, x = (_Pra, c, tic, e, _makes, _perfect, .), and w1 = (_Pra, c, tic, e), w2 = (_makes), w3 = (_perfect), and w4 = (.). Then we align whole words w1, w2, · · · , wm with speech s = (s1, s2, · · · , s|s|) by a forced aligner. Here we use Montreal Forced Aligner2 (MFA).
2https://github.com/MontrealCorpusTools/ Montreal-Forced-Aligner
This provides us time interval 1 ≤ lsi ≤ rsi ≤ |s| for word wi where superscript s stands for speech feature.
Now we have identified m corresponding pairs of speech segments s[lsi : r s i ] and words x[l t i : r t i ] for i = 1, 2, · · · ,m. The representations of them are obtained as follows,
f si = MeanPool(S-Enc(s)[l̃si : r̃ s i ]) (1) f ti = MeanPool(T-Emb(x)[l t i : r t i ]) (2)
where S-Enc is the speech encoder, T-Emb is the text embedding layer, l̃si = lsi |s| |S-Enc(s)| and r̃si = rsi |s| |S-Enc(s)| refer to the relative indices given the audio representation length shrinkage after the speech encoder.
We treat f si and f t i as a positive pair and treat f si and other words in the same batch as negative pairs and we apply multi-class N-pair contrastive loss (Sohn, 2016) on them:
CTR(B) =
− E fsi ,f t i∈B
[ log exp(sim(f si , f t i )/τ)∑
f tj =i∈B exp(sim(f s i , f t j )/τ)
]
(3)
where B is the current batch, τ is the temperature hyper-parameter, sim() is used to measure the distance between two representations, and we use cosine similarity sim(a, b) = a b/‖a‖‖b‖.

heading: 3.4 Training and Inference
text: Cross-Modal Pre-training We first pre-train text embedding and joint Transformer on external MT dataset (e.g., WMT dataset). Then we train the MFA model on ASR data DASR to obtain alignments, and further train our ST model using wordaligned contrastive loss on DASR
LPT = E B⊆DASR [ CTR(B)] , (4)
where CTR(B) is defined in Equation 3. The pretraining stage aims to map speech and text embeddings into a shared semantic space using ASR and MT data.
Multi-task Fine-tuning We fine-tune our model using the multi-task cross-entropy losses, as suggested in Ye et al. (2021), and contrastive loss.
LFT = LCE + λLCTR (5) where
LCE = E (s,x,y)∈DST [ ST + MT + ASR] (6)
LCTR = E B⊆DST [ CTR(B)] . (7)
Cross entropy losses are derived directly from the triplet dataset DST,
ST(s, y) = − logP (y|s) (8) MT(x, y) = − logP (y|x) (9) ASR(s, x) = − logP (x|s). (10)
λ is the hyper-parameter controlling the weight of contrastive loss.
Inference During inference, the model accepts speech frames as input of speech encoder and decodes translation in the target language through beam search. No source transcript text is needed during inference.

heading: 4 Experiments

heading: 4.1 Datasets
text: MuST-C We conduct experiments on the MuSTC dataset3,4 (Di Gangi et al., 2019), one of the largest ST benchmark datasets containing translations from English to 8 languages collected from TED Talks. Each language direction involves around 400 hours of audio recordings. Limited by computing resources, we examine our method on three language directions: EnglishGerman/French/Spanish (En-De/Fr/Es).
MuST-C Low-Resource To examine our method in the extremely low-resource settings, we manually create ASR and ST subsets from the MuST-C En-De training set. Specifically, we build 10-hour, 100-hour and 370-hour ASR subsets and 1-hour and 10-hour ST subsets respectively through random sampling.
IWSLT Low-Resource We also evaluate our method on Maltese to English translation in IWSLT 2023 low-resource track5. We use the officially provided ST triplets as ST data and build the ASR dataset by combining the the audio-transcript part from the official ST data and CommonVoice (Ardila et al., 2020)6. We remove silences and randomly partition the data. Finally we obtain 1 hour of ST triplets and 10 hours of ASR pairs as training and development set and 0.1 hour of ST triplets as test set.
External ASR We also introduce LibriSpeech (Panayotov et al., 2015) as the external ASR dataset. LibriSpeech is the de facto public English ASR benchmark7 containing 960 hours of audiobook speech. We build a 1330-hour English ASR dataset by combining MuST-C and LibriSpeech.
External MT Additionally, we introduce external WMT En-De/Fr/Es datasets (Bojar et al., 2016)
3Released under CC BY NC ND 4.0 International 4Here we refer to MuST-C v1.0. 5https://iwslt.org/2023/low-resource 6We do not use the officially provided ASR dataset due to
its severe audio-transcript misalignment. 7Released under CC BY 4.0
for each language direction to pre-train text embedding and Transformer. We also introduce Flores200 (Costa-jussà et al., 2022), a massively multilingual machine translation dataset, for Mt-En direction. As shown in previous work (Xu et al., 2021; Ye et al., 2021), MT pre-training greatly improves ST performance.
The statistics of datasets above are listed in Appendix A.1.

heading: 4.2 Experimental Setups
text: Model Configurations In MuST-C experiments, we use wav2vec 2.0 base model8 in our speech encoder which is solely pre-trained on 960-hour English audio. It consists of a 7-layer convolutional feature extractor and 12 Transformer encoder blocks with 768 hidden units. Two down-sampling convolutional layers have kernel size 5, stride size 2 and hidden size 512 or 1024 depending on the Transformer hidden size. For En-De/Fr/Es directions, the Transformer encoder-decoder has 6 encoder and decoder layers with hidden size 512, 2048 FFN hidden units and 8 attention heads. For Mt-En direction, it has 12 layers each with hidden size 1024, 4096 FFN hidden units and 16 attention heads.
Preprocessing The input speech is the raw 16-bit 16kHz mono-channel waveform. We filter speech that is either too long (>480k frames) or too short (<1k frames) out. This results in 388/471/480 hours of speech being retained as ST training data for EnDe/Fr/Es directions. We jointly tokenize the transcripts and translations for each language direction using SentencePiece (Kudo and Richardson, 2018) with a vocabulary size set to 10k. To conduct forced alignment required by WACO (see section 3.3), we first remove punctuations and group whole words by identifying special space token in the vocabulary. Then we use the MFA to train forced aligners on DASR to align English speech and words. The amount of ASR data used to train forced aligner is the same as that used for training WACO. Due to vocabulary mismatch between the MFA and our SentencePiece model, a small number of speeches and transcripts (e.g., 18h for En-De) cannot be aligned and we simply ignore them when doing contrastive learning.
Training Transformer and text embedding are pre-trained on the external WMT dataset for En-
8https://dl.fbaipublicfiles.com/fairseq/ wav2vec/wav2vec_small.pt
De/Fr/Es (MT training details can be found in Appendix A.5). For Mt-En direction, we directly initializes the Transformer and text embedding with the NLLB-600M model (Costa-jussà et al., 2022)9 pre-trained on Flores-200. For both cross-modal pre-training and multi-task fine-tuning, we set contrastive temperature τ = 0.2 and optimize our model by Adam optimizer (Kingma and Ba, 2015) (β1 = 0.9, β2 = 0.98) with learning rate 1e-4 and 25k warm-up steps. After the warm-up, the learning rate is decayed following the inverse square root schedule. The effective batch size is 16 million frames. We set dropout rate to 0.1. For pretraining, we save the checkpoints with the best contrastive loss on the validation set. For fine-tuning, we save the checkpoints with the best BLEU on the validation set and average the last 10 saved checkpoints. Also, we set label smoothing to 0.1 for the cross-entropy losses, λ = 0 in low-resource ST and λ = 1 in ST with full data. All models are trained on Nvidia A6000 GPUs.
Inference and Evaluation During inference, we run beam search with beam size 10 and length penalty 0.6/1.0/0.1/0.3 for En-De/Fr/Es and Mt-En directions respectively. For evaluation, we report case-sensitive detokenized BLEU scores on MuSTC tst-COMMON and IWSLT Mt-En test set using sacreBLEU (Post, 2018)10.
Baselines In low-resource ST settings, we compare our method with three baselines: • Base: This baseline ignores DASR and only opti-
mizes cross entropy loss in Equation 6 on DST. 9https://github.com/facebookresearch/fairseq/
tree/nllb 10BLEU signature: nrefs:1|bs:1000|seed:12345|case:mixed| eff:no|tok:13a| smooth:exp|version:2.0.0
• Base+CTC: This baseline, on top of Base, applies CTC loss on DASR to align speech and text representations. In particular, we add a linear layer after the speech encoder to predict the text BPE token at each frame and fix its weight with text embedding. We only include CTC with BPE tokenization here since it performs consistently better than its phoneme counterpart (details in Section 5.2). • ConST: This baseline adds a coarse-grained contrastive loss on DASR on top of Base to reduce modality gap as in Ye et al. (2022b). Instead of word-level alignment, ConST conducts contrastive learning on sentence-level average speech and text embeddings. Hyper-parameters are directly borrowed from Ye et al. (2022b). In regular ST with full MuST-C data, we compare our method with other existing works.

heading: 4.3 Main Results
text: Low-Resource ST Results are shown in Table 1. The ASR data for cross-modal pre-training varies from 10 hours to 1330 hours, and the ST data for multi-task fine-tuning varies from 1 hour to 10 hours. WACO consistently outperforms baseline methods in all data configurations and language directions. In particular, our model achieves a BLEU score of 14.1 for En-De and 13.3 for MtEn with only 1h ST and 10h ASR data and 21.0 for En-De with only 10h ST and 100h ASR data. With 1330h ASR data, WACO even pushes the BLEU score to 17.5 and 22.9. More surprisingly, we find that WACO has a further advantage when using less ASR data. When reducing ASR data from 370 hours to 100 hours, the BLEU score increases (WACO vs Base+CTC, ConST) are enlarged from +2.0,+4.9 to +4.0,+8.9 in 1h ST set-
ting. This demonstrates that WACO is more efficient than the baseline methods especially in lowresource setting.
Regular ST Results are shown in Table 2. Here we are using the entire MuST-C training set as in previous works to enable fair comparison, which means DST has full MuST-C training data. WACO is competitive with previous models such as STEMM and ConST in all three language directions. Note that SpeechUT and STPT achieve the highest BLEU scores in all directions, but both SpeechUT and STPT leverages additional speech data (60k hours) or ASR data (100 hours) and employ a different model architecture.

heading: 5 Analysis

heading: 5.1 Why is Word-level Contrastive Loss Better than Sentence-level Contrastive Loss?
text: Intuitively, only aligning the representations between speech and text at the sentence level cannot assure that model captures the accurate word correspondence between these two modalities.
First, we measure the cosine similarity between speech embedding and transcript text embedding using models (ConST and WACO) pre-trained on 370h ASR dataset and fine-tuned on 1h ST dataset. The embeddings of speech and transcript text are calculated based on Equation 1 and 2 respectively. The result is shown in Table 3. WACO achieves more accurate word-level alignment than ConST (0.65 v.s. 0.44), which indicates WACO can handle word-level details inside a sentence better. Besides, though not optimized for sentence-level alignment, WACO still achieves close sentencelevel similarity with ConST (0.30 v.s. 0.33).
We also show an example in Figure 5 to further demonstrate the importance of such details. From the similarity matrix, we can see that WACO aligns both modalities quite well for all words but ConST struggles on words “that” and “evolve” as highlighted in blue boxes. This directly results in two translation errors of ConST. First, it fails to recover the clause structure implied by “that”. Second, it omits “evolve” entirely in the translation. Though ConST still provides higher sentence similarity than WACO, it fails to understand the subtlety inside the sentence. More examples are in Figure 9.

heading: 5.2 Why is WACO better than CTC?
text: WACO treats words as base units to preserve acoustic boundaries and leverage knowledge of the pretrained MT model, while CTC cannot achieve both merits simultaneously. CTC cannot benefit from word tokenization due to its extremely large vocabulary. To preserve acoustic boundaries, CTC requires phoneme or character tokenization. To leverage pre-trained MT model, CTC requires the same tokenization with MT model, i.e., BPE tokenization, but it has no guarantee on well-behaved acoustic boundaries.
Figure 6 gives a typical example of the token-
to-frame embedding alignment matrix, indicating that WACO learns more accurate alignment compared to CTC with BPE tokenization. Table 3 also substantiates this quantitatively (0.65 and 0.30 v.s. 0.08 and 0.13).
For acoustic tokenizations, we evaluate phoneme tokenization. We use the same phoneme vocabulary and grapheme-to-phoneme package as in (Tang et al., 2022). As shown in Table 4, WACO consistently outperforms CTC with phoneme tokenizations, proving its effectiveness in exploiting pretrained MT knowledge.

heading: 5.3 Comparison with Sequence-Level Knowledge Distillation and Cascade ST
text: One might argue that for the low-resource scenario we described above, sequence-level knowledge distillation (SeqKD, i.e. forward translation) (Kim and Rush, 2016) and building a cascaded system are the most straightforward way to leverage additional data and improve performance. The former trains an E2E model by expanding the ASR data translated from the existing MT model, while the latter builds the ASR and MT models separately. We compare WACO and these two methods in various settings to demonstrate their differences. The implementation details of SeqKD and Cascade ST are described in Appendix A.6.
We show the result with 10h ASR and 1h ST data in Table 5. Results of more data configurations are shown in Table 10 in the Appendix. We vary the amount of other pre-training resources: speechonly data and MT data. Speech-only data is used to pre-train speech encoder (i.e., wav2vec 2.0) and MT data is used to pre-train text embedding and joint Transformer in WACO and MT models in SeqKD and Cascade ST.
All models drops 1∼2 BLEU points when using 10% MT data. Cascade ST performs better than
WACO when initialized with wav2vec 2.0. However, WACO outperforms the cascade system when we train speech encoders from scratch. The ASR model trained from scratch in Cascade ST fails to produce meaningful transcript due to severe overfitting. Surprisingly, both WACO and SeqKD are not sensitive to wav2vec 2.0 initialization. WACO and SeqKD are also complementary to each other, and combining them leads to the highest BLEU score.

heading: 6 Conclusion
text: In this work, we propose WACO to align wordlevel speech and text embeddings. Experiments demonstrate the effectiveness of our method in both low-resource and regular ST settings. Analysis shows that our method can achieve better speechtext alignment which correlates well with its ST performance.
Limitations
There are two main limitations in this work. First, instead of best ST performance given full data, our cross-modal pre-training only aims to demonstrate the effectiveness of our method in the low-resource ST setting. We realize that unified pre-training for both speech and text gradually becomes a dominant paradigm for ST and our future work is to fuse WACO into a joint pre-training framework.
Second, we note that Tang et al. (2022) explores the possibility of pre-training MT models with phoneme tokenizations, though it is unclear if the phoneme-based MT model has an advantage over the BPE-based MT model. We follow the tradition of using the latter one and leave the comparison of them in future works.
Ethics Statement
WACO has the potential to benefit speakers of lowresource languages. For example, their published video or speech can be better translated into other languages, so more viewers in the world can understand them, enabling deeper communication between different cultures. Though WACO may be beneficial to cross-language communication, we do not encourage users to treat the translation generated by the E2E ST model as fully correct since they are far from perfect in practice.

heading: Acknowledgement
text: Siqi Ouyang is supported by UCSB-IEE-Meta Collaborative Research Grant on AI.

heading: A Appendix
text: A.1 Statistics of Datasets We show statistics of MuST-C, IWSLT Mt-En, LibriSpeech and WMT datasets in Table 6,7,8 and 9.
A.2 Impact of Hyperparameter Temperature Figure 7 demonstrates BLEU scores produced by different temperature values
in low-resource ST setting with 10h ASR and 1h ST data. Higher temperature in general leads to higher BLEU scores, but the marginal improvement becomes negligible when τ > 0.5.
Pooling and Layer Here we compare different pooling mechanisms used to aggregate word-level representation and also different layers used to extract representations (before or after joint Transformer encoder). Table 12 shows BLEU scores in low-resource ST setting with 10h ASR and 1h ST data. WACO is not very sensitive to pooling method, but sensitive to the layer selection. Aligning representations extracted before joint encoder delivers much better performance.
A.3 More Examples of WACO versus ConST We show two more examples that WACO achieves more accurate translation than ConST by better speech-text alignment in Figure 9.
A.4 Loss Curves for Cross-Modal Pre-training
We present pre-training loss curves of CTC with both BPE and phoneme tokenizations, and WACO in Figure 8.
A.5 MT Pre-training
We use the same vocabulary and SentencePiece model described in Section 4.2 to tokenize the WMT datasets. The model is optimized with Adam. The learning rate starts at 1e-7, warmed up to 7e-4 by 4k steps and then decays following the inverse square root schedule with a minimum learning rate of 1e-9. The maximum number of tokens in a batch is 8192. We select the checkpoint with the highest BLEU (beam size 4, length penalty 0.6) on the WMT validation set.
A.6 Sequence-Level Knowledge Distillation and Cascade ST
SeqKD We apply the same MT model used to initialize joint Transformer in WACO to translate the transcript of ASR data into target language,
which is German in our case. The translation is produced by beam search with width 10 and length penalty 0.6. After we construct the pseudo ST dataset (10h), we combine it with the real-world ST dataset (1h) and obtain an 11h triplet ST dataset. Then we follow the same fine-tuning procedure in low-resource ST setting, i.e., cross-entropy losses, to obtain the final ST model.
Cascade ST We follow the same fine-tuning procedure used in (Baevski et al., 2020) to fine-tune wav2vec 2.0 small on 10h ASR data to obtain the ASR model. The final transcript is decoded by Viterbi algorithm. We use the same MT model and configuration as in SeqKD to translate the English transcript into German translation.
A.7 Results of More Data Configurations We compare WACO and Cascade ST with or without wav2vec2 in more data configurations as in Table 1 except for 1330-hour ASR. The results are shown in Table 10.
A.8 Word Error Rate of using WACO as an ASR model
The word error rates (WER) of WACO with different data configurations are shown in Table 11.
A.9 Training Efficiency The computation cost of calculating WACO loss function is higher than that of sentence-level methods (e.g., ConST) and in our profiling results, it takes 20∼30% time of a forward pass. However, WACO converges much faster, in terms of number of iterations, than ConST and CTC. In the 100h ASR data case, WACO only needs <25k iterations to converge while both ConST and CTC requires >50k iterations. This makes WACO more time efficient overall.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 7

heading: 3 A2. Did you discuss any potential risks of your work?
text: section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 0 and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section 4

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data we use are all from public available and widely-used benchmarks.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We include the statistics in both section 4 and appendix.
C 3 Did you run computational experiments? section 4 and 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4 and appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4 and 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We realize that unified pre-training for both speech and text gradually becomes a dominant paradigm for ST and our future work is to fuse WACO into a joint pre-training framework. Second, we note that Tang et al. (2022) explores the possibility of pre-training MT models with phoneme tokenizations, though it is unclear if the phoneme-based MT model has an advantage over the BPE-based MT model. We follow the tradition of using the latter one and leave the comparison of them in future works