CTC-based Non-autoregressive Speech Translation
Year: 2023

Abstract:
Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation and low latency. In this paper, we investigate the potential of connectionist temporal classification (CTC) for non-autoregressive speech translation (NAST). In particular, we develop a model consisting of two encoders that are guided by CTC to predict the source and target texts, respectively. Introducing CTC into NAST on both language sides has obvious challenges: 1) the conditional independent generation somewhat breaks the interdependency among tokens, and 2) the monotonic alignment assumption in standard CTC does not hold in translation tasks. In response, we develop a prediction-aware encoding approach and a cross-layer attention approach to address these issues. We also use curriculum learning to improve convergence of training. Experiments on the MuST-C ST benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67×, which is comparable to the autoregressive counterpart and even outperforms the previous best result of 0.9 BLEU points1.

heading: 1 Introduction
text: End-to-end speech translation (E2E ST) has attracted unprecedented attention and achieved dramatic development in recent years (Duong et al., 2016; Berard et al., 2016; Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Wang et al., 2020b,c; Xu et al., 2021; Zhang et al., 2022b). Stand-alone
*Corresponding author. †Work was done while at ByteDance AI Lab. 1The code is available at https://github.com/xuchennlp/
S2T.
modeling reduces the inference latency by almost half compared to cascaded systems, where the automatic speech recognition (ASR) model and the machine translation (MT) model run serially. This helps the application in real scenarios, especially with limited computational resources.
However, this advantage only holds in the context of autoregressive (AR) decoding, where each token is generated depending on the previously predicted results. Non-autoregressive (NAR) generation (Gu et al., 2018), the recently popular decoding method in ASR and MT, makes the inference process fast by predicting the output sequence in parallel, resulting in the E2E ST no longer being superior in terms of inference speed-up.
A natural question arises: can we build a powerful non-autoregressive speech translation (NAST) model? The NAR results in the latest literature are still inferior to the AR counterparts with a large gap of about 2 ∼ 3 BLEU points, even with the iterative refinement process (Inaguma et al., 2021a). In this work, we aim to develop a promising NAST model for comparable performance to the AR model without complex decoding.
We resort to the connectionist temporal classification (CTC, Graves et al., 2006) because of its great success in ASR and MT and the convenience of variable length prediction. CTC is well suited for speech-to-text modeling, where the input sequence is longer than the output. Recent studies show that CTC-based NAR models achieve comparable or even better performance than their AR counterparts, providing insight into the design of the powerful CTC-NAST model.
Our CTC-NAST model is decoder-free and consists of two stacked encoders: an acoustic encoder and a textual encoder. They are guided by CTC to
13321
predict transcription and translation, respectively (Chuang et al., 2021). Then, we carry out a careful and systematic inspection of the underlying issues and address the challenges of CTC-NAST. In particular,
• The conditional independence assumption allows fast inference but omits interdependency across the whole sequence. We identify the prediction-aware encoding (PAE) method underlying the success of a series of studies (Nozaki and Komatsu, 2021; Huang et al., 2022; Higuchi et al., 2021a), which observe preliminary prediction and refine it in the final generation. Following this idea, we predict the CTC result in the intermediate layer and then integrate it into the subsequent encoding.
• Another inherent property of CTC, the monotonic assumption, is valid for ASR but does not hold for translation tasks, where a future word in the target text may be aligned with the earlier part of the source text, especially on distant language pairs (Hannun, 2017). A critical requirement of the decoder-free design is the reordering augmentation (Chuang et al., 2021). As a remedy, we introduce an additional cross-layer attention module, which is complementary to the self-attention module.
Even with the above efforts, NAST is still a difficult task that suffers from heavy modeling burdens. A curriculum learning strategy that guides the training in an easy-to-hard way is significant for better convergence. We replace part of the incorrect prediction with ground truth in PAE to prompt the generation of the whole sequence. In this way, the model relieves the CTC learning burden by observing almost the whole sequence in the early stages, while only a few tokens are replaced as CTC performance improves, ensuring consistency between training and inference.
Our CTC-NAST model is simple, completely parallel, and works well for both similar and distant language pairs. The proposed methods yield a remarkable gain of 3.0 BLEU points on MuST-C En-De, achieving an average BLEU score of 29.5 with an inference speed-up of 5.67×, and even outperforming the best previous AR results by 0.9 BLEU points. We also report competitive results on the more challenging MuST-C En-Ja and FisherCallhome corpus.

heading: 2 Background

heading: 2.1 Connectionist Temporal Classification
text: CTC (Graves et al., 2006) was originally proposed for labeling unsegmented sequences. It learns monotonic alignment between acoustic features and transcriptions, which is valid for cross-modal learning like ASR. CTC helps convergence and allows re-scoring decoding through a lightweight output layer, achieving great success in ASR as an auxiliary loss on top of the encoder (Watanabe et al., 2017; Karita et al., 2019). Given the encoder representation h and the corresponding sequence y, the CTC loss is defined as:
LCTC = −logPCTC(y|h) (1)
where the probability is calculated by marginalizing over all possible alignments Φ(y) between h and y:
PCTC(y|h) = ∑
π∈Φ(y) P(π|h) (2)
CTC has the same conditional independence property as NAR generation, where the probability of the path π is the product of the probability P (πt|ht) at each time step t:
P(Y |X) ≈ T∏
t=1
P(πt|ht) (3)
where T is the length of h.

heading: 2.2 AR and NAR
text: Given a source sequence X = (x1, · · · , xT ′ ), a sequence-to-sequence model predicts the target sequence Y = (y1, · · · , yT ) by conditional distribution:
P(Y |X; θ) = T∏
t=1
PAR(yt|y<t, X; θ) (4)
where θ is the model parameters. This autoregressive generation learns sequential dependency but suffers from high inference latency.
Instead, NAR carries out the conditional independent prediction for parallel inference (Gu et al., 2018):
P(Y |X; θ) = T∏
t=1
PNAR(yt|X; θ) (5)
Although the vanilla NAR model speeds up inference by about 15× (Gu et al., 2018), it is still inferior to the AR counterpart by a large gap.
Researchers have proposed many series of methods to improve the generation quality and investigate a better trade-off between performance and speed in the MT task, such as the iterative decoding method (Lee et al., 2018; Stern et al., 2019; Ghazvininejad et al., 2019; Kasai et al., 2020), latent variable method (Gu et al., 2018; Song et al., 2021; Gu and Kong, 2021), data manipulation method (Zhou and Keung, 2020; Bao et al., 2022; Ding et al., 2021), enhancement based method (Guo et al., 2019; Wang et al., 2019), and semiautoregressive decoding (Ran et al., 2020). There are also some studies to design the architecture of the NAR models, such as the use of CTC for prediction for its ability of variable length prediction (Libovický and Helcl, 2018; Shu et al., 2020; Saharia et al., 2020).
In addition, the NAR generation also shows promising results in ASR task, especially the CTCbased systems (Higuchi et al., 2020, 2021b; Lee and Watanabe, 2021; Nozaki and Komatsu, 2021; Kim et al., 2022).

heading: 2.3 Speech Translation
text: Recently, E2E ST has received a lot of attention due to its direct modeling (Berard et al., 2016). Unlike the conventional cascaded system that decouples the cross-modal and cross-lingual modeling into ASR and MT models respectively (Ney, 1999; Mathias and Byrne, 2006), the end-to-end manner is more elegant and has the potential for fast inference and error-free propagation.
One promising route to improve ST is to develop more adaptive architectures according to the task characteristics. Based on the idea of modeling decoupling, the stacked encoding method divides cross-modal and cross-lingual learning into acoustic and semantic encoders, respectively (Liu et al., 2020; Xu et al., 2021). In this design, the CTC loss for transcription is usually introduced to guide the learning of the acoustic encoder, which significantly helps convergence. In addition, the latent alignment learned in the CTC is used to bridge the two encoders. Liu et al. (2020) shrink the sequence length based on CTC prediction. Xu et al. (2021) introduce an adapter to bridge two encoders by integrating CTC prediction.
Several studies investigate the NAR generation
in ST (Inaguma et al., 2021a,b; Chuang et al., 2021). However, current NAR systems are still inferior to AR counterparts, especially CTC-based systems. Researchers also continue to extend the use of CTC to learn target text as an auxiliary loss of the encoder (Zhang et al., 2022a; Yan et al., 2022). But there is no work to inspect the underlying issues in the CTC modeling of target text in ST. To this end, we study the challenges of building a powerful CTC-based NAST model and then propose corresponding methods. We also extend our method to AR models for a comprehensive exploration.

heading: 3 CTC-NAST
text: Among many well-established NAR designs for ASR or MT models, CTC is particularly suitable for ST modeling because the input length is remarkably longer than its output. In this section, we present CTC-NAST in detail. We first describe the base architecture, then identify and address three underlying challenges. See Figure 1 for an overview of our system.

heading: 3.1 Base Architecture
text: ST aims to translate audio in the source language to text in the target language directly. Let (x; ys; yt) be a training sample of ST, where x is the input speech feature sequence, ys is the corresponding transcription of x, and yt is the translation in the target language. We assume that transcription is always available in our work.
We drop the decoder network and rely only on the CTC-based encoder. Following the design of SATE (Xu et al., 2021; Chuang et al., 2021), we decouple the encoding into an acoustic encoder and a textual encoder in a stack architecture, as shown in Figure 1(a). They are guided by CTC loss for transcription and translation (denoted CTC and XCTC for distinction), respectively.
Formally, given a representation ha of the acoustic encoder output, the CTC loss is calculated as:
LCTC = −logPCTC(ys|ha) (6) Similarly, the XCTC loss is calculated as:
LXCTC = −logPXCTC(yt|ht) (7) where ht is the representation of the textual encoder output.
Then, the training objective is formulated as the interpolation of the two CTC losses:
L = αA · LCTC + αT · LXCTC (8)
where αA and αT are the coefficients of the CTC and XCTC losses, respectively.
Although CTC works well for the NAR ASR model, extending CTC naively to the more challenging ST task is fragile. We claim that CTCNAST can be improved by addressing three issues:
• Conditional independence assumption is an inherent property of CTC, which ignores interdependency with past or future contexts, leading to poor generation (Chan et al., 2020), like repetition and omission errors.
• Although the self-attention network has the modest reordering capability (Chuang et al., 2021), our encoder-only architecture is hard to handle the monotonic assumption, especially for distant language pairs.
• E2E ST already suffers from the heavy burden of cross-modal and cross-lingual mapping, while NAR modeling further aggravates the difficulty and results in poor convergence.

heading: 3.2 Prediction-aware Encoding
text: NAR generation enlarges the search space in inference due to conditional independence (Ran et al., 2021), especially with the long speech sequence of hundreds and thousands of units. A commonlyused solution, incorporating latent variables that contain the initial prediction into modeling, has
been demonstrated to be effective (Lee et al., 2018). In this way, the NAR generation is decoupled as the multiple-step refinement of the target sequence, enabling the model to be aware of the previous prediction.
Inspired by the prior efforts in MT (Huang et al., 2022) and ASR (Nozaki and Komatsu, 2021), we introduce prediction-aware encoding (PAE). The detailed illustration is shown in Figure 1(c). Specifically, given one representation hl outputted by the intermediate encoder layer l, PAE integrates the prediction information (corresponding 1⃝ in the Figure) into the following encoding explicitly by weighting the embedding matrix W over the current CTC distribution (called InterCTC) (Xu et al., 2021):
PAE(hl) = hl + PInterCTC(π|hl) ·W (9) where the weights W are shared in the whole network. Note that we use PAE to augment the learning of both CTC and XCTC.
Since the poor prediction leads to the risk of error propagation, we also optimize the InterCTC loss for guaranteed prediction:
LInterCTC = −logPInterCTC(y|h) (10) In this way, we ensure that CTC predicts well. However, the worse result for XCTC limits the benefits of PAE, which may result in negative effects. We alleviate this issue in Section 3.4.
Now, we re-formulate the training loss in Eq. 8 as:
L = αA · LCTC + αT · LXCTC
+ βA · 1
M
m∑
m=1
LmInterCTC
+ βT · 1
N
N∑
n=1
LnInterXCTC (11)
where M and N are the numbers of the intermediate CTC and XCTC, βA and βT are the corresponding coefficients.

heading: 3.3 Reordering Augmentation
text: Vanilla Transformer generates each token by distributing the weight of the encoder-decoder attention module to the corresponding source part to be translated, which easily handles the order gap between languages. However, CTC modeling faces the intractable issue of reordering the representation into the target language order during encoding. Although previous studies have demonstrated that the MT or ST encoder can capture the global information (Yang et al., 2018; Xu et al., 2021), it is still difficult to rely only on the self-attention module to search the positions that contribute significantly to decoding (Chuang et al., 2021).
To enhance the reordering capability of CTCNAST, we mimic the design of the decoder and introduce cross-layer attention (CLA) module, which is inserted between the self-attention module and the feed-forward module in the specific layers of the textual encoder, as shown in Figure 1(b). Let SA(·, ·, ·) and CLA(·, ·, ·) denote the self-attention and CLA modules, the new Transformer layer j can be formulated as:
h ′ = hj−1 + SA(hj−1, hj−1, hj−1) (12) h ′ = h ′ + CLA(h ′ , hk, hk) (13) hj = h ′ + FFN(h ′ ) (14)
where hk is the representation output from the layer k(k < j).
In this way, CLA offers a remedy for the lacking attention, that captures the information from the bottom layer directly and is complementary to the self-attention module. Now the textual encoder acts as both a stack of the encoder and the decoder of the vanilla encoder-decoder model.
In order to further enhance the ability of CLA, we introduce the drop-net technique. In each layer
containing the CLA module, we drop the selfattention module with a probability pdrop ∈ [0, 1]. Note that the self-attention module always keeps during inference.

heading: 3.4 Curriculum Learning Strategy
text: Even with the auxiliary encoding and improved design architecture, the CTC-NAST model still faces the difficulty of a heavy modeling burden, leading to poor convergence. Inspired by Qian et al. (2021), a curriculum learning strategy is remarkably important to reduce the dependency in the early stage and increase the difficulty along the training process.
As illustrated in Figure 1(c), we replace part of the prediction (corresponding 1⃝ in the Figure) in Eq. 9 with the ground truth (corresponding 2⃝ in the Figure), which mitigates the negative effects of error propagation caused by the poor XCTC performance in PAE and prompts the generation of the whole sequence. Unlike the same lengths between input and output in the decoder, the length of the input acoustic feature is remarkably longer than the corresponding text in CTC. Therefore, we take the best alignment computed by the model as the ground truth (Gu and Kong, 2021; Huang et al., 2022):
π̂ = arg max π∈Φ(y)
P(π|s; θ′) (15)
where θ ′
is the current model parameter. Note that the length of π̂ is the same as the input.
Denote the replacement ratio as r ∈ [0, 1], we uniformly sample a random variable U from [0, 1]:
P̂t = I(U >= r) ∗ pt + I(U < r) ∗ π̂t (16)
where I(·) is the indicator function. However, this strategy results in the inconsistency between training and decoding, where the ground truth is unavailable during decoding. To address this issue, Qian et al. (2021) adaptively determine the replacement ratio depending on the current prediction accuracy. But it does not work for CTC-NAST, as shown in Appendix B.3.
Considering the long input sequence in ST, a lower ratio may not provide sufficient prompt, but a higher ratio may result in a severe gap between training and decoding. Therefore, we limit that only the positions where a wrong prediction (argmax pt ̸= π̂t) occurs are replaced. In this way, we enable the large ratio throughout the whole training process. As the accuracy increases, more and
more positions rely on the model’s predictions, and the guidance to the fewer positions with errors always remains stable for better convergence. We call this method curriculum learning mixing (CLM).
Finally, we smooth the ground truth to obtain a distribution similar to the CTC prediction, where the dominant probability is concentrated on the ground truth position, and the rest is evenly distributed among other tokens.

heading: 3.5 Inference
text: CTC-NAST is a fully parallel decoding model. The inference resembles the training process, except the CLM method is not used. We employ greedy decoding, where CTC picks the tokens with maximum probability in each time-step, then removes the blanks and repeated tokens for final translation.

heading: 4 Extension on AR model
text: Now a natural question arises: can our method proposed for the NAR model be used to improve the AR model? Our method produces better encoder representations for CTC prediction, but there is no evidence to demonstrate that the optimization of the CTC and the cross-entropy in the decoder are completely consistent. Excessive optimization of the encoder may interfere with the learning of the decoder.
To answer it, we adopt these techniques to the
encoder-decoder counterpart (called CTC-Aug ST), to investigate the effects of different architectures. And the training loss is formulated as:
L = LS2S + αA · LCTC + αT · LXCTC
+ βA · 1
M
m∑
m=1
LmInterCTC
+ βT · 1
N
N∑
n=1
LnInterXCTC (17)
where LS2S is the cross-entropy loss of the decoder.

heading: 5 Experiments
text: We evaluate our method on the MuST-C and FisherCallhome benchmarks. Details about the datasets and model settings are described in Appendix A.

heading: 5.1 Main Results
text: The results on the MuST-C corpora in Table 1 show that our method significantly outperforms previous AR and NAR models. We achieve remarkable gains for all language pairs. Here we highlight several major breakthroughs: i) CTC-Aug ST is shown to be effective for the AR models, which gains an average of 0.6 BLEU points over the previous best work even without the augmentation of sequencelevel knowledge distillation (Seq-KD) data. Note that not all proposed methods are used in CTCAug ST (see Section 5.2.2). ii) Our CTC-NAST
models achieve comparable or better performance to the powerful AR counterparts on all 9 language pairs, with a high speed-up of 5.67×. Note that CTC-NAST achieves a higher speed-up under large batch sizes (see Section 5.2.4). iii) Referring to Appendix B.1, the En-Ja translation has a strong demand for reordering capability. Our method also works well on this challenging distant language pair, demonstrating the potential of CTC-NAST.
Similar results on Fisher-Callhome are shown in Table 2. Interestingly, the NAST model outperforms the AR counterpart with 0.3 ∼ 0.4 BLEU points on the out-of-domain Callhome sets. We find that the AR models miss some segments when translating the long sentences, while the CTCNAST models still guarantee good performance, as shown in Appendix B.2. It demonstrates the robustness of our CTC-NAST model.

heading: 5.2 Analysis
text: Next, we study several interesting problems on MuST-C En-De and En-Ja datasets to investigate the effects on similar and distant languages. We present further analyses in Appendix B.

heading: 5.2.1 Performance over Sentence Lengths
text: Figure 2 shows the results of the AR and NAR models with and without the proposed methods on the MuST-C En-De corpus with respect to output lengths. The base NAR model performs much
worse than AR counterpart. But interestingly, unlike the ST model, which has an outlier as sentence length increases, the NAST model maintains stable performance. This is similar to the results on Fisher-Callhome in Appendix B.2.
Our methods bring remarkable gains over different lengths for both AR and NAR models, leading to comparable translation quality when the length is less than 60. In particular, CTC-NAST performs even better than AR models when the length is less than 30. However, the performance gap increases with sentence length. We speculate that very long input acoustic features make it more difficult to model semantic information. Future work (Xu et al., 2023) can focus on enhancing the ability to handle complex acoustic encoding.

heading: 5.2.2 Effects of Each Method
text: We compare the results of each method on AR and NAR models in Table 3. More detailed ablation studies of CLA and CLM are presented in Appendix B.3. The base AR model is trained with auxiliary loss, where CTC on top of the acoustic encoder learns to predict the source text. Interestingly, there are different effects on different models, languages, and training data. All methods are lightweight in both computational cost and parameter quantity.
Introducing the XCTC loss and PAE method achieves better performance in nearly all settings. CLA does not work well on the similar En-De language pair due to the less reordering requirement, but stable improvements on the distant En-Ja language pair. The remarkable results of CLM demonstrate that an adaptive training strategy is important for better convergence of NAR models (Qian et al., 2021).
However, CLM leads to slightly worse or better results for AR models trained on Seq-KD data. We conclude that the optimization of XCTC loss in the encoder interferes with the learning of crossentropy loss in the decoder. Although the XCTC achieves good performance, it does not contribute to the final inference in the encoder-decoder framework. In addition, the performance of the AR model trained on raw En-Ja data drops terribly. Raw data distribution is difficult to learn by CTC, especially for distant En-Ja language pair. In this case, the CLM always provides ground truth in a high ratio to mix, leading to overfitting on the training set and worse performance during inference. Therefore, we only use XCTC and PAE on AR models for stable improvements.
We also notice that the simplified data distribution is crucial for achieving optimal performance with the NAST model. Specifically, the base NAR models, when trained on raw data, significantly un-
derperform models trained on Seq-KD data, with a gap of about 7 BLEU points. By combining proposed methods, we develop a powerful NAR model that narrows the gap to within 2 BLEU points. This result highlights the robustness of CTC-NAST, even in the presence of complex data distributions.

heading: 5.2.3 Error Analysis
text: To identify the weakness of NAR generation, we measure the word error rates (WERs) of AR and NAR models on the MuST-C En-De and En-Ja datasets2. For a token in the target text, the sub error indicates that it is incorrectly translated, and the del error indicates that it is omitted. The ins error indicates that the token not in the target text is translated.
High del error rates show that the dominant disadvantage of the NAST model is missing translation. PAE relaxes the conditional independence assumption, giving better results for En-De but increased sub errors for En-Ja. We speculate that this is because poor CTC prediction introduces excessive errors. CLA is particularly effective at reducing del errors, which is consistent with our motivation to relax the monotonic assumption. And CLM reduces error propagation and improves the
2Although WER is the metric for ASR, it helps to understand the error types of the translation results.
robustness of PAE, achieving consistent improvements.
However, the combination of our methods does not lead to a further reduction in del errors. A possible reason is that the inconsistent learning between CLA and CLM limits the effect of the combination. We will explore better methods to alleviate the missing translation in the future.

heading: 5.2.4 Speed-up vs. Batch Size
text: We examine the speed-up compared to AR models under different batch sizes and beam sizes in Figure 3. Our CTC-NAST model consistently maintains a high speed-up, even with a large batch size of 32. The performance of NAR and AR models is comparable when using a beam size of 1, while our NAR model is more than 5× faster. In addition, our encoder-only design simplifies the inference process, eliminating the need for length prediction or iterative refinement. One promising direction is to develop effective encoding methods that can bridge the length gap between acoustic features and text. This has the potential to reduce the computational cost caused by long sequence modeling.

heading: 6 Conclusion
text: Aiming to combine E2E ST and NAR generation, we propose CTC-NAST, which consists of only two CTC-guided encoders for source and target text prediction, respectively. We identify and address several challenges of CTC-NAST: conditional independence assumption, monotonic assumption, and poor convergence. In this way, our CTC-NAST model outperforms the previous best AR models by 0.9 BLEU points. We believe that we are the first to present a NAST model that achieves comparable or better performance than strong AR counterparts.
Limitations
Although our CTC-NAST model achieves excellent performance, there are still some underlying challenges that remain in the follow-up of our work. Here are some limitations that we intend to resolve in the future:
• The better designs of reordering augmentation and training strategy. Although the proposed CLA and CLM approaches achieve good results by alleviating the monotonic assumption and relieving the modeling burden, combing them can not bring remarkable improvement. More importantly, these two methods fail to stable improvements in encode-decoder architecture. This drives us to investigate the interference of the optimizations between CTC and cross-entropy.
• Combination with the pre-training or multitask learning. Although our methods bring remarkable gains on both AR and NAR models, we do not explore the utilization of external data resources. Although we can use the pre-trained models directly, we expect more effective methods in future work. Theoretically, we need to design NAR ASR and MT models that share the same or similar architectures with the acoustic encoder and textual encoder, respectively. In this way, the NAST model bridges the gap between pre-training and fine-tuning and has more potential for better performance.
• The potential risk for unwritten languages. In our work, we assume that transcription is always available, which is consistent with almost previous studies. Although some datasets have no transcription, we can use a well-trained ASR model to generate pseudo labels. However, it is hard to handle speech translation from unwritten source speech. The supervision of source text is very important for our model. Therefore, we need to develop better methods for stable training.

heading: Acknowledgement
text: The authors would like to thank anonymous reviewers for their insightful comments. This work was supported in part by the National Science Foundation of China (No. 62276056), the National Key
R&D Program of China, the China HTRD Center Project (No. 2020AAA0107904), the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01), the Yunnan Provincial Major Science and Technology Special Plan Projects (No. 202103AA080015), the Fundamental Research Funds for the Central Universities (Nos. N2216016, N2216001, and N2216002), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No. B16009).

heading: A Experimental Settings
text: A.1 Datasets and Preprocessing
We conduct experiments on the MuST-C (Gangi et al., 2019) and Fisher-Callhome ST (Post et al., 2013) datasets. MuST-C is a multilingual speech translation corpus extracted from TED lectures. We test our method on all MuST-C v1 corpora: English (En) to German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro) and Russian (Ru). In addition, we also investigate the results of the distant language pair English-Japanese (En-Ja) corpus in the MuST-C v2 dataset. We select (and tune) the model on the dev set (Dev) and report the results on the tstCOMMON set (Test).
Fisher-Callhome is a Spanish-English speech-totext translation dataset with 138k text pairs. This corpus contains 170 hours of Spanish conversational telephone speech, as well as Spanish transcripts and English translations. Following the recipe of ESPnet (Inaguma et al., 2020), we lowercase all texts, and remove all punctuation marks except apostrophes. We select (and tune) the model on the Fisher-dev set, and report the results on the Fisher-{dev, dev2, test} and Callhome-{devtest, evltest} sets.
Following the preprocessing recipes in the fairseq toolkit3, we remove utterances with more than 3,000 frames or less than 5 frames. We extract the 80-channel Mel filter bank features by a window size of 25ms with a stride of 10ms. The text is tokenized using the scripts of Moses (Koehn et al., 2007) except that the Japanese text uses MeCab4. We learn SentencePiece5 segmentation with a size of 10,000 for MuST-C datasets. We use a shared vocabulary for the source and target languages for MuST-C v1 corpora, the independent vocabulary for the En-Ja corpus. And we use a shared vocabulary with a size of 1, 000 for Fisher-Callhome datasets.
A.2 Model Settings
We implement our method based on the fairseq toolkit (Ott et al., 2019). We use the Adam optimizer with β1 = 0.9, β2 = 0.98, and adopt the default learning schedule in fairseq. We apply dropout with a rate of 0.15 and label smoothing of 0.1 for regularization.
Following previous studies on NAR models, our model is trained by sequence-level knowledge distillation (Seq-KD) (Kim and Rush, 2016) data generated by a small MT model with a beam size of 5. Our NAST model consists of an acoustic encoder with 12 Conformer layers and a textual encoder with 12 Transformer layers. Each layer comprises 512 hidden units, 8 attention heads, and 2048 feedforward sizes. We use PAE in layers 6 and 9 in both the acoustic encoder and the textual encoder. In multitask learning, the weights of αA, αT , βA and βT are all set to 1. We start the cross-layer attention from layer 4 in the textual encoder and take the representation output from layer 3 as the key and value. The ratio for curriculum learning mixing is set to 0.8.
We extend our method to the encoder-decoder model with similar settings, where the textual encoder has 6 Transformer layers and the decoder has 6 layers. In this way, we control the model parameters to about 150M for fair comparisons. The weights of αA and αT are set to 0.2, and the weights of βA and βT are to 0.1. We use PAE in layer 4 in the textual encoder. We start the crosslayer attention from layer 3 and take the representation output from layer 2 as the key and value.
3https://github.com/pytorch/fairseq 4https://github.com/taku910/mecab 5https://github.com/google/sentencepiece
During inference, we average the model parameters on the best 10 checkpoints based on the performance of the development set. We use beam search with a beam size of 5 for the AR model. The decoding speed is measured on the test set with a batch size of 1 on an Nvidia A100 80GB GPU. We run 5 times to calculate the average time. We report case-sensitive SacreBLEU (Post, 2018) on the MuST-C datasets and case-insensitive SacreBLEU on the Fisher-Callhome dataset for standardization comparison across papers.

heading: B More Analysis
text: B.1 Reordering Difficulty
Following the metric in Chuang et al. (2021), we measure the reordering difficulties Rπ on 9 language pairs of MuST-C datasets in Table 5. The higher the value of Rπ, the higher the reordering difficulty between texts from two languages, indicating the high demand for improved reordering capability. The Seq-KD technique reduces the reordering difficulty by simplifying the data distribution, except for En-Ja. The reason is that noisy data leads to poor MT performance on the En-Ja dataset. On this distant language pair, our CTCNAST model still achieves a high BLEU score of 16.2, which is comparable to the AR model with a small gap of only 0.2 BLEU points.
B.2 Results on Out-of-domain Data
We also measure the BLEU scores of AR and NAR models under different output lengths on the Callhome sets in Figure 4. Note that Callhome sets are out-of-domain because we only use the Fisher set for training. Here, BLEU scores of the NAR model are better than those of the AR model in most cases of output length. In particular, when the output length is greater than 50, the performance of the AR model drops sharply, while the performance of
the NAR model keeps stable. This demonstrates that our CTC-NAST has better robustness.
B.3 Ablation Studies
To further verify the effectiveness of our proposed methods, we construct a series of ablation studies on MuST-C En-De and En-Ja datasets. Effects of CLA Table 6 shows the results of the CLA module. CLA improves the reordering capability and complements the self-attention module. However, using the CLA module naively brings only modest improvements. We randomly drop the self-attention module with a probability of 0.1, which provides better regularization and robust improvements. Note that the high drop probability may lead to insufficient training of the selfattention module. These results demonstrate the
effectiveness of the CLA module and drop-net technique. Effects of CLM As shown in Table 7, the straightforward mixed training has produces remarkable gains with a ratio of 0.5 or 0.8 on both En-De and En-Ja datasets. The adaptive strategy in NAR MT does not work in CTC-NAST. This is because the sequence length of the input acoustic feature is very lengthy, and the decreased mixing ratio cannot provide enough cues to facilitate training. For stable training, we only replace positions where wrong predictions arise. In this manner, accurate positions solely rely on self-prediction, guaranteeing consistency between training and decoding. Furthermore, we generate a smooth distribution akin to CTC prediction, in which the ground truth token has a high probability of 0.9, and the probabilities of other tokens sum to 0.1. The combination of these two approaches results in additional and stable improvements.
We also calculate BLEU scores with various mixing ratios in Figure 5. Our CLM approach is superior to the naive mixing method, particularly at a high ratio. In this case, our approach incorporates more revisions solely for incorrect predictions, which facilitates the training process and guarantees consistency.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7.
7 A2. Did you discuss any potential risks of your work? We propose a method for non-autoregressive speech translation, which does not have any risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5 and Appendix A.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5 and Appendix A.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Fairseq is an open-sourced toolkit under MIT license. We implement our method based on it.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5 and Appendix A.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use open datasets and cite the related papers.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.
C 3 Did you run computational experiments? Section 5 and Appendix A.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.2.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 and Appendix B.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.2.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Here are some limitations that we intend to resolve in the future:
• The better designs of reordering augmentation and training strategy. Although the proposed CLA and CLM approaches achieve good results by alleviating the monotonic assumption and relieving the modeling burden, combing them can not bring remarkable improvement. More importantly, these two methods fail to stable improvements in encode-decoder architecture. This drives us to investigate the interference of the optimizations between CTC and cross-entropy. • Combination with the pre-training or multitask learning. Although our methods bring remarkable gains on both AR and NAR models, we do not explore the utilization of external data resources. Although we can use the pre-trained models directly, we expect more effective methods in future work. Theoretically, we need to design NAR ASR and MT models that share the same or similar architectures with the acoustic encoder and textual encoder, respectively. In this way, the NAST model bridges the gap between pre-training and fine-tuning and has more potential for better performance. • The potential risk for unwritten languages. In our work, we assume that transcription is always available, which is consistent with almost previous studies. Although some datasets have no transcription, we can use a well-trained ASR model to generate pseudo labels. However, it is hard to handle speech translation from unwritten source speech. The supervision of source text is very important for our model. Therefore, we need to develop better methods for stable training