SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Results on CVSS-C Fr→En and CVSS-T Fr→En test sets.  $\spadesuit$  indicates results quoted from Huang et al. [9].  $\spadesuit$  indicates results of our re-implementation.  $\dagger$ : target length beam=15 and noisy parallel decoding (NPD).  $T_{\mathrm{phone}}$ ,  $T_{\mathrm{unit}}$ , and  $T_{\mathrm{mel}}$  indicate the sequence length of phonemes, discrete units, and mel-spectrograms, respectively. \*\* means the improvements over S2UT are statistically significant ( $p < 0.01$ ).
ID	Models	Decoding	#Iter	#Param	ASR-BLEU (Fr→En)	Speedup
CVSS-C	CVSS-T
Ground Truth	/	/	/	84.52	81.48	/
Single-pass autoregressive decoding
A1	S2UT [5]	Beam=10	Tunit	73M	22.23	22.28	1.00×
A2	Translatotron [3]	Autoregressive	Tmel	79M	16.96	11.25	2.32×
Two-pass autoregressive decoding
B1	UnitY [7]	Beam=(10,1)	Tphone + Tunit	64M	24.09	24.29	1.43×
B2	Translatotron 2 [6]	Beam=10	Tphone + Tmel	87M	25.21	24.39	1.42×
Single-pass non-autoregressive decoding
C1♣	TranSpeech [9]	Iteration	5	67M	17.24	/	11.04×
C2♣	+ b=15 + NPD†	Iteration	15	67M	18.39	/	2.53×
C3♣	TranSpeech [9]	Iteration	5	67M	16.38	16.49	12.45×
C4♣	+ b=15 + NPD†	Iteration	15	67M	19.05	18.60	3.35×
Two-pass non-autoregressive decoding
D1	DASpeech	Lookahead	1 + 1	93M	24.71**	24.45**	18.53×
D2	(λ = 0.5)	Joint-Viterbi	1 + 1	93M	25.03**	25.26**	16.29×
D3	DASpeech	Lookahead	1 + 1	93M	24.41**	24.17**	18.45×
D4	(λ = 1.0)	Joint-Viterbi	1 + 1	93M	24.80**	24.48**	15.65×
Cascaded systems
E1	S2T + FastSpeech 2	Beam=10	Tphone + 1	49M+41M	24.71	24.49	/
E2	DAT + FastSpeech 2	Lookahead	1 + 1	51M+41M	22.19	22.10	/
E3	(λ = 0.5)	Joint-Viterbi	1 + 1	51M+41M	22.80	22.75	/
E4	DAT + FastSpeech 2	Lookahead	1 + 1	51M+41M	22.68	22.57	/
E5	(λ = 1.0)	Joint-Viterbi	1 + 1	51M+41M	23.20	23.15	/
the linguistic decoder using the speech-to-subword task, followed by pretraining on the speech-to-phoneme task. In the second stage of pretraining, the embedding and output projection matrices of the decoder are replaced and trained from scratch to accommodate changes in the vocabulary. We employ this pretraining strategy for DASpeech, UnitY and Translatotron 2 in the multilingual experiment. We learn the subword vocabulary with a size of 6K using the SentencePiece toolkit.
We also adopt the glancing strategy [25] during training, which shows effectiveness in alleviating the multi-modality problem for NAT. It first assigns target tokens to appropriate vertices following the most probable path  $\hat{A} = \arg \max_{A\in \Gamma}P_{\theta}(Y,A|X)$ , and then masks some tokens. We linearly anneal the unmasking ratio  $\tau$  from 0.5 to 0.1 during pretraining and fix  $\tau$  to 0.1 during finetuning.
Evaluation During finetuning, we save checkpoints every 2000 steps and average the last 5 checkpoints for evaluation. We use the open-source ASR-BLEU toolkit<sup>7</sup> to evaluate the translation quality. The translated speech is first transcribed into text using a pretrained ASR model. SacreBLEU [26] is then used to compute the BLEU score [27] and the statistical significance of translation results. The decoding speedup is measured on the test set using 1 RTX 3090 GPU with a batch size of 1.
Baseline Systems We implement the following baseline systems for comparison. More details about the model architectures and hyperparameters can be found in Appendix A.
- S2UT [5] Speech-to-unit translation (S2UT) model generates discrete units corresponding to the target speech with a sequence-to-sequence model. We introduce the auxiliary task of predicting target phonemes to help the model converge.
<sup>7</sup>https://github.com/facebookresearch/fairseq/tree/ust/examples/speech_to_speech/asn BLEU
- Translatotron [3] Translatotron generates the target mel-spectrogram with a sequence-to-sequence model. We also introduce the auxiliary task of predicting the target phonemes.
- UnitY [7] UnitY is a two-pass model which generates target phonemes and discrete units successively<sup>8</sup>. We remove the R-Drop training [28] for simplification. We pretrain the speech encoder and first-pass decoder on the S2TT task.
- Translatotron 2 [6] Translatotron 2 is a two-pass model which generates target phonemes and mel-spectrograms successively. We enhance Translatotron 2 by replacing LSTM with Transformer, and introducing an additional encoder between two decoders following Inaguma et al. [7]. The speech encoder and first-pass decoder are pretrained on the S2TT task.
- TranSpeech [9] TranSpeech is the first non-autoregressive S2ST model that generates target discrete units in parallel. To alleviate the acoustic multi-modality problem, TranSpeech introduces bilateral perturbation (BiP) to disentangle the acoustic variations from the discrete units. We re-implement TranSpeech following the configurations in the original paper.
- S2T + FastSpeech 2 The cascaded system that combines an autoregressive S2TT model and FastSpeech 2. The S2T model contains 12 Conformer layers and 4 Transformer decoder layers, which is also used in UnitY and Translatotron 2 pretraining.
- DAT + FastSpeech 2 The cascaded system that combines the S2TT DA-Transformer model and FastSpeech 2. Both models are used in DASpeech pretraining.
Table 1 summarizes the results on the CVSS-C  $\mathrm{Fr}\rightarrow \mathrm{En}$  and CVSS-T  $\mathrm{Fr}\rightarrow \mathrm{En}$  datasets. (1) Compared with previous autoregressive models, DASpeech (D1-D4) obviously surpasses single-pass models (A1, A2) and achieves comparable or even better performance than two-pass models (B1, B2), while preserving up to 18.53 times decoding speedup compared to S2UT. (2) Compared with the previous NAR model TranSpeech (C1-C4), DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant advantages in both translation quality and decoding speedup. (3) DASpeech obviously outperforms the corresponding cascaded systems (D1-D4 vs. E2-E5), demonstrating the effectiveness of our expect-path training approach. We also find that the cascaded model prefers larger graph size ( $\lambda = 1.0$  is better) while DASpeech prefers smaller graph size ( $\lambda = 0.5$  is better). We think the reason is that a larger graph size can improve S2TT performance, but it also makes end-to-end training more challenging. We further study the effects of the graph size in Appendix C. (4) On the CVSS-T dataset, which includes target speeches from various speakers, we observe a performance degradation in Translatotron and Translatotron 2 as the target mel-spectrogram becomes more difficult to predict. In contrast, DASpeech still performs well since its acoustic decoder explicitly incorporates variation information to alleviate the acoustic multi-modality, demonstrating the potential of DASpeech in handling complex and diverse target speeches.
Table 2 shows the results on CVSS-C dataset of the multilingual X→En S2ST model. We report the average ASR-BLEU scores on all languages, as well as the average scores on high/middle/low-resource languages<sup>9</sup>. We find that DASpeech still obviously outperforms S2UT but performs slightly worse than Translatotron 2 and UnitY in the multilingual setting, with an average gap of about 1.3 ASR-BLEU compared to Translatotron 2. However, DASpeech has about 13 times decoding speedup compared to Translatotron 2, achieving a better quality-speed trade-off.
Table 2: ASR-BLEU scores on CVSS-C test sets of the multilingual  $\mathrm{X}\rightarrow \mathrm{En}$  S2ST model.
Models	Avg.	High	Mid	Low
S2UT [5]	5.15	16.74	6.24	0.84
UnitY [7]	8.15	24.97	9.78	1.86
Translatotron 2 [6]	8.74	25.92	11.07	2.04
DASpeech (λ=0.5)	+ Lookahead	7.42	22.84	9.51	1.41
+ Joint-Viterbi	7.43	22.80	9.49	1.45
Table 3: ASR-BLEU scores on the CVSS-C  $\mathrm{Fr}\rightarrow \mathrm{En}$  test set with best-path training and expect-path training.
Models	Best	Expect	Δ
DASpeech (λ = 0.5)	+ Lookahead	24.45	24.71	+0.26
+ Joint-Viterbi	24.84	25.03	+0.19
DASpeech (λ = 1.0)	+ Lookahead	24.18	24.41	+0.23
+ Joint-Viterbi	24.46	24.80	+0.34
Note that the original UnitY uses subwords instead of phonemes. Here we use phonemes just for consistency with other systems.
The detailed results of each language pair can be found in Table 6 in Appendix B.
Figure 2: Translation latency of different models across different source speech lengths, categorized into 4 groups based on source speech frame counts. The translation latency is computed as the decoding time on 1 RTX 3090 GPU.
Figure 3: Trade-off between translation quality and decoding speed. X-axis represents the speedup ratio relative to S2UT, and Y-axis represents ASR-BLEU. The upper right represents better trade-off.
4.3 Alternative Training Approach: Best-Path Training
In addition to the expect-path training approach proposed in Section 3.2, we also experiment with a simpler approach: Best-Path Training. The core idea is to select the most probable path  $\hat{A} = \arg \max_{A\in \Gamma}P_{\theta}(Y,A|X)$  via Viterbi algorithm [20], and take the hidden states on path  $\hat{A} = (\hat{a}_1,\dots,\hat{a}_M)$  as input to the acoustic decoder, i.e.,  $\mathbf{z}_i = \mathbf{v}_{\hat{a}_i}$ . As shown in Table 3, best-path training also performs well but is inferior to expect-path training. We attribute this to the fact that when using best-path training, only hidden states on the most probable path participate in TTS training, which may result in insufficient training for the remaining hidden states. In contrast, all hidden states participate in TTS training with our expect-path training, which achieves
better performance. The time complexity of Viterbi algorithm used in the best-path training is also  $\mathcal{O}(ML^2)$ . More details about the best-path training can be found in Appendix E.
Table 4: Average speaker similarity on CVSS-T Fr→Ent test set.
Models	Speaker Similarity
Ground Truth	0.48
Unit-based S2ST
S2UT [5]	0.03
UnitY [7]	0.03
TranSpeech [9]	0.03
Mel-spectrogram-based S2ST
Translatotron [3]	0.04
Translatotron 2 [6]	0.05
DASpeech (λ = 0.5)	+ Lookahead + Joint-Viterbi	0.14 0.10
4.4 Analysis of Decoding Speed
In this section, we provide more detailed analysis of decoding speed. Figure 2 shows the translation latency of different models for speech inputs of different lengths. The results indicate that the translation latency of autoregressive models (S2UT, Translatotron, UnitY, and Translatotron 2) significantly increases with the length of the source speech. In contrast, the translation latency of non-autoregressive models (TranSpeech and DASpeech) are hardly affected by the source speech length. When translating longer speech inputs, DASpeech's decoding speed can reach more than 20 times that of S2UT. Furthermore, we illustrate the quality-speed trade-off of different models in Figure 3. By adjusting the hyperparameter  $\lambda$ , the translation quality and decoding latency will change. Specifically, as  $\lambda$  increases, the decoding latency of the model will increase, and it achieves the best translation quality when  $\lambda = 0.5$ . It is evident that DASpeech achieves the best trade-off between translation quality and decoding latency among all models. We further study the speedup under batch decoding in Appendix D.
4.5 Voice Preservation
In this section, we investigate the voice preservation ability of direct S2ST models on the CVSS-T  $\mathrm{Fr}\rightarrow \mathrm{En}$  dataset, where target speeches are in voices transferred from source speeches. Specifically, we use a pretrained speaker verification model[10] [29] to extract the speaker embedding of the source
10https://github.com/yistLin/dvector
speech and generated target speech. We define the cosine similarity between source and target speaker embeddings as speaker similarity, and report the average speaker similarity on the test set in Table 4. We find that: (1) unit-based S2ST model can not preserve the speaker's voice since discrete units contain little speaker information; and (2) DASpeech can better preserve the speaker's voice than Translatotron and Translatotron 2, since its acoustic decoder explicitly introduces variation information of the target speech, allowing the model to learn more complex target distribution.
Direct Speech-to-Speech Translation Speech-to-speech translation (S2ST) extends speech-to-text translation [30-33] which further synthesizes the target speech. Translatotron [3] is the first S2ST model that directly generates target mel-spectrograms from the source speech. Since continuous speech features contain a lot of variance information that makes training challenging, Tjandra et al. [34], Zhang et al. [35] use the discrete tokens derived from a VQ-VAE model [36] as the target. Lee et al. [5, 37] extend this research line by leveraging discrete units derived from the pretrained HuBERT model [38] as the target. To further reduce the learning difficulty, Inaguma et al. [7], Jia et al. [6], Chen et al. [39] introduce a two-pass architecture that generates target text and target speech successively. To address the data scarcity issue, some techniques like pretraining and data augmentation are used to enhance S2ST [8, 40-44]. Huang et al. [9] proposes the first non-autoregressive S2ST model which achieves faster decoding speed. Our DASpeech extends this line of research and achieves better translation quality and faster decoding speed.
Non-autoregressive Machine Translation Machine translation based on autoregressive decoding usually has a high decoding latency [45]. Gu et al. [10] first proposes NAT for faster decoding speed. To alleviate the multi-modality problem in NAT, many approaches have been proposed [46] like knowledge distillation [47-49], latent-variable models [50, 51], learning latent alignments [52-56], sequence-level training [57, 58], and curriculum learning [25]. Recently, Huang et al. [11] introduce DA-Transformer, which models different translations with DAG to alleviate the multi-modality problem, achieving competitive results with autoregressive models. Ma et al. [59], Gui et al. [60] further enhance DA-Transformer with fuzzy alignment and probabilistic context-free grammar.
Non-autoregressive Text-to-Speech Ren et al. [61], Peng et al. [62] first propose non-autoregressive TTS that generates mel-spectrograms in parallel. FastSpeech 2 [12] explicitly models variance information to alleviate the issue of acoustic multi-modality. Many subsequent works enhance non-autoregressive TTS with more powerful generative models like variational auto-encoder (VAE) [63, 64], normalizing flows [65–67], and denoising diffusion probabilistic models (DDPM) [68, 69]. DASpeech adopts the design of FastSpeech 2 for training stability and good voice quality.
In this paper, we introduce DASpeech, a non-autoregressive two-pass direct S2ST model. DASpeech is built upon DA-Transformer and FastSpeech 2, and we propose an expect-path training approach to train the model end-to-end. DASpeech achieves comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while maintaining up to  $18.53 \times$  speedup compared to the autoregressive model. DASpeech also significantly outperforms previous non-autoregressive model in both translation quality and decoding speed. In the future, we will investigate how to enhance DASpeech using techniques like pretraining and data augmentation.
7 Limitations & Broader Impacts
Limitations Although DASpeech achieves impressive performance in both translation quality and decoding speed, it still has some limitations: (1) the translation quality of DASpeech still lags behind Translatotron 2 in the multilingual setting; (2) the training cost of DASpeech is higher than Translatotron 2 (96 vs. 18 GPU hours) since it requires dynamic programming during training; and (3) the outputs of DASpeech are not always reliable, especially for some low-resource languages.
Broader Impacts In our experiments, we find that DASpeech emerges with the ability to maintain the speaker identity during translation. It raises potential risks in terms of model misuse, such as mimicking a particular speaker or voice identification spoofing.
We thank all the anonymous reviewers for their insightful and valuable comments.
A Details of Baseline Models
In our experiments, we implement five baseline systems using fairseq: S2UT, Translatotron, UnitY, Translatotron 2, and TranSpeech. We reproduce TranSpeech with their open-source implementations<sup>11</sup>. In this section, we mainly introduce the configurations of the other four baseline systems.
Figure 4 shows the model architectures of these models. In terms of model architecture, S2UT and Translatotron are single-pass S2ST models while UnitY and Translatotron 2 are two-pass S2ST models. In terms of predicted targets, S2UT and UnitY predict discrete units while Translatotron and Translatotron 2 predict mel-spectrograms. Below we describe the details of each model. The detailed hyperparameters can be found in Table 5.
(a) S2UT
(b) Translatotron
(c) UnitY
(d) Translatotron 2
Figure 4: Overview of baseline models.
S2UT Our implemented S2UT model includes three parts: a speech encoder, a linguistic decoder, and an acoustic decoder. The speech encoder is the same as DASpeech. The linguistic decoder is appended to the top layer of the speech encoder for multi-task learning, which predicts the target phonemes during training. The acoustic decoder generates the reduced discrete units derived from the 11-th layer of the pretrained mHuBERT model<sup>12</sup>. We do not include other auxiliary tasks and remove CTC decoding in Lee et al. [5] for simplification. The model is trained from scratch for 100k steps. We use beam search with a beam size of 10.
Translatotron The speech encoder and linguistic decoder of Translatotron are the same as S2UT. The acoustic decoder generates mel-spectrograms autoregressively. The pre-net dimension is 32 and the reduction factor of the acoustic decoder is 5. The model is trained from scratch for 100k steps.
UnitY UnitY is a two-pass model that includes four parts: a speech encoder, a linguistic decoder, a text-to-speech encoder, and an acoustic decoder. The architecture of the speech encoder, linguistic decoder, and acoustic decoder are the same as S2UT. The additional text-to-speech encoder is used to bridge the gap in representations between two decoders. We remove R-Drop training for simplification. We first conduct S2TT pretraining and finetune the model for 50k steps. We set the beam size of the first-pass and second-pass decoder to 10 and 1, respectively.
Translatotron 2 The model architecture of Translatotron 2 is similar to UnitY except that the second decoder generates mel-spectrograms rather than discrete units. The reduction factor of the acoustic decoder is set to 5. We first conduct S2TT pretraining and finetune the model for 50k steps. The beam size is set to 10 for the first-pass decoder.
For all the above models, we save checkpoints every 2000 steps and average the last 5 checkpoints for evaluation, which is the same as DASpeech. For S2UT and UnitY, we use the pretrained unit-based HiFi-GAN $^{13}$  vocoder to synthesize waveform. For Translatotron and Translatotron 2, we use the same pretrained HiFi-GAN vocoder as DASpeech.
11https://github.com/Rongjiehuang/TranSpeech 12https://dl.fbaipublicfiles.com/hubert/mhubert_base_vp_en(es_fr_it3_ L11_kml000.bin 13https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_ hifigan/mhubert_vp_en(es_fr_it3_400k_layer11_kml000_lj/g_00500000
Table 5: Hyperparameters of DASpeech and baseline models.
Hyperparameters	S2UT	Translatotron	UnitY	Translatotron 2	DASpeech
Speech Encoder	conv_kernel_sizes	(5, 5)	(5, 5)	(5, 5)	(5, 5)	(5, 5)
encoder_type	conformer	conformer	conformer	conformer	conformer
encoder_layers	12	12	12	12	12
encoder_embedding_dim	256	256	256	256	256
encoder_ffn_embedding_dim	2048	2048	2048	2048	2048
encoder attentions_heads	4	4	4	4	4
encoder_pos_enc_type	relative	relative	relative	relative	relative
depthwise_conv_kernel_size	31	31	31	31	31
Linguistic Decoder	decoder_layers	4	4	4	4	4
decoder_embedding_dim	512	512	512	512	512
decoder_ffn_embedding_dim	2048	2048	2048	2048	2048
decoder attentive_heads	8	8	8	8	8
label_smoothing	0.1	0.1	0.1	0.1	0.0
s2t_loss_weight	8.0	0.1	8.0	0.1	1.0
Text-to-Speech Encoder	encoder_layers	-	-	2	2	-
encoder_embedding_dim	-	-	512	512	-
encoder_ffn_embedding_dim	-	-	2048	2048	-
encoderattention_heads	-	-	8	8	-
Acoustic Decoder	decoder_layers	6	6	2	6	8
decoder_embedding_dim	512	512	512	512	256
decoder_ffn_embedding_dim	2048	2048	2048	2048	1024
decoder attentive_heads	8	8	8	8	4
label_smoothing	0.1	-	0.1	-	-
n_frames_per_step	1	5	1	5	1
unit_dictary_size	1000	-	1000	-	-
var_pred Hidden_dim	-	-	-	-	256
var_pred_kernel_size	-	-	-	-	3
var_pred_dropout	-	-	-	-	0.5
s2s_loss_weight	1.0	1.0	1.0	1.0	5.0
Training	lr	1e-3	1e-3	1e-3	1e-3	1e-3
lr_scheduler	inverse_sqrt	inverse_sqrt	inverse_sqrt	inverse_sqrt	inverse_sqrt
warmup Updates	4000	4000	4000	4000	4000
warmup_init_lr	1e-7	1e-7	1e-7	1e-7	1e-7
optimizer	Adam	Adam	Adam	Adam	Adam
dropout	0.1	0.1	0.1	0.1	0.1
max_tokens	40k×4	40k×4	40k×4	40k×4	40k×8
weight Decay	0.0	0.0	0.0	0.0	0.01
clip_norm	1.0	1.0	1.0	1.0	1.0
max_update	100k	100k	50k	50k	50k
B Detailed Results on CVSS-C X→En Datasets
Table 6 summarizes the detailed results of each language pair on CVSS-C test sets of the multilingual  $\mathrm{X}\rightarrow \mathrm{En}$  S2ST models.
Table 6: Results on CVSS-C test sets of the multilingual X→En S2ST models.
Models	Avg.	High	Fa	It	Mid Ru	Zh	Pt
Fr	De	Ca	Es
S2UT [5]	5.15	19.65	13.35	15.37	18.58	1.43	14.47	7.94	0.93	6.42
UnitY [7]	8.15	27.27	20.81	24.22	27.58	3.63	21.68	10.86	4.16	8.56
Translatotron 2 [6]	8.74	28.04	21.54	25.34	28.77	4.23	23.66	13.41	4.49	9.54
DASpeech (λ = 0.5)	+ Lookahead	7.42	25.43	17.87	22.58	25.49	3.01	20.80	12.96	2.86	7.90
+ Joint-Viterbi	7.43	25.39	18.36	22.33	25.10	2.81	20.76	12.94	3.05	7.89
Models	Low
Nl	Tr	Et	Mn	Ar	Lv	Sl	Sv	Cy	Ta	Ja
S2UT [5]	4.67	0.52	0.36	0.14	0.56	0.39	0.73	1.28	0.66	0.17	0.20
UnitY [7]	10.60	3.79	1.07	0.12	0.78	1.50	0.81	1.38	1.74	0.10	0.15
Translatotron 2 [6]	11.17	4.58	1.12	0.32	1.35	1.37	0.93	1.49	1.50	0.10	0.22
DASpeech (λ = 0.5)	+ Lookahead	9.04	1.75	0.04	0.08	0.64	1.43	1.20	1.33	0.70	0.09	0.29
+ Joint-Viterbi	9.43	1.66	0.07	0.08	0.48	1.48	1.30	1.30	0.85	0.09	0.31
C Effects of the Graph Size
In this section, we investigate how the graph size affects the performance. We vary the size factor  $\lambda$  from 0.25 to 1.5, and measure the translation quality of both the S2TT DA-Transformer model and DASpeech on the CVSS-C Fr→En test set. As shown in Figures 5 and 6, we observe that the performance of S2TT DA-Transformer keeps increasing as the graph size gets larger, which is consistent with the observations in machine translation [11, 59]. However, DASpeech performs best at  $\lambda = 0.5$  and shows a performance drop at larger  $\lambda$ . We speculate that this is because larger graph size makes end-to-end training more challenging. We will investigate this issue in the future.
Figure 5: Phoneme-level BLEU scores of the S2TT DA-Transformer under different size factor  $\lambda$ .
Figure 6: ASR-BLEU scores of DASpeech under different size factor  $\lambda$ .
D Speedup Under Batch Decoding
As Gu and Kong [46] pointed out, the speed benefits of non-autoregressive models may degrade during batch decoding. To better understand this problem, we evaluate the speedup ratio under different decoding batch sizes. As shown in Figure 7, the speedup ratio keeps dropping as the decoding batch size increases. Nevertheless, DASpeech ( $\lambda = 0.5$  with Joint-Viterbi decoding) still achieves more than  $6 \times$  speedup with a decoding batch size of 64 and maintains comparable performance with Translatotron 2.
Figure 7: Speedup ratio compared to S2UT baseline (not shown in the figure) and ASR-BLEU score on the CVSS-C  $\mathrm{Fr}\rightarrow \mathrm{En}$  test set with different batch decoding sizes  $(b\in \{1,8,16,32,64\})$
E Best-Path Training