Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation
Year: 2023

Abstract:
Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in endto-end speech translation (E2E ST). The commonplace “modality gap” between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning. However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance. On the other hand, we find that there has another gap, which we call the “capacity gap”: high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the overfitting. In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for enfr on the MuST-C dataset. Code and models are available at https://github.com/hannlp/TAB.

heading: 1 Introduction
text: End-to-end speech translation (E2E ST) employs a direct model to translate source language speech into target language text, which has low latency and can avoid the “error propagation” problem in traditional cascade methods (Weiss et al., 2017). However, compared to automatic speech recognition (ASR) and machine translation (MT) models used in cascade methods, E2E ST models typically have limited training data (Cattoni et al., 2021), which can result in sub-optimal performance.
Transferring the knowledge from the related tasks (e.g. ASR and MT) is a widely-used approach for E2E ST to achieve optimal performance (Tang et al., 2021; Zhang et al., 2022a). However, the difference between tasks and data makes the transfer process more challenging (Wang et al., 2020). The inconsistency of length and representation between speech and text leads to the “modality gap”
∗Corresponding author.
(Liu et al., 2020), which exists in scenarios where the inputs of the model change, such as in the pre-training and fine-tuning (PT-FT) paradigm (Xu et al., 2021) or in the multi-task learning (MTL) methods (Ye et al., 2021). Thus, the connectionist temporal classification (CTC) (Graves et al., 2006) based adapters (Liu et al., 2020; Xu et al., 2021) have been proposed to transform the original speech output into a text-like sequence. Recently, consistency training methods have achieved promising results by using a better branch, such as the mix-up branch (Fang et al., 2022) or the text branch (Cheng et al., 2022), to promote crossmodal learning and support the original speech output. However, we find that the “modality gap” does not exist throughout the training process in Figure 1. While consistency training methods have some regularization function (Liang et al., 2021; Guo et al., 2022) to help the model overcome over-fitting and be fully trained. A natural question arises: Are modality adaption methods always effective when an E2E ST model is fully trained?
1340
In this work, we aim to investigate how much of the improvement is due to the modality adaption or regularization methods. To achieve this, we adopt the PT-FT and encoder decouple paradigm and establish a framework that incorporates adjustable modality adaption and consistency training methods. Through extensive experiments on the MuSTC en-de and en-fr benchmarks, we observe that:
• The modality adaption method in PT-FT only accelerates the early phase of fine-tuning, but does not provide a significant improvement for a fully trained model.
• We obtained 29.0 and 40.3 on the MuST-C en-de and en-fr datasets, but regularization played a major role, which confirming that the “capacity gap” is more severe than the “modality-gap” in E2E ST.

heading: 2 Our Case Study: TAB

heading: 2.1 Architecture
text: The E2E ST corpus DST usually consists of three parts: the source language speech s, the corresponding transcripts x, and the target translation y. The overall framework, as shown in Figure 2, consists of a speech encoder and a shared transformer. Speech Encoder. The speech encoder encodes the source language speech s into speech output h:
h = ENCspeech(s; θs) (1)
We employ the CTC loss at the top of the speech encoder (Wang et al., 2020; Liu et al., 2020; Xu et al., 2021) to predict an alignment path π by speech output h based on a conditionally independent assumption p(π|h) = ∏|h|t p(πt|ht):
p(πt|ht) = Softmax(Linear(ht; θctc)) (2)
where πt ∈ V+, which is an extended source language vocabulary by introducing a “blank” token. The path π can be mapped to the transcript x by removing all repeated labels and blanks, such an operation is called β. The CTC loss is defined as the negative log probability of all possible alignment paths β−1(x) between h and x:
LCTC = − ∑
π∈β−1(x) logp(π|h) (3)
Shared Transformer. The shared transformer accepts two inputs: the original branch o and the auxiliary branch a, both of equal length. We leave the
details of these to the Section 2.2. The encoder and decoder of the shared transformer are utilized to obtain the final predictions Pj = p(yj |y<j, o; θt) and Qj = p(yj |y<j, a; θt), respectively. The cross-entropy losses are then calculated as follows:
LCEo = − ∑|y|
j=1 logPj (4)
LCEa = − ∑|y|
j=1 logQj (5)

heading: 2.2 Tuning with Auxiliary Branch (TAB)
text: The speech encoder and shared transformer are initialized with pre-trained ASR and MT models. During fine-tuning, we aim to build a text-like auxiliary branch which includes some textual representations like Fang et al. (2022); Zhang et al. (2022b) for modality adaption, and provide an adjustable probability to control the degree of it. To obtain the textual embedding, we utilize the ability of the CTC alignment, where πt = argmax(p(πt|ht)) is an id of V+ that denotes the corresponding CTCpredicted token of the speech feature ht. Shrink. To eliminate the effect of too many “blank” positions in the sequence, we first average the consecutively repeated features (e.g. πi = ... = πj = ci→j) in the speech output h to obtain the original branch o, where ok = (hi + ...+ hj) · 1j−i . Copy & Replace. We copy o to a new sequence a to ensure that the auxiliary branch has the same length as the original branch. Each position ak in the new sequence is then replaced with its CTC predicted embedding Embedding(ci→j) with a probability p∗ if ci→j is not a “blank”. Here, Embedding is an embedding matrix initialized by the pre-trained MT model. The replacement probability p∗ can be adjusted to control the degree of modality adaption, which can be a fixed or a dynamic value, as discussed in Section 4.3. It is important to note that the auxiliary branch provides a regularization function due to the replacement operation or dropout (Liang et al., 2021). This effect will be further discussed in Section 4.4. Fine-tuning strategy. To utilize the auxiliary branch, a consistency loss is introduced to enforce consistency between two output distributions:
LCons = ∑|y|
j=1 D(Pj ,Qj) (6)
where D denotes the loss term. The final loss used in TAB is formulated as follows:
L = LCEo + LCEa + λLCTC + αLCons (7)

heading: 3 Experimental Setup
text: Datasets and Pre-processing. We conducted our experiments on the MuST-C (Gulati et al., 2020) dataset for two language directions: En-De and EnFr. The dev set was used for validation, and the tstCOMMON set was used for reporting our results. For training the English ASR model, we used the LibriSpeech (Panayotov et al., 2015) dataset. The WMT16 En-De and WMT14 En-Fr datasets were used to train the MT models. Table 1 presents the statistics of all the datasets used in our pre-training and fine-tuning processes.
We preprocessed the speech input by extracting 80-dimensional log-mel filterbank features and removing utterances with more than 3,000 frames. The vocabulary, which has a size of 10k, is shared between the source and target languages and was trained using the SentencePiece (Kudo and Richardson, 2018) model from the MuST-C dataset. Model Configuration. All experiments were implemented using the fairseq toolkit (Ott et al., 2019). Two convolutional layers with a stride of 2 were introduced to downsample the input speech features. We used the Conformer (Gulati et al., 2020) as our speech encoder, which consists of 12 layers. Both the text encoder and decoder in the shared trans-
former have 6 layers. Each layer in our model has 512 hidden units, 2048 feed-forward size, and 8 attention heads. The ASR and MT models were pre-trained with external data and fine-tuned with the MuST-C dataset. Training and Inference. We used the Adam optimizer with β1 = 0.9 and β2 = 0.997 in MT, while β2 = 0.98 in ASR and ST, respectively. During ASR pre-training, each batch has up to 800k frames, and the learning rate and warmup were 1.4e-3 and 10000. During MT pre-training, each batch has up to 33k tokens, and the learning rate and warmup were 1e-3 and 8000. During ST fine-tuning, each batch has up to 320k frames, and the learning rate and warmup were 7e-4 and 4000. The hyper-parameter λ was set to 0.3 for both pretraining and fine-tuning. We used dropout with a ratio of 0.1 during pre-training and 0.15 during fine-tuning, and label smoothing with a value of 0.1. All training was stopped early if the loss (ASR and MT) or BLEU (E2E ST) on the dev set did not improve for twenty epochs. During inference, we averaged the model parameters of the best 10 checkpoints for evaluation. We used a beam search with a beam size of 5 for all models. We reported the case-sensitive SacreBLEU (Post, 2018). All models were trained on 4 Titan RTX GPUs.

heading: 4 Results and Discussion

heading: 4.1 Which type of consistency loss is best?
text: The choice of an appropriate consistency loss is crucial in leveraging the knowledge from the auxiliary branch, whether it is due to modality adaption or regularization. We conducted experiments with different loss terms with α = 1 and p∗ = 0.2.
Loss term BLEU dev tst-COMMON avg. None (α = 0) 27.69 27.99 27.84 JSD 27.76 28.49 28.13 KLorig→aux 28.47 28.49 28.48 KLaux→orig 28.26 28.78 28.52 bi-KL 28.43 28.78 28.61

heading: 2000 4000 6000 8000
text: Training Steps
Figure 3: The ratios of the auxiliary branch loss to the original branch loss ℓaux/ℓorig at each training step under different fixed values of p∗. Additionally, we show the uncertainty of the original branch υorig (represented by the blue line) at the same time.
As shown in Table 2, the results indicate that a consistency loss is necessary to improve performance. The Jensen-Shannon Divergence (JSD) loss and the unidirectional-KL loss were found to be slightly worse than the bidirectional-KL (bi-KL) loss. Therefore, we selected the bi-KL loss for the subsequent experiments.

heading: 4.2 Whether & when the modality gap exist?
text: In Figure 3, we present the ℓaux/ℓorig curve of finetuning, which represents the ratio of the auxiliary branch loss ℓaux to the original branch loss ℓorig. The only difference between the two branches is the replaced textual embeddings for modality adaption in the auxiliary branch. We investigate the effect of this operation during fine-tuning under different replacement probabilities.
Our results show that, in the early fine-tuning phase, the ℓaux always lower than ℓorig when p∗ > 0, indicating that the model expects some textual representations like the input in pre-training. However, in the later fine-tuning process, the ℓaux is slightly higher than the ℓorig, suggesting that the
model starts to get confused about replacement operations due to the introduction of some kind of noise by the destruction of the original sequence. Moreover, we find that the maximum p∗ = 1.0 always has the lowest ratio at the beginning and the highest ratio at other times. This confirms that the modality gap exists but not throughout the entire fine-tuning process.

heading: 4.3 Does the modality adaption help?
text: We experimented with different fixed replacement ratios p∗ over the range [0.0, 0.2, 0.6, 1.0] under α = 1.0 in Figure 3. Our results showed that the method with modality adaption p∗ > 0 consistently outperformed p∗ = 0. Furthermore, as observed in Section 4.2, there exists a “tipping point” in the finetuning process where the modality gap disappears. Before this point, we should use a higher p∗, while a lower p∗ should be more effective after this point, rather than using a fixed value. We discovered that the uncertainty of the original branch υorig, which is defined by the normalized entropy of Pj 1, is strongly related to this point, as shown in Figure 3:
vorig = − 1 logV · 1|y| ∑|y| j=1 Pj logPj (8)
where V is the vocabulary size. We then proposed a dynamic replacement probability derived from υorig at each step: p∗ = γ·υorig, where γ is a hyperparameter set to 0.5 in all experiments. When we use a dynamic replacement ratio of p∗ = 0.5 ·υorig, we denote it as p∗ = υ. By adopting this dynamic replacement ratio, we achieved a BLEU score of 28.87 on the MuST-C en-de dataset.

heading: 4.4 Is modality adaption always effective?
text: The consistency loss has been demonstrated to have a regularization effect on two branches with different outputs (Liang et al., 2021; Guo et al., 2022). When p∗ = 0, there is no modality adaption through the replacement operation, but the dropout still causes different outputs for the two branches, the TAB degenerates into a pure regularization method. The hyper-parameter α can be used to control the degree of regularization, where a higher α indicates stronger regularization. By increasing α to 5, it is observed that the gap between the modality adaption method (p∗ = 0.2 or p∗ = υ) and the pure regularization method
1More precise definitions are Pj(t) and υorig(t), with the symbol of the training step “t” omitted for brevity.
(p∗ = 0) decreases (29.05 vs 29.01) in Figure 4, although the pure regularization method always required more epochs to converge, which can be attributed to more complete training. These findings confirm that the modality adaption method on PT-FT can accelerate the early phase of fine-tuning, but does not significantly enhance the final performance when a sufficient level of regularization has been applied to the model. This also highlights the issue of over-fitting being a more serious problem in E2E ST than the “modality gap”, and that better regularization and a longer time for fine-tuning can help to eliminate the “modality gap” problem in the PT-FT paradigm.

heading: 4.5 Final results and comparison of methods
text: The results for the MuST-C dataset are shown in Table 3. The modality adaption method shows an improvement of 1.0 and 0.7 BLEU points over our CTC baseline at a lower consistency level (α = 1). However, the pure regularization method with α = 5 slightly outperforms it (+0.1 BLEU), and outperforms all other methods designed for modality adaption (+0.1 ∼ 1.2 BLEU), except those using HuBERT as the feature extractor. When we combined our modality adaption method with a higher level of consistency (α = 5), further improvement can still be achieved, but not consistently in all languages. Our hypothesis is that the replacement operation in TAB not only releases the modality gap in the early fine-tuning phase but also introduces noise in the later stages. This noise can bring better regularization performance (Guo et al., 2022; Gao et al., 2022) when a higher con-
sistency level is given. In general, regularization brings more improvement in TAB than modality adaption, and a better regularization is helpful for E2E ST models to be fully trained.

heading: 5 Conclusion
text: Through a case study, we have demonstrated that the “modality gap” in the PT-FT paradigm for E2E ST is only present in the early stages of fine-tuning. Although a modality adaption method can accelerate the convergence speed, it does not significantly improve the final performance of a fully trained model. However, the over-fitting and “capacity gap” are more critical issues in E2E ST, and a good regularization technique can help in fully training the model.

heading: Acknowledgement
text: The authors would like to thank anonymous reviewers for their insightful comments. This work was supported in part by the National Science Foundation of China (No. 62276056), the National Key R&D Program of China, the China HTRD Center Project (No. 2020AAA0107904), the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01), the Yunnan Provincial Major Science and Technology Special Plan Projects (No. 202103AA080015), the Fundamental Research Funds for the Central Universities (Nos. N2216016, N2216001, and N2216002), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No. B16009).
Limitations
Although our work has achieved high performance, there are still some limitations that need to be addressed in future work:
• Our work was only carried out under the assumption that there is sufficient ASR and MT data, and relied on transcriptions for CTC loss to perform alignment and predict. This assumption may not hold in real-world scenarios, and we plan to investigate the performance of our approach under more diverse data conditions in the future.
• We only attempted to feed two branches into the shared transformer in order to ensure a fair comparison between the pure regularization method and the modality adaption method. However, this approach may have resulted in sub-optimal regularization performance compared to methods that feed all branches into the whole model, as demonstrated by Liang et al. (2021); Guo et al. (2022); Gao et al. (2022).

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Limitations
Although our work has achieved high performance, there are still some limitations that need to be addressed in future work:
• Our work was only carried out under the assumption that there is sufficient ASR and MT data, and relied on transcriptions for CTC loss to perform alignment and predict. This assumption may not hold in real-world scenarios, and we plan to investigate the performance of our approach under more diverse data conditions in the future. • We only attempted to feed two branches into the shared transformer in order to ensure a fair comparison between the pure regularization method and the modality adaption method. However, this approach may have resulted in sub-optimal regularization performance compared to methods that feed all branches into the whole model, as demonstrated by Liang et al. (2021); Guo et al. (2022); Gao et al. (2022)