Consistency is Key: On Data-Efficient Modality Transfer in Speech Translation
Authors: Hojin Lee
Changmin Lee
seung-won hwang
Venue: EMNLP 2023 Findings (2023)
Keywords: Speech Translation
Cross-modal Transfer
Efficient Training

Abstract: End-to-end approaches have shown promising results for speech translation (ST), but they suffer from its data scarcity compared to machine translation (MT). To address this, progressive training has become a common practice, of using external MT data during the fine-tuning phase. Despite of its prevalence and computational overhead, its validity is not extensively corroborated yet. This paper conducts an empirical investigation and finds that progressive training is ineffective. We identify learning-forgetting trade-off as a critical obstacle, then hypothesize and verify that consistency learning (CL) breaks the dilemma of learning-forgetting. The proposed method, which combines knowledge distillation (KD) and CL, outperforms the previous methods on MuST-C dataset even without additional data, and our proposed consistency-informed KD achieves additional improvements against KD+CL. Code and models are availble at https://github.com/hjlee1371/consistency-s2tt.

Consistency is Key:
On Data-Efficient Modality Transfer in Speech Translation
Hojin Lee
Changmin Lee
Seung-won Hwang
Kakao Brain
Kakao Brain
Seoul National University
lambda. wakaaddbbaaaaacccccnccooo com
louie. cm@kakaabbain..com com
seungwonhesnu.acc ac.
Abstract
Exclusion of external MT leads to MT forgetting
End-to-end approaches have shown promis- ing results for speech translation (ST), but they suffer from its data scarcity compared to machine translation (MT). To address this, progressive training has become common practice, of using external MT data during the fine-tuning phase. Despite of its preva- lence and computational overhead, its valid- ity is not extensively corroborated yet. This paper conducts an empirical investigation and finds that progressive training is ineffective. We identify leoreing- trade-off' as a critical obstacle, then hypothesize and ver- ify that consistency learning (CL) breaks the dilemma of searning-forgettin The proposed method, which combines knowledge distilla- tion (KD) and CL., outperforms the previous methods on MuST-C dataset (Di Gangi et al., 2019) even without additional data, and our pro- posed consistency-informn KD achieves ad- ditional improvements against KD+CL. Code and models availble //gitt. com/hjleel 371 /consistency-s2tt.
Inclusion of external MT leads to ST-MT interference
Figure 1: Schematic diagram 'lllustrating the intrinsic tension in adding MT data for ST training. When ex- ternal MT data added during finetuming (in red), the model experiences ST-MT interference. Conversely, when less external MT data is incorporated (in blue), model tends to forget MT knowledge, resulting suboptimal cross-modal transfer.
the widely used progressive training technique ST (Tang et al., 2021b,a: Ye et al., 2022; Tang et al.. 2022; Ye et al., 2021), where external MT data is continuously integrated during the fine-tuning phase. However, our evaluation brings forth sur- prising results, find that progressive training inadequate and leads suboptimal outcomes. This inadequacy is due to the dilemma in adding MT data for ST training. Using external MT data may ineur interference between the two tasks, but having less external MT data leads forgetting.
1 Introduction
While traditional speech-to-text translation (ST) systems are built by pipelining automatic speech recognition (ASR) and machine translation (MT), end-to-end (E2E) approach recently emerges as promising direction ameliorate error propagation and model complexity problems (Anastasopoulos et al., 2022; Bentivogli et al., 2021). However, E2E ST models encounter data scarcity due to the need for cross-modal annotations, which are less abundant compared to datasets used in related tasks such machine translation.
To break the knot in Figure 1, we first shed light on the overlooked relationship between consistency and forgetting. In addition, introduce novel approach called consistency-informed knowledge distillation (cKD). Our findings and proposed meth- ods are thoroughly evaluated on the MuST-C bench- mark, encompassing various language pairs. The results demonstrate the superior performance and enhanced data efficiency of our approach compared to previous methods.
Our goal to enable effective cross-modal trans- fer from machine translation (MT) models, which have ample training data, to ST models with lim- ited data. In pursuit of this goal, we investigate
"Corresponding author 'Image from PresentationGO
Our contributions are as follows.

---

We reevaluate the validity of the widespread use of progressive training, and find that is ineffective.
LKD is weighted by axD and added the final loss.
Progressive training is ineffective Despite its widespread adoption, we propose that the efficacy of progressive training has been accepted without sufficient empirical evidence and followings high- light our empirical findings against common be- liefs. From Table 1, we can see that progressive training does not improve the ST performance, de- spite expensive computational overhead, contrary to the popular belief. For deeper understanding, also evaluate its MT performance throughout the training using transcription-traniiee translation pair of ST triplet. As depicted in Figure 2, we observe catastrophic forgetting in MT performance when we train ST model without Dext, while it preserves its MT knowledge by training with Dext-
We find that consistency learning (CL) reme- dies the catastrophic forgetting problem, and simple combination of knowledge distillation (KD) and CL achieves promising ST BLEU scores with simple design choice and data- efficient training.
We further push the limit of KD+CL through proposing musistencyyinformee KD (cKD), which utilizes token-level consistency for adaptive weighting of KD.
2 Motivation: Is Progressive Training Effective?
Problem statement and Baseline To motivate, we define our problem and describe progressive learning baseline by Ye et al. (2021), that we use as strong baseline throughout the paper.
Based our observation of catastrophic forget- ting, it might be expected that cross-modal KD would benefit from progressive training, as aug- mented MT data provides a more reliable teacher signal. However, the inclusion of an extra KD objective in Table does not yield significant im- provements. This raises an important question: why does addressing catastrophic forgetting not lead to improved ST performance? It appears that while Dext mitigates forgetting, it also diverts the model's focus from ST, highlighting the inherent tension between learning ST and avoiding MT for- getting.
The speech translation (ST) corpus, denoted as DST = {(ss x, }) consists s (source language speech), x (source language text or transcription), and y (target language text or translation).
Due to the scarcity of speech translation datasets, it is common practice to train ST models jointly with MT ASR subtasks (Tang et al., 2021a,b; Ye train
al., 2021). Similarly, in our approach, our model jointly on ST and MT using multitask cross-cntropy losses, denoted as (C y, and Lc
In progressive training, the MT training is con- tinued during the fine-tuning phase using external data source Dext= y)). Specifically, at each epoch, Dext is randomly downsampled to D and during training, ST triplets (s, y)) MT pairs y) are sampled from the union of Dg and D'xx
Lang.
Models
DSt
+D ext
p-value
De
Baseline
28.07
28.07
0.4145
Es
Baseline
31.35
31.30
0.2686
Fr
Baseline
37.94
38.04
0.1600
De
Base+KD 28.18
28.23
0.2456
Es
Base+KD)
31.52
31.37
0.1186
Fr
Base+KD
38.28
38.33
0.2541
In addition to joint training, we incorporate MT- to-ST online knowledge distillation (KD) proposed by Tang et al. (2021a). For the data triplet (s. (s the KD loss is computed as:
Table 1: ST BLEU results with the baseline and KD training for various translation directions. Dext ad- ditional MT data from external source(e.g. WMT) other than ST triplet datasets.
lyl |
CKD =
P(u = 1yly,,
j=
3 Proposed: Effective Cross-modal Transfer with Consistency Learning and Consistency-iffommed KD
log P(y = vylyie,
where by corresponds to the j-th token of the vo- cabulary. This encourages the ST "student" to learn more finengrained information from the MT "teacher". When combined with baseline systems,
Continual Learning View of Speech Translation ST learning can be viewed from a continual learn- ing perspective, where knowledge is continuously

---

34
ting, hypothesized, we can expect substantial improvements by incorporating cross-modal KD compared to not using it. To achieve further en- hancements, we can go beyond the basic combina- tion of KD and CL. In particular, we introduce a novel loss called consisteney aunistency-informed which leverages more detailed information on con- sistency, as outlined below.
33
BLEE
32
informedd KD (cKD),
31
ST-anly ext-MTT
ST-only CL
30
50000
100000
150000
200000
Steps
ly
LeKD =
P(y = uyly,, x;
Figure 2: Evolution of MT BLEU with transcription- translation pairs from MuST-C En-De dev set.
j=
log P(y -ou=-ulliese
uslyic, s:
accumulated through pre-trained MT, MT-, and ST-finetuning. From this standpoint, progressive training can be interpreted as the generalization rehearsal (Robins, 1995). However, as high- lighted in studies on continual learning (Verwimp et al., 2021) and multitask learning (Lin et al., 2019; Sener and Koltun, 2018), bare replay of previous tasks is not the answer to the stability-pha plaasticiyy dilemma: it only offers suboptimal balance be- tween MT and ST. Therefore, we turn our focus to regularization- based methods such as elastic weight consolidation (EWC) and its variants (Kirk- patrick et al., 2017; Chaudhry et al., 2018; Schwarz et al., 2018; Thompson et al., 2019).
It augments vanilla KD with token-level weighting based on MT consistency matrix cMV Concretely, MT represents the bidirectional KI divergence Cij between two forward pass probabilities for i-th token and j-th vocabulary: (1 uy mylyic,> x) and P2 uylVic: x). Intuitively, cKD can be understood as ignoring inconsistent MT teacher probabilitics at token-level.
4 Results and Analysis
Consistency learning remedies catastrophic for- getting We begin by examining the hypothesis that CL data-efficiently remedies catastrophic for- getting. As illustrated in Figure 2, CL demonstrates remarkable ability to retain MT knowledge even in the absence of additional MT data, thereby con- firming our hypothesis. The final ST BLEU scores, presented in Table 3, further support these findings. Surprisingly. progressive training consistently un- derperforms in all language directions. This sug- gests that progressive training becomes redundant in the presence of CL, as it loses the benefits of preserving MT knowledge while still diverting the models' attention away from ST.
Their main idea is to restrict the gradient of new tasks within the low-curvature "valleys" of the loss landscape of previous task. While traditional EWC approximates the Hessian as the diagonals of the Fisher information matrix Σ, (0, 2,),, we can remove diagonal approximation from EWC through well-known relation between KL. diver- gence and Fisher as
D (KI(po-lleg) (pp)) ~ E, Po -log Po]
(2)
/ 0*)
CL provides a more reliable teacher for KD data-efficient solution for catas- trophic forgetting, we can confidently predict larger gains from knowledge distillation (KD), as ex- plained in 3. To empirically demonstrate this, we train our ST model using simple combination of KD and CL. The results in Table 3 clearly show that this approach leads to greater improvements in all language directions, surpassing the performance of progressive training by significant margin. Furthermore, we observe that the performance gain from KD is more pronounced when combined with CL compared to KD alone (+0.29 BLEU with
While 8* is fixed parameter obtained from prior tasks in original EWC, we can interpret it as cur- rent parameter and 0 = + AO as the parameter of submodel induced by arbitrary perturbations. It recalls recently proposed CL losses such as R- Drop (Liang et al., 2021). 1), and provokes the ques- ameliorate
Thanks
tion that whether the CL framework the intrinsic dilemma of ST. We thus adopt R R-drop our CL framework, and weighted loss aCtCoL is added to the final loss.
Consistency-informed KD the concept of con- sistency is indeed crucial for preventing MT forget-

---

FT Data
Languages Es
Models
Joint PT
Avg.
ST
MT
De
Fr
(Induri al., 2021)
28.88
SpeechT5(Ao et al., 2022)
25.18
35.30
(Tan' et al., 2022)
29.28
33.1
39.7
34.0
SpeachUT et al., 2022)
30.1
33.6
41.4
35.0
JT-S- -T-S-M et ol., 2021a)
26.8
31.0
37.4
31.7
XSTNet(Ye al., 2021)
27.8
30.8
38.0
32.2
Chimera(Han et al., 2021)
27.1
30.6
35.6
31.1
SATE(Xu al., 2021)
28.1
SFEn al., 2022)
28.7
31.0
37.4
32.4
WACO(Ouyang et al., 2022)
28.1
32.0
38.1
32.7
AdaTrans(Zeng et al., 2022)
28.7
38.7
ConST(Ye et al., 2022)
28.3
32.0
38.3
32.8
Ours(Base+KD+CII
29.08
32.13
39.47
33.56
Ours(Basc+cD++CL)
29.27
32.32
39.51
33.70
Table 2: ST BLEU scores MuST-C tst-COMMON for various methods. use OpenSubtitles (Lison and Tiedemann, 2016) for Dext and use additional data augmentation. & is from corresponding github implementations, not from the papers. and ** indicates statistically significant differences between (Base+KD+CL) and (Base+cKD+CL) (p 0.1 and p < 0.05).
Lang.
Models
DSt
+Dext
p-valuc
icantly more data, training complexity, and com- putational resources?. Despite this, our method performs on par with most of the JPT approaches, indicating ample opportunity for further research in developing lightweight strategies for patching modality-specine pretrained models.
**
Base+CL
28.94
28.57
0.0043
Es***
Base+CL
31.83
31.18
0.0001
Fr
Base+CL
39.05
38.79
0.0253
De*
+KD+CL 29.08
28.82
0.0240
Es***
+KD+CL
32.13
31.43
0.0001
Fr*
+KD+CL.
39.47
38.96
0.0001
Simple KD+CL is comparable to well-chosen
Table 3: ST BLEU results with CL training for var- ious translation directions. *** indicates statistically significant differences between Dst and Dşr + Dext (p 0.01).
Dext Some previous works have observed that introducing the spoken domain Dext. such as Open- Subtitles (Lison and Tiedemann, 2016), improves ST BLEU (Ye et al., 2021; Han et al., 2021). To compare our data-efficient method with more com- petitive models, we also conducted intensive ex- periments for En-De using OpenSubtitles as our new Dext. during finetuning. Without CL, models trained with OpenSubtitles achieve higher BLEU scores, which aligns with previous works and demonstrates the importance of domain knowledge. However, with CL, training with well-chosen ternal MT becomes worthless.
CL +0.21 BLEU without CL). This suggests that the improvements achieved through CL are not limited intra-modal regularization, but rather have a broader cross-modke impact. Thus, we can attribute the enhanced performance to better MT teacher originating from the non-forgetting effect of CL.
Additional improvement from cKD and com- parison of methods From Table 2, it is evident that our straightforward combination of KD and CL outperforms the majority of previous methods, even with minimal FT data. Moreover, our pro- posed cKD method achieves additional significant improvements. While we included large-scale joint pretraining (JPT) methods in the comparison, is important to note that these methods require signif-
Considering the difficulties and high computa- tion costs of determining the best D our sug- gested approach provides more practical and effi- cient way of training ST models. Further detailed analysis of the relationship between CL and forget- ting can be found in the appendix E.
FFor detailed information, refer to appendix D

---

Models
DST
+OpenSubritles
p-value
Buropean benchmarks.
Base"*
28.07
28.54
0.0001
+KD**
28.18
28.66
0.0001
+CL
28.94
28.90
0.2942
References
+KD+CL
29.08
28.91
0.1036
Antonios Anastasopoulos, Loic Barrault, Luisa Ben- tivogli, Marcely Zanon Boito, Ondfej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Véra Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nädejde, Satoshi Nakamura, Matteo Negri, Jan Niehues. Xing Niu, John Ortega, Juan Pino, Eliz- abeth Salesky, Jiatong Shi, Matthias Sperber, Se- bastian Stüker, Katsuhito Sudoh, Marco Turchi, Yo- gesh Virkar, Alexander Waibel, Changhan Wang. and Shinji Watanabe. 2022. Findings of the IWSLT 2022 evaluation campaign. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 98-157, Dublin, Ireland (in- per-person and online). Association for Com- putational Linguistics.
Table 4: ST BLEU results with various training con- figurations for En-De, after substituting WMT with OpenSubtitles. *** indicates statistically significant differences between Dyr and Dşr + Dext (p < 0.01).
5 Conclusion
In this paper, we conduct thorough experiments to reexamine the effectiveness and efficiency of progressive training. We identify the key challenge of balancing ST and MT tasks and discover that, when KD and CL. are combined with balance, adding data plays a deterimental role and thus can be omitted for higher data efficiency. Our findings lead us to propose cKD, which dynamically utilizes intra-modal consistency for cross modal KD. Our experiments demonstrate the effectiveness of cKD in terms of both performance and data efficiency. We also provide additional analysis to support our findings.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujic Liu, Tom Ko, Qing Li. Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. SpeechT5: Unified-nee Unified-modal modal encoder-decoder pre-training for spoken language processing. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume Long Papers), pages 5723-5738, Dublin, Ireland. Associa- tion for Computational Linguistics.
6 Limitations
While our method provides a simple and data- efficient way of training. the convergence speed of single model is still slow, as previously re- ported in Liang et al. (2021). Although Beyer et al. (2022) recently studied the advantage of lengthy training schedules in knowledge distillation, KD with a faster convergence remains as an open ques- tion.
Alexei Baevski, Yubao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Sys- tems, volume 33, pages 12449-12466 Curran Asso- ciates, Inc.
Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus direct speech translation: Do the differences still make a differ- ence? In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2873-2887 Online. Association for Computa- tional Linguistics.
Additionally, while our codes and data are sourced from publicly available resources, our pre- trained MT checkpoints are not publicly accessi- ble. Although this choice was necessary ensure a fair comparison with previous studies, leverag- ing public MT checkpoints could potentially en- able more efficient training strategies, especially for low-resource languages. Notably, the availabil- ity of extensive multilingual MT checkpoints, such as those introduced by Fan et al. (2021); Ma et al. (2021), presents opportunity for enhanced train- ing approaches.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Mar- keeva, Rohan Anil, and Alexander Kolesnikov. 2022. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEF.//VV Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10925-10995
Furthermore, the translation directions we con- sidered are quite limited to Indo-Furopean lan- guages (German, Spanish, and French), though this is partially attributed to the scarcity of non-Indo-
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2018. Riemannian

---

walk for incremental learning: Understanding forget- ting and intransigence. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV).
Melbourne, Australia, Association for Computational Linguistics.
xiaobo Liang. Lijun Wu, Juntao Li, Yue Wang. Qi Meng. Tao Qin, Wei Chen, Min Zhang, and Yan Liu. 2021. R-drop: Regularized dropout for neural net- works. In Advances in Neural Information Process- ing Systems, volume 34, pages 10890-10905. Curran Associates, Inc.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017 Min- neapolis, Minnesota, Association for Computational Linguistics.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang. and Sam Tak Wu Kwong. 2019. Pareto multi-task learning. In Neural Information Processing Systems.
Pierre Lison and Jörg Tiedemann. 2016. OpenSub- titles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC*16), pages 923-929, Portorož, Slovenia. European Language Resources Association (ELRA).
Angela Fan, Shruti Bhosale, Holger Schwenk. Zhiyi Ma. Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Çelebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22:107:1-107:44
Ilya Loshchilov and Frank Hutter, 2017. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 7050-7066 Dublin, Ireland. Association for Computational I Linguistics.
Shuming Ma, Li Dong, Shaohan Huang, Dong- dong Zhang, Alexandre Muzio, Saksham Sing- hal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Deltalm: Encoder-decoder pre-training for language generation and translation by aug- menting pretrained multilingual encoders. ArXiv, abs/2106. 13736.
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2214-2225, Online. Association for Computational Linguistics.
Myle Out, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019 Demonstrations.
Sathish Indurthi, Mohd Abhas Zaidi, Nikhil Ku- mar Lakumarapu, Beomseok Lee, Hyojung Han, Seokchan Ahn, Sangha Kim, Chanwoo Kim, and Inchul Hwang. 2021. Task aware multi-task learning for speech to text tasks. In ICASSP 2021- - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7723-7727.
Siqj Ouyang, Rong Ye, and Lei Li. 2022. Waco: Word- aligned contrastive learning for speech translation. ArXiv, abs/2212.09333
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudampur. 2015, Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwinskk. et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):3511-352
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Pupers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Anthony Robins. 1995. Catastrophic forgetting, re- hearsal and pseudorchearsal. Connect. Sci., 7:123- 146.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwing Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress& & compress: A scalable framework for con- tinual learning. In Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80
Taku Kudo. 2018
Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75,

---

of Proceedings of Machine Learning Research, pages 4528 4537..5533.500 PMLR.
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingbo Zhu. 2021. Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation en- coders. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 2619-26300 Online. Association for Computa- tional Linguistics.
Ozan Sener and Vladlen Koltun, 2018. Multi tasti learn- ing multi-objective optimization. In Neural Infor- mation Processing Systems.
Chenze Shao and Yang Feng. 2022. Overcoming catas- trophic forgetting beyond continual learning: Bal- anced training for neural machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2023-2036, Dublin, Ireland. As- sociation for Computational Linguistics.
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- End Speech Translation via Moos- Progressive Training. In Proc. Interspeech 2021, pages 2267- 2271.
Yun Tang. Hongyu Gong. Ning Dong, Changhan Wang. Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 1488-1499, Dublin, Ireland. Association for Computational Linguistics.
Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5099-5113, Seattle, United States. Association for Computational Linguistics.
Xingshan Zeng. Liangyou Li, and Qun Liu. 2022. Adatrans: Adapting with boundary-based shrink- ing for end-to-end speech translation. ArXiv, abs/2212.08911.
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving speech transla- tion by understanding and learning from the auxiliary text translation task. In Proceedings the 59th An- nual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence Natural Language Processing (Volume 1: Long Papers), pages 4252-4261, Online. Association for Computational Linguistics.
Ziqiang Zhang, Long Zhou, Junyi Ao. Shujic Liu. Lirong Dai, Jinyu Li, and Furu Wei. 2022. SpeechUT: Bridging speech and text with hidden- unit for encoder-decoder based speech-text pre- training. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663- 1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yun Tang, Juan Pino, Changhan Wang. Xutai Ma, and Dmitriy Genzel. 2021b. A general multi-task learn- ing framework to leverage text data for speech to text tasks. In ICASSP 2021- 2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6209-6213.
A
Data Statisties
En-De
En-Es
En-Fr
Data
MuST-C
ST
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages 2062-2068, Minneapolis, Min- nesota. Association for Computational Linguistics.
Sents
250k
264k
274k
Data
WMT14
WMT13'
WMT14 4
MT
Sents
4M
14M
36M
Table 5: Statistics of data used for main experiments.
B Implementation Details
Data We used MuST-C (Di Gangi et al., 2019) dataset with three language directions: English (En) to German (De), Spanish (Es), and French (Fr). For external MT data, we used WMT dataset with dif-
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
ferent years for each language pairs: WMT13 for
Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. 2021. Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9385-9394.
"https://mt.. fbt.:////tbt../.../// www.:fkkkeemmmeee eu/must-c/
Phttps:://ww/ http:///wwstttttttts/tt statmt..ttt.. bratmt.org//m. org//wt114 14/
translation- html
'ht //t/:w//wwwww www.st.ttttttttt wrgg/wtt133 13/
translation- tack,

---

En-Es and WMT14 for En-De, En-Fr. For Open- Subtitle experiment, we used v2018 with 18M sentences for finetuning, while pretrained check- points are shared with WMT experiments.
training techniques are built based on this. All models are trained with fairseq" (Ot et al.. 2019) using Nvidia V100 GPUs. Following Liang et al. (2021), size of batches without CL is twice that of with CL for fair comparison. We averaged best 5 checkpoints based on ST BLEU score of MuST-C dev set. At evaluation, we used sacreBLEU" (Post, 2018 10.8). All models are trained with 3 random seeds and concatenated for statistical test through paired bootstrap resampling (Koehn, 2004).
Models We brought the model architecture from Ye et al. (2021) strong baseline systems. Con- cretly, the model comprised of transformer encoder- decoder and modality-specific encoders. For speech inputs, we used 16-bit 16kHz raw signal as our audio input, and is fed into the acoustic en- coder, which is a sequence of Wav2Vec2 (Baevski et al., 2020) and convolutional subsampler. Due to the sequence length difference between speech and text inputs, the 2-layer convolutional subsamplers with kernel size 5 and stride 2 are used. For text in- puts, conventional embedding layer with tokenized subwords used. Output from the modality- specific encoders are subsequentially fed into the shared translation encoder-decoder.
C Relation between Continual Learning and Consistency Learning
In original EWC, Hessians is usually approximated as diagonals of Fisher information matrix F. seen in EWC loss as follows:
LEWC = P
(3)
where 0" the parameter obtained from previous tasks. In the context of ST, thanks to availability of MT data, we can remove diagonal approximation from EWC through well-known relation between KL divergence and Fisher information as
Following recent works with competitive results (Ye et al., 2021, 2022; Han et al., 2021; Fang et al., 2022), we also leveraged pretraining strategies for speech and MT. While several previous works pro- posed various types of pretraining, we only ex- ploited two types of pretraining: speech represen- tation learning and MT. For speech, we used pub- licly available, Librispeech (Panayotov al., 2015) trained Wav2Vec2 (Baevski et al., 2020) with base configuration". For MT, we pretrained encoder- decoder on MT task with external dataset Dext. We used post-norm transformer- base (Vaswani et al., 2017) with shared embedding layer.
D PKl(pp ((pe) E, [log Pg- llog po]
1
(4)
-0 F'
where indicates that the Fisher is calculated at gr.
Concretely, CL can be understood as regular- izing the dropout submodel using full model's curvature, unlike EWC-like regularizations with the following differences: it uses online mini- batch throughout the training, not fixed subset of previous task's dataset (Kirkpatrick et al., 2017) or exponential moving average of mini-batch Fisher (Chaudhry et al., 2018); Fisher is computed at continuously varying parameter 6 not fixed #* Despite these differences, the joint training strategy allows for the approximation of Eq 2 to be con- sidered accurate, the parameters will remain in the low-loss valley of MT throughout the training process. Intuitively speaking, dropout-averaged curvature information regularizes the gradient at each training step in more manner, as opposed to simply summing multitask gradients.
Training & Evaluation For training, unigram tokenizers are firstly trained using sentencepice (Kudo, 2018) with 10k joint vocabularies on the transcriptions and translation pairs from MuST- C. For main experiments, we use AdamW opti- mizer (Loshchilov and Hutter, 2017) with 31 = 0.9 and B2 0.999. Learning rate is 5 x 10 with 25000 warmup steps and inverse square root scheduling. Weight for KD and CL was DKD 0.2 and aCL- 5.0 For MT pretraining, we use AdamW optimizer (3) 0.9, A = 0.98)
with learning rate 7 x 10-4, 4000 warmup steps. Dropout (p = 0.1) and label smoothing (p= 0.1) is applied to both MT pretraining and finetuning. Traplet-based joint training only with D is con- sidered as bascline, and various combinations of
"nttps://githubb http:/ttww..mmmccceoreeeeeeeee/e
hnttps://opus... http:/uus.nnll.ee//peeeeuuu http/penSuuiitles-221111 'https://d1l fttt:::://d..faaiulliiifcc... aav2vec/waa2vee_gma1...g
"BLEU signature: nrefs: 1 |bs: 100000 seed: 12345] case: case:mixeds mnixxe]efff: tok::331 smooth: version:
2.0.0

---

D Comparison with joint pretraining (JPT) methods
34
33
My m
Joint-pretraining (JPT) approaches have critical limitations in low-resource settings. JPT requires the preparation and aggregation of more data from multiple sources, shown in Table 6. They also necessitate additional data augmentation, either at the phoneme level (STPT) or the unit level (SpeechUT). This, in turn, adds complexity to the training pipeline, such as the inclusion of a phoneme transcriptor (STPT) and a lext-to-unit generator (SpeechUT), as illustrated below. In terms of computational efficiency, we also compare the GPU updates required by approach with JPT. Our proposed contribution, which involves interleaving moddality-specifit models in terms of data and computation, leads to significant cost sav- ings.
32
Init with pretrained with forgotten
100000
150000 200000 250000 Steps
Figure 3: Evolution of MT BLEU with different param- oter initialization.
clear that forgotten MT knowledge cannot be restored even with CL. also can be understood through our continual learning interpretation of CL discussed in Section C, that CL loss cannot pro- vide credible curvature information outside of MT low-loss region.
STPT(TTng et al., 2022)
Joint pretraining: 16 A A100, 12 gradient accumulation, 200k updates
Joint finetuning: 8 V100, 3 gradient ac- cumulation, 50k updates
WW Kel of  Www.ertteateeeeeoe
Spearman correlation coefficient
SpeechUT(Zhang et al., 2022)
-0.1
Text-to-unit (T2U) generator: not re-
ported
- Joint pretraining: 32 V100, gradient accumulation, 400k updates
-0.2
Task-specific finetuning: V100, 4 gra- dient accumulation, 50k updates
-0.3
Base CL
Ours
-0.4
25
100 Epoch
125
150
175
200
Joint finetuning: V100, 4 gradient ac- cumulation, 200k updates
Figure 4: Epoch-wise Spearman correlation coefficient between batch order and their losses for IWSLT14 De- En. It is clear that CL alleviates imbalanced training problem.
E
Further analysis
Forgotten MT knowledge cannot be restored Advantage of adopting CL ST can understood as two-fold: the well-known regularization effect already discussed in original paper (Liang et al., 2021). and keeping MT knowledge for successful cross-modal transfer. To verify that BLEU improve- ments cannot be solely explained by the former, tried to fix the model, which already had undergone catastrophic forgetting, and see whether restores MT knowledge. Concretely, we initialize the whole model with final checkpoint from baseline of 2 and retrain it with KD+CL. As shown in Figure 3,
CL also remedies imbalanced training Shao and Feng (2022) recently found that the problem of catastrophic forgetting arises in not only continual learning but also conventional static learning, espe- cially in low-resource regime of complex tasks (c.g. MT). Concretely, they found the imbalanced train- ing problem exists, that the models concentrate more on recently exposed data samples. While we established the relation between CL and catas- trophic forgetting in cross- modal transfer, we fur-

---

Pretrainig Data
Models
Speech
ASR
MT
STPT(Tang et al.. 2022)
60k hours
400 hours
4.5M sentences
SpeechUTCZhang et al., 2022)
1.4k hours
100 hours
4.5M sentences
Ours
960 hours
4.5M sentences
Table 6: Amount of pretraining data required for En-De
ther verify it with conventional MT through the lens of imbalanced training. We trained low-resource MT systems with IWSLT14 De-En with 160k sen- tonces and calculated epoch-wise Spearman cor- relation between order training batches and their respective losses following Shao and Feng (2022), that negative coefficient implies imbalanced train- ing. As Figure shows, CL largely alleviates the problem of imbalanced training even without the use of additional teacher model originally proposed in Shao and Feng (2022). We also gathered epoch- wise normalized losses over the whole training and calculated Spearman correlation for quanti- tative comparison, and it reconfirms our findings (r = 0..1 for baseline vs. r -0.07 for CL).
"hhttps://wsstt.


Reviews:
[{"review_id": "f0HZsFJn3G", "reviewer": "Reviewer_mLFp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1\nOur novelty is, as the reviewer also summarized, a thorough reinvestigation of the commonly held belief in the effectiveness of progressive training, and proposing intra-modal consistency to augment KD. \n\nWe believe the extensive evaluation we conducted across a wide search space, coupled with robust statistical testing is a significant contribution and also could not find an existing finding similar to this, which was encouraged to be mentioned in the review when arguing the lack of novelty, per reviewing policy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 This paper conducts an empirical investigation and finds that progressive training is ineffective and proposes a more effective method.\n\n2 This paper proposes cKD to dynamically utilize intra-modal consistency for cross-modal KD, which is the major novelty of this paper. \n", "reasons_to_reject": "The paper lacks sufficient novelty.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed a method which combines knowledge distillation consistency learning to improve ST. This paper identifies learning-forgetting trade-off as a critical obstacle, then hypothesize and verify that consistency learning (CL) breaks the dilemma of learning-forgetting.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dHKfseIh3n", "reviewer": "Reviewer_zMRf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 & Q2\nWe argue that these two points can be addressed through rewriting. Due to the space limitations of a short paper, we have included many supporting pieces of evidence in the Appendix (C for motivation of CL and E for additional analysis). These can be incorporated into the main text to better motivate CL and support claims with evidence.\n\nRegarding MT-ST interference, it has already been reported in prior works (e.g., section 5.2 of https://arxiv.org/abs/2204.05409). We reconfirmed this in our preliminary experiment, where a significant amount of additional MT data exhibited poor performance and, in some cases, even failed to converge. We will supplement the discussion about it.\n\n### Q3\nWe stress our gains are statistically significant for De and Es, derived from robust and thorough experiments. We employed bootstrap resampling with three random seeds to ensure scientifically meaningful conclusions (see Appendix B for details).\n\n### Q4\nAppendix E supports the improvement is from “avoidance of forgetting”, as forgotten MT knowledge cannot be restored even with CL. It also can be understood through our continual learning interpretation of CL in Appendix C that CL loss cannot provide credible curvature information outside of MT low-loss region.\n\nRegarding the strong results of CL-only in Table 4, these were observed only when in-domain MT data is available (as shown in Table 3). Also, note that even in the case of in-domain MT, cKD outperforms both CL and KD+CL, indicating the importance of intra-modal consistency for a successful KD.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThoroughly evaluates and analyzes the common practice of progressive training for ST, showing it is actually ineffective despite its prevalence. This is an important empirical finding.\n2.\tProposes a simple yet effective combination of consistency learning and knowledge distillation that achieves strong results without requiring extra training data. This provides a valuable data-efficient training approach.\n3.\tIntroduces a novel consistency-informed knowledge distillation method that leverages intra-modal consistency for better inter-modal transfer. This is an interesting modeling contribution.\n", "reasons_to_reject": "1.\tAlthough the proposed method achieves remarkable improvements, the motivation behind introducing CL is unclear. The claim made in lines 164-173 is confusing and difficult for me to comprehend.\n2.\tSome arguments lack supporting evidence. For instance, the authors state in line 131 that \"The external MT diverts the model's focus from ST,\" but there is no analysis provided regarding the interference between MT and ST.\n3.\tThe improvements of the cKD method are limited.\n4.\tCL is indeed an effective regularization method that helps alleviate overfitting. However, it is necessary to demonstrate that the improvements in the MT model are not solely due to regularization but rather to the avoidance of forgetting. Additionally, the significant improvements achieved by using the CL method alone, as shown in Table 4, suggest that the KD technique may not be as important. The better regularization method may be crucial for ST fine-tuning.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the effectiveness of progressive training, a commonly used technique for training end-to-end speech translation (ST) models by leveraging abundant machine translation (MT) data. The main contributions are:\n\n1.\tThe authors re-evaluate progressive training and find it is actually ineffective for improving ST performance, despite its prevalence.\n2.\tThey identify the core issue as the trade-off between learning ST and avoiding catastrophic forgetting of MT knowledge.\n3.\tTo address this, they propose using consistency learning (CL) to alleviate forgetting, allowing better transfer from MT models.\n4.\tThey show a simple combination of CL and knowledge distillation (KD) achieves strong results, outperforming progressive training.\n5.\tThey further improve performance with a novel consistency-informed KD method that leverages token-level consistency information.\n\nOverall, this paper makes important empirical and modeling contributions around effectively leveraging abundant MT data to improve low-resource ST in a data-efficient manner. The proposed CL and consistency-informed KD methods outperform previous techniques.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H0cWD1PEJs", "reviewer": "Reviewer_muUy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1\nBecause our primary goal was a thorough reexamination of prior works, we focused on reporting results with statistical significance  for 3 language pairs. We are experimenting with 8 languages, but due to the resources required for the extensive experiments (24 models for each language direction), will report conclusive significance results to the camera-ready version.\n\n### Q2\n In Appendix D, we already presented a quantitative comparison with JPT methods, demonstrating that our method requires only about 1/10 to 1/5 of the GPU cost when compared to the JPT methods.\n\n### Q3\nWhile our codebase and data are already reproducible being sourced from publicly available resources (e.g., fairseq and WMT), we will also provide an easier access to the data, code, and training scripts to ensure straightforward reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation of the work and of the proposed methods are clear.\nThe proposed approach combining KD and CL helps to deal with catastrophic forgetting and is more efficient than choosing the best D_ext, and is further improved using the cKD loss.", "reasons_to_reject": "The claim that progressive training is ineffective could be better supported with more results from more language pairs (given that there are 8 available from MuST-C).\n\nThe authors also mentioned expensive computational overhead and made a point on training efficiency, thus it could be good to provide some self-contained quantitative comparisons.\n\nIt also seems that some results are difficult to reproduce due to MT checkpoints not being publicly available, if I'm not missing something here.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is focused on methods aiming to deal with the data scarcity issue in end-to-end speech translation (E2E ST). The authors investigated the effectiveness of progressive training, empirically found that it is not as effective due to learning-forgetting trade-off, and proposed a new method combining knowledge distillation and consistency learning to deal with catastrophic forgetting. The authors then propose consistency-informed KD (cKD) using token-level consistency to further improve the method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- l. 117 \"trainscrption-translation\" -> \"transcription-translation\"\n- l. 563, \"paris\" -> \"pairs\"\n\nPresentation:\n- In Table 2, column De, the numbers are not well-aligned (ex. they could be centered around the dot)"}}, {"review_id": "5HGdBSye7y", "reviewer": "Reviewer_8RCh", "summary": "", "questions": "Some work (e.g., SATE, STEMM) has proved that KD is an effective way to bridge the modality gap. Why does it only achieve slight improvement in this work? What is the possible reason?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding the domain problem\n\nOur experiments with in-domain MT are already presented in Table 4. With our proposed combination of KD and CL, progressive training does not bring about significant BLEU gains, which is consistent with out-domain experiments.. Moreover, it is evident that consistency plays a significant role in successful KD, as the inclusion of CL or cKD produces results that are comparable or even superior to the BLEU obtained from the incorporation of carefully selected, in-domain MT data (OpenSubtitles 28.91 vs. KD+CL 29.08 vs. cKD+CL 29.27). Our conclusion is solid, being already supported with both in- and out-of-domain MT data.\n\n### Regarding JPT and training efficiency\n\nThe catastrophic forgetting that occurs in cross-modal transfer is an important research topic in its own right. We have revealed that speech translation suffers from one, then demonstrated through extensive experimentation and analysis that CL can address this.\n\nIn practical terms, as discussed in Appendix D, JPT methods necessitate a substantial amount of data collected from diverse sources and significant computational resources (e.g. 5x~10x GPU cost compared to ours). This implies that JPT methods cannot be generalized well for low-resource situations (e.g., low-resource languages). Our objective of achieving efficient cross-modal transfer without catastrophic forgetting provides additional low-resource and generalizable alternatives to the JPT approaches.\n\n### Regarding the effectiveness of cKD\n\nWe stress our gains are statistically significant for De and Es, derived from robust and thorough experiments. We employed bootstrap resampling with three random seeds to ensure scientifically meaningful conclusions (see Appendix B for details).\n\n### Regarding the effectiveness of KD\n\nAs KD itself does not improve speech translation a lot (JT-S-MT in table 2), for effective cross-modal KD, most of the previous non-JPT studies have utilized supplementary techniques to bridge the gap between modalities (e.g., contrastive learning, cross-modal mixup, or an additional adaptor module). In this work, we found that simple CL can be more effective than complex alternatives shown above (lines 218-221).\n\nOur method not only results in strong performance, but its simplicity also facilitates the seamless incorporation of the aforementioned “gap-bridging” methods. The investigation into its effectiveness remains a subject for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The analyses are based on strong baselines, and the Consistency Learning based on R-Drop shows a significant improvement. \n\n2. The conclusions can benefit the Progressive training method for the speech translation task, making the whole training process more efficient.\n", "reasons_to_reject": "1. The influence of the domain problem is not clearly claimed. I noticed that the XSTNet (Ye et al., 2021) shows that adding MT loss during the fine-tune stage is useful, which is inconsistent with your conclusions. The difference between the two papers is that they use in-domain MT data, while you use out-of-domain MT data. Thus, the influence of the domain problem has not been explored clearly, and whether it will affect your conclusion is unknown. This makes your conclusion not solid enough.\n2. The contribution is limited. As shown in Table 2, joint pretraining methods do not rely on Progressive training. Furthermore, the proposed cKD method is not effective enough. \n3. Since the motivation is to improve training efficiency, there is a lack of comparison about the training time between the baseline and proposed methods.\n", "questions_to_authors": "Some work (e.g., SATE, STEMM) has proved that KD is an effective way to bridge the modality gap. Why does it only achieve slight improvement in this work? What is the possible reason?", "paper_topic_and_main_contributions": "During the fine-tuning stage of the progressive training for the speech translation task, previous work usually utilizes the machine translation (MT) task to help train the speech translation model. The analysis of this work shows that the MT task is not necessary after adding the consistency learning strategy. They adapt the R-drop method as the consistency learning and further improve the knowledge distillation (KD) method called cKD. Their experiments on MuST-C three datasets show the effect of their methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Clarifying the meaning of $\\mathcal{D}_{\\mathrm{ext}}$ can help the readers understand Table 1.\n\n2. Some tables cross the margin.\n\n3. The footnotes in the Appendix are wrong.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows the limitations in progressive training for speech translation (where external text translation data is used during fine-tuning), and then proposes an approach to address this shortcoming. The reviewers agreed that the experiments were sound in showing the problems associated with progressive training as well as the improvements that can be achieved by the new approach. However, the reviewers also had concerns that further analyses could have improved some of the claims in the paper. Some reviewers also had concerns about the novelty of the work, leading to overall ambivalent excitement. Nevertheless, the overall soundness is good.
Decision: Accept-Findings