Towards Zero-shot Learning for End-to-end Cross-modal Translation Models
Authors: Jichen Yang
Kai Fan
Minpeng Liao
Boxing Chen
Zhongqiang Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: Zero-Shot
End-to-End
Speech Translation

Abstract: One of the main problems in speech translation is the mismatches between different modalities. The second problem, scarcity of parallel data covering multiple modalities, means that the end-to-end multi-modal models tend to perform worse than cascade models, although there are exceptions under favorable conditions. To address these problems, we propose an end-to-end zero-shot speech translation model, connecting two pre-trained uni-modality modules via word rotator's distance. The model retains the ability of zero-shot, which is like cascade models, and also can be trained in an end-to-end style to avoid error propagation. Our comprehensive experiments on the MuST-C benchmarks show that our end-to-end zero-shot approach performs better than or as well as those of the CTC-based cascade models and that our end-to-end model with supervised training also matches the latest baselines.

Towards Zero-shot Learning
for End-to-end Cross-modal Translation Models
Jichen Yang'; ; Kai Fan Minpeng Liao", Boxing Chen', Zhongqiang Huang²
'The Hong Kong Polytechnic University
2 Alibaba DAMO Academy
buchen..ynggppe edw.ooluluuuuuu..... hk
[k. fan, minpeng. 1mp, boxing. cbx, z www.gnnellibbbbbiiinccmmmmm www.llibbbbbincc..m
Abstract
language pairs. Solutions have been proposed combat this data problem. In (Liu al., 2020; Xu et al., 2021), an adapter with additional parameters is used during fine-tuning to combine the two pre- trained models of different modalities. The new module, however, only learns from ST data, which is of a greatly reduced quantity. The alignment in building cross-modal representations is also popular topic. Zhang al. (2023) simply concate- nates the representations of different modalities and lets the self-attention learn the cross- modal features. Some solutions deal with this problem through mapping features into fixed-size represen- tations (Reimers and Gurevych, 2019; Feng et al., 2020; Han et al., 2021; Duquenne et al., 2022). The squared error is generally used as the optimization objective (Pham al., 2019; Dinh et al., 2022). They may suffer from information loss when repre- sentations are compressed or constrained prior.
One of the main problems in speech translation is the mismatches between different modali- ties. The second problem, scarcity of paral- lel data covering multiple modalities, means that the end-to-end multi-modal models tend to perform worse than cascade models, although there are exceptions under favorable conditions. To address these problems, we propose an end- to-end zero-shot speech translation model, con- necting two pre-trained uni-modality modules via word rotator's distance. The model retains the ability of zero-shot, which is like cascade models, and also can be trained in an end-to- end style to avoid error propagation. Our com- prehensive experiments on the MuST-C bench- marks show that end-to-end zero-shot ap- proach performs better than as well as those the CTC-based cascade models and that our end-to-end model with supervised training also matches the latest basclines.
In order to overcome both the data and the length problems, we propose a pre-trainable adapter that connects two pre-trained modules. Specifically, we adopt a popular cross-modal ST architecture that can be generalized to many existing works. For the alignment adapter, we employ as loss the Word Rotator's 's Distance (WRD) minimization (Yokoi et al., 2020; Monge, 1781; Kantorovich, 1960; Peyré al., 2019), allowing the adapter to pro- mote the cross-modal representations that match in the space of the semantic encoder. Unlike previ- ous works, this strategy allows us to pre-train the adapter. Meanwhile, instead of mapping to a fixed length, the CTC enables adjustment of the length of the source modality representation dynamically. This step can guarantee the cross-modal representa- tions become features with similar but not exactly the same length, and then our proposed WRD ob- jective with optimal transport (OT) solver can align them properly.
1 Introduction
Speech translation (ST) requires knowledge trans- fer among different modalities, whereas models more often than not perform worse on cross-modal tasks. The ST model in real-world applications is usually a cascade approach that first uses an au- tomatic speech recognition (ASR) system to tran- scribe the speech into text and then uses text ma- chine translation (MT) model. Recent end-to-ent (e2e) ST models remove the need for an explicit ASR, with several practical advantages over the cascade models such reduced latency, reduced eтror propagation, and shorter pipeline.
However, e2e ST models are less competitive than cascade models practice (Zhang et al., 2019; Sperber and Paulik, 2020; Dinh, 2021) because end-to-end data are an order of magnitude less than those for ASR or MT, especially for low-resource
equal contribution. Work was done during Jichen Yang's 's research internship DAMO Academy, Alibabu Group. Corresponding author.
Besides speech translation, our approach can be naturally adapted into image translation. Unlike the

---

E,d,
6468
W.
WHB Loss
(8000
Domain
Sohmax
0590
Shrin Layst
Layer
h
DMC
TUT
00566000
BED
Enooder
Layor
Encoder
Shrne Adspler
x-Va
Layer
Shared
Audii
TGT
Supsample
Transoription
Transtation
Audio
Croze Wode Encoder
Semanto Encodar and Decoday
Tra seenss
ERC
(0)
Archilacuuro
Usago
Tepslisd
Transsdption
Tonnlatonn
Deooder
Croas-medee
Shrrisk
Cross-modial Encodar
Shrink Adaaparr
Grosdemodial Encodeer
Shrink Adapparr
Enoodar
Adaaphir
Samariic
Encedrer
Serratic
Erooder
WHD
Semantc
Enezderr
WHO
Audo//nap
Toweu
(6) Semand: encoder and docoder training
(d) Zaro shot moss-modal traiinn
(e] End-lo--dd finotuning
Figure 1: Overview of our proposed framework. (a) The overall model architecture. (b) Data usage in sero-shot setting and the possible supervised setting. (c) NMT pre-training. (d) Only cross-modal encoder is trained with ASR data, where the semantic encoder is freezing. (e) Fine-tuning if end-to-end data is available.
image assistant translation (Su et al., 2019; Yawei and Fan, 2021), we attempt to translate the text within the image. Our goal is also to align the cross-modal representation between images and texts (Li al., 2022). The related discussion and experiments refer to the Appendix A.4.
2.2 Zero-shot Translation Training
In this phase, we train zero-shot translation model by training the cross-modal encoder alone shown in Figure (d). As the tradition, we apply the recog- nition task with ASR data Dom {(z,: x,)), and adopt a classical ASR architecture with CTC clas- sifier, and optimize the CTC loss Cate
The contributions of this paper are follows: (I) We adopt the WRD loss together with the shrink mechanism to measure two feature sequences in different lengths, enabling the adapter pre-training (2) The pre-trained adapter allows an end-to-end zero-shot ST ability like cascade models.
Besides the regular recognition task, we use Word Rotator's Distance (WRD) (Yokoi et al., 2020) supervise the encoder to generate encod- ing results with less diserepancy across different modalities. We expect to align different modali- ties in the space of the semantic encoder, allowing the seamless transition between the cross- modall encoder and the semantic decoder. To precise, suppose the acoustic encoder output as h, and the CTC distribution as d. = softmax( W.h,), a light- weight adapter shrinks and integrates them. The shrink mechanism (Yi al., 2019; Tian al., 2020; Gaido et al., 2021) is widely employed to remove the representations of blank and repeated tokens. Thus, we consider using the CTC path via effi- cient arg max as the guidance to remove the blank tokens and average the representations of consecu- tively duplicated tokens, as shown in Figure ((a). By denoting the shrunk hidden state and CTC distribution as h, and de, the adapter output is,
(3) Experiments on the MuST-C demonstrate that our end-to-end zero-shot model can match or be slightly better than the CTC-based cascade model (without intermediate post-procee procesrocsssing). The results our end-to-ent training can also match the recent supervised ST baselines.
2
Main Method
We adopt a popular framework in Figure I(a). including cross-modal encoder with a shrink adapter and a semantic encoder/decoder pack.
2.1 Semantic encoder-decoder training
A machine translation model is first pre-trained as illustrated in Figure 1(c). Given machine transla- tion corpus Dint {(x,, y:)), our aim is to obtain semantic encoder Ene, (E,x;) h, and seman- tic decoder Dec; = P(yr h,), where E, the source embedding matrix. The objective the task is defined as the cross entropy loss Lynt-
h = E,d. + W. h.
(1)
where W. is the trainable parameters in the adapter. More details can refer to the implementation. Since

---

En-De v2
En-De
En-Fr
En-Es
Training Data
he
he
he
he
WMT
28.59
27.45
28.62
27.44
40.79
37.54
32.38
38.
WMT MuST-C
33.13
31.99
32.99
31.90
44.14
39.97
36.96
42.04
Table Performance of MT on MuST-C testset (BLEU1). The input is the ground truth of the source transcription.
En-De v2
En-De
En-Fr
En-Es
Average
Model
common
he
common
he
he
common
he
Cap
Zero-Shot: MT is only trained on WMT corpus.
cascade
17.30
27.15
21.29
MultiST
113.99
zero-shot
/
/
6.77
10.85
/
6.75
/
Chimera
zero-shot
22.2
15.3
cascade
22.85
22.27
22.45
22.30
32.60
31.65
26. 14
31.55
Ours
+0.79
zero-shot"
24.00
23.04
23.41
22.94
33.65
32.25
26.48
32.32
Pseudo Zero-Shot: MT trained
WMT and MuST-C parallel corpus.
Tight
cascade
1
/
25.9
25.0
30.2
37.6
-1,325
Integrated
p. zero-shot
25.1
24.4
28.7
35.2
cascade
26.43
25.14
25.21
25.32
34.53
32.63
29.15
34.68
Ours
+0.78
p. zero-shot"
27.39
26.46
26.52
25.46
35.34
33.66
29.46
35.05
Table 2: Zero-Shot ST on MuST-C (BLEU1). TTight Integrated extends our ASR data 2300 hours, and it used
27M En-De and 48M En-Es MT data.
the ASR performance greatly affects the quality of CTC paths (Fan al., 2020), our shrink method dif- fers from previous approaches, where the adapter merges the representations from both before and after the CTC module to reduce error propagation. ha can be regarded as the final audio representa- tion which is ready to be fed into the semantic
WRD emphasizes the semantic similarity be- tween sequences better than Euclidean dis- tance. The common solution is to implement the Inexact Proximal point method for Optimal Trans- port (IPOT) algorithm (Xie et al., 2020) as shown Appendix A. Because of CTC prediction errors. is possible that m * n, but the loss Lurd can circumvent the length discrepancy for alignment. The final loss of the cross-modal training is,
encoder. To alleviate the cross-modal mismatch, we optimize the WRD loss.
Laer = AeteCete + Nurdhurd
(5)
ea Encye {et C
(2)
Ence(h:) {el ........ e
where Acte and Aurd are hyper-parameters. To keep the semantic encoding intact, the semantic encoder including the embedding matrix is frozen, leading to zero-shot translation system naturally from the ASR training.
wrrd D Durd 01).
The detailed WRD loss is defined as follows.
Dur ex) = T"), - 1-cosee e()
2.3 End-to-End Translation Training
where ( denotes the dot-product and cos(,,)) is the cosine similarity. T* is the optimal transport (OT) plan from the following problem.
Once have the triplets supervision dataset Dr = ((),x,)) such as speech-transsrrition translation, it is possible to proceed for the fine- tuning phase as shown in Figure 1(e). Since the zero-shot training loss Eq. (5) is still valid in this phase, we can integrate it into the final end-to-end ST training objective L (2, Lasr.
T arg min(C. C, T) T20
S.t.Tlm = p. T =
where are the normalized ea, e,, Particularly. we represent them two vectors (pi....,,Py and
3 Experiments
!4,..... 2ml
3.1 Datasets and Settings
le
| Σ
ST We conduct our experiments on the three popu- lar language pairs MuST-C (Cattoni et al., 2021): ):
p:
12 1=1
lef
4 =
2y=1
|e:112

---

Num.
En-De v2
En-De
En-Fr
En-Es
Model
Params
commnn
he
common
he
commonn
he
comunoon
be
MIL' (Tang al., 2021b)
31M
23.9
33.1
28.6
FAT-ST (Zheng et al., 2021)
58M
/
25.5
30.8
JT-S-MT* (Tang al., 2021a)
74M
/
26.8
37.4
31.0
Chimera (Han et al., 2021)
165M
27.1
30.6
XSTNET (Ye et al., 2021)
155M
/
27.8
38.0
30.8
STEMM (Fang al., 2022)
155M
28.7
37.4
31.0
FT from zero-shot"
95M
29.22
29.07
28.22
28.22
39.00
37.06
31.96
38.83
FT from p. zero-shot"
95M
29.12
29.74
28.17
28.19
39.05
37.21
32.03
38.89
Table 3: Supervised ST on MuST-C (BLEUt with beam=5) with additional Librispeech and WMT data. MTL uses the hidden dimension 256. # JT-S-MT only uses WMT data. FT from our models marked with Table
English-German (En-De) V1 and V2, English- French (En-Fr), and English-Spanish (En-Es).
Aard 10.
In Table 1, we list the BLEU scores of our pre- trained MT models of the first training stage, both zero-shot and pseudo zero-shot settings. Our main results of zero-shot ST are illustrated in Ta- ble 2. We compare our model with the pioneering zero-shot ST method MultiSLT (Escolano et al., 2021), achieving the zero-shot translation via ASR training with adapter well. We compare another cross-modal alignment method Chimera (Han et al., 2021), which is initially designed for su- pervised ST training but also suitable for zero-shot ST. Clcarly, our system can achieve a minimum gap between the cascade and the end-to-end setups in zero-shot scenario, and our end-to-end zero-shot ST on average performs +0.79 higher than that of the cascade system.
ASR The 960h LibriSpeech English ASR dataset (Panayotov et al., 2015) is mainly used for pre- training the ASR the zero-shot training stage.
MT For En-De and En-Fr, we collect the WMT 2014 data with about 4.5M and 36M parallel sen- tences respectively as in Vaswani al. (2017). For En-Lss, collect the WMT 2013 data of size 28M. Model Details The audio inputs are pre- processed as 80-channel log Mel filterbank coeffi- cients as fairseq'. The cross-modal encoder con- tains two ID convolutional subsampler layers (Syn- naeve et al., 2019) and 12 transformer encoder lay- ers with hidden dimension 512. The MT model is architecture (Vaswani al., 2017). The individual vocabulary includ- ing 10K sub-word units learned by Sentence- Piece (Kudo and Richardson, 2018). All hyper- parameters such as are tuned on En-De V2 and directly applied to other datasets. Additional de- tails can refer to the Appendix A.2.
standard transformer-base
Following Tight Integrated (Dalmia et al., 2021; Bahar et al., 2021), we also conduct pseudo zero-shot ST experiment. In this case, even though each training phase does not directly consume any speech-translation pairs, the overlapped MuST-C transcription data could be seen by both ASR and MT models. The gap between cascade and end-to- end remains unchanged (+0.79 - +0.78) for our model. It is an indication of the stability of our approach to bridging the modality gap.
3.2 Main Results
Zero-shot ST Recent works (Dinh, 2021; Escolano al., 2021) indicate that when large amounts of ASR and MT data dominate the training, the cas- caded ST is better than the direct end-to-end ST. our proposed second phase, the desired ASR training can easily facilitate the building of zero- shot ST model. The MT model is pre-trained with the WMT data alone, preventing the model from accessing the in-domain data of MuST-C. For the ASR training, combine the Librispeech data and the speech-transcriittine pairs in MuST-C give comparable amount of ASR data as in the practical cascade system. Particularly, we set dete = 1 and
Supervised ST In this experiment, we evaluate the performance of our third training phase. We compare our approach only in the unconstrained scenario with the recent end-to-end ST methods that used similar datasets, and the results are sum- marized Table 3.
3.3 Analysis
For zero-shot ST setting, the average gap is 0.79, and the breakdown difference is visualized in the middle panel of Figure 2. The cascade system only has more sentences falling in BLEU interval [o, 10).
pertps://githhee thuo htt/aacebookeseerrcotn hwr.aaabooorrrrrerrrrreeeeeeeree wwamples/speacht to text

---

183
Supervised Loss
BLEU
ST
CTC
WRD
29.22
200
23-
20.94
140
20.86
0.0
43
60
130
Figure 2: En-De v2. Left: BLEU v.s. WRD. Middle: Distribution of BLEU scores. Right: Ablation study of loss.
Model Module
En-De v2
Acoustic Encoder
Adaptor
Semantic Enc/Dec*
common
trainable
trainable
trainable
29.22
trainable
trainable
frozen
25.14
frozen
trainable
trainable
28.94
Table Ablation study on model parameters for supervised training phrase. models are fine-tuned from the zero-shot ST model with BLEU 24.00 in Table 2. AAcosstc encoder includes the CTC layer.
We also plot the relation between BLEU and WRD for each sentence in the tst-CMMON set of En- De v2 (left panel of Figure 2). The overall trend indicates the BLEU score decreases with increasing word rotator's s distance.
direct end-to-end ST model the zero-shot setup that matches the cascade system without additional post-processing, e.g., rescoring via an additional language model. Our method can also achieve com- parable results to recent supervised ST models.
To achieve the zero-shot speech translation, the losses in ASR training are both required. So the ablation study in the right panel of Figure explored the effect of each loss in final end-to-end supervised training. All models are fine-tuned from the zero-shot ST model with BLEU 24.00 in Ta- ble 2. The CTC loss cannot be directly removed since the WRD depends on a reasonable CTC path. Therefore, we optimize the supervised loss with- out CTC loss by freezing the acoustic encoder and CTC layer.
Limitations
The accuracy of IPOT depends the number of loops. Since we unroll the IPOT similar RNN training and apply automatic differentiation on the IPOT algorithm the back-propagale stage, the iterative process will consume more computing resources than directly calculating the Jacobian ma- trix of the OT plan. Besides, though our model is able to work different translation tasks such image translation and speech translation, the hyper- parameters, especially the weights of WRD loss and CTC loss, would be varied on each task. The CTC loss and WRD loss are sometimes conflict- ing, which also requires us to set a pair of proper weights via deep hyper-para searching.
In Table 4, we have another ablation study on whether the model parameters are trainable for the supervised training phase. The result becomes much worse if the semantic encoder/decoder is frozen. The main reason we hypothesize is that since the NMT teacher is frozen, the in-domain MT data is not used. So it's difficult for the NMT decoder adapt the supervised ST data, i.e., the decoder not good language model.
Ethics Statement
After careful review, to the best of our knowledge. have not violated the ACL Ethics Policy.
4 Conclusion
Acknowledgements
In this paper, we present zero-shot architecture that takes better advantage of cascade models, bridging the gap between cascade and end-to-end translation models. By leveraging differentiable shrink adapter and WRD loss, our approach is a
We would like to thank all the anonymous review- ers for their insightful and helpful comments. This work was supported by Alibaba Group through Al- ibaba Research Intern Program.

---

References
translation. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2214-2225.
Parnia Bahar, Tobias Bieschke, Ralf Schlüter, and Her- mann Ney. 2021. Tight integrated end-to-end training for cascaded speech translation. In 2021 IEEE Spo- ken Language Technology Workshop (SLT), pages 950-957. IEEE
L. V. Kantorovich. 1960. Mathematical methods of organizing and planning production. Management Science, 6(4):366-422.
Roldano Cattoni, Mattia Antonino Di Gangi, Luisa Ben- tivogli, Matteo Negri, and Marco Turchi. 2021. Must- multilingual corpus for end-to-end speech trans- lation. Computer Speech & Language, 66: 66:101155. 155.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language P Processing: System Demonstrations, pages 66-71.
Siddharth Dalmia, Brian Yan, Vikas Raunak, Florian Metze, and Shinji Watanabe. 2021. Searchable hid- den intermediates for end-to-end models of decom- posable sequence tasks. In NAACL-HLT.
Lei Li, Kai Fan, and Chun Yuan. 2022. Cross- modal representation learning and relation reasoning for bidirectional adaptive manipulation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3222-3228. International Joint Conferences on Artificial Intelli- gence Organization. Main Track.
Tu Anh Dinh. 2021. shoo speech translation. arXiv preprint arXiv:2107 10700000
Tu Anh Dinh, Danni Liu, and Jan Niehues. 2022. Tack- ling data scarcity in speech translation using zero- shot multilingual machine translation techniques. In ICASSP 2022 2022 IEEF International Confer- ence on Acoustics, Speech and Signal Procexsing (ICASSP), pages
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the modality gap for speech- to-text translation. arXiv preprint arXiv:2010. 14920.
G. Monge. 1 1781. Memoire sur la theorie des deblais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pages 666-704.
Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, and Holger Schwenk. 2022. T-modules: Translation modules for zero-shot cross-modal machine transla- tion. arXiv preprint arXiv:2205.122:1 12216.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr cor- pus based public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5210. IEEF.
Carlos Escolano, Marta R Costa-jusså, José AR Fonol- losa, and Carlos Segura. 2021. Enabling zero- shot multilingual spoken language translation with language-specific encoders and decoders. In 2021 IE..F. Automatic Speech Recognition and Understand- ing Workshop (ASRU), pages 694-701. IEEE
Gabriel Peyré, Marco Cuturi, et al. 2019. Computa- tional optimal transport: With applications to data sci- ence. Foundations and Trends@ in Machine Learn- ing. ((5-6):355-66 607.
Kai Fan, Bo Li, Jiayi Wang, Shiliang Zhang, Box- ing Chen, Niyu Ge, and Zhijie Yan. 2020. Neural zero-inflated quality estimation model for automatic speech recognition system.
Ngoc-Quan Pham, Jan Niehues, Thanh-Le Ha, and Alexander Waibel. 2019. Improving zero-shot trans- lation with language-independerr independent constraints. In Pro- ceedings of the Fourth Conference Machine Trans- lation (Volume 1: Research Papers), pages 13-23, Florence, Italy. Association for Computational Lin- guistics.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng. and Mingx- uan Wang. 2022. Stemm: Self-learning with speech- text manifold mixup for speech translation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 7050-7062.
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2020. manguage-agnostte BERT sentence embedding. CoRR, abs/2007.01
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert Sentence embeddings using siamese bert-networks. CoRR, abs/1908. 10084.
Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. Ctc- based compression for direct speech translation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 690-696.
Matthias Sperber and Matthias Paulik. 2020. Speech translation and the end-to-end promise: Taking stock of where we are. arXiv preprint arXiv:2004.065 0..558.......
Yuanhang Su, Kai Fan, Nguyen Bach, C-C Jay Kuo, and Fei Huang. 2019. Unsupervised multi-modal neural machine translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10482-10491.
Chi Han, Mingxuan Wang. Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text

---

Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Ta- tiana Likhomanenko, Edouard Grave, Vineel Pratap. Anuroop Sriram, Vitaliy Lipichinsky, and Ronan Col- lobert. 2019. End-to-end ast: from supervised to semi-supervised learning with modern architectures. arXiv preprint arXiv: 191 1.08460.
Linlin Zhang, Kai Fan, Boxing Chen, and Luo Si. 2023. A simple concatenation can effectively im- prove speech translation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Shori Papers), pages 1793- 1802.
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving speech transla- tion by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 4252-4266
Pei Zhang, Niyu Ge, Boxing Chen, and Kai Fan. 2019. Lattice transformer for speech translation. Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties, pages 6475- 6484.
Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. 2021. Fused acoustic and text encoding for multimodal bilingual pretraining and speech transla- tion. In International Conference on Machine Learn- ing, pages 12736-12746. PMLR.
Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021b. A general multi-task learn- ing framework to leverage text data for speech to text tasks. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 2209-6213 IEEE
A
Appendix
A.1
WRD based IPOT
Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, Shuai Zhang. and Zhenggi Wen. 2020. bpike-triggered non-autoregressive transfomeer for end-to-end speech recognition. arXiv preprint ar XrOiv:2005.0 2005:000..07033.
Algorithm 1: WRD based IPOT
Input: Maximum iterations T 50,
encoded sequences
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processsinng systems, 30.
{e] {e)))=-- Initialize p with p; = E:= [e
||
Initialize q similar to p.
Initialize C Ci. = cos(ef, e¹).
Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. 2020. A fast proximal point method for computing exact wasserstein distance. In Un- certainty in artificial intelligence, pages 433-453. PMLR.
4 T - 1 s σ 1 m. G.j =c Ci.j. for t = 1,2,
do
Q GOGT T
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju. Tong Xiao, and Jingbo Zhu. 2021. Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation en- coders. In ACL.
p . 0- Q T diag())Qdiag(o) 10 return (C, T)
8
8
9
Kong Yawei and Kai Fan. 2021. Probing multi-modal machine translation with pre-trained language model. In Findings of the Association for Computational Linguistics: ACL-IJCNIP 2021, pages 3689-3699.
IPOT replaces the Bregman divergence (x, = Σi= xy log z E x + > yi with the proximal point iteration, i.e., substitutes the following iterative update for the original optimization problem.
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- End Speech Translation via Cross-Modal Progressive Training. In Proc. Interspeech 2021, pages 2267- 2271.
T (1+1) arg min Main(C, (C,T) + DT, (6) T20
Cheng Yi, Feng Wang, and Bo Xu. 2019. Ecte-docd: An end-to-end structure with ctc encoder and ocd decoder for speech recognition. In INTERSPEECH. pages 4420-4424.
Algorithm shows the detailed implementation of IPOT, where diag(ō) represents the diagonal matrix with 5, as its i-th diagonal element, and O and 8 denote the element-wise matrix multipli- cation and division respectively. The algorithm outlines the forward- steps only. Since each iteration of the algorithm only involves dif- ferentiable operators, we can utilize the automatic
Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. 2020. Word rotator's distance. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2964,,960. Online, Association for Computa- tional Linguistics.

---

differentiation packages (e.g. PyTorch) to back- propagate the gradients like an unrolled RNN, The corresponding implementation can refer to the sub- mitted software.
data 2, The has 2,000 images with Chinese transcriptions and English translations. The BLEU on the test set for the pre-trained NMT 15.87, which not high due to the domain shift.
In particular, we set different weights Awrd = 0, 1. 5, 10, 20, 50 to investigate the effectiveness of the WRD loss, where the model with Awrd 0 reduces to cascade model. The results of the zero-shot Image Translation are shown in Figure 4. illustrates the intuition of how to tune the im- portance of WRD loss. In Figure 5, we visualize the transport plan T and cost matrix C some examples.
A.2 Additional Training Details
As model design in Figure 1, the embedding layer in the adapter shares the weights with the seman- tic source embedding. The beam size is 5 during inference, and we use SacreBLEU in fairseq as evaluation metrics.
For the third phase, supervised ST training, we have multiple tasks in the final objective. For the ST task Lst. some previous works may leverage the MT model and the Librispeech transcription to construct pseudo translation sentences. However, we only use the audio and translation pairs from MuST-C. For the ASR task Letes we only use the audios and transcriptions from MuST-C. For the MT task Lm.. we optimize it on both the MuST-C parallel corpus and WMT data, making the decoder better language model. En-De WMT only has 4.5M sentence pairs and the entire training is still manageable. However, for En-Fr/Es, optimizing the large end-to-end ST model with a huge amount of trainable parameters will cumbersome be- cause the size of WMT data overwhelmingly slows down the training. Therefore, we randomly sample a 10M corpus from the original WMT En-Fr/Es data to train the final supervised loss.
A.3 Additional Experimental Results of ST
Figure 3, we plot the WER scores of the sec- ond training stage, in both zero-shot and pscudo zero-shot settings. The ASR of the cascade sys- tem (i.e., trained with CTC loss only and without semantic encoder) has clearly higher WER than proposed ASR training with additional WRD loss. However, the in-domain MuST-C data do not appear to make significant difference indicated by the orange and the green bars in Figure 3.
A.4 Generalization and Visualization
We also conduct an experiment on zero-shot Image Translation using only OCR data and NMT data to further test the effectiveness of our framework. It is also convenient visualize with image data. The NMT model (i.e. the semantic encoder and decoder) is pre-trained on the WMT 2018 Zh-Fn data (20M) parallel sentences the news domain. We crop 2M text-line images from Chinese OCR
Phetps://gitt thu. www./ccaas/cciiesss.ooo and htttps//ttabbat

---

WER on COMMON
WER on HE
14.0
12.0
ASR for cascade ST
13.5
11.5
ASR for zero-shot ST
ASR for pseudo zero-shot ST
13.0
11.0
WER
12.5
WER
10.5
12.0
10.0
11.5
9.5
11.0
9.0
10.5
8.5
En-De v2
En-De
En-Fr
En-Es
En-De v2
En-De
En-Fr
En-Es
Figure 3: The performance of the ASR as zero-shot ST systems.
Per formance of zero-shot Image Trans lat on
WER
22.5
BLEU (right)
14
20.0
17.5
MER
12.5
20
Weight of WRD
loss
Figure The performance of OCR and zero-shot shot image translation over different weights Awrd-

---

0.12
0.08
00.0
0.6
0.4
-0.2
<eos>
00.00
<bos> 25
<eea>
<eos>
1.2
00,10 10
1..
00.08
-0.8
0. 06
-0.8
0. 04
-0.4
yeos>
<eos>
-0.2
<boo>
1H]
<600>
AP
18
W
1.2
X
0.08
1.0
0.8
0.6
0. 04
69
0
0.2
(oor>
Zooc>
<ooo>
Figure 5: Left Column: visualization of transport plan T Right Column: visualization of cost matrix C. The WRD with OT solver can align ross-modal features with different lengths. y-axis represents the shrunk features. and z-axis represents the transcription features. Since the actual shrunk tokens only represent a 4-pixel wide part of an image, we cut images along blank tokens as a schematic representation. The real shrunk tokens are usually in the middle of the images y-axis. For the cost matrix, the smaller elements mainly distributed the diagonal block regions. It could be the incorrect shrinking segments sometimes aligning with more than one character. For the transport plan matrix, the larger elements are mainly distributed on the diagonal. this way, their products will remain small.


Reviews:
[{"review_id": "u8FjeukAf2", "reviewer": "Reviewer_B6yQ", "summary": "", "questions": "In Table 1:\nDoes \"Ours cascade\" mean the baseline system without the shrink adapter that uses a word sequence (text) to connect a speech recognizer and a translation system?\n\nIn Equation (2), e_a = {e_1^a, ..., t_n^a):\nWhat is the difference between \"e\" and \"t\" in e_1^a and t_n^a?\n\nIn Abstract, \"without introducing any trainable parameters\":\nAre the parameters in the Shrink Adapter not counted here?\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comments. In general, we will fix all typos and make the statement more clear in the revised version.\n\n>Q1: In Table 1: Does \"Ours cascade\" mean the baseline system without the shrink adapter that uses a word sequence (text) to connect a speech recognizer and a translation system? \n\nYes. In the cascade system, the ASR (speech recognizer) and the NMT (translation system) have the same model architecture as our proposed end-to-end approach. Only the shrink adapter is removed. For fair comparison in the zero-shot setups, most previous works (e.g., MultiSLT [1], DCMA [2], Cascade versus Direct[3]) usually adopt similar architecture and similar model size between cascade and end-to-end systems. \n\n[1] MultiSLT: Enabling Zero-Shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoders, (ARSU 2021)\n\n[2] DCMA: Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation, (EMNLP 2022)\n\n[3] Cascade versus Direct: Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? (ACL 2021)\n\n>Q2: In Equation (2), $e_a = (e_1^a, ... ,t_n^a)$ What is the difference between $e$ and $t$ in $e_1^a$ and $t_n^a$? \n\nSorry, this is a typo. It should be $e_a = (e_1^a, ..., e_n^a)$. We will fix this typo in the revised version.\n\n>Q3: In Abstract, \"without introducing any trainable parameters\": Are the parameters in the Shrink Adapter not counted here?\n\nSorry for the unclear description. We originally mean that unlike other approaches that introduce additional trainable parameters during the end-to-end training stage, our adapter has already been pre-trained. Therefore, no additional parameters will be introduced in the fine-tuning stage. We will clarify this statement in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a new method to connect a speech recognizer and a text translation system using an embedded representation that can work without requiring end-to-end data, which is often limited.", "reasons_to_reject": "The text needs major revision for readability. It isn't easy to understand the details. They do not explain many of the variables and the Table entries.", "questions_to_authors": "In Table 1:\nDoes \"Ours cascade\" mean the baseline system without the shrink adapter that uses a word sequence (text) to connect a speech recognizer and a translation system?\n\nIn Equation (2), e_a = {e_1^a, ..., t_n^a):\nWhat is the difference between \"e\" and \"t\" in e_1^a and t_n^a?\n\nIn Abstract, \"without introducing any trainable parameters\":\nAre the parameters in the Shrink Adapter not counted here?\n\n\n", "paper_topic_and_main_contributions": "The authors propose a zero-shot learning method for end-to-end speech translation models. Unlike the cascade models that feed a token (word) sequence output from a speech recognition system to a text-based translation system, the proposed method connects the two systems using a variable-length embedding representation. We can expect robustness to speech recognition errors by using the embedding representation. Their method can train the embedding representation using independent speech recognition and machine translation training data, realizing zero-shot learning without requiring speech-transcript-translation triplet supervision data. It is also possible to fine-tune the system if triplet supervision data is available. The experiments show that the proposed method can match or be slightly better than the CTC-based cascade model. The results of end-to-end training also match the recent supervised speech translation baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Introduction \"with OT solver:\":\n The authors should describe full spelling when they first introduce an abbreviation.\n\nSection 3.1:\nThe cross-model encoder: cross-modal (?)\n\nSection 3.3:\nTable ??\n\n"}, "extra_text": {}}, {"review_id": "R7AZAlPVpx", "reviewer": "Reviewer_aFGi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comment. We will address the main conern \"Table 2 presents a limited number of new end-to-end ST models for comparison.\" \n\nAs the main research area and contribution of our work is zero-shot end-to-end ST, it means the transcriptions in ASR dataset and the source sentences in MT dataset have no overlap (as shown in Figure (b)). However, most end-to-end ST models are not trained in the zero-shot setup. To our best knowledge, we have already included the relevant works in our paper. \n\nSupervised continue training is a by-product of our work, but most of them are not comparable with ours. Because they are directly trained in supervised or semi-supervised learning rather than supervised continue training from a zero-shot model. \n\nIn Table 2, we have listed some recent supervised ST models with similar model size. We still appreciate your suggestions and will include more supervised ST in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a new approach to address the mismatches between different modalities and scarcity of parallel data. Moreover, the experiments were conducted thoroughly, and the model diagrams were beautifully illustrated, making it a good short paper.", "reasons_to_reject": "Table 2 presents a limited number of new end-to-end ST models for comparison. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about End-to-end Cross-modal Translation for Zero-shot.\n\nMain contributions:They propose an end-to-end zero-shot speech translation model to adress the mismatches between different modalities and scarcity of parallel data. Adopting the WRD loss together with the shrink mechanism to measure two feature sequences in different lengths, enabling the adapter pre-training.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rFNE90wbs3", "reviewer": "Reviewer_kgo5", "summary": "", "questions": "A. As mentioned in the weaknesses of the paper, it is understandable that the selected submodules in the cascade models may not be the optimal ones. However, it is evident that the chosen modules are significantly inferior compared to the state-of-the-art (SOTA) counterparts. This raises concerns about the effectiveness of the proposed E2E (End-to-End) model.\nIf we were to employ the SOTA submodules from ASR and MT, along with a much larger amount of data within each modality, the question arises: do the advantages of the proposed method still hold true? \nIn addition, the reference and more detailed description of each submodule is missing, which makes it hard in terms of reproducibility.\n\nB. In each table, what is the common and he? please clarify.\n\nC. While the authors present the ablation study in the table shown in Figure 2, the observed differences between the results may be small. The question arises whether the authors conducted any significance tests to determine the statistical significance of these differences.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments. We will fix the typos in the revised version. The concerns will be addressed first, then the question will be responded. \n\n>Concern 1: SOTA ASR and MT are not used in cascade system.\n\nThanks for raising this issue. In the research line of zero-shot end-to-end ST, the main purpose is not to compare with a cascade system with both SOTA ASR and NMT models. As previous works (e.g., MultiSLT [1], DCMA [2]), similar architecture and similar model size between cascade and end-to-end systems are usually adopted. In this way, two systems can be fairly compared, e.g., the paper \"Cascade versus Direct\"[3] applied systematic comparison between two systems in similar transformer architecture without language model rescoring. In summary, training two systems to approach SOTA but with different architectures and different datasets did not fall into this research area.\n\n[1] MultiSLT: Enabling Zero-Shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoders, (ARSU 2021)\n\n[2] DCMA: Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation, (EMNLP 2022)\n\n[3] Cascade versus Direct: Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? (ACL 2021)\n\n>Concern 2: Considering the comparable performance or slightly better performance on the proposed e2e settings, it could be hard to demonstrate the effectiveness.\n\nThanks for proposing this concern. We will like to address our title or main contribution is exploring the zero-shot end-to-end ST system, and the main result is shown in Table 1. We extracted the the last column from Table 1 in our paper.\n|| Average GAP|\n|:----------------|:----:|\n|previous SOTA|-13.79 / -1.325|\n|ours|+0.79 / +0.78|\n\n**GAP**: BLEU of zero-shot end-to-end system $-$ BLEU of zero-shot cascade system. \nWe can clearly see our approach achieve significant improvement to close the gap between zero-shot cascade and end-to-end systems.\n\n**Questions**\n>A. As mentioned in the weaknesses of the paper, it is understandable that the selected submodules in the cascade models may not be the optimal ones. However, it is evident that the chosen modules are significantly inferior compared to the state-of-the-art (SOTA) counterparts. This raises concerns about the effectiveness of the proposed E2E (End-to-End) model. If we were to employ the SOTA submodules from ASR and MT, along with a much larger amount of data within each modality, the question arises: do the advantages of the proposed method still hold true? In addition, the reference and more detailed description of each submodule is missing, which makes it hard in terms of reproducibility.\n\nAs we explained in the response for concerns, utilizing the SOTA ASR and MT is not the main objective of this area. Our current adapter is designed based on two commonly used ASR and MT architectures,  a CTC-based model and a transformer encoder-decoder model. If similar architecture is used, our proposed adapter should be still valid, e.g., changing the transformer layer to conformer layer. If we use a SOTA model with very different architecture, we may have to re-design the adapter accordingly. \n\nFor the reproducibility, we submitted the code and we will open-source our code upon acceptance. The detailed implementation is in the code folder \"examples/dcm/\". We appreciate your suggestions and will introduce more details in the revised version.\n\n>B. In each table, what is the common and he? please clarify.\n\nIn the widely used ST dataset MuST-C, there are two testsets for each language pair. The dataset names of the two testsets are \"common\" and \"he\", where \"common\" means common talks and \"he\" means an additional testset.\n\n>C. While the authors present the ablation study in the table shown in Figure 2, the observed differences between the results may be small. \nThe question arises whether the authors conducted any significance tests to determine the statistical significance of these differences.\n\nWe did not conduct any significance tests on the ablation study in Figure 2, because our work did not focus on the supervised learning. Our main contribution is the zero-shot end-to-end learning, while supervised continue training is a by-product. Note that supervised continue learning is different from direct supervised learning (other methods shown in Table 2). In addition, the last column in Table 1 shows the gap or the improvement of our approach is significantly better than previous works in relevant research area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes the utilization of the Word Rotator's Distance loss and a shrink adapter to address the multi-modal discrepancy, which is the major novelty of this paper. Interesting results are shown to convince the effectiveness of the method. ", "reasons_to_reject": "The paper is lacking crucial details, and proving the generalization of the proposed method might be difficult due to the selection of different submodules (as indicated in the questions below).\n\nFor example, the ASR system, based on the limited description in the paper, utilizes a CTC loss with some transformer encoder layers, which is considerably weaker compared to the state-of-the-art (SOTA) ASR modules. For instance, using Conformer layers in the encoder has demonstrated significantly better performance on the spectrum input. Furthermore, relying solely on CTC without a language model can result in poor ASR output performance. Similar concerns apply to the MT (Machine Translation) sub-system.\n\nConsidering the comparable performance or slightly better performance on the proposed e2e settings and the selection of submodules, it could be hard to demonstrate the effectiveness of its proposed approach. \nIn addition, based on the results in the table (advantages compared to other existing work), experiments are not convincing enough.", "questions_to_authors": "A. As mentioned in the weaknesses of the paper, it is understandable that the selected submodules in the cascade models may not be the optimal ones. However, it is evident that the chosen modules are significantly inferior compared to the state-of-the-art (SOTA) counterparts. This raises concerns about the effectiveness of the proposed E2E (End-to-End) model.\nIf we were to employ the SOTA submodules from ASR and MT, along with a much larger amount of data within each modality, the question arises: do the advantages of the proposed method still hold true? \nIn addition, the reference and more detailed description of each submodule is missing, which makes it hard in terms of reproducibility.\n\nB. In each table, what is the common and he? please clarify.\n\nC. While the authors present the ablation study in the table shown in Figure 2, the observed differences between the results may be small. The question arises whether the authors conducted any significance tests to determine the statistical significance of these differences.", "paper_topic_and_main_contributions": "The paper proposed an E2E speech translation model designed for a zero-shot setting, considering the scarcity of parallel data during training. Specifically, the speech translation model is constructed by integrating the word rotator's distance loss to connect two pretrained uni-modality parts: ASR (Automatic Speech Recognition) and the Machine Translation (MT) modules. \n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In 3.3 table ref is shown as ???"}, "extra_text": {}}, {"review_id": "diCApPY2im", "reviewer": "Reviewer_ZEUx", "summary": "", "questions": "Question A: Certain design choices are not clear to me. AFAIK best path taken during CTC training can be from all possible CTC alignments instead of being just argmax of choices present at current stage? Why authors particularly chose argmax?\nQuestion B: I felt comparison with cascade based models in which end translation decoder trained end to end would have been more fair.\nQuestion C: I felt comparison of latency with earlier cascade based models could have been good to bring out novelty better.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive comments. We will include the suggested references and describe the details of dataset in the revised version. For the results in Table 1 and 2, we tried our best to keep the experimental setting the same. In the caption of each Table, we've marked the difference against our setting. However, not all papers release their code and checkpoint, so the marks in the caption are mainly based on the description in the original papers.\n\n**Questions**\n>Question A: Certain design choices are not clear to me. AFAIK best path taken during CTC training can be from all possible CTC alignments instead of being just argmax of choices present at current stage? Why authors particularly chose argmax? \n\nThanks for the great suggestion. It's definitely correct and better to use the best CTC path from all possible alignments. In our work, we use $\\arg\\max$ for efficiency. Because $\\arg\\max$ is more friendly for parallelization, leading to faster training speed. In contrast, even dynamic programming is used in searching all possible paths is not efficient for training purpose. However, we appreciate this suggestion and will explore its possibility for inference.\n\n>Question B: I felt comparison with cascade based models in which end translation decoder trained end to end would have been more fair. \n\nWe apologized that we are not sure what exactly this question means. The cascade system includes two independent models: ASR and NMT. The ASR is trained on audio-transcription data, and the NMT is an encoder-decoder architecture trained on parallel (source-target) text corpus. In the zero-shot setting, the source text in MT data and the transcription in ASR data have no overlap. Due to the unavailability of audio-target data, the translation decoder cannot be trained end-to-end.\n\nOur explanation is based on our understanding, and we are open to provide more response during the discussion window.\n\n>Question C: I felt comparison of latency with earlier cascade based models could have been good to bring out novelty better.\n\nThanks for this suggestion. As we mentioned earlier, it is difficult for us to access earlier cascade based models for either code repository or checkpoint. Similarly, for the end-to-end models, it is also difficult to access their resources, so we list the number of parameters. \n\nWe submitted our code as the supplementary material and will open-source our code upon acceptance. We would like to add the latency of our own model in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I really like how end to end model using models from two different domains ASR and MT aligned by word rotator distance. \n2. Problem formulation is well defined and afaik results seems to perform better or as good as current CTC cascade based models without any added overhead of existing cascade models in terms of latency.\n\n", "reasons_to_reject": "1. Paper is hard to follow sometimes. I felt paper has missed some important reference. Like CTC is mentioned multiple times like in line 019, 069, 083 but no reference is provided making it difficult for readers from different domain to follow. Second ST dataset is not properly described in section 3.1 leading to confusion what common and he means in Table 1,2.\n2. For the reported results in table 1 and table 2, how results are obtained for prior work is not clear to me. Is it ensured that all experiments are performed in same settings as in prior works?", "questions_to_authors": "Question A: Certain design choices are not clear to me. AFAIK best path taken during CTC training can be from all possible CTC alignments instead of being just argmax of choices present at current stage? Why authors particularly chose argmax?\nQuestion B: I felt comparison with cascade based models in which end translation decoder trained end to end would have been more fair.\nQuestion C: I felt comparison of latency with earlier cascade based models could have been good to bring out novelty better.", "paper_topic_and_main_contributions": "This paper proposes a novel method for end to end speech translation using only machine translation (MT) and automatic speech recognition (ASR) data in zero shot setting as well as same in fine-tune setting with speech translation (ST) data. For zero shot setting ASR and MT models are trained by using Word Rotator Distance (WRD) to align transcribed representation of speech with translation model in conjuction with usual CTC loss on CTC based speech encoder. For fine tune setting translation model can be trained with source transcripts and target transcripts of ASR data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 248 has missing table reference. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a zero-shot learning approach that bridges the gap between cascade and end-to-end speech translation models using a differentiable shrink adapter and WRD loss.

The reviewers' evaluations are mostly consistent, showing moderate soundness and relatively high excitement scores.
The reviewers are concerned about the readability to understand the details, limited comparisons, and the lack of SOTA ASR system, e.g., Conformer-CTC with an external language model.

The authors answered the reviewers' questions regarding technical details, and the answers sound reasonable to me. 
The authors also explained that the focus of the paper is to bridge the gap between the cascade and end-to-end systems and not to achieve SOTA performance.
This is understandable, but I think that evaluating using Conformer-CTC with a language model would have yielded more reliable results. However, the topic the paper addresses is interesting. The reviewers rated high excitement scores. So, it will be of interest to a large audience. If accepted, I hope that the authors will try to improve the readability as pointed out by the reviewers.
Decision: Accept-Findings