AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
Authors: Xingshan Zeng
Liangyou Li
Qun Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: speech translation
modality adaptation

Abstract: To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.

AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
Xingshan Zeng, Liangyou Li, Qun Liu
Huawei Noah's is Ark Lab
(zeng. xingshan, 1il willllllan liliangyou qun liu] @huawei. com
Abstract
eficial to learn desired features (Liu et al., 2020; Zeng et al., 2021). Initializing the two encoders by pre-trained ASR and MT encoders, respectively, can significantly boost the performance (Xu et al., 2021). However, the modality gap between speech and text might prevent the ST models from effec- tively inheriting the pre-trained knowledge (Xu et al., 2021).
To alleviate the data scarcity problem in End- to-end speech translation (ST), pre-training on data for speech recognition and machine trans- lation is considered as important technique. However, the modality gap between speech and text prevents the ST model from efficiently in- heriting knowledge from the pre- trained mod- els. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experi- ments on the MUST-C dataset demonstrate that Ada TranS achieves better performance than other shrinking- based methods, with higher in- ference speed and lower memory usage. Fur- ther experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.
The modality gap between speech and text can be summarized as two dimensions. First, the length gap the speech features are usually much longer than their corresponding texts (Chorowski et al., 2015; Liu et al., 2020). Second is the representa- tion space gap. Directly fine-tuning MT parameters (semantic encoder and decoder) with speech fea- tures as inputs, which learned independently, would result in sub-optimal performance. Previous work has explored and proposed several alignment objec- tives to address the second gap, e.g., Cross-modal Adaption (Liu et al., 2020), Cross- Reg- ularization (Tang et al., 2021a) and Cross-modal Contrastive Learning (Ye et al., 2022).
1 Introduction
End-to-end speech translation (ST), which directly translates source speech into text in another lan- guage, has achieved remarkable progress in recent years (Berard et al., 2016; Duong et al., 2016; Weiss et al., 2017; Berard et al., 2018; Xu et al., 2021; Ye et al., 2022). Compared to the conventional cascaded systems (Ney, 1999; Mathias and Byrne, 2006), the end-to-end models are believed to have the advantages of low latency and less error propa- gation. well-trained end-to-end model typically needs large amount of training data. However, the available direct speech-translatinn corpora are very limited (Di Gangi et al., 2019). Given the fact that data used for automatic speech recogni- tion (ASR) and machine translation (MT) are much richer, the paradigm of "pre-training ASR and MT data and then fine-tuning on ST" becomes one of the approaches alleviate the data scarcity prob- lem (Bansal et al., 2019; Xu al., 2021).
A shrinking mechanism is usually used to ad- dress the length gap. Some leverage Continuous Integrate-and-Firg (CIF) (Dong and Xu, 2020) shrink the long speech features (Dong et al., 2022; Chang and Lee, 2022), but they mostly work on simultaneous ST and need extra efforts to perform better shrinking. Others mainly depend on the CTC greedy path (Liu et al., 2020; Gaido et al., 2021), which might introduce extra inference cost and lead to sub-optimal shrinking results. AdaTranS uses new shrinking mechanism called boundary-based shrinking, which achieves higher performance.
Through extensive experiments on the MUST- C (Di Gangi et al., 2019) dataset, we show that Ada- TranS is superior to other shrinking-based methods with a faster inference speed or lower memory us- age. Further equipped with alignment objectives, AdaTranS shows competitive performance com- pared to the fatte-o-thhe-ar models.
has been shown that decoupling the ST en- coder into acoustic and semantic encoders is ben-

---

CTC Loss
Pred Loss
ST Loss
2.3 Boundary-based Shrinking Mechanism
CTC Module
Guide Boundary
Semantic Encoder
Previous shrinking mechanisms (Liu et al., 2020; Zeng et al., 2021) mostly depend on a CTC mod- ule (Graves et al., 2006) to produce token-label probabilities for each frame in the speech represen- tations. Then, a word boundary is recognized if the labeled tokens of two consecutive frames are different. There are two main drawbacks to such CTC-based methods. First, the word boundaries are indirectly estimated and potentially affected by error propagation from the token label predictions which are usually greedily estimated by the argmax operation the CTC output probabilities. Second, the token labels are from large source vocabular resulting in extra parameters and computation cost in the CTC module during inference.
Predictor
Softmax
ccussic Encoder
Shrink
Decoder
f
Source Speech
Target Text
Ada TranS Architecture, where the blue mod- ules are initialized with the ASR model and the orange modules are initialized with the MT model. The CTC module (dotted) can be removed during inference.
Figure I:
2 Proposed Model: AdaTrank
2.1 Problem Formulation
An ST corpus is denoted as Dsr ((2r, y)) containing triples of speech, transcription and trans- lation. Here x = (æ1, X2, ..., IT2)) ) a sequence of speech features or waves as speech input, while = (#1, #2, ...,, 2T,) and y = (y1, (u,,y,.... y2...., BT,,) are the corresponding transcription in source language and translation in target language, respectively. T,, T2, and Ty are the lengths of speech, transcrip- tion and translation, respectively, where usually Te > Tg and Tę Tg.
We introduce a boundary- anism to address the two drawbacks. A boundary predictor is used to directly predict the probability of each speech representation being a boundary. which is then used for weighted shrinking. Since the boundary labels on the speech representations are unknown during training, we introduce signals from the CTC module to guide the training of the boundary predictor. The CTC module will be dis- carded during inference. Below shows the details.
shrinking mech-
2.2 Architecture
AdaTranS decouples the ST encoder into an acous- tic encoder and a semantic encoder. To bridge the modality gap between speech and text, an adaptor is usually needed before the semantic encoder. We choose the shrinking operation (Liu et al., 2020; Zeng et al., 2021) our adaptor, where the long speech sequences are shrunk to the similar lengths the transcription based on designed mechanisms (details will be introduced in the next subsection). The shrunk representations are sent to the semantic encoder and ST decoder for output:
CTC module. We first briefly introduce the CTC module. It predicts a path æ = (TT, T,,-.., TTy ) where Tz is the length of hidden states after the acoustic encoder. And E V VU( (Ø) can be either token in the source vocabulary V or the blank symbol ø. By removing blank symbols and con- secutively repeated labels, denoted an operation B, we can map the CTC path to the corresponding transcription. A CTC loss is defined as the proba- bility of all possible paths that can be mapped to the ground-truth transcription 2:
Tg
LCTC = Σ
Σ xCB-1(2)
log log((T/())
(2)
Cst
log p( (1)
[Darl #CB
Der
CTC-guided Boundary Predictor. We propose to use a boundary predictor to replace the CTC module, which has similar architecture but with only three labels. The three labels are <BK> (blank label), <BD> (boundary label) and <OT> (others). respectively. However, the ground-truth labels for training the predictor are unknown. Therefore, we introduce soft training signals based on the output probabilities the CTC module. Specifically, the ground-truth probabilities of each frame t to be
To incorporate extra ASR and MT data, we use the pre- pre-trained ASR encoder to initialize the ST acoustic encoder, and the pre-trained MT encoder and decoder to initialize the ST semantic encoder and decoder, respectively. Both pre-trained models are first trained with extra ASR (or MT) data and then fine-tuned with the in-domain data (the ASR part or MT part in the ST corpus). Figure displays our architecture as well as the training process.

---

Graani
0
B
B
Forced Training. We introduce forced train- ing trick explicitly solve the length mismatch between speech and text representations. During training, we set the threshold dynamically based on the length of 2 to make sure the shrunk repre- sentations have exactly the same lengths as their corresponding transcriptions. Specifically, we first sort the probabilities to be <BD> of all frames in descending order, and then select the T--th one the threshold B.
trush
AK00540
#R)0-10)
ARCALO)
AK00330 3m
#ксол))
Productrr
(710.19)
07(0225)
07(0.24)
057025)
07/034)
03(0.0
R(A.)
810.13
810.00
000.0
Bro.00
8:0.0
CTC
RVO.))
8(0.3)
#1033)
BL0.36
#10.0)
CQUT)
EQ0-2)
600.000
((03))
(Q0.4))
(Q10)
Speech Feslures
Figure 2: An example of CTC probabilities (we assume there are three tokens in vocabulary for simplification), together with the corresponding boundary predictor soft signals and ground labels. We will get different bound- ary detection results based on their label probabilities (the solid and dotted lines). CTC argmax path predicts the wrong boundaries and we get correct boundaries if we set the threshold 0 as 0.5 in boundary predictor.
2.4 Training Objectives
The total loss of our AdaTranS will be:
C LST + a LCTc + B- Pred
(6)
labeled as the three labels are defined as:
where a, 3 are hyper parameters that control the effects of different losses.
f/(<BK>)= = p( =
<((<BD>)=> - ipp p( = 1)p(TT+1 #1
(3)
3 Experiments
14(<OT>) =1
(11-((<BK>)) -1- pb(<BD>)
3.1 Experimental Setup
We conduct experiments on three lan- guage pairs of MUST-C dataset (Di Gangi et al., 2019): English-German (En-De), English-French (En-Fr) and English- Russian (En- (En-Ru). We use the official data splits for train and development and tst- LL-COMMON for test. We use LibriSpeech (Panay- otov et al., 2015) to pre-train the acoustic model. OpenSubtidles0118 18 or WMT14 are used to pre- train the MT model. The data statistics are listed in Table of Appendix A.
Datasets.
Then, the objective for the boundary predictor! is:
7
Pred ΣΣX
p1(11 logp:(1) (4)
[D5rl t=licA where A {<BK>, <BD>, <OT>}. The CTC
module only used in the training process and can be discarded during inference. Since the num- ber labels in the predictor significantly smaller than the size the source vocabulary, the time and computation costs introduced by the predictor are negligible. Figure 2 shows an example to elaborate the advantage of such predictor.
Preprocessing, We use 80D log-mel filterbanks as speech input features and SentencePiece' (Kudo and Richardson, 2018) to generate subword vocab- ulary with a size of 16000 for each language pair. More details please refer the Appendix B.
Weighted Shrinking. For shrinking, we define boundary frames as those with the probabilities of the <BD> label higher than a pre-defined threshold 0. The frames between two boundary frames are defined as one segment, which can be aligned to one source token. Inspired by Zeng et al. (2021). over the frames in one segment weighted by their probabilities of being blank labels to dis- tinguish informative and non-informative frames:
Model Setting, 2020) or Conformer (Gulati et al., 2020) (results in Table 2 are achieved by AdaTranS with Con- former, while the remaining results utilized Conv- Transformer) is used as our acoustic encoder, both containing 12 layers. For the semantic encoder and ST decoder, we follow the general NMT Trans- former settings (i.e., both contain 6 layers). Each Transformer layer has an input embedding dimen- sion of 512 and feed-forward layer dimension of 2048. The whole model contains 107M (140M if
Con--Transformer (Huang et al..
exp(u(1 Pc(<BK>))
Σ
1 Σ Es exp(p(1 Ps (BK>)))
tEseg 1
(5)
where μ > 0 denotes the temperature for the Soft- max Function.
http:///wpss.allleeeepppuubbbbes 9 hhtps:://wwwstttmmm wttp:////wwwwttttt....m wtwp://m..w.gg.mmttt/aaaaaattton https///ghhbbaoooomoooooo/leeeee
Since the training of the predictor highly depends on the quality of the CTC output, the CTC module is also pre-trained.

---

D (%)
Mem Usage
BLEU En-De
BLEU
Model
Speedup
Model
En-Fr
En-De
En-Fr
En-Ru
Shrink
1.00x
26.0
36.8
MT
34.4
44.9
21.3
Fix Shrink
36.7
1.06x
0.74
25.4
36.0
Cascaded Model
28.1
37.0
17.6
CIF-Based
70.3
1.04x
0.74
25.8
36.2
JT-S-MT (Tang al., 2021a) a)
26.8
37.4
CTC- Based
80.2
0.76x
1.77
26.4
36,9
Chimera (Han et al., 2021)
27.1
35.6
17.4
Boundary- -aased
26.7
37.4
XSTNet (Ye al., 2021)
27.1
38.0
18.5
Fooreed Train
81.9
1.06
0.78
26.4
37.1
SATE (Xu et al., 2021)
28.1
Blankk Label
26.3
36.6
STEMM (Fang al., 2022)
28.7
37.4
17.8
ConST (Ye et al., 2022)
28.3
38.3
18.9
Table The results of shrinking-based methods and the corresponding shrinking quality, evaluated with length differences between the shrunk representations and tran- scriptions. D means the length differences are less than or equal to 2. The speedup and memory usage are both tested with batch size of 16, and we only display the relative values for clear comparison.
AdaTranS
28.7
39.2
19.3
-Boundayy bbased Shrink
28.1
38.8
18.8
Table 2: Comparisons with the SOTA models. The first
two rows are the results with our pre-trained MT model and cascading pre-trained ASR and MT models.
efficiency (lower inference speed and higher mem- ory usage) as they introduce extra computation cost producing greedy CTC path a large source vocab- ulary. Our method performs the best in both shrink- ing and translation quality with nice inference ef- ficiency. On the other hand, we also notice that removing forced training trick ("-Forced Train") or weighted-shrinke
use Conformer) parameters. The hyper- haper-parameters in Eq. 6 are set as: = 1.0 and / = 1.0, respec- tively. The temperature of the softmax function in Eq. 5 (1)) is 1.0, while the threshold in the boundary predictor is set to 0.4 during inference Training details please refer to Appendix B.
We apply SacreBLEU for evaluation, where case-sensitive detokenized BLEU reported.
(i.e., "-Blank Label", simply average the frame representations rather than Eq. 5) will affects the translation quality, showing the effectiveness of these two components.
3.2 Experiment results
Table 1 compares different Marinking-based meth- ods in terms of quality and efficiency (for fair com- parison, we use the same architecture pre-trained with the same data and add a CTC loss all the compared models). Besides translation quality, we use length differences between the shrunk repre- sentations and the corresponding transcriptions to evaluate shrinking quality following Zeng et al. (2021). We use inference speedup and memory usage to evaluate efficiency.
Adopting Alignment Objectives. AdaTranS can be further improved with objectives that align speech and text representations (i.e. bridging the representation space gap introduced in Section I). Table 2 shows the results of AdaTraaS equipped with Cross-modal Contrastive (Ye et al., 2022) and knowledge distillation (KD) guided by MT, com- pared to the models that also work on modality alignment objectives (more complete comparisons please refer to Appendix C). The results show that AdaTranS achieves competitive results in all three datasets. We also examine and show the effects of boundary-based shrinking in such setting (*- Boundary-base Shrink")..
For comparisons, the Fix-Shrink method shrinks the speech features with fixed rate (e.g. every frames). The CIF-Based method (Dong et al., 2022) is based on a continuous integrate-and-Fie mech- anism. The CTC- method (Liu et al., 2020) shrinks features based on CTC greedy paths. As can be seen, poor shrinking (Fix-Shrin) and CIF- based) hurts the performance, although with better efficiency. The boundary-bant based shrinking used in AdaTrans, and the CTC-based method achieve bet- ter shrinking quality, with performance improved. However, CTC- method hurts the inference
Influence of the Threshold. We also examine the threshold 0 for the boundary predictor. Figure 3 shows the distribution of the predicted boundary probability (i.e. Pr(<BD>)) for each frame. We find that the boundary predictor is confident (< 0.1 and 0.9) in most cases. However, even though only a small portion of predictions are in the range of 10. 0.9], they significantly affect the BLEU scores when the threshold changes (the red line in Figure 3). The model achieves the best perfor-
'll ayper-parametett are set through grid search based the performance of development set.

---

forms better than other shrinking-basse methods. in terms of both quality and efficiency. It can also be further enhanced by modality alignment objec- tives to achieve state-of fate-of-thheare f-thh-eart results.
ppccntageePCT.1g9 (PCT.)))
PCT.
50
BLEU
28
40
30
26
20
Limitations
24
8.0
0.1 0.2 0.3 0.4
0.5 0.6
0.7 0.8 0.9
1.0
Learning of the proposed boundary predictor. The learning objective of our boundary predictor is constructed by the soft labels from CTC objec- tive. Since the labels are not accurate labels, it is inevitable to introduce errors during training. How- ever, the groundtruth labels for boundary predictor is difficult obtain. One alternative is to use forced alignment tools. That also introduces other prob- lems. First, off-the-shelf foreed alignment tools only support speech in popular languages, which limits the use of the method to other languages. Second, forced alignment also doesn't 't guarantee the correctness of labeling, and we still need to fur- ther approximate the labeling results when apply- ing them to speech features after acoustic encoder (with 4x or 8x downsampling).
Boundary Probability (Threshoid)
Figure 3: The distribution of probabilities for boundary prediction, and the corresponding BLEU scores using different values as threshold in MUST-C En-De test set.
mance when the threshold is around 0.4.
4 Related Work
Numerous techniques have been proposed to adapt speech and text representations in order to miti- gate the modality gap in end-to-end ST. Wang et al. (2020) introduce noise into the text input, while Salesky and Black (2020); Tang et al. a,) employ phoneme sequences for text input. These approaches reduce differences between speech and text input by extending text representations. Con- versely, some research aims at shrinking lengthy speech input by presenting various shrinking strate- gies (Salesky et al., 2019; Dong et al., 2020, 2021; Liu et al., 2020; Gaido et al., 2021; Zeng al., 2021). Our work falls within this category, propos- ing a new approach that offers enhanced effective- ness and efficiency. Others prioritize aligning the speech-text representation space (Liu et al., 2020; Tang et al., 2021a; Xu et al., 2021; Ye et al., 2022; Zhang et al., 2022; Le et al., 2023). CTC alignment's role in enhancing ST is par- ticularly pertinent to our research. Earlier studies have demonstrated that integrating additional CTC loss for multi-task learning or pre-training assists ST model training (Wang al., 2020; Xu et al., 2021). Some also leverage CTC alignment to shrink speech inputs (Salesky et al., 2019; Liu et al., 2020; Gaido et al., 2021; Zeng et al., 2021). Moreover, Le et al. (2023) and Zhang et al. (2023) delve deeper. optimizing CTC objectives for ST.
Sensitivity of the selective threshold. From Fig- ure 3, we can find that the BLEU score is sensitive to the threshold selection, although the boundary predictor is confident in most cases. We leave it to our future work to alleviate this phenomenon.
References
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre-training on high-resouree speech recognition improves low- resource speech-to-text translation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 58 68, Minneapolis, Minnesota. Association Computational Linguisties.
Alexandre Berard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, pages 6224-6228. IEEE.
Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text trans- lation. CoRR, abs/16 0tts/1612.01770
5 Conclusion
This work proposes a new end-to-end ST model called AdaTranS, which uses a boundary predictor trained by signals from CTC output probabilities, to adapt and bridge the length gap between speech and text. Experiments show that AdaTranS per-
Chih Chih-Chiang Chang and Hung-yi Lee. 2022. Exploring continuous integrate-and-fin for adaptive simultane- ous speech translation. In Interspeech 2022, 23rd An- nual Conference of the International Speech Commu-

---

nication Association, Incheon, Korea, 18-22 Septem- ber 2022, pages 5179-5179. ISCA.
for Computational Linguistics: Main Volume, pages 190-696, Online, Association for Computational Lin- guistics.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-basee models for speech recognition. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Pro- cessing Systems 2015, December 7-12, 2015, Mon- treal, Quebec, Canada, pages 577-585.
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Machine Learning, Proceedings of the Twenty-Thirl Interna- tional Conference (ICML 2006), Pittsburgh, Pennsyl- vania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Seriex, pages 369-376, ACM.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli. Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume I (Long and Short Papers), pages 2012-200 2017, Min- neapolis, Minnesota. Association for Computational Linguistics.
Anmol Gulati, James Qin, Chung- -aengg Chiu, Niki Parmar, Yu Zhang. Jiahui Ya, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmentee former for speech recognition. In Interspeech 2020, 21 Annual Conference of the International Speech Communication Association, Virtual Event, Shang- hai, China, 25-29 October 2020, pages 4036-5040. ISCA.
trans-
Linhao Dong and Bo Xu. 2020. CIF: continuous integrate-and-Fie fire for end-to-end speech recognition. In 2020 IEEE International Conference on Acous- fics, Speech and Signal Processing. ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 6079-6083. IEEE.
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages Online. Association for Computational Linguistics.
Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2022. Learning when to translate for streaming speech. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 680-694. Association for Computational Linguistics.
Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen. 2020. Conv-transformer transducer: Low latency, low frame rate, streamable end-to-end speech recognition. In Interspeech 2020, 21 Annual Con- ference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 5001-5005. ISCA.
Qianqian Dong, Mingxuan Wang. Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2020. Consecutive decoding for speech-to-text translation. CoRR, abs/2009.09737..
Qianqian Dong, Yaoming Zhu, Mingxuan Wang. and Lei Li. 2021. Unist: Unified end-to-end model for streaming and non-streaming speech translation. CoRR, abs/2109.07368.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 949-959, San Diego, California. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018 SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties.
Qingkai Fang, Rong Ye, Lei Li. Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). Association for Computational Lin- guistics.
Bamoogg-Hng Le, Hongyu Gong, Changhan Wang. Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023, Pre-training for speech translation: CTC meets opti- mal transport. In International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 18667-18685. PMLR.
Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. CTC-based compression for direct speech translation. In Proceedings of the 16th Con- ference of the European Chapter of the Association
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the modality gap for speech- to-text translation. CoRR, 22..20000.49900

---

Lambert Mathias and William Byrne 2006. Statistical phrase-base speech translation. In 2006 IEEE Inter- national Conference on Acoustics Speech and Signal Processing, ICASSP 2006, Toulouse, France, May 14-19, 2006, pages 561-564. IEEE
lation. In The Thirty-Fourth AAAI Conference on Ar- tificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9161-9168. AAAT Press.
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. eequence-to-sqquene models can directly transcribe foreign speech. CoRR, abs/1703.0m 703.08581.
Hermann Ney. 1999. Speech translation: coupling of recognition and translation. In Proceedings of the 1999 IEFE International Conference on Acoustics, Speech, and Signal Processing, ICASSP '99, Phoenix, Arizona, USA, March 15-19, 1999, pages 517-520. IEEE Computer Society.
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingbo Zhu. 2021. Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation coders. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natu- ral Language Processing (Volume I: Long Pupers). pages 2619-2630 Online. Association for Computa- tional Linguistics.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, pages 5206-5210. IEEE
Rong Ye, Mingxuan Wang. and Lei Li. 2021. End-to- end speech translation via gross-modal progressive training. In Interspeech 2021, 22nd Anmual Con- ference of the International Speech Commuicatioo Association, Brno, Czechia, 30 August 3 September 2021, pages 2267-2271. ISCA.
Blizabeth Salesky and Alan W Black. 2020. Phone features improve speech translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 2388 2397, Online. Association for Computational Linguistics.
Elizabeth Salesky. Matthias Sperber, and Alan W Black. 2019. Exploring phoneme-level speech representa- tions for end-to-end speech translation. In Proceed- ings the 57th Annual Meeting of Association for Computational Linguistics, pages 1835-1841, Flo- rence, Italy. Association for Computational Linguis- tics.
Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5099-5113 13. Association for Computational Linguistics.
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang. Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics /Volume 1: Long Papers). Association for Computational Linguistics.
Xingshan Zeng, Liangyou Li, and Qun Liu, 2021. Real- TranS: End-to-end simultaneous speech translation with convolutional weighted-shrinkee In Findings of the Association for Computational Linguistícs: ACL-IJCNLP 2021, pages 2461 2474. Online. Association for Computational Linguistics.
transformer.
Biao Zhang, Barry Haddow, and Rico Sennrich. 2023. Efficient CTC regularization via coarse labels for end-to-end speech translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023. Dubrovnik, Croatia, May 2-6, 2023, pages 2256-2268, Association for Computational Linguis- tics.
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving speech transla- tion by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4252 4261. Online. Association for Computational Linguistics.
Ziqiang Zhang, Long Zhou, Junyi Ao. Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022. Speechut: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training. In EMNLP2022. Association for Computational Lin- guistics.
Yun Tang, Juan Miguel Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021b. A general multi- task learning framework to leverage text data for speech to text tasks. In IEEE International Con- ference on Acoustics, Speech and Signal Process- ing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 6209-6213 0003. 1E13.
Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020. Bridging the gap between pre- training and fine-tuning for end-to-end speech trans-

---

A Data Statistics.
We also include two SOTA works based on speech- text joint pre-training (ST Joint PT), which shows great improvements by applying complex joint pre- training objectives. It should be noted that it is not in line with the focus of this work and our Ada- TranS might also benefit from them by initializing the modules after such joint pre-training.
Table 3 shows the data statistics of the used datasets. ST datasets all from MUST-C, and LibriSpeech serves as extra ASR data. MT data either comes from OpenSubtiiles2018 or WMT14 following set- tings of previous work.
Corpus
ST(Hours/HSents) ASR(Hours)
MT(#Sentt)
En-De
408/234K
960
18M(OS)
2.2
En
492/280K
960
18M(WMT)
ST Baseline AdaTranS
Fin Ru
489/270K
960
2.5((WMT)
2.0
1.8
Table 3: The statistics for the three language pairs. OS:
OpenSubriles201.. WMT: WMT14.
1.6
1
2
5
6
Attention Layer
B Implementation Details.
Figure 4: The attention entropy of each attention layer for end-to-end ST baseline and our model.
Data Preprocessing. We use 80D log-mel filter- banks as speech input features, which are calcu- lated with 25ms window size and 10ms step size and normalized by utterance-level Cepstral Mean and Variance Normalization (CMVN). All the texts in ST and MT data are preprocessed in the same way, which are case-sensitive with punctuation pre- served. For training data, we filter out samples with more than 3000 frames, over 256 tokens, whose ratios of souree and target text lengths are outside the range [2/3, 3/21. We use SentencePiece (Kudo and Richardson, 2018) to generate subword vocab- ulary for each language pair. Each vocabulary is learned on all the texts from ST and MT data and shared across source and target languages, with total size of 16000.
Better Hource-Targel Alignment. We evaluate the entropy of the cross attention from the ST base- line (i.e. any shrinking) and AdaTranS? Let Oij be the attention weight for a target token yi and source speech feature (after shrinking) zj. the entropy for each target token is defined as [al
E, = - =j=1= Oij log Oij. We then average the attention entropy of all target tokens in the test set. Lower entropy means the attention mechanism more confident and concentrates on the source- target alignment. Figure 4 shows the entropy of diffcrent decoder layers. AdaTranS exhibits con- sistently lower entropy than the ST baseline. This means that our shrinking mechanism improves the learning of attention distributions.
Training Details. We train all the models using Adam optimizer (Kingma and Ba, 2015) with a 0,002 learning rate and 10000 warm-up steps fol- lowed by the inverse square root scheduler. Label smoothing and dropout strategies are used, both set to 0.1. The models are fine-tuned on NVIDIA Tesla V100 GPUs with 40000 steps, which takes about 10 hours in average. The batch size set to 40000 frames per GPU. We save checkpoints every epoch and average the last 10 checkpoints for evaluation with beam size of 10.
Influence of Text Input Representations. Rep- resenting text input with phonemes helps reduce the differences between speech and text (Tang al., 2021a,b). However, word representations and punc- tuation are important for learning semantic informa- tion, which are usually ignored when phonemes are used in prior works. Table 5 shows the MT results when using different text input representations, to- gether with the ST performance that is initialized from the corresponding MT model. We can observe that the performance of downstream ST model is affected by the pre-traie pre-trained MT model. Therefore, instead of following prior phoneme-level work for
More Analysis.
Complete Comparisons with the SOTA models. Table 4 extends the results in Table 2 to include the detailed settings of the compared models, in- cluding external data used and training objectives.
"To fairly compare, we also shrink the speech features of the ST baseline with the same boundaries detected hy our houndary predictor.

---

External Data
Training Detail
BLEU En-Fr
Model
Speech
MT
ST Joint
Align
En-De
En-Ru
MT
34,4
44.9
21.3
Cascaded Model
28.1
37.0
17.6
STPT! (Tang et al., 2022)
V
X
x
29.2
39.7
SpeechUT" (Zhang al., 2022)
V
X
X
30.1
41.4
JT-S-MT (Tang et al., 2021a)
X
X
SAKIA
X
/
26.8
37.4
Chimera (Han al., 1021)
27.1
35.6
17.4
XSTNet (Ye al., 2021)
X
X
27.1
38.0
18.5
SATE (Xu al., 2021)
X
28.1
STEMM (Fang al., 2022)
28.7
37.4
17.8
ConST (Ye et al., 2022)
X
28.3
38.3
18.9
AdaTranS
X
XX
28.7
39.2
19.3
Boundary-based Shrink
X
X
28.1
38.8
18.8
Table 4: The complete comparisons with the SOTA models, including external data used and training objectives. "ST Joint PT" indicates speech-text joint pre-training. "Enc Align" means whether adding objectives to aligning representations of speech and text in encoder. "KD" means whether using knowledge distillation from MT.
SPM SPM w/o Punct Phoneme
MT
Only MUST-C Data PT with Extra Data
34.4
31.6
29.5
ST
Initialized with Model 26.0
25.8
25.6
Table 5: BLEU scores of different text input representa- tions MUST-C En-De. SPM means using the subword units learned from a sentencepiece model, while "w/o Punet" indicates that punctuation is removed.
pre-training the MT, in this work we use subword units with punctuation and incorporate the shrink- ing mechanism to mitigate the length gap.
Data
Extra MT Corpus
With MTL
W/O MTL
En-De
(18M)
28.7
28.3
En-Fr
WMT14 (18M)
38.7
39.2
En-Ru
WMT14 (2.5M)
19.0
19.3
Table BLEU scores of AraTranS with KD using MT multi-task learning or not in different language pairs.
Influence of Multi-task Learning (MTL). Pre- vious work usually applies multi-task learning to- gether with MT task (adding external MT data) to improve performance. However, our experiments (Table 6) show that when using KD, MTL might not be always helpful, especially with MT corpus in different domains (OpenSutitles is in spoken lan- guage domain like ST, but data from WMT14 is mostly in news domain).


Reviews:
[{"review_id": "erQIGDz4WO", "reviewer": "Reviewer_T3ym", "summary": "", "questions": "1) Can boundary predictor loss make CTC predictor better? \n2) What about training with only boundary predictor loss without CTC loss? I mean you can get the word boundary from a pretrained CTC model.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructed reviews! We truly appreciate the time and effort you invested in understanding our work! Below are our responses to your questions:\n\nQ1: Why is it better for the lengths of speech and text representations to be closer? Is it because shrinked speech representation is more similar to text embedding?\n\nA1: We discussed this between lines 46 and 58. To be explicit, under the paradigm of \"pre-training on ASR and MT data, then fine-tuning on ST\", making the lengths of speech representations similar to those of text embeddings can facilitate the transfer of pre-trained knowledge from MT models.\n\nQ2: Does bridging the length gap also narrow the representation space gap?\n\nA2: According to our experiment shown in Figure 4 of Appendix B, shrinking does indeed result in better source-target alignment, which helps to narrow the representation space gap. On the other hand, many other methods also aim to bridge this gap. Our experiment in Table 2 demonstrates that our method is complementary to those approaches.\n\nQ3: Can boundary predictor loss make CTC predictor better?\n\nA3: As both the prediction loss and the CTC loss are jointly trained using a shared acoustic encoder, the learning derived from the boundary prediction loss should influence the CTC predictor. However, we believe this effect is likely to be minor, given that the boundary prediction loss is entirely guided by the CTC loss. Further experimentation may be needed to definitively determine whether this effect is positive or not. \n\nQ4: What about training with only boundary predictor loss without CTC loss? I mean you can get the word boundary from a pretrained CTC model.\n\nA4: We chose to jointly train using both the CTC loss and the boundary predictor loss for two main reasons:\n1) The CTC loss serves as a form of regularization. Retaining this loss helps the acoustic encoder concentrate on modeling speech effectively, thereby benefiting downstream tasks.\n2) Since both loss functions share the same input modules, i.e. the acoustic encoder, joint training is a more efficient approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article proposes an improved shrinking method based on Connectionist Temporal Classification (CTC) to address the length gap between speech and text. Experimental results demonstrate that this approach outperforms other methods in terms of both performance and speed. ", "reasons_to_reject": "The innovation in this work is somewhat lacking, and there is a deficiency in explaining certain experimental phenomena. For instance, why is it better for the lengths of speech and text representations to be closer? Is it because shrinked speech representation is more similar to text embedding? Additionally, does bridging the length gap also narrow the representation space gap?", "questions_to_authors": "1) Can boundary predictor loss make CTC predictor better? \n2) What about training with only boundary predictor loss without CTC loss? I mean you can get the word boundary from a pretrained CTC model.", "paper_topic_and_main_contributions": "This article proposes an improved shrinking method based on Connectionist Temporal Classification (CTC) to address the length gap between speech and text. Experimental results demonstrate that this approach outperforms other methods in terms of both performance and speed. However, the innovation in this work is somewhat lacking, and there is a deficiency in explaining certain experimental phenomena. For instance, why is it better for the lengths of speech and text representations to be closer? Is it because shrinked speech representation is more similar to text embedding? Additionally, does bridging the length gap also narrow the representation space gap?", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sBt5AqJb1Q", "reviewer": "Reviewer_g78k", "summary": "", "questions": "- Is previous work comparable with this work in terms of ASR and MT pre-training? In particular, this work pre-trains ASR and MT models on external ASR and MT data, then fine-tunes these models on ASR and MT data of MuST-C. Are all methods in Table 1 pre-trained in the same way?\n- Preprocessing: why is the choice of vocabulary size 16000? \n- Table 1: What are the model architectures used in these methods? Why used Diff<2 instead of the percentage of lengths between shrunk speech and text?\n- Do you have an intuition why weighted shrinking has a much bigger impact on En-Fr while it has only a marginal effect on En-De?\n- What are the differences in AdaTranS between Table 1 and Table 2 that leads to the gain of 2 BLEU points in Table 2? Is it only thanks to the addition of cross-modal contrastive learning and KD?\n- The label <BD> corresponds to one or multiple consecutive frames? What is the shrinking method in the latter case?\n- Did you try using simpler shrinking operations such as averaging instead of weighted shrinking after the boundary predictor module?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructed reviews! We truly appreciate the time and effort you invested in understanding our work! Below are our responses to your concerns and questions:\n\nQ1: This paper misses important references to highly related work and thus fails to properly position itself with respect to the existing literature.\n\nA1: We appreciate your suggestions regarding the inclusion of important references. However, we believe there might be some misunderstandings concerning our method and some of the referenced works. To clarify, our paper is focused on the specific topic of **shrinking speech features** to address the length gap. Given the paper's short length (only 4 pages), we feel it is neither appropriate nor practical to broaden the scope of related references and to discuss all CTC-related methods. We believe we have included all highly relevant works in our related work discussion and in our comparisons. Concerning the references you mentioned, we would like to make the following arguments:\n\n1. The work by Zhang et al., 2023 (\"Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation\") should not be considered closely related to our method. According to our understanding, this work focuses on reducing the output vocabulary size of CTC modules to improve training efficiency and does not address shrinking speech representation lengths. Therefore, it should not be categorized as \"highly related work.\"\n\n2. We acknowledge that \"shrinking speech features\" is not a concept limited to CIF-based or CTC-based approaches. However, these two approaches serve as representative methods, and comparing our work to them offers meaningful insights into the effectiveness of our technique, especially given that our method aims to address the shortcomings of the CTC-based approach. Additionally, the compressed phoneme-based method mentioned (Salesky et al., 2019, \"Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation\") also employs a CTC-based method for compressing speech features.\n\n3. Regarding phoneme-as-MT-input methods like Tang et al., 2021 (\"Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task\"), these are not shrinking-based methods; they simply leverage phonemes as MT inputs for multi-task learning.\n\n4. Adaptive feature selection (Zhang et al., 2020) is indeed a method that focuses on feature shrinking, but their experiments are conducted on encoders that lack any convolutional downsampling, making a fair comparison with our method challenging.\n\nNevertheless, we will expand our discussion of related work upon acceptance, taking advantage of the one additional page allowed.\n\nQ2: Which method adopts Conv-Transformer and which uses Conformer? Do all methods use the same model architecture in each experimental setting?\n\nA2: I apologize for the lack of clarity. In all the experimental settings except for those shown in Table 2 (and its expanded version in Table 4), we used Conv-Transformer as the acoustic encoder. Additionally, the same pre-trained modules were used across these settings.\n\nQ3: Did you perform KD with a frozen pre-trained MT model or did you add an additional MT loss and perform KD with the MT model being updated during training?\n\nA3: Both methods are used and they have different performance in different language settings. Please refer to Table 6 of Appendix C for more details on choices among different language settings (for results \"W/O MTL\", it means performing KD with a frozen pre-trained MT model; while for results \"With MTL\", it means performing KD with the MT model being updated during training).\n\nQ4: The comparison to SoTA methods has some issues. ...in reality Table 2 only contains \"modality alignment\" methods.\n\nA4: Sorry for the confusion caused. We will revise the description in revision. And thanks for introducing two concurrent works (Le et al. 2023 and Ouyang et al. 2023) and more other important ST references, we will include their results in revision.\n\nQ5: Is previous work comparable with this work in terms of ASR and MT pre-training? Are all methods in Table 1 pre-trained in the same way?\n\nA5: In Table 1, the answer is yes, as we re-implemented all the compared methods in a fair manner, where the only difference among them is how they shrink the speech features. As for the results in Table 2, since most of the listed results differ in the used architectures/datasets, it is difficult to maintain an exact fair comparison and we have tried our best by using similar model size and data amount.\n\nQ6: Why is the choice of vocabulary size 16000?\n\nA6: It is an empirical setup following our previous preliminary experiments.\n\nQ7: Table 1: What are the model architectures used in these methods? Why used Diff<2 instead of the percentage of lengths between shrunk speech and text?\n\nA7: All the methods listed in Table 1 employ a Conv-Transformer architecture for the acoustic encoder. We chose to use the \"Diff<2\" metric as opposed to percentage-based metrics for lengths between shrunk speech and text, following the precedent set in previous works, such as \"Bridging the Modality Gap for Speech-to-Text Translation\" by Liu et al. (2020) and \"RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer\" by Zeng et al. (2021). This metric effectively indicates the number of samples that have been successfully aligned, offering a more intuitive basis for comparison.\n\nQ8: Do you have an intuition why weighted shrinking has a much bigger impact on En-Fr while it has only a marginal effect on En-De?\n\nA8: First of all, we believe that drawing a definitive conclusion at this point may be premature, as we have only conducted a limited comparison between MUSTC En-Fr and En-De. The observed difference in performance could be attributed to several factors, such as data distribution or model architecture. If the conclusion holds true after further validation, we would consider potential explanations such as language similarity or issues related to word reordering.\n\nQ9: What are the differences in AdaTranS between Table 1 and Table 2 that leads to the gain of 2 BLEU points in Table 2?\n\nA9: The differences include: 1) replacing Conv-Transformer with Conformer as acoustic encoder; 2) addition of cross-modal contrastive learning and KD.\n\nQ10: The label <BD> corresponds to one or multiple consecutive frames? What is the shrinking method in the latter case?\n\nA10: It corresponds to one frame. Every time we meet a frame with <BD> label, we should shrink the speech features. In other words, the number of <BD> label predicted should be the final length of the shrunk features.\n\nQ11: Did you try using simpler shrinking operations such as averaging instead of weighted shrinking after the boundary predictor module?\n\nA11: The results \"- Blank Label\" in Table 1 are achieved with averaging, as stated in Line 256-258.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow. It is well-motivated and addresses an important problem in ST which is the speech-text modality gap.\n- The proposed method is effective: yielding better performance than several existing shrinking-based methods being compared in the paper with faster inference speed and memory usage.\n- Competitive performance with current strong ST systems using self-supervised pre-training methods.", "reasons_to_reject": "1. This paper misses important references to highly related work and thus fails to properly position itself with respect to the existing literature:\n\t- Zhang et al., 2023 (\"Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation\") showed that simply coarse labeling CTC using simple heuristics rules such as truncation, division or modulo operations yields comparable-to-better performance than baselines with considerable training speedup.\n\t- Shrinking speech features has been an active research area and not limited to only CIF-based or CTC-based approach. Other methods including compressed phoneme-like features (Salesky et al., 2019. Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation), adaptive feature selection (Zhang et al., 2020. Adaptive Feature Selection for End-to-End Speech Translation), and phoneme-as-MT-input (Salesky et al., 2020. Phone Features Improve Speech Translation and Tang et al., 2021. Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task) should be mentioned as well. In terms of CTC-based shrinking approach, an important and highly related work using very simple heuristics rules (Zhang et al., 2023. Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation) should be cited and included for comparison with the proposed method. This is alternative to the CTC greedy path approach that the authors mentioned.\n\t- Previous work using CTC alignments to improve ST is also highly relevant to the current paper (which also relies on CTC alignments signals) and therefore should be discussed. For example, Yan et al., 2022 (\"CTC Alignments Improve Autoregressive Translation\") and Le et al., 2023 (\"Pre-training for Speech Translation: CTC Meets Optimal Transport\"). In particular, Le et al. 2023 specifically address the speech-text modality gap using CTC (which is the main subject of the current submission), and also propose/discuss different length-matching mechanisms when comparing speech and text sequences.\n    \n2. The experimental settings are not clearly described in some parts. \n\t- The ablation study setting to compare different shrinking method is not clear: which method adopts Conv-Transformer and which uses Conformer (Section 3.1., \"Model setting\" only specifies that AdaTranS uses Conformer in Table 2, but what about Table 1?) Do all methods use the same model architecture in each experimental setting?\n\t- The way to incorporate additional alignment objectives is not presented. For example, it is not straightforward to me how KD guided by MT is added to the architecture in Figure 1. Did you perform KD with a frozen pre-trained MT model or did you add an additional MT loss and perform KD with the MT model being updated during training? \n\t\n3. The comparison to SoTA methods has some issues. The caption of Table 2 is \"Comparisons with the SOTA models...\" but in reality Table 2 only contains \"modality alignment\" methods as stated in the text. Note that even within this \"modality alignment\" subgroup, the authors have missed some recent work such as Le et al. 2023 (Pre-training for Speech Translation: CTC Meets Optimal Transport) and Ouyang et al. 2023 (WACO: Word-Aligned Contrastive Learning for Speech Translation). I would suggest to fix the caption of Table 2 and also include these missing references. Finally, as a minor suggestion, for the full comparison in Table 4 (Appendix C), I would recommend to include important ST references that established new SoTA at the time of their publications: ESPNet-ST (Inaguma et al. 2020), Fairseq-S2T (Wang et al.), Dual decoder Transformer (Le et al. 2020), BiKD (Inaguma et al. 2021), Adapters (Le et al., 2021), TDA (Du et al., 2022), just a few off the top of my head.", "questions_to_authors": "- Is previous work comparable with this work in terms of ASR and MT pre-training? In particular, this work pre-trains ASR and MT models on external ASR and MT data, then fine-tunes these models on ASR and MT data of MuST-C. Are all methods in Table 1 pre-trained in the same way?\n- Preprocessing: why is the choice of vocabulary size 16000? \n- Table 1: What are the model architectures used in these methods? Why used Diff<2 instead of the percentage of lengths between shrunk speech and text?\n- Do you have an intuition why weighted shrinking has a much bigger impact on En-Fr while it has only a marginal effect on En-De?\n- What are the differences in AdaTranS between Table 1 and Table 2 that leads to the gain of 2 BLEU points in Table 2? Is it only thanks to the addition of cross-modal contrastive learning and KD?\n- The label <BD> corresponds to one or multiple consecutive frames? What is the shrinking method in the latter case?\n- Did you try using simpler shrinking operations such as averaging instead of weighted shrinking after the boundary predictor module?", "paper_topic_and_main_contributions": "This paper proposes a novel boundary predictor mechanism to tackle the speech-text modality gap in end-to-end speech translation (ST). This boundary-based shrinking method consists of two parts. First, a boundary predictor module predicts the word boundaries based on the signals from the CTC module (by collapsing the vocabulary into three labels including blank, boundary, and others). Second, a shrinking operation is applied on top of the boundary prediction results to shrink the speech representations so as to mitigate the length mismatch between speech and text sequences. The proposed method is complementary with additional alignment losses.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In addition to the important references cited in \"Reasons to reject\", I think the literature review could be greatly improved (without adding too much space) by briefly discussing the following references:\n- L24: Berard et al., 2016 (\"Listen and translate: A proof of concept for end-to-end speech-to-text translation\") were among the first to work on end-to-end ST (concurrent with Duong et al., 2016 but Duong et al. focused on the alignment and reranking tasks instead of translation).\n- Figure 1: Is the CTC module pre-trained as well? If so, Wang et al., 2020 (\"Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation\") were the first to use ASR pre-training with CTC with the motivation to reduce subnet wastes while Le et al., 2023 (\"Pre-training for Speech Translation: CTC Meets Optimal Transport\") did a comprehensive study on the effect of ASR pre-training CTC.\n- L226: Papineni et al., 2002 (\"Bleu: a method for automatic evaluation of machine transla- tion\"); Post, 2018 (\"A call for clarity in reporting BLEU scores\") proposed BLEU and sacreBLEU as translation metrics.\n- L190: This work belongs to the multi-task learning paradigm in which ST is jointly trained with ASR (and maybe other losses). The landmarks papers in this direction are Anastasopoulos and Chiang, 2018 (\"Tied multitask learning for neural speech translation\") and Sperber et al., 2019 (\"Atention-passing models for robust and data-efficient end-to-end speech translation\").", "Typos_Grammar_Style_and_Presentation_Improvements": "- L143: D_st is not defined previously\n- Table 1: would be more straightforward to use \"Weighted shrinking\" instead of \"Blank label\" as the former term is already used and clearly described in previous text"}}, {"review_id": "BF3N2YdDCs", "reviewer": "Reviewer_FEtD", "summary": "", "questions": "- How accurate is the boundary prediction? Is there an alternative method to assess its accuracy apart from relying on Figure 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive reviews! We truly appreciate the time and effort you've invested in understanding our work.\n\nRegarding your concern about why we didn't train a forced alignment tool when AdaTranS has access to sufficient training data: we actually conducted a similar preliminary experiment. Specifically, we used a pre-trained aligner from the Montreal Forced Aligner (https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner). With this tool, we extracted alignments between speech and transcriptions to guide the reduction of speech features. The results were impressive in terms of the shrinking quality, achieving over 99% as per the \"Diff≤2\" metric shown in Table 1. However, the translation quality did not see any significant improvement: it was 26.5 BLEU for MUSTC En-De, marginally lower than the 26.7 BLEU achieved by our current method.\n\nIn response to your question, \"How accurate is the boundary prediction? Is there an alternative method to assess its accuracy apart from relying on Figure 3?\":\n\nThis remains an open question. As discussed in Section 2.2 and in the Limitations section, there's no precise way to obtain ground-truth labels for each frame, especially after downsampling. Therefore, we can only indirectly assess the \"accuracy.\" Besides evaluating the final quality of downstream tasks (e.g., translation performance, as illustrated in Figure 3), alternative methods for assessment could include:\n\n1. Evaluating the shrinking quality, which is measured by the length differences between the reduced representations and transcriptions (i.e., results shown in Table 1). Generally, better boundary prediction can result in better length alignment.\n\n2. Using pseudo labels. We can pre-label each frame with pretrained speech labelers like CTC modules or forced aligners. However, this method might introduce noises, as no ground-truth labels are available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Unlike earlier methods that indirectly predict boundaries by relying on token outputs from CTC, the AdaTranS architecture employs a direct approach. It predicts the likelihood of word boundaries, mitigating the potential error propagation inherent in such indirect methods.\n- The conducted experiments on the MUST-C dataset reveal that the introduced AdaTranS offers superior performance compared to alternative compression techniques. Notably, AdaTranS achieves faster inference for speech processing while maintaining low memory utilization.\n- The evaluation also encompasses a cascaded approach, highlighting the superiority of the proposed end-to-end method.", "reasons_to_reject": "- The authors assert their decision to not employ a forced alignment tool stems from its support being limited to widely spoken languages. Nevertheless, it remains unclear why training a forced alignment tool couldn't be pursued when AdaTranS has access to sufficient training data.", "questions_to_authors": "- How accurate is the boundary prediction? Is there an alternative method to assess its accuracy apart from relying on Figure 3?", "paper_topic_and_main_contributions": "In this paper, AdaTranS is introduced as a novel method capable of directly converting spoken input from one language into written text of another language, all in a end-to-end manner.\nA novel shrinking mechanism is proposed to address the length mismatch between speech and text features.\nThis involves employing a boundary predictor to assign probabilities to each speech representation, indicating the likelihood of it being a boundary. Subsequently, these probabilities are harnessed for a weighted shrinking process.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vvQf7Om3nM", "reviewer": "Reviewer_kuHR", "summary": "", "questions": "A) To what extent would the use of the pre-trained forced aligners to better train the predictor would help? If we ignore the potential limited availability in some languages of forced aligners (which do not require high amounts of data to train in general), it would provide a better understanding of what is the true potential of the proposed idea.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructed reviews! We truly appreciate the time and effort you invested in understanding our work!\n\nRegarding your concern about \"the proposed boundary prediction model discards very important information coming from the sequentiality of the predicted information\", we believe there might be some misunderstandings about our method:\n\n1. Our method does not discard any information concerning sequentiality. The boundary predictor is based on the output from the acoustic encoder, either the Conv-Transformer or Conformer (as referenced in \"Model setting\" in Section 3.1). Both of these incorporate position information. Moreover, our prediction loss, as described in Eq. 4, is guided by the CTC loss, which is also computed considering the sequential target from the corresponding transcript.\n\n2. We have not omitted any frame-level representations during our modeling. As indicated in Eq. 5, every representation is integrated into our weighted average process to generate the final acoustic features.\n\nIn response to your another concern about \"As such, and without having an understanding of what is the actual 'accuracy' of the soft labels predicted by the module, it is hard to understand if the improvements the authors achieve are really because of the proposed algorithm.\", we'd like to clarify the following:\n\n1. While the precise 'accuracy' of the soft labels is challenging to obtain — and indeed might be a subject for future research — it doesn't render our approximations void. Using the CTC objective for speech labeling is standard in speech recognition. Relying on its implicit guidance for boundary prediction is a natural extension. One contribution of our paper is experimentally proving the efficacy of such an approach.\n\n2. The results in Table 1 lucidly depict the improvements facilitated by our method. To be explicit, we re-implemented all presented methods under identical settings. The sole differentiation lies in their method of shrinking acoustic features. A direct comparison between our \"Boundary-Based\" method and \"No Shrink\" reveals improvements in both translation quality and efficiency, thereby underscoring the potency of our proposed technique.\n\nIn response to your inquiry, \"To what extent would the use of the pre-trained forced aligners to better train the predictor help?\":\n\nWe did try a similar idea during our preliminary experiments. Specifically, we employed a pre-trained aligner sourced from Montreal Forced Aligner (https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner). With this tool, we extracted alignments between speech and transcriptions to guide the shrinking of speech features. The outcome was impressive in terms of shrinking quality, achieving over 99% as per the \"Diff≤2\" metric in Table 1. However, the translation quality did not witness any significant leap: it was 26.5 BLEU for MUSTC En-De, marginally lower than the 26.7 BLEU achieved by our current method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is a simple-yet-effective mechanism for detecting boundaries that has a positive impact on the downstream task. Even if the proposal and results are on a still preliminary stage of exploration, the model could spark conversation in the field.", "reasons_to_reject": "The authors highlight clearly the limitations I saw while reading the paper.\n\nMost notably, I find that the proposed boundary prediction model discards very important information coming from the sequentiality of the predicted information, similarly to how common forced alignment modules commonly used to align speech and audio are implemented. As such, and without having an understanding of what is the actual 'accuracy' of the soft labels predicted by the module, it is hard to understand if the improvements the authors achieve are really because of the proposed algorithm.", "questions_to_authors": "A) To what extent would the use of the pre-trained forced aligners to better train the predictor would help? If we ignore the potential limited availability in some languages of forced aligners (which do not require high amounts of data to train in general), it would provide a better understanding of what is the true potential of the proposed idea.", "paper_topic_and_main_contributions": "This paper proposes a mechanism for detecting word boundaries based on connectionist temporal classification (CTC) predictions. These predictions are useful to reduce the feature length mismatch for audio-text conversion pipelines, applied by the authors to the problem of (end to end) speech-to-text translation. The proposed method operates on each individual CTC prediction, obtaining a probability of", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper addresses the modality gap between speech and text representations when using pre-trained models for speech2text translation. It explores the use of boundary predictions (from CTC) in the shrinking process to reduce gap between speech sequence length and text sequence length. The paper is well-written and method proposed (as well experiments presented) are overall convincing. However the positioning related to previous approaches (to address modality gap for S2T translation) could be improved and this makes evaluation of the originality of the work difficult. 
Moreover, questions were raised about the accuracy of the CTC boundary prediction and the use of pre-trained forced aligners instead. Finally, the way additional alignment objectives are included is unclear and some improvement on this part would be needed.
Decision: Accept-Findings