PromptST: Abstract Prompt Learning for End-to-End Speech Translation
Authors: Tengfei Yu
Liang Ding
Xuebo Liu
Kehai Chen
Meishan Zhang
Dacheng Tao
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Speech-to-Text Translation
Linguistic Probing Benchmark
Prompt Learning
TL;DR: We explore the fusion in S2T and propose PromptST to improve.

Abstract: An end-to-end speech-to-text (S2T) translation model is usually initialized from a pre-trained speech recognition encoder and a pre-trained text-to-text (T2T) translation decoder.
Although this straightforward setting has been shown empirically successful, there do not exist clear answers to the research questions: 1) how are speech and text modalities fused in S2T model and 2) how to better fuse the two modalities?
In this paper, we take the first step toward understanding the fusion of speech and text features in S2T model.
We first design and release a 10GB linguistic probing benchmark, namely Speech-Senteval, to investigate the acoustic and linguistic behaviors of S2T models.
Preliminary analysis reveals that the uppermost encoder layers of the S2T model can not learn linguistic knowledge efficiently, which is crucial for accurate translation.
Based on the finding, we further propose a simple plug-in prompt-learning strategy on the uppermost encoder layers to broaden the abstract representation power of the encoder of S2T models.
We call such a prompt-enhanced S2T model PromptST.
Experimental results on four widely-used S2T datasets show that PromptST can deliver significant improvements over a strong baseline by capturing richer linguistic knowledge.
Benchmarks, code, and scripts are freely available at https://github.com/ytf-philp/PromptST.

PromptST: Abstract Prompt Learning for End-to-End Speech Translation
Tengfei Yu'; Liang Ding", Xuebo Liu'; Kehai Chen',
Meishan Zhang', Dacheng Tao?, Min Zhang' Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China IJD Explore Academy, JD.com, Beijing, China
'hhe University of Sydney, Sydney, Australia
22s151166estu hit. edu. cn, liangding. li liam@gmail. com, dacheng. tao@gmail comm (liuxuebo, chenkehai, zhangmeishan, zhangmin2021 jehit edu...ddd.eo. cn
Abstract
signals into a foreign text without any intermedi- ate output, which has gained increasing popular- ity and obtained great success recently (Anasta- sopoulos and Chiang, 2018; Ansari et al., 2020; Li et al., 2021b; Bentivogli et al., 2021). Since directly modeling speech-to-text mapping is non- trivial, the common practice (Wang et al., 2020c, 2021) trains well-performed S2T system by ini- tializing the encoder and decoder with pre-trained single- moggllty models that are designed for ASR (e.g., wav2vec (Baevski et al., 2020)) and MT (e.g.. mBART (Liu et al., 2020b)), respectively.
An end-to-end speech-to-tee to-text (S2T) transla- tion model is usually initialized from pre- trained speech recognition encoder and a pre- trained text-to-text (T2T) translation decoder. Although this straightforwar setting has been shown empirically successful, there do not ex- ist clear answers the research questions: 1) how are speech and text modalities fused S2T model and 2) how to better fuse modalities? In this paper, we take the first step toward understanding the fusion of speech and text features in S2T model. We first design and release a 10GB linguistic prob- ing benchmark, namely Speech-Senteval, to in- vestigate the acoustic and linguistic behaviors of S2T models. Preliminary analysis reveals that the uppermost encoder layers the S2T model can not learn linguistic knowledge ef- ficiently, which is crucial for accurate trans- lation. Based on the finding, we further pro- pose simple plug-in in prompt-learning strategy on the uppermost encoder layers to broaden the abstract representation power of the en- coder of S2T models. We call such prompt- enhanced S2T model PromptST. Experimen- tal results four widely-used S2T datasets show that PromptST can deliver significant im- provements over strong baseline by captur- ing richer linguistic knowledge. Benchmarks, code, and scripts are freely available at https: //gi thub. com/ytf wttpttfpplpp/p/ppppttppp
While this straightforward setting has been shown empirically successful, there are no clear answers to the research questions: 1) how does the S2T model combine speech and text modalities? 2) how improve the fusion of speech and text modalities in the S2T model? Several attempts have been made to alleviate cross-modal represen- tation discrepancy. For example, Yin et al. (2023) use implicit guidance from external ASR model, Le et al. (2023) optimize the CTC loss at the pre- training stage, Ye et al. (2022) use cross-modal contrastive learning (Rao et al., 2023a). Addition- ally, Fang et al. (2022) mix up the speech and text representation to fuse speech and text modalities from the neural representation perspective. How- ever, these methods have not delved into analyzing the intrinsic properties of model representations.
Different from the modal fusion in the input, this paper takes the first step toward understanding the fusion of speech and text features in the S2T model. Specifically, we design a fine-grained linguistic probing benchmark, namely Speech-Senteedl for the S2T model following Conneau et al. (201 8). Considering the information flow transferring from audio text, the encoder of the S2T model prob- ably learns extract, align, and fuse acoustic fea- tures at the lower layers and then turns to learn the important knowledge, c.g., linguistic propertics, to achieve translation at the high-level layers. To fur-
1 Introduction
Different from cascade of separately trained au- tomatic speech recognition (ASR, Yu and Deng 2016) and machine translation (MT, Luong al. 2016) models, end-to-end speech-to-text transla- tion (S2T, Duong et al. 2016; Bérard et al. 2016) directly translates source-langee Langua-l acoustic speech
Work was done when Tengfei was interning at Ex- plore Academy. Corresponding Author

---

ther understand this, we investigate fine-grained comparison of acoustic and linguistic analysis and leave the vanilla text probing benchmark for the T2T model. The result shows that the pre-trained S2T model fails to effectively learn linguisticc infor- mation at the high-level layers, which brings a gap between speech and text in end-to-end S2T models. Motivated by these findings, argue that broad- ening the representation power of the high-level layers for the pre-trained S2T models is at the core of achieving better performance. Plenty of works have shown that translation models can largely ben- efit from the enriched representation (Wang et al., 2019; Wu et al., 2019; Wei et al., 2020; Liu et al., 2019a, 2020a, 2021e; Sun al., 2022; Zan et al., 2022). To this end, design strategy, abstract prompt, to augment the high-level layers of the pre-trained S2T model, leveraging prompt-learning methods (Li and Liang, 2021). This strategy named PromptST due to its succinct and plug- in properties. We experiment with the method in a widely used pre-trained S2T model (Wang et al., 2021) on CoVoST-2 data sets, spanning English- Germmn,, English- Catallm, English-Arabic and English- Turkish language pairs. PromptST consistently and significantly outperforms strong baseline by an average of +0.4 BLEU Our main contributions are as follows:
approach that jointly performs automatic speech recognition and S2T, while Liu al. (2019b) presents a knowledge distillation technique (Deng et al., 2023) by transferring knowledge from T2T models. However, previous works indicate that their successes heavily rely on large amounts of labeled training data, which is challenging to ac- quire. Recent advancements in pre-trained mod- els, such wav2vec2.0 (Baevski et al., 2020) and mBART (Liu et al., 2020b, 2021c,d), have enabled the utilization of large amounts of unlabeled data for pre-training, followed by fine-tuning on S2T tasks. By using pre-trained weights to initialize the S2T structure, the convergence accuracy and training performance of S2T models can be signifi- cantly improved (Stoian et al., 2020; Wang et al., 2021; Ouyang al., 2022; Yin et al., 2023). In this study, we aim investigate the impact of pre- trained knowledge on end-to-end S2T models.
Interpreting the Neural Network Models Prob- ing task is often designed to facilitate comparison between different models at a fine-grained level. For example, Yang et al. (2019) design a word reordering detection task to evaluate a model's abil- ity to extract word order information. Rogers et al. (2021); Lin et al. (2019) analyze the hierarchy of linguistic information in transformer encoder's hid- den representation. Shi et al. (2016) assess the learned representations machine translation mod- els for syntactic knowledge (Wang et al., 2023). Conneau et al. (2018) introduce a set of tasks to probe the linguistic knowledge encoded in sentence embeddings. These methods help open the black box of networks. In the realm of speech-to-text, Tang et al. (2021) compare auxiliary text transla- tion tasks, and Xu et al. (2021) define the localness of word to analyze the model architecture. In this paper, we extend these ideas and construct 10GB probing benchmark, the learned linguistic properties of S2T models.
We extend earlier works on text probing tasks to speech scenarios, organized by the type of linguistic properties they probe. We pub- licly release our probing benchmark Speech- Senteval, with the hope that will be helpful for further study on linguistic properties of ASR and ST communities.
By acoustic and linguistic probing analy- sis, show there great differences be- tween speech-to-text translation and text-to- text translation, particularly in the heveer-level encoder layers of the models.
to analyze
Prompt Learning Prompting refers to adding a snippet of natural language text to unlabeled data during the pre-training phase. In the case of discrete prompting, discrete information is added to the dataset described in (Schick and Schütze, 2021). For example, discrete prompt to- kens such as "It", "is". "[MASK]" can be used to classify a movie review. To illustrate, given the input text x =AAmazing movie!", the in- put embedding sequence would be formulated as
Based on our findings, we propose a straight- forward prompt learning strategy to enhance the representation capabilities of the higher- level layers in pre-trained S2T models.
2 Related Work
Speech-to-Text Translation Most studies have been conducted to enhance end-to-end S2T mod- els. Le et al. (2020) propose multi-task learning

---

Acoustic Feature
Linguistic Feature
1.46
0.64
0.6
1.44
0.6
Surfce
673
1.2
3.50
0.54
E
0.52
1.38
60
KM
0.22
1.20
3.0
Denon
Tue
Donme
N055c
The
Depth
Depth
6.0
Figure 2: Performance results from acoustic and linguis- tic probing tasks, we align and average based on the S2T layers. The shaded areas denote our primary focus.
0.5
6.0
5.6
1.40
050
654
and top, and analyze the properties learned in each layer. Specifically, we design fine-grained linguis- tic probing benchmark, namely Speech Speech-Senteval, for the S2T model following Conneau al. (2018) and leave the vanilla text probing benchmark for the T2T model to further understand linguistic prop- erties. Subsequently, we focus on different views between acoustic (speech view) and linguistic prop- erties (text view) in different layers of the S2T model in order to explore the acoustic and linguis- tic behaviors, respectively.
5.14
6.70
5.00
5.M
6.66
5.55
6.04
1.00
15
EG
Emcoder Depth
Encoder Depth
Figure 1: Performance on 10 probing tasks evaluate the linguistics information encoded by the S2T and T2T models. These tasks are divided into three categories: surface, syntactic, and semantic. We averagely divide the encoder layers into three levels: bottom, middle, and top. The red spots show the surface and syntactic knowledge learned in the last layer is similar to that of the middle layer, indicating the S2T model can not learn linguistic information well the top layers.
3.1 Speech-Senteval Benchmark
For conducting a comparative experiment in repre- sentation between S2T and T2T, we extend earlier work on linguistic probing tasks to S2T. In detail, we employ the same MLP classifier as Conneau et al. (2018), analyzing a rich hierarchy of linguis- tic properties in the encoder of S2T m
(e(x),e("II") e"""t""))""i""))) e""is"), "e"is"""eee((eaa c("[MASK]'). To reduce manual intervention, Lester et al. (2021); Liu et al. (2021b); Hsu et al. (2023) introduce trainable con- tinuous prompts (Qi et al., 2023) as substitution for natural language prompts for language under- standing tasks. For language generation tasks, Gar- cia and Firat (2022) uses an input template that con- tains a slot for input to control the output of trans- lation models. However, the field of prompting speech translation models remains under-explored. We borrow ideas from this line of research by in- corporating text-enhancement parameters into pre-trained end-to-end S2T model to guide better speech translation.
Speech-Senteval Data To build the data set of probing tasks, we use a publicly available Baidu speech synthesis model2 to convert the text probing data set into audio. An instance consists of 16khz sampling audio, a transcription, and classification label. Considering audio has more complex feature sequences than textual data, higher GPU memory constraints, and experimental costs are required, reduce the size of the data set. The detailed data volume and statistics are shown in Appendix A.
S2T and T2T Results We analyze 10 probing tasks on S2T and T2T respectively and provide detailed S2T statistics in Table 1. To make the analysis more intuitive, we average and smooth the experimental results according the class the probed property belongs to. As shown in Figure 1,
3 S2T Behaviors Analysis
In simple implementation of an end-to-ent S2T model, only the top-most representation is used for decoding, ignoring the interactions among layers (Li et al., 2020). To better understand the fusion of speech and text features, we averagely divide the encoder layers into three levels: bottom, middle,
'The encoder is initilized by using the wav2vec2 O-large model which is pretrained on Libri-960hr and self-trained Libri-light (LV-6k).
Phttps: / //ai baiiu. waiidu.com/teeeeee com/techhspeech/te.

---

Categories
Task
Emb
8
12
16
20
24
Word Content
0.0
1.1
2.2
1.5
8.0
11.0
2.5
Surface
Sentence Length
73.0
69.0
73,0
76.0
79.0
78.0
78.2
Top Constituent
33.4
35.5
57.3
56.9
70.7
64.5
50.2
Synatactic
Tree Depth
25.2
26.9
28.3
25.0
33.5
35.7
41.3
Bigram Shift
52.2
49.5
57.8
64.9
63.8
69.0
67.5
Coordination Inversion
54.7
52.2
51.8
48.5
50.8
58.9
61.6
Object Number
71.6
72.7
76.8
74.7
78.3
80.0
Semantic
Past Present
68.9
64.4
72.0
82.0
82.1
80.8
83.8
Subject Number
70.0
59.2
75.2
83.0
79.3
82.2
83,9
Odd Out
47.7
51.0
50.1
55.1
50.4
53.3
58.3
Table Probing task results for S2T model. We show the analysis results of every four layers.
we compare differences between S2T and T2T with
set. The constructed training sets contain 87,299 training phonemes and 32, 370 validation phonemes extracted from utterances. The possible labels are 60 phone symbols included in TIMIT (excluding the begin/end silence symbol h). Speech-senteval probing proposed
surface, syntactic, and semantic information. The results of our analysis reveal that the T2T model encodes a hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle, and semantic features the top. The S2T system gradually encodes these three kinds of properties as the information propagates through the model's layers.
tell how much the linguistic property S2T model catches. To make comparing easy, we average all tasks by layers.
Gap between Feature Learning Figure 2 presents a detailed comparison of the acoustic and linguistic properties. The results indicate opposing trends, particularly at the high-level layers. The ability of the S2T model to encode linguistic prop- erties increases, while the ability encode acoustic properties decreases gradually. This finding con- firms our hypothesis that S2T models prioritize encoding acoustic representation at low-level layers and shift towards encoding of linguistic rep- resentation at high-level layers.
Notably, it is observed that the learning of the S2T model exhibits fluctuations. Analysis of sur- face and syntactic information reveals clear de- cline in representation after the middle layers, which means the output of the encoder in the S2T model encodes similar information the middle layer. Conversely, this is not present in the T2T model. This suggests that the pre-trained S2T model may not effectively learn linguistic informa- tion at higher layers, potentially due to limitations in model capacity, which brings a gap between speech and text in end-to-end S2T models.
3.3 Discussion
3.2 Comparison: Acoustic and Linguistic
Based on the above-detailed analysis of the differ- ences between (1) S2T and T2T models in Sec- tion 3.1, and (2) acoustic and linguistic features in the S2T model in Section 3.2, it is evident that high-level layers play crucial role in undertaking linguistic representation tasks. However, the stan- dard, pre-trained end-to-end S2T model represents linguistic features sub-optimally at higher encoder layers compared to the T2T model. Considering the importance of augmenting linguistic features for achieving high-quality translation (Dyvik, 1992; Sennrich and Haddow, 2016; Ding and Tao, 2019). believe that enhancing the representation power of high-level layers in S2T models could exploit more linguistic properties and thus have the poten- tial to improve speech translation performance.
For exploring speech and text modalities of the S2T model tine-grineed level, employ acoustic and linguistic probing tasks to discern the differ- ences between the modalities.
Setting We analyze acoustic and linguistic prop- erties by preparing task-oriented datasets, respec- tively. Phonetic probing can tell us how much the acoustic property S2T model catches. Adopting the methodology of Belinkov and Glass (2017), employ the TIMIT dataset, which provides time- segmented phonemes, extract frames from al- ternate encoder layers for the phoneme classifi- cation task. A classifier is then trained on these features and its performance evaluated on a test

---

14 toc- longilt,
Encoder to Decoder Projection
Targel Text
Decoder Impat
Encoder Output
!! ra." [Auddoo longth, Encoder Dimeusion)
Norm
Forward
Layerg Prompes
Add Norm
Livery Prompts
Transformers
Multi-Head Aftention
N
Prompt Representations
Add Norm
Optimization
CNN
Muli--Hea Attention
Encoder
Rew Weveform Input
Shifted Right Target
Figure 3: Architecture of the PromptST and its integration with the transformer block. Pink blocks refer trainable prompt embeddings. For an M- M-layyrs encoder, we add prompt embedding from ( + 1)thh layers. Blue blocks are transformer block embeddings initialized by wav2vec2.0. The decoder N-layer transformer-based model.
4 Abstract Prompt Learning for ST
to zero mean and unit variance. The total stride of the encoder determines the number of time-steps T which are input to the Transformer. To address the issue of dimension inconsistency between the audio and text, we integrate two-layer multilayer perceptron (MLP) projection module between the pre-trained encoder and decoder. Mathematically, the encoder output hand to the decoder input h he is derived from:
Motivated by the analyses in Section 3, augment- ing the representation power of higher encoder lay- ers with plug-and-play strategy is essential to achieve better linguistic knowledge that serves for high-quality speech translation. Besides efficient and plug-in properties, the method is required to enhance the higher layers without disturbing the phonetic information learned in the lower layers. Accordingly, we precisely meet these requirements by introducing a soft prompt strategy (Liu et al., a0) that has been empirically successful in sev- eral natural language understanding tasks but re- mains under-explored in the field of speech trans- lation. Specifically, higher encoder layers in pre- trained ST models will be equipped with learnable soft-prompt representations to capture additional abstract information, e.g., linguistic knowledge. We call the proposed method Abstract Prompt.
h dec = MLP
(0)
(1)
end
The transformed representation, b is integrated into the cross-attention module of the decoder.
4.2 Abstract PromptST
The pink blocks in Figure 3 depict the concept of Abstract PromptST, which involves the incorpora- tion of continuous prompts in the higher layers of the S2T model encoder. The method employed in this study is not novel and can be considered optimization of P-Tuning V2 (Liu al., 2021a). Technically, we formalize the pseudo prompt to- kens as jp...... pm). PromptST maps them into trainable tensors emb ((pI... ... pPm ))- In practice, the transformer architecture generates representa- tions of consistent dimensionality across all lay- ers. This consistency presents a challenge when attempting to establish direct mapping between the prompt representations and the hidden states, as the dimensions of the two may not align. To overcome this challenge, we utilize the attention module in the encoder to facilitate the integration the prompt representations with the hidden states.
4.1 S2T Architecture Description
Speech-to-text task directly translates source- language audio foreign text without any inter- mediate output. Intuitively, encoder-decoder (i.c., Seq2Seq) models are considered particularly suit- able where the input and output sequences are not monotonically aligned.
In this study, ture, consisting of a wav2vec2.0 (Baevski al., 2020) encoder and transformer-based (Vaswani al., 2017) decoder. Given source raw waveform audio, we use the wav2vec2 feature processor to extract raw waveform input, which is normalized
pre-trained S2T architec-

---

Formally, the vanilla self-attention module can be represented mathematically as
Models and Settings Our implementation is based on the HuggingFace (Wolf et al., 2020) speech encoder-decoder models. Following the best configuration of Wang et al. (2021), we use a peauuence-t-sequeeee wav2vec 2.0 model with several convolutional layers followed by a Transformer network. The decoder is 7-layer Transformer network whose embedding size is 256, the number of attention heads is 4, and the FFN dimension 2048.
Attn h onee once w whe enc
(2)
model. The encoder is
where projections w , _ are parameter matri- ces and linlc is the input of the i-th layer.
For the selected i-th layer of the encoder, we concatenate the continuous prompt representations to the keys and values in the self-attention module. A randomly initialized embedding is employed to achieve our objectives:
During model training, we reload the pre-trained models w2e-waavve22-largee- (aiggeeennddecc/arrtt from HuggingFace hub". We use the byte-pair encoding (BPE) (Sennrich et al., 2016) implementation from Huggingface to learn the sub-word segmentation where the vocabulary size of subword tokens V is set as 10K. We train our model with Adam optimizer (Loshchilov and Hutter, 2019) with a learning rate of 5e-5. We apply the label smooth- ing with 0.1 and layer drop with 0.05. We set the batch size as 4, which containing 4.8M tokens. We perform the gradient accumulation trick (Ott et al., 2018) with one update per 16 batch. A masking strategy similar to wav2vee 2.0 is adopted with the mask length set as 5 and the mask probability set as 0.15.
k)ys()) (+) = emb((p).... pml)) w bene
(3)
values emb((p")," , w b
where is the total length of prompt representa- tions and emb(.) denotes a initialized representa- tion. This approach allows the model to assign different weights to different utterances in the input and prompt. Additionally, we discuss the perfor- mance of PromptST by incorporating two-linear module and a ianh activation to reparameterize the prompt representations in Appendix A.2.
For the low-level layers of the encoder, we adopt the vanilla self-attention module. Whereas the self- attention module in high-level layers the encoder is revised into the following form:
Optimization and Evaluation In order ex- plore the influence of the method and length of prompt for speech translation, we discuss the opti- mization and prefix length in Appendix A.2. We choose the prefix length by searching in the inter- val [10, 100) and without MLP reparameterization. During model decoding, beam size of 5. Models are trained for 20K updates and the best checkpoint is selected w.r.t BLEU score (Papineni et al., 2002) on the valid set. All models are fine-tuned with 8 NVIDIA A100 GPUs.
Attn (w enees k)ys values(1)
(4)
To enhance the efficiency of fine-tuning while considering model performance, it is essential to utilize PromptST in conjunction with decoder fine- tuning so that the prompt and the decoder parame- ters can be updated jointly. We discuss the impact of different fine-tune lineituning strategies on model perfor- mance in the Appendix A.3.
beam search with
4.4 Main Results
4.3 Experiment Setup
We use the official baselines for most prior works (Wang et al., 2020b, 2021). which are implemented upon fairseq (Wang al., 2020a) and Hugging- final results the
Data We use the CoVoST-2 ST data set?, which is a large-scale multilingual ST corpus containing both XX-English and English-XC XX translation tasks. It is the largest open data set available to date from the perspective of the total volume and covered languages. Specifically, we choose four language pairs, including English (En) to German (De), Cata- lan (Ca), Arabic (Ar), and Turkish (Tr). Each of them contains 430 hours of annotated data.
Face. Table demonstrates test sets. We first evaluate performance of a strong baseline model (# Model "5"") we analyzed in Sec- tion For a fair comparison, we adopt an identical fine-tuning strategy wherein we fix the encoder and
"ht /hwgg//ggugnneceeeeee wwtacc..ooaaaaee co/facebook 'To evaluate the efficacy of our proposed method, we refrain from utilizing the language model for decoding as adopted in Wang et al. (2021).
Phttps: //gp::////tttttttee hattts///gittuub..m//aaaaeeeetee htt.e///c..omeeeeeoooeeeereerrre

---

ID
Model
Para
En-De
En-Ca
En-Ar
En-Tr
Avg.
Previous results on CoVoST
End-to-end (Wang et ul., 2020b)
13.6
20.2
8.7
8.9
12.9
2
End-to-end (+ (re-ARR) (Wang et al., 2020b)
16.3
21.8
12.1
10.0
15.1
Cascade SOTA (Li et al., 2021b)
19.4
25.0
14.3
11.7
17.6
XMEF-JT: (1. et al., 2021b)
1.1B
25.8
30.9
18.0
17.0
22.9
Our residts
Wav2vec-2.0 seaf-iraining (LV-60k) (Wang al., 2021)
0.3B
26.0
30.7
23.3
Continue Train
0.3B
25,9
33.3
19.3
24.0
7
PromptST
0.3B
26.4
33.7
19.6
17.9
24.4
Table 2: BLEU scores on four language pairs of CoVoST-V2 test set. <t" indicates that the proposed method is
significantly better than continued train results at a significance level (p < 0.05). We also report the Translation Error Rate (Snover et al., 2006) in Appendix A.4.
Selected Layers
En-De
En-Ca
En-Ar
En-Tr
Model
Layers
En-De
En-Ca
En-Ar
En-Tr
0-24 layers
29.9
36.7
23.5
20.4
0:24
28.0
34.9
21.7
19.6
Adapter
20-24 layers
29.8
36.8
23.4
20.6
12-24
27.9
34,9
21.6
19.5
16-24 layers
29.9
36.5
23.7
20.5
PromptST
12-24
30.1
37.4
21.0'
12-24 layers
30.1
37.4'
23.8
21.0
Table 4: BLEU scores of replacing prompt with the adapter on the valid set. "t" indicates our method out- performs the adapter with a high degree of significance (p < 0.05).
Table BLEU scores adding prompt representations different layers the valid set. =*" indicates that 212-24 layers" significantly outperform "0-24 layers" with high degree of statistical significance (p 0.05).
explored high-level layers. For effectiveness and simplicity, we use the upper half encoder layers " "12-24 layers" as the default strategy.
proceed to train the model with the same number of epochs as our proposed method. The continued training "Continue Train" (#Model "6") achieves 21.0 BLEU on average, which is 0.7 points higher than the baseline, showing the current model has not yet attained full convergence.
Impact of Efficient Tuning Strategy As afore- mentioned, one of the reasons to employ the prompting strategy is to enhance the higher layers without disturbing the phonetic information learned in the lower layers. One may wonder whether other efficient tuning approaches, c.g., Adapter (Houlsby et al., 2019; 9; He et al., 2022; Rao et al., 2023b), are also suitable. To answer this doubt, we investigate the impact of replacing the prompt with the adapter and adding length adapter (Li et al., 2021a; Le et al., 2021) between the encoder and decoder reduce speech length. The result in Table 4 shows that adding adapters higher layers shows com- parable performance with that of adding to all lay- ers (*0-24"). Noticeably, adopting the prompt is significantly better than that of the Adapter. The reason may exactly match our guess; the adapter is akin to the series circuit when inserted into the original encoder blocks (He et al., 2021), heav- ily disturbing the knowledgeable information flow from lower layers, e.g., phonetic information. In contrast, prompt tokens can be analogized the parallel circuit to provide auxiliary capacity with- out affecting the original information.
Equipping with our prompt strategy, which adds trainable tokens from 12 to 24 layers Model "7"), the model outperforms the prior approach in all language directions by average +1.1 BLEU points. Notably, our PromptST also surpasses continue train setting by +0.4 BLEU points, showing the effectiveness of our approach.
4.5 Ablation Study PromptST
We evaluate the impact of different components of PromptST, including (1) prompting on different layers, and (2) tuning with other efficient strategies on En-De, En-Ca, En-Ar, and En-Tr datasets.
Impact of Prompting Layer Table shows the results of equipping Abstract Prompt differ- ent layers. Adding soft-prompts from 12th layer ("12-24 layers") consistently outperforms other variants ("0-24 layers", "20-24 layers", and "16-24 layers"), which we attribute to the advantage of abstract prompt enhancing the critical yet under-

---

Surface
Syntactic
Semantic
Model
Avg.
SeLen
WC
TrDep
ToCo
BShif
Tense
SubNm
ObjNm
SoMo
Coln
Baseline
78. 17
2.47
41.28
50.17
67.49
83.77
83.85
81.77
58.29
61.55
60.88
+PromptST
78.86
4.43
39.54
54.25
69,57
83.77
83,33
82,51
57.12
61,58
61.50
Table Results probing tasks on our constructed Speech-Sentewee We evaluate the linguistic properties learned the up-most layer and find that method preserves knowledge with improved average accuracy. We also report the results of every four layers similar to Table 1 in Appendix A.5.
Baseline
Data Set
Baseline
PromptST
A BLEU
PromptST
Ca-En Es-En De-En
17.85
18.51
0.66
18.00
18.79
0.78
20-
15.88
16.27
0.39
36
Table 6: Results of the extremely low-resource dataset. report the results on the valid set.
12-
030 6:60 8.50
Ratio The Training Data
0.70
Baseline
Figure 4: BLEU scores CoVoST-2 En-Ca valid set with different ratios of the training data.
F-meas
0.65
PrompiST
30
0.60
25
20
4.6 Analysis
frog
mid freq
high freq
(10,20) (30,30)
(30,40)
We conduct a comprehensive analysis of our methodology, addressing: (1) its aptitude in learn- ing linguistic properties, (2) its performance robust- ness across varying data scales, and (3) the nuanced benefits beyond the BLEU score.
Frequency
Length
Figure Word frequency and sentence length analysis.
CoVoST2.0 Ca-En, Es-En, and De-En, containing PromptST Obtains Richer Linguistic Properties According to analyses Section 3, our PromptST may broaden the model capacity of higher layers, affecting the linguistic properties learned by the encoder. To verify this, we select the En-De well- trained model and probe its linguistic properties. Table illustrates that our method indeed preserves richer linguistic knowledge with better average ac- curacy (cspecially on the surface and syntactic, i.c., +1.4%), confirming our hypothesis.
only 99 hours of annotated data on average. Table 6 shows that the baseline indeed presents an overall lower absolute BLEU score (<20) compared the large datasets in Table 3 (~~27). However, our PromptST still significantly improves the perfor- mance by an average of 0.61 BLEU, demonstrating PromptST could be a promising plug-in strategy to provide bonuses for any data scales and corre- spondingly pre-trained models.
Fine-Grained Gains Beyond BLEU Score To understand how PromptST improves translation be- yond BLEU, we use compare-m mt (Neubig et al., 2019) to compare our model against baseline terms of word frequencies and sentence lengths. Figure 5 interestingly shows PromptST (1) tends to be more robust to mid-frequency (10-1000) and figh-frequency words (beyond 1000), while the bascline model performs slightly better rare words (less than 10) and (2) facilitates shorter sen- tences, e.g., <20, compared the baseline, These phenomena show PromptST can better meet the needs of daily oral communication relatively short sentences with high-frequent words.
PromptST Robustly Works Across Data Scales To confirm the effectiveness of our method across different data sizes, we further experiment on the En-Ca dataset partition into different data scales (0.1, 0.25, 0.5). As seen in Figure 4, our simple method boost performances for speech translation models consistently and significantly across the different size of datascts, showing the robustness and effectiveness of approach.
Also, researchers may doubt that our approach may fail in extremely low-resource settings where the backbone ST model is not well pre-trained. To dispel this concern, we conduct experiments on

---

5 Conclusion
GXWD002008111331500000 Shenzhen Sci- ence and Technology Program (Grant No. Xuebo Liu was sponsored by CCF-eencent Rhino-Binn Bird Open Research Fund. We would like to thank the anonymous reviewers and meta-reviewer for their insightful suggestions.
In this paper, we take the first step toward under- standing the fusion of speech and text features in the S2T model by probing tasks, taking the text-to- text model as a reference. Specifically, we design and release 10GB linguistic probing benchmark, Speech-Senteval, for the S2T task. We find that the uppermost encoder layers of the S2T model can not learn linguistic knowledge efficiently, which is vital for translation. Based on these insights, we propose a straightforward plug-in prompt-learning strategy, coined as Abstract Prompt, on the high- level layers to broaden the representation ability of the pre-trained ST models. Experimental results on four widely-used datasets show that PromptST can deliver significant improvements over strong baseline by capturing richer linguistic knowledge.
0BSS202210080311200000
References
Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 82- 62-91, New Orleans, Louisiana. Association for Computational Linguis- ties.
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondiej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang. Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Eliz- abeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alexander Waibel, and Changhan Wang- 2020. FINDINGS OF THE IWSLT 2020 EVAL- UATION CAMPAIGN. In Proceedings of the 17th International Conference on Spoken Language Trans- lation, pages 1-34, Online. Association for Compu- tational Linguistics.
Limitations
While the proposed PromptST model augments the representation power of higher layers in the en- coder, it still has some limitations: (I) our analysis primarily emphasizes the linguistic discrepancies in end-to-end S2T models, sidelining the acoustic nature of
perspective; (2) the resource-intense loading pre-trained weights from existing methods; (3) in Section 4.6, our comparison with Adapter exclusively employs the series adapter technique, neglecting the potential advantages of the parallel adapter structure as highlighted in Gállego et al. (2021), which warrants further exploration.
Alexci Baevski, Yubao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Ethics Statement
Yonatan Belinkov and James R. Glass. 2017. Analyz- ing hidden representations in end-to-end automatic speech recognition systems. In Advances Neural Information Processing Systems 30: Annual Confer- ence Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2441-2451.
In conducting our research, we strictly adhere to all applicable laws and regulations. Additionally, we take ethical considerations seriously and adhere the standards of the ACL Ethics Policy. This paper focuses on improving speech translation. To support reproducibility, both the datasets and code used in this study are available to other researchers upon request. We have made every effort to present our findings and conclusions in an objective and accurate manner.
Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus direct speech translation: Do the differences still make a differ- ence? In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joins Conference on Natu- rul Language Processing (Volume I: Long Papers), pages 2873-2887, Online. Association for Computa- tional Linguistics.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (Grant No. 62206076), Shenzhen College Stability Support Plan (Grant Nos. GXWD2022081 173300003,,
Alexandre Bérard, Olivier Pietquin. Laurent Besacier, and Christophe Servan, 2016. Listen and translate: A

---

proof of concept for end-to-end speech-to teecc-tt-teettee trans- lation. In NIPS Workshop on end-to-end learning for speech and audio processing.
adapters. In Findings of the Association for Computa- tional Linguisties: EMNLI 2022, pages 2184-2190, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loie Barrault, and Marco Baroni. 2018. What you can cram into a single S& vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 2126-2136, Melbourne, Aus- tralia. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NL.P In Pro- ceedings of the 36th International Conference on Ma- chine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Hexuan Deng, Liang Ding, Xuebo Liu, Meishan Zhang. Dacheng Tao, and Min Zhang. 2023. Improving simultaneous machine translation with monolingual data. In Proceedings of the AAAI Conference on Artificial Intelligence, 1, pages 27788-17766
Chiaming Hsu, Changtong Zan, Liang Ding, Longyue Wang. Xiaoting Wang. Weifeng Liu, Fu Lin, and Wenbin Hu. 2023. Prompt-learning for cross-lingual relation extraction. arXiv preprint arXiv:2304. 10354.
Liang Ding and Dacheng Tao. 2019. Recurrent graph syntax encoder for neural machine translation, arXiv preprint S.Ki::1988005533
Hang Le, Juan Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 3520-3533, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 949-959, San Diego, California. Association for Computational Linguistics.
Hang Le, Juan Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. 2021. Lightweight adapter tuning for multilingual speech translation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 817-824, Online. Association for Computational Linguistics.
Helge Dyvik. 1992. Linguistics and machine transla- tion. In Proceedings of the 8th Nordic Conference of Computational Linguistics (NODALIDA 1991), pages 67-78, Bergen, Norway, Norwegian Computing Cen- tre for the Humanities, Norway.
Qingkai Fang. Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguistics.
Phuong-Haai Le, Hongyu Gong, Changhan Wang. Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. Pre-training for speech translation: Cte meets optimal transport. arXiv preprint 1.1v:230/.117116
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiest prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3050 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Gerard I Gállego, Ioannis Tsiamas, Carlos Escolano, José AR Fonollosa, and Marta R Costa-jussà. 2021. Eindto-end speech translation with pre-trained mod- els and adapters: Upe at iwslt 2021. In Proceedings of the 18th International Conference on Spoken Lan- guage Translation (IWSLT 2021), pages 110-119.
Xavier Garcia and Orhan Firat. 2022. Using natural language prompts for machine translation. ArXiv preprint, 15s/2202.11822..
Bei Li, Ziyang Wang. Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu. 2020. Shallow-to- to-eep training for neural machine trans- lation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 995-1995- Online. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2021. Towards unified view of parameter-efficient transfer learning. In International Conference on Learning Representa- tions.
Xian Li, Changhan Wang. Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021a. Multilingual speech trans- lation from efficient finetuning of pretrained models. In Proceedings of the 59th Annual Meeting of the
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022. SparseAdapter: An easy approach for improving the parameter-efficiency of

---

Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 827-838.
Xuebo Liu, Longyue Wang. Derek F. Wong, Liang Ding, Lidia S. Chao, and Zhaopeng Tu. 2021e. Understand- ing and improving encoder layer fusion in sequence- to-sequence learning. In International Conference on Learning Representations.
Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 202 Multilingual speech trans- lation with efficient finetuning of pretrained models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Lan- guage Processing ( (Volume 1: Long Papers), pages 827-838.
Xuebo Liu, Derek F. Wong. Yang Liu, Lidia S. Chao, Tong Xiao, and Jingbo Zhu, 2019a. Shared-private bilingual word embeddings for neural machine trans- lation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3613-3622, Florence, Italy. Association for Compu- tational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b. Multilingual denoising pre- pre-training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning:
Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Yuchen Liu, Hao Xiong. Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang. and Chengqing Zong. 2019b. End-to end speech translation with knowledge distil- lation. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Associ- ation, Graz, Austria, 15-19 September 2019, pages 1128- 1132. ISCA.
Yongjie Lin, Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside bert's linguistic knowl- edge. In Proceedings of the 2019 ACI. Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241-253.
Ilya I Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. 20 P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 110.0762..
Thang Luong, Kyunghyun Cho, and Christopher D. Manning. 2016. Neural machine translation. In Proceedings of the 54th Anmual Meeting of the As- sociation for Computational Linguisties: Tutorial Abstracts, Berlin, Germany. Association for Compu- tational Linguistics.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too. ArXiv preprint, abs/2103. 10.32103.
Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020a. Norm-based curriculum learning for neural machine translation. 58th Annual Meeting of the Association for Compu- tational Linguisties, pages 427-436, Online. Associ- ation for Computational Linguistics.
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt. A tool for holistic comparison of language genera- tion systems. ence of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 35-41, Minneapolis, Minnesota. Association for Computational Linguistics.
Proceedings of the
Proceedings of the 2019 Confer-
Xuebo Liu, Longyue Wang, Derek Wong, Liang Ding, Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2021c. On the complementarity between pre- pre-training and back-translation for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2900-2907, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018 Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Siqi Ouyang, Rong Ye, and Lei Li. 2022. Waco: word- aligned contrastive learning for speech translation. arXiv preprint arXi::2212.00335
Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao. Shuming Shi, and Zhaopeng Tu. 2021d. On the copying behaviors of pre- pre-training for neural machine translation. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4265, Online. Association for Computational Linguistics.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-33 1-318, Philadelphia,

---

Pennsylvania, USA, Association for Computational Linguistics.
Teng Sun, Chun Wang, Xuemeng Song, Fuli Feng, and Liqiang Nie. 2022. Response generation by jointly modeling personalized linguistic styles and emotions. In ACM Trans. Multim. Comput. Commun. Appl., pages 52:1-52:20. ACM.
Shuhan Qi. Zhengying Cao, Jun Rao, Lei Wang. Jing Xiao, and Xuan Wang. 2023. What is the limitation of multimodal Ilms? a deeper look into multimodal Ilms through prompt probing. Information Process- ing & Management, 60(6);103510
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021. Improving speech translation by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4262-4261.
Jun Rao, Liang Ding, Shuhan Qi, Meng Fang. Yang Liu, Li Shen, and Dacheng Tao. 2023a. Dynamic contrastive distillation for image-text retrieval. IEEE Transactions on Multimedia, pages 1-13.
Jun Rao, Xv Meng, Liang Ding, Shuhan Qi, Xuebo Liu, Min Zhang. and Dacheng Tao. 2023b. Parameter- efficient and student-friendly knowledge distillation. IEEE Trans. Multim., pages 1-12.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1998-6008
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 1.842-866.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 1tt Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural I Language Processing: System Demon- strations, pages 33-39, Suzhou, China. Association for Computational Linguistics.
Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Machine Translation: Volume I, Research Papers. pages 83 91, Berlin, Germany. Association for Computational Linguisties.
Changhan Wang, Anne Wu, and Juan Pino. 2020b. Cov- ost 2 and massively multilingual speech-to-text trans- lation. ArXiv preprint, abs/2007. 103 10.
Changhan Wang. Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, and Alexis Conneau. 2021. Large- scale self- and semi-supervised learning for speech translation. In Interspeech 2021, 22nd Annual Con- ference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021, pages 2242-2246. ISCA.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, Association for Computational Lin- guistics.
Chengyi Wang. Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020c. Bridging the gap between pre- training and fine-tuning for end-to-end speech trans- lation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9161-9168. AAAI Press.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does stringbbased neural MT learn source syntax? In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1526- 1534, Austin, Texas. Association for Computational Linguistics.
Matthew Snover, Bonnie DOrr, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annota- tion. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Pupers, pages 223-231.
Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li juan Shen, and Changxing Ding. 2023. Can linguis- tic knowledge improve multimodal alignment in vision-language pretraining? ArXiv preprint, 12.22308.12899.
Mihaela C. Stoian, Sameer Bansal, and Sharon Gold- water. 2020. Analyzing ASR pretraining for low- resource speech-to-text translation. In 2020 JEFF International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain. May 4-8, 2020, pages 7909-7913. IEFE
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine

---

translation. In Proceedings of the 57th Anmual Meet- ing of the Association for Computational Linguistics, pages 18 1810-1822, Florence, Italy. Association for Computational Linguistics.
Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022. On the complementar ity between pre-training and randomiinitialization for resource-rich machine translation. In Proceedings of the 29th International Conference on Computational Linguisties, pages 5029-5034, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Xiangpeng Wei, Heng Yu, Yue Hu, Yue Zhang, Rongx- iang Weng, and Weihua Luo. 2020. Multiscale col- laborative deep models for neural machine transla- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 414426,, Online. Association for Computational Lin- guistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-aa natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. 2019. Pay less atten- tion with lightweight and dynamic convolutions. In 7th International Conference Learning Represen- tations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju. Tong Xiao, and Jingbo Zhu. 2021. Stacked acoustic-and-teexee the pre-trained models into speech translation en- coders. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 Itth International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2619-2630,
encoding: Integrating
Baosong Yang. Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the ability of self-attention networks to learn word order. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3635- 3644, Florence, Italy, Association for Computational Linguistics.
Rong Ye. Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5099-5113, Seattle, United States. Association for Computational Linguistics.
Wenbiao Yin, Zhicheng Liu, Chengqi Zhao, Tao Wang, Jian Tong, and Rong Ye. 2023. Improving speech translation by fusing speech and text. arXiv preprint arXiv:2305.144 14042.
Dong and Deng. 2016. Automatic speech recog- nition, volume I. Springer.

---

Tasks
Train
Valid
Test
Cate.
Model
En-De
Word Content
5,000
300
200
997
Random Initialized
1.7
Sentence Length
5,000
300
200
5
Finetune Encoder and Prompt
23.6
Top Constituents
5,000
300
200
20
Finetune Prompt
25.9
Tree Depth
5,000
300
200
7
Finetune Encoder Decoder and Prompt
26.0
Bigram Shift
7,000
300
200
2
Finetune Decoder and Prompt
26.4
Coordination
5,000
300
2
Object Number
5,000
300
200
2
Table 9: BLEU scores for randomly initialized model and different fine-tune strategies on test set.
Past Present
5,000
300
200
2
Subject Number
5,000
300
200
2
Odd Man Out
5,000
300
200
2
found that shorter prompts are sufficient for lan- guage pairs that are closely related (e.g., En-De), However, for long-distance language pairs (e.g., En-Ca, En-Ar, En-Trr), prompts longer than 200 tokens are found to be beneficial.
Table 7: Statisties of Speech Speech-Senteval Senteval benchmark.
Model
En-De
En-Ca
En-Ar
En-
MI.P Encoder
29.8
36.9
23.4
20.7
prefix-40
30.1
37.2
23.6
20.8
prefix-100 100
30.1
37.2
23.6
20.8
prefix-200
30.0
37.2
23.7
21.0
A.3
Pre-training and Efficient Fine-tuning
prefix-300
30,0
37.4
23.8
20.8
To demonstrate the importance of pre-trained mod- cls in an end-to-end S2T model, we conduct exper- iments using the same settings to train a randomly initialized En-De model. As shown Table 9, the use of random initialization results in a low BLEU score, indicating that the performance of the S2T model is heavily dependent on being initialized with pre-trained parameters.
prefix-400
30.0
37.3
23.8
20.8
BLEU scores the prompt length and reparamctrization (prefix-40) on valid set.
Table
A
Appendix
A.1
Speech-Senteval Data Set
We further evaluate the efficiency of various tun- ing strategies, namely: (1) fine-tuning the prompt only, (2) comprehensive fine-tuning encompassing prompt, encoder, and decoder, (3) fine-tuning the encoder with the prompt, and (4) fine-tuning the decoder with the prompt. Table 9 illustrates that fine-tuning the decoder with the prompt achieves the most optimal results. Specifically, jointly fine- tuning the encoder and prompt led to swift overfit- ting. This suggests that exhaustive encoder adjust- ments might not ideal.
For each task, we construct training sets containing training, test and validation data sets. The overall volume of data is presented in Table 7. All sets are balanced, having an equal number of instances in each target class.
A.2 Experiment Optimization
Previous studies have commonly employed a repa- rameterization encoder, such as a multi-layes per- ceptron (MLP), to optimize soft- soft-prompt representa- tions for NLU tasks. As demonstrated by Liu et al. (2021b), the use of an MLP reparameterization can enhance the robustness and performance of models. In order evaluate the performance of the S2T model, we employ simple MLP layer to encode the trainable prefix tokens. However, our experi- ments reveal that the use of an MLP led to negative effects for nearly all language pairs.
A.4 Comparision Translation Error Rate In this study, we evaluate the performance of both the baseline and the PrompST models using the Translation Error Rate (TER) metric. The results, presented in Table 11, demonstrate that when our prompt strategy is applied, the model achieves an average TER score of 57.2 the validation sets. This represents significant reduction in the trans- lation error rate across all language directions. with an average decline of 3.5 points, showing the efficacy of our proposed approach.
Furthermore, the prompt length has been iden- tified a crucial factor in the S2T model. We conducted an extensive search of the optimal prefix length for four language pairs within the range of [40, 400), respectivelyy As depicted in Table 8,
A.5 Speech-Senteval Results PromptST
"Following https: htt.ttttbb...../ httpllc.ooocccaeeeeeeeee con//facbookressarch SentEval SentEval/tree/ma http///eeeeeeeeeeeee ///aat//poohatt the dataset is made available under the BSD 3-Clause License.
Table 10 presents the performance of the S2T model using the PromptST approach on all 10 prob-

---

Categories
Task
Emb
8
12
16
20
24
Word Content
0.4
1.1
2.5
14,4
11.8
14.2
4.4
Surface
Sentence Length
76.5
68.1
76.5
77.3
79.5
80.5
78.9
Top Constituent
32.4
39.5
48.4
62.2
67.6
61.9
54.3
Synatactic
Tree Depth
19.7
28.5
28.0
36.0
31.7
31.2
39.5
Bigram Shift
45.5
49.7
57.2
58.7
71.2
67.3
Coordination Inversion
54.7
46.3
51.2
61.5
59.8
57.3
61.6
Object Number
78.9
80.4
78.1
81.2
78.1
82.5
Semantic
Past Present
67.2
69.0
78.0
81.6
80.8
83.8
83.8
Subject Number
53.7
65.6
70.4
79.9
78.8
81.8
83.3
Odd Out
50.9
47.1
49.8
52.9
53.9
57.1
Table 10: Probing task results for PromptST. We show the analysis results of every four layers.
Model
En-De
En-Ca
En-Ar
En-Tr
Avg.
Baseline
56.8
52.4
67.9
65.5
60.7
PromptST
55.2
48.1
62.9
62.4
57.2
Table 11: Results Translation Error Rate valid set.
ing tasks. We run the analysis every four layers. Our experimental data clearly indicate that using PromptST enhances the model's ability retain complex linguistic information. We specifically ob- serve improvements in capturing surface and syn- tactic properties at layers 12, 16, and 24, with in- creases of +3.2, +1.9, and + +1.4, respectively. These findings suggest that PromptST effectively expands the lcarning capacity of the model's higher layers, positively influencing the types of linguistic prop- erties that the encoder can learn.


Reviews:
[{"review_id": "AckPRYt4zd", "reviewer": "Reviewer_aEu5", "summary": "", "questions": "- Why is the choice of 7 layers for the decoder? \n- Why continued training gives much better result for En-Ca compared to remaining pairs? \n- How the performance of acoustic features are evaluated in Figure 2?\n- It would be interesting to evaluate the performance of this prompting-based plug-in in another popular ST benchmark such as MuST-C where many strong ST systems report the results on.\n- Do you think this approach is complementary to existing strong approaches in building ST systems such as multi-task learning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> Q1.1 *The literature review of methods to mitigate the speech-text modality gap or speech-text fusion techniques seems to be too brief (L57-59). For example, methods such as the increasingly common cascaded encoders approach[1-2], using implicit guidance from external ASR model[3], aligning speech and text sequences at the supervised pre-training stage[4], or joint speech-text pre-training approaches [5-6].*  \n\nWe appreciate the feedback regarding the brevity of our literature review on methods that address the speech-text modality gap and fusion techniques. Recognizing the importance of these references, we will ensure they are thoroughly discussed in the Introduction and Related Work sections of our revised manuscript.\n\nIt's worth emphasizing the unique trajectory our study has taken in contrast to the aforementioned works. Unlike the techniques presented in references [1-5], which prioritize introducing methods prior to embarking on analysis, our methodological approach revolves around a deep analysis that precedes validation of the method. This distinction underscores our key contribution: revealing novel insights into the intricacies of end-to-end models.\n\nIn particular, while reference [6] elucidates the complexity and significance of the ST encoder, our research ventures further. We delve into a nuanced exploration of the ST encoder by assimilating both acoustic and linguistic analyses. This depth of investigation, we believe, sets our work apart and adds a unique dimension to the existing body of literature.\n\n**References**  \n[1] Liu et al., Bridging the Modality Gap for Speech-to-Text Translation. arXiv2020.  \n[2] Yin et al., Improving speech translation by fusing speech and text. arXiv2023.  \n[3] Le et al., Pre-training for Speech Translation: CTC Meets Optimal Transport. arXiv2023.  \n[4] Bapna et al. SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training. arXiv2021.  \n[5] Tang et al. Unified Speech-Text Pre-training for Speech Translation and Recognition. ACL2022.  \n[6] Xu et al. Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders. ACL2021.  \n\n> Q1.2 *Limited comparison to parameter-efficient techniques other than adapter-tuning.* \n\nOur primary objective in this study centers on a comprehensive evaluation of the model through diverse probing tasks. Consequently, our emphasis wasn't on drawing comparisons among various fine-tuning techniques. Rather, we anchored our approach on simple yet efficient fine-tuning mechanisms to corroborate our findings.\n\nHowever, we recognize the potential value of such broader comparisons. Given the robustness of our observations supported by our experiments, we are confident that these insights can be extrapolated and hold promise when integrated with other parameter-efficient techniques. In future iterations of this work, we will certainly consider expanding the comparative spectrum to encompass additional fine-tuning methodologies.\n\n> Q1.3 *Why is the choice of 7 layers for the decoder?*\n\n We grounded our selection in the wav2vec 2.0+self-training (Wang et al., 2021) baseline, a cornerstone in current speech processing research. By aligning with this recognized model configuration, we aimed to facilitate not only a direct and reliable comparison but also to anchor our work within the best practices of contemporary research.\n\n**References**  \n[1] Wang et al., Large-scale self-and semi-supervised learning for speech translation. Interspeech2021.  \n\n> Q1.4 *Why continued training gives much better result for En-Ca compared to remaining pairs?*  \n\nOur hypothesis for the significant improvement observed for En-Ca, when subjected to continued training, stems from a potential under-convergence in the initial checkpoint provided by Wang et al.(2021) specifically for this language pair. The rapid rate of convergence we witnessed during our extended training phase reinforces this belief. This suggests that the model had more room to optimize for En-Ca, and by granting it additional training time, we were able to harness this potential.\n\n> Q1.5 *How the performance of acoustic features are evaluated in Figure 2?*  \n\nTo assess the performance of acoustic features as presented in Figure 2, we undergo a multi-step evaluation:  \n1. We begin by extracting frame-level features from every layer of the encoder.\n2. These extracted features are then employed in a phoneme recognition task, leveraging the TIMIT dataset.\n3. Subsequently, we train a supervised classifier using the aforementioned features.\n4. Finally, the trained classifier's performance is gauged on the test dataset.  \n\nRecognizing the importance of transparency in methodology, we are committed to providing a more granular and comprehensive description of this process in our revised manuscript. \n\n\n> Q1.6 *Evaluate the performance of this prompting-based plug-in in MuST-C dataset.*  \n\nThank you for your constructive suggestion. Currently, we have conducted a preliminary evaluation of PromptST on the MuST-C en-de tst-COMMON dataset, and the results are tabulated as follows:\n\n\n| Continue Train | PromptST |\n|:--------------:|:--------:|\n|      16.8      | **17.1** |\n\n\nIt's important to note that, given our time limitations, this evaluation utilizes a model trained on CoVoST training data, not the MuST-C training data. Despite this constraint, the initial findings are promising, showcasing that PromptST achieves a marginally higher performance than the continue-trained model. We plan to undertake more comprehensive experiments in the future to substantiate our observations.  \n\n> Q1.7 *Do you think this approach is complementary to existing strong approaches in building ST systems such as multi-task learning?*\n\nAbsolutely. We believe that our approach can be seen as a complementary strategy to the strong methodologies currently being developed for ST models.\n\nGiven the pervasive challenge of data scarcity in the domain of speech translation, it's increasingly common to bootstrap models with pre-trained parameters. Notably, the multi-task learning methods as introduced by Ye et al. (2022) and Tang et al. (2021) also employ wav2vec2 for initializing their models, echoing our approach.\n\nWhat bolsters our confidence in our approach's general applicability is its inherent flexibility. Our method is not only amenable to integration with newer models but also the insights we've derived have the potential to propel further research in this domain. We envision our findings enriching the current repertoire of techniques, potentially leading to breakthroughs in ST system performance.\n\n**References**  \n[1] Ye et al., Cross-modal Contrastive Learning for Speech Translation. NAACL2022.  \n[2] Tang et al., Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task. ACL2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper offers an interesting analysis that helps strengthen understandings of the behaviors of acoustic encoder in the ST model. \n- The release of a new probing benchmark would facilitate the analysis of speech encoding methods in the future.\n- The use of prompt learning for ST is novel and well-justified.", "reasons_to_reject": "1) The literature review of methods to mitigate the speech-text modality gap or speech-text fusion techniques seems to be too brief (L57-59). For example, methods such as the increasingly common cascaded encoders approach (Liu et al. 2020 \"Bridging the Modality Gap for Speech-to-Text Translation\", Xu et al. 2021 \"Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders\"), using implicit guidance from external ASR model (Yin et al. 2023 \"Improving speech translation by fusing speech and text\"), aligning speech and text sequences at the supervised pre-training stage (Le et al. 2023 \"Pre-training for Speech Translation: CTC Meets Optimal Transport\"), or joint speech-text pre-training approaches (Bapna et al. 2021 \"SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training\", Tang et al. 2022 \"Unified Speech-Text Pre-training for Speech Translation and Recognition\"), etc.\n2) Limited comparison to parameter-efficient techniques other than adapter-tuning, for example LNA tuning (Li et al., 2020 \"Multilingual speech translation with efficient fine-tuning of pre-trained models\"), MAM adapter (He et al. 2022 \"Towards a Unified View of Parameter-Efficient Transfer Learning \").", "questions_to_authors": "- Why is the choice of 7 layers for the decoder? \n- Why continued training gives much better result for En-Ca compared to remaining pairs? \n- How the performance of acoustic features are evaluated in Figure 2?\n- It would be interesting to evaluate the performance of this prompting-based plug-in in another popular ST benchmark such as MuST-C where many strong ST systems report the results on.\n- Do you think this approach is complementary to existing strong approaches in building ST systems such as multi-task learning?", "paper_topic_and_main_contributions": "This paper investigates the acoustic and linguistic behaviors of ST models using a linguistic probing benchmark newly proposed in this paper. By understanding how the acoustic encoder encodes different types of acoustic and linguistic knowledge, the paper proposes a plug-in prompt-learning strategy to better fuse the speech and text modalities in ST model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- As a minor suggestion, additional references to related findings in previous work would improve the paper, for example, the observation that the modality gap is noticeable in top decoder layers which are important to translation quality (Tang et al. 2021 \"Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task\"), or the hierarchy of linguistic information in Transformer encoder's hidden representation (Lin et al. 2019 \"Open Sesame: Getting inside BERT’s Linguistic Knowledge\", Rogers et al. 2020 \"A Primer in BERTology: What We Know About How BERT Works\"). \n- L473: Le et al. 2021 (Lightweight Adapter Tuning for Multilingual Speech Translation) investigates the use of both sequential and parallel adapters for speech translation.", "Typos_Grammar_Style_and_Presentation_Improvements": "- L193: lack of period.\n- L224: incomplete sentence\n- L243: incorrect period\n- L322: \"an raw waveform input\" -> \"a raw waveform input\"\n- Figure 3: \"For a $M$-layers encoder\" -> \"For an $M$-layers encoder\"\n- L525: \"... by averaging +0.61 BLEU\": a bit confusing"}}, {"review_id": "a9DhzAAInZ", "reviewer": "Reviewer_gne1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> Q2.1 *I am not sure what's the benefit of only using efficient approach on the encoder side. The model seems to fix the encoder but fully fine-tune the decoder if I understand correctly. One of the main reasons of prompt learning is to significantly reduce the trainable parameters during the fine-tuning, but this benefit is not very clear if decoder can be fully fine-tuned. why not just fully fine-tune the entire model instead?*  \n\nThank you for raising this insightful question regarding our model fine-tuning strategy. In our study, as detailed in Section 3, we underscore the pivotal role of the encoder, particularly given its responsibility for managing intricate tasks. From our analyses, we've observed that the higher layers of pre-trained encoders might not be optimally primed to encapsulate linguistic intricacies inherent to speech translation (ST).\n\nOur experimental journey provided several insights:  \n1. *Whole Model Fine-tuning*: Our initial attempts at fine-tuning the entire model illuminated two major issues. First, it proved to be considerably resource-intensive. Second, and more crucially, the results were not as promising as anticipated.\n2. *Prompt-only Fine-tuning*: Given the findings from our initial experiment, we explored fine-tuning only the prompt. However, this approach didn't yield satisfactory performance, potentially stemming from the multifaceted nature of the ST challenge. It became apparent that solely adjusting the prompt couldn't unlock the model's full potential.\n3. *Focused Fine-tuning Strategy*: Based on our earlier learnings, we formulated an optimized strategy. We decided to hold the encoder constant, but fully fine-tune both the decoder and the prompt. To provide a perspective on parameter efficiency: the combined parameters of the decoder and the prompt amount to 17 million, a mere fraction when compared to the encoder's substantial 310 million parameters.\nThis fine-tuned strategy, while being more resource-conservative, also culminated in markedly improved fine-tuning outcomes. Thus, our approach harmoniously melds efficiency with performance, paving the way for successful ST model deployment.\n\n**References**  \n[1] Xu et al. Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders. ACL2021.\n\n\n> Q2.2 *The baseline models are not very strong as the proposed model simply contains more tunable parameters compared with model 5,6. It is expected to perform better than them by nature. To more highlight the benefit of prompt, It might be better to have experiments such as 0) prompt both encoder and decoder but fixing encoder/decoder 1) fine-tune prompt only by fixing both encoder and decoder. 2) fine-tune all prompt, encoder, decoder 3) fine-tune encoder/decoder without prompt.*  \n\nThank you for the thoughtful feedback, and we concur with your viewpoint that assessing our method against a variety of tuning strategies is essential for a holistic understanding.\n\nTo address your concerns, we embarked on comprehensive experiments for the English-to-German translation task. Here's a breakdown of our findings:\n\n***********************************************************\n|             **Strategy**            |   BLEU |\n|:------------------:|:--------:|\n|     finetune encoder and prompt     |   23.6   |\n|           finetune encoder          |   23.8   |\n|     finetune encoder and decoder    |   25.5   |\n|           finetune decoder          |   25.9   |\n|           finetune prompt           |   25.9   |\n| finetune encoder decoder and prompt |   26.0   |\n|     finetune decoder and prompt     | **26.4** |\n************************************************************\n\nThese experiments solidify our understanding in several ways:\n- *Isolated Encoder Fine-tuning*: When solely fine-tuning the encoder, we observed a tendency for rapid overfitting, indicating that this alone might not be the best strategy.\n- *Synergistic Approach*: Our original method of fine-tuning the decoder and prompt showcased the best performance. This aligns with our hypothesis that integrating prompts with the decoder's tuning can provide a balance, enhancing model generalization while mitigating overfitting tendencies.\n- *Prompt's Efficacy*: When tuning only the prompt, the model delivered competitive results, underscoring the prompt's potential to improve performance even without adjusting the core model components.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work is well-motivated by its observation that upper layer of encoder does not encode linguistic information efficiently. The proposed soft prompt is able to address this problem to some extent.\n- it contains several detailed analysis which helps more to understand the model. personally I find the analysis of prompt length to be very interesting\n- it proposes a new dataset/benchmark to probe information from S2T model", "reasons_to_reject": "- while I understand the motivation of the proposed approach, I am not sure what's the benefit of only using efficient approach on the encoder side. The model seems to fix the encoder but fully fine-tune the decoder if I understand correctly. One of the main reasons of prompt learning is to significantly reduce the trainable parameters during the fine-tuning, but this benefit is not very clear if decoder can be fully fine-tuned. (still half of the parameters are trainable). why not just fully fine-tune the entire model instead?\n- The baseline models are not very strong as the proposed model simply contains more tunable parameters compared with model 5,6. It is expected to perform better than them by nature. To more highlight the benefit of prompt,  It might be better to have experiments such as 0) prompt both encoder and decoder but fixing encoder/decoder 1) fine-tune prompt only by fixing both encoder and decoder. 2) fine-tune all prompt, encoder, decoder 3) fine-tune encoder/decoder without prompt", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work first proposes to analyze S2T (speech-to-text) encoder by probing acoustic and linguistic information. A comparison with T2T shows that upper encoder layer might not learn linguistic information efficiently. Motivated by this observation, they proposes to attach soft prompt to the upper encoder layers to be jointly fine-tuned with the decoder. The encoder of proposed model achieves a better performance on the linguistic probing tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FRtvTYG6yr", "reviewer": "Reviewer_A846", "summary": "", "questions": "1. There is another method to initialize the S2T models with pretrained speech model and text translation model: leveraging speech encoder with text encoder on top of it for S2T encoder initialization, and text decoder for S2T decoder initialization (Xu et al. 2021; Fang et al. 2022; ...). Have you tested the results of this method? \n\nReferences:\n\nXu et al. 2021: Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders, ACL2021\n\nFang et al. 2022: STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation, ACL2022", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> Q3.1 *The proposed method seems not aligned with the analyzed resulsts on linguistic evaluation. How can the prompt representations improve the linguistic information learning on top encoder layers? It lacks the corresponding analysis between their proposed model PromptST and the baseline, like similar results in Figure 1. If we only see the results of Table 5, the improvement is not consistent and we cannot see the results with the middle layers. Although the final translation results are improved, it is not clear how the improvement come. Is it only from the extra parameters brought by the prompt embeddings?*\n\nBased on our analysis, we suspect that higher-level layers in the model may struggle to capture linguistic nuances, possibly due to limitations in their capacity. Previous research [1-3] supports the idea that enhanced representations can improve translation quality. In light of this, we introduced additional features to the upper layers of the encoder to augment their representational power.  \n\nFor our proposed PromptST method, we did conduct a detailed analysis, similar to what is presented in Figure 1. Due to space constraints in the current paper, these specific findings are not included. However, we are considering incorporating them in the next version for a more complete understanding. The table below presents the results based on models using the PromptST approach.\n*****************************************************************\n|            |                        |     Emb    |        4       |      8     |       12       |       16       |     20     |       24       |\n|:----------:|:----------------------:|:----------:|:--------------:|:----------:|:--------------:|:--------------:|:----------:|:--------------:|\n|   Surface  |      Word Content      |     0.4    |       1.1      |     2.5    |      14.4      |      11.8      |    14.2    |       4.4      |\n|            |     Sentence Length    |    76.5    |      68.1      |    76.5    |      77.3      |      79.5      |    80.5    |      78.9      |\n| Synatactic |     Top Constituent    |    32.3    |      39.3      |    48.4    |      62.2      |      67.6      |    61.9    |      54.3      |\n|            |       Tree Depth       |    19.7    |      28.5      |    28.0    |      36.0      |      31.7      |    31.2    |      39.5      |\n|            |      Bigram Shift      |    45.5    |      49.7      |    57.2    |      58.7      |      71.2      |    67.3    |      69.6      |\n|  Semantic  | Coordination Inversion |    54.7    |      46.3      |    51.2    |      61.5      |      59.8      |    57.3    |      61.6      |\n|            |      Object Number     |    65.2    |      78.9      |    80.4    |      78.1      |      81.2      |    78.1    |      82.5      |\n|            |      Past Present      |    67.2    |      69.0      |    78.0    |      81.6      |      80.8      |    83.8    |      81.9      |\n|            |     Subject Number     |    53.7    |      65.6      |    70.4    |      79.9      |      78.8      |    81.8    |      83.8      |\n|            |       Odd Man Out      |    55.6    |      50.9      |    47.1    |      49.8      |      52.9      |    53.9    |      55.6      |\n**********************************************\nOur experimental data clearly indicate that using PromptST enhances the model's ability to retain complex linguistic information. We specifically observe improvements in capturing surface and syntactic properties at layers 12, 16, and 24, with increases of +3.2, +1.9, and +1.4, respectively. These findings suggest that PromptST effectively expands the learning capacity of the model's higher layers, positively influencing the types of linguistic properties that the encoder can learn.\n\n**References:**  \n[1] Wu et al., Pay less attention with lightweight and dynamic convolutions. ICLR2019.  \n[2] Wei et al., Multiscale collaborative deep models for neural machine translation.  ACL2020.  \n[3] Wang et al., Learning deep transformer models for machine translation. ACL2019.  \n\n> Q3.2 *The method is only evaluated on CoVoST-v2. Another polular dataset, MUST-C, is not evaluated, which makes the results less persuasive.* \n\nFollowing your constructive suggestion, we have conducted a preliminary evaluation of PromptST on the MuST-C en-de tst-COMMON dataset, and the results are tabulated as follows:\n*******************************************\n| Continue Train | PromptST |\n|:--------------:|:--------:|\n|      16.8      | **17.1** |\n**************************************\nIt's important to note that, given our time limitations, this evaluation utilizes a model trained on CoVoST training data, not the MuST-C training data. Despite this constraint, the initial findings are promising, showcasing that PromptST achieves a marginally higher performance than the continue-trained model. We plan to undertake more comprehensive experiments in the future to substantiate our observations.\n\n> Q3.3 *There is another method to initialize the S2T models with pretrained speech model and text translation model. Have you tested the results of this method?*  \n\nThank you for bringing this to our attention, which we had previously overlooked. We've conducted further experiments to test our method when initialized with other pre-trained models, but these are still in progress due to time constraints. Nonetheless, we remain confident that our approach is applicable to various pre-trained models.  \n\nOur research focuses on a model that is already well-regarded and widely used in the speech-to-text field. We begin by situating it within a broader framework, followed by an in-depth performance analysis. We believe that the contributions of our research go beyond the specific method we've developed. This includes the new benchmark we introduce and the broader conclusions we draw, all of which are designed to make meaningful contributions to future research in this domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introducing a linguistic probing benchmark, nameed Speech-Senteval, for speech models  and conducting an interesting analysis on comparisons between S2T and T2T models accordingly.", "reasons_to_reject": "1. The proposed method seems not aligned with the analyzed resulsts on linguistic evaluation. How can the prompt representations improve the linguistic information learning on top encoder layers? It lacks the corresponding analysis between their proposed model PromptST and the baseline, like similar results in Figure 1. If we only see the results of Table 5, the improvement is not consistent and we cannot see the results with the middle layers. Although the final translation results are improved, it is not clear how the improvement come. Is it only from the extra parameters brought by the prompt embeddings?\n\n2. The method is only evaluated on CoVoST-v2. Another polular dataset, MUST-C, is not evaluated, which makes the results less persuasive.\n", "questions_to_authors": "1. There is another method to initialize the S2T models with pretrained speech model and text translation model: leveraging speech encoder with text encoder on top of it for S2T encoder initialization, and text decoder for S2T decoder initialization (Xu et al. 2021; Fang et al. 2022; ...). Have you tested the results of this method? \n\nReferences:\n\nXu et al. 2021: Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders, ACL2021\n\nFang et al. 2022: STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation, ACL2022", "paper_topic_and_main_contributions": "This paper focuses on the analysis and improvement on end-to-end S2T models when directly finetuned with pretrained speech encoder and text translation decoder, to mitigate the modality gap. The authors first conduct an analysis on several linguistic tasks and find the differences between S2T and T2T models. Then they propose a new model called PromptST to improve the performance following  P-Tuning V2. The results on CoVoST-v2 datasets show the consistent improvement over baselines in terms of both S2T performance and the linguistic evaluation benchmark.\n\nThe contributions include:\n1. Introducing a linguistic probing benchmark, nameed Speech-Senteval, for speech models.\n2. An interesting analysis on comparisons between S2T and T2T models for linguistic tasks.\n3. Proposing PromptST for S2T with improvement over baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xcbDu4t3kE", "reviewer": "Reviewer_LoxQ", "summary": "", "questions": "Since the parallel adapter also shows good results, I wonder how the author proposed approach differs in this case. And based on the references in the paper, it seems that the authors do not use length adapter (1D convolution) to reduce speech length. This comparison may seem unfair, since the length reduction is usually required when the adapter is applied in E2E-ST", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> Q4.1 *There have been previous analyses of the network representation problem [1,2,3]. I also admit that the author's analysis is sufficient, but it is not novel or offers new insights.*  \n\nThank you for bringing this to our attention. While we recognize and respect the contributions made by the cited works, we believe our research offers a distinct perspective in its approach and findings.\n\n- *Preliminary Analysis Before Method Validation*: Unlike [1] and [2], which introduce methods prior to their analyses, our study places a significant emphasis on initiating with an in-depth preliminary analysis. This approach allows us to deeply understand the underpinnings of the problem, which subsequently guides our method validation.\n\n- *In-Depth Exploration of the Encoder*: The work by [3] underscores the complexity and significance of the ST encoder. However, our analysis delves deeper, exploring both the acoustic and linguistic intricacies of the encoder. This granular assessment not only reaffirms the encoder's importance but also unravels nuances previously unexplored.\n\n- *Novel Insights for the Community*: The unique vantage point from which we approached our research has enabled us to unearth fresh insights. These findings have the potential to catalyze further advancements in speech translation and enrich the broader discourse on speech-text bimodality.\n\n**References**  \n[1] Dong et al., Listen, understand and translate: Triple supervision decouples end-to-end speech-to-text translation. AAAI2021.  \n[2] Zhang et al., Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning. TASLP2023.  \n[3] Xu et al., Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders. ACL2021.  \n\n> Q4.2 *The contrast with adapter method in Section 4.5 is not sufficient. In the previous work [1], the combination of length adapter and modal adapter is usually used. It seems that the authors only use the modality adapter in their experiments.*  \n\nWhile we appreciate the potential advantages of a length adapter, incorporating it currently falls outside the scope of our research. In Section 4.5, we focus on confirming our observation that higher encoder layers are less effective at processing linguistic features. To maintain a fair comparison, we align our methodology with a default model configuration, which doesn't include a length adapter. Instead, it employs an encoder-to-decoder projection to address dimension discrepancies between the encoder and decoder. \n\nThrough our research, we strategically positioned series adapters within specific tiers of the encoder layer. Empirical results showcase the efficacy of this approach, particularly when adapters are located at top layers. This strategy underlines our observations, promotes a clearer understanding, and facilitates a meaningful comparison with other methods.\n\nWe hope this clarifies our rationale and the scope of our study.\n\n**References**  \n[1]  Gállego et al., Upc’s speech translation system for iwslt 2021. IWSLT2021\n\n>Q4.3 *The author mentioned that the p-tuning is a parallel method, while the adapter is serial. However, some works [1] also use the parallel adapter structure, showing better results. Since the parallel adapter also shows good results, I wonder how the author proposed approach differs in this case.*\n\nThank you for pointing out the contributions of parallel adapter structures as seen in [1]. In our study, the distinction between serial and parallel methods isn't intended as a hierarchy of effectiveness but rather as two distinct methodologies to validate our observations.\n\nWhile we acknowledge the demonstrated efficacy of the parallel adapter structure in [1], our primary focus in this paper was to employ both parallel (p-tuning) and serial (modal adapter) methods to substantiate specific findings about the behavior and characteristics of the encoder layers. The intention was not necessarily to claim one approach as superior to the other but to shed light on how each method complements our understanding.\n\nThat said, we truly value your suggestion on the potential synergy between serial and parallel methods. This presents an exciting avenue for future research, and we are keen on exploring how a joint approach might further enhance speech translation outcomes.\n\n**References**  \n[1]Tsiamas et al., Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022.  IWSLT2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Motivation is detailed and clear.\n2. The s2t behaviors is well analyzed. Based on the above analysis, the authors find that high-level layers play a crucial role in undertaking linguistic representation tasks. However, the standard, pre-trained end-to-end S2T model represents linguistic features sub-optimally at higher encoder layers compared to the T2T model. \n3. PromptST exceeds previous models and significantly improves model performance.\n4. The experimental analysis part is  comprehensive, which is helpful to understand the mechanism of the proposed method.\n5. Authors publicly release their probing benchmark Speech-Senteval.", "reasons_to_reject": "1. There have been previous analyses of the network representation problem [1,2,3]. I also admit that the author's analysis is sufficient, but it is not novel or offers new insights.\n2. The contrast with adapter method in Section 4.5 is not sufficient. In the pervious work [4], the combination of length adapter and modal adapter is usually used. It seems that the authors only use the modality adapter in therir experiments.\n3. The author mentioned that the p-tuning is a parallel method, while the adapter is serial. However, some works [5] also use the parallel adapter structure, showing better results.\n\n[1] “Listen, Understand and Translate”: Triple Supervision Decouples End-to-end Speech-to-text Translation.\n\n[2] Stacked Acoustic-and-Textual Encoding Integrating the Pre-trained Models into Speech Translation Encoders.\n\n[3] Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning.\n\n[4] UPC’s Speech Translation System for IWSLT 2021, arXiv:2105.04512v1\n\n[5] Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022. https://aclanthology.org/2022.iwslt-1.23.pdf", "questions_to_authors": "Since the parallel adapter also shows good results, I wonder how the author proposed approach differs in this case. And based on the references in the paper, it seems that the authors do not use length adapter (1D convolution) to reduce speech length. This comparison may seem unfair, since the length reduction is usually required when the adapter is applied in E2E-ST", "paper_topic_and_main_contributions": "This paper is about end-to-end speech translation. The author focuses on the following two problems in E2E-ST: 1) how are speech and text modalities fused in S2T model? and 2) how to better fuse the two modalities? Preliminary analysis reveals that the uppermost encoder layers of the S2T model can not learn linguistic knowledge efficiently, which is crucial for accurate translation. Based on the finding, authors propose a prompt-enhanced S2T model, PromptST. Experimental results on four widely-used S2T datasets show that PromptST can deliver significant improvements over a strong baseline by capturing richer linguistic knowledge.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W32ByVPnBh", "reviewer": "Reviewer_68rb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> Q5.1 *As identified by the authors, to me it is a shortcoming that the analysis is carried out purely from the linguistic performance point of view. Considering this is a conference focused on language tasks this is not a deal breaker, but a more balanced analysis would greatly reinforce the potential of impacting other fields.*  \n\nThank you for your positive comments on our paper, and for bringing to our attention the need for a more balanced analysis of the acoustic dimensions in our study. In subsequent research, we plan to explore both acoustic aspects and machine learning in greater depth to offer a more complete analysis of the subject matter.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper carries out a thorough analysis of information and probing details within layers/models that can help others in the community understand their models (and contributions) and the contribution of different input signals better. Overall validation of the technology is robust and convincing.", "reasons_to_reject": "As identified by the authors, to me it is a shortcoming that the analysis is carried out purely from the linguistic performance point of view. Considering this is a conference focused on language tasks this is not a deal breaker, but a more balanced analysis would greatly reinforce the potential of impacting other fields.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an extended approach to probing of ML model information for speech-related tasks (together with a public benchmark). Through the analysis, the authors provide a methodology to understand which information is processed where in the model (the balance between linguistic and acoustic information mostly). As a conclusion, then we see a proposed method (Abstract Prompting) that is demonstrated to enhance the representation capabilities of speech translation models (tested on multiple languages with a positive impact in BLEU scores).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
The paper focuses on analyzing and improving end-to-end (E2E) Speech-to-Text (S2T) models by probing acoustic and linguistic information. The authors introduce a linguistic probing benchmark, Speech-Senteval, and propose the PromptST model, which utilizes prompt learning to address modality gap issues. The analysis of linguistic tasks highlights differences between S2T and Text-to-Text (T2T) models. The proposed model improves S2T performance and linguistic evaluation benchmark on CoVoST-v2 datasets.
 
**Pros:**
 
- The paper introduces a novel linguistic probing benchmark, Speech-Senteval, to analyse the speech models (Reviewer 1, Reviewer 3).
 
- The analysis of linguistic tasks between S2T and T2T models contributes to understanding the behaviors of acoustic encoders (Reviewer 1, Reviewer 2).
 
- The PromptST model's introduction provides an innovative solution for mitigating the modality gap in S2T models (Reviewer 1, Reviewer 2, Reviewer 3, Reviewer 4).
 
- The experimental validation of the proposed approach on CoVoST-v2 datasets demonstrates consistent performance improvement over baselines (Reviewer 3, Reviewer 4).
 
- The paper's methodology of analyzing information processing within models and probing the balance between linguistic and acoustic information is insightful (Reviewer 4).
 
**Cons:**
 
- The literature review related to methods addressing the modality gap or fusion techniques is deemed rather brief by some reviewers, potentially affecting the paper's novelty and thoroughness (Reviewer 1, Reviewer 2).
 
- The benefits of using the proposed efficient approach on the encoder side, while fully fine-tuning the decoder, lack clarity and comparison against alternative strategies (Reviewer 2).
 
- The analysis primarily focuses on linguistic performance, potentially overlooking the broader implications of the proposed model's improvements across various fields (Reviewer 4).
 
- The absence of an evaluation on the MUST-C dataset limits the persuasiveness and generalization of the results (Reviewer 3).
 
- The mechanism by which prompt representations enhance linguistic information learning on the upper encoder layers needs further analysis and clarification (Reviewer 3).
 
Reviewer 1: aEu5,
Reviewer 2: gne1,
Reviewer 3: A846,
Reviewer 4: LoxQ
Decision: Accept-Main