ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation
Authors: Chenyang Le
Yao Qian
Long Zhou
Shujie LIU
Yanmin Qian
Michael Zeng
Xuedong Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: end-to-end speech to text translation
cross-modality learning
joint speech and language training

Abstract: Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pre-trained speech-only and language-only models and optimized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks, achieving a new state-of-the-art average BLEU score of 31.5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set.

ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Tex Text Translation
Chenyang Le"
Yao Qian
Shanghai Jiao Tong University
Microsoft Cloud and AI Redmond, WA, US
Shanghai, China
nethermanpro@ajtu.c edu edu.edu.co
iengiiaa@microooffe com
Long Zhou
Shujie Liu
Yanmin Qian
Microsoft Research Asia
Microsoft Research Asia Beijing, China
Shanghai Jiao Tong University Shanghai, China
Beijing, China
lozhou@microsoff. com
shujliu@miccoooffe com
yanminqian @jw..duu.ucunnuu
Michael Zeng
Xuedong Huang
Microsoft Cloud and AI Redmond, WA, US
Microsoft Cloud and AI Redmond, WA, US xdh@microv www.omfctrooo....mmmmmmmmo com
mzeng@microooot, com
Abstract
Joint ppeech-language training is challenging due to the large demand for training data and GPU consumption, well the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models and op- timized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross cross-modand mosssssy learning into transfer learning and conduct them simul- taneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end to-end speech-to seech-to-text translation tasks, achieving a new seate-of-the-aee average BLEU score of 3 31.5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set."
Introduction
recent years, an end-to-end (E2E) modeling approach has been widely applied to spoken language tasks, e.g., speech translation, speech summarization, speech understanding. etc. It enables us to train a single model by E2E optimization with spoken language task-oriented metrics. The conventional pipeline design generally contains two modules: speech model decodes the input speech into text and language model infers the recognized text to target language in translation task, topic sentence in summarization intent in understanding task. These two modules trained using their own respective criteria, which may not be optimal for each other. The cascaded process probably propagates errors that occurred in the current module to the following module. In addition, other information such as prosodic features contained in the speech signal, which are difficult to quantize using language symbols, can also be beneficial for spoken language tasks.
"The first author conducted this work during internship at Microsoft. 2The code is available at http:///hhhbbooooee/eeeeaapppppp
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Unified speech- languh- pretraining based on Transformer architecture has largely boosted E2E modeling for spoken language tasks [46, 31, 8. 7, 13. 11, 44]. The pretraining conducted jointly on unlabeled speech, unlabeled text, paired speech-to-text and paired text-to-text in multiple languages by using both self-supervised and supervised learning. The unified representations from both speech and text modalities are simultaneously learned via shared model parameters or auxiliary modality matching losses in the pretraining stage. However, on the other hand, pretrained speech-only and language-only models are also becoming increasingly powerful. To mitigate modality gap and achieve the comparable performance of the cascaded module system, unified speech-language pretraining must leverage the same or larger scale of data used in speech-only or language-only model pertaining. This makes the training process very challenging.
In this paper, we propose ComSL: a Composite Speech- Language Model for spoken language tasks. The main contributions of the paper can be summarized as follows:
1. Our composite model fully leverages existing pretrained models, eliminating the need for pre- training with large amounts of data from scratch. It can be directly fine-tuned for downstream tasks and demonstrates its data efficiency.
2. Our proposed mross-modality learning with speech-text mapping/matching on either represen- tations or distributions is only based on the concatenation of paired speech and text. Unlike conventional approaches that use contrastive learning among the modalities, it does not require external or internal aligners to force-align speech and text at the token or word level. This simplifies the implementation and allows it to be incorporated into the fine-tuning stage.
3. We have conducted a comprehensive ablation study on bridging the gap of speech and language representations, including tasks, losses, and strategies, as well as comparisons with previous works.
4. Our model outperforms the SOTA Google USM model [44]. Open AI Whisper model [26], and cascaded non-E2E models by 0.8, 1.8 and 1.0 average BLUE score improvements on the Co 2 evaluation set, respectively.
2 Related Work
Early work for E2E Speech Translation (ST) mainly focused on the practicability of E2E approach as proof of the concept and the study of the network structure of Transformer for ST [9, 39, 28, 14, 15, 4]. But these models are unable to compete with cascaded models that are separately trained on abundant Automatic Speech Recognition (ASR) and Machine Translation (MT) data [29]. The recent achievements to tackle data scarcity in E2E ST can be broadly categorized into three directions: 1) pretraining [6. 30, 40, 42, 38, 1, 21]; 2) bridging modality gap [11, 23, 37, 33, 16, 43, 24, 46, 31, 45, 7, 13. 11, 44]; and 3) data augmentation [18, 19, 5, 20).
Pretraining for E2E ST Pretraining allows a model to be initially trained on either unlabeled data or labeled data from other tasks and thus enables the model be more effectively fine-tuned using labeled data from downstream tasks. The exploration of using pretrained model for E2E ST started with using either pretrained ASR model or MT model [6, 30, 40, 42]. However, neither of them is sufficient for downstream E2E ST. [38] proposed a curriculum pretraining method with an ASR loss as an elementary course, and masked language model loss and a translation loss as two advanced courses. Most recently, speech-language model was jointly pretrained with set of tasks including speech recognition, text translation, speech translation, and etc [46, 31, 8, 7, 13, 11, 44].
Bridging modality gap for E2E ST Text representations capture contextual semantics, syntax, and part-of-speech information, while speech representations capture contextual phonemes, speaker iden- tity, prosody, and other acoustic characteristics. When jointly optimizing these two representations, there may be gap between them due their different focuses. Additionally, speech representations are typically longer than text representations because they are extracted at the frame level while text representations are extracted the token level. To align text and speech representations or learn a mapping between them, text embeddings are up-sampled according to the corresponding phoneme duration in the paired speech. During training, a modality matching loss (MML), such as 1.2 loss, is applied between the up-sampled text representations and speech representations [11]. The mismatched length of speech and text representations was also addressed by repeating the text token input based on CTC results [37] and down-sampling speech representations to match the length of
2

---

<En> will definitely do it.

Los
Leuz
Language Transformer
Language Transformer
Convid
Add Norm
Adapter
Feed Forward
Adapter
Embedding
Speech Transformer
Speech Transfomer
Mel Fealures
Transcriptin tokens
Mel Festures
(a) Training
(h) Inference
Overall architecture of our model ComSL, consisting of speech Transformer Blocks, adapter, and language Transformer blocks. (a) model training with ASR, ST, MT and cross-modality learning (CML) tasks. (b) model inference only for ST tasks.
Figure 1:
text representations [22]. Additionally, techniques such as mixing up speech and text features for input [16], leveraging hidden discrete units extracted from off-line opeech-to-unit and text-to-unit generators [45]. applying cross-attentive regularization to the representations from speech and text encoders [33], using contrastive learning at the word (WACO) and sentence (ConST) levels [24, 43], and employing knowledge distillation from MT to ST [23] have been shown to effectively bridge gap between these two modalities.
Data augmentation ST training data, i..., paired data of source language speech and target language text, is often limited. Data augmentation has been straightforwardly proposed to address this issue. Since the amount of paired speech recognition data is larger than that of ST. transcriptions of source language speech were converted into target language text using high-performance MT system for ST training [41]. Additionally, source language texts from parallel machine translation corpora were synthesized into speech [19]. With forced alignment information, new ST paired data was generated by replacing suffix speech segments and translating combined transcription with MT. Furthermore, the ST training data was augmented by spec-augmentation [5] or mixing at frame, word and sentence
levels [12].
Method
3.1 Problem formulation
End-to-end speech translation directly translates speech from a source language into text in target language, without generating intermediate ASR transcription. most likely label sequence y (y1, U2, ..., yN (e.g., words or characters) of length N in the target language given acoustic feature sequence s = (81, 82 82...., 8T (e.g., Mel Filter Bank features) of length T in the source language. An ST training corpus DST = {(s, x. x.)) usually contains x { 22 ..., ...,TM. TM.. ). the transcription of speech in source language, in addition to y and s. allows us to use MT and ASR as auxiliary tasks in multi-task learning manner during the optimization of E2E ST model, as
can be formulated find the
L Lsr + LMT + LASR
Multi-task learning (MTL) has been proven useful to share common knowledge among different tasks. However, it is non-rival in practice. ASR task can help improve the resolution of speech representations while the requirement of strict alignment between speech and text may negatively impact the performance of language pairs with big differences word order. MT task can serve as teacher to guide the ST task, text-to-text mapping is relatively easier to learn than speech-to-text mapping. However, the mismatch between speech and text modalities may make this guidance less effective.

---

3.2 Model Architecture
Followed by previous work [32, 22, 33), our model is a composite model, which is mainly composed of three components: speech Transformer blocks, an adapter and language Transformer blocks stacked together, shown Figure All of the components except the adapter are initialized by pretrained models. We also leverage multi-task learning for both speech and text inputs in the training stage.
Speech Transformer blocks are initialized with the encoder parameters of Whisper model [26]². Whisper follows standard sequence-to-sequene Transformer architecture [35] and has achieved strong performance on both speech-to-text recognition and translation tasks. It is trained on 680,000 hours of labeled speech recognition data as well as 125,000 hours of labeled X X-en translation data. Here we only take its encoder as powerful speech representation extractor.
Adapter placed between the speech Transformer blocks and the language Transformer blocks. contains two layers, in which each layer is composed of feed-forward module and I-D convolution layer with stride two. achieves four times down-sampling for speech encoder outputs in total. And the non- gon-linearity of the feed-forwar module can speed up the adaptation of speech representations to be fed into language Transformer blocks.
Language Transformer blocks are initialized with mBART model. mBART is composed of 12 layers of encoder and 12 layers of decoder with model dimension of 1024 on 16 heads. It is pretrained on monolingual data and fine-tuned on paired MT data and thus capable of many-to-many translation. All its parameters are used to enhance the translation capability of ComSL.
3.3 Cross-modality learning
In Equation we can feed speech-only input and text-only input into the model to learn the tasks of ST, ASR, and MT, respectively. Additionally, we introduce cross- mooss-modality learning (CML) based on paired speech-text input to minimize the gap between speech and text modalities. Different from the previous speech-text alignment approaches that rely on externally forced-alignmee determine word or other unit boundaries in speech sequences, our approach intrinsically learns cross- modality information during model optimization. As illustrated in Figure 2, the speech embedding, e", from adapter and text embeddings, e", from text tokenizer are concatenated as input to mBART encoder, Enc". After traversing the encoder, the concatenated form is split into 2 and 2² again to pass through mBART decoder, formulating
methods to
2t Enc² (e* (e*@
+ E²
(2)
where ex' is the masked text embedding and denotes concatenation. The following tasks are used for cross-modality learning.
Masked Token Prediction (MTP) The text tokens in the concatenated input are randomly masked by a probability Pmask- The MTP task is to predict the masked tokens in the output the decoder. In practice still let the decoder generate the whole sentence wacher-foreinge but only add loss on the masked tokens.
M
V
CMTP EE t=
1(x,
v&a, <mask>) * log P(x, = v|Iee, 20
D
where D the triplet corpus, M the transcription length, V is the dictionary, o the transcription, r' is the randomly masked transcription, is the speech embedding from the adapter, 1(.) is an indicator function. This task encourages the text encoder to fill in the missing token based on its speech corresponding through self-attention and helps to build strong relation between two modalities.
Speech to Text Mapping (STM) This task is similar to regular sequence-to -esuuuence mapping task (such as ASR or ST ) except that the encoder output hidden state is now conditioned on not only the input speech but also the masked ground transcription x'. The STM task has two losses to
"bapss//gituub.ccooopeaa//iiiiie We use revision efb383b experiment.

---

LieZ
00000
mBART Decoder
Spi
(n- Transformer blocks
Lanar
mBARTT Encoder
Hiddon stales the cxme ayeer
Transfarmer blocks
Spewcch
00.00 w00.0
Figure 2: Illustration cross- rross-modality learning. The speech and text embeddings are concatenated into the mBART encoder for ERM loss and split into the decoder for STM and MTP tasks.
predict source transcription and target translation, respectively. Formally,
M
IV
Csre
ΣΣ
1 (x+ = v) * log P(x; = =0/001 20;
(4)
D
t=1
Cigh = -E
1(y =) * log P(y = =;;
(5)
where N is the translation length. By using this way, can construct hidden representation that is of the same length as the speech tokens but contains rich information learned via the attention on the ground-truth transcription. This hidden representation, E.s should have better performance than the down-sampled representation, e", from the adapter on the top of the speech encoder. Our assumption that the sequence of down- samn-s representations has similar length to that of text, making more suitable as input for the text encoder. However, the underlying down-sampling operation may degrade its performance.
Encoder Representation Matching (ERM) To push the representation of speech-only input, =², closer that of concatenate input, add 1.2 loss between them. We do not directly add I.2 loss between 2, and E, since the variation among them might be too large to learn well. Instead, we add the loss on the hidden state of the k-th Transformer block the text encoder, oz and ag. is hyper-parameter and we
be in experiment.
CERM MSE (at ai)
(6)
Decoder Distribution Matching (DDM) The performance of MT is generally better than that of ST especially when the input of MT is the ground-truth truth transeription, i.e., the human-labeled transcription of speech in the source language. We involve DDM into the loss ST task as
LST = - [[E D 1=
(1-A))
u) + PP( ulyet, 2"; ==: ()]]] #log P(y o/ ())
(7)
where Xa is the weight for DDM and || means stop gradient. The modified ST loss allows ST task to learn not only from the ground-trus -ruth translation but also from the output distribution of MT task. The implementation DDM is similar to the online knowledge distillation proposed in [33].
3.4 Training Strategies
3.4.1 Fine-tuning Language Model
Before the model composition, we fine-tuned the language model with the paired text data the training data. The language model integrated into composite model is mBART-50 [34], which

---

is an extension of the original mBART model by adding 25 languages to support multilingual MT tasks in 50 languages. However, two languages, Welsh and Catalan, in our training corpus are still not covered by mBART-50.. As we observed, mBART has good extension capability so that it can achieve decent performance on these unseen languages by fine-tuning with limited data. We also tried to fine-tune the Whisper model before the model composition. Since Whisper has been trained with data in 100 languages, it has less impact on the final performance.
3.4.2 Multi-task Learning
We train ST, MT, ASR well as CML by multi-task learning. For ASR task, we use simple negative log likelihood loss:
M 1(x, u) log ((x))
CASH =
(8)
For CML, we average their losses from the decoder and add weight to the loss from the encoder:
LCME = 1/3 (CSTM + EisT LMTP) WERM LERM
(9)
Our final loss is the weighted sum these tasks.
Ltotal Wasr CASR + Wst LST + Wint * LMT + WCML * LCML
(10)
3.4.3 Regularization on the MT Output
The language Transformer blocks (i.e., mBART-50) have been fine-tuned with MT tasks on our corpus before the model composition. To prevent MT task from overfitting the multi-task learning for the composite model, we introduce an additional language model, i.e., fine-tuned mBART-50 model, 0', and freeze its parameters during training. We add cross-entropy loss to minimize the difference of their output distributions. This operation is similar to that in ST task, i... using better task as a teacher to guide a relatively worse task.
VI u) +)+ +) olyce, 2 #0) +log Ply ulyen, 2";0
CMT ΣE [1->)) D
(11)
where the output distribution of external fixed model and X₃ the weight for regularization.
3.4.4 Freezing Speech Encoder
The speech and language Transformer blocks are initialized by well-trained speech-onl) and text-only models, while the adapter is randomly initialized. These three components should use different learning rates during the training. To avoid it, we freeze the speech encoder at the first few epochs. This retains powerful speech representations during the early stage of fine-tuning when the gradients are unstable. The language component has the regularization on the MT output that plays a similar role.
Experiments
4.1
Datasets
E2E ST Data We conduct experiments on CoVoST 2 [36] dataset, a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages. It contains around 400 hours of English recordings and 900 hours of recordings from other 21 languages. Our work mainly focuses X-EN, the non-English to English direction.
Pseudo ST Data The training data size for some low-resource language pairs the CoVoST 2 dataset is limited. We add unlabeled translation data into the language directions that contain less than 30 hours of recordings into training in a self-training manner. Mozilla Common Voice [3] (version 11), large-scale multi-lingual ASR dataset that is of the same source VoST 2, is used to extract

---

data. We filter out all the data that is already in CoVoST 2 dataset to prevent repetition and test data leakage. Then we translate their transcriptions into English using the mBART model pre-finetuned on Co 2 dataset. By this way, we add about 300 hours of recordings across 16 languages with paired speech and pseudo translation in text.
4.2 Experimental Setups
Configuration We build two versions of model, named ComSL Medium and ComSL Large. The difference among them that the speech Transformer blocks are initialized with the encoder from Whisper in different sizes. The ComSL. Medium uses Whisper medium encoder that contains 24 layers of Transformer blocks with 1024 hidden dimensions and 16 heads. The ComSI. Large uses a Whisper large encoder that contains 32 layers of Transformer blocks with 1280 hidden dimensions and 20 heads. The two versions all have a two-layer convolution adapter and an mBART model that is initialized by a Abart50-large-man -oatttttt-mm-mmmmm checkpoint." The total parameter size is about 0.9 billion for the medium version and 1.3 billion for the large version.
Training All the recordings are converted to 16 bit 16kHz mono-channel waveform for Mel Filter Bank features and then used inputs to the model. During training, due the memory issue, the input length of audio recording is limited to 11 seconds, which is long enough for cover 99 percent of our training data. Two techniques, deepspeed ZeRo [27] and activation checkpointing [10], are employed to optimize the usage of GPU memory. We save the checkpoint that has highest BLEU score on the validation set. It takes about 3 days to train on 4 Nvidia Tesla V100 GPUs with 32G of memory in each.
Inference and Evaluation During inference, we run beam search with beam size 5, For evaluation, we report detokenized corpus level BLEU scores on CoVoST 2 test set using sacreBLEU [25].
4.3 Main Results
Our main results are shown in Table We follow [7, 13] in dividing the 21 languages of the test set into three groups: High-resource (High), Medium-resouree report the average BLEU score for each group as well as the average for all 21 languages (Avg). ComSL Medium and Large outperform Whisper Medium and Large, achieving BLEU scores of 29.7 and 30.1, respectively. The main factors contributing to this improvement are the replacement of the Whisper decoder with mBART and the application of our training strategies and losses in training the combined Whisper and mBART models. We observe that the performance of ComSI. Medium on par with that of Whisper Large, which has significantly more parameters. It is noteworthy that the BLEU scores for Whisper Medium and Large surpass those reported in the original paper [26), which shows zero-shot performance on the CoVoST testing set. The scores for Whisper models, listed in Table 1, result from fine-tuning Whisper with VoST 2 training data. Our BLEU score for the fine-tuned Whisper is higher than that shown in [44], which may be due to differences in the versions used for the experiments. When pseudo ST data is employed, both ComSL Medium and ComSL Large outperform the current SOTA, i.e., USM [44]. Our ComSL Large reaches a BLEU score of 31.5, which is 0.8 higher than USM with 0.7B fewer parameters. According to the data description in [44], the Common Voice corpus, where we extra data and generate pseudo translations, has also been included in USM training through self-supervised learning. Furthermore, our ComSL Medium and Large single models outperform Cascaded models that first conduct ASR with Whisper and then run MT with mBART-50 using ASR hypotheses as inputs. However, there is still a performance degradation due to ASR errors when compared to using ground- truth transcription as inputs for MT.
(Med), and Low-resource (Low). We
5 Analysis
5.1
Ablation study training tasks/losses and strategies
We conduct an ablation study on training tasks/losses and strategies using ComSL Medium. The results, shown in Table 2, demonstrate how our strategies and tasks steadily improve overall perfor-
'nags://nuggingfae..oofff http://gggggggte.ooooooooooooaaa www.accoooo/mbaartttaaooooooooe. large-5--maane many-to many-to-mann to-my-t------nnnne manmanmtm
'The breakdown of BLEU scores in the different contigurations on each of the 21 language pairs is listed in the Appendix.
7

---

Model
PARAMS
High
Med
Low
Avg
mSLAM [7]
2B
37.8
29.6
18.5
24.8
Maestro [11]
0.6
38.2
31.3
18.4
25.2
Mu2SLAM [13]
0.6
37.0
30.6
23.5
27.1
Whisper Medium (Baseline Medium)
0.8
36.4
32.8
24.3
28.6
Whisper Large (Baseline Large)
1.6
37.0
33.8
25.6
29.7
USM [44]
2B
30.7
Our model trained without pseudo ST data
ComSL Medium
0.9 B
37.41
34.1
25.3f
29.7
ComSL Large
1.3 B
37.74
34.5
25.8
30.1
Our model trained with pseudo ST data
ComSL. Medium
0.9 B
37.3f
35.6
26.61
30.8
ComSL Large
1.3 B
37.6$
35.9
27.53
31.5
Non-E2EST
Ground-truth Transcription mBART-50
0.6 B
42.1
39.9
35.8
38.0
Whisper Medium + mBART-50
1.3 B
37.3
34.5
26.4
30.4
Whisper Large + mBART-50
2.2
37.4
34.6
26.4
30.5
Table 1: Speech translation performance in terms of the average BLEU scores non-English to English (xx-en) directions of the CoVoST 2 testing set. All models including Whisper and mBART-50 are fine-tuned with CoVoST training set. The symbols "&" and "4" are used to indicate that the model performance is significantly better (p<0.01) than the Baseline Medium and Large, respectively.
Description
ST BLEU
ST Delta
MT BLEU
w/ ST task only
26.48
29.13
+ MT task
26.31
-0.17
36.43
+ DDM
27.87
1.56
36.21
+ ASR task
28.29
0.42
35.92
+ Freeze encoder
28.64
0.35
35.28
+ MT regularization
29.40
0.76
37.40
CML Loss
29.69
0.29
37.30
+ Pseudo data
30.77
1.08
37.46
Table 2: Ablation study the training strategies and tasks/losses. We start training the model with ST loss only and then adding a strategy or task/loss over the previous experiment. We present the BLEU score of ST task, well the performance gain of ST task brought by each change.
mance. The model trained with only ST loss achieves a BLEU score of 26.48. Adding the MT task results in a slight drop to 26.31, indicating that directly applying the MT task has little impact on performance. However, when DDM is leveraged, the BLEU score increases significantly to 27.87. The ASR task and corresponding training strategy of freezing speech encoder parameters in the early stage of training improve the BLEU score by 0.42 and 0.35, respectively. MT regularization, which prevents overfitting in the MT task, enhances performance by an additional 0.76 BLEU score. CML, which matches speech and text representations, boosts performance to BLEU score of 29.69. Finally, using pseudo ST data further increases the BLEU score to 30.77. To further investigate the performance of the integrated language model in ComSL, we feed ground-truth speech transcription to it, even though ComSL only requires speech as input for the ST task. The corresponding BLEU score of the MT task shown in Table 2 indicates that our training strategies and tasks continue to reduce the degradation of MT performance from mBART-50, finally achieving BLEU score of 37.46 compared to 38.0 for Ground-trutt Transcription + mBART-50 shown in Table 1.
5.2 Comparison among different cross-modality learning (CML) methods
We compare our mrods-my learning methods with previous works. The model trained with all strategies and tasks except the methods of minimizing modality gap refers to the baseline. The performance of the baseline and those with our CML method and previous MML, ConST, and WACO methods are listed in Table 3. Our CML. reaches 29.69 BLEU score, the best one among all listed methods the table. MML also performs well, achieving a BLEU score of 29.6, slightly lower than ours, but it requires internal alignment at the token level. Calculation of alignment by

---

Method
Forced-A Aliinmmet
Avg BLEU
Baseline
N
29.40
MML [11]
Token level
29.60
ConST [43]
N
29.18
WACO [24]
Word level
29.37
Our CML
N
29.69
Table 3: Comparison between different methods for minimizing modality gap on ComSL Medium. The second column shows what kind of forced-alignment is needed for the method. N denotes 'Not Needed'.
whatever internal or external token-level forced aligner, adds burden to the training process. In our implementation, contrastive-oot based methods (ConST and WACO) fail surpass the baseline model. We conjecture that it may be due to the nature of contrastive loss, which pulls negative and positive samples apart, which may cause an undesirable gradient direction for a supervised pretrained model like Whipser.
We also take an in-depth look the hidden level to see what happens after applying cross-modality learning tasks. Figure shows the similarity matrices of speech and text representations. These representations are taken from the output of the 4th layer in the mBART encoder. A higher number, indicated by darker color in the figure, means that the representations are more similar to each other. Here are our findings, 1) Our model can match text and speech representations fairly well even without explicitly applying any modality gap minimizing methods, as shown in Figure 3-a). We believe this most likely due to the use of knowledge from ASR task, which leaves less room for the CML method to further improve performance. 2) The model trained with our CML unified representation not only for speech and text but also for non-speech tokens like silence. As illustrated by the green box in the figure, non-speech tokens like punctuation (period) and silence are aligned much better than those without CML. This is a unique phenomenon observed in our approach. (We tried plotting matrices using other methods listed in Table 3, but all looked like the left figure.) 3) Our CML can help encode speech information more precisely. For example, in the blue box area, there is compound two-syllable word with very short pause in between. Our model better identifies the short pause and sharpens the distributions of surrounding tokens.
produce
0005
R
032
3008
209-
F11-
(a) w/o Cross- mooss-od learning
(b) w/ Cross-modality learning
Figure 3: Similarity matrices of speech hidden states and text hidden states(softmar alone the text axis). These hidden states are taken from the output of the 4-th text encoder layer.
Conclusions, Limitations and Future Work
In this paper, we presented a Composite Speech and Language (ComSL) model to perform E2E speech-to-text translation tasks. We bridged the gap between speech and text representations through cross-modaad modalltty learning tasks, in addition other auxiliary tasks like speech recognition and machine translation. Our composite model outperformed the constituent speech model (Whisper) or cascaded speech and language models WWhisper+BBAR our model achieved new SOTA performance on the public CoVoSTY evaluation set. However, our approaches have not been exploited on decoder-only based Large Language Models (LLLMs) this study due to their recent public availability and the requirement of large GPU resources. Additionally,
pipeline manner. By adding pseudo ST data,

---

our model has only been evaluated on speech-to-tex translation tasks due to the limitations of currently available E2E datasets. In the future, we plan combine our approaches with Low-Rank Adaptation (LoRA) [17] or other light-weight adapters and use efficient data augmentation techniques to fine-tune a large composite model for various downstream spoken language tasks.
References
[1] Ashkan Alinejad and Anoop Sarkar. Effectively pretraining a speech translation decoder with machine translation data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 8014-8020, 2020.
[2] Junyi Ao, Rui Wang. Long Zhou Chengyi Wang, Shuo Ren. Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. arxiv preprint arXiv:2110.07200 2021.
[3] Rosana Ardila, Megan Branson, Kelly Davis, Michael Hemretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A speech corpus. arXiv preprint 1tXiv.1912.06670, 2019.
[4] Parnia Bahar, Tobias Bieschke, and Hermann Ney. A comparative study on end-to-end speech to text translation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 792-799 799. IEEE, 2019.
[5] Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. On using specaugment for end-to-end speech translation. arXiv preprint arXiv: 1.08876. 2019.
[6] Sameer Bansal, Herman Kamper, Karen Liveseu, Adam Lopez, and Sharon Goldwater. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. arXiv preprint 2018.
[7] Ankur Bapna, Colin Cherry, Yu Zhang, Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiw:2202.01374 2022.
[8] Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati. Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. Slam: A unified encoder for speech and language modeling via speech-text joint pre-training. arXiv preprint arXiv:2110. 10329, 2021.
[9] Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. Listen and translate: A proof of concept for end-to-end speech-to- text translation. arXiv preprint a1Xiv:1612.01744, 2016.
[10] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016.
[11] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. Maestro: Matched speech text representations through modality matching. arXiv preprint arXiv:2204.03409 2022.
[12] Xuxin Cheng. Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. M3st: Mix three levels for speech translation. arXiv preprint arXiv:2212.03657, 2022.
[13] Yong Cheng, Zhang, Melvin Johnson, Wolfgang Macherey, and Ankur Bapna, Mu? Multitask. multilingual speech and language models. arXiv preprint arXiv:2212.09553. 2022.
[14] Mattia A Gangi, Matteo Negri, and Marco Turchi. Adapting transformer to end-to-end spoken language translation. In Proceedings of INTERSPEECH 2019. pages 1133-1137. International Speech Communication Association (ISCA), 2019.
[15] Mattia Antonino Di Gangi, Matteo Negri, Roldano Cattoni, Roberto Dessi, and Marco Turchi. Enhancing transformer for and-to-end speech-to-ter translation. In Proceedings of Machine Translation Summit XVII: Research Track, pages 21 21-31, 2019.
[16] Qingkai Fang, Rong Ye. Lei Li, Yang Feng, and Mingxuan Wang. Stemm: Self-learning with speech-text manifold mixup for speech translation. arXiv preprint arXiv:2203.1044 2022.
[17] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09666 2021.
10

---

[18] Hirofumi Inaguma, Taisuya Kawahara, and Shinji Watanabe. Source and target bidirectional knowledge distillation for end-to-end speech translation. arXiv preprint arXivv2104.06457, 2021.
[19] Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. Leveraging weakly supervised data to improve end-to-end speech-to- text translation. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7180-7184, IEEE, 2019.
[20] Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to-end speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 245-254. Dublin, Ireland, May 2022. Association for Computational Linguistics.
[21] Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. Lightweight adapter tuning for multilingual speech translation. arXiv preprint arXiv:2106.01663, 2021.
[22] Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Bacvski, Alexis Conneau, and Michacl Auli. Multilingual speech translation with efficient finctuming of pretrained models. arXiv preprint arXiv:2010. 12829, 2020.
[23] Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang. Hua Wu, Haifeng Wang, and Chengqing Zong. End-to-end speech translation with knowledge distillation. arXiv preprint arXiv: 19Xiv:1904.009 1904.08775 2019.
[24] Siqi Ouyang, Rong Ye, and Lei Li. Waco: Word-aligned contrastive learning for speech translation. arXiv preprint 12iiv2222..0933 2022.
[25] Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv: 1.XXvv1804.08777 804.08771 2018.
[26] Alce Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint 1rX.v:2212.04356, 2022.
[27] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020.
[28] Elizabeth Salesky, Matthias Sperber, and Alex Waibel. Fluent translations from disfluent speech in end-to-end speech translation. arXiv preprint arXiv:1906.0005: 2019.
[29] Matthias Sperber and Matthias Paulik. Speech translation and the end-to-end promise: Taking stock of where we are. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul 2020.
[30] Mihacla C Stoian, Sameer Bansal, and Sharon Goldwater. Analyzing asr pretraining for low-resource speech-to-texy toxtext translation. In ICASSP 2020-2020 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pages 7909-7913, IEEE, 2020.
[31] Yun Tang. Hongyu Gong, Ning Dong. Changhan Wang. Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, et al. Unified speech-text pre-training for speech translation and recognition. arXiv preprint wxXv:2204.00500% 2022.
[32] Yun Tang, Hongyu Gong, Xian Li, Changhan Wang, Juan Pino, Holger Schwenk, and Naman Goyal. Fst: the fair speech translation system for the iwsl121 multilingual shared task. arXiv preprint arXiv:2 107.09999. 2021.
[33] Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. Improving speech translation by understanding and learning from the auxiliary text translation task. arXiv preprint arXiv:2107 007822. 2021.
[34] Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation with extensible multilingual pretraining and finetuning. 2020.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[36] Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual specch-to-text translation. arXiv preprint 2000:220007.111000 2020.
[37] Chengyi Wang. Yu Wu. Shujie Liu, Zhenglu Yang. and Ming Zhou. Bridging the gap between pre- pre-training and fine-tuning for end-to-end speech translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9161-99 (61-9166,,,8 2020.

---

[38] Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. Curriculum pre-training for end-to-end speech translation. arXiv preprint 10099,, 2020.
[39] Ron I Weiss, Jan Chorowski, Naydeep Jaitly, Yonghui Wu, and Zhifeng Chen, Sequeenceto--sequene models can directly translate foreign speech. arXiv preprint arXiv: 170308888,,, 2017.
[40] Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Qi Ju, Tong Xiao, Jingbo Zhu, et al. Stacked acoustic- and-textual encoding: Integrating the pre-trained models into speech translation encoders. arXiv preprint 0Xiv:2105.05552 2021.
[41] Jian Xue, Peidong Wang, Jinyu Li, and Eric Sun. A weakly-supervised streaming multilingual speech model with truly zero-shot capability. arXiv preprint arXiv:2211.02499 2022.
[42] Rong Ye, Mingxuan Wang, and Lei Li. End-to-end speech translation via cross-modal progressive training. arXiv preprint arXiv:2104. 10380, 2021.
[43] Rong Ye. Mingxuan Wang. and Lei Li. Cross-modal contrastive learning for speech translation. arXiv preprint arXiv:2205.0244 2022.
[44] Yu Zhang. Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang. et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.0103
2023.
[45] Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and speech and text with hidden-unit for encoder-decoder based speech-text pre-training. arXiv preprint #xiv:2210.037330 2022.
Speechut: Bridging
[46] Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. In International Conference on Machine Learning, pages PMLR, 2021.
12


Reviews:
[{"review_id": "aMHjTC0W6R", "reviewer": "Reviewer_tyJC", "summary": "With the goal to improve speech translation task, this work, leverages a multi-task training approach optimizing weighted sum of ASR, ST, MT and cross-modality learning (CML) objectives. Using mBART encoder and decoder Transformer blocks, the CML objective is purposed to better align speech and text modalities. CML involves a masked token prediction (MTP), speech to text mapping (STM), and encoder representation matching (ERM) objectives. \n\nOverall, the proposed arc (ComSL) includes a speech and textual transformer blocks, and a two layered bride adapter module. Before the multi-task step, pre-trained models (mBART and Whisper) are fine-tuned using training data for 50 languages, mainly involving CoVoST2 data set. Results are provided for low, medium and high-resource languages show ComSL's performance improvements over recently proposed multi-modal (text and speech) models such as USM and Whisper.", "questions": "- What was the motivation for not evaluating other tasks, as this model is trained in a multi-task setting, at least the ASR evaluation makes sense.\n- Given the number of objectives involved, particularly for the CML part, how does model training and inference complexity compares? \n- Table 1, all models ... are fine-tuned with CoVoST2 data, does this include USM model? if not, what is the justification for a fair comparison?\n- Table 1, for Non-E2EST, does it make more sense to use Whisper + and a standard MT model trained separately? \n- Given most numbers are quite close, in table 1-3, adding statistical significance could make the results more meaningful, and to the support SOTA assertion.\n- L103 fix unfinished sentence\n- S3.3 how does the split is formulated for z^s and z^x before passing it to mBART decoder? Obviously the split is necessary to optimize the MTP objective using z^x, have you considered directly using e^x without the concat step?", "limitations": "Limitations are included, reflecting the main content of the work. I suggest authors to consider a separate limitation part and to consider societal impact if available.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work, is well motivated suggesting an cross-modality learning objectives, to address the modality gap in the next frontier for a working composite speech and textual model.\n- The results for ST are encouraging, performing similarly or slightly better than recently proposed multi-task based speech-text models like Whisper and USM.", "weaknesses": "- Although, ComSL is well motivated to focus on bridging the modality gap, the pre-training and fine-tuning using in-domain data doesn't seem to deliver significant improvements. With close results, and lack of significance testing, its rather difficult to interpret if the proposed approach is better performing. \n- Given the focus on multi-task learning, the provided evaluation (only for ST task) is unexpected. Adding at least ASR task in to the comparison could have provided more substance to the report.\n- Discussion and analysis part could have focused on ST cases (or examples) where CML improves over other models without using the cross-modality objectives. This would show the importance of introducing the objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. We have provided our responses right after each question.\n\n1.\tWhat was the motivation for not evaluating other tasks, as this model is trained in a multi-task setting, at least the ASR evaluation makes sense.\n\nAnswers: \nDue to space limitations in the full paper, we have reported the ASR performance in the appendix of the supplementary materials. If the lack of evaluation on the ASR task affected your score, we hope you will consider adjusting it. \nTable 6 in the appendix shows that ComSL outperforms finetuned Whisper on the ASR task for some languages, but its overall performance is slightly worse. This phenomenon has also been observed in other studies, and as a result, pretrained models are often finetuned for the ASR task only (instead of multi-task such as both ASR and ST tasks) to achieve better performance, as was done on USM (Reference [44]). \n\n2.\tGiven the number of objectives involved, particularly for the CML part, how does model training and inference complexity compares?\n\nAnswers:\nDuring training, objectives other than speech-to-text translation loss are included. However, these additional objectives are not necessary during the inference stage, so the complexity remains the same as a conventional speech-to-text translation model. Our observations show that training time increases by a factor of 1.2, from 2.5 hours per epoch to 3 hours per epoch using 32 V100 GPUs, when compared to a model trained without CML losses.\n\n3.\tTable 1, all models ... are fine-tuned with CoVoST2 data, does this include USM model? if not, what is the justification for a fair comparison?\n\nAnswers:\n Yes, the USM model has been finetuned. USM is not a public model, but we have cited the BLEU scores from its paper (Reference [44]). Section 3.3.2 of this paper indicates that the USM model evaluated on CoVoST2 was finetuned using the CoVoST2 training set and text translation data as available. Additionally, USM for ASR and ST tasks were finetuned separately, so the results of the ASR and ST tasks shown in the USM paper were obtained from different finetuned USM models.\n\n4.\tTable 1, for Non-E2EST, does it make more sense to use Whisper + and a standard MT model trained separately? \n\nAnswers: \nWe are not entirely sure if we understand your question correctly. The mBART-50 model (please check footnote 3 on page 7) that we used is already a MT model. We finetuned this model on CoVoST2 text data. For the first row under Non-E2EST in Table 1, we fed the ground truth transcription to the mBART model to measure its performance, simulating a cascade system with a perfect ASR model output. For the second and third rows, we used Whisper as the ASR model for fair comparison. To the best of our knowledge, mBART-50 is the best pretrained translation model on CoVoST text data, even better than most LLMs (most LLMs are not finetuned for translation).\n\n5.\tGiven most numbers are quite close, in table 1-3, adding statistical significance could make the results more meaningful, and to the support SOTA assertion. \n\nAnswers:\nThank you for your suggestions. We have done significance tests for the results in table 1-3.\n\nFor Table 1, please refer to the attached PDF file in the “global” response to all reviewers. Our tests show that, regardless of whether pseudo ST data is added, our ComSL Medium/Large models significantly outperform their corresponding baseline Medium/Large models (i.e., finetuned Whisper) on high-resource and mid-resource languages, with a significance level of p<0.01. For low-resource languages, where the training data for most languages is around or less than two hours, the addition of pseudo ST data has a significant impact on improving ST performance. We suspected that limited training data for a specific language might be causing issues, such as limited vocabulary coverage, which could affect the efficiency of our proposed methods. This is why we added pseudo ST data only to mid-resource and low-resource languages. Please refer to the appendix for statistics on the pseudo data.\n\nTable 2 shows that adding only the MT task can negatively impact performance. So we conducted significance tests for all tasks/losses except for the MT task. The test results show that the CML loss can significantly improve ST performance on high-resource and mid-resource languages, with a significance level of p<0.05, but not on low-resource languages. All other tasks/losses significantly improve performance on all languages.\n\nFor Table 3, our significance tests show that minimizing the modality gap has a significant impact on performance for high-resource and mid-resource languages. However, the differences between various methods of modality gap minimization are not significant. In our multi-task learning approach, we conducted several tasks, including AST, ASR, MT, CML, and others. The shared model parameters are updated using both speech and text tasks, which reduces the modality gap between speech and text to a certain extent. As a result, the explicit modality gap minimization loss may not be as significant as expected, especially after the DDM (AST and MT distribution matching) loss was employed as a complementary method for reducing the modality gap.\n\n6.\tL103 fix unfinished sentence\n\nAnswer: \nWe will fix it.\n\n7.\tS3.3 how does the split is formulated for z^s and z^x before passing it to mBART decoder? Obviously the split is necessary to optimize the MTP objective using z^x, have you considered directly using e^x without the concat step?\n\nAnswers: \nYes, we have considered this. The mBART used in our ComSL has been finetuned using text from the CoVoST 2 training set, so the improvement from adding a text task in ComSL training is marginal. For the MTP task, the goal is to match speech and text representations through self-attention. If we were to directly use e^x, it would become a pure text denoising task and would reduce the contribution to minimize modality gap.\n"}, {"review_id": "1PlWJCZmJi", "reviewer": "Reviewer_KhJu", "summary": "This work presents a speech-language model built from both speech-only and language-only pretrained models. By compositing pre-trained models from 2 modalities, the authors show that a data-efficiency for spoken language tasks can be achieved. In particular, the authors proposed a few cross-modality loss functions which are designed to  build a strong relationship between 2 modalities. The authors demonstrated that their method is able to achieve a new sota BLEU on the CoVoST2 evaluation task. ", "questions": "- After the finetuning on CoVoST2, I think the model can still perform ASR ? But it would be great that the authors to clearly state that (or explain why it cannot). ", "limitations": "I think the authors have addressed limitations adequately. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well written. It explains the motivations and methods clearly. Besides great overall results, it also presents a detailed analysis including ablation study and in-detail examination of usage of the cross-modality. ", "weaknesses": "- It is a bit disappointing to see from the result the cross modality tasks, which is the main contribution in this work, does not give significant difference in terms of BLEU score (Table 2, 29.40 --> 29.69 due to CML loss). I believe the follow up improvement from pseudo data can be obtained without CML loss. \n\n- It would be better if the authors can clearly state how many parameters are in the pre-trained ASR encoder, the LM (mBART) and in the adapters; and which parts are finetuned or fixed during training. \n\n- Though the model is trained with both ASR and AST loss, the authors only evaluate  the model on the AST / MT tasks. It would be great if the author can provide a benchmark on ASR as well. \n\n- It is not immediately clear to me that whether the comparison of this work to USM / whisper is fair. It seems to be both whisper and USM does not finetune on CoVoST2 tasks, while they both use CoVoST2 tasks as an out-of-domain evaluation. On the other hand, whisper is able to perform both ASR and AST using the same model with just different prompt, while the authors did not reveal their ASR performance. \n\nThere are also a recent paper relevant to this work: \n\nRubenstein, Paul K., et al. \"AudioPaLM: A Large Language Model That Can Speak and Listen.\" arXiv preprint arXiv:2306.12925 (2023). \n\nUnderstandably, this paper appears after NeuralPS submission deadline.  The authors may consider to cite it since it achieves a new SOTA on CoVoST2. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. We have provided our responses right after each question.\n\n1.\tAfter the finetuning on CoVoST2, I think the model can still perform ASR ? But it would be great that the authors to clearly state that (or explain why it cannot).\n\nAnswers: You are correct. The ComSL can perform ASR after the finetuning on CoVoST2. Due to space limitations in the full paper, we reported the ASR performance in the appendix enclosed in the supplementary materials, along with the ST and MT tasks for each of the 21 languages in the CoVoST 2 evaluation set. We also included statistics on pseudo data and experimental details in the appendix. If the lack of an evaluation of the ASR task affected your score on our paper, we hope you will consider adjusting it. \n\nTable 6 in the appendix shows that ComSL outperforms finetuned Whisper on the ASR task for some languages, but its overall performance is slightly worse. This phenomenon has also been observed in other studies, and as a result, pretrained models are often finetuned for the ASR task only (instead of multi-task such as both ASR and AST tasks) to achieve better performance, as was done on USM (Reference [44]).\n\n2.\tIt is a bit disappointing to see from the result the cross modality tasks, which is the main contribution in this work, does not give significant difference in terms of BLEU score (Table 2, 29.40 --> 29.69 due to CML loss). I believe the follow up improvement from pseudo data can be obtained without CML loss.\n\nAnswers: \nWe have done a significance test on comparing with/without CML, which shows CML has a significant impact on performance for high-resource and mid-resource languages with a significance level of p<0.05, but not on low-resource languages. During training, multiple tasks, including AST, ASR, MT, and CML, are conducted. The shared model parameters are updated not only using the speech task but also the text task. This reduces the modality gap between speech and text to a certain extent, and as a result, the CML loss might not be as significant as expected, especially after the DDM (AST and MT distribution matching) loss was employed as complementary method for reducing the modality gap. In addition, we suspected that limited training data of low-resource languages might be causing issues, such as limited vocabulary coverage, which could affect the efficiency of our proposed methods.\nOur experimental results show that the CML loss can significantly improve performance on high-resource languages, while adding more pseudo-ST data after applying the CML loss does not provide additional benefits. However, we have not yet tested the alternative approach of adding pseudo-ST data first and then applying the CML loss on low-resource languages. We will consider conducting this comparison in future research.\n\n3.\tIt would be better if the authors can clearly state how many parameters are in the pre-trained ASR encoder, the LM (mBART) and in the adapters; and which parts are finetuned or fixed during training.\n\nAnswers: The parameter sizes for different components are as follows: 1)Whisper medium encoder: 0.3B; 2) Whisper large encoder: 0.7B; 3)mBART (encoder+decoder): 0.6B; 4) Adapter: 30M. During the first third of the training steps, we only fixed the parameters of pre-trained ASR encoder (i.e., Whisper encoder). Other than that, all parameters were fine-tuned.\n\n4.\tThough the model is trained with both ASR and AST loss, the authors only evaluate the model on the AST / MT tasks. It would be great if the author can provide a benchmark on ASR as well.\n\nAnswer is the same as that of first question.\n\n5.\tIt is not immediately clear to me that whether the comparison of this work to USM / whisper is fair. It seems to be both whisper and USM does not finetune on CoVoST2 tasks, while they both use CoVoST2 tasks as an out-of-domain evaluation. On the other hand, whisper is able to perform both ASR and AST using the same model with just different prompt, while the authors did not reveal their ASR performance.\n\nAnswers:  Actually, the BLEU scores for Whisper and USM reported in Table 1 were achieved by finetuning the models on the CoVoST 2 training set. We finetuned Whisper models and achieved BLEU scores of 28.6 and 29.7 for Whisper medium and Whisper large, respectively, which are higher than those reported in the original Whisper paper (Reference [26]). As for the USM model, since it is not a public model, we cited the BLEU scores from its paper (Reference [44]). Section 3.3.2 of this paper indicates that the USM model evaluated on CoVoST2 was finetuned using the CoVoST2 training set and text translation data such as WMT or TED talks as available. Therefore, our comparison is fair in the context of in-domain finetuning/evaluation. Additionally, we reported the ASR performance in the appendix enclosed in the supplementary materials.\n\n6.\tThere are also a recent paper relevant to this work: Rubenstein, Paul K., et al. \"AudioPaLM: A Large Language Model That Can Speak and Listen.\" arXiv preprint arXiv:2306.12925 (2023). Understandably, this paper appears after NeuralPS submission deadline. The authors may consider to cite it since it achieves a new SOTA on CoVoST2.\n\nAnswers: Thank you for bringing this to our attention. We will be sure to cite this most recent and relevant work on leveraging LLMs to further improve performance in our paper.\n"}, {"review_id": "zzvdoHaOG5", "reviewer": "Reviewer_7Di5", "summary": "This paper proposes a composite speech-language model for speech-to-text (ComSL) translation. ComSL first leverages existing pre-trained models for initialization, including Whisper speech recognition model and mBART machine translation model. And then proposed several modality alignment methods that do not require additional tools. Extensive experiments on CoVoST2 demonstrate the superiority of the proposed method, which also achieves a new SOTA performance.", "questions": "See weaknesses", "limitations": "Not applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides insight into designing large-scale spoken language models.\n2. This paper proposes several methods based on concatenated embeddings to align the representation of speech and text.\n3. The proposed method outperforms previous work and achieves a new SOTA performance.", "weaknesses": "1. The training process may be unstable. As shown in the appendix, multiple losses are distributed by various weights from 0.1 to 0.8. The authors do not explain how these hyperparameters are determined, and how much the impact is. A solid system is expected.\n2. Although this paper achieves a new SOTA performance, it mainly relies on the combination of several common techniques. Therefore, the contribution is limited. To achieve SOAT performance, ComSL is trained with pseudo ST data, resulting in unfair comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. We have provided our responses right after each question.\n\n1.\tThe training process may be unstable. As shown in the appendix, multiple losses are distributed by various weights from 0.1 to 0.8. The authors do not explain how these hyperparameters are determined, and how much the impact is. A solid system is expected.\n\nAnswers:  Thank you for bringing this to our attention. We will make sure to clarify it in the revised version of our paper. Unlike training from scratch, our ComSL model weights were initialized with pre-trained speech and language models, which greatly stabilized the training process. Although we employed various weights for multiple losses, we set most weights empirically or referred to values used in other studies. Our consideration for relatively high or low weights depended on the contributions of the losses. For example, we set a high weight for DDM since the performance gap between AST and MT was large, and relatively smaller weights for other tasks like MT since MBart had already been fine-tuned before being combined with Whisper. We also tried tuning the weights within a small range, such as DDM from 0.8 to 1, and observed that its impact on performance was marginal. We did not perform a “brute force” hyperparameter tuning as it was unaffordable.\n\n2.\tAlthough this paper achieves a new SOTA performance, it mainly relies on the combination of several common techniques. Therefore, the contribution is limited. To achieve SOAT performance, ComSL is trained with pseudo ST data, resulting in unfair comparisons.\n\nAnswers:  \n\na)\tOur contributions are outlined in the final paragraph of the introduction section. With the emergence of pre-trained speech and language models, we proposed an approach that combines two publicly available models and fine-tunes them directly for downstream tasks, along with cross-modality learning, which is typically used during the pre-training stage. We hope that our approach will offer a new avenue for research in academia, where large model pre-training is not feasible due to limited computational resources.\n\nb)\tThe improvement of ComSL comes from both the new approach/losses and the use of pseudo ST data. Please refer to the attached PDF file in the “global” response to all reviewers jointly for the results of significance tests, which show that, regardless of whether pseudo ST data is added, our ComSL Medium/Large models significantly outperform their corresponding baseline Medium/Large models (i.e., finetuned Whisper) on high-resource and mid-resource languages, with a significance level of p<0.01. Nowadays, it is very challenging to conduct apple-to-apple comparisons due to differences in the amount of training data and model architecture/size. Most leaderboards/benchmarks do not require the listing of training data. Additionally, the Pseudo ST data we used was extracted from the Common Voice corpus, which was also included in USM pre-training in the form of self-supervised learning, according to the data description in the USM paper.\n"}, {"review_id": "zQDlxRCqzb", "reviewer": "Reviewer_nRjc", "summary": "Authors propose a speech-text pretrained model for spoken language tasks which leverages already existing pre-trained speech and language models. Modality mapping/alignment is based on a concatenation of paired speech-text (no need of word level alignment). The model is evaluated on a speech-to-text translation (S2TT) task (on CoVoST2 dataset) and slightly outperform previous models such as Whisper for S2TT.\n\n\n\n\n", "questions": "-In what aspect does your method really differ from SpeechT5, SLAM, etc ?\n\n-\"Speech Transformer blocks are initialized with the encoder parameters of Whisper model\" why using Whisper speech encoder only and not XLS-R (multilingual wav2vec) or HuBERT ?\n\n-\"Different from the previous speech-text alignment approaches that rely on externally forced-alignment methods to determine word or other unit boundaries in speech sequences,\" => Do SpeechT5 & SLAM need external word-level forced-alignment (i don't think so)\n\n-(tab 1) does the improvement really comes from the new approach/losses proposed or does it come only from the use of pseudo ST data ?", "limitations": "Limitation mentioned are the use of mBART encoder decoder instead of large decoder only models (with bigger architectures)", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-Good performance reported on CoVoST speech-to-text translation tasks\n\n-An approach that leverage existing pre-trained models (whisper speech encoder; mBART text2text) to build efficient ST systems\n\n", "weaknesses": "-Difference/positioning related to previous speech-text pretrained models should be improved: in what aspect does your method really differ from SpeechT5, SLAM, etc ?\n\n-Improvement over Whisper Large when model is trained without pseudo ST data is tiny (and maybe not significant) so does the improvement really comes from the new approach/losses proposed or does it come only from the use of pseudo ST data ?\n\n-Only Whisper speechencoder is used, why ? Authors could have also chosen wav2vec multilingual (XLS-R) or HuBERT speech encoders", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. We have provided our responses right after each question.\n\n1.\tIn what aspect does your method really differ from SpeechT5, SLAM, etc ?\n\nAnswers: \nOur methods differ from those used in SpeechT5, SLAM, etc. in the following ways: \n\nA) The cross-modality learning (CML) is employed in different training stages. In previous works such as SpeechT5 and SLAM, CML is implemented in the pre-training stage, while ours is used in the fine-tuning stage together with fine-tuning task losses.\n\nB) Different CML methods/objectives are utilized. In SpeechT5, its technique for joint pre-training involves utilizing vector-quantized embeddings as a bridge to align the speech and text representations through a shared codebook and mixing up contextual representations and quantized latent representations to guide the quantizer to use cross-modal information. In SLAM, although it also includes a joint input of speech and text, it predicts masked text or speech spans with BERT or w2v-BERT on top of the encoder. \n\nIn contrast, our approach does not require quantization for speech input or an MLM loss for encoders. Therefore, our approach is completely different from SpeechT5 and SLAM. Our approach is more flexible, requires less data as it is applied during the fine-tuning stage, and can be used for adaptation between any composite transformer-based speech and language model.\n\n2.\t\"Speech Transformer blocks are initialized with the encoder parameters of Whisper model\" why using Whisper speech encoder only and not XLS-R (multilingual wav2vec) or HuBERT ? \n\nAnswers: We believe that our methods are also applicable to XLS-R or HuBERT. HuBERT was pre-trained using Libri-light (60k hours), a collection of spoken English audio without labels. As a result, HuBERT is not a multi-lingual model and requires much more multi-lingual fine-tuning data to produce decent translation performance. On the other hand, XLS-R was pre-trained on massive multi-lingual data in a self-supervised learning manner. However, Whisper prevails in multi-lingual tasks recently since it was trained with much more supervised data and achieved better performance. We chose to fine-tune Whisper as a very strong baseline and explore the potential of our approaches. Considering that Whisper uses a standard transformer architecture, we believe it is representative of transformer-based speech encoders. If given adequate computational resources, we will try HuBERT or XLS-R as well in the future.\n\n3.\t\"Different from the previous speech-text alignment approaches that rely on externally forced-alignment methods to determine word or other unit boundaries in speech sequences,\" => Do SpeechT5 & SLAM need external word-level forced-alignment (i don't think so)\n\nAnswers:  You are correct. SpeechT5 and SLAM do not require external word-level forced alignment. We will rephrase this sentence to make it clear that it refers to the MML loss in USM and the word-aligned contrastive loss in WACO, which are used for comparisons in Table 3.\n\n4.\t(tab 1) does the improvement really comes from the new approach/losses proposed or does it come only from the use of pseudo ST data?\n\nAnswers: The improvement comes from both the new approach/losses and the use of pseudo ST data. Please refer to the attached PDF file in the “global” response to all reviewers jointly for the results of significance tests, which show that, regardless of whether pseudo ST data is added, our ComSL Medium/Large models significantly outperform their corresponding baseline Medium/Large models (i.e., finetuned Whisper) on high-resource and mid-resource languages, with a significance level of p<0.01. In addition, we did not use any pseudo ST data for high-resource languages. For more information on the statistics of the pseudo data, please refer to the appendix.\n\n"}]

Author rebuttal:
We would like to thank all reviewers for taking time and effort to review our paper. We appreciate all your valuable comments and suggestions. To the common concerns and questions, our responses are listed here.

1. ASR evaluation

Due to space constraints in the full paper, we have included the ASR performance results in the appendix of the supplementary materials. This appendix also contains the ST and MT task results for each of the 21 languages in the CoVoST 2 evaluation set. Please review the ASR results in the supplementary materials. We will consider including a summary of these results in the revised version of the full paper. If the absence of an evaluation of the ASR task had an impact on the score you gave our paper, we hope you consider adjusting it accordingly.

Calculating WER for multi-lingual ASR outputs depends on the methods of text normalization and word segmentation for some languages. We were unable to find a standard and publicly available tool for this purpose, so we developed our own method and used it to compare the ASR performance of the models we built, namely Whisper Large finetuned and ComSL Medium/Large. It is important to note that the WER numbers we reported cannot be directly compared with those reported in other publications. 

2. Statistical significance test 

We have updated Table 1 to include the results of our statistical significance tests, which can be found in the attached PDF file. Our tests show that, regardless of whether pseudo ST data is added, our ComSL Medium/Large models significantly outperform their corresponding baseline Medium/Large models (i.e., finetuned Whisper) on high-resource and mid-resource languages, with a significance level of p<0.01. For low-resource languages, where the training data for most languages is around two hours, the addition of pseudo ST data has a significant impact on improving ST performance. We suspected that limited training data for a specific language might be causing issues, such as limited vocabulary coverage, which could affect the efficiency of our proposed methods. This is why we added pseudo ST data only to mid-resource and low-resource languages. Please refer to the appendix for statistics on the pseudo data.

3. Main contributions/differences 

In addition to the main contributions summarized in the last paragraph of the introduction section, we would like to highlight our proposed approach that combines two publicly available pre-trained speech and language models and fine-tunes them directly for downstream tasks. This approach also incorporates cross-modality learning, which is typically used during the pre-training stage. We believe that our approach offers a new avenue for research in academia, where large model pre-training may not be feasible due to limited computational resources. 


Meta-review: This research introduces ComSL, a composite speech-language model, which integrates pre-trained speech and text models to enhance speech translation. By incorporating cross-modality learning objectives and alignment methods without additional tools, the model achieves improved data efficiency and forms a stronger inter-modality relationship. Extensive evaluations on the CoVoST2 dataset indicate that ComSL surpasses existing benchmarks, setting a new state-of-the-art performance.

Many concerns were addressed in the rebuttal. The main critical review was from tyJC, who praised the method but had concerns about the evaluation, namely the lack of ASR results. The authors provided ASR results in the rebuttal. That influenced tyJC to increase their score.

I therefore recommend accept. The authors shall update their camera ready with all the changes promised in the rebuttal.
Decision: Accept (poster)