The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation
Authors: Mutian He
Philip N. Garner
Venue: EMNLP 2023 Findings (2023)
Keywords: spoken language understanding
multilinguality
bayesian transfer learning
TL;DR: Pretraining on speech translation and preserving the knowledge from pretraining helps end-to-end spoken language understanding, particularly on cross-lingual transfer scenarios.

Abstract: End-to-end spoken language understanding (SLU) remains elusive even with current large pretrained language models on text and speech, especially in multilingual cases. Machine translation has been established as a powerful pretraining objective on text as it enables the model to capture high-level semantics of the input utterance and associations between different languages, which is desired for speech models that work on lower-level acoustic frames. Motivated particularly by the task of cross-lingual SLU, we demonstrate that the task of speech translation (ST) is a good means of pretraining speech models for end-to-end SLU on both intra- and cross-lingual scenarios. 

By introducing ST, our models reach higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks. To verify the effectiveness of our methods, we also create new benchmark datasets from both synthetic and real sources, for speech summarization and low-resource/zero-shot transfer from English to French or Spanish. We further show the value of preserving knowledge for the ST pretraining task for better downstream performance, possibly using Bayesian transfer regularizers.

The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation
Mutian Hel Philip N. Garner
Idiap Research Institute, Martigny, Switzerland
2 Ecole Polytechnique Fédérale de Lausanne, Switzerland (mutian. he, phil. garnergrrrrrrreee Jeidia...p.p....
Abstract
text infilling, and token deletion (Lewis al., 2020). While the rather simple NSP has been re- placed by sentence permutation, document rotation (Lewis et al., 2020), and sentence order prediction (Lan et al., 2020). All those efforts introduced more challenges in the pretraining phase to mine stronger semantic supervision signals out of unlabelled data.
End-to-end spoken language understanding (SLU) remains elusive even with current large pretrained language models on text and speech, especially in multilingual cases. Machine trans- lation has been established training objective on text as enables the model to capture high-level semantics of the input utterance and associations between differ- ent languages, which is desired for speech mod- els that work on lower-level acoustic frames. Motivated particularly by the task of cross- lingual SLJ,, we demonstrate that the task of speech translation (ST) is good means of pre- training speech models for end-to-end SLU on both intra- and cross-lingual scenarios.
powerful pre-
Such semantic-rich supervision is particularly relevant for pretrained spoken language models like wav2vec2 (Baevski al., 2020) and HuBERT (Hsu et al., 2021) based on MLM on (sub-)phonetic units from lower-level audio signals, which are less infor- mative and require models to carry out additional labor on acoustics. Therefore, their high-level ca- pacities are more restricted. This may explain why automatic speech recognition (ASR) models fine- tuned upon them with paired data still have role in fully end-to-end (E2E) SLU, often pretrained feature extractor (Seo et al., 2022: Arora al., 2022). Unlike the cascaded SLU which ASR produces transcripts for text processing, in such E2E systems ASR as an auxiliary or additional pre- training task provides strong supervision to explic- itly link audio to representations that correspond to the denser and semantic- richer textual space, which is valuable for downstream understanding tasks.
By introducing ST, our models reach higher per- formance baselines monolingual and multilingual intent classification as well as spo- ken question answering using SLURP, MINDS- 14, and NMSQA benchmarks. To verify the effectiveness of our methods, we also create new benchmark datasets from both synthetic and real sources, for speech summarization and www-resource/zero-shhoo transfer from English to French Spanish. We further show the value of preserving knowledge for the ST pretrain- ing task for better downstream performance, possibly using Bayesian transfer regularizers.
On texts, self-supervised objectives are rather effective thanks enormous text data with high information density, but supervised tasks are still used in many cases, machine translation (MT) be- ing an often-seen one. pioneer of the current PTLM paradigm, CoVe (McCann et al., 2017), is a seq2seq model pretrained on MT that achieved the then -ftt----athtrrr on various downstream tasks. Belinkov et al. (2020) further validate lan- guage capabilities of MT morphological, syn- tactic, and semantic levels, and T5 (Raffel et al., 2020) uses an ensemble of supervised tasks includ- ing MT. Furthermore, when trained with inputs of multiple languages, the model encoder may align and push representations for inputs in different lan-
1 Introduction
Modern artificial intelligence is characterized by large pretrained language models (PTLMs) with strong language capabilities to be adapted to var- ious downstream tasks. The success of PTLMs rests darefielly-desigeed pretraining tasks to be- stow the capability we expect on the model. Cur- rent PTLMs mostly trained self-supervised tasks, which started from masked language mod- elling (MLM) and next sentence prediction (NSP) in BERT (Devlin et al., 2019), but recently evolved into more difficult ones such as whole word (Cui et al., 2021) or span masking (Joshi et al., 2020),

---

guages with similar meaning together to have the same output in the target language, thanks to the guidance from paired data (Johnson et al., 2017: Schwenk and Douze, 2017). With this semantic- centric language agnosticity, such an encoder can achieve few/zero-shot transfer to another language in downstream tasks (Eriguchi et al., 2018).
Ab
Light Turnoff
Nice meet you
Query Weather
Queryy_Watther
|| fonctionne très bien
Query Currency
mBART Decoder
Classifier
Inspired by those works, we hypothesize that the counterpart of multilingual MT on speech, i.e., E2E multilingual speech translation (ST) that directly maps speech of various languages to texts in other languages, will also be effective a pretraining task on E2E SLU, for three critical advantages:
mBART Encoder
Encoder
Target Embed
Adaptor
XLSR-53 (lower helf)
1. It requires high-level understanding as in- terpreter must "understand" the utterance before interpreting it into a different language, unlike ASR that transcribes speech verbatim and MLM on pho- netic units that needs less semantic understanding.
dpo
enchanté
fermer la lumière
It works very well
iquel est le taux de change: entre l'euro et dollar?
Downstream SLU
should gloves
Cross-lingual (Limitedino training data)
It captures long-term dependency and a global view of the full input, in contrast to ASR and MLM which can often be resolved with local context.
tonight
Figure I: Our framework of ST-aided SLU, by connect- ing pretrained XISR and mBART fine-tuned for ST following Li et al. (2021), and then reusing the ST en- coder, transferred to downstream SLU tasks like intent classification with stacked classifier also from PTLMs.
It enables better cross-lingus transfer in com- parison with multilingual ASR models and self- supervised PTLMs without the supervision that promotes language agnosticity.
Admittedly, ST data is only available in a lim- ited number of language pairs, but for each covered language, there are infinite number of diverse down- stream SLU tasks with only rich data in English. It is a practical need enroll various such languages an English-only model trained on each specific SLU task. Therefore, as shown in Figure we may pretrain the model on speech translation between English and the target language French in both di- rections (i.e. En+FFr), and then fine-tune on down- stream tasks with an additional classifier, reusing the encoder. We show the benefit of our method a variety of tasks for semantic understanding of speech, including mono- & multilingual intent clas- sification (IC), spoken question answering (SQA), as well as speech summarization, for which we create a synthetic dataset following Huang et al. (2022). Then we show the strong advantage on cross-lingual transfer to French. All the experi- ments are focused on comparing ST with other tasks like ASR as the pretraining or auxiliary task verify our core hypothesis above. In addition, to show that our method applies other languages as well, we also conducted experiments using Spanish as the target language. This is evaluated by creating the French and Spanish version of the English IC
benchmark SLURP (Bastianelli et al., 2020), using both real and synthetic sources. On all the tasks, our approach outperforms previous baselines and ASR pretraining, often by large margin.
Furthermore, unlike knowledge for self- supervised objectives loosely connected to target SLU tasks, knowledge to handle tasks with closer link to semantics such ST will be more valuable, following our core hypothesis. Hence it should be helpful to preserve such knowledge instead of direct fine-tuning with the risk of catastrophic forgetting. Therefore, we introduce multi-task learning as well as Bayesian regularizers for knowledge preservation, namely L2-SP (Li et al., 201 8b) and EWC (Kirkpatrick et al., 2017), which show benefits especially in low-resource cases. To summarize, our contributions are three-fold:
1. We demonstrate the effectiveness of speech translation pretraining on multiple SLU tasks, es- pecially in cross-lingual transfer cases.
2. We confirm the value of preserving ST pre- training knowledge for downstream tasks and the capability of Bayesian regularizers to achieve that. 3. We build several new datasets for speech summarization and cross-lingual SLU.

---

Our code, models, and datasets will be released at https:/ //ss:///tttt www..hh..bc.. wtt.coo///iiac idiap/ translation-aided-Slll
ASR WER!
TEDx
MuST-C
CoVoST2
ASR
16.58%
8.62%
13.67%
ASR+ST
15.82%
8.28%
13.62%
2 Model Pretraining
ST BLEUt
ST
29.27%
36.30%
31.34%
As in Figure 1, we first build speech transla- tor using architecture established by Li et al. (2021), that connects pretrained models on speech and text with a CNN adaptor: Audio signals are fed into the lower half of the multilingual wav2vec2, XLSR-53 (Conneau et al., 2021), to extract the (sub-)phonetic representations into a 220xmdowssampled sequence. The upper half (12 layers) of XLSR is discarded for computational ef- ficiency, those parameters are found focused on MLM pretraining and less useful for downstream tasks (Zhu al., 2022). Given the phonetic level embeddings produced by the half XLSR, the task is similar to machine translation to map them to the output text, for which we leverage the MT model based on mBART (Liu et al., 2020). While the length of this sequence is still much longer than the corresponding text. To better align it with typ- ical textual embeddings in mBART inputs, 3-layer Ex-downsampling CNN adaptor is inserted. A target embedding is then prepended specify the target language or task, similar to the target token used in mBART. To promote language ag- nosticity, we do not indicate the source language. Furthermore, it has been found that explicitly pro- moting language agnosticity may help zero-shot transfer (Arivazhagan et al., 2019), hence we' 've also attempted add language adversarial train- ing on the encoder outputs during pretraining and fine-tuning, using a language classifier of 2-layer MLP to predict the language of the input speech, with gradient reversal layer to explicitly align the representations between different languages.
ASR+ST
31.19%
37.18%
31.93%
Table 1:
Test results on the cleaned pretraining datasets given by word error rate (WER) for ASR and BLEU score for ST, with French inputs for TEDx and CoV- oST2, and English inputs for MuST-C.
transcripts, we use the same ST dataset for ASR training to build multilingual (En+Fr) ASR model. We've ve tried to jointly train on +SR+ST in a multi- task manner as well. With a total of >700 hours paired speech data, we achieve satisfactory results on the pretraining tasks as indicated in Table 1, and ASR+ST training shows better performance com- pared the single-task ones. Starting from the ASR and ST models, we further add Spanish por- tion from the same set of ST datasets. As result, we obtain an ST model supporting both Ent+Fr and Ent> (Spanish), and a tri-lingual En+Fr+Es ASR model, both with similar satisfactory results, details available in Appendix E.
3 Downstream Adaptation
3.1 Tasks
We then fine-tune the whole model on a variety of direct downstream tasks as follows.
SLURP is large and challenging English SLU dataset recently proposed, with 72.2k real speech recordings and 69.3k synthetic audio for broad range of speech commands given to voice assis- tants. We use its IC labels to classify the input into 18 scenarios and 46 actions.
Based on the architecture. we fine-tune the model using a combination of the En->Fr portion of MuST-C (Gangi et al., 2019), and the Fr- portion of TEDx (Salesky et al., 2021), both de- rived from TED talks, plus the Fr-xEn portion of CoVoST2 (Wang et al., 2021a) based on general sentences in Common Voice (Ardila et al., 2020), with texts further cleaned and sentences that are too long or contain foreign characters removed. Unlike Li et al. (2021), the whole model is fine-tuned for best pretraining results. To compare, we also exper- iment with pretraining on the task of ASR instcad. As the data are paired with both translations and
MINDS-14 is multilingual IC dataset for bank- ing scenarios with 14 types of intents in 14 lan- guages with around 600 utterances per language, and we use four subsets (en-AU, en-GB, en-US, and fr-FR) under a 3:2:5 train-dev-test split in X XTREME-S (Conneau et al., 2022). The rather scarce training data demand data iata-efficient multilin- gual modelling.
NMSQA or Natural Multi-Speakerr Question An- swering is spoken QA dataset consisting of audio for the questions and segmented context articles from SQuAD (Rajpurkar et al., 2016), with 97.6k

---

question-answer pairs given in >300 hours of syn- thetic audio from 12 speakers produced by Amazon TTS, coupled with a 60-speaker real test set of 2.7 hours of recordings. In this task, the goal is sim- ilar to textual QA to predict the correct span in the spoken context audio that answers the ques- tion, and the performance is measured by Audio Overlapping Score (AOS) (Li et al., 2018a), de- fined as AOS X n / UY, in which X is the predicted audio span and Y the ground truth.
ment. We then concatenate the question audio with each segment in the spoken article as model in- puts, and pick the predicted answer span from the segment with the highest answer existence likeli- hood. For these two tasks, the pretrained decoder is simply discarded. While speech summarization is more similar to ASR and distinct from other down- stream tasks that the model first needs to capture general meaning of the speech as encoded repre- sentations, and then generate a textual summary by the decoder, which demands a seq2seq architecture identical to the ST/ASR pretraining task. Hence we reuse the whole ancoder-decoder model and formulate the task as generation in an extra "tar- get language". With the needs to both understand the general meaning and generate in the same lan- guage, we hypothesize that combining ASR and ST will lead to the best results.
Spoken Gigaword is the synthetic spoken ver- sion of the summarization or headline generation task Gigaword (Rush et al., 2015), proposed by Huang et al. (2022), aimed at generating a brief headline from short piece of English spoken news. As was not released, we follow their method to filter the data and create a synthetic dataset of 131.5 1,.5 hours of audio using Google TTS from 9 neural voices in en-US, with 50k training samples, Ik val- idation samples, and 385 test samples, as result of filtering out the frequent noise in the test set.
Furthermore, mentioned above, direct model fine-tuning may lead to catastrophic forgetting of the knowledge on ASR ST and harm semantic understanding capabilities. Hence we also tried a multi- -tsk joint training approach on both the pre- training and target task. Results are compared be- tween the model pretrained with ST, ASR, or both, or one directly derived from self-supervised pre- training without further supervision (None), plus other baselines. More, the recent Whisper (Rad- ford et al., 2023) is trained on multiple speech tasks including ASR and ST, which matches our idea despite not aiming at SLU, Hence we also try fine-tune the Whisper encoder, using the medium version with size similar to our encoder.
Synthetic data are used in those established datasets for training and evaluation. Despite be- ing possibly different from real data, it is observed that they are often reliable to reflect model perfor- mances and well correlated with real cases.
3.2 Methods
For these downstream tasks, reuse the encoder further pretrained ST/AS (with French, unless otherwise stated). It should be noted that the 12- layer mBART encoder we use is slightly smaller than half XLSR. So when connected, the total en- coder size and the computational cost to fine-tune it is comparable with fine-tuning the whole original XLSR. Upon the encoder we stack 3-layer trans- former. which is also transferred from a PTLM. As for IC, we use layer 2-4 from pretrained XLM-R (Conneau et al., 2020) for possibly better under- standing capabilities, stacked with linear classifier heads over mean-pooled outputs. Particularly, for SLURP in which the intent consists of a scenario and an action, two heads are used. As for SQA, we use layer 2-4 of pretrained Longformer (Belt- agy et al., 2020), a PTLM dedicated for long ut- terances due to the length of each segment in the data, as in Lin et al. (2022). Two linear classi- fiers are then applied each frame to predict the start and end of the span, along with an answer existence classifier over mean mcan-pooled outputs to predict if the answer exists in the provided seg-
3.3 Results
English IC Following the previous works, we report the test accuracy on SLURP as in Table 2. It can be observed that the models with ST pre- training outperform those trained on ASR only, while adding ASR ST pretraining makes lim- ited improvements, though it gives better WER and BLEU during pretraining; it is the same case for the model with the extra Spanish ST task introduced in pretraining. However, ASR does help consider- ing the None model directly fine-tuned from self- supervised PTLMs without any additional pretrain- ing. By joint training with both the pretraining and downstream task, results are consistently improved. However, despite being a strong ASR+ST model, Whisper is found not suitable for fine-tuning on SLURP in this way as shown by the low accuracy.

---

Pretraining task
Accuracyt
thogonally for further improvement, and we main- tain advantages on cross-lingual transfer and pos- sibly also generative tasks by reusing a pretrained seq2seq decoder, as elaborated below.
ASR
87.38%
w/ Joint training
88.37%
ST
87.84%
w/ Joint training
89.35%
Multilingual IC We then report the accuracy on MINDS- 14 as in Table 3 on four languages plus the average accuracy across languages, compared to a baseline directly fine-tuned from XLSSR The results are consistent with the monolingual case that ST pretraining can significantly improve the performance SLU tasks, that joint training is beneficial, and that adding ASR gives limited gains.
w/ Joint training Es
89.59 %
ST+ASR
87.75%
w/ Joint training
89.43%
None
84.80%
Whisper
80.39%
ESPnet-SLU (Arora al., 2022)
86.30%
CTI (Seo et al., 2022)
86.92%
Generative IC+SF (Wang et al., 2021b)
Spoken QA We compare our methods with re- sults reported by Lin et al. (2022), including the results from cascaded pipeline that fine-tunes Longformer upon transcripts from way2vec2-baue ASR, and the DUAL approach that fine-tunes Long- former upon units pre-extracted by frozen Hu- BERT, hence not fully end-to-end. For fair compar- ison, we fine-tune the classifier built by layers 2-4 of Longformer and the top 5 layers of the mBART encoder, while the rest of the model is frozen and used as feature extractor, so that they have com- parable number of trainable parameters with the baselines. Therefore ments on joint training in this task most shared parameters are frozen. The results reported the more recent T5lephone (Hsu et al., 2023) including the E2E and cascaded approach are also mentioned, though they are almost twice as large as other mod- els. All the baselines enjoy view of the whole article, while in our experiments we use model that works a shorter context window with the question and each segment in the article individ- ually, in order to have an end-to-end architecture given our computational resources that is consistent with other experiments. Therefore, the baselines possess a strong advantage over ours. However, as shown in Table 4, the additional pretraining stage leads to better results compared to all the E2E base- lines, which further demonstrates the advantage of approach. Particularly, ST considerably im- proves the performance and could successfully beat the cascaded system reported by Lin et (2022) in the more challenging test portion.
based on wav2vec2
87. 87.13%
based on HuBERT
89.38%
CIF-PT (Dong et al., 2023)
91.43%
Table 2: SLURP test results of our models, fine-tuned from wav2vec2, compared baselines without addi- tional supervised pretraining reusing Whisper the encoder, as well as results reported in literatures.
This might be explained by the fact that Whisper is trained En ASR and X-En ST but not for En-+X ST. Our hypothesis is that the ST pretrain- ing on specific language would enhance the se- mantic understanding capabilities the model in that language, which may not help Whisper much on the English SLURP benchmark. Also, Whis- per is trained on 30-second chunks, while SLURP contains more shorter utterances.
do not conduct experi-
HuBERT, used multiple baselines in Table 2, has been found stronger on various downstream tasks compared to wav2vec2. Owing to the lack of a multilingual HuBERT (large) model, we rely on the multilingual wav2vec2 as our acoustic encoder. However, reach much better results compared many notable baselines, including the approach of jointly generating the intents and slots (Wang et al., 2021b), with 87.13% accuracy, the highest among wav2vec2-baad based baselines. We also reach slightly higher accuracy than its HuBERT version, which was the previous state-of state-of-the-a of-the-art. The very recent CIF-PT (Dong et al., 2023), concurrent with ours also injects more semantic signal, but by learn- ing frame-to-token alignments on the encoder and then distilling from PTLMs, significantly pushing the state-of-the-an on this monolingual benchmark. Nevertheless the method is distinct from ours, rais- ing the possibility of applying both methods or-
Speech summarization We report the results compared between different auxiliary tasks as in Ta- ble using the ROUGE- 1/2/L. metries (Lin, 2004). In the experiments, we observed that simply fine- tuning the model rapidly leads to overfitting, hence

---

Pretraining task
en-AU
en-GB
en-US
fr-FR
Average
ASR
95.7%
97.3%
96.5%
95 95.2%
96.2%
w/ Joint training
96.3%
98.3%
98.2%
93.7%
96.6%
ST
96.9%
99.0%
98.2%
97.8%
98.0%
w/ Joint training
97.3%
98.7%
99.3%
98.2%
98.3%
ST+ASR
95.4%
98.3%
97.5%
95.6%
96.7%
w/ Joint training
96.3%
98.3%
98.9%
98.5%
98.0%
XLSR (Lozhkov, 2022)
92.4%
93.2%
93.3%
94.4%
93.3%
Table 3: Test accuracies for models MINDS-14 multilingual IC, comparing with directly fine-tuning the full XLSR model. Both ST pretraining and joint training show benefits.
4 Cross-lingual Transfer
Pretraining task
dev
test
ASR
54.6%
53.0%
For cross-lingual transfer, IC models trained on SLURP are then applied on/fine-tuned French/Spanish data below:
ST
58.2% %
59.4%
ST+ASR
57.8%
58.0%
DUAL E2E
48.5%
49.1%
Datasets A French version of SLURP, SLURP- Fr, is created to evaluate the cross-lingual trans- fer capabilities of the model, which is based on MASSIVE (FitzGerald et al., 2023), translation of SLURP texts into multiple languages. With the same input domain and output categories, shoo transfer becomes possible. We first produce the au- dio for the 16.5k French samples MASSIVE with a 7:2:1 train-dev-test split using Google TTS from four different WaveNet- based speakers. Then we invite two native French speakers to read out total of 477 randomly-selected category-balanced held-out utterances, forming the real test set. To mimic SLURP, we record the audio indoors with two microphones under both near-field and far-field conditions. We also define a 100-shot per category subset with 4.5k samples in total to simulate a con- dition with even lower resource. SLURP-Es is created in way similar to SLURP-Fr with 16.5k Spanish samples in MASSIVE, though are able to create real set.
Cascaded
58.3%
57.4%
ByTSlephone E2E
53.3%
Cascadeed
59.2%
70.5%
Table 4: AOS (1) scores for models on NMSQA, com- pared to baselines reported in DUAL (Lin et al., 2022) well as the much larger BYTSlephone model (Hsu et al., 2023). The pretraining tasks prove helpful, par- ticularly ST pretraining, which may reach performance close better certain cascaded system.
we perform joint-training only, and use a spe- cial target embedding to indicate the summariza- tion task. ASR helpful on the summarization task as the ST+ASR model consistently outper- forms the ST one, while the ST one is still bet- ter than the ASR-only model, signifying the im- portance of the semantic understanding capability brought by ST pretraining. In addition, com- pare with a cascaded baseline that first transcribes the inputs with our ASR model, which introduces WER of 9.19 % and 8.9% on dev and test respec- tively. Then we leverage a BART-based model fine- tuned on the full textual Gigaword with ROUGE- 0111377228//858834... to produce the summaries. When applied to the relatively simple utterances in Spoken Gigaword, it reaches higher performance dev, which suggests the challenges for E2E sys- tems in our benchmark, though the gap is narrow compared to our E2E approach with ST+ASR pre- training, and on the noisier test set our E2E models consistently get much better results.
The advantage of our method on cross-lingual transfer evaluated under the full- data, 100-shot, and zero-shot cases, using different pretraining strategies compared to the None model trained on SLURP without further supervision but directly upon the multilingual self-supervised pre- trained models. Hence it is noteworthy that all the compared models have been pretrained in mul- tilingual way. As given in Table 6 French, ex- tra multilingual ST/ASR supervision consistently leads to better results on different data amounts. ST pretraining outperforms ASR, similar to previ-
Experiments

---

dev
test
Joint task
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-1
ROUGE-2
ROUGE-L
ASR
40.16
18.39
37.69
35.90
16.25
33.77
ST
40.61
18.95
38.23
36.70
16.39
34.47
ST+ASR
41.39
19.50
38.83
37.63
17.80
35.20
None
21.49
7.44
20.37
18.39
6.16
17.41
Cascaded
42.00
21.42
39.60
32.24
15.03
30.14
Table 5: ROUGE (f) scores for models on Spoken Gigaword speech summarization. ST still proves beneficial. while best results could be obtained by combining ASR for this task of generating summaries in the same language.
Full
100-shot test
Zero-shot
Pretrain task
dev
test
real
dev
real
dev
test
real
ASR
83.3%
84.0%
79.7%
69.6%
69.0%
71.3%
39.0%
39.8%
39.4%
ST
85.2%
86.1 %
84.9 %
78.1
77.0%
79.0%
58.9%
58.9%
56.6%
ST+ASR
85.8%
85.7%
82.4%
78.0%
77.0%
78.8%
63.9%
62.6%
59.1% %
ST+Adv. 86.4%
84.9%
84.1 %
78.3% %
78.1% %
80.9%
67.0% %
67.7%
63.7
None 75.9%
74.0%
65.4%
57.5%
52.4%
53.5%
15.3%
16.1%
13.6%
Table 6: Results on SLURP-Fr cross-lingual IC transferred from SLURP with different data amounts. Comparing different supervised pretraining (or w/o additional supervision), results highlight ST and adversarial training.
None
ASR
ST
features to the classifier. In addition, build cascaded system which first translates the speech into English text using our ST model, with BLEU scores of 26. 2e..08/26.34/2111 1,56 on dev/test/real. Then BART-based textual SLURP model with 85.7% test accuracy is used. This essentially zero-shot system gives a competitive 62.2% real accuracy, which poses challenges to the future development of E2E cross-lingual models. The model on Span- ish, based on En+Fr+Es ST/ASR pretraining along with training SLURP in an identical protocol, shows similar results as Table 7, indicating the applicability our approach to other languages.
Full
dev 75.6%
83.7%
85.6%
test 75.1%
82.5%
84.5%
100-shot
dev 64.4%
76.5%
80.3%
test 63.8%
75.1%
79.6%
Zero-shot
dev 12.3%
39.7%
55.2%
test 12.8%
39.1%
54.4%
Table 7: Results SLURP-Es cross-lingual IC trans- ferred from SLURP with different data amounts.
ous experiments, while ST+ASR joint pretraining brings some improvements in the zero-shot case. Notably, the gap between ASR and ST models becomes larger with fewer data, especially with zero shot, which implies the importance of ST on cross-lingual transfer. Accuracy on the real near-field speech is reported, which correlates well with those on synthetic ones, indicating that perfor- mance on synthetic speech is reliable for evaluation. The ST+Adv. model incorporates language adver- sarial training during pretraining and fine-tuning further promote language agnosticity men- tioned above, which outperforms other models in most cases, particularly with zero shot, implying the usefulness of language adversarial training and the importance of language agnosticity of input
Pretraining Knowledge Preservation
Methods As mentioned above, the knowledge to perform ASR/ST that connects speech and semantic-rich texts could be valuable for down- stream tasks, which motivates us to use joint train- ing above that maintains the performance on the pretraining tasks. This is verified by the consid- erable performance improvement or gap between the joint and single models. However, it is com- putationally intensive and requires access the pretraining data. To alleviate the gap without joint training, we intend to explore Bayesian trans- fer/continual learning regularizers that limit the pa- rameter shift by applying a prior on the parameters,

---

STRASR
353.
Jnhn
$6.25
$3.3
365
56.33
302
10.32
902
(din 98:02
00.22
24
274
516
07:37
57.6 $6.72
797.5
075
575
97.54
472
672
964-
$3.30
Je:20es
Figure 2: Results for Bayesian transfer regularizers when applied different tasks, with the goal of mitigating the gap between the performance of single-task and joint-training models, indicated by the lower and the upper horizontal lines. The x-axis indicates the regularization weight for EWC/L2-SP, and the y-axis the accuracy. The regularizers bring positive effects on the data-scarce MINDS- 14 task, but not on SLURP.
based on the Laplacian approximation of posterior parameter distributions in pretraining (MacKay, 1992). Particularly, the L2-SP method formulate the prior isotropic Gaussian distribution with the pretrained parameters 0g the mean and iden- tical variance for all parameters, which leads to an L2 regularization term with weight a centered at Oo in the loss for the maximum likelihood estimation of the parameters (Li et al., 2018b). While elas- tic weight consolidation (EWC) (Kirkpatrick et al., 2017) considers the variance of each parameter 0g decided by the Fisher diagonal F;, which could be further estimated using squared gradients by aver- aging over the stochastic gradient descent (SGD) trajectory. However, for optimization we use the Adam algorithm
the weight aF; above 1c-2, and use EWC weights of 2e2, 2e4, 2e6, and 2e7 to roughly match the magnitude of those for the L2-SP regularizer.
Results shown in Figure 2, and for MINDS- 14 the average accuracies are reported. In the case of SLURP, possible that the amount of data is already sufficient that the preservation of the pre- training knowledge could be helpful only if it is carried out in fully adaptive way, namely joint training. Therefore, the regularizers lead to limited help
harm to the accuracy when the weight is large. However, under the low-resource condi- tion in MINDS-14, both regularizers are effective. As in Li al. (201 although being more flexi- ble and adaptive, EWC doesn't 't necessarily lead better transfer learning. This is consistent with our observations: Both regularizers can successfully overcome the accuracy gap or even go beyond the joint training model under appropriate weight, while the best regularizer varies in different cases, though the more adaptive EWC has chance reach better results as the MINDS- ST+ASR case. this way, we demonstrate the effectiveness of Bayesian parameter peaetterrrrreeiiingeg regularizers for transfer learning on such large pretrained models.
(1) that already computes é,, an exponential moving average of squared gradients (Kingma and Ba, 2015), close to linear averaging with smoothing parameter B2 0.999. Hence we reuse them to set the per-parameter weight aF;; for regularization. For both methods, the hyperparameter is used control the strength of the knowledge preserva- tion or the restraint the parameter update. See Appendix A for more theoretical explanations.
0, 10-0111- a- m//(Vo +e,,
6 Related Work
Experiments We experiment with these regu- larizers, targeted on ST-pretraining on SLURP
Translation as an auxiliary task has been found that representations from MT models cap- ture various aspects of the input utterance such as syntax (Shi al., 2016), morphology (Belinkov et al., 2017), and also semantic inferences (Po- liak et al., 2018; Belinkov al., 2020). Hence MT has been established as a pretraining task as in CoVe (McCann et al., 2017) for various down- stream tasks. But unlike this paper, recent works
and MINDS-14 14, plus the ST+ASR pretraining on MINDS-14 14 which has considerable 1.32% ac- curacy gap. We try to various weights for L2-SP regularization ranging from le-5 le-2, Then we inspect the distribution of the approxi- mated F,, which ranges from le-20 to le-5 as in Appendix For optimization stability we clamp

---

on the direction has been focused on multilingual and cross-lingual cases, starting from attempts to reuse MT representations as sentence embeddings for text classification (Shi et al., 2016; Lu et al., 2018), and, particularly often, for semantic sim- ilarity and bi-text mining (Schwenk and Douze, 2017; V Vázquez et al., 2019; Raganato et al., 2019; Artetxe and Schwenk, 2019). As for pretraining PTLMs be fine-tuned, MT proves effective for downstream cross-lingual tasks on few-shot and zero-shot transfer (Eriguchi et al., 2018), while often accompanied with similar tasks like trans- lation language modelling (Conneau and Lample, 2019; Kale et al.. 2021), cross-lingual MLM (Chi et al., 2021), and dictionary denoising (Reid and Artetxe, 2022). Particularly, MT has been used as auxiliary task for tross-lingual intent classifica- tion on texts (Schuster et al., 2019; Siddhant et al.. 2020; van der Goot et al., 2021), and is widely used on cross-lingual generation, including summariza- tion (Zhu et al., 2019; Cao et al., 2020; Xu et al., 2020; Takase and Okazaki, 2022), simplification (Mallinson et ul., 2020), question generation (Chi et al., 2020), and data-to-text generation (Kale and Roy, 2020).
tuning as in L2-SP for image classification (Li al., 2018b), though similar regularizers have been used on MT (Barone et al., 2017) and ASR (Liao, 2013). More recently, Shwartz-Ziv et al. (2022) propose to approximate the prior using SGD trajectory as in SWAG (Maddox et al., 2019) for transfer learning.
7 Conclusion
We confirm our hypothesis that speech translation can be a powerful pretraining and joint-training means for various end-to-end models on tasks in- volving semantic understanding of speech, Partic- ularly, it benefits multilingual scenarios and cross- lingual transfer, including the zero-shot case. We also create two new datasets for the above tasks. Furthermore, we demonstrate the effectiveness of Bayesian regularizers to preserve the knowledge from pretraining for downstream tasks.
Limitations
Some of the limitations of our paper are:
1. The best results are mostly achieved with multi-task learning, which adaptively preserves the knowledge from the pretraining task, but much slower, computationally intensive, and energy con- suming. Therefore we explore the regularizers from continual learning for knowledge preserva- tion, while there are some other continual learn- ing approaches (e.g. Learning without Forgetting, Gradient Episodic Memory) that might be helpful. Also, we haven't 't explored alternative regulariza- tion approaches and light-weight tuning.
End-to-end SLU Cascaded SLU methods work on ASR transcripts, for which error propagation is major challenge (Chang and Chen, 2022; Cheng al., 2023a). Hence recently end-to-end meth- ods have gained popularity (Serdyuk et al., 2018; Haghani et al., 2018), especially with the perfor- mance gap compared with cascaded systems miti- gated in many cases thanks to the PTLM paradigm. Besides directly fine-tuning existing PTLMs on speech (Wang et al., 2021b; Arora et al., 2022), there also explorations for end-to-end interface connect pretrained models on speech and text (Saxon et al., 2021: Seo al., 2022; Raju al., 2022), as well as joint speech-text modelling, pre- training, or distillation (Chuang et al., 2020; Chung et al., 2021; Kim et al., 2021; Villatoro-Tello et al., 2023; Dong et al., 2023), prompt tuning for PTLMs (Gao et al., 2022: Chang et al., 2022), combining PTLM features (Cheng et al., 2023b), and multitask learning with ASR (Huang et al., 2022).
2. On the monolingual case (i.e. on SLURP), despite getting much better result under fair com- parison with the alternative training methods and other baselines based on wav2vec2, our result is only slightly better than the HuBERT-based gener- ative approach (Wang et al., 2021b), which is the state-of-the-ee before us. Very recently, CIF-PT (Dong et al., 2023), parallel with our work in time, reaches 1.9% higher than both types of models, marking a new ofate-of-the-arts. This approach ap- pears to be orthogonal to ours and the two methods might be jointly applied to the SLU model to reach even better results, but this is left for future work.
3. The dataset we built is relatively small and with limited number of real samples.
Bayesian transfer learning Viewing the pre- trained model not as point estimation but distri- bution critical for continual learning as in EWC (Kirkpatrick et al., 2017), and the idea has been also applied to transfer learning to regularize fine-
Ethics Statement
We honor the ACL Code of Ethics. Particularly, as our work involves data collection, we go through a

---

formal process at the institution for collecting au- dio data, strictly follow the general and local rules for data protection, and receive full consent of par- ticipants to process and release the data. Since cross-lingual transfer is highlighted in our work, the work could have positive societal impacts for the application of speech and language technology in the non-English population. We believe that there is little chance for the method to be misused, except in cases of misusing SLU, such as mass surveillance. We also emphasize the reproducibil- ity, and will release relevant code and models.
Yonatan Belinkov, Nadir Durrami, Fahim Dalvi, Hassan Sajjad, and James R. Glass. 2017. What do neural machine translation models learn about morphology? In ACL 2017, pages 861 ACL
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James R. Glass. 2020. On the linguistic representational power of neural machine translation models. Comput. Linguistics, 46(1):1-52.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long doeummo transformer. CoRR. abs/2004.05150.
Yue Cao, Xiaojun Wan, Jin-ge Yao, and Dian Yu. 2020. MultiSumm: Towards unified model for multi- lingual abstractive summarization. In AAAI 2020, pages 11-18, AAAI Press.
A Acknowledgements
This work received funding under project SteADI, Swiss National Science Foundation grant 197479.
Kai-Wei Chang, Wei-Cheng Tseng. Shang-Wen Li, and Hung-yi Lee. 2022. An exploration of prompt tuning generative spoken language model for speech pro- cessing tasks. In Interspeech 2022, pages 5005-5009. ISCA.
References
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common Voice: A massively- multilingual speech corpus. In LREC 2020, pages 0218-42222 European Language Resources Associa- tion.
Ya-Hsin Chang and Yun-Nung Chen. 2022. Contrastive learning for improving ASR robustness in spoken language understanding. In Interspeech 2022, pages 8458-3462 ISCA.
Xuxin Cheng. Bowen Cao, Qichen Ye. Zhihong Zhu, Hongxiang Li, and Yuexian Zou. 2023a. ML-LMCL: Mutual learning and large-margin contrastive learn- ing for improving ASR robustness in spoken lan- guage understanding. In Findings of ACL 2023, pages 6492-6522 0002-6505.44 ACL.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019. The missing ingredient in zero-shot neural machine translation. CORR, abs/1903.07091.
Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang, Yushi Ueda, Yifan Peng, Yuekai Zhang, Sujay Kumar, Karthik Ganesan, Brian Yan, Ngoc Thang Vu. Alan W. Black, and Shinji Watanabe. 2022. ESPnet-SLUS Advancing spoken language un- derstanding through ESPnet. In ICASSP 2022, pages 7177-7171. IEEE.
Xuxin Cheng, Zhihong Zhu, Ziyu Yao, Hongxiang Li, Yaowei Li, and Yuexian Zou. 2023b. GhostT5: Gen- erate More Features with Cheap Operations to Im- prove Textless Spoken Question Answering. In Inter- speech 2023, pages 1134-1138. ISCA.
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, and Furu Wei. 2021. mT66 Multilingual pretrained text-to-text transformer with translation pairs, In EMNLP 2021, pages 1671-1683, ACL.
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguistics, 7:597-610. 610.
Zewen Chi, Li Dong, Furu Wei, Wenhui Wang. Xian- Ling Mao, and Heyan Huang. 2020. Cross-lingual natural language generation via pre-training. AnAAAI 2020, pages 77770-7577.. AAAI Press.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vee 2.0: A framework for self-supervised learning of speech representations. In NeurlPS 2020.
Yung-Sung Chuang, Chi-Liang Liu, Hung-yi Lee, and Lin-Shan Lee. 2020. SpeechBERT: An audio-and text jointly learned language model for end-to-end spoken question answering. In Interspeech 2020, pages 4168-4468 41724 ISCA.
Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. Regularization techniques for fine-tuning in neural machine transla- tion. In EMNLP 2017, pages 1489-1494, ACL.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojan- ski, and Verena Rieser. 2020. SLURP: A spoken lan- guage understanding resource package. In EMNLP 2020, pages 7252-7262 ACL.
Yu-An Chung, Chenguang Zhu, and Michael Zeng. 2021. SPLAT: Speech-language joint pre-training for spoken language understanding. NAACL-HLT 2021, pages 1897-1907. ACL.

---

Alexis Conneau, Alexei Baevski, Ronan Collobert, Ab- deIrahman Mohamed, and Michael Auli. 2021. Un- supervised cross-lingual representation learning for speech recognition. In Interspeech 2021, pages 2426- 2430. ISCA.
Heting Gao, Junrui Ni, Kaizhi Qian, Yang Zhang, Shiyu Chang, and Mark Hasegawa-Johnsse 2022. WayPrompt: Towards few-shot spoken language un- derstanding with frozen language models. In Inter- speech 2022, pages 2738-2742 ISCA.
Alexis Conneau, Ankur Bapna, Yu Zhang, Min Ma. Patrick von Platen, Anton Lozhkov, Colin Cherry, Ye Jia, Clara Rivera, Mihir Kale, Daan van Esch, Vera Axelrod. Simran Khanuja, Jonathan H. Clark, Orhan Firat, Michael Auli, Sebastian Ruder, Jason Riesa, and Melvin Johnson. 2022. XTREME-S: Evaluating cross- looss-lingual speech representations. In Interspeech 2022, pages 3248-3252 ISCA.
Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro J. Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters. 2018. From audio to semantics: Approaches to end-to-end spoken language understanding. In IEEE Spoken Language Technology Workshop, SLT 2018, pages 720-726. LEEE
Chan-Jan Hsu, Ho-Lam Chung, Hung-yi Lee, and Yu Tsao. 2023. TSlephone: Bridging speech and text sulf-supervised models for spoken language un- derstanding via phoneme level T5. In ICASSP 2023, volume abs/2211 1.00586.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning scale. In ACL 2020, pages $440-8451. ACL.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 29:3451-3466
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In NeurIPS 2019, pages 7057-7067.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for Chinese BERT. SEEEACM Trans. Audio Speech Lang. Process., 29:3504-3514.
Zhiqi Huang, Milind Rao, Anirudh Raju, Zhe Zhang. Bach Bui, and Chul Lee. 2022. MTL-SLT: Multi-task learning for spoken language tasks. In 4th Workshop on NLP for Conversational AI, ConvAI ConA/@ACLL 2022. pages 120-130. ACI..
Jacob Devlin, Ming-Was Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT 2019, pages 4171-4186. ACL.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Tho- rat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff' Hughes, and Jeffrey Dean. 2017. Google's 's multilingual neural machine translation sys- tem: Enabling zero-shot translation. Trans. Assoc. Comput. Linguistics, 5:339-351.
Linhao Dong, Zhecheng An, Peihao Wu, Jun Zhang, Lu Lu, and Zejun Ma. 2023, CIF-PT: bridging speech and text representations for spoken language understanding via continuous integrate-and- pre- training. In Findings of ACL 2023, pages 8894-8907. ACL.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics, 8:64-77.
Akiko Eriguchi. Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. CoRR, abs/1809.00 004666.
Mihir Kale and Scott Roy. 2020. Machine translation pre-training for data-to-text generation A case study in Czech. the 13th International Conference on Natural Language Generation, INLG 2020. pages 91 91-96.44 ACL
Jack FitzGerald, Christopher Hench, Charith Peris. Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath. Laurie Crist, Misha Britan, Wouter Leeuwis, Gökhan Tür, and Prem Natara- jan. 2023. MASSIVE: A IM-example multilin- gual natural language understanding dataset with 51 typologically-difg diverse languages. In ACI. 2023, pages 4277-4302. ACI..
Mihir Kale, Aditya Siddhant, Rami Al-Rfou, Linting Xue, Noah Constant, and Melvin Johnson. 2021. I. nmT5 Is parallel data still relevant for pre-training massively multilingual language models? In ACL/JCNLPP 2021, pages 683-691. ACL.
Seonghin Kim, Gyuwan Kim, Seongjin Shin, and Sang- min Lee. 2021. Two-stage textual knowledge distilla- tion for and-to-end spoken language understanding. In ICASSP 2021, pages 1463-7467, IEEE
Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: A multilingual speech translation corpus. In NAACL-HIT 2019, pages 2012-2017. ACL.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015.

---

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- al. 2017. Overcom-
David MacKay, 1992. A practical bayesian frame- work for backpropagation networks. Neural Comput.. 4(3):448472.
nieszka Grabska-Barwinsker ing catastrophic forgetting in neural networks. the National Academy of Sciences, 11((13):551-3322
Wesley J. Maddox Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. 2019. A simple baseline for Bayesian uncertainty in deep learning. In NeurlPS 2019, pages 13132-13143
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR 2020. OpenReview.net.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2020. Zero-shot crosslingual sentence simplification. In EMNLP 2020, pages 5109-5126. ACL.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising aequeenee-o--eequence pre-training for natural language generation, translation, and com- prehension. In ACL 2020, pages 7871-7880. ACL.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Con- textualized word vectors. In NIPS 2017, pages 6294- 6305.
Razvan Pascanu and Yoshua Bengio. 2014. Revisiting natural gradient for deep networks. In ICLR 2014.
Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung- yi Lee. 2018: 8a. Spoken SQuAD: A study of miti- gating the impact of speech recognition errors on listening comprehension. In Interspeech 2018, pages 3459-3463. ISCA.
Adam Poliak, Yonatan Belinkov, James R. Glass, and Benjamin Van Durme. 2018. On the evaluation of semantic phenomena in neural machine translation using natural language inference. NAACL-HLT 2018, pages 513-523. ACL.
Xian Li, Changhan Wang. Yun Tang, Chau Tran, Yuging Tang. Juan Miguel Pino, Alexei Baevski, Alexis Con- neau, and Michael Auli. 2021. Multilingual speech translation from efficient finetuning of pretrained models. In ACI/IJCNIP 2021, pages 827-838. ACL.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brack- man, Christine McLeavey, and Ilya Sutskever. 2023, Robust speech recognition via large-scale weak su- pervision. In ICML 2023, volume 202 of Machine Learning Research, pages 28492-285188 18. PMLR.
Xuhong Li, Yves Grandvalet. and Franck Davoine. 2018b. Explicit inductive bias for transfer learn- ing with convolutional networks. In ICML 2018, volume 80 of Machine Learning Research, pages 2830-2839. PMLR.
Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Rex., 21: 11::1-4
Hank Liao. 2013. Speaker adaptation of context depen- dent deep neural networks. In ICASSP 2013, pages 7947-7951. IEEE
Alessandro Raganato, Raúl Vázquez. Mathias Creutz, and Jörg Tiedemann. 2019. An evaluation of munguage-agnostic atteeetattetion--e based representa- tions machine translation. In the 4th Workshop on Representation Learning for NLP, RepLANLP@ACL 2019, pages 27-32. ACL.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, ACL.
Guan-Ting Lin, Sung Chuang. Ho-Lam Chung, Shu-Wen Yang, Hsuan-Jui Chen, Shuyan Annie Dong. Shang-Wo Wen Li. Abdelrahman Mohamed, Hung-yi Lee, and I. Lin-Shan Lee, 2022. DUAL: Dis- crete spoken unit adaptive learning for textless spo- ken question answering. In Interspeech 2022, pages 5165-5169. ISCA.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP 2016. pages 2383 2392. ACL.
Anirudh Raju, Milind Rao, Gautam Tiwari, Pranav Dheram, Bryan Anderson, Zhe Zhang, Chul Lee, Bach Bui, and Ariya Rastrow, 2022. On joint training with interfaces for spoken language understanding. In Interspeech 2022, pages 1253-1257. ISCA.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726-742.
Machel Reid and Mikel Artetxe. 2022. PARADISE: Exploiting parallel data for multilingual sequence-to- sequence pretraining. In NAACL 2022, pages 800- 810. ACL
Anton Lozhkov. 2022.
Hugging Face: anton-
l/xtreme_s xxsss_ss000m minds14. 4.0
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhard- waj. Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation. In WMT 2018, pages 84-92. ACL.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In EMNLP 2015, pages 379- 389. ACI.

---

Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Dou- glas W. Oard, and Matt Post. 2021. The multilingual tedx corpus for speech recognition and translation. In Interspeech 2021, pages 3655-3659. ISCA.
Raúl Vázquez, Alessandro Raganato, Jörg Tiedemann. and Mathias Creutz. 2019. Multilingual NMT with language-independent attention bridge. In the 4th Workshop on Representation Learning for NLP. RepLANLP@ 2019, pages 33-39. ACL.
Michael Saxon, Samridhi Choudhary, Joseph P. McKenna, and Athanasios Mouchtaris. 2021. End-to- end spoken language understanding for generalized voice assistants. In Interspeech 2021, pages 4738-
Esaú Villatoro-Tello, Srikanth R. Madikeri, Juan Zuluaga-Gomez, Bidisha Sharma, Seyyed Saeed Sar- fjoo, Iuliia Nigmatulina, Petr Motlfcek, Alexei V. Ivanov, and Aravind Ganapathiraju. 2023, Effective- ness of text, acoustic, and lattice-basee representa- tions in spoken language understanding tasks. ICASSP 2023.
4742. ISCA.
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In NAACI-HHII 2019, pages 3795-388555 3805. ACL
Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 202 CoVoST 2 and massively multilingual speech translation. In Interspeech 2021, pages 2247-2251. ISCA.
Holger Schwenk and Matthijs Douze. 2017. Learn- ing joint multilingual sentence representations with neural machine translation. In 2nd Workshop on Rep- resentation Learning for NLP, Rep4NLP@ ACL 2017, pages 157-167. ACL.
Yingzhi Wang, Abdelmoumene Boumadane, and Abdel- wahab Heba. 2021b. A fine-tuned wav2vee 2.0/Hu- BERT benchmark for speech emotion recognition. speaker verification and spoken language understand- ing. CoRR, abs/2111 11.0273..
Seunghyun Seo, Donghyun Kwak, and Bowon Lee. 2022. Integration of pre-trained networks with con- tinuous token interface for end-to-end spoken lan- guage understanding. In ICASSP 2022, pages 7152- 7156. IEEE.
Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng. and Xuedong Huang. 2020. Mixed-lingual pre-traine -rrinininng for gross-lingall summarization. In AACL/JCNNP 2020, pages 536-541. ACL.
Han Zhu, Li Wang, Gaofeng Cheng, Jindong Wang, Pengyuan Zhang, and Yonghong Yan. 2022. Wav2vec-S: Semi-supervised pre-training for low- resource ASR. In Interspeech 2022, pages 4870- 4874. ISCA.
Dmitriy Serdyuk. Yongqiang Wang. Christian Fuegen, Anuj Kumar, Baiyang Liu, and Yoshua Bengio. 2018. Towards end-to-end spoken language understanding. ICASSP 2018, pages 5754-5758. IEEE.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string- bring-based neural MT learn source syntax? In EMNLP 2016, pages 1526-1534. ACI.
Junnan Zhu, Qian Wang, Yining Wang, Zhou, Ji- ajun Zhang, Shaonan Wang. and Chengqing Zong 2019. NCLS: Neural cross-lingual summarization. In EMNLP-IJCNLP 20/9, pages 3052-3062. ACL.
Ravid Shwartz-7iv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and An- drew Gordon Wilson. 2022. Pre-train your loss: Easy Bayesian transfer learning with informative priors. In NewrlPS 2022.
Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Ari, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. 2020. Evaluating the cross- lingull effectiveness of massively multilingual neural chine translation. In AAAI 2020, pages 8854-8861.
AAAI Press.
Sho Takase and Naoaki Okazaki. 2022. Multi-task learn- ing for cross lingull abstractive summarization. In LREC 2022, pages 3008-3016. European Language Resources Association.
Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet Üstün, Marija Stepanovic, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, and Bar- hara Plank. 2021. From masked language model- ing to translation: Non-english auxiliary tasks im- prove zero-shot spoken language understanding. In NAACL-HLLT 2021, pages 2479-2497. ACL.

---

A Bayesian Transfer Learning
perform Taylor expansion for the log likelihood log f() = log Dp) near the parameters after pretraining, namely 0g. which is assumed to be near to the optimum, making V f(6)) = O. Hence,
As the standard machine learning configura- tion, we determine the parameters by optimiz- ing loss with an L2 regularizer, i.c. minimizing O) + all012 for the parameters 0 E R given data D = {(x,)) y)} and the hyperparameter a, in which the cross-entropy loss L corresponds the negative log-likelihood loo p(y(0) of the label upon model outputs. This can be formu- lated as maximum likelihood estimation (MLE) of by maximizing p(g D), which is equal to log logr(D(() + log p(0) log P(D) by Bayes' theo- rem. With constant p(D) and a zero-mean isotropic Gaussian prior (0, o² /) on & with scalar a, the optimization objective corresponds to
log [(0) = log f(0)) + log [(0))(0 0o)
+ 2 (0- 0)) Ro) log (6)) (6 80) +...
log. (00) + 2 0o) Hog f (00) 0o)
(3)
Therefore, through a second-order expansion, p()) is approximated by a Gaussian distribu- tion corresponding to the negation of the quadratic term above, with Og being the mean and the Hes- sian matrix corresponding to the inverse covariance. To estimate the Hessian matrix, we use Bayes' the- orem and take flat prior Dp. forming
log D) x log p(10) + log p(0)
= logg(D|0) + log(N((: (0)
1
x C(D;0)
03
82 log p(0) Dp)
20
Hoog 1 (0)
(2) Hence L2 regularization can be viewed giving an isotropic cero-mean Gaussian prior the model parameters that assigns higher probability to close- to-zero parameters, with a larger a indicating a smaller scalar o². While instead of zero, L2-SPP (Li et al., 2018b) proposes to limit the parameter shift from the pretrained ones during fine-tuning by assigning Gaussian prior N (0p, o²) centered pretrained parameters 6g. which has been found lead to better downstream performance. Nevertheless, is an verlsimmliiiiation of the prior as different parameters are unequal and some parameters are more critical for the performance on the pretraining task than others. The impor- tance of parameter can be represented by poste- rior distribution p()) near Og on the pretraining data Dp that corresponds to the pretraining loss ())::)) x p())) In this way, elastic weight consolidation (EWC) (Kirkpatrick al., 2017) as- signs a Gaussian prior N (0,, o² with diagonal covariance o; according to the estimated posterior distribution (i.e. loss landscape) of 0g on the pre- training task. A parameter 0; with larger impact to the L(Dp; 0) will have sharper p,)) and smaller of 1/(aF.), thus less flexibility in fine- tuning, lower variance in the fine-tuning prior, and higher weight for its L2 regularizer under the goal of preserving knowledge for the pretraining task. To estimate this posterior distribution or loss landscape on the pretraining data D,, we can
I log pl (D))))
(4)
8 log p(x
E px))()))
002
While the Fisher information matrix can be written
as
82 log p(x=(0)
F= -Ez-p(( (0)
(5)
802
Therefore, the posterior distribution of the pa- rameter on the pretraining task is approximated by a Gaussian distribution with the mean μ O and the inverse covariance Σ-1 F. The Fisher matrix can then be estimated by squared gradients as in Pascanu and Bengio (2014), and EWC further simplifies it by only considering diagonal terms.
B Implementation Details
We pretrain the model following the common set- tings in the field on single 24GB V100 GPUs using the Adam optimizer with a learning rate schedule of 20k linear warmup steps from to 1e-4, fol- lowed by an inverse-sqrt decay to 3e-5. Models are selected and early stopping is performed according to the WER or BLEU on the dev set. 5-beam search is used during evaluation. The PTLMs use are the 24-layer "large" versions provided by Hugging Face. A dynamic batching strategy adopted ac- commodate input utterance with different lengths. Accompanied with gradient accumulation, an av- erage batch size of 25 with 500 target tokens

---

SLURP Script
Label
ASR
ST
ST+ASR
is there a meeting on my calendar this afternoon
quurndar,query
calendar,set
calendar.query
calendar,query
look for apple pie recipe can you really see russia from alaska
cooking.recipe qa, factoid
qa,stock alaov
qa,recipe
cooking.recipe general.quirky
qa.factoid
are there morning shows available
calendar,query
weather.query
recommendation,
lists.query
events
Table 8: Examples of SLURP IC benchmark and predictions produced by different models.
per step is used. The wav2vec2 part is frozen for the first 10k steps, and utterances shorter than 0. or longer than 10s are not used during the first 20k steps. The L2 regularization with a=5c-3 is applied to the weights, except the Bayesian trans- fer learning experiments. The setting is similar for the fine-tuning cases except that the encoder is frozen during the initial steps, and for models a 1:3 ratio between data for the pretraining and target task is used. While for smaller datasets including MINDS-14, 14, SLURP-Fr, and Spoken Gi- gawords. the data ratio, dropout rate, and learning rate schedule are further tuned avoid overfitting. We also build and compare with several cascaded pipelines based on our ST model, for which we directly use the model outputs with beam search without external LM as the model already lever- ages strong language model. More details could be found from the source code.
train
dev
test
real
#Samples
11514
2033
2974
477
Avg. sec (Fr)
2.47
2.44
2.44
2.35
Avg. see (Es)
3.03
3.01
3.01
Table 9: Statistics of the SLURP-Fr/Es datasets.
train
dev
test
#Samples
50000
1000
385
Mean length (sec)
9.21
9.30
9.24
Article word count
23.9
24.0
24.0
Headline word count
7.8
8.1
8.0
Table 10: Statistics of the Spoken Gigaword dataset.
speakers to read the held-out samples from MAS- SIVE. The dataset size and mean length (in sec- onds) of the utterances are given in Table 9.
C Examples
As for speech summarization, we follow MTL- SLT (Huang et al., 2022) build the synthetic spoken version of Gigaword (Rush et al., 2015), ing 9 speakers from Google TTS, with total 131.5H audio. We follow the data split in the original Gi- gaword dataset with a small and noisy test split (which we further filtered) and randomly sample from the train and dev split. The resultant size and mean length of the utterances are given in Table 10.
Several examples in the SLURP IC benchmark and the predictions from different models are provided here in Table 8 for more direct demonstration for the understanding capability of the models.
D Dataset Details
Three new datasets are introduced in this work. Among them, SLURP-Fr our main dataset for experiments on cross-lingual transfer, while we ad- ditionally carry out a series of experiments on Span- ish (Es) to show that our methods work on more than one language. For the synthetic portion of SLURP-Fr/ES, we built the dataset based on MAS- SIVE, textual translation of SLURP, each using speakers from Google TTS, with total 11.3H and 13.9H audio respectively. Therefore the contents are identical to MASSIVE. While for the real por- tion of SLURP-Fr, we leverage two native French
E Spanish Experiments
Similar to the default type of models using only data with English and French, we first introduce the Spanish data to the En+Fr ASR model or the Ent+Fr ST model to pretrain En+Fr+ES ASR model well a Fn<+Fr+En<++Es ST model, with satisfactory results as Table 11. Both ASR and ST models are then fine-tuned with joint training on SLURP, reaching 87.63% and 89.59% accuracy respectively. Then they are used for cross- -ingual

---

TEDx
MuST-C
CoVoST2
ASR WER|
15.05%
8.94%
11.34%
ST BLEUT
25.84%
30.06%
33.90%
Table II:
Test results of the model with Spanish data added the cleaned pretraining datasets given by word error rate (WER) for ASR and BLEU score for ST, with Spanish inputs for TEDx and CoVoST2, and English imputs for MuST-C.
550
7.5
10.0
12.5
15.0
17.5
20.0
22.5
22.5
20.0
17.5
15.0
12.5
10.00
-.5
Figure 3: The distribution of the estimated Fisher diag- onals shown in heat map, with x-axis for the means of the log squared gradients of each weight or bias, and y-axis the standard deviation.
transfer to SLURP-ES.
F EWC Weight Distribution
The distributions of the log estimated Fisher di- agonals for each weight matrix bias vector are illustrated in Figure It can observed that most weights are concentrated around le-5 to le-10, and they are close to each other as the standard devi- ations are at the similar magnitude. Hence with a=le-7, most weights will reach the le-2 clamping threshold. The exceptions are the biases for key projection in attention modules, which correspond the lower-left cluster and have much smaller weights.


Reviews:
[{"review_id": "okwvDlzLcE", "reviewer": "Reviewer_LTf6", "summary": "", "questions": "1. What is Es stand for in Table2?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We have tried to make the manuscript more consistent and hypothesis based, which addresses some concerns.  Here we also address some criticisms directly:\n\n* Reasons To Reject:\n\n>1. Limited Generalizability: The efficacy of the proposed ST pretraining method could be limited to specific language pairs.\n\nAdmittedly,  ST data is only available in some language pairs, as mentioned in L108~114 in the manuscript. However, for each covered language, there are a large number of diverse downstream SLU tasks with only rich data in English. Therefore, it is a practical need to enroll various such languages to an English-only system trained on each specific SLU task. This goal of cross-lingual transfer of specific English SLU models to other languages is one of the main motivation of our work. Furthermore, following our hypothesis, ST pretraining could also enhance the results on the source language (En), which is validated by our experiments.\n\nIt is also noteworthy that, in order to show that our method does not only work on French as the target language, we had an additional series of experiments using Spanish (_Es_) as the target: ST/ASR pretraining, training on SLURP (En), and then transfer to the target language with SLURP-Es. Positive results with better accuracy on SLURP (Table 2) and similar strong cross-lingual performance (Table 6) indicate that our methods could be applied to different target languages.\n\n\n>2. Lacking Quantitative Discussion: While the paper presents quantitative results and comparisons, it lacks a comprehensive qualitative discussion on the specific strengths and weaknesses of the proposed ST-pretraining approach. For example, compared to ST, ST+ASR performance degrade across all the dataset, but the authors fail to discuss it.\n\nWe have added discussion to the manuscript to address this concern. In summary, our hypothesis is that ST pretraining endows the model stronger semantic understanding capabilities compared to ASR or acoustic pretraining, while additional multi-tasking with the less semantic-demanding ASR may have possibly negative but limited impact, unless it is favorable for some specific downstream task. This matches the experimental results: ST+ASR leads to performance close to ST in most cases (slightly worse in SLURP but slightly better w/ Joint training, worse than ST in MINDS-14 but the gap is small w/ Joint training, slightly worse in NMSQA, varied in SLURP-Fr), but consistently better than ASR-only models. While for the specific task of speech summarization which is more similar to ASR to produce textual summarization in English, ST+ASR could get the best results. \n\n\n\n>3. Lacking Qualitative Analysis: While the paper demonstrates the superiority of ST-pretrained models over baseline methods, it falls short in providing a detailed quantitative comparison with ASR pretrained and acoustic pretrained models. A more comprehensive analysis directly comparing the performance of ST-pretrained models against ASR pretrained models and acoustic pretrained models would strengthen the paper's claims. For example, providing concrete examples where the ST-pretrained models succeed while other ASR pretrained models fail would be beneficial in highlighting the specific advantages of the proposed approach and addressing potential concerns about the lack of quantitative comparison.\n\nOur core hypothesis is that ST pretraining will lead to better SLU performance in intra-lingual, cross-lingual, and multi-lingual situations, compared to baseline models using ASR pretraining and only self-supervised/acoustic-textual pretraining. Our ST-pretraining approach is directly compared with ASR and self-supervised pretrained models in all our experiments. Similar clarification and emphasis of the results and their implications to our hypothesis have been added to the manuscript.\n\nAll the evaluated benchmarks provide examples of the scenarios where the proposed ST pretraining approach outperforms ASR and self-supervised pretraining, as reflected by the accuracy numbers. While several typical examples from SLURP are shown below and added to the appendix:\n\n\n| SLURP Script                                           | Label          | ASR          |       ST       |     ST+ASR     |\n|--------------------------------------------------|----------------|---------------|:---------------------:|:--------------:|\n| is there a meeting on my calendar this afternoon | calendar,query | calendar,set  |     calendar,query    | calendar,query |\n| look for apple pie recipe                        | cooking,recipe | qa,stock      |       qa,recipe       | cooking,recipe |\n| can you really see russia from alaska            | qa,factoid     | alarm,remove  |       qa,factoid      | general,quirky |\n| are there morning shows available                | calendar,query | weather,query | recommendation,events | lists,query    |\n\n\n\nBelow are some examples from the Spoken Gigaword dataset:\n\n> Article: arsenal manager arsene wenger was charged by the football association (fa) on monday with improper conduct over a reported comment about manchester united striker ruud van nistelrooy.\n> \n> Label: wenger charged with improper conduct\n> \n> ASR: wenger charged over van nistelrooy comments\n> \n> ST: wenger charged with misconduct over manchester united comment\n> \n> ST+ASR: arsenal manager winger charged with improper conduct\n>\n> Article: india's top nuclear expert shrugged off antinuclear protests by indians on hiroshima day, saying the activists should instead shout slogans in washington and moscow, a newspaper reported saturday.\n> \n> Label: top nuclear scientist shrugs off indian antinuclear protests\n> \n> ASR: india says activists should instead shoot slogans in washington moscow\n> \n> ST: india's top nuclear expert calls for anti-nuclear protests\n> \n> ST+ASR: india's top nuclear expert warns against anti-nuclear protests\n>\n> Article: syria and lebanon decided on monday to boost border controls and anti-terrorism coordination, as the two neighbors took a new step to strengthen ties since diplomatic relations were established.\n> \n> Label: syria and lebanon to boost border anti-terror controls\n> \n> ASR: syria lebanon boost border controls\n> \n> ST: syria lebanon to boost border controls\n> \n> ST+ASR: syria lebanon to boost border controls and anti-terrorism coordination\n>\n> Article: austrian women in leading positions complained about lingering male domination in their society in a meeting tuesday with visiting us first lady hillary rodham clinton.\n> \n> Label: austrian women complain to mrs. clinton about male domination by roland prinz\n> \n> ASR: austrian women leaders meet clinton\n> \n> ST: austrian women complain about lingering male domination\n> \n> ST+ASR: austria women complain about lingering male-domination in society\n\n\n\n\n>4. The experiments are not unified across datasets: The paper's lack of a consistent experimental setup across different datasets may introduce confounding factors that could affect the validity of the results. Without standardized experiments, it becomes difficult to draw definitive conclusions and assess the true impact of the ST-pretraining approach on SLU performance. For example, only joint training +Es is applied on SLURP dataset. No joint training applied on Spoken QA.\n\nThank you for pointing out that, and we try to further clarify the point below and in the manuscript: As for the response to point 3 above, the core experiments (ST vs ASR/self-supervised pretraining) are consistent across the datasets. While there are several additional experiments to support our hypothesis in other aspects. For example, an additional series of +Es experiments is used to show that our method works not only in the French language as mentioned in L128. While as mentioned in L361~362, most of the encoder layers are frozen in the Spoken QA experiment to make a fair comparison with the baseline (DUAL) regarding the model size while maintain a model architecture consistent with experiments on other tasks. Therefore, joint training is less meaningful in this situation since most parameters shared between the source task and the target QA task are frozen and will not be trained. \n\n\n>5. Redundant context in the paper: The paper may suffer from redundant context in certain sections. This redundancy can obscure the main contributions or key takeaways. For example, the first paragraph in the introduction introduce the self-supervised learning method for NLP pretrained model, but it's irrelevant to the paper. Also, in section 5, the authors spend lots of space introducing L2-SP and EWC algorithm for pretrained knowledge preservation, but put the results of speech summarization to appendix. They should put the details of L2-SP and EWC to appendix but summarization results in main article instead. Additionally, essential methodologies like Adversarial training in the cross-lingual transfer setting should be elaborated upon in the main article\n\nThis was also raised by other reviewers, and clearly makes more sense; we have updated the manuscript to move details on self-supervised learning on NLP and pretrained knowledge preservation to the related work and appendix, while the results on speech summarization and explanations on adversarial training to the main article. While it is noteworthy that adversarial training is an extra trick to enhance our technique and does not affect our conclusion.\n\n\n* Questions For The Authors:\n\n>1. What is Es stand for in Table2?\n\nEs stands for Spanish. Additional clarification is added to the manuscript.\n\n* Typos Grammar Style And Presentation Improvements:\n>1. Instead of None, maybe trained from scratch is better term in each table.\n\nThank you for the suggestion and we have updated the manuscript accordingly. It should be noted that the _None_ models use acoustic and textual pretraining based on XLSR-53 and mBART, instead of completely training from scratch on the target task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Sound assumption:\nThe paper builds upon a sound assumption by proposing to utilize speech translation (ST) as a pretraining method for speech models in end-to-end spoken language understanding (SLU) tasks. The authors' assumption, supported by evidence, suggests that ST can effectively capture high-level semantic information and cross-lingual associations, making it a promising approach to enhance SLU performance.\n\n2. Strong Empirical Validation:\nThe paper presents thorough experimental validation of the proposed ST-pretraining approach on multiple datasets, including new benchmarks. The reported results consistently show that ST-pretrained models outperform ASR and acoustic pretrained models, indicating the effectiveness of the proposed approach.\n\n3. Insights into Cross-Lingual Transfer:\nThe paper provides valuable insights into the cross-lingual transfer capabilities of the ST-pretrained models, particularly in zero-shot cases. This feature is of great practical importance, showcasing the potential for the proposed method to handle multilingual tasks effectively, even with limited training data in new languages.", "reasons_to_reject": "1. Limited Generalizability: The efficacy of the proposed ST pretraining method could be limited to specific language pairs.\n\n2. Lacking Quantitative  Discussion: While the paper presents quantitative results and comparisons, it lacks a comprehensive qualitative discussion on the specific strengths and weaknesses of the proposed ST-pretraining approach. For example, compared to ST, ST+ASR performance degrade across all the dataset, but the authors fail to discuss it.\n\n3. Lacking Qualitative Analysis: While the paper demonstrates the superiority of ST-pretrained models over baseline methods, it falls short in providing a detailed quantitative comparison with ASR pretrained and acoustic pretrained models. A more comprehensive analysis directly comparing the performance of ST-pretrained models against ASR pretrained models and acoustic pretrained models would strengthen the paper's claims. For example, providing concrete examples where the ST-pretrained models succeed while other ASR pretrained models fail would be beneficial in highlighting the specific advantages of the proposed approach and addressing potential concerns about the lack of quantitative comparison. \n\n4. The experiments are not unified across datasets: The paper's lack of a consistent experimental setup across different datasets may introduce confounding factors that could affect the validity of the results. Without standardized experiments, it becomes difficult to draw definitive conclusions and assess the true impact of the ST-pretraining approach on SLU performance. For example, only joint training +Es is applied on SLURP dataset. No joint training applied on Spoken QA. \n\n5. Redundant context in the paper: The paper may suffer from redundant context in certain sections. This redundancy can obscure the main contributions or key takeaways. For example, the first paragraph in the introduction introduce the self-supervised learning method for NLP pretrained model, but it's irrelevant to the paper. Also, in section 5, the authors spend lots of space introducing L2-SP and EWC algorithm for pretrained knowledge preservation, but put the results of speech summarization to appendix. They should put the details of L2-SP and EWC to appendix but summarization results in main article instead. Additionally, essential methodologies like Adversarial training in the cross-lingual transfer setting should be elaborated upon in the main article\n\n\n\n\n\n", "questions_to_authors": "1. What is Es stand for in Table2?", "paper_topic_and_main_contributions": "The paper introduces a novel approach to pretraining speech models for end-to-end spoken language understanding (SLU) tasks using speech translation (ST). The authors argue that ST enables the models to capture high-level semantic information from input utterances and establish connections between different languages, which is not adequately achieved by pretrained spoken language models that focus on lower-level audio features. ST exhibits the ability to capture long-term dependencies instead of just local context information, making it more suitable for SLU tasks compared to automatic speech recognition (ASR). Additionally, ST demonstrates superior cross-lingual transfer capabilities, even outperforming multilingual ASR in zero-shot cases.\n\nTo validate their hypothesis, the authors conduct experiments on five different datasets, including two newly proposed datasets. The results demonstrate that introducing ST as a pretraining method leads to improved performance compared to baseline approaches and ASR-pretrained models. Though the results vary across datasets, the authors showcase the value of preserving ST-pretrained knowledge through Bayesian regularization, particularly under low-resource conditions.\n\nIn summary, the paper presents a promising approach to enhancing SLU performance by leveraging ST as a pretraining technique for speech models. The experiments on various datasets provide strong evidence in favor of the proposed method's effectiveness. The findings offer valuable insights into advancing SLU tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Instead of None, maybe trained from scratch is better term in each table."}, "extra_text": {}}, {"review_id": "NUKcoaox1w", "reviewer": "Reviewer_bU2d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We have tried to make the manuscript more consistent and hypothesis based, which addresses some concerns.  Here we also address some criticisms directly:\n\n* Reasons To Reject:\n\n> Some experimental settings, analysis and comparisons are not always clearly explained, and could benefit from some reorganization. For instance, in Table 4, the DUAL approach and the approach in the paper are using different types spans (document/segment), thus seem hardly directly comparable. Also, the caption claims that ST pretraining is outperforming the cascaded system, but it is not always the case (on dev).\n\nThank you for your comments and we have tried to provide more discussions in the manuscript. To summarize, our core hypothesis is that ST pretraining could introduce stronger semantic understanding capabilities to the model compared to ASR and self-supervised pretraining. Therefore all our experiments compare these 3 types of pretraining across different tasks, which show consistent results supporting our hypothesis. \n\n\n\nSpecifically, the baseline DUAL model enjoys an advantage of the view on the whole document, while the scope of our model is limited to each segment in order to have an end-to-end architecture consistent with other experiments, different from DUAL using pre-extracted features from HuBERT on audio chunks. Hence we have a strong prior that the comparison is actually unfair to our model, and the surprisingly better results of ours further demonstrate the strong advantage of our model. We use the AOS results reported in the original DUAL paper with the document-level view. Unfortunately we are unable to fully reproduce DUAL at the moment using their code, or to further adapt the DUAL model to segment-level span and to report the performance of DUAL with segment-level view only.\n\n\n\nWe acknowledge that ST pretraining has a close but not better results compared to the cascaded model (58.2% vs 58.3%), and we have updated the manuscript to clarify that. However it is noteworthy that in many previous papers only _test_ results on NMSQA are reported, since the test split is more challenging with real audios from multiple speakers. Therefore our better results on the test split are more meaningful.\n\n\n> The writing and presentation are not always easy to follow; the work may benefit from some better organization of the presentation flow. For instance, the investigation of speech summarization seems quite relevant to the main contributions described in the paper.\n\nThis was also raised by other reviewers, and clearly makes more sense; we have updated the manuscript to move details on self-supervised learning on NLP and pretrained knowledge preservation to the related work and appendix, while the results on speech summarization to the main article. Also we further emphasize our hypotheses and the empirical evidence we use to support them.\n\n* Typos Grammar Style And Presentation Improvements:\n\n>Grammar:\n>l.183: \"we've also attempt\" -> \"attempted\"\nStyle:\n>l.172, the sentence starting with 'While...' is not clearly connected to the sentence it's supposed to contrast with.\n\nWe have fixed them in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach using ST for pretraining SLU tasks is well-motivated and the motivations are clearly presented. \n\nThe work tested the pretrained model with several downstream tasks, the preservation of pretrained knowledge and created synthetic datasets for two tasks.\n\nThe authors also stated that the code and models will be open-sourced.", "reasons_to_reject": "Some experimental settings, analysis and comparisons are not always clearly explained, and could benefit from some reorganization. For instance, in Table 4, the DUAL approach and the approach in the paper are using different types spans (document/segment), thus seem hardly directly comparable. Also, the caption claims that ST pretraining is outperforming the cascaded system, but it is not always the case (on dev).\n\nThe writing and presentation are not always easy to follow; the work may benefit from some better organization of the presentation flow. For instance, the investigation of speech summarization seems quite relevant to the main contributions described in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to use speech translation for pretraining E2E spoken language understanding models for both intra and cross-lingual scenarios.\n\nThe approach improved the performance on a series tasks and benchmarks, and the paper also proposes new benchmark datasets for \tspeech summarization and low/zero-resource cross-lingual transfer.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Grammar:\n- l.183: \"we've also attempt\" -> \"attempted\"\n\nStyle:\n- l.172, the sentence starting with 'While...' is not clearly connected to the sentence it's supposed to contrast with."}, "extra_text": {}}, {"review_id": "7P9TvJNJnf", "reviewer": "Reviewer_1hZ9", "summary": "", "questions": "1. Could you explain why whisper is not suitable for fine-tuning on SLURP?\n2. Could you provide the statistics of your synthetic datasets?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We have tried to make the manuscript more consistent and hypothesis based, which addresses some concerns. Here we also address some criticisms directly:\n\n* Reasons To Reject:\n\n>1. For each task, the paper only provides the experimental results of one dataset. For example, the paper could provide the results on ATIS and TREC6 for English SLU as in [1].\n\nWe choose SLURP as the benchmark for evaluation on the English intent classification (IC) task, since SLURP is the only recently-proposed benchmark on the standard IC task that remains challenging, while the state-of-the-art in other previous well-known datasets like ATIS, SNIPS, and FSC has long reached 99% accuracy, making the empirical comparison largely random. To simplify the problem and to emphasize the evaluation of the pretrained encoder's semantic understanding capability, we largely focus on the classification task without the need of building a task-specific decoder.  Since we work on end-to-end SLU from audio, benchmarks like TREC6 with only textual inputs are not suitable.\n\nIt is also noteworthy that we perform experiments on English IC (SLURP), multi-lingual IC (MINDS-14) and cross-lingual IC (SLURP-Fr/Es), and MINDS-14 contains English portion. Therefore, as for the IC task, various situations and aspects have been investigated using multiple datasets.\n\n>2. The paper would be better if it provided some cases to show the effectiveness of your method.\n\nThe effectiveness of our methods has been largely demonstrated by the accuracy number. While several typical examples from SLURP are shown below and added to the appendix:\n\n| SLURP Script                                     | Label          | ASR           |          ST           |     ST+ASR     |\n| ------------------------------------------------ | -------------- | ------------- | :-------------------: | :------------: |\n| is there a meeting on my calendar this afternoon | calendar,query | calendar,set  |    calendar,query     | calendar,query |\n| look for apple pie recipe                        | cooking,recipe | qa,stock      |       qa,recipe       | cooking,recipe |\n| can you really see russia from alaska            | qa,factoid     | alarm,remove  |      qa,factoid       | general,quirky |\n| are there morning shows available                | calendar,query | weather,query | recommendation,events |  lists,query   |\n\n\n\nBelow are some examples from the Spoken Gigaword dataset:\n\n> Article: arsenal manager arsene wenger was charged by the football association (fa) on monday with improper conduct over a reported comment about manchester united striker ruud van nistelrooy.\n> \n> Label: wenger charged with improper conduct\n> \n> ASR: wenger charged over van nistelrooy comments\n> \n> ST: wenger charged with misconduct over manchester united comment\n> \n> ST+ASR: arsenal manager winger charged with improper conduct\n>\n> Article: india's top nuclear expert shrugged off antinuclear protests by indians on hiroshima day, saying the activists should instead shout slogans in washington and moscow, a newspaper reported saturday.\n> \n> Label: top nuclear scientist shrugs off indian antinuclear protests\n> \n> ASR: india says activists should instead shoot slogans in washington moscow\n> \n> ST: india's top nuclear expert calls for anti-nuclear protests\n> \n> ST+ASR: india's top nuclear expert warns against anti-nuclear protests\n>\n> Article: syria and lebanon decided on monday to boost border controls and anti-terrorism coordination, as the two neighbors took a new step to strengthen ties since diplomatic relations were established.\n> \n> Label: syria and lebanon to boost border anti-terror controls\n> \n> ASR: syria lebanon boost border controls\n> \n> ST: syria lebanon to boost border controls\n> \n> ST+ASR: syria lebanon to boost border controls and anti-terrorism coordination\n>\n> Article: austrian women in leading positions complained about lingering male domination in their society in a meeting tuesday with visiting us first lady hillary rodham clinton.\n> \n> Label: austrian women complain to mrs. clinton about male domination by roland prinz\n> \n> ASR: austrian women leaders meet clinton\n> \n> ST: austrian women complain about lingering male domination\n> \n> ST+ASR: austria women complain about lingering male-domination in society\n\n\n\n>3. It is a little unprofessional to put all the experimental results of speech summarization in the appendix.\n\nThis was also raised by other reviewers; we have updated the manuscript to move details on self-supervised learning on NLP and pretrained knowledge preservation to the related work section and the appendix, while the results on speech summarization to the main article. \n\n* Questions For The Authors:\n\n>1. Could you explain why whisper is not suitable for fine-tuning on SLURP?\n\nIt is very difficult to determine the reason why a baseline model is not working well, but as for Whisper there are several possible reasons. \n\n1. Whisper is trained on En ASR and X->En ST but not for En->X ST. Our hypothesis is that the ST pretraining on a specific language would enhance the semantic understanding capabilities of the model in that language, which may not help Whisper much on the English SLURP benchmark. \n2. Whisper is trained on 30s chunks. When used to transcribe relatively short segments, Whisper often produces transcription with significant error, or simply hallucinates. While the SLU datasets we are using contain mostly short (<10s) utterances, hence this domain shift may affect Whisper's performance. The situation could be partially alleviated by padding all of them to 30s, but it would lead to a significant waste of computational power that we are unable to afford.\n\nWe nevertheless report the results of the Whisper-based model as it is an important concurrent pretraining work.\n\n\n\n>2. Could you provide the statistics of your synthetic datasets?\n\nThank you for pointing out this missing part in our manuscript, and we have added the explanation and statistics of our new datasets below.\n\nSLURP-Fr is our main dataset for experiments on cross-lingual transfer, while we additionally carry out a series of experiments on Spanish (Es) to show that our methods work on more than one language.  For the synthetic portion of SLURP-Fr/Es, we built the dataset based on MASSIVE, textual translation of SLURP, each using 4 speakers from Google TTS, with total 11.3H and 13.9H audio respectively. Therefore the contents are identical to MASSIVE. While for the real portion of SLURP-Fr, we leverage two native French speakers to read the held-out samples from MASSIVE. The dataset size and mean length of the utterances are given below:\n\n\n|                        | train |  dev | test | real |\n|------------------------|-------|:----:|:----:|:----:|\n| #Samples               | 11514 | 2033 | 2974 |  477 |\n| Mean length (sec) - Fr | 2.47  | 2.44 | 2.44 | 2.35 |\n| Mean length (sec) - Es | 3.03  | 3.01 | 3.01 |   /  |\n\n\n\n\nAs for speech summarization, we follow [MTL-SLT](https://aclanthology.org/2022.nlp4convai-1.11/) to build the synthetic spoken version of [Gigaword](https://huggingface.co/datasets/gigaword), using 9 speakers from Google TTS, with total 131.5H audio. We follow the data split in the original Gigaword dataset with a small and noisy test split (which we further filtered) and randomly sample from the train and dev split. The resultant size and mean length of the utterances are given below:\n\n\n|                        | train |  dev | test |\n|------------------------|-------|:----:|:----:|\n| #Samples               | 50000 | 1000 |  385 |\n| Mean length (sec) | 9.21  | 9.30 | 9.24 |\n| Article word count      | 23.9  | 24.0 | 24.0 |\n| Headline word count      | 7.8   | 8.1  | 8.0  |\n\n\n\n\n* Missing References:\n\n>[1] Chang Y H, Chen Y N. Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding[J]. arXiv preprint arXiv:2205.00693, 2022. [2] Hsu C J, Chung H L, Lee H, et al. T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5[J]. arXiv preprint arXiv:2211.00586, 2022.\n\nThank you for the suggestions and we have added the references accordingly.\n\n\n\n* Typos Grammar Style And Presentation Improvements:\n\n> In line 227, \"with ∼600 utterances\"\n\nWe have fixed them in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides the experimental results on a large number of downstream tasks.\n2. The method achieves consistent performance improvement on these downstream tasks.\n3. The paper builds some new datasets, which may be helpful to the community.", "reasons_to_reject": "1. For each task, the paper only provides the experimental results of one dataset. For example, the paper could provide the results on ATIS and TREC6 for English SLU as in [1]. \n2. The paper would be better if it provided some cases to show the effectiveness of your method.\n3. It is a little unprofessional to put all the experimental results of speech summarization in the appendix.\n\n[1] Chang Y H, Chen Y N. Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding[J]. arXiv preprint arXiv:2205.00693, 2022.", "questions_to_authors": "1. Could you explain why whisper is not suitable for fine-tuning on SLURP?\n2. Could you provide the statistics of your synthetic datasets?", "paper_topic_and_main_contributions": "1. The paper demonstrates the effectiveness of speech translation pretraining on multiple SLU tasks, especially in cross-lingual transfer cases.\n2. The paper confirms the value of preserving ST pre-training knowledge for downstream tasks and the capability of Bayesian regularizers to achieve that.\n3. The paper builds several new datasets for speech summarization and cross-lingual SLU.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 227, \"with ∼600 utterances\""}, "extra_text": {"Missing_References": "[1] Chang Y H, Chen Y N. Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding[J]. arXiv preprint arXiv:2205.00693, 2022.\n[2] Hsu C J, Chung H L, Lee H, et al. T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5[J]. arXiv preprint arXiv:2211.00586, 2022."}}]

Author rebuttal:

Meta-review: The paper presents a new approach to using speech translation (where data is relatively abundant) for pre-training E2E spoken language understanding models for both intra and cross-lingual scenarios. While not a revolutionary novel idea, the approach improves performance on a series of existing tasks and benchmarks (setting SOTA performance levels on some), and the paper also proposes new benchmark datasets for speech summarization and low/zero-resource cross-lingual transfer. Authors also commit to releasing the code and models as open-sourced, which will create an interesting and relevant test-bed for future development. Authors responded well to reviewers (whose main criticism is the organization of the paper, and the presentation of results, sometimes asking for more detail, which the authors generally provide in the response), and commit to incorporating feedback and improve readability, which I think may even allow including the approach in the main conference, certainly in Findings.
Decision: Accept-Findings