Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis
Authors: Jianqiao Lu
Wenyong Huang
Nianzu Zheng
Xingshan Zeng
Yu Ting Yeung
Xiao Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: joint speech-text learning
spoken language understanding
speech recognition
TL;DR: a joint speech-text learning approach for E2E spoken language understanding and speech recognition

Abstract: Training a high performance end-to-end speech (E2E) processing model requires an enormous amount of labeled speech data, especially in the era of data-centric artificial intelligence. However, labeled speech data are usually scarcer and more expensive for collection, compared to textual data.
We propose Latent Synthesis (LaSyn), an efficient textual data utilization framework for E2E speech processing models.
We train a latent synthesizer to convert textual data into an intermediate latent representation of a pre-trained speech model. These pseudo acoustic representations of textual data augment acoustic data for model training.
We evaluate LaSyn on low-resource automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. 
For ASR, LaSyn improves an E2E baseline trained on LibriSpeech train-clean-100, with relative word error rate reductions over 22.3\% on different test sets. 
For SLU, LaSyn improves our E2E baseline by absolute  4.1\% for intent classification accuracy and 3.8\% for slot filling SLU-F1 on SLURP, and absolute 4.49\% and 2.25\% for exact match (EM) and EM-Tree accuracies on STOP respectively.
With fewer parameters, the results of LaSyn are competitive to published state-of-the-art works. 
The results demonstrate the quality of the augmented training data.

Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis
Jianqiao Lu Wenyong Huang Nianzu Zheng", Xingshan Zeng', Yu Ting Yeung² & Xiao Chen
'The University of Hong Kong HHuawei Noah's Ark Lab jqluêcs, hku. hk, wenyong. wwwnghhuawei.ccoommm com
Abstract
text with ASR, then pipes the text output the nat- ural language understanding (NLU) model for text analysis. An end-to-end (E2E) speech processing system leverages a single model which takes the in- put speech and performs spoken language process- ing tasks simultaneously. E2E models draw increas- ing attention due to less computational complexity and error propagation mitigation (Shen et al., 2021;
Training a high performance end-to-end speech (E2E) processing model requires an enormous amount of labeled speech data, especially in the era of data-centric artificial intelligence. How- ever, labeled speech data are usually scarcer and more expensive for collection, compared textual data. We propose Latent Synthesis (LaSyn), an efficient textual data utilization framework for E2E speech processing models. We train latent synthesizer convert textual data into an intermediate latent representation of pre-trained speech model. These pseudo acoustic representations of textual data aug- ment acoustic data for model training. We eval- uate LaSyn on low-resource automatic speech recognition (ASR) and spoken language under- standing (SLU) tasks. For ASR, LaSyn im- proves F2E baseline trained LibriSpeech train-clean- 100, with relative word error rate reductions over 22.3% on different test sets. E2E baseline
Tian and Gorinski, 2020; Sharma et al., 2021; Lu- gosch al., 2020; Wang et al., 2020; Chen et al., 2021b), However, a challenge E2E model train- ing the collection enormous annotated spoken data, which are significantly more expensive to collect compared with the text-only counterpart. In contrast, for a cascaded system, the ASR mod- ule and NLP module trained separately with paired spesch-transcription data and annotated tex- tual data respectively. Separated types of data are usually more readily available and thus lower data collection costs. As the amount of high quality training data is critical for an E2E model, a strategy to alleviate the inadequate spoken data problem with more abundant textual data.
For SLU, LaSyn improves by absolute 4. for intent classification ac- curacy and 3.8% for slot filling SLU-FI on SLURP, and absolute 4,49% and 2.25% for ex- act match (EM) and EM-Tree accuracies on STOP respectively. With fewer parameters, the results of LaSyn are competitive to published sfatc-of-the-ar works. The results demonstrate the quality of the augmented training data.
Two approaches have been proposed for utilizing textual data for E2F. speech models in the literature. The first is modality conversion which utilizes text-to-speech (TTS) system to convert text into speech (Laptev et al., 2020). The disadvantage is the requirement for high-quality expressive TTS system. Another approach unified representa- tion learning for matching latent representations of speech and text with alignment losses (Bapna et al., 2021; Chen et al., 2022a). Given the signifi- cant difference between speech and text, aligning the hidden latent space of the two modalities is challenging-
1 Introduction
In the data-centric artificial intelligence era, large quantity and high quality training data essen- tial for good performance of natural language pro- cessing (NL.) models including speech processing models. A conventional speech processing system is usually cascaded with an automatic speech recog- nition (ASR) module and an NLP module. For ex- ample, in spoken language understanding (SLU) which predicts semantic information from speech input, the system first transcribes input speech into
We propose Latent Synthesis (LaSyn), method to utilize text-only data for E22 speech processing models. LaSyn can be seen as an integration of the above two ideas. We train a latent synthesis model which synthesizes textual data into an intermediate
"Leading co-authors with equal contribution.
Work done during an intership at Huawei.

---

latent representation of pre-trained speech model. Compared modality conversion, speech latent representation contains fewer details and redun- dancy than the original speech signal, thus is easier to synthesize. Compared to unified representation learning, instcad of aligning two modalities of huge difference, LaSyn learns to map the text into the latent representation of speech directly.
This paper is organized as follows. In the next section, we discuss related works of LaSyn. In Section 3, we discuss the model structure and train- ing of LaSyn. We present experimental setup and results in Section 4, and ablation studies on SLU tasks in Section 5. Finally, we conclude our work in Section 6.
Related Works
We evaluate LaSyn low-resource ASR and SLU tasks. Low-resource ASR has gained big progress with the advancement of cupessuee speech pre- pre-training (Bacvski et al., 2020; Hsu et al., 2021; Huang et al., 2022). Further performance im- provement still relies on external language models (Baevski et al., 2020). We show that LaSyn al- lows an E2E ASR model to utilize text-only data effectively without external language models, and outperforms ASR models with external language models. We further evaluate LaSyn on two publicly available datasets for SLU tasks, namely SLURP (Bastianelli et al., 2020) and Spoken Task Oriented Semantic Parsing (STOP) (Tomasello et al., 2022). LaSyn achieves comparable performance to the otate-of-the-art (SOTA) SLU models but with sig- nificantly fewer model parameters. We summarize contributions as follows:
In this section, we discuss the prior works of modal- ity conversion and unified representation learning related to LaSyn.
Modality conversion: that TTS data augmentation improves ASR per- formance in a low-resource setting. Sun et al. (2020) further shows that the diversity and qual- ity of the TTS system are important for ASR data augmentation. Chen et al. (2022b) demonstrates similar representations derived from synthesized speech help downstream ASR tasks. Lugosch al. (2020) confirms the effectiveness of speech synthe- sis for E2E SLU models, either as a sole source of training data or as a form of data augmentation. Thomas et al. (2021) utilizes artificially synthe- sized speech to adapt SLU model based on a recurrent neural network transducer. Huang et al. (2020b) demonstrates the effectiveness of multi- speaker TTS system under low-resource SLU setting. Kharitonov et al. (2023) decouples the text- to-semantic and comantic-to-acouse tasks to real- ize a multi-speakerr text-to-speech system. LaSyn generates pseudo acoustic representations from text without requiring vocoder for speech waveform generation.
Laptev et al. (2020) shows
We propose LaSyn, an efficient textual data uti- lization framework for E2E speech processing models. The framework enables cross-modal knowledge transfer from text to F2E speech pro- cessing models through latent synthesis.
We design implementations for latent synthe- sizer which is the core of LaSyn framework: a fixed-projection latent synthesizer, and a dif- fusion latent synthesizer which applies recent progress of generative model, diffusion proba- bilistic model (Ho et al., 2020; Song et al., 2020).
Unified representation learning: Ao et al. (2021) extends the idea T5 poses Speech-T5 with cross-modal vector quan- tization in a shared discrete latent space. Kim et al. (2021) learns multi-modal alignment with two cross-modal pre-training tasks of masked lan- guage modeling and conditioned language model- ing. Qian et al. (2021) unifies a pre-trained ASR en- coder for speech and pre-trained language model encoder for text into a transformer decoder. Sato et al. (2022) introduces adaptation branch to embed acoustic and linguistic information in the same latent space. Thomas et al. (2022) trains an RNN-T model both on speech and text inputs. Zhang et al. (2022a) introduces two alternative dis- crete phoneme-une unit and coidenn-uni tokenizers to
T5 (Raffel et al., 2020) and pro-
By improving an E2E ASR model through tex- tual data utilization with LaSyn, we achieve com- petitive results on a low-resource ASR setup than published supervised ASR models which utilize textual data through an external language model.
With LaSyn, we demonstrate E2E SLU models can be improved with diverse set of textual NLP tasks, including NLU, information extrac- tion (IE), named entity recognition (NER), and masked language modeling (MLM). We achieve competitive results to published SOTA works on two publicly available SLU datasets, with signifi- cantly fewer model parameters.

---

Tabis
Decoler
Modkl
Layas
Speech Latent Encoder
Lalent Synthesarr
Layess
be windy
lomormw?
Figure I: The architecture of LaSyn framework.
Figure 2: Speech Latent Encoder and Guiding Net from a pre-trained ASR model.
bridge speech and text modalities. MAESTRO (Chen et al., 2022a) learns unified representations text and speech through sequence matching and duration prediction. Chung et al. 2) attempts to align the individually learned text and speech em- bedding via adversarial training and refinement procedure. SpeechUT (Zhang et al., 2022b) lever- ages hidden units as the bridge between the speech encoder and the text decoder. SpeechGPT(Zhang al., 2023) upplies modality-adapttate pertaining and cross-modal instruction fine-tuning to perceive and generate text and speech information by mapping text rep- resentation directly into the pseudo acoustic latent space of pre-trained speech model.
3.2.2 Latent Synthesizer
We then train latent synthesizer to project textual data into same speech latent space of the speech latent encoder. Latent synthesizer allows utilizing training samples from textual data, which the core of the LaSyn framework. We explore two implementations of the latent synthesizer.
ASR LO8S
content. LaSyn connects
Guiding Not.
Caradiestt
Fbuu--Prfection
Lakent Syathosto
3 Method
3.1 Architecture
G2P
The LaSyn framework is illustrated in Fig. 1. The framework has 3 components: speech latent en- coder which maps speech data to corresponding speech latent representation, latent synthesizer that projects text into the speech latent space, and backbone model which trained with either speech latent representations or pseudo acoustic latent rep- resentations from text.
Will be windy botorrow!
Figure 3: Training process Fixed- Sreed-Projection Latent Synthesizer. We freeze parameters of Guiding Net.
projeepoooetiion Latent Synthesizer: train a fixed-projection latent synthesizer with the help of a guiding net. The guiding net is also obtained from the pre-trained ASR model as illustrated in Fig. 2. Note that the guiding net is frozen in this stage. The training procedure is illustrated in Fig. 3. We optimize fixed-projection latent synthe- sizer to generate latent representations which are recognizable as input of the guiding net. As the name suggests, the fixed- suod-proecction latent synthe- sizer learns a fixed one-to-one projection between
3.2
Training procedure
3.2.1
Speech Latent Encoder
Speech latent encoder is obtained from pre- trained speech processing model, which is a pervised ASR model as illustrated in Fig. this work. The parameters of speech latent encoder are frozen the latter training stages to fix the speech latent space.

---

text data and speech latent representation. The training objective is defined as follows,
NSE
Text Encoder
UNet
argmin LASR
Go Gc(Po((2P()))
(I)
Laker
Encoder
where Go and Pa represent the guiding network and the fixed- projepproeetionn latent synthesizer respec- tively, Φ represents the parameters of the latent syn- thesizer, is the text input, and G2P is grapheme- to-phoneme module. Casn is the same loss func- tion of the pre-trained ASR model, such as trans- ducer loss (Graves, 2012) or cross-entropy loss for attention-based encoder-decoder (AED) (Vaswani et al., 2017).
soludl tomorrow?"
a-P(p) e-M(01))
Figure 4: Diffusion Latent Synthesizer training. The gray color indicates that the Speech Latent Encoder and Autoencoder frozen during training.
distribution, X(o) is the weighting factor of noise levels. We employ classifier-free diffusion guid- ance (Ho and Salimans, 2022) to control latent quality and text alignment when sampling from the diffusion latent synthesizer.
Diffusion Latent Synthesizer: We also experi- ment with diffusion probabilistic models (DPM) (Ho et al., 2020) as the latent synthesizer. DPMs have achieved great success in TTS (Popov et al., 2021; Chen et al., 2021a) and lext-conditioned im- age synthesis (Nichol et al., 2021; Saharia al., 2022) recently. We use the formulation of DPM proposed Karras et (2022). Diffusion latent synthesizer generates latent representations by sam- pling an initial latent representation from a noise distribution and iteratively denoising the sample using denoising model (hhnoisy o. where Anoisy represents the noisy latent at the current step, denotes the conditional text. The denois- ing model is composed of UNet (Ronneberger al., 2015) and a text encoder as shown in Fig. To reduce the complexity of the diffusion model, we train an autoencoder to compress the latent rep- resentation and use the lower-dimensional latent representation as the target of the diffusion latent synthesizer, similar to Rombach et al. (2022). For succinctness, we do not depict the training of au- toencoder in Fig. 4. The training objective is to minimize,
3.2.3 Backbone Model and Dual-modality Training
After we train the latent synthesizer, we train the backbone model. We freeze the speech latent en- coder and the latent synthesizer during backbone model training. We utilize both text and speech data in training. The backbone model takes input latent features from either speech latent encoder latent synthesizer. We formulate both text-to-text and speech-to-text tasks as unified sequence-to- sequence problem and refer to as dual-modality training. The training loss specific to each task, i.e., transducer loss for ASR, and cross-entropy loss for SLU. The amount of textual data is usually significantly larger than speech data. We first train the backbone model with textual data. Then we train the backbone model with both text and speech data.
4
Experiments
4.1 Training Data
E p((h)))))))
(oo)l + oe;e,o) -1112
4.1.1 ASR
(2)
We apply a 100-hour subset (train-clean-Iu) 1000 of LibriSpeech (Panayotov et al., 2015) as low- resource labeled speech data. We use the transcrip- tion of the whole 960-hour LibriSpeech training split (LS-960) text-only data.
where h is clean latent representation, p(h,ee e) rep- resents the training data distribution of latent-text pairs. The latent-text pairs are derived from paired speech-text dataset and speech latent en- coder which converts the speeches into latent rep- resentations. p(a) the distribution of noise levels that defines the corruption schedule (Karras et al., 2022). p(e) E A(, 1) is the standard normal
4.1.2 SLU
We evaluate LaSyn on two challenging SLU datasets, SLURP (Bastianelli et al., 2020) and

---

Task
Dataset
windy tateocnewt
CLINC150 (Lumsn 2019)
Redwood (Larson and Leach, 2022)
Hackbooe
Maxeel
GOOGLE-DSTCS (Rastogi al., 2020)
Leyzer (Sowafiski and Janicki, 2020)
HINT3 (Arom al., 2020)
Ladoed
NLU
Charbot-Corpus (Bramm al., 2017)
atllnndw
MultiWOZ. (Zang 2020)
BANKING77 (Casanecva 2020)
W.RRI (MA PrO IMASKI
LIAUMRAT RETRS
FEWSHCTWOZ (Pong 2020)
Mask
ATIS 2010)
Schema (Rastogi et al., 2019)
CrossNER (Liu et 2020)
WNUT17 (Derezynski ot al, 2017)
NER
CoNLL-2003 (Tjong Kim Sang De Moukder, 2003)
Figure MLM utilizing unlabeled text data. [MASK] denotes the masked position.
CoNLL-2004 (Careeuus and Marquez,
OntoNotes (Weischedel et al., 2013)
IE
SCIERC (Luan al., 2018)
4.2 Model and Training Setups
Table I: Extra NLP datasets for SLU experiments.
4.2.1 ASR
For ASR pre-training, we use Transformer Trans- ducer model (Tian et al., 2019; Yeh et al., 2019; Zhang et al., 2020). We apply 128ddimensional log-mel filterbank with 20 ms window length and 10 ms frame rate as input acoustic feature. We interleave strided-convolttion in the encoder to gradually down-sample the input speech as illus- trated in Fig. 5, which reduces computation ef- fectively with negligible performance degradation (Peddinti al., 2018; Han al., 2020; Huang al., 2020a). This model pre-trained with train-clean 100. SpecAugment (Park et al., 2020) is applied to avoid overfitting. This pre-trained model is also our E2E ASR baseline. We obtain a speech latent encoder from this pre-trained model.
Channel multiplier
1,1,1,1
Dropout
0.1
Number of channels
256
Number of residual blocks
1
Self attention resolutions
[4,
Table 2: Hyper-parameters UNet model
STOP (Tomasello al., 2022). SLURP is substan- tially larger and linguistically more diverse than previous SLU datasets. STOP a recently released dataset that is the largest and the most complex SLU dataset. We also leverage a diverse set of NLP text datasets from different tasks, including natu- ral language understanding (NLU), named entity recognition (NER), and information extraction (IE). The extra NLP text datasets listed in Table 1.
For latent synthesizers, we evaluate both fixed- projection latent synthesizer and diffusion latent synthesizer. The crejjproooction latent synthesizer is composed 1-D convolutional layers of 5 filters with a kernel size of 5, We observe that a sim- ple model structure is sufficient. We train the dif- fusion latent synthesizer with train-clean- 100. The text encoder is composed of two convolution layers followed by the two-layer transformer. The number of channels 256. The UNet model is adapted for 1-D sequence processing. The hyper-parameters of the UNet model are listed in Table We use a small model such that the latent synthesizer gener- ates the pseudo acoustic latent representations on the fly during dual-modality training.
Troasforaur Layors
Selwumpling Canvalation
Loe Prome att
Transtoraer Layees
Frush tave
Selssmpling Cancolation
The backbone model is the same the guid- ing net in Fig. 2. To utilize textual data in dual- modality training the backbone model, we de- sign a task similar to masked language modeling
Figure 5: Encoder architecture of the ASR model. The frame rate input denoted as '10/40/80 ms'.

---

(MLM) (Devlin al., 2018) as illustrated in Fig. 6. We randomly mask 30% of imput phonemes con- verted by g2pE' according to CMUDicr', and train the backbone model to predict the corresponding words.
NLU Heshhepprng
indett peDooo lid marse sivrgeing
NER people
Iad tie bas emooel
Mede
We note that the parameters of the guiding net are frozen in latent synthesizer training. If we do not provide textual data for backbone model train- ing, just update the E2E baseline with extra epochs with frozen speech latent encoder.
grgpt optrrcatti preeaeee srepiiis problizea
methedd rale opterinian tank sooce and grpbicc problens
Intestrrsce quaSS
Figure 7: Dual-modality training for SLU with LaSyn. The output labels of different tasks are converted text sequences as shown in the right blocks. Meta values such as slot type and entry type are in bold.
4.2.2 SLU
We apply an attention-based encoder-decoder model for ASR pre-training. The pre-trained ASR model trained with LS-960 and SLURP speech data. The structure of the encoder is similar to the one in ASR experiments described in section 4.1.2. We apply 6-layer, 256-dimensional Transformer the decoder. We evaluate the implementa- tions of the latent synthesizer similar to ASR ex- periments. For exed-projection latent synthesizer, the configuration the same as ASR experiments. We apply text transcription of LS-960 for training. For difffuson latent synthesizer, we use LS-960 paired speech-text training data. The backbone model shares the same model structure as the guid- ing net in Fig. 2. We also initialize the parame- ters from the guiding net. We train the backbone model with multiple tasks, including SLU, NLU, NER, and IE. We convert the annotation of all the datascts to a text-sequence format as illustrated in Fig. 7. We formulate all the tasks as a unified -o-euneee----quecce problem.
test
Model
LM
clean
other
Hybrid DNN/HMM (Litsher et 201,
4-gram LSTM
LAS (Puk
5.5
16.9
Conforee-CCCC (Watanabe et 2022)
7.7
20.6
Conformer- CaCaae-CCAAAAtttiiiiio (Witarahe M00)
19.3
Conformer Transducer (Witassbe 2022)
7.8
19.8
TTS augm. (Laptev 2000)
TTS data (Laytes 2020)
LSTM
4.3
13.5
EZE baseline (ouns)
20.1
LaSyn FixedProj LFR (ours)
4.5
17.1
LaSyn-tixedProj (ours)
4.5
16.1
LaSyn-Diflusion (ours)
Table 3: Low-resource ASR results trained with train- clean-100 100 split of LirbiSpeech. We compare LaSyn with published supervised methods. We report WER (%) on dev/test sets.
pared to the E2E baseline. We notice that the im- provement on test-clean is more significant than test-other. Both the fixed-projection latent syn- thesizer and the diffusion latent synthesizer are trained with train-clcan- 100 which contains only clean speech. We speculate that the limited va- ricty of training data train-clean-10 biases ASR performance toward clean speech.
We note that the model structure of the E2F bascline model is the same as the LaSyn model, but the latent synthesizer is disabled. The E2E baseline model does train with any additional textual data. We fine-tune the E2E baseline model with SLU task after ASR pre-training.
We also observe that the performance of the model with fixed-projection latent synthesizer (LaSyn-FixedProj) is only slightly worse than LaSyn-Diffusion. The result is surprising, the fixed-pree -xojepproecttoon latent synthesizer is simpler than the diffusion latent synthesizer. The diffusion la- tent synthesizer may need further hyper- parametrr tuning, or may need more training data for better performance. The utilizes a low frame rate speech latent encoder as illustrated in Fig. 5. The performance is slightly worse than the LaSyn-FixedProi on test-other.
4.3
ASR Results
The experimental results of ASR are shown in Ta- ble 3. We first compare LaSyn models with our E2E baseline which achieves comparable perfor- mance to conformer-based models. The only differ- ence is that the LaSyn models are trained with ad- ditional textual data. The LaSyn-Diffusion model, which uses diffusion latent synthesizer, achieves 40.5% and 22.3% relative WER reductions on test- clean and test-other of Librispeech test sets com-
model
Compared to published supervised ASR mod-

---

Model
Params
IC
SF (SLU-FF)
Inten:roll,, narce-edoo Slauos
Intentplay mdio. redio sliles
Intentylog ridio, namesskile slalco
(ACC %)
alatoo
ESPact-SLU (Arora et al., 2022)
300 M
86,3
71.9
PT-hbt-base (Wäng et al., 2021)
2 M
87.5
75.3
Brneline
LaSyn
-i-bbb--argo (Wing stal. 2021)
300
89.4
78.4
Textrall Corpos
E21 Baseline (nurs)
37.8 M
84.4
74.7
LaSyn-Diffusion Diffusion (ours)
37.8 M
87.4
77.3
oldies station
does not occur
can you find the ulcies sialion play alties stabin
LaSyn-FixedProj (ours)
37.8 M
88.5
78.5
Table 4: Results on SLURP dataset. We report accuracy (ACC%) for task and SLU-FI for SF task.
Figure 8:
An example of LaSyn output from SLURP test set. The target "oldies station" does not appear in SLU training data while LaSyn utilizes knowledge from the textual corpus. Meta values such "intent" and slot type are italicized.
els that utilize text data through external language models, LaSyn models perform better without an external language model (LM). Compared to the published methods using TTS for data augmenta- tion, the performance of LaSyn models are signif- icantly better without external LM. Given the existence of real-world scenarios with limited la- beled speech data, such as minority languages and specific domains, our proposed method offers a novel approach to developing ASR applications.
the test set that LaSyn performs better than our baseline. An example is shown in Fig. 8. Our E2E baseline model fails for the slot "Oldies Station", as this phrase never occurs in the SLURP training set. In contrast, LaSyn model correctly predicts the slot value. This phrase is included in the textual corpora. The text knowledge is transferred to SLU model with the LaSyn framework. The baseline E2E SLU model does not get the proprietary term 'Oliies Station' while LaSyn predicts this unique vocabulary successfully.
4.4 SLU Results
4.4.1 SLURP
The experimental results of SLURP are shown in Table 4. We report accuracy for intent classification (IC), and (Batta(Basii et al., 2020) for slot filling (SF).
4,4.2 STOP
We present our results of STOP in Table 5. Com- pared to our E2E baseline, LaSyn-FixedProi im- proves EM accuracy and EM-Trec accuracy on the test set by absolute 4.49% and 2.25% respec- tively, again suggesting that there is effective cross- modality text knowledge transfer.
We first compare LaSyn models with our E2E bascline. Compared to the E2E baseline, LaSyn- FixedProj improves IC accuracy and SF SLU-F1 by absolute 4.1 1% and 3.8% respectively. The re- sult suggests that knowledge of textual NIP data is effectively transferred to SLU model. LaSyn- Diffusion performs slightly worse than LaSyn- FixedProj. We believe that with further hyper- parameter tuning and more training data, the per- formance of diffusion latent synthesizer should be further improved.
We further compare our results with STOP- E2E and STOP- JTOP-Cascaded (Tomasello et al., 2022). STOP-E2E is an encoder-decoder based Trans- former model fine-tuned from an E2E ASR model. The E2E ASR model is fine-tuned from HuBERT Base (Hsu al., 2021). STOP- Cascaded is caded system composed of an ASR system fine- tuned from wav2vec2.0 Base (Baevski et al., 2020) and an NLU model fine-tuned from BART Base model (Lewis et al., 2019). LiSyn-FixedProj per- forms slightly better than STOP-E2E with 0.25% and 2.63% absolute improvement of EM and EM- Tree accuracies on the test set respectively. How- ever, compared to while LaSyn-FixedProo is competitive EM-Tree accuracy, EM accuracy is slightly inferior. The number of parameters in LaSyn models is much fewer. We expect performance improvement with more model parameters.
We further compare the LaSyn models with pre- viously published E2E SLU results. The published models are fine-tuned from HuBERT (Hsu al., 2021) Base (95 M parameters) or Large (300 M parameters). The performance of LaSyn-FixedProj is comparable to ESPnet-SLU (Arora et al., 2022) and Pi--bt-base (Wang et al.. 2021). The IC ac- curacy of LaSyn-FixedProy is slightly worse than EF-hbt-large (Wang et al., 2021), but the number of parameters is 8 times fewer.
on the test set,
To understand how LaSyn improves our baseline E2E SLU model, we further analyze samples from

---

Model
# Params
dev
Test EM-
Model
Text Training data
STOP (dev) (EM FA-Twe))
STOP (test)
EM EM-Tre
(RM RM-Treel
STOP-E2R
120 baseline
64,02/ 82.84
64.99/822
90 M
63.82/83.8
69 82.87
(Tomssollo al. 1003)
NLU NER
68.99/ 86.
09 29.40/85:45
(Tomescllo al, 2002)
STOP.Casesdol
230 M
72/43/ 96.58
72.36 85.77
LaSyn
IE
68.55 85.555/55.65 85.65
68.43/85. 85.55
69 09.24 15.005
68 68.88/ 84.99
E2E Buseline (ours)
37.8
64,02 82.84
64.99/ 82.25
NLU + NER
69.33/ 86.24
69.18/85.50
LaSyn-Diffusion (ours)
37.8 M
67.91 85.57
68..3 84,92
LaSyn-FixxdProj (ours)
Table 7: Results of LaSyn trained with text data of differ- NL. tasks. We report EM and EM-Trre accuracies (%) on STOP dev and test sets.
Table 5: Results STOP dataset. We report the EM and EM-Tree accuracies (%) on dev and test sets.
SLURP (IC/SF)
STOP (dev) (KM / EM-Tree)
STOP (test) (EM KM--rree
Model
(ten)) (EM/EM-Tre)
Model
(C: SF)
(EM EM- Tree)
E2E Baseline
84.4/747
64/12/ 82 84
64 64.99/ 82,
Baseline
$4,7744.7
64.02 82.84
64.99/82.25
LaSyn (Acoustic Aug.)
86.9/76.0
67.69/85.8
LaSyn FroedFroj
labelled
$8.5 77855
09.33 86
19.48/335.0
JSSyy-FasceProj
unlahelled
76,1
66.13 /82399
66,40/ 92.33
Table 8: Results of acoustic augmentation with latent synthesizer. We report IC (ACC%) and SF (SLU-F1) for SLURP, EM and EM-Tree accuracies (%) for STOP.
Table 6: Ablation study of unlabeled text data, We report results on SLURP test set, STOP dev and test sets.
5.3 Latent Synthesizer as Acoustic Augmentation
5
Ablation Study
We experiment with using the fixed-proeection la- tent synthesizer for acoustic augmentation. We ex- tract the transcription and the annotation from the SLU dataset to form an NLU dataset. When train- ing the backbone model, we apply both the SLU and the NLU datasets in dual-modality training. As the NLU dataset is derived from the SLU dataset, the latent synthesizer does not introduce extra tex- tual content. Pseudo speech latent representations from the latent synthesizer are considered as an augmentation of the original speech latent repre- sentation.
5.1 Training with Unlabeled Textual Data
Plain text data without annotation more abun- dant than annotated NLP data. We experiment with SLU training with unlabelled textual data. We prepare the unlabelled text data by striping the an- notation labels of the NLP datasets and keeping the input text. We apply the MLM task described in section 4.1.2 to utilize the unlabeled textual data, We evaluate LaSyn models with projeprooection latent synthesizer. The results are listed in Table 6.
The results show that LaSyn still benefits from unlabeled text, compared to our E2E baseline on both SLURP and STOP datasets. With unlabeled text and MLM tasks, LaSyn achieves an absolute improvement of 1.6 % and 0.9 % on IC and SF tasks on SLURP dataset, 2.19 %. and 0.46% on EM and EM-Tree improvement is not as significant using labeled textual data, data collection is further simplified with unlabelled textual data.
As shown in Table 8, SLU performance im- proves significantly over the E2E baseline but does not reach the level of Table 7 which utilizes extra NLP datasets. Further enriching the diversity of pseudo acoustic latent is the potential to improve SLU performance.
STOP test set. While the
Conclusion
We present LaSyn, a framework which enables efficient textual data utilization for E2E speech pro- cessing. By converting text into pseudo acoustic la- tent representation with a latent synthesizer, cross- modality knowledge transfer from textual data to E2E speech processing models is achieved. For the low-resource ASR task with Librispeech, LaSyn achieves relative WER reduction from 22.3% 40.5% on test sets, compared to our E2E baseline with the same model structure. The results are com- petitive to published works which utilize textual data through external language models. For SLU
5.2 Training with Diverse NLP Tasks
We do an ablation to observe the effect of train- ing LaSyn with textual data from a diverse set of NLP tasks. The results are shown in Table 7. We observe that including each NLP task brings sub- stantial improvement over the E2E baseline. As the NLU task is the most relevant SLU, perfor- mance improvement is the most significant. When we combine all the NLP tasks, there is marginal further performance improvement.

---

tasks, LaSyn improves over our E2E baseline by absolute 4.1% and 3.8% for for IC accuracy and SF SLU-FI on SLURP, and absolute 4.49% and 2.25% of EM and EM-Tree accuracies on STOP. The results are competitive to published SOTA works with much fewer model parameters. Future improvement of latent synthesizer should further bridge the gap between speech and textual modal- ity, which we leave as next step.
Workshop on Insights from Negative Results in NLP. pages 100-105.
Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xu- ankai Chang, Yushi Ueda, Yifan Peng. Yuekai Zhang, Sujay Kumar, Karthik Ganesan, Brian Yan, et al. 2022. ESPnet-SILUU: Advancing spoken language understanding through ESPret. In 2022 IEEE Inter- national Conference on Acousticx, Speech and Signal Processing (ICASSP 2022), pages 7167-7171.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed. and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Sys- tems, volume 33, pages 12449-12460 12460.
Limitations
The core of our method is the generation of pseudo acoustic representation from text input. We fo- cus on generating consistent latent sequences effec- tively. We only evaluate two latent synthesis meth- ods, including fixed-proeection and diffusion latent synthesizers. There are other probable methods for latent generation, such as generative adversarial networks (GAN) (Goodfellow et al., 2020). Com- pared with TTS which generates audible speech suitable for human judgment, there is no subjective method to evaluate the quality and intelligibility of generated pseudo acoustic representation from the proposed framework, which is main limita- tion. The design of reasonable quality indicators of acoustic representation would be meaningful for future work. Moreover, we have not evaluated the proposed latent synthesis framework on other phonological systems such as tonal languages like Chinese. The effectiveness of the framework on tonal languages is not guaranteed.
Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. 2021. SLAM: A unified encoder for speech and language model- ing via speech-text joint pre-training. arXiv preprint arXiv: I 10. 10329.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojan- ski, and Verena Rieser. 2020. Slurp: A spoken lan- guage understanding resource package. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7252-7262.
Daniel Braun, Adrian Hernandez-Mendez, Florian Matthes, and Manfred Langen. 2017. Evaluating natural language understanding services for conver- sational question answering systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 174-185.
Xavier Carreras and Lluís Mårquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role la- beling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL- 2004) at HL.T-NACII 2004, pages 89-97.
Ethics Statement
In this paper. we only use publicly available datasets for experiments. Our experiments do not involve any subjective tests or human data anno- tations. In the experiments, the latent synthesis framework does not produce any audible speech content. We do not apply any specific speaker in- formation during training and inference.
lñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on NLP for ConvAl- ACL 2020.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mo- hammad Norouzi, Najim Dehak, and William Chan. 2021a. WaveGrad 2: Iterative refinement for text-to- speech synthesis. arXiv preprint (1..:i211..0966666
Yixin Chen, Weiyi Lu, Alejandro Mottini, Li Erran Li, Jasha Droppo, Zheng Du, and Belinda Zeng. 202 lb. Top-down attention in end-to-end spoken language understanding. In 2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP 2021), pages 61 6199-6203.
References
Junyi Ao, Rui Wang. Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei. et al. 2021. SpeechTs: Unified-modk encoder- decoder pre-training for spoken language processing. arXiv preprint arxiv: 10.07205.
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. 2022a. Maestro: Matched speech text representations through modality matching. arXiv preprint arXiv:2204.03400
Gaurav Arora, Chirag Jain, Manas Chaturvedi, and Kru- pal Modi. 2020. HINT3: Raising the bar for intent detection in the wild. In Proceedings of the First

---

Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhu- vana Ramabhadran, Pedro Moreno, and Gary Wang. 2022b. TisApretrain 2.0: Advancing the use of text and speech in ast pretraining with consistency and contrastive losses. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7677-7681.
speech pre-training, In International Conference on Learning Representations.
Yinghui Huang, Hong-Kwang Kuo, Samuel Thomas, Zvi Kons, Kartik Audhkhasi, Brian Kingsbury. Ron Hoory, and Michael Picheny. 2020b. Leveraging unpaired text data for training end-to-end speech-to- intent systems. In 2020 IEEE' International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP 2020), pages 7984-7988.
Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass. 2018, Unsupervised cross-modal align- ment of speech and text embedding spaces. Advances neural information processing systems, 31.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. 2022. Elucidating the design space of diffusion-basett generative models. arXiv preprint arXiv:2206.00364.
Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 140-147.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. 2023. Speak, read and prompt: High-fidelity text- to-speech with minimal supervision. arXiv preprint (rxxi:2302.0354:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint 180.0008..
Minjeong Kim, Gyuwan Kim, Sang- Woo Lec, and Jung- Woo Ha. 2021. ST-BERT: Cross-modal language model pre-training for end-to-end spoken language understanding. In 2021 IEEE International Confer- ence on Acoustics. Speech and Signal Processing (ICASSP 2021), pages 7478-7482.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Communications of the ACM,
Alex Graves. 2012. Sequence transduction with recurrent neural networks. arXiv preprint
Aleksandr Laptev, Roman Korostik, Aleksey Svischev, Andrei Andrusenko, Ivan Medennikov, and Sergey Rybin. 2020. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. arXiv preprint arXiv:2005.071:
121.111
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang. and Yonghui Wu. 2020. ContextNet: Improving convolutional neural networks for auto- matic speech recognition with global context. In Interspeech 2020, pages 3610-3614.
Stefann Larson and Kevin Leach. 2022. Redwood: Using collision detection to grow a large-scale intent classi- fication dataset. arXiv preprint arXiv:2204.0544
Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evalua- tion dataset for intent classification and out-of- prediction. arXiv preprint Xiviv: 1909.02027.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Ad- vances in Neural Information Processing Systems, volume 33, pages 6840-6851.
Jonathan Ho and Tim Salimans. 2022. Classifier- free diffusion guidance. arXiv:2207. 12598.
arXiv preprint
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence teeeeetto-segggeee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arxiv:1910 13461.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Zi- wei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020. CrossNER: Evaluating cross- domain named entity recognition. ArXiv preprint arXiv:2012.04373.
Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen. 2020a. Conv- Transforsfrr Transducer: Low latency, low frame rate, streamable end-to-end speech recognition. In Interspeech 2020, pages 5001- 5005.
Yi Luan, Lubeng He. Mari Ostendorf, and Hannanch Hajishirzi. 2018. Multi-task identification of entities. relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 2232-323..
Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, and Qun Liu. 2022. SPIRAL: Sulf-super perturbation-invariee representation learning for

---

Loren Lugosch, Brett Meyer, Derek Nowrouzezahrai, and Mirco Ravanelli. 2020. Using speech synthesis to train end-to-end spoken language understanding models. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), pages 8499-8503.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. RWTH asr systems for LibriSpeech: Hybrid vs Attention. In Interspeech 2019, pages 231-235.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint 1XXiv:190.00585.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: To- wards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741
A Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Schema- guided dialogue state tracking task at dste8, arXiv preprint arXiv:2002.0135)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-00655.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5210- IEEE.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241.
D. S. Park. Y. Zhang, C. Chiu, Y. Chen, B. Li, W. Chan, Q. Le, and Y. Wu. 2020. SpecAugment on large scale datasets. In 2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP 2020), pages
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, $ Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 1487.
Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung- Cheng Chiu, Bo Li. Yonghui Wu, and Quoe V. Le. 2020. Improved noisy student training for automatic speech recognitoon. In Interspeech 2020, pages 2817- 2821.
Hiroaki Sato, Tomoyasu Komori, Takeshi Mishima. Yoshihiko Kawai, Takahiro Mochizuki, Shoei Sato, and Tetsuji Ogawa. 2022. Text-only domain adap- tation based on intermediate ctc. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol- ume 2022, pages 2208-22111
Vijayaditya Peddinti, Yiming Wang, Daniel Povey, and Sanjeev Khudanpur. 2018. Low latency acoustic modeling using temporal convolution and LSTMs. IEEE Signal Processing Letters, 25(3):373-377
Baolin Peng. Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task- oriented dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 172-182, Online. Association for Computational Lin- guistics.
Bidisha Sharma, Maulik Madhavi, and Haizhou Li. 2021. Leveraging acoustic and linguistic embed- dings from pretrained speech and language models for intent classification. In 202/ IEEÉ International Conference on Acoustics, Speech and Signal Process- ing (ICASSP 2021). pages 7498-7502.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. 2021. Grad-TTS: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning (ICML), pages 1599-8608.
Yilin Shen, Yen-Chang Hsu, Avik Ray, and Hongxia Jin. 2021. Enhancing the generalization for intent classification and out-of-domain detection in slu. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 2443- 2453.
Yao Qian, Ximo Biany, Yu Shi, Naoyuki Kanda, Leo Shen. Zhen Xiao, and Michael Zeng. 2021. Speech- language pre-training for end-to-end spoken language understanding. In 2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP 2021), pages 7458-7462.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through

---

stochastic differential equations. arXiv:2011.13456.
arXiv preprint
Pengwei Wang, Liangehen Wei, Yong Cao, Jinghui Xie, and Zaiqing Nie. 2020. Large-scale unsupervised pre-training for end-to-end spoken language under- standing. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), pages 7999-8003.
Marcin Sowański and Artur Janicki. 2020. Leyzer: A dataset for multilingual virtual assistants. In Inter- national Conference on Text, Speech, and Dialogue, pages 477-486.
Yingzhi Wang. Abdelmoumene Boumadane, and Abdel- wahab Heba. 2021. A fine-tumed wav2vee 2.0/hu- bert benchmark for speech emotion recognition. speaker verification and spoken language understand- ing, arXiv preprint arXiv:2111.02735.
Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg. Bhuvana Ramab- hadran, and Yonghui Wu. 2020. Generating diverse and natural text-to-speech samples using a quantized fine-graine VAE and autoregressive prosody prior. In 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP 2020), pages
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson En- rique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsuhasa Ochiai. Official Results of Conformer- Models from ESPNET [online]. 2022.
Samuel Thomas, Brian Kingsbury, George Saon, and Hong- Kwang J Kuo. 2022. Integrating text inputs for training and adapting rnn transducer asr models. In ICASSP 2022-2022 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 8127-8131, IEEE
Ralph Weischedel, Martha Palmer, Mitchell Marcus. Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni- anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 19C2013T19. Linguistic Data Consortium.
Samuel Thomas, Klong-Kwang Kuo, George Saon, Zoltán Tüske, Brian Kingsbury, Gakuto Kurata, Zvi Kons, and Ron Hoory. 2021. Rnn transducer models for spoken language understanding. ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7493- 7497. IEEE.
Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kal- gaonkar, Yongqiang Wang, Due Le, Mahaveer Jain, Kjell Schubert, Christian Fuegen, and Michael L Seltzer. 2019. Trunsforme--Trandducee speech recognition with self-attention. arXiv preprint 1xiv:1910.12977.
End-to-end
Yusheng Tian and Philip John Gorinski. 2020. Improv- ing end-to-end speech-to-intes classification with Reptile. In Interspeech 2020, pages 891-895.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ. 2.2: A dialogue dataset with addi- tional annotation corrections and state tracking base- lines. In Proceedings of the 2nd Workshop on Natu- ral Language Processing for Conversational AI, ACL 2020, pages 107-
Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, and Zhengqi Wen. 2019. aelfnattention transducers for end-to-end speech recognition. In Interspeech 2019, pages 4395 4399.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction the CoNLL-2003 shared task: eegguage-indepedeed named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning HLT-NAACL 2003, pages 142- 147.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross- modal conversational abilities. arXiv preprint arXiv:2305.1 1000.
Paden Tomasello, Akshat Shrivastava, Daniel Lazar, Po- Chun Hsu, Due Le, Adithya Sagar, Ali Elkahky, Jade Copet, Wei-Ning Hsu, Yossi Adi, Robin Algayres, Tu Ahn Nguyen, Emmanuel Dupoux, Luke Zettle- moyer, and AbdeIrahman Mohamed. 2022. STOP: A dataset for spoken task oriented semantic parsing. arXiv preprint arXiv:2207. 10644.
Q. Zhang, H. Lu, H. Sak. A. Tripathi, E. McDermott, S. Koo, and S. Kumar. 2020. Transformer Trans- ducer: A streamable speech recognition model with transformer encoders and RNN-T loss. In 2020 I.E. International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), pages 7829-7833.
Ziqiang Zhang, Sanyuan Chen. Long Zhou, Yu Wu. Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, et al. 2022a. SpeechLM: En- hanced speech pre-training with unpaired textual data. arXiv preprint arXiv:2209.15329
Gokhan Tur, Dilek Hakkani-Tüir, and Larry Heck. 2010.
What is left to be understood in ATIS? In 2010 IEEE Spoken Language Technology Workshop (SLT), pages 19-24.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
Ziqiang Zhang. Long Zhou, Junyi Ao. Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022b. SpeechUT: Bridging speech and text with hidden- unit for encoder-decoder based speech-text pre- training. In Proceedings of the 2022 Conference on

---

Empirical Methods in Natural Language Processing, pages 1663-1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.


Reviews:
[{"review_id": "NCkwmCINrz", "reviewer": "Reviewer_1c8Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and comments. We next provide explanations for your concerns.\n\n> The ASR experiments are limited. Librispeech is a widely used dataset, but there are various more competitive models (e.g. Wave2Vec2, HuBERT etc) than the baselines in the paper.\n\nOur method is complementary to speech pre-training as LaSyn is initialized with a Transformer-based ASR which can be fine-tuned from a pre-trained model. \nThe official result (WER) of Wav2Vec 2.0 BASE in low-resource ASR experiment is 6.1\\% / 13.3\\% for test-clean / other without external LM. While Wav2Vec 2.0 utilizes 960-hour unlabelled speech for pre-training, our LaSyn framework in contrast utilizes 960-hour text transcription (4.4\\% / 15.6\\%). Both frameworks exhibit their own strength on test clean / other respectively.  \n\n> Moreover, since Librispeech is clean read speech data and not a very challenging dataset, experimenting on it alone may not reflect the ASR performance comprehensively.\n\nWe acknowledge that testing on more varieties of ASR corpora for ASR task would be more comprehensive. We consider ASR task as the first stage of traditional pipe-lined SLU system. We include the ASR task with LibriSpeech 100-hour data to evaluate the soundness of LaSyn design on basic speech processing. We further conducted more comprehensive experiments on SLU tasks and demonstrated the effectiveness of our LaSyn framework.\n\n> When targeting low-resource languages/domains, the availability of G2P are also be limited (section 3.2.2).\n\nWe acknowledge the importance of addressing the G2P availability in low-resource settings. We will incorporate this perspective in the Limitation section in the revised manuscript. We note that a G2P model can built with various methods other than deep learning, as long as pronunciation dictionaries of the low-resource languages are available.\n\n> Reproducibility: 2: Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.\n\nAll datasets for experiments in this paper are publicly available. We will release the source code along with the parameters in the Appendix once accepted.\n\n> Missing References\n\nWe will make revisions to the paper to include the missing references you mentioned. We believe that incorporating them into our paper will better support and enrich our perspectives.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Two novel designs of latent synthesizer are tailored to the end goal of text-to-speech representation simulation.\n2. The effectiveness of the approach is empirically proven by promising results on multiple tasks/datasets.\n3. Good paper writing and clear presentation.", "reasons_to_reject": "1. The ASR experiments are limited. Librispeech is a widely used dataset, but there are various more competitive models (e.g. Wave2Vec2, HuBERT etc) than the baselines in the paper. Moreover, since Librispeech is clean read speech data and not a very challenging dataset, experimenting on it alone may not reflect the ASR performance comprehensively.\n2. When targeting low-resource languages/domains, the availability of G2P are also be limited (section 3.2.2).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The researchers introduce a framework called LaSyn, designed to efficiently utilize textual data for end-to-end (E2E) speech processing models. This framework facilitates the transfer of cross-modal knowledge from text to E2E speech processing models through latent synthesis. Two implementations of the latent synthesizer are designed as the core of LaSyn: a fixed-projection latent synthesizer and a diffusion latent synthesizer, which incorporates recent advancements in diffusion models. By leveraging LaSyn to enhance E2E speech models with textual data, the researchers achieve competitive results multiple ASR/SLU tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Relevant papers on improving/adapting latent speech representation with text only data.\n\n1. Text-Only Domain Adaptation Based on Intermediate CTC. https://www.isca-speech.org/archive/pdfs/interspeech_2022/sato22_interspeech.pdf\n2. Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models. https://arxiv.org/pdf/2202.13155.pdf"}}, {"review_id": "KHWRlK3vI0", "reviewer": "Reviewer_Paaz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and valuable feedback.\n\n> A comparison with data augmentation methods utilizing discrete acoustic units like SpeechUT is absent. These discrete units simplify speech by emphasizing phonetic and semantic information. An alternative to latent synthesizers involves training Transformer models to generate these discrete units from text [Popuri+ 2022]. While this approach necessitates an additional module to map discrete units to the continuous space, prior research has demonstrated quality enhancements through this technique.\n\nSpeechUT leverages speech pre-training such as HuBERT for speech-to-unit task. The reported results from LaSyn do not utilize speech pre-training but can be initialized with fine-tuned ASR from pre-trained models. The amount of utilized text information is not the same. Therefore the results are not directly comparable. We have already included SpeechUT in the related work and are considering expanding the discussion in the revised edition.\n\nMethods utilizing discrete units, which have been demonstrated quality enhancements, and LaSyn utilizing continuous latent features are both promising and worth exploring.\n\nMethods that utilize discrete units involve discretizing speech signals, which requires the construction of speech-to-unit and text-to-unit generators, e.g., the k-means model learned from HuBERT for speech, and Transformer models for text-to-unit conversion. \n\nIn contrast, our LaSyn method directly generates continuous latent features from text by learning a latent synthesizer, e.g., a diffusion model or a fixed projection model.\n\n> Comparisons with TTS augmentation are unfair due to the encoder architectures (Transformer vs Conformer). Consequently, the results don't definitively indicate the superior data augmentation method.\n\nWe emphasize that our models are not based on Conformer.\n\nOur models are still based on Transformer even though we add interleaved strided-convolutions between transformer layers for downsampling. We compared our baseline model with Conformer-based ASR and found that the performance is comparable as listed in Table 3. The LaSyn models perform significantly better than our baseline model and Conformer-based ASR.\n\n> The investigation into the efficacy of external language model fusion is lacking. Since the text data employed in data augmentation originates from the same source as LM training data (specifically Librispeech 960h in this study), we seek clarification on two aspects: 1) the relative effectiveness of the proposed method compared to shallow fusion, and 2) whether the proposed method complements shallow fusion\n\nWe compared with the published ASR results with LM in Table 3. Our results of LaSyn models without LM are better or comparable. While LM is a validated approach to utilizing text data, decoding with LM introduces additional complexity and computational costs. As our LaSyn model, when performing ASR decoding, is the same as conventional Transducer-based ASR, the model is compatible with shallow fusion and can utilize LM if available.\n\n> The results in Section 4.2.1 highlight the dependence of data augmentation effectiveness on the domains of training data used for latent synthesizers. This restriction confines the applicability of the proposed method. To investigate domain transferability, employing an ASR model trained on an out-of-domain dataset for initializing the Guiding Net would be desirable.\n\nIn the ASR experiments, the guiding net is trained only with train-clean-100 split of LibriSpeech. We still observe performance improvement on test-other dataset. Covariate shift is observed in the test-other dataset as they are recorded in a more complex background environment compared with test-clean dataset.\n\n> Missing References\n\nWe will make revisions to the paper to include the missing references you mentioned.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is novel to some extent. If the authors conducted more experiments in the speech translation task, the claims would be stronger.", "reasons_to_reject": "・A comparison with data augmentation methods utilizing discrete acoustic units like SpeechUT is absent. These discrete units simplify speech by emphasizing phonetic and semantic information. An alternative to latent synthesizers involves training Transformer models to generate these discrete units from text [Popuri+ 2022]. While this approach necessitates an additional module to map discrete units to the continuous space, prior research has demonstrated quality enhancements through this technique.\n\n・Comparisons with TTS augmentation are unfair due to the encoder architectures (Transformer vs Conformer). Consequently, the results don't definitively indicate the superior data augmentation method.\n\n・The investigation into the efficacy of external language model fusion is lacking. Since the text data employed in data augmentation originates from the same source as LM training data (specifically Librispeech 960h in this study), we seek clarification on two aspects: 1) the relative effectiveness of the proposed method compared to shallow fusion, and 2) whether the proposed method complements shallow fusion.\n\n・The results in Section 4.2.1 highlight the dependence of data augmentation effectiveness on the domains of training data used for latent synthesizers. This restriction confines the applicability of the proposed method. To investigate domain transferability, employing an ASR model trained on an out-of-domain dataset for initializing the Guiding Net would be desirable.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel data augmentation framework aimed at enhancing low-resource automatic speech recognition (ASR) and spoken language understanding (SLU) by leveraging unpaired text data. The authors introduce latent synthesizers, which generate continuous latent features from text. These latents correspond to the output features of the speech encoder, enabling the latent synthesizer to produce simplified representations compared to traditional mel-spectrogram and waveform approaches. The study investigates fixed-projection and diffusion-based models as potential latent synthesizers. Experimental evaluations demonstrate that the proposed augmentation technique leads to substantial improvements in both ASR and SLU quality.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "・Popuri et al., Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation, INTERSPEECH, 2022.\n\n・Siuzdak et al., WavThruVec: Latent speech representation as intermediate features for neural speech synthesis, INTERSPEECH, 2022."}}, {"review_id": "fPkXoEJGeO", "reviewer": "Reviewer_tecw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind and detailed review.\n\n> Not a strong reason to reject, but I am left wondering if the impact of the proposed approach happens because the latent acoustic representation trained with the datasets is incomplete (in terms, for example, of linguistic coverage). As there is an abundance of speech-text paired datasets in the public domain, it could be that the benefits of the approach are restricted to lower-resource languages in which the technique was untested, leading to a lower impact of the proposed knowledge.\n\nIn this work, we focus on the low-resource setting as there are real-world scenarios where labeled speech data are still limited, e.g., minority languages and specific domains. For example, the STOP dataset used in our SLU experiment is the largest public SLU dataset so far which contains only 218 hours of paired data.\n\nIt is interesting whether the proposed LaSyn method is effective in the high-resource setting and we are considering exploring it in future work.\n\n> Missing References \n\nWe will make revisions to the paper to include the missing references you mentioned. We believe that incorporating them into our paper will better support and enrich our perspectives.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes an interesting formulation to include unpaired data to speech models. While there has been an abundant recent amount of work to incorporate unscripted audio (audio without text pairs) there has been limited recent work on the 'other side of the coin'. This has been mostly carried out by leveraging pre-trained Language Models, so in this paper the authors propose a lighter approach to use linguistic information in a cross-modal way. This is demonstrated to have a positive impact across multiple downstream tasks, which reinforces the value of linguistic information in multiple speech-related tasks beyond the conventional domains.", "reasons_to_reject": "Not a strong reason to reject, but I am left wondering if the impact of the proposed approach happens because the latent acoustic representation trained with the datasets is incomplete (in terms, for example, of linguistic coverage). As there is an abundance of speech-text paired datasets in the public domain, it could be that the benefits of the approach are restricted to lower-resource languages in which the technique was untested, leading to a lower impact of the proposed knowledge.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework for leveraging pure textual data as a way to improve the modeling capability of speech tasks that originally required paired audio-text pairs. This makes it possible to grow the representation capability and accuracy of models by leveraging linguistic information proyected into the acoustic latents. This, the authors hypothesize that promotes cross-modal (linguistic->acoustic) knowledge transfer. Results are demonstrated on both speech recognition and spoken natural language understanding tasks, improving over baselines and meeting state of the art performance on smaller models and (paired) datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I believe the paper would benefit from covering the problem also from the point of view of acoustic->linguistic cross-modal training, such as https://arxiv.org/pdf/2302.03540.pdf. Even if the focus is not on how to properly develop and train speech latent encoders, there is potential interaction and extended impact in covering both literatures.\n\nI also miss references to the recent trends such as SpeechGPT models that are potential competitors (or beneficiaries?) of this approach https://arxiv.org/pdf/2305.11000.pdf"}}]

Author rebuttal:

Meta-review: This paper presents a lightweight framework to utilize text for training speech processing models. In particular, it adds a latent synthesizer module that maps text into speech embedding for training a shared backbone model that takes this embedding as input. Two implementations (fixed projection and diffusion) of the latent synthesizers are presented. Effectiveness of the approach is verified on speech recognition (Librispeech) and spoken language understanding (SLURP and STOP).

All reviewers find the approach novel and can be easily reproduced. The proposed approach also demonstrates strong performance over supervised baselines and other semi-supervised baselines on multiple speech processing tasks. 

The main caveat of the manuscript is the lack of comparison with semi-supervised baselines that utilizes text data (only compared with TTS data augm. (Laptev et al., 2020)). While I agree with the authors in the rebuttal that SpeechUT may not be a suitable baseline because it is built upon HuBERT utilizing unpaired speech, there are still prior studies such as [1,2]. Specifically, [1] trains a text-to-encoder (TTE) model that predicts ASR embeddings from text and uses that for training shared components. This is highly related to the proposed method, but I acknowledge that proposed methods does have novelty by leveraging diffusion model.

[1] Hayashi, Tomoki, et al. "Back-translation-style data augmentation for end-to-end ASR." 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018.
[2] Liu, Alexander H., Hung-yi Lee, and Lin-shan Lee. "Adversarial training of end-to-end speech recognition using a criticizing language model." ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.
Decision: Accept-Findings