Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM
Authors: Eliya Nachmani
Alon Levkovitch
Roy Hirsch
Julian Salazar
Chulayuth Asawaroengchai
Soroosh Mariooryad
Ehud Rivlin
RJ Skerry-Ryan
Michelle Tadmor Ramanovich
Venue: ICLR 2024 poster (2024)
Keywords: Speech Continuation
Spoken Question Answering
Primary area: general machine learning (i.e., none of the above)
TL;DR: Spoken question answering and speech continuation leveraging pre-trained language model operating in the spectrogram domain

Abstract: We present Spectron, a novel approach to adapting pre-trained large language models (LLMs) to perform spoken question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture. Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original LLM as demonstrated through spoken QA datasets. We release our audio samples and spoken QA dataset via our website.

Published as a conference paper at ICLR 2024
SPOKEN QUESTION ANSWERING AND SPEECH CONTINUATION USING SPECTROGRAM-POWERE LLM
Eliya Nachmani' Alon Levkovitch-.3... Roy Hirsch", Julian Salazar', Chulayuth Asawaroengchai', Soroosh Mariooryad', Ehud Rivlin?, RJ Skerry-Ryan', Michhelle Tadmor Ramanovich'
Google Research, "Verily AI, "Tel-Aviv Univeristy {eliyn, alevkovitch, royhirsch| @google.com
ABSTRACT
We present Spectron, novel approach to adapting pre-trained large language mod- els (LLMs) to perform spoken question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end- spectrograms, simplifying our architecture. Key
to-end and operates directly to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, en- abling 'cross-modal" chain-of-thought within single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original L.M demonstrated through spoken QA datasets. We release our audio samples and spoken QA dataset via our website
INTRODUCTION
The goal of natural language processing (NLP) is develop computational models that understand and generate human language. By capturing the statistical patterns and structures of text-based natural language, language models can predict and generate coherent and meaningful sequences of words. Combined with the Transformer model architecture (Vaswani et al. 2011)). large language models (LLMs) trained on web-scale amounts of text, with proportionate compute and size, have demonstrated remarkable success in NLP tasks (Devlin et al., [219 Brown et al., 2020 Chowdhery et al., 2022
Zhang et al.) 2022a; Scao et al.) 2022; Zeng et 2023). However, transferring these abilities to spoken human language remains a challenging frontier. Spoken dialog systems remain cascade of separately trained automatic speech recognition (ASR), natural language understanding (NL.)) and generation (NLG), and sext-to-speech (TTS) systems (Gorin et al. 1997 Jokinen & McTear 2009), with LLMs now playing the role a combined NLU and NLG system. However, such cascades introduce latency and additional mechanisms for propagating and rendering non-verbal cues like speaker identity and prosody. Recently, spoken language models (Lakhotia et al.) 202T Kharitonov et al. 2022) and other generative audio models (Dhariwal et al. Hawthorne et al. 2022 Borsos et al. 2023; AAgsstinelli et al. (2033) have emerged as a promising avenue for generative speech modeling. These works quantize audio representations (Hsu al. 2021 Chung al. |021 Zeghidour et al. 2022] Défossez et al.) (022) into learned discrete tokens compatible with the same next-token cross-entropy objective as text LLMs, step that (Nguyen et al. (002)) argued as necessary for generative quality. In this paper, we introduce Spectron, novel spoken language model that:
Directly process spectrograms as both input and output. Spectron leverages the audio capabilities of pre-trained speech encoder through the use of intermediate projection layers.
'Equal contribution.
Work done during an internship at Google.
https//michellccaannoiicc htt.://mwwncciiiiu.c.oooooonnmom

---

Published as a conference paper at ICLR 2024
Cross-entropy loss Lux
Reconstruction loss Lawan
Pre-trained token
Post-net
Pre-trained language model (Transformer decoder)
Connector layer
Pre-net
Pre-trained speech (Conformer encoder)
Yp
Ye
X
(3s prompt split)
Spectrogram
Original
spectrogram
Text transcript (prompt+cont.)
Figure 1: Spectron connects the encoder of speech recognition model with a pre-trained Transformer decoder language model. At training time, we take speech utterances and split their audio into prompt and continuation. From the prompt speech features, the full (prompt and continuation' s) transcript must be reconstructed, as well as the continuation's 's speech features via newly introduced pre- and post-net speech modules. At inference time, only a prompt provided; the prompt's transcription, text continuation, and speech continuations are all generated by the model.
Demonstrably transfer generative ability from a pre-trained LLM, as shown by competitive performance in semantic coherence and spoken question answering over other end-to-end spoken language models.
To quantify this transfer of knowledge, we also introduce two benchmarks for the nascent spoken QA task, which synthesize from Web Questions (Berant et al. 2013) and generations from LLaMA (Touvron et al. (023). Audio samples and our LLaMA dataset can be found on the project website given the first page.
Our work shows that the inductive biases from pre-trained speech encoder and a language model decoder enable end-to-end training and fftte-of-the-are performance without sacrificing representa- tional fidelity. Key to this is a novel end-to-end training objective which implicitly supervises speech recognition, text continuation, and conditional speech synthesis in a joint manner. The language model transcribes and generates text continuations, acting as an 'intermediate scratchpad" (Nye et al.) 202T Wei et al. (022) be conditioned on for audio generation. A novel spectrogram regression loss also supervises the model to match the higher-order temporal and feature deltas of the ground truth, based on the idea that the derivatives of the ground truth express rich, longer-range information about the shape of the signal. Our overall scheme is summarized in Figure and described in the rest of this work.
2 RELATED WORK
The dominant approach to spoken language modeling is to use compact discrete speech represen- tations. This allows the application of text-based language models to speech data. Typically, these representations created by clustering the outputs of speech encoder using K-means and taking the centroids as tokens. The resulting discrete sequences can be easily modeled using Transformer architectures (Vaswani al. 2017). Below notable examples of works using this approach; comparison table is also presented in Appendix A.2
Generative Spoken Language Modeling (GSLM; Lakhotia al. 2021 offers baseline system that operate on units quantized from pre- trained audio representations, such as HuBERT (Hsu et al.) 2021) The quantized units processed by a Transformer-basel model. An additional unit-to-speech
2

---

Published as a conference paper at ICLR 2024
decoder converts the generated units into spectrograms. Spectron's approach is more simple and explicit, where a single model is imput with spectrograms and outputs raw spectrograms.
TWIST (Hassid et al., 2023) uses the same unit-to-speech and speech-to-unit systems GSLM, but warm-starts the spoken language model from a text-based language model. They show that this warm-start improves overall metries and convergence speed, with reasonable performance StoryCloze tasks (though notably degraded from the text model). For the textual language model, they use the state-of-the-art and open-weight OPT (?) and LLaMA models up 7B (13B camera-ready time) and show that spoken language model improvements scale.
AudioLM (Borsos al. 2023) utilizes two kinds of quantized representations: w2v-BERT (Chung) it al. )021) as semantic tokens, and SoundStream (Zeghidour et al. 2022) as acoustic tokens. SoundStream embeddings undergo discretization using residual vector quantization (RVQ), resulting in hierarchy of vector quantizers. AudioLM utilizes three transformer models, each corresponding different layer of token generation. As Spectron does not involve any quantization, our method naturally preserves the input's semantic and acoustic characteristics. Furthermore, Spectron offers single model trained with a unified objective, as opposed to the multiple components of AudioLM.
SpeechGPT (Zhang et al. 2023a) adapts LLaMA-7B to perform speech tasks by using both discrete speech representations and text. They introduce the SpeechInstruet dataset which they use for instruction tuning. SpeechGPT is trained in 3 different steps: modality adaptation, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning (using LoRA; Hu et al. 2022). The obtained model is capable of generating both speech and text, well following instructions in both modalities. Our method, in comparison, is trained using a single reconstruction step and uses only public datasets for integration. Despite not using a curated dataset or specialized prompts, we demonstrate competitive performance on spoken question answering and superior results for speech continuation.
As for related works in other tasks: Spoken language understanding (SLU): A number of recent studies explore the usage of pre-trained language models (LMs) for different SLU tasks. Goog et al. (2023), Zhao et al. (2023), ) and Liu et al. (2023a) fine-tuned LMs on audio data to perform speech-to-tes question answering tasks, that is, answer textual questions about directly-input audio. Fathullah et al. (2023) showed that adding an audio encoder to an LM and training with LoRA enables the LM to perform automatic speech recognition (ASR). Zhang et al. (2022b) aligned text and audio tokens to perform a large number of SLU tasks. |Peng al. (2023) showed that LMs can be used to answer text questions about spoken properties of language. Spectron, in comparison, produces both textual outputs and spectrograms using single autoregressive decoder. Multi-modal text-speech training: Ao et al. (2022) performed joint training on speech and text data to perform multiple tasks, such text-to-speech (TTS) and ASR. |en al. (2019) used unsupervised pre-training text and speech data to perform TTS for low-resource languages. Textual-guided audio generation: Liu al. (023b) used LMs generate audio scripts and interacting with audio-creation APIs. Huang et al. 2023) augmented the ChatGPT input/output interface to invoke ASR / /TS tools.
APPROACH
3.1
ARCHITECTURE
We propose a novel architecture for direct speech continuation. The architecture is initialized with a pre-trained speech encoder denoted as È and a pre-trained language decoder denoted as LM. The encoder is prompted with a speech utterance as input, which it encodes into continuous linguistic features. These features are fed into the decoder optimized to jointly minimize a cross-entropy loss (for speech recognition and transcript continuation) and novel reconstruction loss (for speech continuation). During inference, one provides spoken speech prompt, which is encoded and then decoded to give both text and speech continuations.
prefix, and the whole encoder-decoder is
3.1.1 INPUT PRE- PROCESSIIN
During training, the proposed model uses supervised speech utterances, which are pairs of speech x and transcripts y for training. The speech input, denoted xr, is spectrogram that is split into two
3

---

Published as a conference paper at ICLR 2024
segments position
3p = sss
Te 2>5-
The first segment Zp (which we call the prompt) is fed into the speech encoder & to give continuous representations that condition the LM. The second segment ze (the continuation) is used later for spectrogram reconstruction loss. SpecAugment (Park et al.. 2019) is applied for data augmentation. The corresponding transcripts y can be also split at position (5)):
Vp M5o(s))
Ve V>(((>)
(2)
where (b)) maps the feature index 8 x to its text token index y. Note that ((s) is not needed for training losses.
3.1.2 SPEECH ENCODER
The speech encoder E is a 6000-000P0 -ara-maree Conformer encoder (Gulati et al. 2020) pre-trained on web-scale data (12M hours: /Zhng et al. 20235). It takes the spectrogram of the source speech as input, generating a hidden representation that incorporates both linguistic and acoustic information. The input spectrogram is first subsampled using a convolutional layer and then processed by series of Conformer blocks, Each Conformer block consists of a feed-forward layer, a self-attention layer, convolution layer, and second feed-forward layer. The outputs of the total encoder & are passed through layer P that projects the hidden representations into the embedding dimension of the language model. We denote these final embeddings
2 ((E(x))).
(3)
3.1.3 LANGUAGE MODEL.
We use prefix decoder language models with 350M or IB parameters trained in the manner of PaLM (Google 2023), which denote as LM(-), The LM receives the encoded features of the prompt Im
Tp as a prefix. Note that this is the only connection between the speech encoder and the LM decoder; i... there no cross-attention between the encoder and the decoder. This late-stage integration is consistent with work in ASR, which found that joint fine-tuning of a pre-trained speech encoder and a pre-trained LM decoder into a sequence-to sequence-to-sequence sequence model can improve performance, even if the integration occurs as a single final layer (Deng et al.) (021]: more layers did not improve performance, which they attribute having sufficiently powerful text representations. During training, the decoder is teacher- forced to predict the text transcription yp, text continuation ye: and speech embeddings x To convert the speech embeddings to and from spectrograms, we introduce lightweight modules h and tr described in the next section. In all, we get next-step predictions for the concatenation of these text tokens and embeddings:
Dpp Dc. 4- LM(2,T [Mp: Ves x!I).
By having the same architecture decode the intermediate text and the spectrograms, we gain two benefits. First, we benefit from the pre-training of the LM in the text domain continue the prompt in the text domain before synthesizing the speech. Secondly, the predicted text serves as intermediate reasoning, enhancing the quality of the synthesized speech, analogous to improvements in text-based language models when using intermediate scratchpads (Nye et al. 2021 or chain-of-thought (CoT; Wei al. 2022).
3.1.4 ACOUSTIC PROJECTION LAYERS
To enable the language model decoder to model speech features, we employ multi-layer perceptron (MLP), the pre-net hrve to project the ground truth spectrogram speech continuations ze to the language model dimension z h1 (,.)) This pre-net h compresses the spectrogram input He into lower dimension, creating a bottleneck that aids the decoding process. This bottleneck mechanism prevents the model from repetitively generating the same prediction in the decoding process, as demonstrated in previous work (Shen et al. 20188) To project &e from the language model dimension to the spectrogram dimension, the model employs post-net hpust which is also MLP. This projection is represented by t. = (1pss((p))
Both hre and pess two-layer MLPs. Additionally, the input text sequence [yp. Mel is padded at the beginning with a "start of sequence" (sos) token, while the output sequence is padded with an "end of sequence" (eos) token at the final position.
4

---

Published as a conference paper at ICLR 2024
3.2 TRAINING OBJECTIVE
The training methodology of the proposed approach is depicted in Figure uses two distinct loss functions: (1) cross-entropy loss, employed for both speech recognition and transcript continuation, and (2) regression loss, employed for speech continuation. During training, all parameters are updated (speech encoder E. projection layer Pa. language model LM, pre-net h and post-net h
3.2.1 SPEECH RECOGNITION AND TRANSCRIPT CONTINUATION
The first loss term is combination of speech recognition loss LASR and transcript continuation loss LLM, which are given by:
LASR 0p) CE(Up: Up), (LM Mc) CE((e:
(5)
where CE denotes cross-entropy, which quantifies the dissimilarity between the predicted distribution over Up: Ve. and the corresponding ground truth distribution over Yp Me. This objective increases the likelihood of the text [yp, yo] under the conditional distribution modeled by the LM.
3.2.2 SPEECH CONTINUATION
The speech continuation objective is formulated as a regression task, predicting each frame's spec- trogram channels independently given previous frame spectrogram predictions and the ASR and LM context. To promote convergence and improve modeling power, we apply & and & regression losses on the spectrogram (Shen et al. (020). These losses applied to the feature-deltas of the spectrogram, and to the time-deltas the spectrogram up to order K, giving "(discrete) derivative loss" terms. That is, for tensor of dimension T x F we define:
A time (2) - [.::T.)) .) Teal (z) FI.1FF- Pl,M:));
L 2') = || 21111 + ||= 211
(7)
For ground truth spectrogram T. and the predicted spectrogram &., the speech continuation loss is a combination of three objectives:
Cs(re, Re) (1+2 &c),
(8)
CiEe, Rc) L142 Afeat (xc) feat (+c)),
(9)
C,(x, = Σ (12((A) time (xc). A time (fe))
(10)
k=1
The overall speech continuation loss is thus given by:
RRoon (Ies Rc) = La(xe, &c) + Ci(we, 2c) + Lo( Rc).
(11)
3.2.3 OVERALL LOSS
Using the above notation, our objective is: Liotat (ar, y) CASR(Un) úp) + LIM(Ue, Mc) + CRecon. (Vey St).
(12)
Since LASR and LLM are cross-entropy losses and since = [gps Mel (Eq 2).
the overall speech
recognition and transcript continuation loss can be written as:
CAsk(yp 0p) + LLM(Ue: Ue) CE(y, M) = Lee(y, 1)
(13)
where ý the concatenation of ÿp and g... This simplifies the overall loss to:
Lusa (2,, y) Lcr 9) + A (Iez ic);
(14)
where X, is weighting coefficient. This simplification eliminates the necessity of the text-speech time alignment Ø(s). Our approach can be seen as jointly optimizing three capabilities:
Speech recognition (CASR): The combined model learns transcribe speech audio into text. As use a pre-trained speech encoder and a pre-trained language model, this objective encourages the alignment and integration of each model's 's functionality.
5

---

Published as a conference paper at ICLR 2024
Transcript continuation (CLM): This reuses, maintains, and leverages the language model's ability to generate natural text as learned from its training scheme, for example, dialogue for a chat-optimized LM. Depending the utterance, the decoder may further learn to use paralinguistic cues from the prompt speech to favor certain completions.
Conditional speech synthesis (CRecom.) We reuse the language model's autoregressiive generation ability and direct it toward spectrogram reconstruction. As the teacher-forced transcript is available and the most "accessible" feature, the decoder learns perform text-to-speech. In this way, the model can synthesize the LM's arbitrary textual continuations at inference time, including words not found in training. Finally, we expect that good spectrogram-lest level continuations require the preservation of speaker, prosody, and channel effects from the original speech prompt.
3.3 INFERENCE
In inference, the speech prompt 2p is encoded by the speech encoder E, then projected by P. to the LM's dimension to give =he (Eq. 3). Utilizing ahe and the start-of-sentence (sos) token, the language model decodes text in an autoregressive manner: ÿ LM([Z)] sos)) until eos is emitted, where is concatenation of the predicted transeript and continuation [üp- De]- Following this, the language model decodes a spectrogram in an autoregressive manner. It predicts the next spectrogram feature estimate (t) using prompt features = text prediction and past estimated spectrogram features dc(5 ( 1). Past spectrogram estimates fo(S L 1) projected to the language model dimension: < t 1) = ht (f.( t 1)). Then, &²(()) is predicted at step t: (t(() [M sos, G, 20((5 t- ) The decoded output (t) is then projected the spectrogram domain using h Bc(t) h (1 (t)). Finally, a vocoder converts the predicted spectrogram f..
into a waveform signal.
EXPERIMENTS AND RESULTS
4.1 DATA AND PRE-PROCESSING
To empirically evaluate the performance of the proposed approach, conducted experiments the Libri-Light dataset (Kahn et al., 2020). Libri-Light is a 60k hour English dataset consisting of unlabelled read speech from Libri Vox audiobooks. For our training objective, the dataset was transcribed using a NST (Park et al. 2020) model trained on LibriSpeech (960 hours). We used a frozen neural vocoder, WaveFit (Koizumi al. 2022). with its default hyperparameters to convert the predicted spectrograms into raw audio. Our proposed model was trained using 64 TPUv4 chips (Jouppi et al., 2023), over duration of 48 hours. We give comprehensive table of hyperparameters in Appendix I. We consider predetermined set of 3-second prefixes denoted as 8 = 3sec. During training utterances with a length of less than 3 seconds are discarded. For Libri-Light, only 0.04% of utterances are less than seconds. To evaluate utilize the test-clean test set from LibriSpeech (Panayotov et al., 2013). We employ the first 3 seconds of each utterance in the test set a prompt to the models, excluding the ground truth transcripts. For semantic and acoustic quality, Spectron was trained with LM of 350 million parameters, while for the question answering task, Spectron was trained with LM of 1 billion parameters. The two models are identical except for the LM. In Sections 4.2.1 and 4.2.2 a 350M LM was used; in Section 4.2.3 1B LM was used.
model and the baseline models during testing,
4.2 BASELINES
We compare our method against existing spoken language models:
GSLM: We evaluate their best model, the HuBERT-L6 configuration with 200 token units, for conditional speech continuation. The model was trained on a filtered subset of Libri-Light (Rivière & Dupoux ))21) AudioLM: We utilize the Libri-Light trained model described in their work. The two A models we compare against differ in the number of SoundStream residual vector quantizer (RVQ) layers they generate. One model generates the top layers (3-RVQ), while the other model generates all 12 layers (12-RVQ). TWIST: We evaluate both the OPT-1.3B and LLaMA-7B- initialized versions of their models, which were trained towards quantized HuBERT representations.

---

Published as a conference paper at ICLR 2024
Their models were trained on Libri-Light, Spotify podcasts (Clifton et al. 2020). The People's Speech (Galvez et al. 2021 I and VoxPopuli (Wang et al. 30 SpeechGPT: We evaluate their open-sourced model, which based upon the LLaMA-7B model with HuBERT speech representations. This model termed SpeechGPI-7B-com, and was trained using all 3 training stages in SpeechGPT. The model
was trained using the Libri-LLight and SpeechInstruct datasets.
4.2.1 SEMANTIC QUALITY
We employ the log-pens pee-perpexxtty metric to evaluate the semantic quality of the speech output from the models. We use state-of-the-ard Conformer ASR system (Zhang et al.) 2023b) trained on proprictary English-only dataset to transcribe the speech continuation. Subscquently, we compute the log-perplexity of the predicted transcripts using GPT-2 medium (Radford et al. 2019) via the open- source transformer library (Wolf et al. 2020). The results presented in Table demonstrate the performance gains of our method compared to previous approaches such as GSLM, where our method achieves improvement of 170.91 in perppeeplextt.. Furthermore, when compared to the state-of-the-arr AudioLM method, our approach outperforms both the 3-RVQ and 12-RVQ variants, exhibiting enhancements of 12.88 and 14.20 respectively. Moreover, the results in Table T reveal that our method exhibits improved performance compared to existing cascade methods.
4.2.2 ACOUSTIC QUALITY
We consider two metries to capture acoustic quality and speaker consistency, respectively: Natu- ralness Mean Opinion Score (N-MOS; Nguyen et al.) 2023): This is reported solely for speech continuations. Human evaluators are tasked with assigning a rating on five-point scale to denote the perceived naturalness of given speech utterance, spanning from I (indicative of poor quality) to (indicative of excellent quality). Tests were conducted using 20 randomly sampled utterances from the LibriSpeech test-clean test set. 30 raters participated in the tests. The prompts to the raters. Avg. speaker similarity: We compute the speaker similarity between the input prompt and its generated continuation using the speaker encoder of the PnG-NAT TTS model (Morioka et al. 2022). We compute the speaker embeddings of both and measure the cosine similarity between each pair of embeddings. We report the average across the entire test set.
not available
As seen in T our approach performs better than GSLM in terms of N-MOS with an improvement of 0.55 absolute. When compared to AudioLM, our approach comparable to the 3-RVQ version and slightly inferior to the 12-RVQ version, with a decrease of 0.19 in N-MOS. One can see in Table ( that the results of TWIST are similar those of GSLM, and that Spectron outperforms the 1.3B and 7B versions by 0.4 and 0.65 respectively. SpeechGPT performs slightly inferior to Spectron, which outperforms it by a score of 0.3. 3 presents the results for average speaker similarity. Our method demonstrates significant improvement of 0.31 over the GSLM method. When compared to AudioLM, our method outperforms both the 3-RVQ and 12-RVQ versions, with increases of 0.05 and 0.07 in average speaker similarity, respectively. Moreover, comparing TWIST 1.3B and 7B, the proposed method improve the average speaker similarity by 0.18 and 0. respectively. These results indicate that comparable acoustic quality be achieved with Spectron's simpler approach. Our model is trained end-to-end and utilizes the universal speech representation encoded by spectrograms. Note that SpeechGPT does not intend to preserve speaker identity, which is why its average speaker similarity is lower.
Table Log- perplexity for completions of LibriSpeech utterances given a 3-second prompt. Lower is better.
Method
Log-perplese (L)
GSLM
296.99
AudioLM (3-RVQ)
138.96
AudioLM (12-RVQ)
140.28
TWIST (1.3B)
229.53
TWIST (7B)
170.81
SpeechGPT
136.42
Spectron (350M)
126.08
7

---

Published as a conference paper at ICLR 2024
Table 2: Naturalness Mean Opinion Score (N-MOS; Mean = SE) for completions of LibriSpeech utterances.
Table 3: Average Speaker Similarity metric
for completions of LibriSpeech utterances.
Method
Speaker Sim. (1)
Method
N-MOS (1)
GSLM
0.11
GSLM
3.13 + 0.32
AudioLM (3-RVQ)
0.37
AudioLM (3-RVQ)
3.61 0.29
M (12-RVQ)
0.35
AudioLM (12-RVQ)
3.87 = 0.32
TWIST (1.3B)
0.24
TWIST (1.3B)
3,28 - 0.24
TWIST (7B)
0.23
TWIST (7B)
3,03 = 0.22
SpeechGPT
0.05
SpeechGPT
3.38 = 0.30
Spectron (350M)
0.42
Spectron (350M)
3.68 + 0.29
Ground Truth
4.23 + 0.33
4.2.3 QUESTION ANSWERING
We propose examining whether the models can continue spoken sentences or questions with the appropriate answer. This can can be viewed as spoken generative QA; the correct answer must be produced out of infinite possible speech continuations. Note that except for SpeechGPT, all other methods (including ours) have not seen instruction data and are thus evaluated in a zero-shot fashion for spoken question answering. Given that the various spoken language models evaluated with 3-second input contexts, we use TTS (via the publicly-availeee available Google Cloud TTS service, voice to synthesize questions that fit within this duration. The questions drawn from an existing set and a new test set which we name LLLMA- I.LaMA-Questione Questions. WebQuestions (Berant al. 2013) is an open-ended text QA NLP dataset. The dataset contains open-ended questions that are answerable via the Freebase database and are centered around a single named entity. LLaMA- Questions is an open-domain world knowledge QA dataset that we synthesized using LLaMA-7B. We prompted the model to provide questions and short answers regarding various topics. Overall, we gathered 300 questions in this manner and generally verified the answers. Answer accuracy: We use Conformer ASR system (Zhang et al. (023b) to transcribe the answers of the models. If the text answer is contained in the transcript, we count the answer as being correct (as zero-shot models are merely continuing the prefix audio).
The results presented in Table demmonstaate the performance of the proposed model in comparison to other existing models. Specifically, the proposed model exhibits an accuracy of 22.9% on LLaMA- Questions, while SPeechGPT achieves a comparable accuracy of 21.9%. Note that contrast to SpeechGPT's utilization of larger model architecture comprising 7 billion parameters, proposed method uses a more modest billion parameter LM for comparable results. In contrast, TWIST models with 1.3 billion and 7 billion parameters demonstrate lower accuracies of 1% and 0.5% respectively. Upon careful examination, it becomes evident that these models predominantly generate completions of input questions rather than providing substantive answers. AudioLM 3-RVQ. AudioLM 12-RVQ and GSLM achieved accuracy of 7%, 6.7% and 4%, respectability, which is likely due the fact that the underlying Transformer architecture not pre-trained on large language model. Similarly, on the Web Questions test set, the proposed model attains an accuracy of 6.1%. while SpeechGPT yields comparable accuracy of 6.5%. Again, TWIST models with 1.3 billion and 7 billion parameters achieve accuracies of 0.7% and 1.1% respectively, further reinforcing the observed trend of completion-centric behavior rather than direct question answering. Additionally, models such AudioLM 3-RVQ, AudioLM 12-RVQ, and GSLM exhibit accuracies of 2.3%, 2.3%, and 1.5% respectively, which can likely be attributed to the absence of pre-training on a large-scale language model within the underlying Transformer architecture.
Audio samples and our spoken QA dataset can be found on the project website
4.2.4 ABLATION ANALYSIS
To understand the individual impacts various components within the proposed approach, an ablation study was conducted. We measure the log-perplexity over the test-clean test set of the LibriSpeech
8

---

Published as a conference paper at ICLR 2024
Table 4: Accuracy (%) on spoken question answering datasets.
Method
Web Questions (1)
LLaMA-Questions (1)
Zero-Shot
GSLM
1.5
4.0
AudioLM (3-RVQ)
2.3
7.0
AudioLM (12-RVQ)
2.3
6.7
TWIST (1.3B)
0.7
1.0
TWIST (7B)
1.1
0.5
SpeechGPT (7B)
6.5
21.9
X
Spectron (IB)
6.1
22.9
dataset (Panayotov et al. 2015). This study involved removing each specific component in isolation. (i) Disabled intermediate loss on text ("-Cce") (ii) removed spectrogram derivative loss ("-(C; 2,"" (iii) removed pre-training of the language model LM, letting it train from scratch (iv) removed pre-training of the speech encoder & and training from scratch (v) removed pre-training of both the speech encoder & and language model LM, training the entire model from scratch. The findings are summarized Table 5 The results demonstrate that each of the aforementioned components contributes to the overall performance enhancement of the proposed approach. Notably, the ASR & LM cross-entropy loss CcE and the spectrogram derivative loss L + C, have the most significant impact, leading degradation of 661.81 and 588.35 in the log-perplexity score, respectively. Furthermore, the incorporation of the pre-trained speech encoder and pre-trained language model exhibits discernible decline in performance, resulting in degradation of 87.17 and 75.63 in the log-perplexity score, respectively. Notably, when both the speech encoder and pre-trained language model removed, degradation of 118.31 in the tog-perplexity score is observed.
Table 5: Ablation analysis.
Model
Log-peeplexity (L)
Proposed Spectron (350M)
126.08
LCE
714.43
(Lr + L.)
787.89
Pre-trained LM
201.71
Pre-trained speech encoder
213.25
Pre-trained LM & speech encoder
244.39
5 LIMITATIONS AND FUTURE WORK
The limitation of our work is the high time and space complexity of generating spectrogram frames. Since spectrogram frames are computed with rate 12.5 ms, generation of long speech utterances is not possible. We hypothesize that potential solutions include generating multiple spectrogram frames from each hidden representation. Another limitation is that text and spectrogram decoding processes not parallelizable. This hinders the ability to Spectron in streaming scenarios and introduces small latency between audio input and output. We leave the development of a parallelized decoding algorithm for future work. We further recognize that biases in the pre-trained language model may be sustained in our model, we refer to Google (2023) for detailed discussion of ethical considerations for text-based language models.
CONCLUSION
We proposed Spectron, neural direct speech continuation model that can be trained end-to-end and operates in the spectrogram domain. We showed that a pre-trained language model can be given speech recognition and generation capabilities post-hoc, by fine-tuning continuation tasks using pre-trained speech encoder and a novel training objective. The result is a model that benefits from the pre-training of both models and outperforms previous spoken language models various metrics.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors would like thank Heiga Zen, Neil Zeghidour, Eugene Kharitonov, Tal Schuster, Bryan Richter, Christian Frank, Marco Tagliasacchi, Nadav Bar, and the rest of the Google Research team for helpful discussions and previous work on data preparation. The contribution of Alon Levkovitch is part of his Ph.D. thesis research conducted at Tel-Aviv University.
REFERENCES
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi. Matthew Sharifi, Neil Zeghi- dour, and Christian Havnø Frank. Musicl. Generating music from text. CoRR, abs/2301.11325,, 2023.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang. Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. SpeechTS: Unified-modal encoder- decoder pre-training for spoken language processing. In ACL (1), pp. 5723-5738. Association for Computational Linguistics, 2022.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common Voice: A massively- multilingual speech corpus. In LREC, pp. 4218 4222. European Language Resources Association, 2020.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In EMNLP, pp. 1533-1544, ACL, 2013.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:2523-2553 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim. Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. CoRR, abs/2204.02311,, 2022.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Chent Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In ASRU, pp. 244-250. IEEE, 2021.
Ann Clifton, Aasish Pappu, Sravana Reddy, Yongze Yu, Jussi Karlgren, Ben Carterette, and Rosic Jones. The Spotify Podcasts dataset. arXiv preprint arXiv:2004.0477 2020.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. CoRR, 0bs/2210.13438 2022.
10

---

Published as a conference paper at ICLR 2024
Keqi Deng, Songjun Cao, Yike Zhang, and Long Ma. Improving hybrid CTC/attention end-to-end speech recognition with pretrained acoustic and language models. In ASRU, pp. 76-82. IEEE, 2021.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of
deep bidirectional transformers for language understanding, In NAACL-HLT (1), pp. 4171-4186. Association for Computational Linguistics, 2019.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. CoRR, abs/2005, 00041,, 2020.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. arXiv preprint arXiv:2307. 11700 2023.
Daniel Galvez, Greg Diamos, Juan Torres, Keith Achorn, Juan Felipe Cerón, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. The People's Speech: A large- scale diverse english speech recognition dataset for commercial usage. In NeurlPS Datasets and Benchmarks, 2021.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305. 2023.
Google. PaLM technical report, 2023.
peim2teccreport.
Allen L. Gorin, Giuseppe Riccardi, and Jeremy H. Wright. How may help you? Speech Commun., 23(1-2):113-27,, 1997.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: aanoolutionauugmente transformer for speech recognition. INTERSPEECH, pp. 5036-5040. ISCA, 2020.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Textu- ally pretrained speech language models. In NeurIPS, 2023.
Curtis Hawthorne, Andrew Jaegle, Catalina Cangea, Sebastian Borgeaud, Charlic Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew M. Botvinick, Ian Simon, Hannah Sheahan, Neil Zeghidour, tean-Baptiss Alayrac, João Carreira, and Jesse H. Engel. General- General-purpose. long-context autoregressive modeling with perceiver AR. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 3335-8558. PMLR, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 19:3451-3460 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR. OpenRe- view.net, 2022.
Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang. Jinglin Liu, et al. AudioGPT: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304. 12995, 2023.
Kristiina Jokinen and Michael MeTear. Spoken dialogue systems. Synthesis Lectures on Human Language Technologies, 201):1-151, 2009.
Norman P. Jouppi, George Kurian, Sheng Li. Peter C. Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David A. Patterson. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In ISCA, pp. 82:1-82:14 ACM, 2023.
11

---

Published as a conference paper at ICLR 2024
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Serre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhoma- nenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. Libri- Light: A benchmark for ASR with limited or no supervision. In ICASSP, pp. 7669-7673. IEEE, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 100301, 2020.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text- free prosody-aware generative spoken language modeling. In ACL (!), pp. 8666-8681, Association for Computational Linguistics, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Yuma Koizumi, Kohei Yatabe, Heiga Zen, and Michiel Bacchiani. WaveFit: an iterative and non- autoregressive neural vocoder based on mxed-point iteration. In SLT, pp. 884-891. IEEE, 2022.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Proc. NeurIPS, 33:17022-17033, 2020.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte,
Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for
Computational Linguistics, 9:1336 1354, 2021. doi: 10.1162/tac a 00430. URL https
//aclanthology.c wrg. taa.taacc11.. 1.79
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Miguel Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech- to-speech translation on real data. In NAACL-HLT, pp. 860-8 872. Association for Computational Linguistics, 2022.
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding LLaMA: Advancing toxt-to-musie generation with question answering and captioning. arXiv preprint arXiv:2308. / 1276, 2023a.
Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Meng Cui, Qiushi Huang, Jinhua Liang, Yin Cao, Qiuqiang Kong, Mark Plumbley, al. WavJoumey: Compositional audio creation with large language models. arXiv preprint arXiv:2307. 14335, 2023b.
Nobuyuki Morioka, Heiga Zen, Nanxin Chen, Yu Zhang, and Yifan Ding. Residual adapters for few-shot text-to-speed speaker adaptation. arXiv preprint arXiv:2210.15800 2022.
Tu Anh Nguyen, Benoît Sagot, and Emmanuel Dupoux. Are discrete units necessary for spoken language modeling? IEEE J. Sel. Top. Signal Process., 16(6):1415-1423, 2022.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11: 250-266, 2023.
Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/21 1iss22122.01111 14, 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus bused on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 5206-52012 5210. IEFF, 2015.
12

---

Published as a conference paper at ICLR 2024
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph. Ekin D Cubuk, and Quoe Le. SpecAugment: A simple data augmentation method for automatic speech recognition. In INTERSPEECH, 2019.
Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung- Chengg Chiu, Bo Li, Yonghui Wu, and Quoc V Le. Improved noisy student training for automatic speech recognition. arXiv preprint arXiv:2005.09629 2020.
Linkai Peng, Baorian Nuchged, and Yingming Gao. Spoken language intelligence of large language models for language learning. arXiv preprint arXiv:2308. /4536, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8)99 2019.
Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Almost unsupervised text to speech and automatic speech recognition. In International conference machine learning, pp. 5410-5419. PMLR, 2019.
Morgane Rivière and Emmanuel Dupoux. Towards unsupervised learning of speech features in the wild. In 202/ IEEE Spoken Language Technology Workshop (SLT), pp. 156-163. IEEE, 2021.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue. Christopher Klamm, Colin Leong, Daniel van Strien, David Ifcoluwa Adelani, and et al. BLOOM: A 176B- parameter open-access multilingual language model. CoRR, abs/2211.055
2022.
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang. RJ-Skerrv Ryan, Rif A. Saurous, Yannis Agionyrgiannakis, and Yonghui Wu. Natural TTS synthesis by conditioning wavenet on Mel spectrogram predictions. In ICASSP, pp. 4779-4783. IEEE, 2018
Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non- attentive Tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling. arXiv preprint arXiv:2010.0443 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et LLaMA: Open and efficient foundation language models. arXiv preprint 2rX11:2302.13971,2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In ACL 202/-59th Annual Meeting of the Association for Computational Linguisties, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe V. Le, and Denny Zhou. -fain-of-thhughh prompting elicits reasoning in large language models. In NewrIPS, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-we natural language processing. In EMNLP (Demos). pp. 38-45. Association for Computational Linguistics, 2020.
13

---

Published as a conference paper at ICLR 2024
Shu-Way Wen Yang. Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi. Xuankai Chang, Guan- Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang. Shuyan Dong, Shang- Wen Li, Shinji Watanabe, Abdelrah- man Mohamed, and Hung-yi Lee. SUPERB: speech processing universal performance benchmark. In Interspeech, pp. 119--1198. ISCA, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- Stream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30: 495-507, 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM- 130B: an open bilingual pre-traines model. In ICLR. OpenReview.net, 2023.
Dong Zhang. Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In EMNLP (Findings), pp. 15757-15773. Association for Computational Linguisties. 2023a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. CoRR, ab/22205010688 2022a.
Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghami, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara N. Sainath, Pedro J. Moreno, Chung-Chor Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. Google USM: Scaling automatic speech recognition beyond 100 languages. CoRR, abs/2303.01037, 2023b.
Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, et al. SpeechLM: Enhanced speech pre-training with unpaired textual data. arXiv preprint ar arXiv:2209.15322 2022b.
Zihan Zhao, Yiyang Jiang, Heyang Liu, Yanfeng Wang, and Yu Wang. LibriSQA: Pioneering free-form and open-ended spoken question answering with a novel dataset and framework. arXiv preprint arXiv:2308 10390 2023.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 TABLE OF
Table 6: Model byyer-pprameee used in the experiments. ("xn": layers)
Input & Oulput
Sample rate (Hz)
16,000
Mel channels
128
Mel lower band (Hz)
20
Mel upper band (Hz)
8,000
Frame size (ms)
50.0
Frame step (ms)
12.5
SpecAugment
Freq blacks
Time blocks
Freq mask max bins
27
Time mask max frames
40
Time block length ratio
0.05
Speech Encoder
Conformer dims
1024
Attention heads
Conv kernal size
(3, 3)
Conv stride size
(2, 2)
Language Model
Transformer (dim layers)
1024
Dim per head
64
Hidden dims
4096
Num heads
Vocab size
256,000
WaveFit vocoder
Iterations
5 [5, 5, 2, 3 [160, 32, 512] [400, 80, 1024]
UBlock upsampling factors
STFT loss resolutions
Hann win size, frame shift, FFT size res 1 Hann win size, frame shift, FFT size res 2 Hann win size, frame shift, FFT size res 3 Multi-periol periop discriminator
1800, 160, 2048
Kong et al. 2020
Multi-g perii-ppeiod discriminator loss weight
Training
Optimizer
Adam (Kingma 2015
Learning rate schedule
Vaswan et al. 2017
Learning rate (peak)
3.5 x 10
Warm-up steps
8K
Batch size
128
Continuation loss weight X
0.1
Derivative loss order K
3
A.2 EXTENDED COMPARISON TO PREVIOUS METHODS
Given the variation in pre-trained models across the methods discussed in this paper, we find it fit to conduct a more comprehensive comparison of their performance. This detailed comparison is presented Table 7 Regarding the Word Error Rate (WER) assessment for ASR systems, the WER scores are sourced from the SUPERB benchmark paper (Yang et al. 2021 WERs are reported on the LibriSpeech test-clean test set. It's important to note that these scores rely solely on the speech encoder type due to limited data availability for all utilized models. For instance, models such as mHuBERT (Lee et al. 2022) employed in SpeechGPT and the New Frequency HuBert adapted and trained within TWIST exist solely tokenization models and lack dedicated ASR model forms.
15

---

Published as a conference paper at ICLR 2024
The performance comparison of speech encoders referenced in various methods within this paper depicted in Table 7 Notably, the performance of the speech encoders on the LibriSpeech test set comparable. However, concerning the language models (LMs) utilized, more variation is evident among the methods. LMs span spectrum, ranging from larger models such as SpeechGPT and TWIST employing 7B LMs, intermediate-sized models like Spectron and AudioL employing approximately IB LMs, and finally, GSLM utilizing smaller 200M parameter LM. It is widely acknowledged that LM performance significantly influenced by model size (Kaplan et al. 2020). Moreover, diverse datasets have been employed across these systems, including LibriSpeech, Libri- Light. SpeechInstruct, VoxPopuli (Wang al. (021). Common Voice (Ardila et al. 2020), Spotify (Clifton al. 2020). and The People's s Speech (People: People:GGalvez
Galvez et al. 2021).
Table 7: Comparison of different models mentioned in this paper.
Detail
Spectron
SpeechGPT
TWIST
GSLM
AudioLM
LM #Params
IB / 350M
7B
IB/7B
~00M
0.9B
LM Type
PaLM-2
LLaMA
OPT/LLaMA
None
None
Speech Encoder
USM
mHuBERT
T-HuBERT
HuBERT
Vav2Vec
Speech Encoder
600M
317M
317M
317M
600M
#Params
WER of Speech Encoder
3.1
2.94
2.94
2.94
3.1
LibriSpeech VoxPopuli
Speech Encoder
Web-scale LibriSpeech
LibriSpeech Libri-Light
VoxPopuli 100k
Common Voice
Libri-Light
Dataset
Spotify
Fisher
LibriSpeech
LibriSpeech Speech Instruct
Spotify
Training Dataset
Libri-Light
Libri-Light
Libri-Light
People
VoxPopuli
#Training
Examples
60k
60k + 38k
150k
60k
60k
16


Reviews:
[{"review_id": "Q0iMzWBA5i", "reviewer": "Reviewer_6UiY", "summary": "This paper proposes a new training scheme to build a large speech and language model. The idea is to split the paired speech utterance into the first 3-second segment and the rest. The first task is to predict the corresponding sentence given the first 3-second segment. This corresponds to 1) performing ASR corresponding to the first 3-second audio segment and 2) predicting the rest of the sentence conditioned by the first 3-second audio segment and its transcriptions, corresponding to the language modeling task if we ignore the first 3-second audio segment. Note that the actual implementation does not require the split in the text part. The second task is to perform the speech continuation task of the rest of the audio segment conditioned on the first 3-second audio segment and previously estimated sentence. Thus, this training scheme holds ASR, text continuation, and speech continuation tasks with a single training framework with the standard paired speech utterances. This model is built upon various speech and text pre-trained models and is fine-tuned with the public speech database based on Libri-Light. The paper also has multiple comparisons with the other methods for speech continuation and spoken QA tasks and shows the superiority of the proposed method.", "questions": "- Can you evaluate the ASR performance of this method? It would probably not be nice due to the over-prediction function in the ASR prediction phase (it will predict more than what was spoken), but it would be an interesting result to report. \n  - I have the same question for TTS. In this case, due to the lack of conditions, it may not work at all (?).\n- Why is it 3 seconds? Would it robustly work even if we throw more than (or less than) 3-second spoken prompts?\n- Similarly, how did you deal with sentences that are less than 3 seconds during training? Can you discard them or concatenate neighboring sentences to make it longer than 3 seconds?\n- How about combining multiple sentences and using the original first sentence as a chunk instead of a 3-second chunk? In this case, we would also have access to the transcription corresponding to the spoken prompt part easily. We can have more precise control of the text output to perform ASR inside the framework explicitly.\n- Around equation (3), $\\mathcal{P}_s$: why does $\\mathcal{P}_s$ depend on position $s$?\n- In section 4.1, the last sentence, \"For semantic and acoustic quality, Spectron was trained with a model of 350 million parameters, while for the question answering task, Spectron was trained with a model of 1 billion parameters,\" I could not understand it. Do you mean you use 350M models for Sections 4.2.1 and 4.2.2 and 1 B models for Section 4.2.3? How are they different?\n- Is the question-answering task performed with zero-shot without fine-tuning it? Please clarify it.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Building a speech foundation model by leveraging an LLM is a hot topic in ML and AI. Also, providing more powerful understanding capability for speech models is desired.\n- A novel training algorithm to (implicitly) perform ASR, text continuation, and speech continuation tasks with the standard paired speech utterances.\n- The paper shows a strong performance compared with other speech LM methods.", "weaknesses": "- The paper lacks the reproducibility and accessibility of the results due to several pre-training models, which are not publicly available or cannot be reproduced due to the inaccessible training data (e.g., WaveFit and state-of-the-art Conformer ASR system (Zhang et al., 2023b)). I appreciate your efforts in mitigating the issue (e.g., the use of Libriright and the release of the SQA test set), but I think the paper still has this issue.\n- Due to the above issue, it is not clear whether the superiority of the proposed method compared with other speech LMs comes from their novel training schemes or strong pre-trained models. Thus, the effectiveness, especially for the comparisons with other speech LMs, is weak.\n- Clarity: Section 2 requires more improvements. There are many different aspects between the related studies and the proposed method, and it is not clear what is the advantage of this method. I recommend you rewrite Section 2 to categorize the different aspects (e.g., model architecture, training method, pre-training data, etc.) and emphasize the distinction between the proposed method and others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xKTT8I591x", "reviewer": "Reviewer_hbZt", "summary": "This paper introduces a new spoken language model that operates on spectrograms and can simultaneously perform speech recognition, speech continuation, and text continuation (auto-regressive language modeling). The authors propose a model with a speech encoder, an LLM decoder, and a post network that takes in a sequence of speech input to produce its transcription, textual completion, and speech completion.  Experiments on LibriLight show that the proposed Spectron obtains better acoustic and semantic quality compared to other work. They also extend the approach to Spoken Question Answering and show that Spectron can do this almost as well as SpeechGPT.", "questions": "A. No details are provided on MOS tests - could the authors share some details ?\n\nB. What was the WER of the Librispeech model used to obtain pseudo-labels on LibriLight ? Do the authors have any idea about label quality ? \n\nC. The paper says \"we use TTS to synthesize questions that fit within this duration\" - what does this mean? Do you truncate long questions or modify the durations of phones/words to fit within 3s? Or do you drop questions that are longer than 3s?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is clear and simple. Experimental results demonstrate benefits from this approach compared to others. \n\n2. The idea of prompting order imposed that the authors claim resembles CoT is interesting.", "weaknesses": "1. The MOS evaluation of synthesis is bereft of details - it is important to mention the number of raters and the number of examples rated. The lack of information makes it hard to contextualize reported MOS scores. Further, since this paper is evaluating speech continuation as opposed to text-to-speech synthesis (TTS), it is unclear if standard MOS is the best evaluation strategy. Do the raters hear the original prompt before hearing the continuation from the model? Does the MOS in this case measure naturalness of the continuation or something else ?\n\n2. The CoT idea is interesting as I mentioned before, but this paper could benefit from some analysis on how inter-dependent these outputs are in the trained model. That is, if the transcription contains errors, does that necessarily lead to poor text or speech continuation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tktImsFr9Z", "reviewer": "Reviewer_zUkq", "summary": "This paper proposes a novel approach to adapt the pre-trained LLMs for spoken question answering and speech continuation. Specifically, a pre-trained speech model as well as some projection layers such as connector layer, pre-net, and post-net are utilized to process speech signals in the same way as text tokens in the LLMs. In this way, the adapted LLMs can take speech inputs and generate speech outputs and train speech recognition, text continuation, and speech synthesis using only paired speech-text pairs in an end-to-end fashion, where the text transcripts can be treated as a cross-modal chain-of-thought. The proposed spectrogram-powered LLMs surpasses existing spoken language models in speaker preservation, semantic coherence, as well as spoken question answering.", "questions": "Will Spectron still work when using a LLM that has 100B parameters? Currently, Spectron uses 1B language model (PaLM 2), and that is much smaller than state-of-the-art LLMs. In addition, what happens if you use Whisper (https://cdn.openai.com/papers/whisper.pdf) to transcribe Libri-Light and obtain training transcripts $y$ ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality:** The concepts and methods of Spectron is novel. It adapts exiting pre-trained LLMs for spoken language modeling, and trains an end-to-end system that takes speech inputs and generates speech outputs.\n\n**Quality:** This paper has solid experimental results. The proposed system doesn't achieve state-of-the-art results in some metrics though.\n\n**Clarity:** This paper is well-organized and well-written.\n\n**Significance:** The impact of the paper is good. However, the scale of the proposed system is relatively small compared to OpenAI voice-version ChatGPT.", "weaknesses": "I don't see major weaknesses. It would be great if the authors can also compare their system with OpenAI voice-version ChatGPT (https://openai.com/blog/chatgpt-can-now-see-hear-and-speak).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s01dLzV90u", "reviewer": "Reviewer_Qrsa", "summary": "In this paper, the authors leverage a pretrained Language Model (LLM) and a pretrained speech encoder to tackle spoken question answering and speech continuation tasks. The proposed model processes spectrograms directly as inputs and outputs while utilizing a novel end-to-end objective for training. This objective implicitly supervises speech recognition, text continuation, and conditional speech synthesis tasks. Comprehensive experimental results validate the semantic and acoustic quality of the proposed model, showcasing its efficacy.", "questions": "1. Could you please provide additional information about the training and inference in the Spoken QA task? Does SPECTRON undergo training for the spoken QA task, whereas the baselines, such as AudioLM and GSLM, does not?\n\n2. In the provided Spoken QA demos, SpeechGPT tends to generate lengthier and more detailed answers, while SPECTRON prefers more concise responses. Could the comparable performance of both models be attributed to the nature of the QA task, such as the prevalence of shorter answers in the test set?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work introduces a new end-to-end training paradigm for spoken language modeling, efficiently employing the pretrained Language Model (LLM) to enhance semantic quality.\n2. The experimental results, along with the audio samples in the supplementary material, confirm the semantic and acoustic performance of the model.\n3. The availability of the released test set offers the research community a valuable benchmark for assessing the semantic quality of spoken language models.\n4. The presentation is clear and easy to understand.", "weaknesses": "1. The association between generated text and speech can be further evaluated. For instance, performing a Word Error Rate (WER) test using speech, [xp, xc], and transcriptions [yp, yc] would be advantageous. In addition, replacing the text with a sequence of null tokens of the same length can help showcase the impact of text on semantic quality. These analyses would lead to a better understanding of the role that text plays in the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)