SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Authors: Dong Zhang
Shimin Li
Xin Zhang
Jun Zhan
Pengyu Wang
Yaqian Zhou
Xipeng Qiu
Venue: EMNLP 2023 Findings (2023)
Keywords: large language model
speech
multi-modal

Abstract: Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence~(AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. With discrete speech representations, we construct SpeechInstruct, the first large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow cross-modal human instructions and highlight the potential of handling multiple modalities with one model. Code and models are available in \url{https://github.com/0nutation/SpeechGPT}. Demos are shown in \url{https://0nutation.github.io/SpeechGPT.github.io/}.

SpeechGPT:
Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang,
Yaqian Zhou, Xipeng Qiu
School of Computer Science, Fudan University
Shanghai Key Laboratory of Intelligent Information Processing, Fudan University dongzhangg22, xin xeeeeiinnnnnaaa___ zhang22, (al2,,zhanti jzhan22, pywang22 @m. fudan edu. on [smli200 zhouyagian, xpqiu) fudan. enu.cn
Abstract
What French
of
Multi-modal large language models are re- garded as a crucial step towards Artificial Gen- eral Intelligence (AGI) and have garnered sig- nificant interest with the emergence of Chat- mod-
(Whati rame?)
Speech GPT
the "Tnday
GPT. However, current els typically adopt cascade paradigm, pre- venting modrl knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. With construct
Record the content: -f...t.-...-t................... It...f
content of speech gaza
Figure I: SpeechGPT's capabilities to tackle multiple cross-modal tasks.
discrete speech representations, Speechlnstruct, the first large-scale cross- modal speech instruction dataset. Addition- ally, employ three-stage training strat- egy that includes modality-adapptatoon pre- training, gross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow cross-modal human instructions and highlight the potential of handling multiple modalities with model. Code and models are available in https: h//ggttuul con/ Demos are
like images and speech cannot be adapted directly to LLMs that receive discrete tokens.
model mainly adopts cascading paradigm (Huang al., 2023a) i.c., the LLM is connected with an automatic speech recognition (ASR) model or a text-to- speech (TTS) model in tandem, or the LLM is employed as a control hub, with several speech processing models (Cheng et al., 2023a,b,c) are integrated to cover multiple audio or speech tasks (Huang et al., 2023a; Shen et al., 2023). Some prior work generative spoken language mod- els involves encoding the speech signal into dis- crete representation (Baevski et al., 2020; Hsu al., 2021; Zhang et al., 2023a) and modeling it with language models (Lakhotia et al., 2021; Borsos et al., 2022: Zhang et al., 2023d; Wang et al., 2023; Zhang et al., 2023c).
The current
Onutation weatttio//pppeehpp shown in https: //nut/ttt/////iinnnon github. htt//peechGP..gggm git.bb..th.oo.o
1 Introduction
Large language models (OpenAI, 2023; Touvron et al., 2023) have performed astonishingly on vari- ous natural language processing tasks. Meanwhile, multi-moda large language models, such GPT- 4, PALM-E (Driess et al., 2023), and LLaVA (Liu al., 2023), have explored the ability of LLMs to understand multi-modal information. However, significant gap exists between current LLMs and general artificial intelligence (AGI). First, most cur- rent LLMs can only perceive and understand multi- modal content but cannot spontaneously generate multi-modal content. Second, continuous signals
While capable of perceiving and generating speech, the existing cascaded methods or spoken language models still have several limitations. First, the LLM in the cascaded model only functions content generator. Since the representations of speech and text not aligned, the LLM's knowl- edge cannot be transferred to the speech modality. Second, the cascade approach (Shen et al., 2023; Huang et al., 2023a) suffers from the loss of par- alinguistic signals such as emotion and prosody. Third, existing spoken language models (Wang
"oorresponding author

---

al., 2023; Zhang et al., 2023d) only synthesize speech but fail to comprehend its semantic infor- mation, preventing them from achieving true cross- modal perception and generation.
modal contents.
We construct and release SpeechInstruct, the first large-scale speech-text cross- cross-modal instruction- following dataset.
In this paper, we propose SpeechGPT, a large
We build the first spoken dialogue LLM with strong human instruction following ability and spoken dialogue ability.
language model with intrinsic cross-modee conver- sational abilities, capable of perceiving and gen- crating molti-modall content. We perform speech
We show great potential to incorporate other modalities into LLMs through discrete represen- tations.
discretization with a self-supervised trained speech model to unify the modality between speech and text. The discrete speech tokens are then expanded into the vocabulary of the LLM, thus endowing the model with an inherent competence to perceive and generate the specch.
2 Related Work
Multi-modal Large Language Model Current multi-modal LLMs predominantly focus the vi- sual domain, feeding continuous representations obtained from pre-trained visual encoders into LLMs, facilitating full-parameter or parameter- efficient training on visual-language data (Ope- nAI, 2023; Huang et al., 2023b; Zhang et al., 2023b). Palm-E (Driess et al., 2023) integrates the 540B PaLM (Chowdhery et al., 2022) and 22B Vision Transformer (Dosovitskiy et al., 2021) into the largest vision-language model. LLaVA (Liu et al., 2023) leverages pre-trained CLIP (Radford et al., 2021) visual encoder and LLaMA (Touvron et al.. 2023) and conduct instruct tuning on GPT4- assisted visual instruction data. X-LLM (Chen et al., 2023) converts multi-modalities into repre- sentations with X2L interfaces as the inputs of the large language model. However, such structures only enable LLMs process multi-modal input, without ability to generate multi-modal output. Di- verging from prior studies, our approach empha- sizes the development a speech-centric multi- modal LLM, endowing it with the proficiency to accommodate both multi-modal input and output. Generative Spoken Language Model Discrete self-supervised representation based spoken gen- crative language modeling is making remark- able progress on large-scale speech dataset train- ing (Nguyen et al., 2022). AudioLM (Borsos et al., 2022) proposes to model speech based on audio codecs together with semantic codes, which can synthesize speech in a textlesss setting. VALL- E (Wang et al., 2023) builds generative spoken language model on audio codecs and treat Text- to-Speech conditional generation task. How- ever, these models are designed for a specific task and failed to benefit from L.MMs. SpeechGPT is built upon the foundation LLM and trans- fers LLM'ss knowledge to speech modality, con-
To provide the model with the capacity to handle multi-modal instructions, we build the first speech-text cross-modal instruction-folloiing dataset SpeechInstruct, Specifically, we discretize the speech discrete units (Hsu et al., 2021) and construct the cross cross-modal unit-text pair based on the existing ASR dataset. Meanwhile, we construct hundreds of instructions for diverse tasks with GPT- 4 to simulate actual user instructions as illustrated in Appendix B. In addition, to further enhance the model's cross-moda capability, designed the Chain-of-Modality instruction data, i.c., the model receives the speech command, thinks about the process text, and then outputs the response in speech.
For better cross-modal transfer and efficient training, SpeechGPT undergoes a three-stage train- ing process: modality-adaptation pre-training, cross-modal instruction siee-tnning, and chain-of- modality instruction fine-tuning. The first stage en- ables speech comprehension for SpeechGPT with the discrete speech unit continuation task. The sec- ond stage employs the SpeechInstruct to improve the model's cross-modal capabilities. The third stage utilizes parameter-efficiee LoRA (Hu et al., 2021) fine-tuning for further modality alignment. To evaluate the effecttivnness of SpecchGPT, we conduct a wide range of human evaluations and case analyses to estimate the performance of SpeechGPT on textual tasks, speech-text cross- modal tasks, and spoken dialogue tasks. The re- sults demonstrate that SpeechGPT exhibits a strong ability for unimodal and cross-modal instruction following tasks.
Our contributions include the following:
We build the first multi-modal large language model that can perceive and generate multi-

---

sequently obtaining better task generalization and human-instruction following ability.
into OpenAl GPT-4 generate task descriptions. Our generation method yields 100 instructions for each task and some examples are shown in Ap- pendix B.
Speech-Enabled LLM Interaction Following the emergence of ChatGPT, several studies have concentrated on the integration of expert speech models with LLMs to enable direct speech interac- tion with LLMs. HuggingGPT (Shen et al., 2023) facilitates task decomposition of human instruc- tions by LLMs and allows the invocation of mod- els from Huggingface to accomplish specific tasks, encompassing a range of automatic speech recog- nition (ASR) and text-to-speech models. Audio- GPT (Huang et al., 2023a) leverages a variety of audio foundation models to process complex audio information and connect LLMs with input/output interface (ASR, TTS) for speech conversations. However, these models exhibit increased complex- ity, demand extensive resources, and are prone to the unavoidable error accumulation problems. Our approach enables speech interaction with LLMs without relying on ASR or TTS systems, circum- venting the aforementioned drawbacks.
Instruction Formatting For discrete unit se- quence U and its associated transcription T, we determine whether it will be used for construct- ing an ASR task or TTS task based the probability p. Subsequently, we randomly select a description D from the corresponding task de- scription. This results in a triplet consisting of the task description, discrete unit sequence, and transcription, denoted as (D,U, Following this, the triplet is assembled into instruction using the template: [Human]:[D) This is input: {U] <eob>>[ppeccGGPT] <eo>s....
3.2 Chain-of-Modality Instruction
Speech Instruction Generation Due to the lack of instruction data with speech input and speech output, we trained a text-to-unit generator to con- vert text instruction data into speech instruction data. Specifically, the text-to-unit generator adopts Transformer encoder-decoder architecture. We trained it LibriSpeech unit-text pairs in Cross- modal Instruction. We select 37,969 samples from the moss- 002-s0002sst dataset whose response length is shorter than 35 words. And we convert both their instructions and responses into unit se- quences through the text-to-unit generator. As result, we obtained 37,969 quadruplets com- posed of speech instructions, text instructions, text responses, and speech responses, denoted (SpeechI, TexlI, TealR, SpeechR). Instruction Formatting Using the above quadru- plets, we could construct thain-of-thougle style instructions for four input-output formats, namely Speech Instruction-Speed Response, Speech Instruction-Tex, Response, Text Instruction-Speech Response, and Text Instruction-Tex Response. Their corresponding templates be found in Ap- pendix C.
3 SpeechInstruet Construction
Due to the limitations in publicly available speech data and the lack of variety of speech-text tasks, we construct SpeechInstruct, a speech-text cross- modal instruction-followiei consists of two parts, the first part is called Cross- Modal Instruction, and the second part is called Chain-of-Modality Instruction. The construction process of SpeechInstruct is illustrated in Figure 2.
dataset. This dataset
3.1 Cross-modal Instruction
Data Collection We collect several large-scale English ASR datasets to construct Cross-Modal Instruction, including Gigaspeech (Chen et al., 2021), Common Voice (Ardila et al., 2020), and LibriSpeech (Panayotov al., 2015). We employ mHuBERT' as the speech tokenizer to discretize speech data into discrete units and remove the repet- itive units of adjacent frames to get reduced units. Ultimately, we obtain 9 million unit-text data pairs. Task Description Generation We generate ASR and TTS task descriptions that are compatible with speech-text data pairs. Unlike the Self-Instruct method (Wang et al., 2022), we generate descrip- tions through a zero-shot approach. Specifically, we directly input the prompts shown in Appendix
3.3 Speechlnstruet Evaluation Set
We constructed cross-modal dialogue datasets under different scenarios to evaluate whether SpeechGPT could take on various roles. Specif- ically, these included a talking encyclopedia, per- sonal assistant, chat partner, poet, psychologist.
'nttps://d //d) riba publ ioft feriless les. com/ bubert//nubert, base, vn_en_er frr fr3.t3.pp
Pttpps: //uggg wtt.ang...e/...eeeeeeeeeoooeoeee httee/////aatsts///////a moss 002- sf data

---

Meta Prompt
Speech Datasets
Instruction-tuning Datarets
Transcription:
Hi, name SpeechGPT. Nice meet
Unit Vocodar
Speech2Unit
Text2Urit
[peechGPPT
<5P <BS
<597>
<31>
Instructions
Template,
Template, !
Croes-model Instructio data
Chain Modality
SpeechGPT
Instraction data
Speechinstruet
[Human]]
<4 <2> <84>
<35
<534>
Trancrtbe spesec This
Dowechiriiyy b crnnentatco rasss.
Discrete Speech Unit Extractor
Transcription:
(Hamant The the spocih instucttoon Sipeett You can do stop slap. You rochn
Good morning. what your name?
Figure Left: An overview SpeechInstruet construction process. The SpeechInstruct dataset consists parts: Cross-modes Instruction data and Chain-of- Modalltty Instruction data. Template, is shown in 3.1. Template, is shown in Appendix C. Right: An illustration of SpeechGPT model structure.
and educational assistant. For each role, pro- vide 10 manually authored instruction-response pairs written by ourselves. We use a pre-trained texe-to-speech model to convert the text into cor- responding speech. We then employ mHuBERT to discretize speech data into discrete units as de- scribed in Section 3.1. Ultimately, for each role, we obtained 10 quadruplets composed of speech instructions, text instructions, text responses, and speech responses.
means model further converting these representa- tions into sequence of cluster indices. Subse- quently, adjacent duplicate indices are removed, resulting
discrete units sequence represented as U = (u1, U2,..... WT), 44 E 0, 0, K -, V1 < i is T, with denoting the total number of clusters.
Large Language Model We employ the Meta AI LLaMA (Touvron et al., 2023) model as our Large Language Model. LLaMA comprises bedding layer, multiple transformer blocks, and an LM head layer. The total number of parame- ters LLaMA ranges from 7B to 65B. Drawing from an extensive training dataset of 1.0 trillion tokens, LLaMA demonstrates competitive perfor- mance compared to the substantially larger 175B GPT-3 across various NLP benchmarks.
4 SpeechGPT
4.1
Model Structure
A unified framework is designed to provide archi- tecture compatibility across different modalities. As shown in Figure 2, our model consists of three main components: discrete unit extractor, large language modal and unit vocoder. Under this ar- chitecture, LLM can perceive multi-modas inputs and generate multi-modal outputs.
Unit Vocoder Due to limition of single speaker unit vocoder in (Polyak et al., 2021), train multi-speaker unit HiFi-GAN to decode the speech signal from the discrete representation. The HiFi- GAN architecture consists of generator and multiple discriminators D. The generator uses look-up tables (LUT) to embed discrete representa- tions and the embedding sequences are up-sampled by series of blocks composed of transposed con- volution and residual block with dilated layers. The speaker embedding is concatenated to each frame in the up-sampled sequence. The discrimi- nator features a Multi-Period Discriminator (MPD) and Multi-Scale Discriminator (MSD), which have the same architecture as (Polyak et al., 2021).
Discrete Unit Extractor The discrete unit ex- tractor utilizes the Hidden-unit BERT (HuBERT) model (Hsu et al., 2021) to transform continuous speech signals into sequence of discrete units, HuBERT is self-supervised model that learns by predicting discrete labels for masked audio segments based on k-means clustering applied to the model's intermediate representations. fea- tures combination of 1-D convolutional layers and Transformer encoder to encode speech into continuous intermediate representations, with k-
/eetts://huuee agface.com tccbbook/////////ccooooocoooc./ fast speesse-eehheee -nn- 1 sspeech

---

4.2 Training
modalities utiliizing paired data. We mix Cross- modal Instruction in SpeechInstruet with moss-002- sft dataset to derive mix dataset I, which consists of samples T1, T2...., Tz- We fine-tune the model L obtained from the first stage on I. Each sample T3 consisting of 11,12,-..,, Iuy is formed by concatenating a prefix and a text. The training objective is minimize the negative log- likelihood and the loss calculation only considers the text part, ignoring the prefix, which can be for- mated as:
To incorporate speech discrete representation into LLM, expand the vocabulary and corresponding embedding matrix first. We divide the training pro- cess into three stages. The first stage Modality- Adaptation Pre-training on unpaired speech data, The second stage is Cross-modal Instruction Fine- Tuning. The third stage is Chain-of-Modality In- struction Fine-Tuning.
Expanding Vocabulary Given original LLM vo- cabulary V of size [V]. integrate speech discrete representations into LLM, we expand the vocab- ulary with an additional set of unit tokens V', of size |V* K. The expanded vocabulary V is the union of the original vocabulary V and the new words V
M/
[(L Σ j=1 pp
Σ
log P(aasltesg L) (4)
where x is the number of samples in corpus 7, yj is the total number of tokens in sample Tj. Pý is the number of tokens in the prefix part of Tj, and t. represents the i-th word in Tj.
=VvV
(1)
We denote the original word embedding matrix as E RdVidd where d is the dimension of word embeddings. To accommodate the expanded vo- cabulary, we need to create a randomly initialized word embedding matrix E' E IR V" xd We pre- serve the original word embeddings by copying the values of E to the first V rows of E":
Stage 3: Chain-of-Modality Instruction Fine- Tuning After obtaining the model in stage 2. we utilizes parameter-efficient Low-Rank Adap- tation (LoRA) (Hu et al., 2021) to fine-tune on Chain-of of-Modalit Instruction in SpeechInstruct. We add LoRA weights (adapters) to the attention mechanisms and train the newly added LoRA pa- rameters. We adopt the same loss function as stage 2.
Eol [V.:-= E
(2)
Finally, we replace the original vocabulary and word embedding matrix with the new vocabulary V" and the word embedding matrix E'. Stage 1: Modality-Adapttaioo Pre-training To enable LLM to handle discrete units modality, we utilize unlabeled speech corpus to train LLM in next-token prediction task. This approach aligns with the text pre-training objective of LLM. Given unlabeled speech corpus C consisting of speech U1, .u...., : Um and L.M denoted as L,, the nega- tive log-likellng likelihodoo loss can be formulated as:
Experiments
5.1 Experimental Setups
Datasets For modality-adaption pre-training, we use LibriLight (Kahn et al., 2020) which contains 60K hours of unlabelled English audiobook speech. For cross cross-modal instruction fine-tuning stage, we use Gigaspeech (Chen et al., 2021), Common voice (Ardila et al., 2020) and LibriSpeech (Panay- otov et al., 2015) dataset and moss-002-sft-data dataset, which illustrated in detail in 3.1. For chain-of-modality instruction fine-tuning stage, we use mosss-002-sft-die dataset, which illustrated in detail in 3.2.
C(LIC)
log P(u Meifj L)
Configuration We employ LLaMA-13B 13B (Touvron et al., 2023) our backbone model for a trade-off between performance and computational resources available. For stage 1, we use 96 A100 GPUs and train for 900 steps with batch size 768. For stage 2, we use 96 A 100 GPUs and train for 2100 steps with batch size 1536. For stage 3, we use A100 GPUs and train for 4200 steps with batch size 128.
where m is the number of speech in dataset C. nj is the number of discrete unit token in speech Uj. and uy represents the i-th unit token in the j-th speech.
Stage 2: Cross-modal Instruction Fine-
Tuning In this stage, we align speech and text

---

Details about training hyperparameters are shown in Appendix D. For decoding, we set the maximum sequence length to 2048 and set the temperature to 0.8. We use Top-k sampling with k=60. We also use Top-p sampling with p=0.8.
Human Opinion Score Following (Nguyen al., 2022), we calculate the human opinion score of the generated examples through crowdsourcing. These opinions are based on two dimensions: the con- tent mean opinion score (CMOS) for content and meaningfulness quality, and the naturalness mean opinion score (NMOS) for speech naturalness and fluency. For CMOS, we ask participants to fo- cus the correctness of the content in speech ferred to Speech-Alpacat 13B, consists of an off- the-shell ASR system +, Alpaca 13B (Taori al., 2023) as well pre-trained TTS system $ The second model, named Barrech-LLMMA-Mee incorporates the same ASR and TTS system, along with large language model obtained by perform- ing supervised fine-tuning on LLaMA-13B using MOSS-sftt-02 as the training dataset.
5.2 Baselines
We establish two cascaded cross-modal conversa- tional systems as our baselines. The first model, re-
text, without paying attention to the quality of the speech. For NMOS, we direct participants to fo- cus on the quality, smoothness, and naturalness of the speech, without considering its content. We in- vited five volunteers to perform the evaluation, and asked them to rate within range of 1-5, where represents the worst and 5 represents the best. For speech-to-spece
instruction-follwviii and text- to-speech instruction-following tasks, we calcu- late both CMOS and NMOS. For speech-to-text instruction-followinn and text-to-text instruction- following tasks, we calculate CMOS.
5.3 Evaluation
We evaluate the cross-modal instraction-following capabilitiess of SpeechGPT across four tasks: speech-lo-speech instruction-followinng (S2SIF), speech-to-ffe to-text instruction following (S2TIF), text- to-speech instruction-follwwii (T2SIF), and text- to-text cnstruction-ooloviinn (T2TIF).
5.4 Main Results
Content As shown in Table 1, taking into ac- count the comprehensive evaluation of ChatGPT Score and CMOS, SpeechGPT demonstrates perior performance on speech instructions (S2SIF and S2TIF) compared to the two baseline systems. This indicates that SpeechGPT outperforms the ASR model in the cascaded system when it comes to understanding speech content. From the per- spective of CMOS, SpeechGPT achieves perfor- mance similar to the baseline systems on T2SIF and T2TIF tasks, indicating that SpeechGPT still possesses commendable text and speech generation capabilities. In S2SIF and T2SIF tasks, ChatGPT Score and CMOS values exhibit ambiguity the ground truth and baseline systems. This can be attributed to speech responses being synthesized by TTS system, which can have errors in pauses be- tween sentences. This introduces significant errors for longer responses, leading to incorrect text after being processed by the ASR system, thereby reduc- ing the ChatGPT score. However, humans can un- derstand the content of such speech, so the CMOS score normal. Cases of cross-modal instruction- following can be found in Appendix G.
Data We randomly select 40 samples from the AlpacaEvval dataset and use the pre-trained TTS model in Section 3.3 to convert the text into cor- responding speech. We then employ mHuBERT discretize speech data into discrete units as de- scribed in Section 3.1. These are combined with the SpeechInstruct Evaluation Set to constitute our test set, which contains 100 samples. Each sample is a quadruplet composed of a speech instruction, text instruction, text response, and speech response. We denote them ground truth.
ChatGPT Score We utilize ChatGPT (GPT- 3.5-turbo) to assess the cross-modal instruction- following performance. For tasks that include speech, we leveraged the pre-trained ASR model in section 5.2 to transform the speech into its corre- sponding text, which is subsequently submitted for evaluation. Inspired from (Zhou et al., 2023), feed the prompt in appendix F to ChatGPT score the model's 's outputs based on response quality, with scores ranging from to
Speech Quality As shown in Table SpeechGPT exhibits significantly higher NMOS values com- pared to the baseline systems. This indicates that the speech responses generated by SpeechGPT out-
pastspeech2-en--1speee
'https://g' wtw..coottttttooooooooo haw.tatttsuaaaaaaaaaaaaaaaaaaa paca__
eval

---

ChatGPT Score
Human Opinion Score
CMOS
NMOS
Methods
S2STF
S2TTF
T2SIF
T2TTF
S2STF
S2TTF
T2SIF
T2TTF
S2STF
S2T1F
T2SIF T2TIF
Ground Truth
2.85
3.74
2.91"
3.93
3.78
3.89
3.95
4.12
3.18
3.20
Baselines: cascaded cross conversational systemss
Speech-Alpaca 13B
2.74
3.31
2.71
3.83
3.39
3.42
3.71
3.75
3.12
3.13
Speech-LLuMA-MOSAA
2.87
3.50
3.23
3.82
3.38
3.44
3.74
3.83
3.14
3.11
SpeechGPT
3,42
3.52
3,53
3.64
3.42
3.49
3.57
3.69
3.65
3.62
Table Main Results of SpeechGPT. S2SIF refers to speech-to-speech instruction-folloving S2TIF is speech-to- instruction-followingg T2SIF denotes text-to-speech instruction-following and T2TIF represents text-to-text cntructiio-foolowingg ChatGPT score obtained through ChatGPT evaluatation. CMOS refers to content mean opinion score. NMOS denotes naturalness mean opinion score. *: The ChatGPT Score for speech responses in Ground Truth is due to them being synthesized by TTS system, which can have errors in pauses between sentences. This introduces significant errors for longer responses, leading to incorrect text after being processed by the ASR system, thereby reducing the score. However, humans can understand the content of such speech, so the CMOS score is normal.
Training
Inference
ChatGPT Score
200
Standard
Standard
2.15
From scratch
Standard
CoM
2.12
From LLaMA
CoM
Standard
2.35
160
CoM
CoM
3.42
ASR-PPL
140
Table 2: ChatGPT Score on speech-to speech-to-speech
120
instruction -olllwiinn task. CoM refers to chain-of- modality prompting and Standard denotes standard prompting.
100
80
perform the TTS system in the cascaded system in terms of audio quality and prosody. More detailed speech prosody analysis are located in Section ??.
28
88
Tokens Seen
Figure 3: ASR-PPL of speech continue task on 100 ut- terances from LibriSpeech test-clean set. From scratch refers to model pre-trained from randomly randomly-initialized initialized parameters. From LLaMA denotes model pre-trained from LLaMA.
6 Analysis
6.1
Chain-of-modality prompting matters
Table 2 shows ChatGPT Scores on speech-to- speech instruction-follooing task for models uti- lizing standard prompting and shain-of-modality prompting during training and inference stages re- spectively. Standard prompting refers to directly obtaining a speech response from speech in- struction without transitioning through an inter- mediate text form. The template can be located in Appendix For standard prompting training, we use this template to construct training data. We discovered that if standard prompting is used, the performance is rather poor when either stan- dard prompting Mhain-of-modally, prompting is prompt-
cates that chain-of-modallee prompting matters in both training and inference. We think chain-of- modality prompting decomposes the complex task into easy tasks, allowing the model to complete them step by step, which reduces the difficulty.
6.2 Can text knowledge benefit speech modality?
SpeechGPT originates from a text pre-trained model, LLaMA. Nonetheless, the question remains whether the knowledge from the text modality can contribute beneficially to the speech modality. To resolve this, utilize speech continuation task which assesses the model's capability generate coherent and semantically accurate speech. We compare the performances of two models on this task: one model is pre-trained from LLaMA, while
used for inference. If ing employed during training, ChatGPT Score sees an enhancement, and when the inference also applics Mfain-offmmdalit prompting, there a huge improvement in performance. This indi-

---

instruction-follooiing ability of SpeechGPT with LLaMA-MOSS-002. LLaMA-MOSS-002 is ob- tained by performing supervised fine-tuning on LLaMA-13B using MOSS-sft-002 as the training dataset. This ensures that both models have been exposed to the same amount of text data. We evalu- ated both models using the test set from Section 5.3, As depicted in Figure 4, with increase in training samples, both LLaMA-MOSS-ON and SpeechGPT's ChatGPT Score gradually improve. Although SpeechGPT consistently remains lower than LLaMA-MOSS-002. the performance gap between them gradually decreases. When the train- ing samples reach 40,000, the performance of the two models becomes very similar. This suggests that SpeechGPT still retains text capability, We attribute this to the large parameter size of the 13B model, enabling learn new speech modal- ity while preserving text capability without catas- trophic forgetting.
4.0
ChalGPT Score
3.0
2.5
2.0
1.5
SpeechGPT
LLaMA-MOSS-002
10000
15000
20000
25000
30000
35000
40000
Samples
Figure ChatGPT Score on text-to-text instruction- following task. LLaMA-MOSS-OU0 is obtained by per- forming supervised fine-tuning on L.LMA-13B using MOSS-sft-002 as the training dataset.
the other model is trained from scratch.
We utilize LibriSpeech test-clean set for evalua- tion, where we randomly select 100 utterances, and use the first seconds of each utterance as prompt. The 3-second speech prompt is converted into dis- crete units by mHuBERT. The model takes the prompt input and generates continuation of dis- crete units, which are subsequently converted back into speech by discrete unit vocoder. To assess the semantic quality of the speech continuation, we employ ASR-PPL metric. This involves transcrib- ing the speech continuation into text using the ASR system in Section 5.2 and calculating the perplexity the transcripts using GPT-3.5 text-devinci-00s model. As shown in Figure 3, we observe a continu- ous decrease in ASR-PPL as the training tokens in- crease. The ASR-PPL of the model initialized from LLaMA consistently remains lower than that of the model pre-trained from scratch. This indicates that text pre-trained model provides warm initial- ization and speech modality can benefit from text knowledge. We believe the reason for this is that even though the modeling granularity of speech and text is different, they model the same content information. This leads to certain degree of similarity in the sequence structure, which aids in knowledge transfer.
7 Conclusion
This work presents SpeechGPT, large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. To alleviate the scarcity of instruction datasets in current speech domain, propose SpeechInstruct, the first speech-text cross-modal cestructioo-folloiiini dataset. To ob- tain improved cross-modal performance, we adopt a three-stage training paradigm to obtain the fi- nal SpeechGPT. Experimental results indicate that SpeechGPT achieves promising results in vari- ous unimodal cross-modal instruction-following tasks and demonstrate that combining discrete speech tokens into the language model is promis- ing direction.
Limitation
Despite SpeechGPT exhibiting impressive cross- modal instruction following and spoken dialogue abilities, still presents certain limitations: 1) Due to the audio discretization technique constraints, SpeechGPT does not explicitly model the paralin- guistic information included in the speech signal. 2) Since SpeechGPT generates speech responses via the Mhain-of-MMdallln it needs initially gen- erate speech units after text tokens, which increases decoding time. However, by improving the capa- bilities of the foundation model, SpeechGPT may generate speech units directly without noticeably
6.3 Does SpeechGPT Sacrifice Text Capability as a Trade-off?
Initialized form LLaMA, SpeechGPT is capable of preceiving and generating speech after train- ing on large scale speech data. However, does SpeechGPT sacrifice text capability as a trade-off? To draw conclusions, we compared the text-to-text

---

degrading its performance. 3) SpeechGPT is not evaluated in the multi-turn scenario as the length of one round already close to the maximum length of the model due to the long speech unit sequences. We believe this issue can be addressed by either in- creasing the maximum length the model can handle or employing more effective speech discretization techniques.
Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. 2023b. M 3 st: Mix at three levels for speech translation. In ICASSP 2023-2023 IEEE International Conference on Acous- lics, Speech and Signal Processing (ICASSP), pages IEEE.
Xuxin Cheng, Zhihong Zhu, Ziyu Yao, Hongxiang Li, Yaowei Li, and Yuexian Zou, 2023c. GhostT5: Gen- erate More Features with Cheap Operations to Im- prove Textless Spoken Question Answering. In Proc. INTERSPEECH 2023, pages 1134-1138.
Acknowledgements
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov. Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
We thank Rong Ye and Fuliang Weng for the care- ful guidance and revisions to the paper and thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027).
References
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty. Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively- multilingual speech corpus.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems,
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eu- gene Kharitonov, Olivier Pictquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasac- chi, and Neil Zeghidour: 2022. Audiolm: a language modeling approach to audio generation.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale.
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu Xu, and Bo Xu. 2023. X- Ilm: Bootstrapping advanced large language models by treating multi-modalltie as foreign languages.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multmmodal language model. arXiv preprint arXiv:2303.03378
Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su. Daniel Povey, Jan Trmaa, Junbo Zhang, Mingjie Jin, San- jeev Khudanpur. Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao. Yongqing Wang. Yujun Wang, Zhao You, and Zhiyong Yan. 2021. Gi- gaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3400 3460..
Xuxin Cheng, Bowen Cao, Qichen Ye, Zhihong Zhu, Hongxiang Li. and Yuexian Zou. 2023a. MI-Imel: Mutual learning and aage-margua contrastive learn- ing for improving asr robustness in spoken language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6492-
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Rongjie Huang. Mingze Li, Donghhao Yang. Jia- tong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu,
6505.

---

Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. 2023a. Audiogpt: Understanding and generating speech, music, sound, and talking head.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- ford alpaca: An instruction-folloiing llama model. https:/ httr//gitttuu www.ttub.com/ttttt..eee comm/tatuu-aa sab/ Stanford_alpane
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorek, Vishrav Chaudhary, Subhojit Som. Xia Song, and Furu Wei. 2023b. Language not all you need: Aligning perception with language models.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXivv2302. 13971.
J. Kahn. M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.E. Mazare, Karadayi, V. Liptchinsky, R. Col- lobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, 2020. Libri- light: A benchmark for ASR with limited or super- vision. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Neural codec language models are zero-shot text speech synthesizers.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu. Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, AbdeIrahman Mohamed, et al. 2021. On generative spoken lan- guage modeling from raw audio. Transactions of the Association for Computational Linguisties, 9:1336- 1354.
Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang. and Yaqian Zhou. 2023a. DUB: Discrete unit back- translation for speech translation. In Findings of the Association for Computational Linguisties: ACL 2023, pages 7147-7164, Toronto, Canada. Associa- tion for Computational Linguistics.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jac Lee. 2023. Visual instruction tuning, arXiv preprint arXiv:2304.0848.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Qiao. 2023b. Lllmma-dapterr: Efficient fine-tuning of language models with zero-init attention.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Filkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, AbdeIrahman Mo- hamed, and Emmanuel Dupoux. 2022. Generative spoken dialogue language modeling.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023c. Speechtokenizer: Unified speech tokenizer for speech large language models.
OpenAI. 2023. Gpt-4 technical report.
Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023d. Speak foreign languages with your own voice: Cross-lingual neural codec language mod- eling.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur. 2015, Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
Chunting Zhou, Pengfei Liu, Puxin Xu, Stini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self- supervised representations.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving tasks with chatgpt and friends in huggingface.

---

A Prompts to Generate Task Description
ASR:
You are asked come up with of 100 diverse task instructions about automatic speech recognition, which about recognizing speech.
Here are the requirements:
These instructions should be to instruct someone to recognize the content of the following speech.
1.
2. Try not to repeat the verb for each instruction to maximize diversity.
3. The language used for instruction also should be diverse. For example, you should combine questions with imperative instructions.
4. The type of instructions should be diverse.
5. The instructions should be English.
6. The instructions should be sentences long. Fither an imperative sentence or question is
permitted.
List of 100 tasks:
TTS:
You are asked to come up with set of 100 diverse task instructions about text speech, which is about recognizing speech.
Here are the requirements:
1.
These instructions should instruct someone to recognize the content of the following speech.
2. Try not to repeat the verb for each instruction to maximize diversity.
3. The language used for instruction also should be diverse. For example, you should combine questions with imperative instructions.
4. The type of instructions should be diverse.
5. The instructions should be in English.
6. The instructions should be to sentences long. Either imperative sentence question is permitted.
List of 100 tasks:

---

B Examples of Task Description
ASR:
Begin by converting the spoken words into written Can you transcribe the speech into a written format?
Focus on translating audible content into text.
Transcribe the speech by carefully listening to it.
Would you kindly write down the content of the speech?
Analyze the speech and create written transcription.
Engage with the speech to produce text-based version.
Can you document the speech in written form?
Transform the spoken words into text accurately.
How about putting the speech's content into writing?
TTS:
Can you please read this sentence out loud?
Recite the following words as if you were speaking normally.
Project your voice to clearly articulate this statement.
Would you mind speaking these words naturally possible?
Whisper the given sentence softly.
Enunciate each word in this sentence with precision. How would you express this sentence in conversational tone?
Could you please relay the message below verbally? Emphasize the key points while reading the sentence.
Sing the text provided in a melodic voice.

---

C Chain-of-Modality Instructions Templates
Speech Instruction-Speech Response:
[Human]: This speech instruction: {Speechl). And your response should be speech. You can do it step by step. You can first transcribe the instruction and get the text Instruc- tion. Then you can think about the instruction and get the text response. Last, you should speak the response aloud <eoh>. [SpeechGPT]: [tq] (Textl); [ta] [TextR];; [ua] (SpeechR <enee>> <eoa>.
Speech Instruction-Tess Response:
[Human]: This is speech instruction: {Speechl). And your response should be text. You can do it step by step. You can first transcribe the instruction and get the text instruction. Then you think about the instruction and get the text response. <eoh>, [SpeechGPT]: [tq] (TTxtt); [ta] <eoa>.
Text Instruction-Speech Response:
[Human]: This is a text instruction: (TextI). And your response should be speech. You can do it step by step. You can think about the instruction and get the text response. Then you should speak the response aloud <eoh>. [SpeechGPT]: [ta] (TextR); [ua] [SpeechR) <<eo>>.
Text Instruction-Tess Text Response:
[Human]: This is text instruction: (TextI). And your response should be text. You can think about the instruction and get the text response. [SpeechGPT]: [ta] (TextR) <eoa>>
D Hyperparameters
Stage
Stage 2
Stage 3
Batch size
768
1536
128
Peak learning rate
2e-4
2e-4
2e-4
Max length
1024
512
1024
Training steps
900
4000
4200
LoRA rank
LoRA alpha
16
Trainable parameters
13B
13B
6M
Training device
96 x A100
96 x A100
8 x A A100
Table 3: SpeechGPT training hyperparameters.

---

E
Standard Prompting Templates
Speech Instruction-Specet Response:
[Human]: This is speech instruction: {Speechl). And your response should be speech <coh>.
[SpeechGPT]: [ua] (opecchR<<eass
Speech Instruction-Text Response:
[Human]: This is a speech instruction: (Speechl). And your response should be text. <eoh>.
[SpeechGPT]: [ta] [TextR <eoa>.
Text Instruction-Speech Response:
[Human]: This is text instruction: (Textl). And your response should be speech <eoh>.
[SpeechGPT]: [ua] [SpeechR<<eoa>>
Text Instruction- Text Response:
[Human]: This is text instruction: (TTxtl). And your response should be text. [SpeechGPT]:
[ta] [TextR J <eoa>.

---

F
ChatGPT Score Evaluation Prompt
You are evaluating response that has been submitted for instruction, using specific set of standards. Below is the data:
[BEGIN DATA]
#*#
[Instruction]: inst
***
[Response]: resp
[Criterion]: helpfulness:
"1": "Not helpful The generated text is completely irrelevant, unclear, incomplete. It does provide any useful information to the user."
"2": "Somewhat helpful The generated text has some relevance to the user's question, but it may unclear or incomplete. It provides only partial information, or the information provided may not be useful for the user's needs."
"3": "Moderately helpful The generated text is relevant to the user's question, and it provides clear and complete answer. However, it may lack detail or explanation that would be helpful for the user."
"4": "Helpful The generated text is quite relevant the user's question, and it provides clear, complete, and detailed answer. offers additional information or explanations that are useful for the user. However, some of the points of the response are somewhat repetitive or could be combined for greater clarity and concision"
"5": "Very helpful The generated text highly relevant to the user's question, and provides clear, complete, and detailed answer. It offers additional information, explanations, or analogies that are not only useful but also insightful and valuable to the user. However, the structured of the response is not well-organized and there is no clear progression or logical sequence of different points in the response."
***
[END DATAJ
Does the response meet the criterion? You should only write out your score this format: "My
score is:

---

G
Cases of Cross-modal Instruction Following
Instruction: Can you transcribe the speech into a written format? Input: Speech clip (Transcripts: I'm afraid there are no signs here said he.)
Output: Text: I'm afraid there are no signs here said he.
Instruction: Listen to the speech and write down its content.
Input: Speech clip (Transcripts: Did anyone know that these proofs would be there saved the printer.)
Output: Text: Did anyone know that these proofs would be there no one saved the printer.
Instruction: Would you mind speaking these words as naturally as possible?
Input: Text: Today a sunny day and I'm happy to be here.
Output: Speech clip (Transcripts: Today is a sunny day and I'm happy to be here.)
Instruction: Would you please speed-read the following sentence?
Input: Text: am large language model that can listen and speak, member of Fudan University, and glad talk with you.
Output: Speech clip (Transcripts: am large language model that can listen and speak, member of Fudan University, and glad to talk with you.)

---

Instruction: Speech clip (Transcripts: Can you show me what is the correct English pronunciation of apple?)
Response:
Speech clip (Transcripts: Sure, the correct pronunciation of apple (app-ull].)
Instruction: Speech clip (Transcripts: Write five-line poem that praises spring)
Response: Speech clip (Transcripts: The sun out, the sky is blue, The flowers bloom and the birds sing too, The days are long, the nights are short, The spring is here and it's so sweet and warm!)
Instruction: Speech clip (Transcripts: My girlfriend is angry with What should I do now?)
Response: Speech clip (Transcripts: It is understandable that your girlfriend is angry with you. It is important to understand why she is angry and to try to resolve the issue. You can start by having conversation with her and trying to understand her perspective.)
Instruction: Speech clip (Transcripts: Give some advice about my dinner.)
Speech clip (Transcripts: Sure, here are some suggestions: 1. Make salad with fresh vegetables. 2. Grill some lean meat. 3. Roast vegetable- based dish like a cauliflower casserole. Make a soup with vegetables and legumes. 5. Try recipe.)
Response:
Instruction: Speech clip (Transcripts: Do you know Stephen Curry?)
Speech clip (Transcripts: Yes, I do know Stephen Curry.He is an American professional baskethall player, who currently plays for Golden States Warriors. He two-time NBA most valuable player and four-time NBA all star.)
Response:
Instruction: Speech clip (Transcripts: How can go to the moon?)
challenging task that requires lot of planning and resources. To do this, you will need to develop a spacecraft that can withstand the extreme conditions moon's s atmosphere, design mission plan, and secure the necessary funding and personnel. Additionally, you will need consider the ethical implications of such mission.)
Response: Speech clip (Transeripts: Going the
Instruction: Speech clip (Transcripts: How can cheat my parents?
Response: Speech clip (Transcripts: Cheating your parents is not good idea. It can damage your relationship with them and can have long-term consequences.)


Reviews:
[{"review_id": "dYmKJbL0Tc", "reviewer": "Reviewer_bU86", "summary": "", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Thanks for pointing out this enhancement. Stage 1 equips LLM to understand speech discrete units, i.e., learn the probability distribution of the units. Stage 2 enables the LLM to better align the two modalities by using paired data and enable better knowledge transfer of textual modalities. Stage 3 then equips the model with the ability to better understand and follow multimodal instructions, thus enabling instruction comprehension and response for both modalities. We conduct ablations study on the importance of three training stages. Due to limitations in computational resources, we performed ablation studies on the 7B version of SpeechGPT, focusing on its three key stages: s1, s2, and s3, which correspond to stage1, stage2, and stage3, respectively. We then conducted a human evaluation of the resultant models, specifically targeting their performance in the Speech-to-Speech Instruction Following (S2SIF) task. Our findings, summarized in the accompanying table, reveal the critical roles that both stage2 and stage3 play in SpeechGPT's effectiveness. Without these stages, the model struggles to carry out the S2SIF task proficiently. This is in line with our expectations, as stage2 facilitates speech-text alignment and the transfer of text-based capabilities to speech, while stage3 enables the model to manage cross-modality dialogue tasks. The absence of stage1 also leads to a decline in S2SIF performance, suggesting that stage1 serves as a useful foundation for stage2.\n| Stage | S2SIF (CMOS) |\n|----------|----------|\n| s1+s2+s3 | 3.37 |\n| -s1 | 2.94 |\n| -s2 | 0.12|\n| -s3 | 0.53 |\n2. Thank you for your careful suggestions, we will revise them.\n3. Due to the length of the text, we did not introduce more experiments in the text. We will add more experiments on speech translation, speech summarization task etc. in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper constructs SpeechInstruct, the first large-scale cross-modal speech instruction dataset.\n2. The authors build the first multi-modal large language model that can perceive and generate multimodal content.\n3. This paper is well-organized. ", "reasons_to_reject": "1. The role of each stage in the training is unclear. The authors should add more motivational detail in the method section.\n2. The component name in the method section (discrete unit extractor, large language modal and unit vocode) should be unified to that (discrete unit extractor, SpeechGPT and unit vocode) in the FIgure 2.\n3. The authors can consider using automated metrics and human evaluation of test model performance on more tasks.\n", "questions_to_authors": "See weakness.", "paper_topic_and_main_contributions": "This paper proposes SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2CvPwkkmxs", "reviewer": "Reviewer_9UAb", "summary": "", "questions": "1. Can you provide more insights into the choice of discrete speech representations over continuous representations? Have you tried to analyze how this affect the model's performance compared to existing approaches?\n2. How did you ensure the quality and diversity of the SpeechInstruct dataset?\n3. Have you run some experiments to validate the importance of the three stage training approach that you adopted? Any comparisons with the training regime followed or visual multi modal LLMs such as Palm-E?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. To the best of our knowledge, we are among the first to explore large-scale language models with over 7B parameters that integrate both speech and text modalities using a unified lexicon. While there are other concurrent works in this area such as Google's AudioPaLM and OpenAI's TWIST. We are also the first to build large cross-modal conversational large languaga models that can both understand and generate cross-modal outputs.\n2. Thanks for your great suggestion. Due to restrictions on text length, we did not include more trials about robustness. The upcoming version’ll include more experiments with actual human speech situations.\n3. One significant motivation for choosing discrete speech representations is to enable the model to accept speech input while also being capable of generating speech output. In the realm of speech generation, the classification tasks required for discrete representations are inherently simpler than the regression tasks required for continuous representations. This provides us with a distinct advantage and differentiates our work from other multimodal LLMs. Additionally, the uniformity of discrete speech representations with textual symbols makes it easier to scale up the model's architecture. We believe that this scalability can also be harnessed in multimodal LLMs, as they may follow the same scaling laws that have been observed in unimodal models like GPT-3, thus offering greater potential.\n4. Initially, while generating instructions using GPT-4, we specified in the prompt that the data should be of high quality and diverse. Subsequently, we manually cleaned and filtered the data to ensure its quality. To guarantee diversity in the speech data, we selected datasets from different sources: LibriSpeech, which is derived from audiobooks; Gigaspeech, sourced from web scraping; and Common Voice, from manual recordings. As no relevant datasets were available at the time of the paper's completion, no comparisons could be made.\n5. Due to limitations in computational resources, we performed ablation studies on the 7B version of SpeechGPT, focusing on its three key stages: s1, s2, and s3, which correspond to stage1, stage2, and stage3, respectively. We then conducted a human evaluation of the resultant models, specifically targeting their performance in the Speech-to-Speech Instruction Following (S2SIF) task. Our findings, summarized in the accompanying table, reveal the critical roles that both stage2 and stage3 play in SpeechGPT's effectiveness. Without these stages, the model struggles to carry out the S2SIF task proficiently. This is in line with our expectations, as stage2 facilitates speech-text alignment and the transfer of text-based capabilities to speech, while stage3 enables the model to manage cross-modality dialogue tasks. The absence of stage1 also leads to a decline in S2SIF performance, suggesting that stage1 serves as a useful foundation for stage2.\n| Stage | S2SIF (CMOS) |\n|----------|----------|\n| s1+s2+s3 | 3.37 |\n| -s1 | 2.94 |\n| -s2 | 0.12|\n| -s3 | 0.53 |\nThe vast majority of current visual multi-modal Language Models (LLMs) employ a structure that connects a pretrained visual encoder to a large language model. However, this architecture is limited to accepting multi-modal inputs without the capability to produce multi-modal outputs. Our architecture features both modality-specific encoders and decoders, enabling it to generate multi-modal outputs. This provides a significant advantage in terms of capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Introduces the SpeechInstruct dataset, (most likely the first) large-scale speech-text cross-modal instruction-following dataset, which fills the limitations in publicly available speech data and enhances the reproducibility of the proposed approach.\n* Presents a well-defined three-stage training process, including modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning, which contributes to efficient and effective cross-modal transfer.\n* Empirical evaluations and case analyses demonstrate SpeechGPT's strong performance in textual tasks, speech-text cross-modal tasks, and spoken dialogue tasks. Both human and automate evaluations are used to validate the claims.", "reasons_to_reject": "Below are a few reasons to reject : \n* Lack of novelty: The proposed model and methodology might not significantly differ from existing approaches, and the contributions may not be substantial enough to warrant acceptance.\n* Inadequate Discussion on Robustness: The paper does not address potential vulnerabilities or adversarial attacks on SpeechGPT. Nor does it evaluate using actual human speech as input. (They use TTS generated audio as inputs to evaluate). To strengthen the paper, the authors should investigate and discuss the model's robustness to perturbations and adversarial inputs, as this is a critical concern in real-world applications.", "questions_to_authors": "1. Can you provide more insights into the choice of discrete speech representations over continuous representations? Have you tried to analyze how this affect the model's performance compared to existing approaches?\n2. How did you ensure the quality and diversity of the SpeechInstruct dataset?\n3. Have you run some experiments to validate the importance of the three stage training approach that you adopted? Any comparisons with the training regime followed or visual multi modal LLMs such as Palm-E?", "paper_topic_and_main_contributions": "This paper introduces SpeechGPT, a multimodal large language model with text and speech capabilities. Specifically, a model that can not only process speech and text inputs but also generate speech and text outputs effectively. Additionally, the author's also propose an instruction tuning dataset for speech/text called as SpeechInstruct. \nOverall, the paper's main contributions are computationally-aided linguistic analysis on different permutations of speech and text modality instructions and the generated outputs and an NLP engineering experiment. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XXX5hSj3et", "reviewer": "Reviewer_mbz5", "summary": "", "questions": "Question 1:\nFor the results of T2TIF in Table 1, are Speech-Alphaca-13B and Speech-LLaMa-MOSS-002 just using language models such as Alphaca-13B and LLaMa-MOSS-002 without using speech respectively?\n\nQuestion 2:\nTable1 shows that the results in NMOS of SpeechGPT are generally high and therefore considered to be natural. On the other hand, the results of Text input (T2SIF, T2TIF) in CMOS of SpeechGPT are lower than Speech-LLaMA-MOSS-002 . In the case of Text input when using Speech-LLaMA-MOSS-002, I think T2TIF is only using LLaMA-MOSS-002 without speech (ASR and TTS) and T2SIF is only using LLaMA-MOSS-002+TTS without ASR. Does the difference in performance of CMOS mean that instruction tuning with LLM like LLaMA-MOSS-002 can lead to more content and meaningful learning than learning via the Speech modality?\n\nQuestion 3:\nSpeech-LLaMA-MOSS-002 learns Instruction by using MOSS-002. On the other hand, SpeechGPT learns Instruction by using SpeechInstruct. Have you ever tried to train SpeechGPT from models that have already been trained on MOSS-002, instead of only using SpeechInstruct for LLaMa? I think it might avoid a lack of learning about content and meaning.\n\nQuestion 4:\nIn terms of ChatGPT Score, the Ground Truth of T2SIF is lower than that of Baselines. It is considered that the quality of TTS has an influence as authors mentioned. If so, isn't ChatGPTScore unreliable because it cannot evaluate correctly when TTS is included? In Table 1, only Ground Truth is annotated with *, but is there any possibility that Baselines and SpeechGPT are also unreliable as evaluation values? For example, although the SpeechGPT for S2SIF shows high performance, but if ChatGPTScore cannot provide reliable evaluation results due to TTS, is it possible that not only  the low GroundTruth value is unreliable, but also the high SpeechGPT for S2SIF and T2SIF are unreliable?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. They are both connected with off-the-shell ASR and TTS system. We explain this in Section 5.2：Speech-Alpaca-13B, consists of an off-the-shell ASR system, Alpaca 13B (Taori et al., 2023) as well as a pre-trained TTS system. Speech-LLaMA-MOSS-002 incorporates the same ASR and TTS system, along with a large language model obtained by perform ing supervised fine-tuning on LLaMA-13B using MOSS-sft-002 as the training dataset. \n2. Yes, for instruction-following tasks where text serves as the input, the performance of SpeechGPT is lower than that of Speech-LLaMA-MOSS-002. We discuss this issue in Section 6.3, where the results indicate that although SpeechGPT still retains some text capability, there is a loss in performance. We believe that obtaining instruction-following capabilities is most effectively done through textual modality. This is because speech contains additional information, such as timbre and emotion, that can interfere with content-based capabilities.\n3. As we mentioned in Section 4.2, we mix the Cross-modal Instruction in SpeechInstruct with the moss-002-sft dataset to create a mixed dataset. Therefore, SpeechGPT also learns instruction-following capabilities from MOSS-002.\n4. When TTS (Text-to-Speech) is included, ChatGPTScore is not as reliable as CMOS. Thank you for pointing out that baselines should also be annotated with *; we will make this improvement in the paper. However, the speech output generated by SpeechGPT does not go through a TTS system, making its speech quality reliable. As a result, its T2SIF (Text-to-Speech Instruction Following) and S2SIF (Speech-to-Speech Instruction Following) scores are higher. Overall, for the evaluation of T2SIF and S2SIF, CMOS scores are more reliable than ChatGPTScore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea to incorporate other modalities into LLMs by using SpeechInstruct dataset is new and interesting. The authors experimentally confirm that text knowledge is beneficial to speech modalities and that SpeechGPT maintains text ability. Further improvements can be expected based on this study.", "reasons_to_reject": "The ChatGPT scores for S2SIF and T2SIF may be unreliable when using TTS, and the values in Table 1 may mislead readers.", "questions_to_authors": "Question 1:\nFor the results of T2TIF in Table 1, are Speech-Alphaca-13B and Speech-LLaMa-MOSS-002 just using language models such as Alphaca-13B and LLaMa-MOSS-002 without using speech respectively?\n\nQuestion 2:\nTable1 shows that the results in NMOS of SpeechGPT are generally high and therefore considered to be natural. On the other hand, the results of Text input (T2SIF, T2TIF) in CMOS of SpeechGPT are lower than Speech-LLaMA-MOSS-002 . In the case of Text input when using Speech-LLaMA-MOSS-002, I think T2TIF is only using LLaMA-MOSS-002 without speech (ASR and TTS) and T2SIF is only using LLaMA-MOSS-002+TTS without ASR. Does the difference in performance of CMOS mean that instruction tuning with LLM like LLaMA-MOSS-002 can lead to more content and meaningful learning than learning via the Speech modality?\n\nQuestion 3:\nSpeech-LLaMA-MOSS-002 learns Instruction by using MOSS-002. On the other hand, SpeechGPT learns Instruction by using SpeechInstruct. Have you ever tried to train SpeechGPT from models that have already been trained on MOSS-002, instead of only using SpeechInstruct for LLaMa? I think it might avoid a lack of learning about content and meaning.\n\nQuestion 4:\nIn terms of ChatGPT Score, the Ground Truth of T2SIF is lower than that of Baselines. It is considered that the quality of TTS has an influence as authors mentioned. If so, isn't ChatGPTScore unreliable because it cannot evaluate correctly when TTS is included? In Table 1, only Ground Truth is annotated with *, but is there any possibility that Baselines and SpeechGPT are also unreliable as evaluation values? For example, although the SpeechGPT for S2SIF shows high performance, but if ChatGPTScore cannot provide reliable evaluation results due to TTS, is it possible that not only  the low GroundTruth value is unreliable, but also the high SpeechGPT for S2SIF and T2SIF are unreliable?", "paper_topic_and_main_contributions": "This paper proposes SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. The authors construct SpeechInstruct, the large-scale cross-modal speech instruction dataset. And also, they employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The main contribution of this paper is to show the potential of the model based on the three-stage training strategy with SpeechInstruct dataset than the two cascaded cross-modal conversational systems as baselines.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a multimodal SpeechGPT. The paper also creates a SpeechInstruct dataset. Reviewers see merit in these contributions, despite the architecture itself not bringing novelty.
Decision: Accept-Findings