LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Authors: Qingkai Fang
Shoutao Guo
Yan Zhou
Zhengrui Ma
Shaolei Zhang
Yang Feng
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
speech interaction
speech-to-speech
speech-language models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: LLaMA-Omni is a low-latency and high-quality end-to-end speech interaction model, which can simultaneously generate both text and speech responses based on speech instructions.

Abstract: Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

Published as a conference paper at ICLR 2025
LLAMA-OMNI: SEAMLESS SPEECH INTERACTION WITH LARGE LANGUAGE MODELS
Qingkai Fang Shoutao Guol.5, Yan Zhou Zhengrui Ma
Shaolei Zhang!. Yang Feng
1.2.3+
Key Laboratory of Intelligent Information Processing
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
2 Key Laboratory of AI Safety, Chinese Academy of Sciences
3 University of Chinese Academy of Sciences, Beijing, China (f Fanggingkai21B, fengyang] ac.
ABSTRACT
Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to tra- ditional text-based interaction. However, there is still lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel end-to-end model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omin inte- grates pretrained speech encoder, a speech adaptor, an LLM, and streaming speech decoder. It eliminates the need for speech transcription, and can simulta- neously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3. 1-8B- Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200k which includes 200K speech instructions and corresponding speech responses, with a style that better matches the character- isties of speech interaction scenarios. Experimental results show that compared to previous speech-languas models, I.LMAA-Omi provides better responses in both content and style, with response latency low 236ms. Additionally, training I.LaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future
INTRODUCTION
Large language models (LLMs). represented by ChatGPT (OpenAl 2022). have become powerful general-purpose task solvers, capable of assisting people in daily life through conversational inter- actions. However, most LLMs currently only support text-based interactions, which limits their application in scenarios where text input and output are not ideal. Recently, the emergence of GPT- 40 (OpenAI (2244) has made possible interact with LLMs through speech, responding to user's instruction with extremely low latency and significantly enhancing the user experience. However, there is still a lack of exploration the open-source community on building such speech interaction models based on LLMs. Therefore, how achieve low-latency and high-quality speech interaction with LLMS is a pressing challenge that needs to be addressed.
The simplest way to enable speech interaction with LLMs is through cascaded system based on automatic speech recognition (ASR) and text-to-speech (TTS) models, where the ASR model tran- scribes the user's speech instruction into text, and the TTS model synthesizes the LLM's response into speech. However, since the cascaded system sequentially outputs the transcribed text, text re- sponse, and speech response, the overall system tends have higher latency. In contrast, some multimodal speech-language models have been proposed (Zhang el al. 2023 Rubenstein et al.
"Coreeponning Author: Yang Feng.
Model: http://ss///oooggoofece///////// Audio Sampleehhtpess//icnnpgggihhhh omni demo/
3-1 1-88--8888888 Omni

---

Published as a conference paper at ICLR 2025
simul Canecus generation
LLaMA-Omni
you give advicer on wriling NLP papers!)
Cortatady! Writing high- NLP paper requires
Figure 1:
LLaMA-Omni can simultaneously generate text and speech responses based the speech instruction, with extremely low response latency.
2023j, which discretize speech into tokens and extend the LLM's vocabulary to support speech in- and output. Such speech- laeech-language models theoretically generate speech responses directly from speech instructions without producing intermediate text, thereby achieving extremely low re- sponse latency. However, in practice, direct speech-to-speech generation can be challenging due to the complex mapping involved, so is common to generate intermediate text to achieve higher generation quality (Zhang et al. 2023). although this sacrifices some response latency.
In this paper, we propose a novel model architecture, I.AMA-Ommi, which enables low-latency and high-quality interaction with LLMs. LLaMA-Omni consists of speech encoder, a speech adaptor, an LLM, and streaming speech decoder. The user's speech instruction is encoded by the speech encoder followed by the speech adaptor, and then input into the LLM. The LLM decodes the text response directly from the speech instruction, without first transcribing the speech into text. The speech decoder is autorrrrrreeeevvee (NAR) streaming Transformer (Ma al. 2023), which takes the output hidden states of the LLM as input and uses connectionist temporal classification (CTC; Graves al. 2006a) to predict the sequence of discrete units corresponding to the speech response. During inference, the LLM autoregressively generates the text response, the speech decoder si- multancously generates the corresponding discrete units. To better align with the characteristics of speech interaction scenarios, construct dataset named InstructSSS-206 by rewriting existing text instruction data and performing speech synthesis. Experimental results show that LLaMA- Omni can simultaneously generate high-quality text and speech responses with latency low as 236ms. Additionally, compared to previous speech-language models like SpeechGPT (Zhang et al.) 2023 LLaMA-Omni significantly reduces the required training data and computational resources, enabling the efficient development of powerful speech interaction models based on the latest LLMs.
2 MODEL: LLAMA-OMN
In this section, we introduce the model architecture of I.LMAAQmmi. As shown in Figure 2 it consists of speech encoder, a speech adaptor, an LLM, and a speech decoder. We denote the user's speech instruction, text response, and speech response X Y", and Y respectively.
2.1 SPEECH ENCODER
We use the encoder of Whisper-large-V (Radford al. 2023) the speech encoder E. Whisper is a general-purpose speech recognition model trained on a large amount of audio data, and its encoder capable of extracting meaningful representations from speech. Specifically, for the user's speech instruction X the encoded speech representation is given by H { where = [h,,.. hy] the speech representation sequence of length N. We keep the speech encoder's parameters frozen throughout the entire training process.
2.2 SPEECH ADAPTOR
To enable the LLM to comprehend the input speech, we incorporate trainable speech adaptor AA that maps the speech representations into the embedding space of the LLM. Following Ma et al. (0024b). our speech adaptor first downsamples the speech representations H to reduce the sequence length. Specifically, every consecutive frames are concatenated along the feature dimension:
H' hi..., h [N/R]] where h [bx((-1))+ ① hax(i-1)42 ① hexi]
(1)
httpss//huugginnffccc..ooo htt.::/c..ooo...aaaaaaaaaaaaaaa.
2

---

Published as a conference paper at ICLR 2025
* la Latency
1ill qil
+
Vocoder
Large Language Model
Speech Adaptor
A
Speech Encoder
aimul genezacion
Speech Decoder
Stage
Upsample
Lerc
Wizing
Se-ligh
Speech Decoder
Large Language Model
Upsample
Largo Language Model
Speech Adaptor
Speech Adaptor
Speech Encoder
LLaMA-Omni
Speech Encoder
Con
uningg NUP popere!)
Stage
Figure 2: Left: Model architecture of LLaMA-Ommi Right: Illustration of the two-stage training strategy for I.aMA-Ommi.
Next, H' is passed through a 2-layer perceptron with ReLU activation between the linear layers, resulting in the final speech representation S. The above process can be formalized follows:
$ = A(H) Le
2.3 LARGE LANGUAGE MODEL
Dubey et al. 2024) as the LLM M, which is currently the state- of-the-art open-source LLM. It has strong reasoning capabilities and is well-aligned with human preferences. The prompt template P(-) is shown Appendix A The speech representation sequence is filled into the position corresponding <speech>, and then the entire sequence P(S) is imput into the LLM. Finally, the LLM autoregressively generates the text response y 1 MM directly based on the speech instruction and is trained using cross-entropy loss:
We use -Jama-3.1-8B-Instiee
M
LLLM
log Ply P(S),
(3)
2.4
SPEECH DECODER
For the speech response SS we first follow Zhang al. 2023 discretize the speech into discrete units. Specifically, we use the pretrained HuBERT (Hsu et al. 202T) model to extract continuous representations of the speech, and then convert these representations into discrete cluster indices using a K-means model. Subsequently, consecutive identical indices are merged into single unit, resulting the final discrete unit sequence Yu = [ot' vf, v. {0, 1,.., - 11,W where K is the number of clusters, and L is the length of discrete unit sequence. To synthesize waveforms based on discrete units, we adopt a unit-based HiFi-GAN vocoder with a duration pre- dictor (Polyak et al. 202T). It first predicts the duration of each discrete unit and repeats them to model prosody, and then generates the waveform based on discrete units.
1), VI S S L,
To generate speech responses simultaneously with text responses, add a streaming speech de- coder 'D after the LLM. It consists of several standard Transformer (Vaswani et al., (017) layers with the same architecture LLaMA (Dubey al. 2024). each containing causal self-attention module and a feed-forward network. Similar to Ma et al. (2024)); /Zhang et al. (2024b). the speech decoder runs in non-autoregressive manner, which takes the output hidden states from the LLM as input, and generates the discrete unit sequence corresponding to the speech response. Specifically,
https::/huggingfffe http:n/g.........eeooooooooo ama aaaaaa/Metaaa
88H Intruct

---

Published as a conference paper at ICLR 2025
the output hidden states corresponding to the text response denoted C [c1 .... Carl, where c, = MM((()) Y2).. We first upsample each hidden state into a chunk by a factor of X, resulting in
an upsampled hidden state sequence C C1 _,__...x where غ; C///A-- Next, C is fed into the speech decoder DD and the output hidden state sequence is denoted as O : OX-M]- We use connectionist temporal classification (CTC: Graves et al. (006a) to align with the discrete unit sequence Y Specifically, CTC extends the output space with special blank token e:
OO) = coftmax(Wo; + b) Va, E {0, (0.1...., K - 1,e),
(4)
where W E (((+)))dd and b E I are weights and biases of the linear layer, and the sequence A = [a1,..., 02-M] is known as the alignment. To model the variable-lenghe mapping between input and output, CTC introduces a collapsing function ((A), which first merges all consecutive repeated
tokens in A and then eliminates all blank tokens e. For instance: 3((1, 1,2, e, 2, 2]3 = [1, 2, 2.3. During training, CTC performs marginalization over all possible alignments as follows:
X-M
Lere = 1e-llgg log P(Y =loog
Σ O(A log Σ
II P(a. O). (5) 1=1
AES-1(Y")
AEA-I(Y") 1=1
where (YU( denotes all possible alignments of length A M that can be collapsed Yu The alignment is modeled in a non-autoregressive way. During inference, we select the best alignment
A arg m P(A(O), and apply the collapsing function to obtain the discrete unit sequence 3(A*), which is then fed into the vocoder to synthesize waveform.
2.5 TRAINING
Algorithm 1: Inference Process
As shown in Figure we adopt a two-stage train- ing strategy for LLaMA-Omni In the first stage, we train the model to generate responses directly from the speech instructions. Specifically, the speech en- coder is frozen, and the speech adaptor and the LLM are trained using the objective LITM in Eq. (3). The speech decoder is not involved in training during this stage. In the second stage, we train the model to gen- crate speech responses. During this stage, the speech encoder, speech adaptor, and LLM all frozen, and only the speech decoder is trained using the objective Lere in Eq.
Input: speech instruction X Output: text outputs Y", units Y waveform outputs y²
Model: speech encoder E, speech adaptor A. LLM .M, speech decoder D, vocoder V.
Require: Minimum chunk size for units f2 Initialization: i i= L.j = 0, Y = D. ye = -l. = [1 [.
S A(E(X*)); while v.=, * (EOS) do c; + M(P(S), Y2):
uf + arg max, , P(u (P()),
2.6 INFERENCE
Y" Y + y.
Č - UUUpaapple(c.):
During inference, the L.M autoregressively gener- ates the text response based on the speech instruction.
O t- D(()::
A + arg m A P(A[O);
Meanwhile, since our speech decoder uses causal at- tention, once the LLM generates a text response pre- fix Si. the corresponding upsampled hidden states
Y t B(A");
Y -j 2 S then
v + V(Y Ys + ys + +y²
Cex+ can be fed into the speech decoder to generate partial aligmment Agx-s, which in turn yields the dis- crete units corresponding to the generated text prefix. To further enable streaming synthesis of speech wave- forms, when the number of generated units reaches a pre-defined chunk size 22 we input this unit segment into the vocoder to synthesize speech segment, which then immediately played to the user. As result, users can start listening to the speech response without waiting for the complete text response to be generated, ensuring low response latency that is not affected by the length of the text response. Algorithm [ describes the above process. Additionally, since the speech decoder uses non- non-autoregressive modeling, the alignment corresponding to each text token v." specifically (1-(1-))+ 1:A-i is generated in parallel within the chunk. Therefore, the decoding speed for generating both text and speech responses simultancously is not significantly different from the speed of generating text response alone.
j + Y
end
i - i 1;
end
j
< Y then S < v(r
y5 +
end
4

---

Published as a conference paper at ICLR 2025
3 CONSTRUCTION OF SPEECH INSTRUCTION DATA: INSTRUCTS2S-2000
To train LLaMA-Omni we need triplet data consisting of <speecch instruction, text response, speech response>. However, most publicly available instruction data is in text form. Therefore, struct speech instruction data based on existing text instruction data through the following process:
Instruction Rewriting Since speech input has different characteristics compared text input, we rewrite the text instructions according to the following rules: (1) Add appropriate filler words (such as "hey", "so", "uh", "um", etc.) the instructions simulate natural speech patterns. (2) Convert non-text symbols in the instructions (such as numbers) into their corresponding spoken forms to ensure correct synthesis by TTS. (3) Modify the instructions be relatively brief without excessive verbiage. We use the Llama-3-70BB IOS-Instr model to rewrite the instructions according to these rules, The prompt can be found in Appendix
Step I:
Step 2:
Response Generation In speech interactions, existing responses from text instructions are not suitable for direct use as speech instruction responses. This is because, in text-based inter- actions, models tend to generate lengthy responses, using complex sentences and possibly including non-verbal elements like ordered lists or parentheses. However, in speech interactions, concise yet informative responses are typically preferred (Cho et al. 2024). Therefore, we use the Llama-3- 70B-Instruct model to generate responses for speech instructions according to the following rules: (I) The response should not contain content that cannot be synthesized by the TTS model, such as parentheses. ordered lists, etc. (2) The response should be very concise and to the point, avoiding lengthy explanations. The prompt can be found Appendix A
Step 3: Speech Synthesis After obtaining the instructions and responses suitable for speech in- teractions, we need to further convert them into speech using TTS models. For the instructions, to make the synthesized speech sound more natural, we use the Cosy Voice-300M-SFT (Du et al. 2024) mode randomly selecting either male or female voice for each instruction. For the responses, use the VITS (Kim et al. 2021 mode trained on the LJSpeech (Ito & Johnson, 2017) dataset to synthesize the responses into a standard voice.
For the basic text instructions, we collect around 50K in- structions from the Alpaca dataset" (Taori et al. 2023). which covers wide range of topics. Additionally, we gather around 150K instructions from the UltraChat dataset" (Ding al., 2023), which primarily consist of questions about the world. Note that UltraChat is a large- scale multi-turn conversation dataset, but we only select the first 150K entries and use only the first-round in- struction. Using the above datasets and data processing pipeline, we ultimately obtain 200K speech instruction data, referred to as InstructS2S-200K. The detailed sta- tistical information is listed in Table[]
Table I: Statistical information of the InstructS2-20001 dataset.
Statistic
Value
Speech Instruction Duration
418h
Speech Response Duration
1058h
Avg. Speech Instruction Duration
7.5s
Avg. Speech Response Duration
19.0s
Avg. Text Instruction Length
21.7
Avg. Text Response Length
39.5
Avg. Unit Sequence Length
553.6
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets For the training data, we use the InstructS2S-2009 dataset mentioned Section 3 which includes 200K speech instruction data. To extract discrete units corresponding to the target
speech, we use a prw--ereiedde------a-a K-means quantize which has learned 1000 clusters from the HuBERT
3-700-------- Instruut
https: http:e//itttbb..ccooo/aaaaaaaaaa
httpa://gghuuu.oo///mmm11/111111 http:///stttttttaaaaaaaapaaaaaa aatptsss--abbb/ppppp paca
https://d1. http:///.....aaaaaaaaaaaoaaoao/r
base_vp. _a_e____e___________e___________ _p__n__s_____ aa__f____. i
M
km0000.bin
5

---

Published as a conference paper at ICLR 2025
10 Kong et al. 2020 Polyak et al. 2021 is used to synthesize discrete units into waveform. For the evaluation data, we select two subsets from Alpaca- Eva Li et al., 2023): helpful base and vicuna, as their questions are more suitable for speech interaction scenarios. We remove questions related to math and code, resulting in a total of 199 instructions. To obtain the speech version, we use the Cosy Voicce-300MSSF model to synthesize the instructions into speech. We refer to this test set as MnstructS2S-Evar in the following sections.
features. The pretrained HiFi-GAN vocoder
Model Configuration We use the encoder of Whisper-largo-v3 as the speech encoder, and use -lama-3.1-8B-Instruct the LLM. The speech adapter performs 4 downsampling the speech representations. The speech decoder consists of 2 Transformer layers with the same architecture as LLaMA, with hidden dimension of 4096, 32 attention heads, and feed-forward network di- mension of I 11008, which contains 425M parameters. The upsample factor À is set to 25, For the minimum unit chunk size input to the vocoder, we Ω +00 in the offline scenario, meaning we wait for the entire unit sequence to be generated before inputting it to the vocoder for speech synthesis. In the streaming scenario, we adjust the value of 52 within the range of [10, 20, 40, 60, 80, 100] to control the response latency of the model.
Training LLaMA-Omai follows a two-stage training process. In the first stage, we train the speech adapter and the LLM with a batch size of 32 for epochs. We use cosine learning rate scheduler with the first 3% of steps for warmup, and the peak learning rate is set to 2c-5. In the second stage, we train the speech decoder, using the same batch size, number of steps, and learning rate scheduler as the first stage, but with the peak learning rate set to 2e-4. The entire training process takes approximately 65 hours on 4 NVIDIA L.40 GPÜs.
4.2 EVALUATION
Since LLaMA-Omni can generate both text and speech responses based on speech instructions, we evaluate the model's performance two tasks: speech-to-text instruction-folloiing (S2TIF) and speech-to-speech instruction-followingg (S2SIF). We use greedy search to ensure reproducible experimental results. For the S2SIF task, we further evaluate the model in two scenarios: offline and streaming. In the offline scenario, the model generates the text response first, and then synthesizes the complete speech. In the streaming scenario, the speech response is generated simultaneously with the text response. We use the following metrics to evaluate the model:
ChatGPT Score To evaluate the model's ability to follow speech instructions, we use GPT- 4o (OpenAl 2024) to score the model's responses. For the S2TIF task, scoring is based on the transcribed text of the speech instructions and the model's 's text response. For the S2SIF task, we first transcribe the model's speech responses into text using the Whisper-large-33 model, and then score the same manner as the S2TIF task. GPT-4o gives a score between 1 and 5 based on factors such helpfulness, relevance, fluency, and suitability for speech interaction. The detailed prompt for evaluation can be found in Appendix A
To evaluate the alignment between text and speech responses, we use the Whisper- large-v3 model to transcribe the speech responses into text, and then calculate the Word Error Rate (WER) between the transcribed text and the text response, which is referred to as ASR-WER.
ASR-WER
UTMOS To evaluate the quality the generated speech, we utilize Mean Opinion Score (MOS) prediction model called UTMOS (aakii et al. 2022). which is capable of predicting the MOS score of the speech to assess its naturalness.
Latency The response latency is a key metric for speech interaction models, referring to the time interval between the input of speech instruction and the start of the speech response, which has significant impact on user experience. We measure the latency on NVIDIA L40 GPU.
http:://h.aatapppuuleeceoceeooaa http://ei.e....ooooooooeeeeeeeea "https://d1
bifigan/mhuberse vp en es fr it3 400k laverll km1000 15/g 00500000
httppe//ggthhaccco///reeee//gee pacappcaaaa eva
https:://ittubb.co//ttaaaa

---

Published as a conference paper at ICLR 2025
Table 2: Results on the InstructS2S-Evw benchmark in the offline scenario. A refers the differ- ence in ChatGPT Score between the S2SIF and S2T1F tasks.
ChatGPT Score
Model
ASR-WER +
UTMOS t
S2TIF
S2SIF
A
SpeechGPT
2.98
2.19
0.79
45.00
3,8958
SALMONN Orea
3.44
3.40
0.04
3.78
3,8286
Qwen2-Audio + Orea
3.47
3.38
0.09
6.77
3.6119
LLaMA-Omni
3.99
3.47
0.52
10.82
3.9296
Speech Rate
To measure the speech rate of the generated speech, we use the metric Words Per Second (WPS), which represents the average number of words per second in the generated speech.
4.3 BASELINE SYSTEMS
We include the following speech-language models baseline systems:
SpeechGPT SpeechGPT (Zhang et al., 2023) a speech-language model that supports both speech input and output. We use the chain-of-modality prompting adopted in the original paper for decoding, which sequentially outputs the text instruction, text response, and speech response based on the speech instruction.
SALMONN + Orca SALMONN TTang et al., 2024) is LLM capable of accepting speech and audio inputs and responding with text, enabling to perform the S2TIF task. For the S2SIF task, we add TTS model called Orca [33 after SALMONN. Orca is an industrial TTS model that supports both streaming and offline speech synthesis, delivering excellent performance. This integrated system enables speech synthesis to begin concurrently with the generation of the text response.
2024 a powerful general-purpon audio un- Qwen2-Audio + Orca Qwen2-Audio (Chu et al.
derstanding model capable of performing various audio-related tasks, including the S2T1F task. We also build cascaded system with Qwen2-Audio and Orca to complete the S2SIF task.
When using Orca for streaming speech synthesis, need set word chunk size θ, which means that speech synthesis is triggered every time θ new words arrive. In our experiments, we varies e within the range of [1, 3, 9] control the response latency of cascaded systems.
4.4 RESULTS IN THE OFFLINE SCENARIO
Table presents the results the InstructS2S-Eval benchmark in the offline scenario. For the S2TIF task, LLaMA-Ommi achieves the highest ChatGPT Score, significantly outperforming the baseline systems. We attribute this to two key factors. First, model is built upon the latest Llama-3.1-8P Instruct model, leveraging its strong text instruction followiiinn capabilities. Second, InstructS2S-2001 dataset effectively aligns the model with speech interaction scenarios, suring that its responses are both high-quality and well-suited to speech contexts. In comparison, SALMONN and Qwen2-Audio, as speech-to-ter models, have not been aligned with speech inter- action scenarios. As a result, their responses often include formatted content and a lot of redundant explanations, making them less suitable for such contexts.
For the S2SIF task, LLaMA- still achieves the highest ChatGPT score among all models. We observe that scores for the S2SIF task are generally lower than those for the S2TIF task. This decline is primarily due to errors introduced during the speech synthesis process and the reliance on an ASR model for evaluation. The two cascaded baseline systems, which use industrial TTS models for speech synthesis, exhibit the smallest decline in scores. For the end-to-end models, L.aMA- Omni demonstrates a smaller drop in score compared to SpeechGPT, indicating that LLaMA-Omni has stronger speech generation capabilities. This can be further verified by the ASR- WER metric:
httpps: httptp:://githhbb.co/////////

---

Published as a conference paper at ICLR 2025
[1aMA Crmi
LLMMA (Ottins)
CharGP Soore
ASR WER
30
SALMONN Orza (Otliid)
LLAMA Onei
Ques2-Aadia Out
LLaMh-Ormi (Ollic))
OwenI-Nadia Osex (Delinc)
SALMONN
SporckCP
Speckke (ifflidd
Qucel Aaals (Offlad
SparaCeT
Spareace (Oiiinc)
10
200
300
100
600
5000
8000
200
400
500
700
6000
8000
Latenny (ms)
Latency (ms)
(a) ChatGPT Score against latency.
(b) LAR-WER against latency.
3.8
3.2
Words Per Second (WPS)
2.8
UIMOS
LLaMM-Oai
2.6
MONN
3.2-
2.0
SALMONN Oreeiffline
SALMONN Otta
Queil: Audio Onez
SALMONN Orea (ffllin)
Qpesi-Autte deea (Utaac))
Audio Crea
Spand (iilia
SpoddGPP
SpectrPTT (Difio))
1.8
Lutency (ms)
Latency (mms)
(c) UTMOS against latency.
(d) WPS against latency.
Figure 3: Results on the InstructS2S- Eval benchmark in the streaming scenario. We report the
ChatGPT Score, ASR-WER, UTMOS, and WPS under different latency conditions.
the cascaded systems achieve the lowest ASR-WER, while LLaMA-Omni's -SR-WER is slightly higher but still acceptable. In contrast, SpeechGPT's ASR-WER is significantly higher, suggesting poor alignment between its speech and text responses. Although the alignment between LLaMA- Omni's speech and text responses is slightly lower than that of the cascaded systems, this is primarily due to the fact that LLaMA-Ommi's speech decoder is trained on only approximately 1K hours of training data, LLaMA-Ommi performance could be further improved. Finally, the UTMOS metric shows that LLaMA-Onin generates satisfactory speech quality, slightly surpassing other baseline models.
data, which is far less than the industrial TTS model. We believe that with
4.5 RESULTS IN THE STREAMING SCENARIO
In the streaming scenario, the model generates speech responses simultaneously while generating text responses. For SpeechGPT and LLaMA-Omni, the response latency controlled by adjusting the unit chunk size Ω2 whereas for cascaded systems, it is managed by adjusting the word chunk size O. In Figure we present the ChatGPT Score, ASR-WER WER, UTMOS and WPS results of all models under different latency conditions. Firstly, we examine the results of LLaMA-Ommn under different latency conditions and observe that LLaMA-Omi can achieve minimum latency of 236ms (with
= 10), which is even lower than GPT-4o's 's average audio latency 320ms. As latency increases (2 grows larger), we notice slight decrease in the ChatGPT Score, while ASR-WER shows minor improvement. We believe this is mainly because the vocoder may handle shorter unit sequences more reliably than longer ones, as it is typically trained shorter sequences. However, when Ω is smaller, the speech is divided into more segments for synthesis, which increases discontinuities in the speech. This leads to decline in speech quality, and consequently, decrease in the UTMOS score is observed. In addition, the speech generated by LLaMA-Omin maintains almost consistent speech rate across different latency conditions. In summary, L.LMAA-mmi achieves relatively stable
8

---

Published as a conference paper at ICLR 2025
1Lalisebeen Wes
LioMs Levn]
lir
SALVONN
SALMONN
Q+ sula
Quea Aa
20
Meee
(a) Helpfulness.
(b) Naturalness.
Figure 4: Results of human evaluation from the perspectives of helpfulness and naturalness.
performance under varying latency conditions. At a latency of 563ms (S2 100), the results of all metrics are relatively close to those in the offline scenario.
Next, we compare LLaMA-Omni with other baseline systems. For SpeechGPT, its decoding la- tency is significantly higher than other systems (>4500ms) due to the sequential generation of text instructions and text responses before generating units. Additionally, the quality of its generated responses is relatively poor. For cascade systems based on streaming TTS, we observe that both Qwen2-Audio + Orea and SALMONN + Orea can achieve latencies below 300ms. However, they exhibit two notable issues. Firstly, in the streaming scenario, their ChatGPT Score and ASR-WER show significant gaps compared to the offline scenario, indicating that streaming speech synthesis at the word level tends to introducing additional errors. Secondly, as shown in Figure dj, when the latency low, the overall speech rate of cascaded systems drops sharply. This is mainly because the pauses between words become more frequent, resulting in reduced naturalness and coherence of the speech. In contrast, LLaMA-Omen -anii's 's streaming unit generation is performed within the end-to-end model, requiring only cascaded vocoder to complete the streaming As result, the overall prosody and rhythm of the speech remain almost unchanged across different latency levels. This demonstrates the advantage of end-to-end models over cascade systems based on streaming TTS. This can be more intuitively experienced by listening to the provided audio samples. We provide the numerical results of all models in the streaming scenario in Appendix C
toveefoorooo conversion.
4.6 HUMAN EVALUATION
To better understand human preferences for model responses in real-time speech interaction sce- narios, we further conduct a human evaluation. Specifically, we perform side-by-side comparisons between LLaMA-Omni (Ω = 40, latency=347ms) and two cascaded systems: SALMONN+Orea (O = 3. latency=352ms) and Qwen2-Audio+Orca (6 = 3, latency=420ms). For each comparison, randomly select 20 speech instructions and collect the speech responses generated by the two models. We then invite participants to evaluate all samples from two key perspectives: helpfulness and naturalness. Helpfulness evaluates whether the model follows the instructions and provides appropriate responses, while naturalness evaluates the fluency and natural quality of the generated speech. Participants compare each pair of samples rate them as win, tie, or lose for both aspects. Finally, each pairwise model comparison results in a total of 100 votes. The results, shown in Fig- ure 5, demonstrate that LLaMA- -mmn achieves higher win rate compared the cascade systems in both helpfulness and naturalness, confirming that LLaMA-Omni generates responses that better align with human preferences.
RELATED WORK
Speech/Audio Language Models With the success of language models in the field of natural lan- guage processing (Brown et al. 2020). researchers have begun exploring how to model speech or audio using language models. Early work attempted train language models on semantic tokens or acoustic tokens of audio, enabling the generation of audio without the need for text (Lakhotia et al.) 2021 Nguyen et al. 2023 Borsos et al. 2023). Furthermore, by jointly training speech tokens and text, decoder-only models like VALL-E (Wang et al.) 2023b) and VioLA (Wang et al.) 2023c) can perform tasks such as speech recognition, speech translation, and speech synthesis. However, the above models are not built upon LLMs. To harness the power of LLMs, many studies explore how to build ppeech-language models based on LLMs like LLaMA, which can be further divided into two types. The first type, represented by SpeechGPT (Zhang al. 2023 2024a), AudioPaLM (Ruben- stein et al. (023), and Any (Zhan et al.] 2024), involves creating native multimodal speech- models by adding speech tokens the L.M 's vocabulary and continuing pretraining using speech

---

Published as a conference paper at ICLR 2025
and text data. However, this approach typically requires a large amount of data and substantial computational resources. The second type typically involves adding a speech encoder before the LLM and finetuning the entire model to equip it with speech understanding capabilities (Shu et al.
2023
Deshmukh et al. (023)) such as speech recognition (Fathullah et al., 2024a Yu et al., 2024 2024). speech translation (Wu et al. [023] Wang et al. 2023a
Ma et al. [202bb] [Hono et al.
Chen
ct al.
2024), or other general speech-to-text tasks (Chu et al.) 2023; Tang et al., 2024 Chu et al.
2024
Fathullah et al., 2024b Das et al. 2024; Hu et al. 2024). However, these approaches typi-
cally focus only on speech or audio understanding without the ability to generate them. Compared to previous work, LLaMA-Ommi equips the LLM with both speech understanding and generation capabilities, enabling it to perform general speech instraction-followee
tasks. Recently, some con- temporary works such as Mini-Omni (Xie & Wu 2024) and Moshi (Défossez et al.] (024) have also focused on speech interactions with L.Mss, aiming to improve the quality of speech response by simultancously generating both text and speech. Compared to them, our advantages include: (1) LLaMA-Ommi is built upon the latest LLM --ama-3.1-8B-Ins-t.
model, which has stronger reasoning capabilities; (2) We utilize CTC to adaptively learn the alignment between speech and text responses, eliminating the need to pre-align speech and text during training; (3) Our training only uses 200K data samples, which is several orders of magnitude less than theirs, and the training requires only 4 GPUs for days, making the training cost significantly lower.
Simultaneous Generation Streaming generation aims to begin producing output before the en- tire input is received. This capability is crucial for maintaining synchronization between speak- ers and listeners in various scenarios, such as streaming speech recognition and simultaneous in- terpretation. In the case of large language models, having a streaming speech synthesis compo- nent can significantly reduce latency between the model and its users. Popular streaming gener- ation methods fall into three main categories: monotonie-attention-bne
methods (Raffel et al.) 2017). CTC-based methods (Graves et al.) (006b)), and Transducer-base methods (Graves] 2012). Monotonic-attention -based methods modify the traditional attention- based sequence-to-sequeee framework (Bahdanaul 2014) to support streaming generation. These methods rely on an exter- nal module to manage the READ/WRITE policy, which can be either fixed (e.g.. Wait-k (Ma al., 2018 or adaptive (e.g.. MMA (Ma et al.] 2019). EDAtt (Papi et al.] 2022). Seg2Seg (Zhang & Feng. 2024 CTC-based methods add blank symbol to the target vocabulary represent WAIT action. Streaming inference is achieved by removing adjacent repetitive tokens and blank symbols, which has proven effective in simultaneous interpretation and streaming speech synthesis (Ma et al.. [023 Zhang et al. 2024b Ma et al.) (024a). Transducer-basee methods are designed to bridge the gap between the non-autoregressive nature of CTC-based methods and the autoregressive dependency between target tokens. These approaches introduce an additional predictor to capture token depen- dencies, and their variants have also shown strong performance simultaneous interpretation (Liu ett al. 2021 Tang et al. 2023) and streaming speech synthesis (Chen et al. 202Ib. For the stream- ing TTS task, many studies employ straightforward lookahead strategies built upon models like Tacotron 2 (Shen et al. 2018), such as waiting for few future words (Ma al. 20200] Stephenson cct al., (020) or leveraging a language model to predict several future words (Sacki et al.) [021a]b] Liu etueta al.] 2022), enabling improved streaming speech synthesis. Dekel et al. (2024) achieves stream- ing speech synthesis simultaneously with the output stream of L.LMs by employing two cascaded streamable models that sequentially generate phonemes and speech in real-time.
6 CONCLUSION
In this paper, we propose an innovative model architecture, LLaMA-Omni, which enables low- latency and high-quality speech interaction with LLMs. LLaMA-Omni is built upon the latest Llama-3.1-8P 1-0-a-----------tteuee- model, with the addition of a speech encoder for speech understanding and a streaming speech decoder that can generate both text and speech responses simultaneously. To align the model with speech interaction scenarios, we construct speech instruction dataset InstructionS2S-200K. which contains 200K speech instructions along with the speech responses. Experimental results show that, compared to previous speech-language models, LLaMA-Ommn de- livers superior responses in both content and style, with a response latency as low as 236ms. More- over, training LLaMA-Omi Ammn requires less than 3 days on 4 GPUs, enabling rapid development of speech interaction models based on the latest LLMs. In the future, we plan to explore enhancing the expressiveness of generated speech responses and improving real-time interaction capabilities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their insightful and valuable comments. This paper is supported by National Natural Science Foundation of China (Grant No.62376260).
REFERENCES
Dzmitry Bahdanau. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0e 0473, 2014.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523-2255 2533, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL htteps://proceedd neuri caperper fi [stle/pprer/2222 File/ F122:357ccarrr001111111222222222
Jiawei Chen, Xu Tan, Yichong Leng, Jin Xu, Guihua Wen, Tao Qin, and Tie- Yan Liu. Speech-t: Transducer for text to speech and beyond. Advances in Neural Information Processing Systems, 34:6621-6633, 2021.
Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, and Satoshi Nakamura. LLaST: Improved end- to-end speech translation system leveraged by large language models. In Lun-Wei Ku, An- dre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lin- guistics ACL 2024, pp. 6976-6987, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL Eindings-acl.. 416
Hyundong Justin Cho, Nicolaas Paul Jedema, Leonardo F. R. Ribeiro, Karishma Sharma, Pedro Szekely, Alessandro Moschitti, Ruben Janssen, and Jonathan May. Speechworthy instruction- tuned language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 10652-10677 Miami, Florida, USA. November 2024. Association for Computational Linguis- tics.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07911 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin. Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint ar0iv:2407. 10..44977.999 2024.
Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, et al. Speechverse: A large-scale generalizable audio language model. arXiv preprint arXiv:2405.08295 2024.
Alexandre Défossez, Laurent Mazaré, Manu Orsini. Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dia- logue. Technical report, Kyutai, September 2024. URL http::tttt:/ttttt http://///ouaaoooo//osooth. org/Moshi. ppf

---

Published as a conference paper at ICLR 2025
Avihu Dekel, Slava Shechtman, Raul Fernandez, David Haws, Zvi Kons, and Ron Hoory. Speak while you think: Streaming speech synthesis during text generation. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 11931- 11935. JEFE, 2024.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090-18108, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling nigh-quality instructional conversations. arXiv preprint urXiv:2305. 14233, 2023.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint arXiv:2407.0544 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2177 22777127775 2024.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin. Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadcokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13351-13355. IEEE, 2024a.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Ma- hadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Audiochatllama: Towards general- purpose speech abilities for 1lms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Papers), pp. 5522 5532, 2024b.
Alex Graves. Sequence transduction with recurrent neural networks 1rXiv:1211.377 2012.
arXiv preprint
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: Labelling unsegmented sequence data with recurrent neural networks, In Pro- ceedings of the 23rd International Conference on Machine Learning, ICML '06. pp. 369-376, New York, NY, USA, 2006a. Association for Computing Machinery. ISBN 1595933832 143844. 143891. http::ttt////tt /ttp://////..orr//...1111////110
doi:
10. 1111/111114//// 1143891
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006b.
Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki. and Kei Sawada. In- tegrating pre-trained speech and language models for end-to-end speech recognition. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 13289-13305, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL httpt::////// http///aaantttlll.oooorg///////e
Findings acl. 787
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia. Ruslan Salakhutdinov, and AbdeIrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00653 2024.
Keith Ito and Linda Johnson.
The lj speech dataset. bttps://keithito. com/
I T.a-speech-Datasete 2017.
12

---

Published as a conference paper at ICLR 2025
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp- 5530-5540. PMLR, 2021.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversar- ial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In-
formation Processing Systems, volume 33, pp. 17022-17033. Curran Associates, Inc.,
2020. URL https:://roceeeete ngs www.insur..... cc/paper..pppreeeppe
wwteeerfffeeepppprrrrreee pasper/pp/////
File/ Rlle:/c667Sc00776662226666666623
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354 2021.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cedtuuction-followiin models. https::/
com coomttatsu-laa com./tatsso-aabaapaaaaaa _aval] 2023.
Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen. Cross attention augmented transducer networks for simultaneous translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 39-55, 55, 2021.
Danni Liu, Changhan Wang, Hongyu Gong, Xutai Ma, Yun Tang, and Juan Pino. From start finish: Latency reduction strategies for incremental speech synthesis in simultaneous speech-to-speech translation. In Interspeech 2022, pp. 1771-1775, 2022. doi: 10.21437/lneerspecchh2222 2022- 10568.
Mingho Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, et al. Stacl: Simultaneous translation with im- plicit anticipation and controllable latency using prefx-to-preff framework. arXiv preprint arXiv:1810.08330 08..8000.89, 2018
Mingbo Ma, Baigong Zheng, Kaibo Liu, Renjie Zheng, Hairong Liu, Kainan Peng, Kenneth Church, and Liang Huang. Incremental cext-to-speech synthesis with prefix-to-preee framework. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pp. 3886-3896, 2020.
Xutai Ma, Juan Pino, James Cross, Liezl Puzon, and Jiatao Gu. Monotonic multihead attention. arXiv preprint 1909. 12406, 2019.
Zhengrui Ma. Shaolci Zhang, Shoutao Guo, Chenze Shao, Min Zhang, and Yang Feng. Non- autoregressive streaming transformer for simultaneous translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5177-5190, 2023.
Zhengrui Ma. Qingkai Fang, Shaolei Zhang, Shoutao Guo, Yang Feng, and Min Zhang. A non-autoregressive generation framework for end-to-end simultancous speech-to-any translation. arXiv preprint wwX.x:2440600633777 2024a.
Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang. Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. An embarrassingly simple approach for Ilm with strong capacity. arXiv preprint 2rXiv:2402.08846, 2024b.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, AbdeIrahman Mohamed, and Emmanuel Dupoux. Generative spoken dialogue language modeling. Transactions of the Association for Com- putational Linguistics, 11:250-266, 2023. doi: 10.1162/acl URL https://
aclanthology org wtw.://.00000222..1111111.
OpenAl. Introducing chatgpt, 2022. URL http:///ss///pppaaa..c/////l//e/
OpenAL. Hello gpt-4o, 2024. URL htt.:h:pp.///ppaa.cco///ooooeeeo
Sara Papi, Matteo Negri, and Marco Turchi. Attention as guide for simultaneous speech translation. arXiv preprint 0720:222/..07850, 2022.
13

---

Published as a conference paper at ICLR 2025
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech Resynthesis from Discrete Disentangled Self- Superviuee Representations. In Proc. Interspeech 2021, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-28518. PMLR, 2023.
Colin Raffel, Minh-Tharn Luong, Peter Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. In International conference on machine learning, pp. 2837-2846. PMLR, 2017.
Paul K Rubenstein, Chulayuth Asawaroengchai, Due Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.1299 2023.
Takaaki Saeki, Shinnosuke Takamichi, and Hiroshi Saruwatari. Incremental text-to-speech synthesis using pseudo lookahead with large pretrained language model. IEEE Signal Processing Letters, 28:857-861, 2021a.
Takaaki Sacki, Shinnosuke Takamichi, and Hiroshi Saruwatari. Low-latency incremental text-to- speech synthesis with distilled context prediction network. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 749-756. IEEE, 2021b.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. In Interspeech 2022, pp. 4521 4525, 2022. doi: 2t2244///neerpppccch2222222
2022--439...
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvr- giannakis, and Yonghui Wu. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783, 2018. doi: N0.1109/CCASSP.20188 1.. 8,88616886.. 368.
Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. Llasm: Large language and speech model. arXiv preprint arXiv:2308. 15930, 2023.
Brooke Stephenson, Laurent Besacier, Laurent Girin, and Thomas Hueber. What the future brings: Investigating the impact of lookahead for incremental neural its. In Interspeech, 2020. URL
https://ap3 wemanteiiin wccho ar www.aa.r.oposoosatar//aa 221507498
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference Learning Representations, 2024. URL https: 7 //openreview.
Yun Tang, Anna Sun, Hirofumi Inaguma, Xinyue Chen, Ning Dong, Xutai Ma, Paden Tomasello, and Juan Pino. Hybrid transducer and attention based encoder-decoder modeling for speech- to-text tasks. In Proceedings of the 6/st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12441-12455, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnstruction-folloiiin llama model. https:://iitee wtw/ seatsu-labbb http:aaaaoordaaaaaaaaaeaeaa
2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL haarae/55550aaae1010oooaaaaaaaaa
fi 21le/355e224547deee99
14

---

Published as a conference paper at ICLR 2025
Chen Wang. Minpeng Liao. Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. Blsp: Bootstrapping eeguage-speecee pre-training via behavior align- ment of continuation writing. arXiv preprint 1rXXi:2300900016.. 2023a.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.00 3it::2330.021,,,, 2023b.
Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305. 16107, 2023c.
Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang. Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, al. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. 16725, 2024.
Wenyi Yu, Changli Tang, Guangzhi Sun. Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma. and Chao Zhang. Connecting speech encoder and large language model for asr. In ICASSP 2024- 2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 12637-12641. IEEE, 2024.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang. Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, and Xipeng Qiu. AnyGPT: Unified multimodal LLM with discrete sequence modeling. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisticy (Volume 1: Long Papers), pp. 9637-9662, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1/226644/1/2024. acl-long.522 URL https://ac www.ollnnthhhooyooooooooooo org. 2024 ac 211224.cl-loo long.52.222.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp- 15757-15773, Singapore, December 2023. As- sociation for Computational Linguisties. doi: 10.18653/v 10133///1222...rm 1411665//222...idnng-ommpp...pp0 win.inng--emmlp.005.. URL http:://aclaanthoogy.ooooooo
f findings emn.p 10155
Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechgpt-gen: Scaling chain-of-informaaiin speech generation. arXiv preprint arXiv:2401. 13527, 2024a.
Shaolei Zhang and Yang Feng. Unified segnent-to-segment framework for simultaneous sequence generation. Advances in Neural Information Processing Systems, 36, 2024.
Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. Stream- speech: Simultancous speech-to-speech translation with multi-task learning. arXiv preprint arXiv:2406.0300 03049, 2024b.
15

---

Published as a conference paper at ICLR 2025
A PROMPT
Prompt template of AaaMAAAOm
< lbbein of text >> Istart header id >systen< end header id You are a helpful language and speech assistant. You are able to understand the speech content that the user provides, and assist the user with variety of tasks using natural language. < eot_id|>
[tstat.header >user</ (end_header.dd/>
<speech>
Please answer the questions in the user's input speech. < cott_idl>>
sttat_header _id Sassistant
Prompt for ChatGPT Scoring (Model: GPT-44)
need your help to evaluate the performance of several models in a speech interaction scenario. The models receive the user's speech input and respond with speech output. For evaluation purposes, both the user's speech input and the model's speech response have been transcribed into text using Automatic Speech Recognition (ASR). Your task is rate the model's responses based the provided user input transcription [Instruction] and model's output transcription [Response]. Please consider factors such as helpfulness, relevance, fluency, and suitability for speech interaction in your evaluation, and provide single score on scale from 1 5.
Below are the transcription of user's 's instruction and models' response: ### [Instruction]: [instruction]
[Response]: [response]
After evaluating, please output the scores in JSON format: explanations.
(score: ...}. You don't need to provide any
Prompt for Instruction Rewriting (Model: Llama-3- 70B-70BB-nnnnt
Below is instruction data containing the user's instruction. would like generate speech version of this instruction for training a large language model that supports speech input. Therefore, please rewrite my instruction data according to the following requirements:
1. Modify the instruction to simulate human speech, adding fillers as appropriate (but not too many "you know', 'like', etc.).
The question should not contain content that cannot be synthesized by the TTS model. Numbers should be written in English words rather than Arabic numerals. The question should be relatively brief without excessive verbiage.
[instruction]: finstruction]
Please output in JSON format as follows: ("question": {question}
Prompt for Response Generation (Model: -lama-3-70B-Instnuee
Below the transcribed text user's speech query. Please provide a response to this question, which will be converted to speech using TTS. Please follow these requirements for your response:
1. Your response should not contain content that cannot be synthesized by the TTS model, such as paren- theses, ordered lists, etc. Numbers should be written in English words rather than Arabic numerals. 2. Your response should be very concise and to the point, avoiding lengthy explanations.
[instruction]: {instruction]
Please output in JSON format as follows: {"response": {response)
B CASE STUDY
To intuitively understand the differences in responses from different models, provide example Table 3 It can be observed that the response Qwen2-Audio are quite lengthy and include elements such line breaks and parentheses that cannot be synthesized into speech. The response
16

---

Published as a conference paper at ICLR 2025
Table Responses from different models for the instruction: "How do I wrap a present neatly?"
Model
Response
Qwen2-Audio
Wrapping a present neaty can a tricky. but with some practice and patience, you can master the art of wrapping presents beautifully. Here are some steps follow;)n)nl. Start with clean, flat surface: Find clean, flat surface where well. (al,. Gather your supplies: You will need wrapping paper, gift wrap scissors, a ruler or measuring tape, and any other decorations you want to use (such as ribbon, bows, or stickers). (n' n3, Meusure the width and length of the gift. Make to add little extra
work on wrapping your present. A large table or countertop works
gift:
Use the ruler measuring lape space uround the edges for wrapping. in Cut the wrapping puper: Cut the wrapping puper to the size you measured. If you have multiple pieces of wrapping paper, layer them together so that they fit neatly sides
around the An Fold paper: Begin by folding one crease well. and then fold the other side over the top. This creates simple, yet attractive fold. n6. Add decorations: If you like, you can add decorations such as ribbon, bows, or stickers to the gift. Simply attach bottom Bring the left side of the paper up over gift, tuck it in tightly. and press down. Repeat this process the right side of the gift. bringing the right side of the puper up over the gill, tucking it in tightly, and pressing it down. down.'n',n 'n. Finish the edges: Once you have wrapped the gift, trim off any excess puper along edges using the gift wrap scissors. You can also add additional touches such bow decorative label complete the look. nnnnRemmeeerr the wrapping present neatly to take your time and pay atlention to detail. With practice, you'll 'Il be able to create heautiful wrapping paper presentations every time!
wrapping paper over the gift,
them to the top the gift using glue or double-sided tape. ( Wrap the gift: Starting of gift, place piece wrapping paper
SALMONN
To wrap present neatly, you can start by laying out clean and surface to work on. Then, place the present
surface and wrap the paper in half and then in half again, so that the top and bottom flaps meet in the middle. Then, fold the left and right flaps in towards the center, so that they moet at the top and bottom of the present. Finally, seal the present by pressing the flaps together and using some tape to secure them.
piece wrapping paper that slightly larger than present. Fold
SpeechGPT
Wrapping present neatly requires a few simple steps. First, you should fold puper hulf and then unfold it. Then, you should place the present in the center the paper and fold the paper around Finally, you should tie the ends of the puper with ribbon or string.
L.AMA-Ommi
To wrap a present neatly, start with flat surface, place the gift in the center, and fold one edge of the wrapping paper over the gifl, crease it, and tape down. Repeat this process with the remaining edges, making sure align evenly smooth any air bubbles.
from SALMONN is also a bit long. The style SpeechGPT's response is more appropriate for speech interaction scenarios, but the amount of information contained in its responses is less. In contrast, the response given by LLaMA--mmi is more detailed and helpful while maintaining concise style, outperforming previous models in speech interaction scenarios.
C NUMERICAL. RESULTS IN THE STREAMING SCENARIO
We present the numerical results of all models in streaming scenarios in Tables 4 and
Table 4: Numerical results of LLaMA-Omni in the streaming scenario.
Latency (ms) Vocoder
Ω
ChatGPT Score
ASR-WER
UTMOS
WPS
LLM
Total
10
206.03
30.15
236. 18
3.54
9.84
3.2304
2.76
20
236.18
45.23
281.41
3.56
9.91
3.4748
2.75
40
301.51
45.23
346.73
3.52
10.37
3.6688
2.74
60
361.81
50.25
412.06
3.52
10.47
3.7549
2.74
80
432.16
55.28
487.44
3.50
10.70
3.7858
2.73
100
497.49
65.33
562.81
3,49
10.71
3,8242
2.74
Offline
1542.71
211.06
1753.77
3.47
10.82
3,9296
2.73
17

---

Published as a conference paper at ICLR 2025
Table Numerical results of SpeechGPT the streaming scenario.
Latency (ms)
R
ChatGPT Score
ASR-WER
UTMOS
WPS
LLM
Vocoder
Total
10
4899.50
30.15
4929.65
2.16
44.85
2.7099
1,95
20
5005.03
30.15
5035.18
2.22
42.03
3.1920
1.86
40
5547.74
40.20
5587.94
2.17
43.88
3.5106
1.89
60
6005.03
45.23
6050.25
2.16
43.35
3.6293
1.84
80
6160.80
55.28
6216.08
2.21
43.98
3.6970
1.87
100
8301.51
65.33
8366.83
2.20
43.74
3.7397
1,86
Offline
17919.60
170.85
18090.45
2.19
45.00
3,8958
1.85
Table Numerical results of SALMONN Orca the streaming scenario.
Latency (ms)
ChatGPT Score
ASR-WER
UTMOS
WPS
LLM
TTS
Total
212.45
19.71
232.16
3.28
6.64
3.0947
1.86
3
316.59
35.08
351.67
3.09
8.65
3.7338
2.91
5
428.79
32.08
460.87
3.04
9.23
3.7750
3.18
7
536.47
45.90
582.37
3.06
8.49
3.7972
3.20
9
659.57
69.38
728.95
3.23
7.07
3.8060
3.22
Offline
4274.48
1049.59
5324.07
3.40
3.78
3.8286
3.31
Table 7: Numerical results of Qwen2-Audio + Orca in the streaming scenario.
Latency (ms) TTS
ChatGPT Score
ASR-WER
UTMOS
WPS
LLM
Total
289.46
19.15
308.61
2.79
25.25
2.8597
1.90
3
381.66
38.19
419.85
2.95
13.30
3.5529
2.91
5
470.55
38.64
509.19
2,93
13.00
3,5865
3.08
7
568.22
52.95
621.17
3.07
10.59
3.5739
3.11
675.55
81.02
756.57
3.14
9.81
3.6016
3.13
Offline
7062.93
2361.49
9424.42
3.38
6.77
3.6119
3.30
18


Reviews:
[{"review_id": "OtwhKMyIu6", "reviewer": "Reviewer_MWRj", "summary": "This paper proposed a Speech-LLM model that can do speech-to-speech conversation with good latency and quality. The central design enables the low latency feature is the proposed streaming speech decoder.", "questions": "please resolve the concerns in Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This is probably the first public work that connects a speech-to-text Speech-LLM with a streaming text-to-speech model so as to enable low latency speech outputs.", "weaknesses": "1. The speech-to-text component of this model is not new (relevant works like SALMONN and Qwen2-audio) and the main addition is the text-to-speech component. The latter has a clear connection with the streaming TTS model research but the paper does not spend enough content to acknowledge and compare to this line of research. To elaborate on this, the paper should compare the proposed method with prior research in this space and motivate this unique CTC based design. The paper should also replace the baseline of SALMONN+TTS with SALMONN+streaming TTS so as to have a fair comparison and show the value of the proposed component. \n2. The \"response latency\" used in this paper is misleading. It only considers the latency from streaming speech decoder, but in fact the non-streaming speech-to-text Speech-LLM also has latency which should be considered as \"response latency\". The paper fails to point this out and didn't give the overview of the latency breakdown given the proposed design \n3. The audio quality of the proposed method is not good from the MOS score and is not compared with any external works, which challenges the value of this proposed CTC based design, especially considering there have been prior works in the streaming TTS area. It's hard to understand whether the proposed method is better or worse.\n4. \"On the other hand, the ASR model we use, Whisper-large-v3, has strong robustness. Even when the speech is somewhat discontinuous with smaller Ω,\" If this ASR model cannot identify the discontinuity and relevant quality issue, it may not serve well as a way to evaluate the intelligence of the generated speech. This is a limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NNAIm1gOlU", "reviewer": "Reviewer_ppgU", "summary": "This paper proposes LLaMA-Omni, an end-to-end model architecture designed for low-latency, high-quality speech interaction with large language models (LLMs). LLaMA-Omni integrates multiple components: a pre-trained speech encoder, a speech adaptor, a large language model, and a streaming speech decoder. To enable more natural, human-like speech interactions, the authors introduce InstructS2S-200K, a dataset of 200K speech instructions and responses crafted in a conversational, oral style, to fine-tune the model for speech-based dialogue. Experimental results show that LLaMA-Omni achieves low-latency speech responses while reducing training costs compared to similar models.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Conversational Data with InstructS2S-200K:** The proposed InstructS2S-200K dataset addresses a notable gap in conversational, oral-style speech instruction data. By providing a large, purpose-built dataset tailored to natural, interactive speech patterns, this work supports more effective alignment of LLM responses with human conversational norms. \n2. **Cost-Effective Training Process:** The two-stage training process adopted in LLaMA-Omni demonstrates a practical approach to reducing training costs. Compared to prior end-to-end training methods, this staged training setup optimizes resource use, enabling faster model development without compromising performance.", "weaknesses": "1. **Lack of Novelty:** While the authors suggest that they proposed a \"novel model architecture,\" it appears to primarily build on a combination of existing methods without a clear breakthrough in architecture: \n   1) The connection of a speech encoder to an LLM via a speech adaptor has been widely explored in prior research. \n   2) The streaming TTS module seems to be directly based on previous designs, specifically following the approach outlined in [1]. \n   \n    In summary, while LLaMA-Omni successfully assembles a functional pipeline for speech-based interactions with LLMs, the architecture seems to build on prior work without delivering significant innovation in model design, which somewhat limits the paper’s impact from a research innovation standpoint. Further discussion is needed to emphasize the unique contribution of this work.\n2. **Questionable Experiment Results:** Some experiment results in this paper is questionable:\n   1) **ChatGPT-Score in Table 2:** While LLaMA-Omni shows improved performance over baseline models in ChatGPT-Score, this result is potentially misleading. The evaluation prompt used (detailed in Appendix A) was crafted to reward concise responses and penalize extraneous information. This prompt design aligns closely with the prompts used to construct the InstructS2S-200K dataset, creating a circular evaluation condition where the model is likely to excel by design. A more objective and meaningful approach would involve conducting a human evaluation to capture human preferences more accurately. \n   2) **WER & CER in Table 2:** In Table 2, Qwen2-Audio + TTS shows an unexpectedly high WER of 55.72%, yet it achieves a relatively moderate S2SIF score of 2.32/2.58, which implies a more acceptable response quality. This discrepancy suggests an inconsistency, as such a high WER would typically indicate a low intelligibility level that should correlate with a poor S2SIF score. This contradiction casts doubt on the reliability of the WER measurements or, potentially, the effectiveness of the S2SIF scoring methodology when used alongside TTS outputs.\n    \n        A likely explanation for these irregularities lies in the TTS model used, VITS, which is primarily trained on **short audio clips** [2]. This limitation may hinder VITS’s ability to process longer inputs effectively, causing high error rates when it encounters the longer outputs of models like Qwen2-Audio and SALMONN. Sending unchunked, lengthy responses directly to VITS without segmentation could result in degraded performance, as evidenced by the elevated WER and CER scores. \n\n        To address these limitations and obtain more accurate metrics, it would be more appropriate to adopt a TTS model trained for longer utterances or to chunk input text before passing it to VITS. Such adjustments would likely yield more realistic WER and CER values and provide a fairer basis for comparing baseline models.\n    3) **Average decoding time in Table 4:** The decoding time reported in Table 4 does not convincingly demonstrate the superiority of the proposed streaming decoding architecture. The average decoding time is measured for generating complete speech responses; however, since baseline methods produce longer responses, they naturally require more time for decoding. This dependency on response length makes the reported decoding times trivial and limits their effectiveness in showcasing any advantages of the proposed approach. To provide a fair and meaningful comparison, it would be more appropriate to compare with Qwen2-Audio + streaming TTS model and report decoding time per token rather than total decoding time. \n3. **Limited speaker support:** While InstructS2S-200K provides valuable conversational data, it has limitations due to the lack of speaker diversity. The input speech in this dataset is synthesized using CosyVoice-300M-SFT, which represents a single voice and does not capture the variability in accents, pitch, tone, and other speaker characteristics found in real-world interactions. This limitation restricts LLaMA-Omni’s adaptability to diverse user-profiles and reduces its effectiveness in applications requiring personalized or varied speaker responses. Integrating a broader range of voices could significantly improve LLaMA-Omni’s applicability in real-world scenarios.\n\n\n[1] Fang Q, Ma Z, Zhou Y, et al. CTC-based Non-autoregressive Textless Speech-to-Speech Translation[J]. arXiv preprint arXiv:2406.07330, 2024.\n\n[2] Kim J, Kong J, Son J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech[C]//International Conference on Machine Learning. PMLR, 2021: 5530-5540.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vMxc16lwyh", "reviewer": "Reviewer_DXz7", "summary": "This work presents an end-to-end spoken dialogue model LLAMA-Omini, which integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming TTS module. The model can directly understand continuous speech features and can simultaneously generate text and speech responses directly with low latency. To achieve alignment between different modules, this work also proposes a pipeline for constructing speech-to-speech dialogue data, with a style that better matches the characteristics of speech interaction scenarios.", "questions": "1. Could the author please emphasize the contribution of this article?\n   - If it is a streaming TTS module, there is a lack of comparison with other streaming generation methods (autoregressive TTS models)\n   - if it is the speech instruction dataset, there is a lack of human evaluation to prove its greater alignment with human preference, and the limitations of TTS synthesized speech input also restrict its application\n   - if it is an open-source seamless spoken dialogue model, evaluation on real speech benchmarks is also needed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and presents a promising approach to seamless speech interaction with LLMs. The reported response latency of 226ms is impressive and demonstrates the value for real-world applications.\n\n2. The authors have made a valuable contribution by creating the InstructS2S-200K dataset, which is tailored to speech interaction scenarios. The core motivation for building this dataset is that \"in speech interactions, concise yet informative responses are typically preferred\", which has not been considered in previous work.", "weaknesses": "1. Although this work presents a seamless spoken dialogue model, the choice of connecting a LLM with a CTC-based streaming TTS module is not well-motivated.\n    - Autoregressive TTS models naturally support streaming decoding, but this paper does not discuss or compare with the traditional autoregressive decoding.\n\n    - The CTC-based streaming TTS module incorporates ideas from StreamSpeech [1] and achieves streaming output by segmenting generated units into chunks. However, it is not evident what benefits the end-to-end model offers over a straightforward cascaded solution where the text response is segmented into chunks and then fed into an offline TTS model.\n\n2. Another contribution of this work is the constructed dataset InstructS2S-200K, but there is a lack of sufficient experiments to demonstrate the advantages of this dataset.\n\n   - First of all, the speech input in this dataset is synthesized using CosyVoice-300M-SFT, which can only synthesize standard speech from limited speakers. This may lead to the model only being able to handle tts-synthesized input speech, lacking speaker generalization. Therefore, in order to demonstrate the practical value of this dataset, evaluation on human recorded speech is necessary. For example, ASR and ST performance in academic dataset (LibriSpeech, CoVoST), or evaluation on general benchmarks (such as AIRBench [2]) is necessary to demonstrate the model's understanding of real speech. But this work is only evaluated on their TTS-synthesized InstructS2S-Eval.\n\n   - One advantage of this dataset is that the response is accurate and concise, which is more suitable for the speech interaction scenario. However, in order to prove this statement, human evaluation is very necessary to demonstrate that humans would prefer shorter replies. In Table 1, the author uses GPT4 as a judge, but in Appendix A, we can see that the author has incorporated the need for short and accurate responses into the GPT4 evaluation prompt. This can only prove that the model's responses are accurate and concise, but cannot prove that they are more in line with human preferences.\n\n3. There are some unconvincing evaluation metrics and results analysis.\n\n   - In Table 1, a biased prior has been introduced in the evaluation prompt of GPTscore (see 2.b), therefore the content and style scores are not reasonable.\n\n   - In Table 1, the pipeline models SALMOON/Qwen2-Audio + TTS have very poor alignment scores. Although the authors explained in Line 392-393 that the responses of these two models contain characters that cannot be synthesized into speech, we can see in the case study in Table 5 that SALMMON does not have any unsynthesizable characters, and Qwen2-Audio only has \\n that is unsynthesizable. I speculate that the very poor alignment score here is due to the fact that the TTS model used only supports synthesizing speech for around 10 seconds, while SALMOON and Qwen2-Audio generate very long responses that exceed the maximum supported length of the TTS model. I believe the authors need to provide a detailed description of how they used the TTS model, or to segment long text responses into shorter chunks to fed into TTS model to ensure fairness in comparison.\n\n   - Table 4 reports the total time for completing a single response, and it is quite apparent that shorter responses result in shorter completion times. However, this does not provide any new insights or perspectives to demonstrate the advantages of this work.\n\n4. The author emphasizes that LLAMA-Omini is an interactive dialogue model, or a GPT-4o-like model, and therefore needs to demonstrate its ability for multi-turn conversations. However, evaluations have only been conducted on single-turn responses. I think that quantitative metrics or qualitative demos are necessary.\n\n[1] Zhang, Shaolei, et al. StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning. ACL 2024.\n\n[2] Yang, Qian, et al. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. arxiv 2402.07729.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vv51UdMb6X", "reviewer": "Reviewer_4A7w", "summary": "This paper introduces LLama-Omni, a novel end-to-end architecture that enables real-time interaction with LLMs through speech. Llama-Omni has a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. Speech is fed continuously into the LLM, while text is generated discretely. Then, each text unit is replicated a fixed number of times and aligned with HuBERT k-means units using non-autoregressive CTC. Once the units accumulate to a preset number, they form a chunk and feed to a vocoder for speech reconstruction. This non-autoregressive architecture supports simultaneous text and speech generation from spoken instructions with very low latency, while achieving very low training overhead.", "questions": "- Is there a risk of audio artifacts, such as \"popping\", at the boundaries when using the vocoder to generate speech chunks? If so, how does the model mitigate these issues?\n\n- Could you provide more detailed comparisons between LLaMA-Omni and other speech-interactive LLMs to evaluate whether introducing speech as an input/output modality affects the model’s reasoning and language comprehension abilities? Specifically, does enabling speech interaction in a text-based LLM reduce its core language understanding and intelligence? Personally, I evaluated another GPT-4o-like model, mini-Omni, and found that their method to introduce the speech modality significantly reduced the language model's core intelligence. Although the authors repeatedly claimed in the paper that their method could \"retain the original model’s language capabilities with minimal degradation,\" they provided no evaluation to substantiate this claim.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- LLama-Omni carries out a novel non-autoregressive architecture for simultaneous text and speech generation, providing a fresh approach among recent GPT-4o-like models. It achieves a sufficiently low latency of 226ms and high-quality speech interaction, making it suitable for real-time scenarios.\n\n- LLaMA-Omni can be integrated with any open-source LLM at minimal training cost, requiring only 4 GPUs and less than 3 days to train. This makes it accessible for academic research, allowing more institutions and researchers to use high-quality, speech-interactive LLMs.\n\n- One notable strength is that LLaMA-Omni largely preserves the intelligence of pre-trained LLMs.", "weaknesses": "- One limitation is that the LLM outputs text only. While this preserves its reasoning abilities, it losses richer features such as emotional nuance and speaker identity.\n\n- As the authors mentioned in Section 4.4, the cascaded use of VITS TTS in SALMONN and Qwen2-Audio may introduce errors affecting S2SIF and alignment due to propagated TTS inaccuracies, making this comparison less of an apple-to-apple evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)