LLaMA-Omni: Seamless Speech Interaction with Large Language Models

## Abstract

Abstract Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future. 1 1 1 Code and model are available at https://github.com/ictnlp/LLaMA-Omni .

## 1 Introduction

Large language models (LLMs), represented by ChatGPT (OpenAI, ), have become powerful general-purpose task solvers, capable of assisting people in daily life through conversational interactions. However, most LLMs currently only support text-based interactions, which limits their application in scenarios where text input and output are not ideal. Recently, the emergence of GPT-4o (OpenAI, ) has made it possible to interact with LLMs through speech, responding to user’s instruction with extremely low latency and significantly enhancing the user experience. However, there is still a lack of exploration in the open-source community on building such speech interaction models based on LLMs. Therefore, how to achieve low-latency and high-quality speech interaction with LLMs is a pressing challenge that needs to be addressed.

The simplest way to enable speech interaction with LLMs is through a cascaded system based on automatic speech recognition (ASR) and text-to-speech (TTS) models, where the ASR model transcribes the user’s speech instruction into text, and the TTS model synthesizes the LLM’s response into speech. However, since the cascaded system sequentially outputs the transcribed text, text response, and speech response, the overall system tends to have higher latency. In contrast, some multimodal speech-language models have been proposed (Zhang et al., ; Rubenstein et al., ), which discretize speech into tokens and extend the LLM’s vocabulary to support speech input and output. Such speech-language models theoretically can generate speech responses directly from speech instructions without producing intermediate text, thereby achieving extremely low response latency. However, in practice, direct speech-to-speech generation can be challenging due to the complex mapping involved, so it is common to generate intermediate text to achieve higher generation quality (Zhang et al., ), although this sacrifices some response latency.

In this paper, we propose a novel model architecture, LLaMA-Omni, which enables low-latency and high-quality interaction with LLMs. LLaMA-Omni consists of a speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. The user’s speech instruction is encoded by the speech encoder followed by the speech adaptor, and then input into the LLM. The LLM decodes the text response directly from the speech instruction, without first transcribing the speech into text. The speech decoder is a non-autoregressive (NAR) streaming Transformer (Ma et al., ), which takes the output hidden states of the LLM as input and uses connectionist temporal classification (CTC; Graves et al., ) to predict the sequence of discrete units corresponding to the speech response. During inference, as the LLM autoregressively generates the text response, the speech decoder simultaneously generates the corresponding discrete units. To better align with the characteristics of speech interaction scenarios, we construct a dataset named InstructS2S-200K by rewriting existing text instruction data and performing speech synthesis. Experimental results show that LLaMA-Omni can simultaneously generate high-quality text and speech responses with a latency as low as 226ms. Additionally, compared to previous speech-language models like SpeechGPT (Zhang et al., ), LLaMA-Omni significantly reduces the required training data and computational resources, enabling the efficient development of powerful speech interaction models based on the latest LLMs.

Figure: Figure 1: LLaMA-Omni can simultaneously generate text and speech responses based on the speech instruction, with extremely low response latency.
Refer to caption: x1.png

## 2 Model: LLaMA-Omni

In this section, we introduce the model architecture of LLaMA-Omni. As shown in Figure 2, it consists of a speech encoder, a speech adaptor, an LLM, and a speech decoder. We denote the user’s speech instruction, text response, and speech response as $X^{S}$, $Y^{T}$, and $Y^{S}$ respectively.

### 2.1 Speech Encoder

We use the encoder of Whisper-large-v3(^2^22[https://huggingface.co/openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)) (Radford et al., ) as the speech encoder $\mathcal{E}$. Whisper is a general-purpose speech recognition model trained on a large amount of audio data, and its encoder is capable of extracting meaningful representations from speech. Specifically, for the user’s speech instruction $X^{S}$, the encoded speech representation is given by $\mathbf{H}=\mathcal{E}(X^{S})$, where $\mathbf{H}=\left[\mathbf{h}_{1},...,\mathbf{h}_{N}\right]$ is the speech representation sequence of length $N$. We keep the speech encoder’s parameters frozen throughout the entire training process.

### 2.2 Speech Adaptor

To enable the LLM to comprehend the input speech, we incorporate a trainable speech adaptor $\mathcal{A}$ that maps the speech representations into the embedding space of the LLM. Following  Ma et al. (), our speech adaptor first downsamples the speech representations $\mathbf{H}$ to reduce the sequence length. Specifically, every $k$ consecutive frames are concatenated along the feature dimension:

$$ $\mathbf{H}^{\prime}=\left[\mathbf{h}^{\prime}_{1},...,\mathbf{h}^{\prime}_{ \left\lfloor N/k\right\rfloor}\right],\text{where}\ \mathbf{h}^{\prime}_{i}= \left[\mathbf{h}_{k\times(i-1)+1}\oplus\mathbf{h}_{k\times(i-1)+2}\oplus\cdots \oplus\mathbf{h}_{k\times i}\right].$ (1) $$

Next, $\mathbf{H}^{\prime}$ is passed through a 2-layer perceptron with ReLU activation between the linear layers, resulting in the final speech representation $\mathbf{S}$. The above process can be formalized as follows:

$$ $\mathbf{S}=\mathcal{A}(\mathbf{H})=\text{Linear}(\text{ReLU}(\text{Linear}( \text{DownSample}(\mathbf{H})))).$ (2) $$

Figure: Figure 2: Left: Model architecture of LLaMA-Omni. Right: Illustration of the two-stage training strategy for LLaMA-Omni.
Refer to caption: x2.png

### 2.3 Large Language Model

We use Llama-3.1-8B-Instruct(^3^33[https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)) (Dubey et al., ) as the LLM $\mathcal{M}$, which is currently the state-of-the-art open-source LLM. It has strong reasoning capabilities and is well-aligned with human preferences. The prompt template $\mathcal{P}(\cdot)$ is shown in Figure 3. The speech representation sequence $\mathbf{S}$ is filled into the position corresponding to <speech>, and then the entire sequence $\mathcal{P}(\mathbf{S})$ is input into the LLM. Finally, the LLM autoregressively generates the text response $Y^{T}=[y^{T}_{1},...,y^{T}_{M}]$ directly based on the speech instruction and is trained using cross-entropy loss:

$$ $\mathcal{L}_{\text{LLM}}=-\sum_{i=1}^{M}\log P(y^{T}_{i}|\mathcal{P}(\mathbf{S }),Y^{T}_{<i}).$ (3) $$

Figure: Algorithm 1 Inference Process

### 2.4 Speech Decoder

For the speech response $Y^{S}$, we first follow  Zhang et al. () to discretize the speech into discrete units. Specifically, we use the pretrained HuBERT (Hsu et al., ) model to extract continuous representations of the speech, and then convert these representations into discrete cluster indices using a K-means model. Subsequently, consecutive identical indices are merged into a single unit, resulting in the final discrete unit sequence $Y^{U}=[y^{U}_{1},...,y^{U}_{L}],y^{U}_{i}\in\{0,1,...,K-1\},\forall 1\leq i\leq
L$, where $K$ is the number of clusters, and $L$ is the length of discrete unit sequence. The discrete units can be converted into waveform with an additional unit-based vocoder $\mathcal{V}$ (Polyak et al., ).

To generate speech responses simultaneously with text responses, we add a streaming speech decoder $\mathcal{D}$ after the LLM. It consists of several standard Transformer (Vaswani et al., ) layers with the same architecture as LLaMA (Dubey et al., ), each containing a causal self-attention module and a feed-forward network. Similar to Ma et al. (); Zhang et al. (), the speech decoder runs in a non-autoregressive manner, which takes the output hidden states from the LLM as input, and generates the discrete unit sequence corresponding to the speech response. Specifically, the output hidden states corresponding to the text response are denoted as $\mathbf{C}=[\mathbf{c}_{1},...,\mathbf{c}_{M}]$, where $\mathbf{c}_{i}=\mathcal{M}(\mathcal{P}(\mathbf{S}),Y^{T}_{<i})$. We first upsample each hidden state into a chunk by a factor of $\lambda$, resulting in an upsampled hidden state sequence $\widehat{\mathbf{C}}=[\widehat{\mathbf{c}}_{1},...\widehat{\mathbf{c}}_{
\lambda\cdot M}]$, where $\widehat{\mathbf{c}}_{i}=\mathbf{c}_{\lfloor i/\lambda\rfloor}$. Next, $\widehat{\mathbf{C}}$ is fed into the speech decoder $\mathcal{D}$, and the output hidden state sequence is denoted as $\mathbf{O}=[\mathbf{o}_{1},...,\mathbf{o}_{\lambda\cdot M}]$. We use connectionist temporal classification (CTC; Graves et al., ) to align $\mathbf{O}$ with the discrete unit sequence $Y^{U}$. Specifically, CTC extends the output space with a special blank token $\epsilon$:

$$ $P(a_{i}|\mathbf{O})=\text{softmax}(\mathbf{W}\mathbf{o}_{i}+\mathbf{b})[a_{i}] ,\forall a_{i}\in\{0,1,...,K-1,\epsilon\},$ (4) $$

where $\mathbf{W}\in\mathbb{R}^{(K+1)\times d}$ and $\mathbf{b}\in\mathbb{R}^{K+1}$ are weights and biases of the linear layer, and the sequence $A=[a_{1},...,a_{\lambda\cdot M}]$ is known as the alignment. To model the variable-length mapping between input and output, CTC introduces a collapsing function $\beta(A)$, which first merges all consecutive repeated tokens in $A$ and then eliminates all blank tokens $\epsilon$. For instance: $\beta([1,1,2,\epsilon,\epsilon,2,3])=[1,2,2,3]$. During training, CTC performs marginalization over all possible alignments as follows:

$$ $\mathcal{L}_{\text{CTC}}=-\log P(Y^{U}|\mathbf{O})=-\log\sum_{A\in\beta^{-1}(Y ^{U})}P(A|\mathbf{O})=-\log\sum_{A\in\beta^{-1}(Y^{U})}\prod_{i=1}^{\lambda \cdot M}P(a_{i}|\mathbf{O}),$ (5) $$

where $\beta^{-1}(Y^{U})$ denotes all possible alignments of length $\lambda\cdot M$ that can be collapsed to $Y^{U}$. The alignment is modeled in a non-autoregressive way. During inference, we select the best alignment $A^{*}=\arg\max_{A}P(A|\mathbf{O})$, and apply the collapsing function to obtain the discrete unit sequence $\beta(A^{*})$, which is then fed into the vocoder to synthesize waveform.

Figure: Figure 3: Prompt template of LLaMA-Omni.

### 2.5 Training

As shown in Figure 2, we adopt a two-stage training strategy for LLaMA-Omni. In the first stage, we train the model to generate text responses directly from the speech instructions. Specifically, the speech encoder is frozen, and the speech adaptor and the LLM are trained using the objective $\mathcal{L}_{\text{LLM}}$ in Eq. (3). The speech decoder is not involved in training during this stage. In the second stage, we train the model to generate speech responses. During this stage, the speech encoder, speech adaptor, and LLM are all frozen, and only the speech decoder is trained using the objective $\mathcal{L}_{\text{CTC}}$ in Eq. (5).

### 2.6 Inference

During inference, the LLM autoregressively generates the text response based on the speech instruction. Meanwhile, since our speech decoder uses causal attention, once the LLM generates a text response prefix $Y^{T}_{\leq i}$, the corresponding upsampled hidden states $\widehat{\mathbf{C}}_{\leq\lambda\cdot i}$ can be fed into the speech decoder to generate a partial alignment $A_{\leq\lambda\cdot i}$, which in turn yields the discrete units corresponding to the generated text prefix. To further enable streaming synthesis of speech waveforms, when the number of generated units reaches a pre-defined chunk size $\Omega$, we input this unit segment into the vocoder to synthesize a speech segment, which is then immediately played to the user. As a result, users can start listening to the speech response without waiting for the complete text response to be generated, ensuring low response latency that is not affected by the length of the text response. Algorithm 1 describes the above process. Additionally, since the speech decoder uses non-autoregressive modeling, the alignment corresponding to each text token $y^{T}_{i}$, specifically $A_{\lambda\cdot(i-1)+1:\lambda\cdot i}$, is generated in parallel within the chunk. Therefore, the decoding speed for generating both text and speech simultaneously is not significantly different from the speed of generating text alone.

## 3 Construction of Speech Instruction Data: InstructS2S-200K

To train LLaMA-Omni, we need triplet data consisting of <speech instruction, text response, speech response>. However, most publicly available instruction data is in text form. Therefore, we construct speech instruction data based on existing text instruction data through the following process:

#### Step 1: Instruction Rewriting

Since speech input has different characteristics compared to text input, we rewrite the text instructions according to the following rules: (1) Add appropriate filler words (such as “hey”, “so”, “uh”, “um”, etc.) to the instructions to simulate natural speech patterns. (2) Convert non-text symbols in the instructions (such as numbers) into their corresponding spoken forms to ensure correct synthesis by TTS. (3) Modify the instructions to be relatively brief without excessive verbiage. We use the Llama-3-70B-Instruct(^4^44[https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)) model to rewrite the instructions according to these rules. The prompt can be found in Appendix A.

#### Step 2: Response Generation

In speech interactions, existing responses from text instructions are not suitable for direct use as speech instruction responses. This is because, in text-based interactions, models tend to generate lengthy responses, using complex sentences and possibly including non-verbal elements like ordered lists or parentheses. However, in speech interactions, concise yet informative responses are typically preferred (Anonymous, ). Therefore, we use the Llama-3-70B-Instruct model to generate responses for speech instructions according to the following rules: (1) The response should not contain content that cannot be synthesized by the TTS model, such as parentheses, ordered lists, etc. (2) The response should be very concise and to the point, avoiding lengthy explanations. The prompt can be found in Appendix A.

#### Step 3: Speech Synthesis

After obtaining the instructions and responses suitable for speech interactions, we need to further convert them into speech using TTS models. For the instructions, to make the synthesized speech sound more natural, we use the CosyVoice-300M-SFT (Du et al., ) model(^5^55[https://github.com/FunAudioLLM/CosyVoice](https://github.com/FunAudioLLM/CosyVoice)), randomly selecting either a male or female voice for each instruction. For the responses, we use the VITS (Kim et al., ) model(^6^66[https://github.com/jaywalnut310/vits](https://github.com/jaywalnut310/vits)) trained on the LJSpeech (Ito & Johnson, ) dataset to synthesize the responses into a standard voice.

For the basic text instructions, we collect around 50K instructions from the Alpaca dataset(^7^77[https://huggingface.co/datasets/tatsu-lab/alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca)) (Taori et al., ), which covers a wide range of topics. Additionally, we gather around 150K instructions from the UltraChat dataset(^8^88[https://github.com/thunlp/UltraChat](https://github.com/thunlp/UltraChat)) (Ding et al., ), which primarily consist of questions about the world. Note that UltraChat is a large-scale multi-turn conversation dataset, but we only select the first 150K entries and use only the first-round instruction. Using the above datasets and data processing pipeline, we ultimately obtain 200K speech instruction data, referred to as InstructS2S-200K.

## 4 Experiments

### 4.1 Experimental Setups

#### Datasets

For the training data, we use the InstructS2S-200K dataset mentioned in Section 3, which includes 200K speech instruction data. To extract discrete units corresponding to the target speech, we use a pre-trained K-means quantizer(^9^99[https://dl.fbaipublicfiles.com/hubert/mhubert_base_vp_en_es_fr_it3_L11_km1000.bin](https://dl.fbaipublicfiles.com/hubert/mhubert_base_vp_en_es_fr_it3_L11_km1000.bin)), which has learned 1000 clusters from the HuBERT features. The pretrained HiFi-GAN vocoder (Kong et al., ; Polyak et al., ) is used to synthesize discrete units into waveform. For the evaluation data, we select two subsets from Alpaca-Eval(^10^1010[https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval)) (Li et al., ): helpful_base and vicuna, as their questions are more suitable for speech interaction scenarios. We remove questions related to math and code, resulting in a total of 199 instructions. To obtain the speech version, we use the CosyVoice-300M-SFT model to synthesize the instructions into speech. We refer to this test set as InstructS2S-Eval in the following sections.

#### Model Configuration

We use the encoder of Whisper-large-v3 as the speech encoder, and use Llama-3.1-8B-Instruct as the LLM. The speech adapter performs a 5$\times$ downsampling on the speech representations. The speech decoder consists of 2 Transformer layers with the same architecture as LLaMA, with a hidden dimension of 4096, 32 attention heads, and a feed-forward network dimension of 11008, which contains 425M parameters. The upsample factor $\lambda$ is set to 25. For the minimum unit chunk size $\Omega$ input to the vocoder, we set $\Omega=+\infty$ in the main experiment, meaning we wait for the entire unit sequence to be generated before inputting it to the vocoder for speech synthesis. In subsequent experiments, we will analyze how adjusting the value of $\Omega$ can control response latency, as well as the trade-off between latency and speech quality.

#### Training

LLaMA-Omni follows a two-stage training process. In the first stage, we train the speech adapter and the LLM with a batch size of 32 for 3 epochs. We use a cosine learning rate scheduler with the first 3% of steps for warmup, and the peak learning rate is set to 2e-5. In the second stage, we train the speech decoder, using the same batch size, number of steps, and learning rate scheduler as the first stage, but with the peak learning rate set to 2e-4. The entire training process takes approximately 65 hours on 4 NVIDIA L40 GPUs.

### 4.2 Evaluation

Since LLaMA-Omni can generate both text and speech responses based on speech instructions, we evaluate the model’s performance on two tasks: speech-to-text instruction-following (S2TIF) and speech-to-speech instruction-following (S2SIF). We use greedy search to ensure reproducible experimental results. The model is evaluated from the following aspects:

#### ChatGPT Score

To evaluate the model’s ability to follow speech instructions, we use GPT-4o (OpenAI, ) to score the model’s responses. For the S2TIF task, scoring is based on the transcribed text of the speech instructions and the model’s text response. For the S2SIF task, we first transcribe the model’s speech responses into text using the Whisper-large-v3 model, and then score it in the same manner as the S2TIF task. GPT-4o gives scores on two aspects: content and style. The content score evaluates whether the model’s response adequately addresses the user’s instruction, while the style score assesses whether the model’s response style is suitable for speech interaction scenarios. The detailed prompt can be found in Appendix A.

#### Speech-Text Alignment

To evaluate the alignment between text responses and speech responses, we use the Whisper-large-v3 model to transcribe the speech responses into text, and then calculate the Word Error Rate (WER) and Character Error Rate (CER) between the transcribed text and the text response. We refer to these metrics as ASR-WER and ASR-CER, respectively.

#### Speech Quality

To evaluate the quality of the generated speech, we utilize a Mean Opinion Score (MOS) prediction model called UTMOS(^11^1111[https://github.com/tarepan/SpeechMOS](https://github.com/tarepan/SpeechMOS)) (Saeki et al., ), which is capable of predicting the MOS score of the speech to assess its naturalness. We refer to this metric as the UTMOS score.

#### Response Latency

The latency is a key metric for speech interaction models, referring to the time interval between the input of a speech instruction and the start of the speech response, which has a significant impact on user experience. Additionally, we calculate the number of words already generated in the text response when the speech response begins, referred to as the #lagging word.

### 4.3 Baseline Systems

We include the following speech-language models as baseline systems:

#### SpeechGPT

SpeechGPT (Zhang et al., ) is a speech-language model that supports both speech input and output. We use the chain-of-modality prompting adopted in the original paper for decoding, which sequentially outputs the text instruction, text response, and speech response based on the speech instruction.

#### SALMONN (+TTS)

SALMONN (Tang et al., ) is a LLM capable of accepting speech and audio inputs and responding with text, enabling it to perform the S2TIF task. For the S2SIF task, we add a VITS TTS model after SALMOON to generate speech responses in a cascaded manner.

#### Qwen2-Audio (+TTS)

Qwen2-Audio (Chu et al., ) is a powerful general-purpose audio understanding model capable of performing various audio-related tasks, including the S2TIF task. We also build a cascaded system with Qwen2-Audio and VITS to complete the S2SIF task.

### 4.4 Main Results

**Table 1: ChatGPT scores for S2TIF and S2SIF tasks on the InstructS2S-Eval benchmark, along with the alignment scores between speech and text responses. Here we set $\Omega=+\infty$ for the S2SIF task.**
| Model | S2TIF | S2SIF | Alignment |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Content $\uparrow$ | Style $\uparrow$ | Content $\uparrow$ | Style $\uparrow$ | ASR-WER $\downarrow$ | ASR-CER $\downarrow$ |  |
| SpeechGPT | 2.59 | 3.15 | 1.58 | 1.81 | 47.62 | 37.26 |
| SALMONN (+TTS) | 2.57 | 2.79 | 2.46 | 2.84 | 21.77 | 20.25 |
| Qwen2-Audio (+TTS) | 2.73 | 2.64 | 2.32 | 2.58 | 55.72 | 53.09 |
| LLaMA-Omni | 3.23 | 3.81 | 2.69 | 3.12 | 11.61 | 7.59 |

Table 1 presents the main results on the InstructS2S-Eval benchmark. First, for the S2TIF task, from the content perspective, LLaMA-Omni shows significant improvement compared to previous models. This is mainly because LLaMA-Omni is developed based on the latest Llama-3.1-8B-Instruct model, leveraging its strong text instruction-following capabilities. From the style perspective, SALMONN and Qwen2-Audio receive lower scores, as they are speech-to-text models. Their output style is not aligned with speech interaction scenarios, often producing formatted content and containing a lot of redundant explanations. In contrast, SpeechGPT, as a speech-to-speech model, achieves a higher style score. Similarly, our LLaMA-Omni attains the highest style score, indicating that after being trained on our InstructS2S-200K dataset, the output style has been well-aligned with speech interaction scenarios. For the S2SIF task, LLaMA-Omni also outperforms previous models in both content and style scores. This further confirms that LLaMA-Omni is capable of effectively addressing user’s instructions with speech in a concise and efficient manner.

Additionally, in terms of alignment between speech and text responses, LLaMA-Omni achieves the lowest ASR-WER and ASR-CER scores. In contrast, SpeechGPT performs poorly in aligning speech and text responses, likely due to its sequential generation of text and speech. The speech-text alignment of cascaded systems, such as SALMONN+TTS and Qwen2-Audio+TTS, is also suboptimal, primarily because the generated text responses may contain characters that cannot be synthesized into speech. This issue is especially evident in Qwen2-Audio, which occasionally outputs Chinese characters, introducing errors in the speech responses. In comparison, LLaMA-Omni achieves the lowest ASR-WER and ASR-CER scores, demonstrating a higher degree of alignment between generated speech and text responses, and further validating the advantage of our approach in simultaneously generating both text and speech responses.

### 4.5 Trade-off between Speech Quality and Response Latency

**Table 2: Latency, speech-text alignment and speech quality under different unit chunk sizes.**
| Chunk Size $\Omega$ | Latency (ms) | #Lagging Word | ASR-WER $\downarrow$ | ASR-CER $\downarrow$ | UTMOS $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| 10 | 226.13 | 1.82 | 10.44 | 6.94 | 3.2304 |
| 20 | 256.28 | 2.83 | 10.51 | 6.98 | 3.4748 |
| 40 | 326.63 | 4.68 | 10.99 | 7.24 | 3.6688 |
| 60 | 391.96 | 6.47 | 11.17 | 7.27 | 3.7549 |
| 80 | 467.34 | 8.29 | 11.40 | 7.45 | 3.7858 |
| 100 | 527.64 | 10.02 | 11.48 | 7.45 | 3.8242 |
| $+\infty$ | 1924.62 | 41.40 | 11.61 | 7.59 | 3.9296 |

LLaMA-Omni can simultaneously generate both text responses and discrete units corresponding to the speech response. As described in Section 2.6, to further enable streaming waveform generation, when the number of generated discrete units reaches a certain chunk size $\Omega$, the unit chunk is fed into the vocoder to synthesize and play the speech. By adjusting the value of $\Omega$, we can control the system’s latency, where a smaller $\Omega$ corresponds to lower system latency. When $\Omega=+\infty$, it equates to waiting for all units to be generated before synthesizing the speech. At the same time, the value of $\Omega$ also affects the quality of the generated speech. A smaller $\Omega$ means that the speech is divided into more segments for synthesis, which may result in discontinuities between the segments, potentially reducing the overall coherence of the speech.

To better understand the impact of $\Omega$, we explore the system’s latency, the alignment between speech and text responses, and the quality of the generated speech under different $\Omega$ settings. As shown in Table 2, when $\Omega$ is set to 10, the system’s response latency is as low as 226ms, which is even lower than GPT-4o’s average audio latency of 320ms. At this point, the speech response lags by an average of 1.82 words at the start. When $\Omega$ is set to $+\infty$, the latency increases to around 2 seconds. For the ASR-WER and ASR-CER metrics, we are surprised to find that as the chunk size increases, the error rates also increase. We believe there may be two reasons for this. On one hand, the vocoder may handle short unit sequences more reliably than long ones, as it is typically trained on shorter sequences. On the other hand, the ASR model we use, Whisper-large-v3, has strong robustness. Even when the speech is somewhat discontinuous with smaller $\Omega$, it has little impact on ASR recognition accuracy. Therefore, we further evaluate the naturalness of the generated speech using the UTMOS metric. It shows that as $\Omega$ increases, the naturalness of the speech improves, since discontinuities in the speech decrease. In summary, we can adjust the value of $\Omega$ based on different scenarios to achieve a trade-off between response latency and speech quality.

### 4.6 Decoding Time

**Table 3: Average decoding time (s) of different models on S2TIF and S2SIF tasks.**
| Model | S2TIF | S2SIF |
| --- | --- | --- |
| SpeechGPT | 4.28 | 25.60 |
| SALMONN | 4.78 | / |
| Qwen2-Audio | 8.42 | / |
| LLaMA-Omni | 1.49 | 1.92 |

Table 3 lists the average decoding times of different models on the S2TIF and S2SIF tasks. For the S2TIF task, SpeechGPT needs to first output the text instruction and then the text response, while SALMONN and Qwen2-Audio tend to produce lengthy responses. In contrast, LLaMA-Omni provides concise answers directly, resulting in significantly lower decoding times, with an average of only 1.49 seconds per instruction. For the S2SIF task, SpeechGPT serially outputs the text and speech responses, resulting in a decoding time approximately 6 times longer than when generating only text responses. In contrast, LLaMA-Omni outputs both text and speech responses simultaneously and employs a non-autoregressive architecture for generating discrete units. As a result, the total generation time only increases by 1.28 times, demonstrating LLaMA-Omni’s advantage in decoding speed.

### 4.7 Case Study

**Table 4: Responses from different models for the instruction: “How do I wrap a present neatly?”**
| Model | Response |
| --- | --- |
| Qwen2-Audio | Wrapping a present neatly can be a bit tricky, but with some practice and patience, you can master the art of wrapping presents beautifully. Here are some steps to follow:\n\n1. Start with a clean, flat surface: Find a clean, flat surface where you can work on wrapping your present. A large table or countertop works well.\n\n2. Gather your supplies: You will need wrapping paper, gift wrap scissors, a ruler or measuring tape, and any other decorations you want to use (such as ribbon, bows, or stickers).\n\n3. Measure the gift: Use the ruler or measuring tape to measure the width and length of the gift. Make sure to add a little extra space around the edges for wrapping.\n\n4. Cut the wrapping paper: Cut the wrapping paper to the size you measured. If you have multiple pieces of wrapping paper, layer them together so that they fit neatly around the gift.\n\n5. Fold the paper: Begin by folding one of the sides of the wrapping paper over the gift, crease it well, and then fold the other side over the top. This creates a simple, yet attractive fold.\n\n6. Add decorations: If you like, you can add decorations such as ribbon, bows, or stickers to the gift. Simply attach them to the top of the gift using glue or double-sided tape.\n\n7. Wrap the gift: Starting at the bottom of the gift, place a piece of wrapping paper over it. Bring the left side of the paper up over the gift, tuck it in tightly, and press it down. Repeat this process on the right side of the gift, bringing the right side of the paper up over the gift, tucking it in tightly, and pressing it down.\n\n8. Finish the edges: Once you have wrapped the gift, trim off any excess paper along the edges using the gift wrap scissors. You can also add additional touches such as a bow or decorative label to complete the look.\n\nRemember, the key to wrapping a present neatly is to take your time and pay attention to detail. With practice, you’ll be able to create beautiful wrapping paper presentations every time! |
| SALMOON | To wrap a present neatly, you can start by laying out a clean and flat surface to work on. Then, place the present on the surface and wrap it in a piece of wrapping paper that is slightly larger than the present. Fold the paper in half and then in half again, so that the top and bottom flaps meet in the middle. Then, fold the left and right flaps in towards the center, so that they meet at the top and bottom of the present. Finally, seal the present by pressing the flaps together and using some tape to secure them. |
| SpeechGPT | Wrapping a present neatly requires a few simple steps. First, you should fold the paper in half and then unfold it. Then, you should place the present in the center of the paper and fold the paper around it. Finally, you should tie the ends of the paper with a ribbon or string. |
| LLaMA-Omni | To wrap a present neatly, start with a flat surface, place the gift in the center, and fold one edge of the wrapping paper over the gift, crease it, and tape it down. Repeat this process with the remaining edges, making sure to align them evenly and smooth out any air bubbles. |

To intuitively understand the differences in responses from different models, we provide an example in Table 4. It can be observed that the response of Qwen2-Audio are quite lengthy and include elements such as line breaks and parentheses that cannot be synthesized into speech. The response from SALMONN is also a bit long. The style of SpeechGPT’s response is more appropriate for speech interaction scenarios, but the amount of information contained in its responses is less. In contrast, the response given by LLaMA-Omni is more detailed and helpful while maintaining a concise style, outperforming previous models in speech interaction scenarios.

## 5 Related Work

#### Speech/Audio Language Models

With the success of language models in the field of natural language processing (Brown et al., ), researchers have begun exploring how to model speech or audio using language models. Early work attempted to train language models on semantic tokens or acoustic tokens of audio, enabling the generation of audio without the need for text (Lakhotia et al., ; Nguyen et al., ; Borsos et al., ).
Furthermore, by jointly training speech tokens and text, decoder-only models like VALL-E (Wang et al., ) and VioLA (Wang et al., ) can perform tasks such as speech recognition, speech translation, and speech synthesis. However, the above models are not built upon LLMs. To harness the power of LLMs, many studies explore how to build speech-language models based on LLMs like LLaMA, which can be further divided into two types. The first type, represented by SpeechGPT (Zhang et al., ; ) and AudioPaLM (Rubenstein et al., ), involves creating native multimodal speech-text models by adding speech tokens to the LLM’s vocabulary and continuing pretraining using speech and text data. However, this approach typically requires a large amount of data and substantial computational resources. The second type typically involves adding a speech encoder before the LLM and finetuning the entire model to equip it with speech understanding capabilities (Shu et al., ; Deshmukh et al., ), such as speech recognition (Fathullah et al., ; Yu et al., ; Ma et al., ; Hono et al., ), speech translation (Wu et al., ; Wang et al., ; Chen et al., ), or other general speech-to-text tasks (Chu et al., ; Tang et al., ; Chu et al., ; Fathullah et al., ; Das et al., ; Hu et al., ). However, these approaches typically focus only on speech or audio understanding without the ability to generate them. Compared to previous work, LLaMA-Omni equips the LLM with both speech understanding and generation capabilities, enabling it to perform general speech instruction-following tasks. Additionally, LLaMA-Omni has a low training cost, making it convenient for development based on the latest LLMs.

#### Simultaneous Generation

Streaming generation aims to begin producing output before the entire input is received.
This capability is crucial for maintaining synchronization between speakers and listeners in various scenarios, such as streaming speech recognition and simultaneous interpretation.
In the case of large language models, having a streaming speech synthesis component can significantly reduce latency between the model and its users.
Popular streaming generation methods fall into three main categories: monotonic-attention-based methods (Raffel et al., ), CTC-based methods (Graves et al., ), and Transducer-based methods (Graves, ).
Monotonic-attention-based methods modify the traditional attention-based sequence-to-sequence framework (Bahdanau, ) to support streaming generation.
These methods rely on an external module to manage the READ/WRITE policy, which can be either fixed (e.g., Wait-k (Ma et al., )) or adaptive (e.g., MMA (Ma et al., ), EDAtt (Papi et al., ), Seg2Seg (Zhang & Feng, )).
CTC-based methods add a blank symbol to the target vocabulary to represent a WAIT action.
Streaming inference is achieved by removing adjacent repetitive tokens and blank symbols.
To leverage the strengths of attention-based methods, CTC-based approaches often use a chunk-based non-autoregressive architecture (Ma et al., ), which has proven effective in simultaneous interpretation and streaming speech synthesis (Zhang et al., ; Ma et al., ).
Transducer-based methods are designed to bridge the gap between the non-autoregressive nature of CTC-based methods and the autoregressive dependency between target tokens.
These approaches introduce an additional predictor to capture token dependencies, and their variants have shown strong performance in simultaneous interpretation (Liu et al., ; Tang et al., ).
More recently, researchers have begun adopting decoder-only large language models for streaming generation tasks (Seide et al., ; Guo et al., ), and extending them to an interruptible duplex model (Ma et al., ).

## 6 Conclusion

In this paper, we propose an innovative model architecture, LLaMA-Omni, which enables low-latency and high-quality speech interaction with LLMs. LLaMA-Omni is built upon the latest Llama-3.1-8B-Instruct model, with the addition of a speech encoder for speech understanding and a streaming speech decoder that can generate both text and speech responses simultaneously. To align the model with speech interaction scenarios, we construct a speech instruction dataset InstructionS2S-200K, which contains 200K speech instructions along with the speech responses. Experimental results show that, compared to previous speech-language models, LLaMA-Omni delivers superior responses in both content and style, with a response latency as low as 226ms. Moreover, training LLaMA-Omni requires less than 3 days on 4 GPUs, enabling rapid development of speech interaction models based on the latest LLMs. In the future, we plan to explore enhancing the expressiveness of generated speech responses and improving real-time interaction capabilities.

## Appendix A Prompt

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: please resolve the concerns in Weaknesses
soundness: 2
strengths: This is probably the first public work that connects a speech-to-text Speech-LLM with a streaming text-to-speech model so as to enable low latency speech outputs.
summary: This paper proposed a Speech-LLM model that can do speech-to-speech conversation with good latency and quality. The central design enables the low latency feature is the proposed streaming speech decoder.
weaknesses: 1. The speech-to-text component of this model is not new (relevant works like SALMONN and Qwen2-audio) and the main addition is the text-to-speech component. The latter has a clear connection with the streaming TTS model research but the paper does not spend enough content to acknowledge and compare to this line of research. To elaborate on this, the paper should compare the proposed method with prior research in this space and motivate this unique CTC based design. The paper should also replace the baseline of SALMONN+TTS with SALMONN+streaming TTS so as to have a fair comparison and show the value of the proposed component. 
2. The "response latency" used in this paper is misleading. It only considers the latency from streaming speech decoder, but in fact the non-streaming speech-to-text Speech-LLM also has latency which should be considered as "response latency". The paper fails to point this out and didn't give the overview of the latency breakdown given the proposed design 
3. The audio quality of the proposed method is not good from the MOS score and is not compared with any external works, which challenges the value of this proposed CTC based design, especially considering there have been prior works in the streaming TTS area. It's hard to understand whether the proposed method is better or worse.
4. "On the other hand, the ASR model we use, Whisper-large-v3, has strong robustness. Even when the speech is somewhat discontinuous with smaller Ω," If this ASR model cannot identify the discontinuity and relevant quality issue, it may not serve well as a way to evaluate the intelligence of the generated speech. This is a limitation.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: See above.
soundness: 3
strengths: 1. **Conversational Data with InstructS2S-200K:** The proposed InstructS2S-200K dataset addresses a notable gap in conversational, oral-style speech instruction data. By providing a large, purpose-built dataset tailored to natural, interactive speech patterns, this work supports more effective alignment of LLM responses with human conversational norms. 
2. **Cost-Effective Training Process:** The two-stage training process adopted in LLaMA-Omni demonstrates a practical approach to reducing training costs. Compared to prior end-to-end training methods, this staged training setup optimizes resource use, enabling faster model development without compromising performance.
summary: This paper proposes LLaMA-Omni, an end-to-end model architecture designed for low-latency, high-quality speech interaction with large language models (LLMs). LLaMA-Omni integrates multiple components: a pre-trained speech encoder, a speech adaptor, a large language model, and a streaming speech decoder. To enable more natural, human-like speech interactions, the authors introduce InstructS2S-200K, a dataset of 200K speech instructions and responses crafted in a conversational, oral style, to fine-tune the model for speech-based dialogue. Experimental results show that LLaMA-Omni achieves low-latency speech responses while reducing training costs compared to similar models.
weaknesses: 1. **Lack of Novelty:** While the authors suggest that they proposed a "novel model architecture," it appears to primarily build on a combination of existing methods without a clear breakthrough in architecture: 
   1) The connection of a speech encoder to an LLM via a speech adaptor has been widely explored in prior research. 
   2) The streaming TTS module seems to be directly based on previous designs, specifically following the approach outlined in [1]. 
   
    In summary, while LLaMA-Omni successfully assembles a functional pipeline for speech-based interactions with LLMs, the architecture seems to build on prior work without delivering significant innovation in model design, which somewhat limits the paper’s impact from a research innovation standpoint. Further discussion is needed to emphasize the unique contribution of this work.
2. **Questionable Experiment Results:** Some experiment results in this paper is questionable:
   1) **ChatGPT-Score in Table 2:** While LLaMA-Omni shows improved performance over baseline models in ChatGPT-Score, this result is potentially misleading. The evaluation prompt used (detailed in Appendix A) was crafted to reward concise responses and penalize extraneous information. This prompt design aligns closely with the prompts used to construct the InstructS2S-200K dataset, creating a circular evaluation condition where the model is likely to excel by design. A more objective and meaningful approach would involve conducting a human evaluation to capture human preferences more accurately. 
   2) **WER & CER in Table 2:** In Table 2, Qwen2-Audio + TTS shows an unexpectedly high WER of 55.72%, yet it achieves a relatively moderate S2SIF score of 2.32/2.58, which implies a more acceptable response quality. This discrepancy suggests an inconsistency, as such a high WER would typically indicate a low intelligibility level that should correlate with a poor S2SIF score. This contradiction casts doubt on the reliability of the WER measurements or, potentially, the effectiveness of the S2SIF scoring methodology when used alongside TTS outputs.
    
        A likely explanation for these irregularities lies in the TTS model used, VITS, which is primarily trained on **short audio clips** [2]. This limitation may hinder VITS’s ability to process longer inputs effectively, causing high error rates when it encounters the longer outputs of models like Qwen2-Audio and SALMONN. Sending unchunked, lengthy responses directly to VITS without segmentation could result in degraded performance, as evidenced by the elevated WER and CER scores. 

        To address these limitations and obtain more accurate metrics, it would be more appropriate to adopt a TTS model trained for longer utterances or to chunk input text before passing it to VITS. Such adjustments would likely yield more realistic WER and CER values and provide a fairer basis for comparing baseline models.
    3) **Average decoding time in Table 4:** The decoding time reported in Table 4 does not convincingly demonstrate the superiority of the proposed streaming decoding architecture. The average decoding time is measured for generating complete speech responses; however, since baseline methods produce longer responses, they naturally require more time for decoding. This dependency on response length makes the reported decoding times trivial and limits their effectiveness in showcasing any advantages of the proposed approach. To provide a fair and meaningful comparison, it would be more appropriate to compare with Qwen2-Audio + streaming TTS model and report decoding time per token rather than total decoding time. 
3. **Limited speaker support:** While InstructS2S-200K provides valuable conversational data, it has limitations due to the lack of speaker diversity. The input speech in this dataset is synthesized using CosyVoice-300M-SFT, which represents a single voice and does not capture the variability in accents, pitch, tone, and other speaker characteristics found in real-world interactions. This limitation restricts LLaMA-Omni’s adaptability to diverse user-profiles and reduces its effectiveness in applications requiring personalized or varied speaker responses. Integrating a broader range of voices could significantly improve LLaMA-Omni’s applicability in real-world scenarios.


[1] Fang Q, Ma Z, Zhou Y, et al. CTC-based Non-autoregressive Textless Speech-to-Speech Translation[J]. arXiv preprint arXiv:2406.07330, 2024.

[2] Kim J, Kong J, Son J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech[C]//International Conference on Machine Learning. PMLR, 2021: 5530-5540.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Could the author please emphasize the contribution of this article?
   - If it is a streaming TTS module, there is a lack of comparison with other streaming generation methods (autoregressive TTS models)
   - if it is the speech instruction dataset, there is a lack of human evaluation to prove its greater alignment with human preference, and the limitations of TTS synthesized speech input also restrict its application
   - if it is an open-source seamless spoken dialogue model, evaluation on real speech benchmarks is also needed.
soundness: 3
strengths: 1. The paper is well-written and presents a promising approach to seamless speech interaction with LLMs. The reported response latency of 226ms is impressive and demonstrates the value for real-world applications.

2. The authors have made a valuable contribution by creating the InstructS2S-200K dataset, which is tailored to speech interaction scenarios. The core motivation for building this dataset is that "in speech interactions, concise yet informative responses are typically preferred", which has not been considered in previous work.
summary: This work presents an end-to-end spoken dialogue model LLAMA-Omini, which integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming TTS module. The model can directly understand continuous speech features and can simultaneously generate text and speech responses directly with low latency. To achieve alignment between different modules, this work also proposes a pipeline for constructing speech-to-speech dialogue data, with a style that better matches the characteristics of speech interaction scenarios.
weaknesses: 1. Although this work presents a seamless spoken dialogue model, the choice of connecting a LLM with a CTC-based streaming TTS module is not well-motivated.
    - Autoregressive TTS models naturally support streaming decoding, but this paper does not discuss or compare with the traditional autoregressive decoding.

    - The CTC-based streaming TTS module incorporates ideas from StreamSpeech [1] and achieves streaming output by segmenting generated units into chunks. However, it is not evident what benefits the end-to-end model offers over a straightforward cascaded solution where the text response is segmented into chunks and then fed into an offline TTS model.

2. Another contribution of this work is the constructed dataset InstructS2S-200K, but there is a lack of sufficient experiments to demonstrate the advantages of this dataset.

   - First of all, the speech input in this dataset is synthesized using CosyVoice-300M-SFT, which can only synthesize standard speech from limited speakers. This may lead to the model only being able to handle tts-synthesized input speech, lacking speaker generalization. Therefore, in order to demonstrate the practical value of this dataset, evaluation on human recorded speech is necessary. For example, ASR and ST performance in academic dataset (LibriSpeech, CoVoST), or evaluation on general benchmarks (such as AIRBench [2]) is necessary to demonstrate the model's understanding of real speech. But this work is only evaluated on their TTS-synthesized InstructS2S-Eval.

   - One advantage of this dataset is that the response is accurate and concise, which is more suitable for the speech interaction scenario. However, in order to prove this statement, human evaluation is very necessary to demonstrate that humans would prefer shorter replies. In Table 1, the author uses GPT4 as a judge, but in Appendix A, we can see that the author has incorporated the need for short and accurate responses into the GPT4 evaluation prompt. This can only prove that the model's responses are accurate and concise, but cannot prove that they are more in line with human preferences.

3. There are some unconvincing evaluation metrics and results analysis.

   - In Table 1, a biased prior has been introduced in the evaluation prompt of GPTscore (see 2.b), therefore the content and style scores are not reasonable.

   - In Table 1, the pipeline models SALMOON/Qwen2-Audio + TTS have very poor alignment scores. Although the authors explained in Line 392-393 that the responses of these two models contain characters that cannot be synthesized into speech, we can see in the case study in Table 5 that SALMMON does not have any unsynthesizable characters, and Qwen2-Audio only has \n that is unsynthesizable. I speculate that the very poor alignment score here is due to the fact that the TTS model used only supports synthesizing speech for around 10 seconds, while SALMOON and Qwen2-Audio generate very long responses that exceed the maximum supported length of the TTS model. I believe the authors need to provide a detailed description of how they used the TTS model, or to segment long text responses into shorter chunks to fed into TTS model to ensure fairness in comparison.

   - Table 4 reports the total time for completing a single response, and it is quite apparent that shorter responses result in shorter completion times. However, this does not provide any new insights or perspectives to demonstrate the advantages of this work.

4. The author emphasizes that LLAMA-Omini is an interactive dialogue model, or a GPT-4o-like model, and therefore needs to demonstrate its ability for multi-turn conversations. However, evaluations have only been conducted on single-turn responses. I think that quantitative metrics or qualitative demos are necessary.

[1] Zhang, Shaolei, et al. StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning. ACL 2024.

[2] Yang, Qian, et al. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. arxiv 2402.07729.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - Is there a risk of audio artifacts, such as "popping", at the boundaries when using the vocoder to generate speech chunks? If so, how does the model mitigate these issues?

- Could you provide more detailed comparisons between LLaMA-Omni and other speech-interactive LLMs to evaluate whether introducing speech as an input/output modality affects the model’s reasoning and language comprehension abilities? Specifically, does enabling speech interaction in a text-based LLM reduce its core language understanding and intelligence? Personally, I evaluated another GPT-4o-like model, mini-Omni, and found that their method to introduce the speech modality significantly reduced the language model's core intelligence. Although the authors repeatedly claimed in the paper that their method could "retain the original model’s language capabilities with minimal degradation," they provided no evaluation to substantiate this claim.
soundness: 4
strengths: - LLama-Omni carries out a novel non-autoregressive architecture for simultaneous text and speech generation, providing a fresh approach among recent GPT-4o-like models. It achieves a sufficiently low latency of 226ms and high-quality speech interaction, making it suitable for real-time scenarios.

- LLaMA-Omni can be integrated with any open-source LLM at minimal training cost, requiring only 4 GPUs and less than 3 days to train. This makes it accessible for academic research, allowing more institutions and researchers to use high-quality, speech-interactive LLMs.

- One notable strength is that LLaMA-Omni largely preserves the intelligence of pre-trained LLMs.
summary: This paper introduces LLama-Omni, a novel end-to-end architecture that enables real-time interaction with LLMs through speech. Llama-Omni has a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. Speech is fed continuously into the LLM, while text is generated discretely. Then, each text unit is replicated a fixed number of times and aligned with HuBERT k-means units using non-autoregressive CTC. Once the units accumulate to a preset number, they form a chunk and feed to a vocoder for speech reconstruction. This non-autoregressive architecture supports simultaneous text and speech generation from spoken instructions with very low latency, while achieving very low training overhead.
weaknesses: - One limitation is that the LLM outputs text only. While this preserves its reasoning abilities, it losses richer features such as emotional nuance and speaker identity.

- As the authors mentioned in Section 4.4, the cascaded use of VITS TTS in SALMONN and Qwen2-Audio may introduce errors affecting S2SIF and alignment due to propagated TTS inaccuracies, making this comparison less of an apple-to-apple evaluation.

Meta-review: This paper proposed LLaMA-Omni, a model architecture designed for low-latency speech interaction with LLM which is a very important research area  with the representative works such as GPT-4o and Moshi. The proposed architecture integrates a pretrained speech encoder, an adaptor, an LLM, and a streaming speech decoder.  With such a structure, the model can simultaneously generates text and speech responses directly from speech instructions with the latency as low as 236ms. To align the model with speech interaction scenarios, the authors also constructed a dataset with 200K speech instructions and corresponding speech responses. Experiment results showed that LLaMA-Omni is superior than previous speech LLMs. 

The paper presents extensive experiments in offline and streaming scenarios. The proposed work stands out due to its low-latency streaming processing, unlike most speech LLMs. In their rebuttal, the authors used an industrial TTS model to strengthen the baseline for cascaded systems, making the LLaMA-Omni's improvements more convincing. The initial submission has the issues of calculating ASR-WER metrics and using ChatGPT scoring which were pointed out by reviewers. These were all fixed during the rebuttal. Furthermore, the authors have conducted human evaluation to compare different models. 

Overall, after revision, the paper is much stronger than the initial submission. Streaming speech-LLM with low latency is an area with very limited works at this moment. The proposed work is a good plus to the field.

In summary, this paper's strength lies in its focus on streaming speech LLM scenarios. The proposed method demonstrates superior performance compared to the cascaded system in low-latency situations, which is highly beneficial for spoken chat applications. However, a noted weakness is that the proposed method does not consistently outperform the cascaded system across all configurations, with occasional instances where the cascaded baseline exhibits an unusually high word error rate.

Decision: Accept (Poster)